k8s/AzureAKS

Links
2024 AKS CNI + Cilium
SP - Service principal - used by the k8s cluster
- NOTE: $/!\$ only valid for 1 year - rotate credentials
  - Saved on azure node(vm) at /etc/kubernetes/azure.json
  - If created with az aks create, saved on laptop at ~/.azure/aksServicePrincipal.json
  - got to delete SP, on cluster deletion, not done for you. query for your clusters servicePrincipalProfile.clientId and then delete it using the az ad sp delete
- Give it roles/access e.g.
  - Access to Azure Container registry to pull images
  - Access storage in different RG
  - Access IP in different RG

Create AKS

RG_NAME=myResourceGroup-NP
CLUSTER_NAME=myAKSCluster
LOCATION=canadaeast

Create the AKS cluster and specify *azure* for the network plugin and network policy.

```azurecli
az aks create \
    --resource-group $RG_NAME \
    --name $CLUSTER_NAME \
    --node-count 1 \
    --network-plugin azure \
    --network-policy azure

AKS RBAC

Assign role to AD group

AKS_ID=$(az aks show \
    --resource-group myRGroup \
    --name myAKSCluster \
    --query id -o tsv)

APPDEV_ID=$(az ad group create \
    --display-name appdev \
    --mail-nickname appdev \
    --query objectId -o tsv)

az role assignment create \
  --assignee $APPDEV_ID \
  --role "Azure Kubernetes Service Cluster User Role" \
  --scope $AKS_ID

AKS Autoscaler and NodePools

GitHub - https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/expander/priority/readme.md
Explained - https://www.thorsten-hans.com/aks-cluster-auto-scaler-inside-out/#why-should-you-use-cluster-auto-scaler
Azure doc - https://learn.microsoft.com/en-us/azure/aks/cluster-autoscaler#using-the-autoscaler-profile
e.g. ConfigMap - https://docs.giantswarm.io/advanced/spot-instances/azure/ondemand-fallback/

Use priorities to pick preferd pool order, From https://github.com/Azure/AKS/issues/2359

az aks update --subscription xxx --resource-group xxx --name xxx --cluster-autoscaler-profile expander=priority

AKS spot node pool

k8s node run with Taints: kubernetes.azure.com/scalesetpriority=spot:NoSchedule