I'm trying to configure a Load Balancer for my AKS server using Bicep/ARM. I am using the NGinx Ingress Controller in kubernetes and it does seem to work but when I first spin things up I am encountering an error.
Mainly I'm wondering what is the equivalent ARM or Bicep template for this step in the Azure documentation?
https://docs.microsoft.com/en-us/azure/aks/static-ip#create-a-service-using-the-static-ip-address
az role assignment create \
--assignee <Client ID> \
--role "Network Contributor" \
--scope /subscriptions/<subscription id>/resourceGroups/<resource group name>
I'm using Bicep and have created my AKS server like this for example:
resource ExampleKubernetes 'Microsoft.ContainerService/managedClusters@2021-07-01' = {
// ...
}
I'm then adding a role assignment to the kubelet identity like so:
var NetworkContibutor = '4d97b98b-1d4f-4787-a291-c67834d212e7'
resource AssignNetworkContributorToKubelet 'Microsoft.Authorization/roleAssignments@2020-08-01-preview' = {
name: guid(resourceGroup().id, ExampleKubernetes.id, NetworkContibutor)
dependsOn: [
ExampleKubernetes
]
properties: {
roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', NetworkContibutor)
principalType: 'ServicePrincipal'
principalId: ExampleKubernetes.properties.identityProfile.kubeletidentity.objectId
}
}
Which seems to work, I can see the Role assigned to the managed principal in the dashboard... but the Service in kubernetes seems to fail with a permission issue still:
Error syncing load balancer: failed to ensure load balancer: Retriable: false,
RetryAfter: 0s, HTTPStatusCode: 403, RawError: Retriable: false, RetryAfter:
0s, HTTPStatusCode: 403, RawError:
{"error":{"code":"AuthorizationFailed","message":"The client
'<some guid A>' with object id
'<some buid A>' does not have authorization to perform
action 'Microsoft.Network/publicIPAddresses/read' over scope
'/subscriptions/<subid>/resourceGroups/example/providers/Microsoft.Network/publicIPAddresses/example'
or the scope is invalid. If access was recently granted, please refresh your
credentials."}}
What's weird is that later on at some point it seems to just magically work. That error says "retriable false" and it does seem like the service doesn't retry but a subsequent deploy of NGinx to kubernetes will then cause it to retry and suddenly boom its working.
It just seems like the error message is telling me there is some non-deterministic delay of role propagation... So my questions are:
- Is that right? Is it in fact just a delay and my code is basically right?
- Am I using the right principalId? Or is that actually unnecessary?
- Is there a way for me to force those role updates to propagate? I could have a CLI step in between if I needed to. How can I wait to install my ingress controller which connects to the LB after the permissions are ready?