Score:0

Kubernetes network polices are not enforced unless the network-plugin daemon-sets are restarted. Why?

br flag

I have only one network policy in my cluster in prod namespace that allows only ingress rules. The network plugin is weave-net. No rules are configured for Egress so I am expecting egress traffic will be blocked. But until I restart the network daemon-set pods the rule has no effect. I know by best practices I should have default ingress and egress rules. But I want to understand the reason of this behavior. Is this step always required to restart the network-plugin pods?

1. Network Policy Definition

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-network-policy
  namespace: prod
spec:
  ingress:
  - {}
  podSelector:
    matchLabels:
      run: prod-nginx
  policyTypes:
  - Ingress
  - Egress

2. Checking the netpol object

Name:         test-network-policy
Namespace:    prod
Created on:   2021-06-06 10:16:50 +0000 UTC
Labels:       <none>
Annotations:  <none>
Spec:
  PodSelector:     run=prod-nginx
  Allowing ingress traffic:
    To Port: <any> (traffic allowed to all ports)
    From: <any> (traffic not restricted by source)
  Allowing egress traffic:
    <none> (Selected pods are isolated for egress connectivity)
  Policy Types: Ingress, Egress

3. Testing egress traffic to nginx server (This is unexpected to my understanding)

Note: 10.39.0.5 is the IP of the nginx server running in 'test' namespace

Command : kubectl -n prod exec -it prod-nginx -- curl http://10.39.0.5 | grep successfully #egress
Response: <p>If you see this page, the nginx web server is successfully installed and

4. Restarted the weave-net pods

5. Retesting egress connection to same nginx server (expected)

Note: 10.39.0.5 is the IP of the nginx server running in 'test' namespace

Command: kubectl -n prod exec -it prod-nginx -- curl http://10.39.0.5 | grep successfully #egress**
Response: No connection
matt_j avatar
in flag
It shouldn't be necessary to restart the `weave-net` Pods. Which Pod has the IP address `10.39.0.5`, can you describe it ?
Rajesh Dutta avatar
br flag
The IP 10.39.0.5 belongs to a nginx server which is running in a different namespace ('test')
Rajesh Dutta avatar
br flag
I am sorry for not mentioning this. I have updated the question adding a Note about the IP. Thanks for your effort.
Score:0
in flag

I would like to show you that restarting the weave-net Pods isn't required for NetworkPolicy to take effect.

Your test-network-policy NetworkPolicy is applied to Pods with the label run=prod-nginx in the prod Namespace and allows all ingress traffic and denies all egress traffic.

I will create an example to illustrate how it works.


First, I created the prod-nginx & prod-test Pods and tested the connectivity with no NetworkPolicy deployed:

# kubectl run prod-nginx --image=nginx -n prod
pod/prod-nginx created

# kubectl run prod-test --image=nginx -n prod
pod/prod-test created
    
# kubectl get pod -o wide -n prod
NAME         READY   STATUS    RESTARTS   AGE   IP          LABELS
prod-nginx   1/1     Running   0          37s   10.44.0.1   run=prod-nginx
prod-test    1/1     Running   0          11s   10.44.0.2   run=prod-test

# kubectl exec -it prod-nginx -n prod -- curl 10.44.0.1 | grep -i success
<p>If you see this page, the nginx web server is successfully installed and

# kubectl exec -it prod-nginx -n prod -- curl 10.44.0.2 | grep -i success
<p>If you see this page, the nginx web server is successfully installed and

Everything works fine, so let's deploy a test-network-policy NetworkPolicy and test again:

# cat netpol.yml 
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-network-policy
  namespace: prod
spec:
  ingress:
  - {}
  podSelector:
    matchLabels:
      run: prod-nginx
  policyTypes:
  - Ingress
  - Egress
  
# kubectl apply -f netpol.yml 
networkpolicy.networking.k8s.io/test-network-policy created

We can see that the prod-nginx Pod can't connect to other Pods but can connect to itself:
NOTE: A pod cannot block access to itself (see: Network Policies documentation)

# kubectl exec -it prod-nginx -n prod -- curl 10.44.0.1 | grep -i success
<p>If you see this page, the nginx web server is successfully installed and
   
# kubectl exec -it prod-nginx -n prod -- curl 10.44.0.2 | grep -i success
command terminated with exit code 7

Now let's create a stage-nginx Pod in the stage namespace and check if the prod-nginx Pod can connect to it:

# kubectl run stage-nginx --image=nginx -n stage
pod/stage-nginx created

# kubectl get pod -o wide -n stage
NAME          READY   STATUS    RESTARTS   AGE   IP          
stage-nginx   1/1     Running   0          20s   10.44.0.6   

# kubectl exec -it prod-nginx -n prod -- curl 10.44.0.6 | grep -i success
command terminated with exit code 7

We have verified that the egress rule is working properly and restarting the weave-net Pods is not required.

Rajesh Dutta avatar
br flag
Thanks for the effort. I have updated the question with a Note about the IP.
Rajesh Dutta avatar
br flag
Thanks for the effort. I have updated the question with a Note about the IP. Now regarding the problem. Please try connecting a server running in a different namespace. In my case I am still seeing the problem. You can try multiple times enabling and disabling egress and see if each time the network policy is behaving as expected without restarting the weave-net servers. Actually the problem also exists with Ingress but I chose to explain with egress. and so far what I observed, this is happening only when you specify a policy type(egress in our use case) with no rules.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.