nslookup weird behavior in 2 pods on the same node in kubernetes

I have 2 containers running in the same namespace.

• in-pod dns of the pod xwing does not resolving host name but will resolve FQDN
• but in-pod dns of the pod utils resolve it properly.

✗  kgpo  
NAME                         READY   STATUS    RESTARTS   AGE
deathstar-7848d6c4d5-g248j   1/1     Running   0          15m
deathstar-7848d6c4d5-qnsp8   1/1     Running   0          15m
tiefighter                   1/1     Running   0          15m
utils                        1/1     Running   0          11m
xwing                        1/1     Running   0          15m

➜ kgsvc         
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
deathstar    ClusterIP   172.20.41.112   <none>        80/TCP    15m
kubernetes   ClusterIP   172.20.0.1      <none>        443/TCP   46d

➜ k ice image               
PODNAME                     CONTAINER  PULL    IMAGE                          TAG
deathstar-7848d6c4d5-g248j  deathstar  Always  docker.io/cilium/starwars      -
deathstar-7848d6c4d5-qnsp8  deathstar  Always  docker.io/cilium/starwars      -
tiefighter                  spaceship  Always  docker.io/tgraf/netperf        -
utils                       utils      Always  saritasallc/network-multitool  -
xwing                       spaceship  Always  docker.io/tgraf/netperf        -

the following pods:

• xwing (docker.io/tgraf/netperf) and
• utils (saritasallc/network-multitool)

pods show the same /etc/resolv.conf values, yet they behave differently for nslookup query.

➜ kex xwing -- cat /etc/resolv.conf  
search default.svc.cluster.local svc.cluster.local cluster.local ec2.internal
nameserver 172.20.0.10
options ndots:5

➜ kex utils -- cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local ec2.internal
nameserver 172.20.0.10
options ndots:5

and utils pod can query service name deathstar but xwing pod can't. Only by FQDN of the service:

➜ kex utils -- nslookup deathstar  
Server:     172.20.0.10
Address:    172.20.0.10#53

Name:   deathstar.default.svc.cluster.local
Address: 172.20.41.112

➜ kex xwing -- nslookup deathstar
nslookup: can't resolve '(null)': Name does not resolve

➜ kex xwing -- nslookup deathstar.default.svc.cluster.local
nslookup: can't resolve '(null)': Name does not resolve
nslookup: can't resolve 'deathstar': Name does not resolve
command terminated with exit code 1

Name:      deathstar.default.svc.cluster.local
Address 1: 172.20.41.112 deathstar.default.svc.cluster.loca

both pods are running on the same eks node.

After enabling logs in coredns I see this:

// kex utils -- nslookup deathstar (success)
[INFO] 10.120.133.213:45918 - 30670 "A IN deathstar.default.svc.cluster.local. udp 53 false 512" NOERROR qr,aa,rd 104 0.000298996s
[INFO] 10.120.133.213:45293 - 18650 "AAAA IN deathstar.default.svc.cluster.local. udp 53 false 512" NOERROR qr,aa,rd 146 0.000405998s

// kex xwing -- nslookup deathstar (failure)
[INFO] 10.120.134.84:32840 - 45185 "AAAA IN deathstar. udp 27 false 512" NXDOMAIN qr,rd,ra 102 0.003295317s
[INFO] 10.120.134.84:32840 - 44875 "A IN deathstar. udp 27 false 512" NXDOMAIN qr,rd,ra 102 0.003334905s