1

My microk8s cluster is running on a centos8 vm and I have some issues with DNS resolution of my pods. Nameserver are at x.x.x.101 and x.x.x.100 both are ping able from within the pods and I can also ping 8.8.8.8

nslookup from within a pod looks like this:

root@debug-7857894f66-mnklp:/# nslookup kubernetes.default Server: 10.152.183.10 Address: 10.152.183.10#53 Name: kubernetes.default.svc.cluster.local Address: 10.152.183.1

coredns config looks like this:

apiVersion: v1 data: Corefile: ".:53 {\n errors\n health {\n lameduck 5s\n }\n ready\n \ log . {\n class error\n }\n kubernetes cluster.local in-addr.arpa ip6.arpa {\n pods insecure\n fallthrough in-addr.arpa ip6.arpa\n }\n \ prometheus :9153\n forward . x.x.x.101 x.x.x.100 \n cache 30\n \ loop\n reload\n loadbalance\n}\n" kind: ConfigMap metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","data":{"Corefile":".:53 {\n errors\n health {\n lameduck 5s\n }\n ready\n log . {\n class error\n }\n kubernetes cluster.local in-addr.arpa ip6.arpa {\n pods insecure\n fallthrough in-addr.arpa ip6.arpa\n }\n prometheus :9153\n forward . x.x.x.101 x.x.x.100 \n cache 30\n loop\n reload\n loadbalance\n}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"EnsureExists","k8s-app":"kube-dns"},"name":"coredns","namespace":"kube-system"}} creationTimestamp: "2021-08-31T08:57:27Z" labels: addonmanager.kubernetes.io/mode: EnsureExists k8s-app: kube-dns name: coredns namespace: kube-system resourceVersion: "2420090" selfLink: /api/v1/namespaces/kube-system/configmaps/coredns uid: 471b258a-253d-4b51-aaf7-7e934ab300d1

the /etc/resolv.conf in my pods looks like this:

search default.svc.cluster.local svc.cluster.local cluster.local xxx.xxxxx nameserver 10.152.183.10 options ndots:5

when I look at the logs of my kube-dns with $ microk8s kubectl logs --namespace=kube-system -l k8s-app=kube-dns I get the following response:

[INFO] 10.1.107.105:47549 - 5288 "AAAA IN www.google.com. udp 36 false 512" NOERROR - 0 0.000256103s [ERROR] plugin/errors: 2 www.google.com. AAAA: read udp 10.1.107.127:51486->x.x.x.101:53: read: no route to host

DNS service is up:

$ microk8s kubectl get svc --namespace=kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE [...] kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP,9153/TCP 21d

DNS endpoints are exposed:

$ microk8s kubectl get endpoints kube-dns --namespace=kube-system NAME ENDPOINTS AGE kube-dns 10.1.107.127:53,10.1.107.127:53,10.1.107.127:9153 21d

The virtual machine the cluster is running on is using the same nameserver without a problem. Is this something I need to bother our admins about, or am I missing something that is wrongly configured?

EDIT:

Okay I tried out a bit more and there is some weird behavior that I think has to do with me incorrectly creating the coredns config file. I completely reset the cluster and tried again, still same error when I use 8.8.8.8 or 8.8.4.4 as DNS servers. But, when I run the following command: $ microk8s enable dns:x.x.x.101,x.x.x.100 finally it works. Then I tried to configure for this with $ microk8s kubectl -n kube-system edit configmap/coredns and I add both DNS to the config it stops working, and I again get the error: [ERROR] plugin/errors: 2 www.google.com. AAAA: read udp 10.1.107.127:51486->x.x.x.101:53: read: no route to host so what is wrong with my configuration that is automatically correctly set with $ microk8s enable dns:x.x.x.101,x.x.x.100 ?

2nd Edit

I tried to use dig here even when I specify a DNS server it is not working. Is there any reason ping could go through but DNS is blocked by the system? It only happens on the microk8s cluster, host system runs fin docker runs fine ... here are the printouts: this is from within the pod:

root@debug-865cb7fb4-wfhw4:/# dig www.google.com ; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> www.google.com ;; global options: +cmd ;; connection timed out; no servers could be reached root@debug-865cb7fb4-wfhw4:/# dig @x.x.x.101 www.google.com ; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @x.x.x.101 www.google.com ; (1 server found) ;; global options: +cmd ;; connection timed out; no servers could be reached root@debug-865cb7fb4-wfhw4:/# dig @8.8.8.8 www.google.com ; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @8.8.8.8 www.google.com ; (1 server found) ;; global options: +cmd ;; connection timed out; no servers could be reached root@debug-865cb7fb4-wfhw4:/# dig @x.x.x.100 www.google.com ; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @x.x.x.100 www.google.com ; (1 server found) ;; global options: +cmd ;; connection timed out; no servers could be reached

and this is from the host system:

$ dig www.google.com ; <<>> DiG 9.11.20-RedHat-9.11.20-5.el8_3.1 <<>> www.google.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25735 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4000 ;; QUESTION SECTION: ;www.google.com. IN A ;; ANSWER SECTION: www.google.com. 113 IN A 142.250.185.228 ;; Query time: 0 msec ;; SERVER: x.x.x.101#53(x.x.x.101) ;; WHEN: Fri Oct 08 15:10:21 CEST 2021 ;; MSG SIZE rcvd: 59 $ dig @8.8.8.8 www.google.com ; <<>> DiG 9.11.20-RedHat-9.11.20-5.el8_3.1 <<>> @8.8.8.8 www.google.com ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3924 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;www.google.com. IN A ;; ANSWER SECTION: www.google.com. 300 IN A 142.250.185.228 ;; Query time: 34 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Fri Oct 08 15:10:49 CEST 2021 ;; MSG SIZE rcvd: 59 $ dig @x.x.x.101 www.google.com ; <<>> DiG 9.11.20-RedHat-9.11.20-5.el8_3.1 <<>> @x.x.x.101 www.google.com ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60305 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4000 ;; QUESTION SECTION: ;www.google.com. IN A ;; ANSWER SECTION: www.google.com. 70 IN A 142.250.185.228 ;; Query time: 0 msec ;; SERVER: x.x.x.101#53(x.x.x.101) ;; WHEN: Fri Oct 08 15:11:04 CEST 2021 ;; MSG SIZE rcvd: 59

I have no idea what seems to be going on ....

Improve this question
5
  • Which version of Kubernetes did you use? Commented Sep 22, 2021 at 13:17
  • I'm using Kubernetes v1.22 Commented Sep 23, 2021 at 6:20
  • Please provide your pod yaml file. Commented Sep 24, 2021 at 12:28
  • I rebuild the cluster in an infrastructure that is fully under my control, there DNS works. Commented Sep 27, 2021 at 6:50
  • It seems to be an issue connected to iptables and the firewall on the host, if I disable both the problem goes away I'll post and answer as soon as I know how to correctly configure these tools. Commented Oct 12, 2021 at 11:26

1 Answer 1

Reset to default
0

I have posted a community wiki answer for better visibility. As Thagor has mentioned in the comment, problem is resolved and was related to external DNS:

I rebuild the cluster in an infrastructure that is fully under my control, where DNS works.

Try disabling the iptables and the firewall on the host and see if you are able to do via coredns config.

More information about DNS you can find in the official documentation.

Improve this answer
2
  • Hello @Thagor. Does this answer your question? Commented Sep 28, 2021 at 11:53
  • hey i updated my first post turns out DNS works if i set it with $ microk8s enable dns:x.x.x.100,x.x.x.101 but as soon as I try to do it via the coredns config it fails, and I have no idea why. Commented Oct 5, 2021 at 13:11

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.