Kubernetes pods cannot resolve private IP addresses within the cluster running weave CNI

Question

Service definition

apiVersion: v1
kind: Service
metadata:
  name: zevrant-oauth2-service-db
spec:
  ports:
    - port: 5432
      targetPort: 5432
  selector:
    app: zevrant-oauth2-service-db

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: zevrant-oauth2-service-db-deployment
  labels:
    app: zevrant-oauth2-service-db
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zevrant-oauth2-service-db
  template:
    metadata:
      labels:
        app: zevrant-oauth2-service-db
    spec:
      imagePullSecrets:
        - name: regcred
      volumes:
        - name: database
          nfs:
            server: 192.168.0.127
            path: /i-data/420aa917/nfs/keys
      nodeSelector:
        architecture: arm
      containers:
        - name: zevrant-oauth2-service-db
          image: postgres:latest
          volumeMounts:
            - name: database
              mountPath: /storage/keys/
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: oauth-db-password
                  key: password
            - name: PGDATA
              value: /storage/keys/db/$ENVIRONMENT/oauth2/
            - name: POSTGRES_USER
              value: zevrant
            - name: POSTGRES_DB
              value: oauth2
          ports:
            - containerPort: 5432

cluster detail information

zevrant@master-node:~$ kubectl get svc -o wide -n kube-system|grep dns 
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   10h   k8s-app=kube-dns

zevrant@master-node:~$ kubectl get pod -o wide -n kube-system|grep dns
coredns-66bff467f8-nq5jv              1/1     Running   1          150m   10.32.0.7       zevrant       <none>           <none>
coredns-66bff467f8-tljmr              1/1     Running   1          10h    10.40.0.1       master-node   <none>           <none>

 kubectl get svc -o wide 
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE   SELECTOR
ingress-nodeport-develop    NodePort    10.99.251.215   <none>        8080:30124/TCP   10h   app=zevrant-home-ui
zevrant-oauth2-service-db   ClusterIP   10.97.75.171    <none>        5432/TCP         9h    app=zevrant-oauth2-service-db

coredns logs for both master and worker node consist of

.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.7
linux/amd64, go1.13.6, da7f65b

query for destination service

zevrant-home-ui@zevrant-home-ui-deployment-79c7dbb78-gksn8:/$ dig @10.96.0.10 zevrant-oauth2-service-db                    

; <<>> DiG 9.11.3-1ubuntu1.12-Ubuntu <<>> @10.96.0.10 zevrant-oauth2-service-db
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65289
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;zevrant-oauth2-service-db. IN  A

;; ANSWER SECTION:
zevrant-oauth2-service-db. 5    IN  A   92.242.140.2

;; Query time: 14 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Sun Jul 12 23:43:07 UTC 2020
;; MSG SIZE  rcvd: 95

/etc/resolve.conf

nameserver 10.96.0.10
search zevrant-home-services-develop.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

destination pod can hit the service (which is forwarded to itself) but UI pods cannot establish a connection to the service or destination pod via IP address. Any thoughts as to why this might be?

EDIT #1

postgress logs

2020-07-13 15:34:27.394 UTC [1] LOG:  starting PostgreSQL 12.3 (Debian 12.3-1.pgdg100+1) on arm-unknown-linux-gnueabihf, compiled by gcc (Debian 8.3.0-6) 8.3.0, 32-bit
2020-07-13 15:34:27.395 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2020-07-13 15:34:27.395 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2020-07-13 15:34:28.825 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-07-13 15:34:29.401 UTC [26] LOG:  database system was shut down at 2020-07-13 15:31:07 UTC
2020-07-13 15:34:29.508 UTC [1] LOG:  database system is ready to accept connections

weave report

{
    "Ready": true,
    "Version": "2.6.5",
    "VersionCheck": {
        "Enabled": true,
        "Success": false,
        "NewVersion": "",
        "NextCheckAt": "2020-07-13T19:06:21.030492043Z"
    },
    "Router": {
        "Protocol": "weave",
        "ProtocolMinVersion": 1,
        "ProtocolMaxVersion": 2,
        "Encryption": false,
        "PeerDiscovery": true,
        "Name": "ea:99:0e:11:d5:ae",
        "NickName": "master-node",
        "Port": 6783,
        "Peers": [
            {
                "Name": "ea:99:0e:11:d5:ae",
                "NickName": "master-node",
                "UID": 4593396642225438236,
                "ShortID": 2254,
                "Version": 4,
                "Connections": [
                    {
                        "Name": "1a:21:59:47:9a:5e",
                        "NickName": "zevrant",
                        "Address": "192.168.0.207:38089",
                        "Outbound": false,
                        "Established": true
                    },
                    {
                        "Name": "6a:6f:c4:0b:db:46",
                        "NickName": "raspi-03-02",
                        "Address": "192.168.0.42:6783",
                        "Outbound": true,
                        "Established": true
                    }
                ]
            },
            {
                "Name": "6a:6f:c4:0b:db:46",
                "NickName": "raspi-03-02",
                "UID": 13536333655574378111,
                "ShortID": 3959,
                "Version": 13,
                "Connections": [
                    {
                        "Name": "1a:21:59:47:9a:5e",
                        "NickName": "zevrant",
                        "Address": "192.168.0.207:45287",
                        "Outbound": false,
                        "Established": true
                    },
                    {
                        "Name": "ea:99:0e:11:d5:ae",
                        "NickName": "master-node",
                        "Address": "192.168.0.73:50257",
                        "Outbound": false,
                        "Established": true
                    }
                ]
            },
            {
                "Name": "1a:21:59:47:9a:5e",
                "NickName": "zevrant",
                "UID": 14270350061285030667,
                "ShortID": 3515,
                "Version": 4,
                "Connections": [
                    {
                        "Name": "6a:6f:c4:0b:db:46",
                        "NickName": "raspi-03-02",
                        "Address": "192.168.0.42:6783",
                        "Outbound": true,
                        "Established": true
                    },
                    {
                        "Name": "ea:99:0e:11:d5:ae",
                        "NickName": "master-node",
                        "Address": "192.168.0.73:6783",
                        "Outbound": true,
                        "Established": true
                    }
                ]
            }
        ],
        "UnicastRoutes": [
            {
                "Dest": "6a:6f:c4:0b:db:46",
                "Via": "6a:6f:c4:0b:db:46"
            },
            {
                "Dest": "1a:21:59:47:9a:5e",
                "Via": "1a:21:59:47:9a:5e"
            },
            {
                "Dest": "ea:99:0e:11:d5:ae",
                "Via": "00:00:00:00:00:00"
            }
        ],
        "BroadcastRoutes": [
            {
                "Source": "ea:99:0e:11:d5:ae",
                "Via": [
                    "6a:6f:c4:0b:db:46",
                    "1a:21:59:47:9a:5e"
                ]
            },
            {
                "Source": "1a:21:59:47:9a:5e",
                "Via": null
            },
            {
                "Source": "6a:6f:c4:0b:db:46",
                "Via": null
            }
        ],
        "Connections": [
            {
                "Address": "192.168.0.42:6783",
                "Outbound": true,
                "State": "established",
                "Info": "fastdp 6a:6f:c4:0b:db:46(raspi-03-02)",
                "Attrs": {
                    "mtu": 1376,
                    "name": "fastdp"
                }
            },
            {
                "Address": "192.168.0.207:38089",
                "Outbound": false,
                "State": "established",
                "Info": "fastdp 1a:21:59:47:9a:5e(zevrant)",
                "Attrs": {
                    "mtu": 1376,
                    "name": "fastdp"
                }
            },
            {
                "Address": "192.168.0.100:6783",
                "Outbound": true,
                "State": "failed",
                "Info": "dial tcp :0-\u003e192.168.0.100:6783: connect: connection refused, retry: 2020-07-13 15:53:46.887472731 +0000 UTC m=+60329.814440915",
                "Attrs": null
            },
            {
                "Address": "192.168.0.82:6783",
                "Outbound": true,
                "State": "failed",
                "Info": "dial tcp :0-\u003e192.168.0.82:6783: connect: connection refused, retry: 2020-07-13 15:47:56.145112536 +0000 UTC m=+59979.072080717",
                "Attrs": null
            }
        ],
        "TerminationCount": 0,
        "Targets": [
            "192.168.0.82",
            "192.168.0.42",
            "192.168.0.100",
            "192.168.0.207"
        ],
        "OverlayDiagnostics": {
            "fastdp": {
                "Vports": [
                    {
                        "ID": 0,
                        "Name": "datapath",
                        "TypeName": "internal"
                    },
                    {
                        "ID": 1,
                        "Name": "vethwe-datapath",
                        "TypeName": "netdev"
                    },
                    {
                        "ID": 2,
                        "Name": "vxlan-6784",
                        "TypeName": "vxlan"
                    }
                ],
                "Flows": [
                    {
                        "FlowKeys": [
                            "EthernetFlowKey{src: ca:03:86:42:44:3b, dst: 1a:21:59:47:9a:5e}",
                            "InPortFlowKey{vport: 1}",
                            "UnknownFlowKey{type: 23, key: 0000, mask: 0000}",
                            "UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
                            "UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
                            "UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}"
                        ],
                        "Actions": [
                            "SetTunnelAction{id: 0000000000dbb8ce, ipv4src: 192.168.0.73, ipv4dst: 192.168.0.207, ttl: 64, df: true}",
                            "OutputAction{vport: 2}"
                        ],
                        "Packets": 24,
                        "Bytes": 4230,
                        "Used": 59955261
                    },
                    {
                        "FlowKeys": [
                            "UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
                            "UnknownFlowKey{type: 23, key: 0000, mask: 0000}",
                            "UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
                            "EthernetFlowKey{src: ea:99:0e:11:d5:ae, dst: 01:00:5e:7f:ff:fa}",
                            "InPortFlowKey{vport: 1}",
                            "UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}"
                        ],
                        "Actions": [
                            "SetTunnelAction{id: 0000000000f778ce, ipv4src: 192.168.0.73, ipv4dst: 192.168.0.42, ttl: 64, df: true}",
                            "OutputAction{vport: 2}",
                            "SetTunnelAction{id: 0000000000dbb8ce, ipv4src: 192.168.0.73, ipv4dst: 192.168.0.207, ttl: 64, df: true}",
                            "OutputAction{vport: 2}",
                            "OutputAction{vport: 0}"
                        ],
                        "Packets": 40,
                        "Bytes": 12240,
                        "Used": 60041269
                    },
                    {
                        "FlowKeys": [
                            "TunnelFlowKey{id: 00000000008cedbb, ipv4src: 192.168.0.207, ipv4dst: 192.168.0.73}",
                            "InPortFlowKey{vport: 2}",
                            "UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
                            "UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}",
                            "EthernetFlowKey{src: 1a:21:59:47:9a:5e, dst: ca:03:86:42:44:3b}",
                            "UnknownFlowKey{type: 23, key: 0000, mask: 0000}",
                            "UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}"
                        ],
                        "Actions": [
                            "OutputAction{vport: 1}"
                        ],
                        "Packets": 24,
                        "Bytes": 2106,
                        "Used": 59955261
                    },
                    {
                        "FlowKeys": [
                            "UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
                            "EthernetFlowKey{src: ee:7c:a4:ba:4d:12, dst: ca:03:86:42:44:3b}",
                            "UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}",
                            "UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
                            "InPortFlowKey{vport: 2}",
                            "TunnelFlowKey{id: 00000000008cedbb, ipv4src: 192.168.0.207, ipv4dst: 192.168.0.73}",
                            "UnknownFlowKey{type: 23, key: 0000, mask: 0000}"
                        ],
                        "Actions": [
                            "OutputAction{vport: 1}"
                        ],
                        "Packets": 5,
                        "Bytes": 414,
                        "Used": 59953149
                    },
                    {
                        "FlowKeys": [
                            "UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
                            "UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
                            "UnknownFlowKey{type: 23, key: 0000, mask: 0000}",
                            "UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}",
                            "EthernetFlowKey{src: 42:8b:89:88:2b:c3, dst: ca:03:86:42:44:3b}",
                            "InPortFlowKey{vport: 2}",
                            "TunnelFlowKey{id: 00000000008cedbb, ipv4src: 192.168.0.207, ipv4dst: 192.168.0.73}"
                        ],
                        "Actions": [
                            "OutputAction{vport: 1}"
                        ],
                        "Packets": 4,
                        "Bytes": 398,
                        "Used": 59942269
                    },
                    {
                        "FlowKeys": [
                            "EthernetFlowKey{src: ca:03:86:42:44:3b, dst: ee:7c:a4:ba:4d:12}",
                            "InPortFlowKey{vport: 1}",
                            "UnknownFlowKey{type: 23, key: 0000, mask: 0000}",
                            "UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
                            "UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
                            "UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}"
                        ],
                        "Actions": [
                            "SetTunnelAction{id: 0000000000dbb8ce, ipv4src: 192.168.0.73, ipv4dst: 192.168.0.207, ttl: 64, df: true}",
                            "OutputAction{vport: 2}"
                        ],
                        "Packets": 5,
                        "Bytes": 1032,
                        "Used": 59953149
                    },
                    {
                        "FlowKeys": [
                            "UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
                            "EthernetFlowKey{src: ca:03:86:42:44:3b, dst: 8a:0a:d2:ae:d3:97}",
                            "UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
                            "InPortFlowKey{vport: 1}",
                            "UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}",
                            "UnknownFlowKey{type: 23, key: 0000, mask: 0000}"
                        ],
                        "Actions": [
                            "SetTunnelAction{id: 0000000000dbb8ce, ipv4src: 192.168.0.73, ipv4dst: 192.168.0.207, ttl: 64, df: true}",
                            "OutputAction{vport: 2}"
                        ],
                        "Packets": 6,
                        "Bytes": 1592,
                        "Used": 59954109
                    },
                    {
                        "FlowKeys": [
                            "UnknownFlowKey{type: 23, key: 0000, mask: 0000}",
                            "UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
                            "TunnelFlowKey{id: 00000000008cedbb, ipv4src: 192.168.0.207, ipv4dst: 192.168.0.73}",
                            "UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
                            "EthernetFlowKey{src: 8a:0a:d2:ae:d3:97, dst: ca:03:86:42:44:3b}",
                            "InPortFlowKey{vport: 2}",
                            "UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}"
                        ],
                        "Actions": [
                            "OutputAction{vport: 1}"
                        ],
                        "Packets": 6,
                        "Bytes": 542,
                        "Used": 59954109
                    },
                    {
                        "FlowKeys": [
                            "UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}",
                            "UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
                            "UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
                            "EthernetFlowKey{src: be:ae:7d:4e:72:5a, dst: ff:ff:ff:ff:ff:ff}",
                            "TunnelFlowKey{id: 00000000008cef77, ipv4src: 192.168.0.42, ipv4dst: 192.168.0.73}",
                            "InPortFlowKey{vport: 2}",
                            "UnknownFlowKey{type: 23, key: 0000, mask: 0000}"
                        ],
                        "Actions": [
                            "OutputAction{vport: 1}",
                            "OutputAction{vport: 0}"
                        ],
                        "Packets": 10,
                        "Bytes": 1784,
                        "Used": 60044189
                    },
                    {
                        "FlowKeys": [
                            "InPortFlowKey{vport: 1}",
                            "UnknownFlowKey{type: 23, key: 0000, mask: 0000}",
                            "UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
                            "EthernetFlowKey{src: ca:03:86:42:44:3b, dst: 42:8b:89:88:2b:c3}",
                            "UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}",
                            "UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}"
                        ],
                        "Actions": [
                            "SetTunnelAction{id: 0000000000dbb8ce, ipv4src: 192.168.0.73, ipv4dst: 192.168.0.207, ttl: 64, df: true}",
                            "OutputAction{vport: 2}"
                        ],
                        "Packets": 4,
                        "Bytes": 1016,
                        "Used": 59942269
                    },
                    {
                        "FlowKeys": [
                            "UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
                            "InPortFlowKey{vport: 2}",
                            "EthernetFlowKey{src: 3a:96:00:f9:20:d1, dst: ff:ff:ff:ff:ff:ff}",
                            "TunnelFlowKey{id: 00000000008cef77, ipv4src: 192.168.0.42, ipv4dst: 192.168.0.73}",
                            "UnknownFlowKey{type: 23, key: 0000, mask: 0000}",
                            "UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
                            "UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}"
                        ],
                        "Actions": [
                            "OutputAction{vport: 1}",
                            "OutputAction{vport: 0}"
                        ],
                        "Packets": 3,
                        "Bytes": 1149,
                        "Used": 59998761
                    }
                ]
            },
            "sleeve": null
        },
        "TrustedSubnets": [],
        "Interface": "datapath (via ODP)",
        "CaptureStats": {
            "FlowMisses": 12234
        },
        "MACs": [
            {
                "Mac": "ee:7c:a4:ba:4d:12",
                "Name": "1a:21:59:47:9a:5e",
                "NickName": "zevrant",
                "LastSeen": "2020-07-13T15:43:17.292214531Z"
            },
            {
                "Mac": "ca:03:86:42:44:3b",
                "Name": "ea:99:0e:11:d5:ae",
                "NickName": "master-node",
                "LastSeen": "2020-07-13T15:43:17.292142775Z"
            },
            {
                "Mac": "ea:99:0e:11:d5:ae",
                "Name": "ea:99:0e:11:d5:ae",
                "NickName": "master-node",
                "LastSeen": "2020-07-13T15:43:17.292177972Z"
            },
            {
                "Mac": "8a:0a:d2:ae:d3:97",
                "Name": "1a:21:59:47:9a:5e",
                "NickName": "zevrant",
                "LastSeen": "2020-07-13T15:43:17.292261033Z"
            },
            {
                "Mac": "3a:96:00:f9:20:d1",
                "Name": "6a:6f:c4:0b:db:46",
                "NickName": "raspi-03-02",
                "LastSeen": "2020-07-13T15:43:17.292295962Z"
            },
            {
                "Mac": "be:ae:7d:4e:72:5a",
                "Name": "6a:6f:c4:0b:db:46",
                "NickName": "raspi-03-02",
                "LastSeen": "2020-07-13T15:43:17.292272079Z"
            },
            {
                "Mac": "1a:21:59:47:9a:5e",
                "Name": "1a:21:59:47:9a:5e",
                "NickName": "zevrant",
                "LastSeen": "2020-07-13T15:43:17.292200309Z"
            },
            {
                "Mac": "42:8b:89:88:2b:c3",
                "Name": "1a:21:59:47:9a:5e",
                "NickName": "zevrant",
                "LastSeen": "2020-07-13T15:43:17.292226062Z"
            }
        ]
    },
    "IPAM": {
        "Paxos": null,
        "Range": "10.32.0.0/12",
        "RangeNumIPs": 1048576,
        "ActiveIPs": 2,
        "DefaultSubnet": "10.32.0.0/12",
        "Entries": [
            {
                "Token": "10.32.0.0",
                "Size": 393216,
                "Peer": "1a:21:59:47:9a:5e",
                "Nickname": "zevrant",
                "IsKnownPeer": true,
                "Version": 75
            },
            {
                "Token": "10.38.0.0",
                "Size": 131072,
                "Peer": "6a:6f:c4:0b:db:46",
                "Nickname": "raspi-03-02",
                "IsKnownPeer": true,
                "Version": 15
            },
            {
                "Token": "10.40.0.0",
                "Size": 262144,
                "Peer": "ea:99:0e:11:d5:ae",
                "Nickname": "master-node",
                "IsKnownPeer": true,
                "Version": 6
            },
            {
                "Token": "10.44.0.0",
                "Size": 1,
                "Peer": "6a:6f:c4:0b:db:46",
                "Nickname": "raspi-03-02",
                "IsKnownPeer": true,
                "Version": 1
            },
            {
                "Token": "10.44.0.1",
                "Size": 262143,
                "Peer": "ea:99:0e:11:d5:ae",
                "Nickname": "master-node",
                "IsKnownPeer": true,
                "Version": 0
            }
        ],
        "PendingClaims": null,
        "PendingAllocates": null
    }
}

Pods are deployed into the same namespace as such

NAME                                                    READY   STATUS             RESTARTS   AGE
zevrant-home-ui-deployment-79c7dbb78-gksn8              1/1     Running            0          16h
zevrant-home-ui-deployment-79c7dbb78-zgp4c              1/1     Running            0          16h
zevrant-oauth2-service-db-deployment-6f7c7ccdb5-5g599   1/1     Running    

    0          19m

What do you mean by "UI pod"? I ask because if the UI is serving something like a react app, that is running on the client machine, and if it is trying to access a back-end service running in a pod, you would need to make sure ingress would allow that traffic in — Brian Pursley, Jul 13 '20 at 01:25
i'm execing into the pod to test the ability to connect to the database ingress is setup however unnecessary in this scenario — Zevrant, Jul 13 '20 at 13:12
Can you use `weave report` to generate a dump and post it in your question? https://www.weave.works/docs/net/latest/troubleshooting/#producing-a-json-report — Brian Pursley, Jul 13 '20 at 14:09
Also `weave status dns` might be helpful https://www.weave.works/docs/net/latest/troubleshooting/#listing-dns-entries — Brian Pursley, Jul 13 '20 at 14:10
Ah my bad, Weave DNS is disabled for use in Kubenetes, which handles its own DNS. — Brian Pursley, Jul 13 '20 at 19:11

Rico · Answer 1 · 2020-07-13T14:34:17.333

2

Based on the dig output you shared zevrant-oauth2-service-db is resolving to 92.242.140.2 but it looks like the IP address of your K8s service is 10.97.75.171 (ClusterIP) (based on the output you shared too).

If you hit 10.97.75.171 5432 you should be able to access your Postgres database, provided that you don't have any Kubernetes Network Policy and/or firewall blocking access. Make sure you that in your Postgres config you are binding the server to 0.0.0.0 otherwise if it's something like localhost you will only be able to get to it from the pod.

So the question is what is 92.242.140.2? Wny is coredns responding to a query to zevrant-oauth2-service-db with 92.242.140.2? Is there a DNS forwarder configured in coredns? Is there a default domain configured that is not part of svc.cluster.local?

edited Jul 13 '20 at 14:34

answered Jul 13 '20 at 04:48

Rico

58,485
12
111
141

So even using the old address of the service and the pod itself I am unable to establish a connection. The IP address in question is my isp DNS server. If there is a dns forwarder set up how would I disable/remove this. – Zevrant Jul 13 '20 at 12:24
This depends on the config of the forwarder. But the workaround is just to use more of the same to force coredns to resolve to your Kubernetes address `zevrant-oauth2-service-db`. For example. `zevrant-oauth2-service-db..svc.cluster.local` Regarding not being able to access the db, I updated the answer. You need to make sure you bind to `0.0.0.0` – Rico Jul 13 '20 at 14:37
currently the database is configured with listen_addresses = '*' and host all all 0.0.0.0/0 trust – Zevrant Jul 13 '20 at 15:21
what pod are you using to test the pod to pod connectivity? – Rico Jul 13 '20 at 15:30
i am using zevrant-home-ui-deployment-79c7dbb78-gksn8 as shown above – Zevrant Jul 13 '20 at 15:56
i do not it should all be default settings – Zevrant Jul 13 '20 at 16:31
Did you build the docker/container image? Are you sure it's exposing 5432? – Rico Jul 13 '20 at 16:36
i did not it is runnning postgres:latest – Zevrant Jul 13 '20 at 17:10
So your original question was about DNS. maybe this has shifted to connectivity? Then, is weave configured the way it's supposed to? Are you kube-proxy processes running correctly? – Rico Jul 13 '20 at 17:19
i am seeing a lot of `WARN: 2020/07/13 17:41:19.507556 UDP connection from 0.0.0.0:68 to 255.255.255.255:67 blocked by Weave NPC.` – Zevrant Jul 13 '20 at 17:42
also seeing ` k8s.io/client-go/informers/factory.go:135: Failed to watch *v1.Endpoints: Get https://192.168.0.73:6443/api/v1/endpoints?allowWatchBookmarks=true&labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=115538&timeout=7m46s&timeoutSeconds=466&watch=true: dial tcp 192.168.0.73:6443: connect: connection refused` on the database server – Zevrant Jul 13 '20 at 17:44

score 1 · Accepted Answer · edited Aug 26 '20 at 08:53

1

The solution ended up being to run the following commands on the master and all slave nodes. This is needed any time there are changes to the networking settings on the cluster.

sysctl net.ipv4.conf.all.forwarding=1
 iptables -P FORWARD ACCEPT
 iptables -P FORWARD ACCEPT
 iptables -P OUTPUT ACCEPT
 systemctl stop kubelet
 systemctl stop docker
 iptables --flush
 iptables -tnat --flush
 systemctl start kubelet
 systemctl start docker

edited Aug 26 '20 at 08:53

Community

1
1

answered Jul 13 '20 at 19:13

Zevrant

115
1
9

You should have asked this issue on a different question, now my answer looks like its answering something different because you changed your question after I posted the initial answer – Rico Jul 13 '20 at 19:28
How did you arrive at this solution? – Brian Pursley Jul 13 '20 at 19:53
I kept seeing ` k8s.io/client-go/informers/factory.go:135: Failed to watch *v1.Endpoints: Get 192.168.0.73:6443/api/v1/…: dial tcp 192.168.0.73:6443: connect: connection refused` in the dns logs doing a google search led me to a three year old issue that was supposedly fixed – Zevrant Jul 14 '20 at 17:44
yeah, IP forwarding was disabled. IP forwarding enablement is pretty standard in Kubernetes nodes. – Rico Jul 14 '20 at 23:22

Kubernetes pods cannot resolve private IP addresses within the cluster running weave CNI

2 Answers2