OVN-Kubernetes - My pods can not communicate in the same namespace

Issues with pod communication in a namespace

I have multiple pods running in the same namespace, but they cannot communicate with each other.

When sitea-frontend-8798d9cdc-sq7dk or siteb-frontend-66f9f55ddc-97jkg in the frontend namespace try to reach each other, communication just hangs.

Debug the Traffic Flow

To trace the path with ovnkube-trace, simply pass in the -src-namespace, -src pod name, -dst-namespace and -dst pod name:

./ovnkube-trace -tcp -src-namespace frontend -src sitea-frontend-8798d9cdc-77jj7 -dst-namespace frontend -dst siteb-frontend-66f9f55ddc-9qcwr --loglevel=4

What does ovnkube-trace do and how is it going to help us determine why our pods cannot communicate?

It is a wrapper tool that takes the CLI inputs, looks at your cluster, gets all of the IP and MAC information and builds an ovn-trace command to actually trace the traffic flow.

Running the First Trace

From your execution environment, run the following ovn-trace.

As you can see, the ovn-trace is far more complex than then ovnkube-trace:

  1. worker-5.ocpv.tamlab.rdu2.redhat.com → is the OVN datapath which is a logical router or switch associated to the workloads

  2. inport → is the OVN port for instance we are tracing from

  3. eth.src → source pod mac address

  4. eth.dst → destination pod mac address

  5. ip4.src → source pod IP address

  6. ip4.dst → destination pod IP address

  7. tcp.dst → destination port

  8. tcp.src → source port

ovn-trace --no-leader-only  --db unix:/var/run/ovn/ovnsb_db.sock worker-5.ocpv.tamlab.rdu2.redhat.com 'inport=="frontend_sitea-frontend-8798d9cdc-77jj7" && eth.src==0a:58:0a:81:06:0f && eth.dst==0a:58:0a:81:06:01 && ip4.src==10.129.6.15 && ip4.dst==10.129.6.17 && ip.ttl==64 && tcp.dst==80 && tcp.src==52888'

When the ovn-trace runs, you get a full accounting of how that packet would traverse the SDN. Because we don’t need to analyze the whole thing, we’re going to focus on the last egress section.

egress(dp="worker-5.ocpv.tamlab.rdu2.redhat.com", inport="stor-worker-5.ocpv.tamlab.rdu2.redhat.com", outport="frontend_siteb-frontend-66f9f55ddc-9qcwr")
---------------------------------------------------------------------------------------------------------------------------------------------------------
 2. ls_out_pre_acl (northd.c:6095): ip, priority 100, uuid c1ee92b5
    reg0[0] = 1;
    next;
 3. ls_out_pre_lb (northd.c:6305): ip, priority 100, uuid 2f4aa83e
    reg0[2] = 1;
    next;
 4. ls_out_pre_stateful (northd.c:6336): reg0[2] == 1, priority 110, uuid bcf2d2d5
    ct_lb_mark;

ct_lb_mark /* default (use --ct to customize) */
------------------------------------------------
 5. ls_out_acl_hint (northd.c:6437): !ct.new && ct.est && !ct.rpl && ct_mark.blocked == 0, priority 4, uuid cfae0de0
    reg0[8] = 1;
    reg0[10] = 1;
    next;
 8. ls_out_acl_action (northd.c:7346): reg8[30..31] == 0, priority 500, uuid 9ba0e4eb
    reg8[30..31] = 1;
    next(6);
 8. ls_out_acl_action (northd.c:7346): reg8[30..31] == 1, priority 500, uuid a72c3c38
    reg8[30..31] = 2;
    next(6);
 6. ls_out_acl_eval (northd.c:7175): reg8[30..31] == 2 && reg0[10] == 1 && (outport == @a3640229366109469754), priority 2000, uuid e5f2cb4b
    reg8[17] = 1;
    ct_commit { ct_mark.blocked = 1; ct_label.obs_point_id = 0; };
    next;
 8. ls_out_acl_action (northd.c:7319): reg8[17] == 1, priority 1000, uuid b201ae3a
    reg8[16] = 0;
    reg8[17] = 0;
    reg8[18] = 0;
    reg8[30..31] = 0;

Analyzing the Trace

What we see is an abrupt end with no clear indication of success or failure.

Focusing in, you can see that during ls_out_acl_eval we commit ct_mark.blocked = 1.

When ct_mark.blocked is set to 1, it means that the connection is no longer allowed by the policy and anything in reply will be dropped.

 6. ls_out_acl_eval (northd.c:7175): reg8[30..31] == 2 && reg0[10] == 1 && (outport == @a3640229366109469754), priority 2000, uuid e5f2cb4b
    reg8[17] = 1;
    ct_commit { ct_mark.blocked = 1; ct_label.obs_point_id = 0; };
    next;

So how do we determine why the connection was not allowed? Let’s inspect the database and find out.

Let’s start by running the following 2 commands to look at the list of logical router (lr) and logical switch (ls) objects in the database.

ovn-nbctl lr-list
Logical Router objects, defined by a UUID and name.
9316827c-90ed-4943-bb63-1480ffbcffac (GR_worker-5.ocpv.tamlab.rdu2.redhat.com)
07fef835-dcce-4db0-93df-5515f3dd0f49 (ovn_cluster_router)
ovn-nbctl ls-list
Logical Switch objects, defined by a UUID and name.
cc63f8ab-f7ff-4906-a15b-fb7b24c62eb2 (br.ex.localnet_ovn_localnet_switch)
1524ecbe-0704-4e9f-b97d-9a09e984ef02 (cluster_udn_drenard.udn.tamlab_ovn_layer2_switch)
4168b09e-0580-4d54-9ef2-3c5dc361a0bf (ext_worker-5.ocpv.tamlab.rdu2.redhat.com)
26008754-5629-419b-87cb-bf9fbbf9ff7f (join)
57b99e34-e732-4ce1-8244-eb507ee02ffe (ovs.bridge.vlan530_ovn_localnet_switch)
8763ad07-147d-4f38-b87f-41b5c152a7bb (transit_switch)
16ccd735-09f4-41c1-a546-2df09f22c2f7 (vlan.636.localnet_ovn_localnet_switch)
217a444d-ef97-4d51-89d8-581d673e3cf0 (worker-5.ocpv.tamlab.rdu2.redhat.com)

From the output, you will notice one of the logical switches, worker-5.ocpv.tamlab.rdu2.redhat.com is the switch we used in our ovn-trace. So let’s take a closer look at that switch by listing at the attached logical switch ports (lsp).

Because a cluster is going to have a lot of ports, we focus in with a grep on our specific pods. If you are interested, feel free to run the command again without the grep to see the full list.

ovn-nbctl lsp-list worker-5.ocpv.tamlab.rdu2.redhat.com | grep 'siteb\|sitea\|network-policy-demo-monitor\|external'
Logical Switch Port objects, defined by UUID and name, where the name maps to pod
0d633506-aad3-4136-a1bf-70f86eed3a9d (backend_sitea-backend-796cf44b87-l7mhb)
b1fbaafe-32e7-400d-926a-c31b55a7b0d2 (backend_siteb-backend-5b765679b5-l4gp6)
5512943c-a0ce-4ce8-9ff4-c3edf9f6b994 (external_virt-launcher-external-vm-x7gnx)
3db761b5-3239-426b-8ba7-7d8aea2b12e9 (frontend_sitea-frontend-8798d9cdc-77jj7)
d22f7f6b-31c0-49e1-a81d-d5c7197885b7 (frontend_siteb-frontend-66f9f55ddc-9qcwr)
8a80737e-7db1-4558-ab46-0b2e65440a7d (network-policy-demo-monitor_monitor-956f5b8d7-gkmnn)

Now that we have taken a look and validated our logical switch, logical router and logical switch port objects, we can come back to our trace. You can see traffic going to outport == @a3640229366109469754 before it is marked as dropped.

Each outport corresponds to a portgroup and one is created for every unique NetworkPolicy in your cluster. Every pod selected by the podSelector of that NetworkPolicy is added to the portgroup port list.

Let’s take a look at our problematic portgroup.

ovn-nbctl find Port_Group name="a3640229366109469754"
_uuid               : 5a0d0fe2-bece-44df-9990-66f323e26f19
acls                : [0cbe6f52-9cf1-4584-9237-7c23f65c82f1, 1fdd654d-f251-4e58-b4ce-d57804f1ba9b]
external_ids        : {direction=Ingress, "k8s.ovn.org/id"="default-network-controller:NetpolNamespace:frontend:Ingress", "k8s.ovn.org/name"=frontend, "k8s.ovn.org/owner-controller"=default-network-controller, "k8s.ovn.org/owner-type"=NetpolNamespace}
name                : a3640229366109469754
ports               : [3db761b5-3239-426b-8ba7-7d8aea2b12e9, d22f7f6b-31c0-49e1-a81d-d5c7197885b7]

From the output there are 2 pieces of important information:

  1. acls → there are 2 UUIDs → [0cbe6f52-9cf1-4584-9237-7c23f65c82f1, 1fdd654d-f251-4e58-b4ce-d57804f1ba9b]

  2. ports → there are 2 UUIDs → [3db761b5-3239-426b-8ba7-7d8aea2b12e9, d22f7f6b-31c0-49e1-a81d-d5c7197885b7]

What you will notice is that the 2 ports UUIDs correspond to the UUIDs of the logical switch port entries for the 2 frontend pods from the lsp-list output above. This confirms that the portgroup is influencing our 2 pods.

Next, let’s take a look at the 2 access control list (ACL) objects.

ovn-nbctl list ACL 0cbe6f52-9cf1-4584-9237-7c23f65c82f1
_uuid               : 0cbe6f52-9cf1-4584-9237-7c23f65c82f1
action              : drop
direction           : to-lport
external_ids        : {direction=Ingress, "k8s.ovn.org/id"="default-network-controller:NetpolNamespace:frontend:Ingress:defaultDeny", "k8s.ovn.org/name"=frontend, "k8s.ovn.org/owner-controller"=default-network-controller, "k8s.ovn.org/owner-type"=NetpolNamespace, type=defaultDeny}
label               : 0
log                 : false
match               : "outport == @a3640229366109469754"
meter               : acl-logging
name                : "NP:frontend:Ingress"
options             : {}
priority            : 1000
sample_est          : []
sample_new          : []
severity            : []
tier                : 2
ovn-nbctl list ACL 1fdd654d-f251-4e58-b4ce-d57804f1ba9b
_uuid               : 1fdd654d-f251-4e58-b4ce-d57804f1ba9b
action              : allow
direction           : to-lport
external_ids        : {direction=Ingress, "k8s.ovn.org/id"="default-network-controller:NetpolNamespace:frontend:Ingress:arpAllow", "k8s.ovn.org/name"=frontend, "k8s.ovn.org/owner-controller"=default-network-controller, "k8s.ovn.org/owner-type"=NetpolNamespace, type=arpAllow}
label               : 0
log                 : false
match               : "outport == @a3640229366109469754 && (arp || nd)"
meter               : acl-logging
name                : "NP:frontend:Ingress"
options             : {}
priority            : 1001
sample_est          : []
sample_new          : []
severity            : []
tier                : 2

You can see that the second ACL, 1fdd654d-f251-4e58-b4ce-d57804f1ba9b, is matching on arp || nd, so we can ignore it as we are sending tcp requests.

Focusing in on the output from the first ACL 0cbe6f52-9cf1-4584-9237-7c23f65c82f1, we can see a few key points:

  1. action → drop

  2. direction → to-lport

  3. external_ids → frontend:Ingress:defaultDeny

  4. match → outport == @a3640229366109469754

This tells us that any traffic that is forwarded to a logical port that matches outport == @a3640229366109469754 will be dropped due to a defaultDeny policy in the frontend namespace.

Looking back at our setup, we know that our cluster adds a default-deny NetworkPolicy to every new Namespace and that aligns to the policy above that is impacting our pod to pod communication.

Resolving the Connectivity Issue

Now that we know the issue, we can easily fix that with a new NetworkPolicy to allow traffic between pods in the namespace.

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-same-namespace
  namespace: frontend
spec:
  podSelector: {}
  ingress:
  - from:
    - podSelector: {}
  policyTypes:
  - Ingress

Because this is a static lab, we cannot directly demonstrate the resolution of the issue. So you can see the difference, we have included the output and analysis of a successful trace after applying the above NetworkPolicy.

Looking at the new trace, focusing in on the same egress section, there are 2 key differences.

egress(dp="worker-5.ocpv.tamlab.rdu2.redhat.com", inport="stor-worker-5.ocpv.tamlab.rdu2.redhat.com", outport="frontend_siteb-frontend-66f9f55ddc-9qcwr")
---------------------------------------------------------------------------------------------------------------------------------------------------------
 2. ls_out_pre_acl (northd.c:6095): ip, priority 100, uuid c1ee92b5
    reg0[0] = 1;
    next;
 3. ls_out_pre_lb (northd.c:6305): ip, priority 100, uuid 2f4aa83e
    reg0[2] = 1;
    next;
 4. ls_out_pre_stateful (northd.c:6336): reg0[2] == 1, priority 110, uuid bcf2d2d5
    ct_lb_mark;

ct_lb_mark /* default (use --ct to customize) */
------------------------------------------------
 5. ls_out_acl_hint (northd.c:6437): !ct.new && ct.est && !ct.rpl && ct_mark.blocked == 0, priority 4, uuid cfae0de0
    reg0[8] = 1;
    reg0[10] = 1;
    next;
 8. ls_out_acl_action (northd.c:7346): reg8[30..31] == 0, priority 500, uuid 9ba0e4eb
    reg8[30..31] = 1;
    next(6);
 8. ls_out_acl_action (northd.c:7346): reg8[30..31] == 1, priority 500, uuid a72c3c38
    reg8[30..31] = 2;
    next(6);
 6. ls_out_acl_eval (northd.c:7127): reg8[30..31] == 2 && reg0[8] == 1 && (ip4.src == {$a16799780576945479038} && outport == @a3002123605838622674), priority 2001, uuid a0914472
    reg8[16] = 1;
    next;
 8. ls_out_acl_action (northd.c:7314): reg8[16] == 1, priority 1000, uuid b2d321e2
    reg8[16] = 0;
    reg8[17] = 0;
    reg8[18] = 0;
    reg8[30..31] = 0;
    next;
11. ls_out_check_port_sec (northd.c:5904): 1, priority 0, uuid a87e1d41
    reg0[15] = check_out_port_sec();
    next;
12. ls_out_apply_port_sec (northd.c:5912): 1, priority 0, uuid 7e9dcb6d
    output;
    /* output to "frontend_siteb-frontend-66f9f55ddc-9qcwr", type "" */

  1. The ls_out_acl_eval is different and is no longer dropping the packet

6. ls_out_acl_eval (northd.c:7127): reg8[30..31] == 2 && reg0[8] == 1 && (ip4.src == {$a16799780576945479038} && outport == @a3002123605838622674), priority 2001, uuid a0914472

  1. We see a successful output to our destination pod → / output to "frontend_siteb-frontend-66f9f55ddc-9qcwr", type "" /

If we take it a step further, we can take a look at our new ls_out_acl_eval with ip4.src == {$a16799780576945479038} && outport == @a3002123605838622674.

The ip4.src is a hash corresponding to an address set:

ovn-nbctl list address_set a16799780576945479038

On the address_set you can see the addresses list, which contains the IPs of our 2 frontend pods.

_uuid               : d183e0fe-c0b7-4ebd-ae2a-4cf73ed8fd18
addresses           : ["10.129.6.15", "10.129.6.17"]
external_ids        : {ip-family=v4, "k8s.ovn.org/id"="default-network-controller:PodSelector:frontend_LS{}:v4", "k8s.ovn.org/name"="frontend_LS{}", "k8s.ovn.org/owner-controller"=default-network-controller, "k8s.ovn.org/owner-type"=PodSelector}
name                : a16799780576945479038

In your execution environment, you can run a list address_set with no UUID to see a list of all address sets in the cluster.

ovn-nbctl list address_set

The outport, like above, corresponds to a portgroup.

ovn-nbctl find Port_Group  name="a3002123605838622674"

Upon inspection, you can see that this portgroup comes from our new NetworkPolicy → NetworkPolicy:frontend:allow-same-namespace.

_uuid               : a12b4d2a-7b29-453b-8bf6-d8d4aae8221b
acls                : [1a4a5833-d2f1-4391-a4e6-439acb31910a]
external_ids        : {"k8s.ovn.org/id"="default-network-controller:NetworkPolicy:frontend:allow-same-namespace", "k8s.ovn.org/name"="frontend:allow-same-namespace", "k8s.ovn.org/owner-controller"=default-network-controller, "k8s.ovn.org/owner-type"=NetworkPolicy}
name                : a3002123605838622674
ports               : [3db761b5-3239-426b-8ba7-7d8aea2b12e9, d22f7f6b-31c0-49e1-a81d-d5c7197885b7]

On that portgroup, a new ACL exists.

ovn-nbctl list ACL 1a4a5833-d2f1-4391-a4e6-439acb31910a

This ACL now has allow-related for any to-lport traffic that matches the ip4.src and outport for our pods in the frontend Namespace.

_uuid               : 1a4a5833-d2f1-4391-a4e6-439acb31910a
action              : allow-related
direction           : to-lport
external_ids        : {direction=Ingress, gress-index="0", ip-block-index="-1", "k8s.ovn.org/id"="default-network-controller:NetworkPolicy:frontend:allow-same-namespace:Ingress:0:None:-1", "k8s.ovn.org/name"="frontend:allow-same-namespace", "k8s.ovn.org/owner-controller"=default-network-controller, "k8s.ovn.org/owner-type"=NetworkPolicy, port-policy-protocol=None}
label               : 0
log                 : false
match               : "ip4.src == {$a16799780576945479038} && outport == @a3002123605838622674"
meter               : acl-logging
name                : "NP:frontend:allow-same-namespace:Ingress:0"
options             : {}
priority            : 1001
sample_est          : []
sample_new          : []
severity            : []
tier                : 2

Let us know if you have any questions or feel free to move on to the next section!