Orphan `tap*_urunc` left after urunc restart in Kubernetes, preventing NetworkSetup

[sidneychang]

Summary

When urunc exits while the Pod's network namespace remains, previously created tap*_urunc devices can persist in the namespace in a NO-CARRIER / DOWN state. Current startup treats the mere existence of a TAP as an active unikernel and refuses to create a new TAP, causing network setup to fail.

Impact

• urunc Pod restart/retry can fail to configure networking because leftover TAPs block creation of a fresh TAP.
• Only affects TAPs created by urunc (naming pattern tap*_urunc). Must avoid deleting other CNI/user interfaces.

Steps to reproduce

1. Deploy the test Pod

Apply the test manifest and wait for the Pod to reach Running:

kubectl apply -f nginx-urunc.yaml
kubectl get pods

output:

deployment.apps/nginx-urunc created
service/nginx-urunc created

NAME                           READY   STATUS    RESTARTS   AGE
nginx-urunc-67f8694dd6-874rc   1/1     Running   0          55s

2. Locate the QEMU process for the urunc Pod
   List QEMU processes on the host and record the PID of the Pod’s QEMU instance (PID 1374168 in this example):

ps aux | grep qemu

output:

root     1374168 35.0  0.0 840108 85048 ? Ssl 05:28 0:00 /usr/bin/qemu-system-x86_64 ... -net tap,ifname=tap0_urunc ...

3. Inspect network interfaces inside the Pod netns

Enter the QEMU network namespace and list interfaces:

nsenter -t 1374168 -n ip link

output:

1: lo: <LOOPBACK,UP,LOWER_UP> ...
2: eth0@if288: <BROADCAST,MULTICAST,UP,LOWER_UP> ...
3: tap0_urunc: <BROADCAST,MULTICAST,UP,LOWER_UP> ...

At this point:

eth0 is UP, LOWER_UP
tap0_urunc is UP, LOWER_UP

The Pod is functioning normally

4. Force-kill the QEMU process (simulate a crash)

kill -9 1374168

Check Pod status:

kubectl get pods

output:

NAME                           READY   STATUS   RESTARTS   AGE
nginx-urunc-67f8694dd6-874rc   0/1     Error    0          65s

Kubernetes then automatically restarts the Pod:

NAME                           READY   STATUS    RESTARTS     AGE
nginx-urunc-67f8694dd6-874rc   1/1     Running   1 (6s ago)   69s

5. Locate the new QEMU process after restart
   List QEMU processes again and record the new PID (PID 1374761 in this example):

ps aux | grep qemu

output:

root     1374761  1.2  0.0 838364 83976 ? Ssl 05:29 0:00 /usr/bin/qemu-system-x86_64 ...

6. Inspect interfaces in the new QEMU ns

nsenter -t 1374761 -n ip link

output:

1: lo: <LOOPBACK,UP,LOWER_UP> ...
2: eth0@if288: <BROADCAST,MULTICAST,UP,LOWER_UP> ...
3: tap0_urunc: <NO-CARRIER,BROADCAST,MULTICAST,UP> ... state DOWN

Observed state after restart:

eth0 remains UP, LOWER_UP

tap0_urunc still exists but is now NO-CARRIER and state DOWN

7. Observe urunc error logs

Relevant log messages from urunc:

Failed to setup network :unsupported operation: can't spawn multiple unikernels in the same network namespace

The Pod restart succeeds at the Kubernetes level, but network setup inside urunc fails due to the presence of the pre-existing tap0_urunc device.

Reason

In kubernetes setups when a pod is getting restarted, the network namespace (created by the pause container) remains active and hence the tap0_urunc device still exists. Therefore, when urunc (re)creates the container it identifies the tap0_urunc device and it does not recreates it.

[cmainas]

Thank you @sidneychang for opening this issue. We will try to reproduce it and get back to you.

[sidneychang]

Thank you @sidneychang for opening this issue. We will try to reproduce it and get back to you.

Hi! I’m currently working on a fix for this issue and will submit a PR shortly.
The current reproduction steps are a bit rough — I’ll take some more time tomorrow to refine and clarify them.

[cmainas]

Hello @sidneychang ,

I was able to reproduce this issue.

Thank you for finding it.

[cmainas]

Hello @sidneychang , I think by merging the related PRs in this issue, the bug should be fixed. Could you verify that?

[sidneychang]

Hello @cmainas ,

From my side, it also appears to be resolved.

(base) root@sev:~/zxd/urunc# nsenter -t 631638 -n ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth0@if1112: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
Error: Peer netns reference is invalid.
    link/ether 52:26:29:dc:a6:68 brd ff:ff:ff:ff:ff:ff link-netnsid 0
4: tap0_urunc: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 16:4c:81:34:76:7f brd ff:ff:ff:ff:ff:ff
(base) root@sev:~/zxd/urunc# kill -9 631638
(base) root@sev:~/zxd/urunc# nsenter -t 631638 -n ip link
nsenter: cannot open /proc/631638/ns/net: No such file or directory
(base) root@sev:~/zxd/urunc# ps aux | grep qemu | grep urunc
root      631517  0.0  0.0   9452  3088 ?        Ss   19:20   0:00 /bin/bash -O extglob -c snap=$(command cat <&3) && builtin shopt -s extglob && builtin eval -- "$snap" && { builtin set +u 2>/dev/null || true; builtin export PWD="$(builtin pwd)"; builtin eval "$1"; }; COMMAND_EXIT_CODE=$?; dump_bash_state >&4; builtin exit $COMMAND_EXIT_CODE -- kubectl get pods -l run=nginx-qemu-urunc -w
root      631531  0.0  0.0 5477952 39148 ?       Sl   19:20   0:00 kubectl get pods -l run=nginx-qemu-urunc -w
root      637087  9.8  0.0 475500 64524 ?        Ssl  19:30   0:00 /opt/urunc/bin/qemu-system-x86_64 -m 268M -L /usr/share/qemu -cpu host -enable-kvm -display none -vga none -serial stdio -monitor null -smp 1 -kernel /unikernel/app-nginx_kvm-x86_64 -net nic,model=virtio,macaddr=52:26:29:dc:a6:68 -net tap,script=no,downscript=no,ifname=tap0_urunc -initrd /unikernel/initrd -append Unikraft   netdev.ipv4_addr=10.244.0.47 netdev.ipv4_gw_addr=10.244.0.1 netdev.ipv4_subnet_mask=255.255.255.0 vfs.rootfs=initrd -- -c /nginx/conf/nginx.conf
(base) root@sev:~/zxd/urunc# nsenter -t 637087 -n ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth0@if1112: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
Error: Peer netns reference is invalid.
    link/ether 52:26:29:dc:a6:68 brd ff:ff:ff:ff:ff:ff link-netnsid 0
5: tap0_urunc: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether ce:92:82:68:ee:25 brd ff:ff:ff:ff:ff:ff
(base) root@sev:~/zxd/urunc# kubectl get pods
NAME                                READY   STATUS    RESTARTS      AGE
nginx-qemu-urunc-7d5ff8c499-c9gg9   1/1     Running   1 (36s ago)   11m

However, I noticed there was an error. Based on further inspection, it seems that the current data path is no longer going through the expected veth + tap + tc filter pipeline.

Specifically:

The Pod is reachable and serving traffic normally:

(base) root@sev:~/zxd/urunc# kubectl get pods -o wide
NAME                                READY   STATUS    RESTARTS        AGE   IP            NODE   NOMINATED NODE   READINESS GATES
nginx-qemu-urunc-7d5ff8c499-c9gg9   1/1     Running   1 (4m46s ago)   15m   10.244.0.47   sev    <none>           <none>
(base) root@sev:~/zxd/urunc# curl 10.244.0.47
<!DOCTYPE html>
<html>
<head>
  <title>Hello, world!</title>
</head>
<body>
  <h1>Hello, world!</h1>
  <p>Powered by <a href="http://unikraft.org">Unikraft</a>.</p>
</body>

No tc filters are present on either interface:

(base) root@sev:~/zxd/urunc# nsenter -t 637087 -n tc filter show dev eth0
(base) root@sev:~/zxd/urunc# nsenter -t 637087 -n tc filter show dev tap0_urunc

both return empty.
I'm not sure whether this behavior is related to my environment — I ran the tests on a freshly installed machine with urunc.

[cmainas]

Hello @sidneychang , the tc filters are attached in the ingress handle (ffff). The tc filter show command above seems to use the default root handle (1). If you specify the the ffff handle, you should be able to see the filters.

$ sudo nsenter -t 2887056 -n tc filter show dev eth0 parent ffff:
filter protocol all pref 49152 u32 chain 0 
filter protocol all pref 49152 u32 chain 0 fh 800: ht divisor 1 
filter protocol all pref 49152 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 terminal flowid ??? not_in_hw 
  match 00000000/00000000 at 0
	action order 1: mirred (Egress Redirect to device tap0_urunc) stolen
	index 2 ref 1 bind 1

[sidneychang]

Hi @cmainas,

Thanks — you're right. I checked with parent ffff: and can see the filters now.

Everything looks fine on my side.