Re: [kata-dev] vsock & network namespaces in Kata
Hi Stefano, On Fri, 2019-12-13 at 09:35 +0100, Stefano Garzarella wrote:
Hi Sebastien,
On Thu, Dec 12, 2019 at 09:32:18PM +0000, Boeuf, Sebastien wrote:
On Wed, 2019-12-11 at 18:27 +0100, Stefano Garzarella wrote:
On Thu, Dec 5, 2019 at 9:07 AM Boeuf, Sebastien < sebastien.boeuf@intel.com> wrote:
On Wed, 2019-12-04 at 17:36 +0000, Montes, Julio wrote:
Do you think could be an easy change in the Kata runtime? I need to look better, but if you already know the answer you'll save me some time :)
ok, this means all kata components must be in the same network namespace.
fortunately, kata-shim and qemu run in the same network namespace, so the change should be easy, just move kata-proxy and kata- runtime into the same net namespace
Well in case of vsock, we don't need kata-proxy, the shim being directly connected to the VM through vsock. And the kata-shim runs in the same netns as the VMM, so we're good from this perspective.
In case of containerd-shimv2, we don't have kata-shim anymore, the shim being a simple thread part of the kata-runtime process, which means kata-runtime needs to enter the netns, and that's where it might be pretty complex. IIRC, Julio did some experiments to run the kata-runtime in a set of namespaces, but I don't think we merged it, did we?
Looking in the code seems that kata-runtime uses the "vhostfd" parameter of the vsock device, this means that the vhost device (/dev/vhost- vsock) is opened by the runtime and then the file descriptor is passed to QEMU. This is great for this case because the netns assigned to the device is the one of the process that opened the /dev/vhost-vsock.
I tried containerd-shimv2 with the vhost-vsock modified to support netns and seems to work well.
Instead with v1 it doesn't work, maybe because kata-runtime and kata-shim run in two different netns. (or maybe I did something wrong, but it works disabling vsock or using the vhost-vsock without netns)
Does that make sense?
Yes I think your experiment and the result makes sense.
;-)
In case of v1, kata-shim runs in the same netns as QEMU, but kata- runtime runs in a separate one (the default netns). Later when kata- shim tries to access the fd, because it runs in a different netns than kata-runtime, the permission is denied.
Do you think can easly changed?
As Julio suggested, maybe we can move kata-runtime in the same netns.
As I said earlier, this is something that could work, and I think Julio had a POC on this. The point is to make sure we run the entire process into the netns. A possible hack would be to enter the netns when we open /dev/vhost- vsock, but that also means we need to enter the netns anytime the kata- runtime tries to communicate with the agent. Not sure this is a viable solution.
Otherwise, is it possible to open the /dev/vhost-vsock in kata- runtime, then pass it to kata-shim that pass it to QEMU?
From a pure technical standpoint, I'd say yes, but problem is, we would modify the shim and the kata-runtime/kata-shim interaction quite a lot. The kata-shim has been designed to be as simple as possible and act as a thin wrapper between containerd/crio and the container running in the VM.
So if we can solve this easily, it seems that there are no big issues for Kata with this new feature that I would like to introduce on Linux.
I agree, I think that eventually if Kata wants this feature to be supported, it will work out of the box for v2, and depending on the need for v1, the changes might be introduced.
In case of v2, there's only one process, the containerd-shimv2, which again runs on the default netns, but later will be the one trying to connect to the vhost-vsock fd, which is why it will succeed. In this case there's no more additional process connecting through vsock, it is simply a different thread from containerd-shimv2.
At first, I didn't get the fact that the netns authorized to access vsock had to be the one where the device was opened from. That's why my analysis was completely reversed :)
I think was my fault, I didn't know that Kata uses the 'vhostfd' parameter. So I always thought QEMU was opening it and that got you confused, sorry ;-)
Thanks, Stefano
Thanks, Sebastien --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
On Mon, Dec 16, 2019 at 07:30:40AM +0000, Boeuf, Sebastien wrote:
Hi Stefano,
On Fri, 2019-12-13 at 09:35 +0100, Stefano Garzarella wrote:
Hi Sebastien,
On Thu, Dec 12, 2019 at 09:32:18PM +0000, Boeuf, Sebastien wrote:
On Wed, 2019-12-11 at 18:27 +0100, Stefano Garzarella wrote:
On Thu, Dec 5, 2019 at 9:07 AM Boeuf, Sebastien < sebastien.boeuf@intel.com> wrote:
On Wed, 2019-12-04 at 17:36 +0000, Montes, Julio wrote:
Do you think could be an easy change in the Kata runtime? I need to look better, but if you already know the answer you'll save me some time :)
ok, this means all kata components must be in the same network namespace.
fortunately, kata-shim and qemu run in the same network namespace, so the change should be easy, just move kata-proxy and kata- runtime into the same net namespace
Well in case of vsock, we don't need kata-proxy, the shim being directly connected to the VM through vsock. And the kata-shim runs in the same netns as the VMM, so we're good from this perspective.
In case of containerd-shimv2, we don't have kata-shim anymore, the shim being a simple thread part of the kata-runtime process, which means kata-runtime needs to enter the netns, and that's where it might be pretty complex. IIRC, Julio did some experiments to run the kata-runtime in a set of namespaces, but I don't think we merged it, did we?
Looking in the code seems that kata-runtime uses the "vhostfd" parameter of the vsock device, this means that the vhost device (/dev/vhost- vsock) is opened by the runtime and then the file descriptor is passed to QEMU. This is great for this case because the netns assigned to the device is the one of the process that opened the /dev/vhost-vsock.
I tried containerd-shimv2 with the vhost-vsock modified to support netns and seems to work well.
Instead with v1 it doesn't work, maybe because kata-runtime and kata-shim run in two different netns. (or maybe I did something wrong, but it works disabling vsock or using the vhost-vsock without netns)
Does that make sense?
Yes I think your experiment and the result makes sense.
;-)
In case of v1, kata-shim runs in the same netns as QEMU, but kata- runtime runs in a separate one (the default netns). Later when kata- shim tries to access the fd, because it runs in a different netns than kata-runtime, the permission is denied.
Do you think can easly changed?
As Julio suggested, maybe we can move kata-runtime in the same netns.
As I said earlier, this is something that could work, and I think Julio had a POC on this. The point is to make sure we run the entire process into the netns.
Cool! @Julio, let me know if you have a POC working. I can try with my changes.
A possible hack would be to enter the netns when we open /dev/vhost- vsock, but that also means we need to enter the netns anytime the kata- runtime tries to communicate with the agent. Not sure this is a viable solution.
Perhaps it might be enough to enter into the netns only during the open. If the fd is not closed, it remains associated with the netns, so there is no need to re-enter for communication.
Otherwise, is it possible to open the /dev/vhost-vsock in kata- runtime, then pass it to kata-shim that pass it to QEMU?
From a pure technical standpoint, I'd say yes, but problem is, we would modify the shim and the kata-runtime/kata-shim interaction quite a lot. The kata-shim has been designed to be as simple as possible and act as a thin wrapper between containerd/crio and the container running in the VM.
Got it. If the hack above works, there's no point in complicating the architecture.
So if we can solve this easily, it seems that there are no big issues for Kata with this new feature that I would like to introduce on Linux.
I agree, I think that eventually if Kata wants this feature to be supported, it will work out of the box for v2, and depending on the need for v1, the changes might be introduced.
Great! Thank you very much for your collaboration, Stefano
Hi Stefano
Cool! @Julio, let me know if you have a POC working. I can try with my changes.
I only have a PoC for mntns, but I can help you with the netns, I'll send you the patch once it's ready - Julio ________________________________ From: Stefano Garzarella <sgarzare@redhat.com> Sent: Monday, December 16, 2019 6:16 AM To: Boeuf, Sebastien <sebastien.boeuf@intel.com>; Montes, Julio <julio.montes@intel.com> Cc: kata-dev@lists.katacontainers.io <kata-dev@lists.katacontainers.io> Subject: Re: [kata-dev] vsock & network namespaces in Kata On Mon, Dec 16, 2019 at 07:30:40AM +0000, Boeuf, Sebastien wrote:
Hi Stefano,
On Fri, 2019-12-13 at 09:35 +0100, Stefano Garzarella wrote:
Hi Sebastien,
On Thu, Dec 12, 2019 at 09:32:18PM +0000, Boeuf, Sebastien wrote:
On Wed, 2019-12-11 at 18:27 +0100, Stefano Garzarella wrote:
On Thu, Dec 5, 2019 at 9:07 AM Boeuf, Sebastien < sebastien.boeuf@intel.com> wrote:
On Wed, 2019-12-04 at 17:36 +0000, Montes, Julio wrote:
Do you think could be an easy change in the Kata runtime? I need to look better, but if you already know the answer you'll save me some time :)
ok, this means all kata components must be in the same network namespace.
fortunately, kata-shim and qemu run in the same network namespace, so the change should be easy, just move kata-proxy and kata- runtime into the same net namespace
Well in case of vsock, we don't need kata-proxy, the shim being directly connected to the VM through vsock. And the kata-shim runs in the same netns as the VMM, so we're good from this perspective.
In case of containerd-shimv2, we don't have kata-shim anymore, the shim being a simple thread part of the kata-runtime process, which means kata-runtime needs to enter the netns, and that's where it might be pretty complex. IIRC, Julio did some experiments to run the kata-runtime in a set of namespaces, but I don't think we merged it, did we?
Looking in the code seems that kata-runtime uses the "vhostfd" parameter of the vsock device, this means that the vhost device (/dev/vhost- vsock) is opened by the runtime and then the file descriptor is passed to QEMU. This is great for this case because the netns assigned to the device is the one of the process that opened the /dev/vhost-vsock.
I tried containerd-shimv2 with the vhost-vsock modified to support netns and seems to work well.
Instead with v1 it doesn't work, maybe because kata-runtime and kata-shim run in two different netns. (or maybe I did something wrong, but it works disabling vsock or using the vhost-vsock without netns)
Does that make sense?
Yes I think your experiment and the result makes sense.
;-)
In case of v1, kata-shim runs in the same netns as QEMU, but kata- runtime runs in a separate one (the default netns). Later when kata- shim tries to access the fd, because it runs in a different netns than kata-runtime, the permission is denied.
Do you think can easly changed?
As Julio suggested, maybe we can move kata-runtime in the same netns.
As I said earlier, this is something that could work, and I think Julio had a POC on this. The point is to make sure we run the entire process into the netns.
Cool! @Julio, let me know if you have a POC working. I can try with my changes.
A possible hack would be to enter the netns when we open /dev/vhost- vsock, but that also means we need to enter the netns anytime the kata- runtime tries to communicate with the agent. Not sure this is a viable solution.
Perhaps it might be enough to enter into the netns only during the open. If the fd is not closed, it remains associated with the netns, so there is no need to re-enter for communication.
Otherwise, is it possible to open the /dev/vhost-vsock in kata- runtime, then pass it to kata-shim that pass it to QEMU?
From a pure technical standpoint, I'd say yes, but problem is, we would modify the shim and the kata-runtime/kata-shim interaction quite a lot. The kata-shim has been designed to be as simple as possible and act as a thin wrapper between containerd/crio and the container running in the VM.
Got it. If the hack above works, there's no point in complicating the architecture.
So if we can solve this easily, it seems that there are no big issues for Kata with this new feature that I would like to introduce on Linux.
I agree, I think that eventually if Kata wants this feature to be supported, it will work out of the box for v2, and depending on the need for v1, the changes might be introduced.
Great! Thank you very much for your collaboration, Stefano
Hi Stefano This patch [1] moves the runtime to the sandbox's (qemu) net namespace. This patch is a hotfix that should be used only for testing - Julio [1] - https://github.com/devimc/kata-runtime/commit/656d518da223ecff9a9086be514073... ________________________________ From: Montes, Julio <julio.montes@intel.com> Sent: Monday, December 16, 2019 11:14 AM To: Stefano Garzarella <sgarzare@redhat.com>; Boeuf, Sebastien <sebastien.boeuf@intel.com> Cc: kata-dev@lists.katacontainers.io <kata-dev@lists.katacontainers.io> Subject: Re: [kata-dev] vsock & network namespaces in Kata Hi Stefano
Cool! @Julio, let me know if you have a POC working. I can try with my changes.
I only have a PoC for mntns, but I can help you with the netns, I'll send you the patch once it's ready - Julio ________________________________ From: Stefano Garzarella <sgarzare@redhat.com> Sent: Monday, December 16, 2019 6:16 AM To: Boeuf, Sebastien <sebastien.boeuf@intel.com>; Montes, Julio <julio.montes@intel.com> Cc: kata-dev@lists.katacontainers.io <kata-dev@lists.katacontainers.io> Subject: Re: [kata-dev] vsock & network namespaces in Kata On Mon, Dec 16, 2019 at 07:30:40AM +0000, Boeuf, Sebastien wrote:
Hi Stefano,
On Fri, 2019-12-13 at 09:35 +0100, Stefano Garzarella wrote:
Hi Sebastien,
On Thu, Dec 12, 2019 at 09:32:18PM +0000, Boeuf, Sebastien wrote:
On Wed, 2019-12-11 at 18:27 +0100, Stefano Garzarella wrote:
On Thu, Dec 5, 2019 at 9:07 AM Boeuf, Sebastien < sebastien.boeuf@intel.com> wrote:
On Wed, 2019-12-04 at 17:36 +0000, Montes, Julio wrote:
Do you think could be an easy change in the Kata runtime? I need to look better, but if you already know the answer you'll save me some time :)
ok, this means all kata components must be in the same network namespace.
fortunately, kata-shim and qemu run in the same network namespace, so the change should be easy, just move kata-proxy and kata- runtime into the same net namespace
Well in case of vsock, we don't need kata-proxy, the shim being directly connected to the VM through vsock. And the kata-shim runs in the same netns as the VMM, so we're good from this perspective.
In case of containerd-shimv2, we don't have kata-shim anymore, the shim being a simple thread part of the kata-runtime process, which means kata-runtime needs to enter the netns, and that's where it might be pretty complex. IIRC, Julio did some experiments to run the kata-runtime in a set of namespaces, but I don't think we merged it, did we?
Looking in the code seems that kata-runtime uses the "vhostfd" parameter of the vsock device, this means that the vhost device (/dev/vhost- vsock) is opened by the runtime and then the file descriptor is passed to QEMU. This is great for this case because the netns assigned to the device is the one of the process that opened the /dev/vhost-vsock.
I tried containerd-shimv2 with the vhost-vsock modified to support netns and seems to work well.
Instead with v1 it doesn't work, maybe because kata-runtime and kata-shim run in two different netns. (or maybe I did something wrong, but it works disabling vsock or using the vhost-vsock without netns)
Does that make sense?
Yes I think your experiment and the result makes sense.
;-)
In case of v1, kata-shim runs in the same netns as QEMU, but kata- runtime runs in a separate one (the default netns). Later when kata- shim tries to access the fd, because it runs in a different netns than kata-runtime, the permission is denied.
Do you think can easly changed?
As Julio suggested, maybe we can move kata-runtime in the same netns.
As I said earlier, this is something that could work, and I think Julio had a POC on this. The point is to make sure we run the entire process into the netns.
Cool! @Julio, let me know if you have a POC working. I can try with my changes.
A possible hack would be to enter the netns when we open /dev/vhost- vsock, but that also means we need to enter the netns anytime the kata- runtime tries to communicate with the agent. Not sure this is a viable solution.
Perhaps it might be enough to enter into the netns only during the open. If the fd is not closed, it remains associated with the netns, so there is no need to re-enter for communication.
Otherwise, is it possible to open the /dev/vhost-vsock in kata- runtime, then pass it to kata-shim that pass it to QEMU?
From a pure technical standpoint, I'd say yes, but problem is, we would modify the shim and the kata-runtime/kata-shim interaction quite a lot. The kata-shim has been designed to be as simple as possible and act as a thin wrapper between containerd/crio and the container running in the VM.
Got it. If the hack above works, there's no point in complicating the architecture.
So if we can solve this easily, it seems that there are no big issues for Kata with this new feature that I would like to introduce on Linux.
I agree, I think that eventually if Kata wants this feature to be supported, it will work out of the box for v2, and depending on the need for v1, the changes might be introduced.
Great! Thank you very much for your collaboration, Stefano
On 2019/12/17 05:51, Montes, Julio wrote:
Hi Stefano
This patch [1] moves the runtime to the sandbox's (qemu) net namespace. This patch is a hotfix that should be used only for testing
- Julio
[1] - https://github.com/devimc/kata-runtime/commit/656d518da223ecff9a9086be514073...
+func (n *Network) Set(netNSPath string) error { + span, _ := n.trace(context.Background(), "run") + defer span.Finish() + + runtime.LockOSThread() + defer runtime.UnlockOSThread() + + targetNS, err := ns.GetNS(netNSPath) + if err != nil { + return err + } + + if err := targetNS.Set(); err != nil { + return err + } + + return nil +} kata shimv2 has two Ms. I don't think it can work by just setting one thread's netns. The result is that netns is randomly set for shimv2 goroutines. containerd-shim-v2/service.go: cmd.Env = append(os.Environ(), "GOMAXPROCS=2") Cheers, Tao -- Into something rich and strange.
Hi Julio, On Mon, Dec 16, 2019 at 09:51:56PM +0000, Montes, Julio wrote:
Hi Stefano
This patch [1] moves the runtime to the sandbox's (qemu) net namespace. This patch is a hotfix that should be used only for testing
Thanks for the patch! I tried it, but I have the same behaviour (v1 doesn't work, v2 works). I did a minimal debugging with the old vsock modules, where v1 works well, checking the netns: $ ip netns cnitest-3c2d7774-b4e1-1901-6c8a-fb20b857898b (id: 0) $ ip netns pids cnitest-3c2d7774-b4e1-1901-6c8a-fb20b857898b 1343 1357 $ ps -p 1357 1343 PID TTY STAT TIME COMMAND 1343 ? Sl 0:21 /usr/bin/qemu-vanilla-system-x86_64 -name sandbox-b00b4ff14f15714027d76221d316ad778cfda3cb5e104421c5942c2e04adc752 -uuid 1cee59f0-ff07-4602-af2e-d25962de412f -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/vc/vm/b00b4ff14f15714027d76221d316ad778cfda3cb5e104421c5942c2e04adc752/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=8979M -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=true,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/b00b4ff14f15714027d76221d316ad778cfda3cb5e104421c5942c2e04adc752/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/kata-containers/kata-containers-image_clearlinux_1.10.0-alpha1_agent_c87f497312.img,size=134217728 -device virtio-scsi-pci,id=scsi0,disable-modern=true,romfile= -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng,rng=rng0,romfile= -device vhost-vsock-pci,disable-modern=true,vhostfd=3,id=vsock-1011671291,guest-cid=1011671291,romfile= -device virtio-9p-pci,disable-modern=true,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/b00b4ff14f15714027d76221d316ad778cfda3cb5e104421c5942c2e04adc752,security_model=none -netdev tap,id=network-0,vhost=on,vhostfds=4,fds=5 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,disable-modern=true,mq=on,vectors=4,romfile= -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -object memory-backend-ram,id=dimm1,size=2048M -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinuz-4.19.75.55-59.1.container -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro ro rootfstype=ext4 quiet systemd.show_status=false panic=1 nr_cpus=4 agent.use_vsock=true systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket -pidfile /run/vc/vm/b00b4ff14f15714027d76221d316ad778cfda3cb5e104421c5942c2e04adc752/pid -smp 1,cores=1,threads=1,sockets=4,maxcpus=4 1357 pts/2 Ssl+ 0:00 /usr/libexec/kata-containers/kata-shim -agent vsock://1011671291:1024 -container b00b4ff14f15714027d76221d316ad778cfda3cb5e104421c5942c2e04adc752 -exec-id b00b4ff14f15714027d76221d316ad778cfda3cb5e104421c5942c2e04adc752 -terminal It seems that only qemu and kata-shim are in the same netns. Maybe I did something wrong, can you double check? If you want, I can provide you a Fedora VM with new vsock kernel modules. Thanks, Stefano
Hi Stefano I think we'll need a more complex solution [1] (unless someone else has a better patch). Here [1] you can find a version of kata-runtime that supports all namespaces (yes, including mnt namespace!!) unlike upstream kata-runtime Since you want to use it for testing net ns, I just enabled it, but if you need other namespace, just uncomment it out from the list [2] I only tested v1 (no netmom, no vmtemplating) with docker let me know if it works for you [1] - https://github.com/devimc/kata-runtime/tree/nodelete/namespaces [2] - https://github.com/devimc/kata-runtime/blob/556f11ec6e8a5f98973b9023f1746888... ________________________________ From: Stefano Garzarella <sgarzare@redhat.com> Sent: Tuesday, December 17, 2019 6:47 AM To: Montes, Julio <julio.montes@intel.com> Cc: Boeuf, Sebastien <sebastien.boeuf@intel.com>; kata-dev@lists.katacontainers.io <kata-dev@lists.katacontainers.io>; Peng Tao <tao.peng@linux.alibaba.com> Subject: Re: [kata-dev] vsock & network namespaces in Kata Hi Julio, On Mon, Dec 16, 2019 at 09:51:56PM +0000, Montes, Julio wrote:
Hi Stefano
This patch [1] moves the runtime to the sandbox's (qemu) net namespace. This patch is a hotfix that should be used only for testing
Thanks for the patch! I tried it, but I have the same behaviour (v1 doesn't work, v2 works). I did a minimal debugging with the old vsock modules, where v1 works well, checking the netns: $ ip netns cnitest-3c2d7774-b4e1-1901-6c8a-fb20b857898b (id: 0) $ ip netns pids cnitest-3c2d7774-b4e1-1901-6c8a-fb20b857898b 1343 1357 $ ps -p 1357 1343 PID TTY STAT TIME COMMAND 1343 ? Sl 0:21 /usr/bin/qemu-vanilla-system-x86_64 -name sandbox-b00b4ff14f15714027d76221d316ad778cfda3cb5e104421c5942c2e04adc752 -uuid 1cee59f0-ff07-4602-af2e-d25962de412f -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host,pmu=off -qmp unix:/run/vc/vm/b00b4ff14f15714027d76221d316ad778cfda3cb5e104421c5942c2e04adc752/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=8979M -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=true,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/b00b4ff14f15714027d76221d316ad778cfda3cb5e104421c5942c2e04adc752/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/kata-containers/kata-containers-image_clearlinux_1.10.0-alpha1_agent_c87f497312.img,size=134217728 -device virtio-scsi-pci,id=scsi0,disable-modern=true,romfile= -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng,rng=rng0,romfile= -device vhost-vsock-pci,disable-modern=true,vhostfd=3,id=vsock-1011671291,guest-cid=1011671291,romfile= -device virtio-9p-pci,disable-modern=true,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/b00b4ff14f15714027d76221d316ad778cfda3cb5e104421c5942c2e04adc752,security_model=none -netdev tap,id=network-0,vhost=on,vhostfds=4,fds=5 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,disable-modern=true,mq=on,vectors=4,romfile= -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -object memory-backend-ram,id=dimm1,size=2048M -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinuz-4.19.75.55-59.1.container -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro ro rootfstype=ext4 quiet systemd.show_status=false panic=1 nr_cpus=4 agent.use_vsock=true systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket -pidfile /run/vc/vm/b00b4ff14f15714027d76221d316ad778cfda3cb5e104421c5942c2e04adc752/pid -smp 1,cores=1,threads=1,sockets=4,maxcpus=4 1357 pts/2 Ssl+ 0:00 /usr/libexec/kata-containers/kata-shim -agent vsock://1011671291:1024 -container b00b4ff14f15714027d76221d316ad778cfda3cb5e104421c5942c2e04adc752 -exec-id b00b4ff14f15714027d76221d316ad778cfda3cb5e104421c5942c2e04adc752 -terminal It seems that only qemu and kata-shim are in the same netns. Maybe I did something wrong, can you double check? If you want, I can provide you a Fedora VM with new vsock kernel modules. Thanks, Stefano
Hi Julio, On Tue, Dec 17, 2019 at 8:09 PM Montes, Julio <julio.montes@intel.com> wrote:
Hi Stefano
I think we'll need a more complex solution [1] (unless someone else has a better patch).
Here [1] you can find a version of kata-runtime that supports all namespaces (yes, including mnt namespace!!) unlike upstream kata-runtime
Since you want to use it for testing net ns, I just enabled it, but if you need other namespace, just uncomment it out from the list [2]
I only tested v1 (no netmom, no vmtemplating) with docker
Thanks for the patches. I tried it, but unfortunately, it doesn't work with the vsock modules with netns support. This is the log, I don't know if can be useful: https://pastebin.com/mG7CfUrV With the old modules (vsock without netns) it works but looking at the netns I'm still seeing only kata-shim and qemu: $ sudo ip netns pids cnitest-1939e49f-19af-3b72-625d-01db153a9fa9 1308 1321 $ ps -p 1308 1321 PID TTY STAT TIME COMMAND 1308 ? Sl 0:05 /usr/bin/qemu-vanilla-system-x86_64 -name sandbox-efd90c46d5b8989886 1321 pts/2 Ssl+ 0:00 /usr/libexec/kata-containers/kata-shim -agent vsock://357155431:1024 This is the log with working kata and vsock modules without netns support: https://pastebin.com/cdHiV5ZV Let me know if I can do other tests or if you want a VM with the vsock modules with netns support. Thanks, Stefano
Hi Stefano
I tried it, but unfortunately, it doesn't work with the vsock modules with netns support.
that's weird, because with these patches whole process (including threads) is moved to the net namespace Q. Do you know if this new implementation of vsock/netns supports persistent namespaces?
With the old modules (vsock without netns) it works but looking at the netns I'm still seeing only kata-shim and qemu:
you don't see kata-runtime, because it's not running, it spawns to kata-shim and qemu and finish its execution
Let me know if I can do other tests or if you want a VM with the vsock modules with netns support.
yes, please - Julio ________________________________ From: Stefano Garzarella <sgarzare@redhat.com> Sent: Thursday, December 19, 2019 4:49 AM To: Montes, Julio <julio.montes@intel.com> Cc: Boeuf, Sebastien <sebastien.boeuf@intel.com>; kata-dev@lists.katacontainers.io <kata-dev@lists.katacontainers.io>; Peng Tao <tao.peng@linux.alibaba.com> Subject: Re: [kata-dev] vsock & network namespaces in Kata Hi Julio, On Tue, Dec 17, 2019 at 8:09 PM Montes, Julio <julio.montes@intel.com> wrote:
Hi Stefano
I think we'll need a more complex solution [1] (unless someone else has a better patch).
Here [1] you can find a version of kata-runtime that supports all namespaces (yes, including mnt namespace!!) unlike upstream kata-runtime
Since you want to use it for testing net ns, I just enabled it, but if you need other namespace, just uncomment it out from the list [2]
I only tested v1 (no netmom, no vmtemplating) with docker
Thanks for the patches. I tried it, but unfortunately, it doesn't work with the vsock modules with netns support. This is the log, I don't know if can be useful: https://pastebin.com/mG7CfUrV With the old modules (vsock without netns) it works but looking at the netns I'm still seeing only kata-shim and qemu: $ sudo ip netns pids cnitest-1939e49f-19af-3b72-625d-01db153a9fa9 1308 1321 $ ps -p 1308 1321 PID TTY STAT TIME COMMAND 1308 ? Sl 0:05 /usr/bin/qemu-vanilla-system-x86_64 -name sandbox-efd90c46d5b8989886 1321 pts/2 Ssl+ 0:00 /usr/libexec/kata-containers/kata-shim -agent vsock://357155431:1024 This is the log with working kata and vsock modules without netns support: https://pastebin.com/cdHiV5ZV Let me know if I can do other tests or if you want a VM with the vsock modules with netns support. Thanks, Stefano
On Thu, Dec 19, 2019 at 4:25 PM Montes, Julio <julio.montes@intel.com> wrote:
Hi Stefano
I tried it, but unfortunately, it doesn't work with the vsock modules with netns support.
that's weird, because with these patches whole process (including threads) is moved to the net namespace
Q. Do you know if this new implementation of vsock/netns supports persistent namespaces?
With the old modules (vsock without netns) it works but looking at the netns I'm still seeing only kata-shim and qemu:
you don't see kata-runtime, because it's not running, it spawns to kata-shim and qemu and finish its execution
Let me know if I can do other tests or if you want a VM with the vsock modules with netns support.
yes, please
Just an update: I shared with Julio the VM and he found that I was testing the wrong kata-runtime :-( The patched v1 provided by Julio works well with vsock netns :-) So, I'll continue in this way to push netns support upstream and I'll keep you updated. Thanks, Stefano
participants (4)
-
Boeuf, Sebastien
-
Montes, Julio
-
Peng Tao
-
Stefano Garzarella