Kata on Fedora experiment (modular kernel)
Hi, I wanted to share some thoughts after running Kata on Fedora. I built Kata from source and used the Fedora rootfs together with a vanilla modular kernel. This configuration is similar to what I imagine official Fedora RPMs would eventually use, because they probably want to use a stock Fedora kernel instead of building a special monolithic kernel for Kata.
The Fedora rootfs was built like this:
# KERNEL_MODULES_DIR=/lib/modules/$(uname -r) OS_VERSION=28 ./rootfs.sh fedora
My understanding is Kata wants *either* an initramfs or a disk image. Modular kernels typically need to load drivers before they can access the disk, so I opted for the initramfs even though it is several hundred megabytes. The initramfs could probably be shrunk significantly with some work on the Fedora rootfs script.
This configuration required several tweaks to get Kata running. I'm not going to create a patch series or send a pull request, because at this stage these are dirty hacks that cannot be merged. But it's likely that other people will encounter these issues, so I'm including them below.
Stefan
---
* kata-agent assumes initramfs is equivalent to running the agent as init. In our case kata-agent *is* in an initramfs but *not* init, so it calculates the wrong value for noPivotRoot:
diff --git a/agent.go b/agent.go index ec01250..182c2de 100644 --- a/agent.go +++ b/agent.go @@ -830,7 +832,7 @@ func realMain() { running: false, // pivot_root won't work for init, see // Documention/filesystem/ramfs-rootfs-initramfs.txt - noPivotRoot: os.Getpid() == 1, + noPivotRoot: true, // TODO check initramfs, not pid=1! kata-agent might have been started by systemd form an initramfs os.Getpid() == 1, subreaper: r, pciDeviceMap: make(map[string]string), deviceWatchers: make(map[string](chan string)),
* kata-agent assumes AF_VSOCK should be used if the guest kernel has vsock support. A generic kernel like the Fedora kernel will have AF_VSOCK but that doesn't mean the virtio-serial device should be ignored. Perhaps change the check to see if /dev/virtio-ports/agent.channel.0 is present?
diff --git a/channel.go b/channel.go index b4f8977..34ef292 100644 --- a/channel.go +++ b/channel.go @@ -29,25 +29,29 @@ type channel interface { }
func newChannel() (channel, error) { + // This is broken, it ignores the virtio-serial agent.channel.0 device + // in favor of AF_VSOCK support, even on systems that are not going to + // use AF_VSOCK but happen to have the drivers installed. + // Check for vsock support. - vSockSupported, err := isAFVSockSupported() - if err != nil { - return nil, err - } - - if vSockSupported { - // Check if vsock socket exists. We want to cover the case - // where the guest OS can support vsock, but the runtime is - // still using virtio serial connection. - exist, err := vSockPathExist() - if err != nil { - return nil, err - } - - if exist { - return &vSockChannel{}, nil - } - } +// vSockSupported, err := isAFVSockSupported() +// if err != nil { +// return nil, err +// } +// +// if vSockSupported { +// // Check if vsock socket exists. We want to cover the case +// // where the guest OS can support vsock, but the runtime is +// // still using virtio serial connection. +// exist, err := vSockPathExist() +// if err != nil { +// return nil, err +// } +// +// if exist { +// return &vSockChannel{}, nil +// } +// }
return &serialChannel{}, nil }
* The Fedora initramfs and modular kernel rely on udev to load the virtio_console.ko driver. Make sure to start kata-agent *after* the agent device appears. Note this change will break AF_VSOCK if you choose to use that instead of virtio-serial!
diff --git a/kata-agent.service.in b/kata-agent.service.in index 861757d..5395047 100644 --- a/kata-agent.service.in +++ b/kata-agent.service.in @@ -2,6 +2,8 @@ Description=Kata Containers Agent Documentation=https://github.com/kata-containers/agent Wants=kata-containers.target +BindsTo=dev-virtio\x2dports-agent.channel.0.device +After=dev-virtio\x2dports-agent.channel.0.device
[Service] # Send agent output to tty to allow capture debug logs
* Make sure the rootfs has modprobe and udev. Needed by the modular kernel.
diff --git a/rootfs-builder/fedora/config.sh b/rootfs-builder/fedora/config.sh index a0ca15a..19fbc7b 100644 --- a/rootfs-builder/fedora/config.sh +++ b/rootfs-builder/fedora/config.sh @@ -14,4 +14,4 @@ PACKAGES="iptables" #Optional packages: # systemd: An init system that will start kata-agent if kata-agent # itself is not configured as init process. -[ "$AGENT_INIT" == "no" ] && PACKAGES+=" systemd" || true +[ "$AGENT_INIT" == "no" ] && PACKAGES+=" systemd systemd-udev" || true
Hi Stefan,
Nice experiment. I notice that you got some trouble with the vsock code, thanks for reporting this. We're planning to add a proper support of vsock pretty soon because it's clearly not there yet.
Thanks, Sebastien ________________________________________ From: Stefan Hajnoczi [stefanha@redhat.com] Sent: Friday, June 29, 2018 10:42 AM To: kata-dev@lists.katacontainers.io Cc: Lokesh Mandvekar Subject: [kata-dev] Kata on Fedora experiment (modular kernel)
Hi, I wanted to share some thoughts after running Kata on Fedora. I built Kata from source and used the Fedora rootfs together with a vanilla modular kernel. This configuration is similar to what I imagine official Fedora RPMs would eventually use, because they probably want to use a stock Fedora kernel instead of building a special monolithic kernel for Kata.
The Fedora rootfs was built like this:
# KERNEL_MODULES_DIR=/lib/modules/$(uname -r) OS_VERSION=28 ./rootfs.sh fedora
My understanding is Kata wants *either* an initramfs or a disk image. Modular kernels typically need to load drivers before they can access the disk, so I opted for the initramfs even though it is several hundred megabytes. The initramfs could probably be shrunk significantly with some work on the Fedora rootfs script.
This configuration required several tweaks to get Kata running. I'm not going to create a patch series or send a pull request, because at this stage these are dirty hacks that cannot be merged. But it's likely that other people will encounter these issues, so I'm including them below.
Stefan
---
* kata-agent assumes initramfs is equivalent to running the agent as init. In our case kata-agent *is* in an initramfs but *not* init, so it calculates the wrong value for noPivotRoot:
diff --git a/agent.go b/agent.go index ec01250..182c2de 100644 --- a/agent.go +++ b/agent.go @@ -830,7 +832,7 @@ func realMain() { running: false, // pivot_root won't work for init, see // Documention/filesystem/ramfs-rootfs-initramfs.txt - noPivotRoot: os.Getpid() == 1, + noPivotRoot: true, // TODO check initramfs, not pid=1! kata-agent might have been started by systemd form an initramfs os.Getpid() == 1, subreaper: r, pciDeviceMap: make(map[string]string), deviceWatchers: make(map[string](chan string)),
* kata-agent assumes AF_VSOCK should be used if the guest kernel has vsock support. A generic kernel like the Fedora kernel will have AF_VSOCK but that doesn't mean the virtio-serial device should be ignored. Perhaps change the check to see if /dev/virtio-ports/agent.channel.0 is present?
diff --git a/channel.go b/channel.go index b4f8977..34ef292 100644 --- a/channel.go +++ b/channel.go @@ -29,25 +29,29 @@ type channel interface { }
func newChannel() (channel, error) { + // This is broken, it ignores the virtio-serial agent.channel.0 device + // in favor of AF_VSOCK support, even on systems that are not going to + // use AF_VSOCK but happen to have the drivers installed. + // Check for vsock support. - vSockSupported, err := isAFVSockSupported() - if err != nil { - return nil, err - } - - if vSockSupported { - // Check if vsock socket exists. We want to cover the case - // where the guest OS can support vsock, but the runtime is - // still using virtio serial connection. - exist, err := vSockPathExist() - if err != nil { - return nil, err - } - - if exist { - return &vSockChannel{}, nil - } - } +// vSockSupported, err := isAFVSockSupported() +// if err != nil { +// return nil, err +// } +// +// if vSockSupported { +// // Check if vsock socket exists. We want to cover the case +// // where the guest OS can support vsock, but the runtime is +// // still using virtio serial connection. +// exist, err := vSockPathExist() +// if err != nil { +// return nil, err +// } +// +// if exist { +// return &vSockChannel{}, nil +// } +// }
return &serialChannel{}, nil }
* The Fedora initramfs and modular kernel rely on udev to load the virtio_console.ko driver. Make sure to start kata-agent *after* the agent device appears. Note this change will break AF_VSOCK if you choose to use that instead of virtio-serial!
diff --git a/kata-agent.service.in b/kata-agent.service.in index 861757d..5395047 100644 --- a/kata-agent.service.in +++ b/kata-agent.service.in @@ -2,6 +2,8 @@ Description=Kata Containers Agent Documentation=https://github.com/kata-containers/agent Wants=kata-containers.target +BindsTo=dev-virtio\x2dports-agent.channel.0.device +After=dev-virtio\x2dports-agent.channel.0.device
[Service] # Send agent output to tty to allow capture debug logs
* Make sure the rootfs has modprobe and udev. Needed by the modular kernel.
diff --git a/rootfs-builder/fedora/config.sh b/rootfs-builder/fedora/config.sh index a0ca15a..19fbc7b 100644 --- a/rootfs-builder/fedora/config.sh +++ b/rootfs-builder/fedora/config.sh @@ -14,4 +14,4 @@ PACKAGES="iptables" #Optional packages: # systemd: An init system that will start kata-agent if kata-agent # itself is not configured as init process. -[ "$AGENT_INIT" == "no" ] && PACKAGES+=" systemd" || true +[ "$AGENT_INIT" == "no" ] && PACKAGES+=" systemd systemd-udev" || true
Hi Stefan,
Thanks for sharing your experiments. Your approach to integrate Kata shows some points improve to support this use case.
I did not consider in the past use the host kernel as the kata kernel but definitely is a use case we could support.
A couple of questions related on how to mange the kernel and image when you add Fedora RPMs.
Are you going to add a symlink to allow the kata runtime always find the latest installed kata kernel ?
How are you planning to update the modules if a kernel changes ? The image will by generated after a new kernel update? Or every time the kernel changes you are going to update the kernel from package?
On Fri, Jun 29, 2018 at 06:42:27PM +0100, Stefan Hajnoczi wrote:
Hi, I wanted to share some thoughts after running Kata on Fedora. I built Kata from source and used the Fedora rootfs together with a vanilla modular kernel. This configuration is similar to what I imagine official Fedora RPMs would eventually use, because they probably want to use a stock Fedora kernel instead of building a special monolithic kernel for Kata.
The Fedora rootfs was built like this:
# KERNEL_MODULES_DIR=/lib/modules/$(uname -r) OS_VERSION=28 ./rootfs.sh fedora
My understanding is Kata wants *either* an initramfs or a disk image. Modular kernels typically need to load drivers before they can access the disk, so I opted for the initramfs even though it is several hundred megabytes. The initramfs could probably be shrunk significantly with some work on the Fedora rootfs script.
This configuration required several tweaks to get Kata running. I'm not going to create a patch series or send a pull request, because at this stage these are dirty hacks that cannot be merged. But it's likely that other people will encounter these issues, so I'm including them below.
Stefan
- kata-agent assumes initramfs is equivalent to running the agent as init. In our case kata-agent *is* in an initramfs but *not* init, so it calculates the wrong value for noPivotRoot:
diff --git a/agent.go b/agent.go index ec01250..182c2de 100644 --- a/agent.go +++ b/agent.go @@ -830,7 +832,7 @@ func realMain() { running: false, // pivot_root won't work for init, see // Documention/filesystem/ramfs-rootfs-initramfs.txt
noPivotRoot: os.Getpid() == 1,
noPivotRoot: true, // TODO check initramfs, not pid=1! kata-agent might have been started by systemd form an initramfs os.Getpid() == 1, subreaper: r, pciDeviceMap: make(map[string]string), deviceWatchers: make(map[string](chan string)),
- kata-agent assumes AF_VSOCK should be used if the guest kernel has vsock support. A generic kernel like the Fedora kernel will have AF_VSOCK but that doesn't mean the virtio-serial device should be ignored. Perhaps change the check to see if /dev/virtio-ports/agent.channel.0 is present?
diff --git a/channel.go b/channel.go index b4f8977..34ef292 100644 --- a/channel.go +++ b/channel.go @@ -29,25 +29,29 @@ type channel interface { }
func newChannel() (channel, error) {
// This is broken, it ignores the virtio-serial agent.channel.0 device// in favor of AF_VSOCK support, even on systems that are not going to// use AF_VSOCK but happen to have the drivers installed.// Check for vsock support.
vSockSupported, err := isAFVSockSupported()if err != nil {return nil, err}if vSockSupported {// Check if vsock socket exists. We want to cover the case// where the guest OS can support vsock, but the runtime is// still using virtio serial connection.exist, err := vSockPathExist()if err != nil {return nil, err}if exist {return &vSockChannel{}, nil}}+// vSockSupported, err := isAFVSockSupported() +// if err != nil { +// return nil, err +// } +// +// if vSockSupported { +// // Check if vsock socket exists. We want to cover the case +// // where the guest OS can support vsock, but the runtime is +// // still using virtio serial connection. +// exist, err := vSockPathExist() +// if err != nil { +// return nil, err +// } +// +// if exist { +// return &vSockChannel{}, nil +// } +// }
return &serialChannel{}, nil}
- The Fedora initramfs and modular kernel rely on udev to load the virtio_console.ko driver. Make sure to start kata-agent *after* the agent device appears. Note this change will break AF_VSOCK if you choose to use that instead of virtio-serial!
diff --git a/kata-agent.service.in b/kata-agent.service.in index 861757d..5395047 100644 --- a/kata-agent.service.in +++ b/kata-agent.service.in @@ -2,6 +2,8 @@ Description=Kata Containers Agent Documentation=https://github.com/kata-containers/agent Wants=kata-containers.target +BindsTo=dev-virtio\x2dports-agent.channel.0.device +After=dev-virtio\x2dports-agent.channel.0.device
[Service] # Send agent output to tty to allow capture debug logs
- Make sure the rootfs has modprobe and udev. Needed by the modular kernel.
diff --git a/rootfs-builder/fedora/config.sh b/rootfs-builder/fedora/config.sh index a0ca15a..19fbc7b 100644 --- a/rootfs-builder/fedora/config.sh +++ b/rootfs-builder/fedora/config.sh @@ -14,4 +14,4 @@ PACKAGES="iptables" #Optional packages: # systemd: An init system that will start kata-agent if kata-agent # itself is not configured as init process. -[ "$AGENT_INIT" == "no" ] && PACKAGES+=" systemd" || true +[ "$AGENT_INIT" == "no" ] && PACKAGES+=" systemd systemd-udev" || true
kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
On Fri, Jun 29, 2018 at 01:28:55PM -0500, Jose Carlos Venegas Munoz wrote:
Thanks for sharing your experiments. Your approach to integrate Kata shows some points improve to support this use case.
I did not consider in the past use the host kernel as the kata kernel but definitely is a use case we could support.
A couple of questions related on how to mange the kernel and image when you add Fedora RPMs.
Lokesh is working on Fedora RPMs. I will leave the questions to Lokesh, except to say that reusing kernel and QEMU packages is attractive because it means fewer builds, fewer packages, and a smaller test matrix.
Are you going to add a symlink to allow the kata runtime always find the latest installed kata kernel ?
How are you planning to update the modules if a kernel changes ? The image will by generated after a new kernel update? Or every time the kernel changes you are going to update the kernel from package?
Stefan
Can a stock kernel with all the probing still intact ever perform as well as a custom one with all the non relevant stuff stripped out? Subsecond pod starting is very handy.
Thanks, Kevin ________________________________________ From: Stefan Hajnoczi [stefanha@redhat.com] Sent: Monday, July 02, 2018 9:13 AM To: Jose Carlos Venegas Munoz Cc: Lokesh Mandvekar; kata-dev@lists.katacontainers.io Subject: Re: [kata-dev] Kata on Fedora experiment (modular kernel)
On Fri, Jun 29, 2018 at 01:28:55PM -0500, Jose Carlos Venegas Munoz wrote:
Thanks for sharing your experiments. Your approach to integrate Kata shows some points improve to support this use case.
I did not consider in the past use the host kernel as the kata kernel but definitely is a use case we could support.
A couple of questions related on how to mange the kernel and image when you add Fedora RPMs.
Lokesh is working on Fedora RPMs. I will leave the questions to Lokesh, except to say that reusing kernel and QEMU packages is attractive because it means fewer builds, fewer packages, and a smaller test matrix.
Are you going to add a symlink to allow the kata runtime always find the latest installed kata kernel ?
How are you planning to update the modules if a kernel changes ? The image will by generated after a new kernel update? Or every time the kernel changes you are going to update the kernel from package?
Stefan
* Fox, Kevin M (Kevin.Fox@pnnl.gov) wrote:
Can a stock kernel with all the probing still intact ever perform as well as a custom one with all the non relevant stuff stripped out? Subsecond pod starting is very handy.
It's an interesting question; hopefully it can get close if we: a) Configure the virtual hardware appropriately b) Pass kernel command line options to tell the probes to get out of the way c) Maybe pass firmware/ACPI data for the same type of thing. (Not investigated)
Dave
Thanks, Kevin ________________________________________ From: Stefan Hajnoczi [stefanha@redhat.com] Sent: Monday, July 02, 2018 9:13 AM To: Jose Carlos Venegas Munoz Cc: Lokesh Mandvekar; kata-dev@lists.katacontainers.io Subject: Re: [kata-dev] Kata on Fedora experiment (modular kernel)
On Fri, Jun 29, 2018 at 01:28:55PM -0500, Jose Carlos Venegas Munoz wrote:
Thanks for sharing your experiments. Your approach to integrate Kata shows some points improve to support this use case.
I did not consider in the past use the host kernel as the kata kernel but definitely is a use case we could support.
A couple of questions related on how to mange the kernel and image when you add Fedora RPMs.
Lokesh is working on Fedora RPMs. I will leave the questions to Lokesh, except to say that reusing kernel and QEMU packages is attractive because it means fewer builds, fewer packages, and a smaller test matrix.
Are you going to add a symlink to allow the kata runtime always find the latest installed kata kernel ?
How are you planning to update the modules if a kernel changes ? The image will by generated after a new kernel update? Or every time the kernel changes you are going to update the kernel from package?
Stefan
kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
-- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
On Mon, Jul 02, 2018 at 07:32:13PM +0100, Dr. David Alan Gilbert wrote:
- Fox, Kevin M (Kevin.Fox@pnnl.gov) wrote:
Can a stock kernel with all the probing still intact ever perform as well as a custom one with all the non relevant stuff stripped out? Subsecond pod starting is very handy.
I haven't benchmarked it. What is the command-line to start a second container in the existing VM? By default Kata launches a new VM.
It's an interesting question; hopefully it can get close if we: a) Configure the virtual hardware appropriately
kata-runtime already does this in the sense that only necessary devices are configured.
b) Pass kernel command line options to tell the probes to get out of the way
kata-runtime already does this. Here is the kernel command-line:
tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug panic=1 initcall_debug nr_cpus=4 ip=::::::<blob>::off::
initcall_debug is a strange one since it can slow things down by printing lots of output to the console. I haven't measured how much difference it makes though.
c) Maybe pass firmware/ACPI data for the same type of thing. (Not investigated)
Firmware startup time can definitely be reduced. Up until now people have custom-compiled SeaBIOS to get great startup times, but similar times could be achieved with runtime detection.
Stefan
On Mon, Jul 02, 2018 at 07:32:13PM +0100, Dr. David Alan Gilbert wrote:
- Fox, Kevin M (Kevin.Fox@pnnl.gov) wrote:
Can a stock kernel with all the probing still intact ever perform as well as a
custom one with all the non relevant stuff stripped out? Subsecond pod starting is very handy.
I haven't benchmarked it. What is the command-line to start a second container in the existing VM? By default Kata launches a new VM.
I think Alan was saying 'boot times <1s', not 'a second boot', if I'm interpreting you both correctly... (an aside - afaik, you need to use k8s to launch a second container in a VM - that is two containers in a pod (sandbox in Kata). You cannot do this from Docker afaik).
initcall_debug nr_cpus=4 ip=::::::<blob>::off::
initcall_debug is a strange one since it can slow things down by printing lots of output to the console. I haven't measured how much difference it makes though.
Hah, I suspect initcall_debug is actually a hangover from measuring and trying to reduce boot-time. I suspect that one can go - but I've never measured if it makes a difference.
Whilst I'm here - one more thought for boot time - some time back I measured that booting with vcpus=2 is quicker than booting with vcpus=1 (my theory is the kernel can then parallelize the init call sequences for instance. And, in fact, iirc, vcpus=2 was the optimal, as if you add more vcpus they then take more time for initialization). I think we moved to having a default of vcpus=1 when we went to hotplug vcpus - it might be worth measuring and considering if a default of vcpus=2 gives us a boot win - but it will likely add a little complexity to the hotplug math (/cc Julio)
Graham
--------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
On Tue, Jul 03, 2018 at 08:55:00AM +0000, Whaley, Graham wrote:
On Mon, Jul 02, 2018 at 07:32:13PM +0100, Dr. David Alan Gilbert wrote:
- Fox, Kevin M (Kevin.Fox@pnnl.gov) wrote:
Can a stock kernel with all the probing still intact ever perform as well as a
custom one with all the non relevant stuff stripped out? Subsecond pod starting is very handy.
I haven't benchmarked it. What is the command-line to start a second container in the existing VM? By default Kata launches a new VM.
I think Alan was saying 'boot times <1s', not 'a second boot', if I'm interpreting you both correctly... (an aside - afaik, you need to use k8s to launch a second container in a VM - that is two containers in a pod (sandbox in Kata). You cannot do this from Docker afaik).
Okay, thanks for the k8s hint. I was asking how to launch another container inside a sandbox because I wasn't aware of a way to do it using Docker.
Stefan
participants (6)
-
Boeuf, Sebastien
-
Dr. David Alan Gilbert
-
Fox, Kevin M
-
Jose Carlos Venegas Munoz
-
Stefan Hajnoczi
-
Whaley, Graham