Re: [kata-dev] update on kata in fedora, and some questions
Hi Cole, see inline for more clarifications. Thanks Marco (PS: I'm using another address, @suse.com doesn't deliver to kata-dev anymore) On Tuesday, September 10, 2019 1:45 AM, Cole Robinson wrote:
Requiring content from a non-distro network source during a package build process is definitely a distro no no, at least for Fedora. Generally builds aren't expected to require network access for one thing
Right now in the existing fedora kata-osbuilder package, the initrd and image are actually built at package install time, so that we can use the host installed kernel and modules, and not have to ship a separate kernel (which is also against Fedora policies). If we pull kata-agent from git at package install time, this means the package has a runtime dep on git and golang, which is not very friendly.
So while indeed kata-agent doesn't really have any usage on the host, the most distro friendly way I can think to handle this is to make kata-agent its own package, install the binary and systemd files in /usr/libexec or somewhere else out of the way, and have the initrd build process access those on the host FS.
This is what I am doing for openSUSE (package link in my first email): - The image package sources include tarballs of both the agent and of the osbuilder source code. - The spec file contains a `BuildRequires: go` line. - The %build step runs the `make initrd` build target, to build the agent and to generate an initrd using dracut. - The %install steps copies the initrd inside the final rpm. - There's not need to run any complex %post step on the host.
For Fedora it seems like /init doesn't matter for the boot process at all and nothing seems to set it by default. Are those strictly required for SUSE? There's probably some set of commands that will work for both
When using the dracut build method, these sets of links are created here: https://github.com/kata-containers/osbuilder/blob/master/rootfs-builder/root... If I recall correctly, I just tried to replicate the set of links created using the non-dracut build method. Feel free to open a PR to improve this and add support for Fedora. BTW, a.t.m. dracut is unit tested with a openSUSE container only, adding Fedora and other would be nice.
* osbuilder-0003-*: This extends the dracut config with the list of kernel driver module names we need in the initrd. Not all virtio devices are listed here because in Fedora some of them are already compiled in (virtio_pci, virtio_rng at least). The list is: 9p 9pnet_virtio vmw_vsock_virtio_transport virtio_console virtio_blk virtio_scsi virtio_net. I presume this will stay downstream only
Nice, I think it makes sense to have a predefined list of virtualization specific kernel modules to add. You can create a PR for that. I am wondering however what could happen if some of these modules are already built-in in the kernel file.
Cool!
From dracut perspective it doesn't seem to have any direct impact. I just tested adding virtio_pci which is built in on Fedora, no complaints. Adding an invalid module name though will make systemd-module-load service report failure, but that's just noise
After those bits are applied, I build osbuilder with:
# cat b.sh IMAGEOUT="$(PWD)/kata-containers-initrd.img" AGENTSRC="/root/go/src/github.com/kata-containers/agent" KVERSION="5.0.9-301.fc30.x86_64" export AGENT_SOURCE_BIN=$AGENTSRC/kata-agent rm -rf dracut_overlay $IMAGEOUT make BUILD_METHOD=dracut DRACUT_KVERSION=$KVERSION initrd
Understandably that KVERSION needs to match a kernel installed on the host, and it needs to match the kernel referenced in configuration.toml. IMAGEOUT is the location of the generated initrd, which is also referenced in /etc/kata-containers/configuration.toml. Then I can test with:
# podman run -it --runtime=/usr/bin/kata-runtime alpine sh
(with selinux disabled on the host though, kata seems to require it for podman usage last I checked).
So with that working, the major pieces that need to be finished so that kata can at least be tested with native Fedora packages are:
* Update the kata-osbuilder package to use the above info * Update all the existing kata-* packages to latest versions * Package kata-agent and get it accepted into Fedora * Finish the kata-runtime package submission and get it accepted into Fedora
After that, the important pieces IMO are:
* Get non-x86 support working in Fedora
* Find a way to package some kind of docker config snippet so 'docker --runtime=kata-runtime' works out of the box. On Fedora the simplest way I found was to add '--add-runtime kata-runtime=path/to/kata-runtime' in /etc/sysconfig/docker OPTIONS=, but that's not really a programmatic operation
* Find a way to get podman to not require selinux=disabled which is a really big hammer. Hopefully it's not hard to make it not error if selinux=permissive which can be changed at runtime.
* Get a minimal qemu build into fedora. Christophe and I have some work on this already: https://github.com/c3d/fedora-qemu-mini https://github.com/crobinso/fedora-qemu-mini
Now some misc questions that I have after poking around at this:
* Is initrd preferred vs image boot? I didn't really try to get image boot working. Should I?
Nope, it's not preferred. But it is packaging friendly.
Yeah thinking about this more it's definitely not memory friendly to use an initrd. I tried the image building and got the fs generating, but the Fedora kernel needs some more drivers built in to make it work without an initrd. I just filed a Fedora kernel bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1750581
* AGENT_INIT=no is the default, meaning /sbin/init == systemd, not /sbin/init == kata-agent. Is that generally the preferred config? When I was flailing around trying to get anything to work I tried AGENT_INIT=yes but didn't get anywhere
Agent init may give you a smaller image size and a faster boot time. However, you will not be able to load initrd kernel modules when using dracut-initrds, so you'll need all the features built into the kernel.
Makes sense
* At first I spent a lot of time trying to get the old style rootfs scripts to work with the Fedora kernel. But I was hitting some confusing situations. The generated initrd fs would have kata agent and the services in them, but nothing was adding it to basic.target.wants; current code in rootfs.sh this only happens if using the dracut method. So understandably kernel+initrd wouldn't work in that case, because nothing was starting the kata-agent... except when I combined that initrd with the kernel from the kata-linux-container package, the kata container would start up fine! Which really perplexes me... does that kernel have some particular knowledge about starting kata-agent or one of the associated services? Or does it trigger some different systemd behavior somehow?
- When the sandbox is started using a regular image, the kernel cmdline includes "systemd.unit=kata-containers.target" See here:
https://github.com/kata-containers/runtime/blob/master/pkg/katautils/create....
- When using initrd instead, it is supposed that the init == agent. However for the dracut generated initrd, the kata-containers.target is added to basic.target.wants. See here:
https://github.com/kata-containers/osbuilder/blob/master/rootfs-builder/root...
Great, thanks for the pointers
Thanks, Cole
On 9/10/19 6:11 AM, Marco Vedovati wrote:
Hi Cole, see inline for more clarifications.
Thanks Marco
(PS: I'm using another address, @suse.com doesn't deliver to kata-dev anymore)
On Tuesday, September 10, 2019 1:45 AM, Cole Robinson wrote:
Requiring content from a non-distro network source during a package build process is definitely a distro no no, at least for Fedora. Generally builds aren't expected to require network access for one thing
Right now in the existing fedora kata-osbuilder package, the initrd and image are actually built at package install time, so that we can use the host installed kernel and modules, and not have to ship a separate kernel (which is also against Fedora policies). If we pull kata-agent from git at package install time, this means the package has a runtime dep on git and golang, which is not very friendly.
So while indeed kata-agent doesn't really have any usage on the host, the most distro friendly way I can think to handle this is to make kata-agent its own package, install the binary and systemd files in /usr/libexec or somewhere else out of the way, and have the initrd build process access those on the host FS.
This is what I am doing for openSUSE (package link in my first email):
- The image package sources include tarballs of both the agent and of the osbuilder source code.
- The spec file contains a `BuildRequires: go` line.
- The %build step runs the `make initrd` build target, to build the agent and to generate an initrd using dracut.
- The %install steps copies the initrd inside the final rpm.
- There's not need to run any complex %post step on the host.
Thanks for the info. I didn't see the package link in your first mail, that's very useful. Your method is definitely simpler, but I do not think it is acceptable per Fedora's guidelines, because the package is shipping a copy of the kernel. Yes it's just a copy of the distribution packaged kernel but from Fedora perspective that doesn't matter. It also has the downside that any time the distro kernel is updated, like for security issues, you will want to ensure the kata-osbuilder package is rebuilt as well so it's not bundling a potentially vulnerable kernel Also I'm curious, what is kernel-kvmsmall? Does it not have modules and everything is built in? Fedora doesn't have anything like that, there's just one build.
For Fedora it seems like /init doesn't matter for the boot process at all and nothing seems to set it by default. Are those strictly required for SUSE? There's probably some set of commands that will work for both
When using the dracut build method, these sets of links are created here:
https://github.com/kata-containers/osbuilder/blob/master/rootfs-builder/root...
If I recall correctly, I just tried to replicate the set of links created using the non-dracut build method.
Feel free to open a PR to improve this and add support for Fedora.
Okay, will do.
BTW, a.t.m. dracut is unit tested with a openSUSE container only, adding Fedora and other would be nice.
Sure I'll take a look at the tests
* osbuilder-0003-*: This extends the dracut config with the list of kernel driver module names we need in the initrd. Not all virtio devices are listed here because in Fedora some of them are already compiled in (virtio_pci, virtio_rng at least). The list is: 9p 9pnet_virtio vmw_vsock_virtio_transport virtio_console virtio_blk virtio_scsi virtio_net. I presume this will stay downstream only
Nice, I think it makes sense to have a predefined list of virtualization specific kernel modules to add. You can create a PR for that. I am wondering however what could happen if some of these modules are already built-in in the kernel file.
Cool!
From dracut perspective it doesn't seem to have any direct impact. I just tested adding virtio_pci which is built in on Fedora, no complaints. Adding an invalid module name though will make systemd-module-load service report failure, but that's just noise
I will send a PR too extending the drivers= list, maybe we can get away with having a shared list and we don't need to differentiate based on host distro. Thanks, Cole
On 9/10/19 4:37 PM, Cole Robinson wrote:
On 9/10/19 6:11 AM, Marco Vedovati wrote:
Hi Cole, see inline for more clarifications.
Thanks Marco
(PS: I'm using another address, @suse.com doesn't deliver to kata-dev anymore)
On Tuesday, September 10, 2019 1:45 AM, Cole Robinson wrote:
Requiring content from a non-distro network source during a package build process is definitely a distro no no, at least for Fedora. Generally builds aren't expected to require network access for one thing
Right now in the existing fedora kata-osbuilder package, the initrd and image are actually built at package install time, so that we can use the host installed kernel and modules, and not have to ship a separate kernel (which is also against Fedora policies). If we pull kata-agent from git at package install time, this means the package has a runtime dep on git and golang, which is not very friendly.
So while indeed kata-agent doesn't really have any usage on the host, the most distro friendly way I can think to handle this is to make kata-agent its own package, install the binary and systemd files in /usr/libexec or somewhere else out of the way, and have the initrd build process access those on the host FS.
This is what I am doing for openSUSE (package link in my first email):
- The image package sources include tarballs of both the agent and of the osbuilder source code.
- The spec file contains a `BuildRequires: go` line.
- The %build step runs the `make initrd` build target, to build the agent and to generate an initrd using dracut.
- The %install steps copies the initrd inside the final rpm.
- There's not need to run any complex %post step on the host.
Thanks for the info. I didn't see the package link in your first mail, that's very useful.
Your method is definitely simpler, but I do not think it is acceptable per Fedora's guidelines, because the package is shipping a copy of the kernel. Yes it's just a copy of the distribution packaged kernel but from Fedora perspective that doesn't matter. It also has the downside that any time the distro kernel is updated, like for security issues, you will want to ensure the kata-osbuilder package is rebuilt as well so it's not bundling a potentially vulnerable kernel
Also I'm curious, what is kernel-kvmsmall? Does it not have modules and everything is built in? Fedora doesn't have anything like that, there's just one build.
kernel-kvmsmall is a KVM virtualization oriented kernel "flavor", that is minimal but comes with most of the virtio stuff set to "=y". I decided to vendor in the kernel binary inside the kata package, because having a dependency on the kernel package was giving some side effects (i.e., in the case of kvmsmall, if you install it on the host then it's used to boot the system). Finally, on OBS, the kata package is automatically rebuilt every time a kernel rebuild is detected.
For Fedora it seems like /init doesn't matter for the boot process at all and nothing seems to set it by default. Are those strictly required for SUSE? There's probably some set of commands that will work for both
When using the dracut build method, these sets of links are created here:
https://github.com/kata-containers/osbuilder/blob/master/rootfs-builder/root...
If I recall correctly, I just tried to replicate the set of links created using the non-dracut build method.
Feel free to open a PR to improve this and add support for Fedora.
Okay, will do.
BTW, a.t.m. dracut is unit tested with a openSUSE container only, adding Fedora and other would be nice.
Sure I'll take a look at the tests
* osbuilder-0003-*: This extends the dracut config with the list of kernel driver module names we need in the initrd. Not all virtio devices are listed here because in Fedora some of them are already compiled in (virtio_pci, virtio_rng at least). The list is: 9p 9pnet_virtio vmw_vsock_virtio_transport virtio_console virtio_blk virtio_scsi virtio_net. I presume this will stay downstream only
Nice, I think it makes sense to have a predefined list of virtualization specific kernel modules to add. You can create a PR for that. I am wondering however what could happen if some of these modules are already built-in in the kernel file.
Cool!
From dracut perspective it doesn't seem to have any direct impact. I just tested adding virtio_pci which is built in on Fedora, no complaints. Adding an invalid module name though will make systemd-module-load service report failure, but that's just noise
I will send a PR too extending the drivers= list, maybe we can get away with having a shared list and we don't need to differentiate based on host distro.
Thank you! Marco
Thanks, Cole
participants (2)
-
Cole Robinson
-
Marco Vedovati