Open Stack

On 9/9/19 12:09 PM, Marco Vedovati wrote:
> Hi Cole,
> it's nice to see more distros trying to have Kata pkgs!
> 
> I can leave you some feedbacks inline, based on the work I did to
> package Kata for openSUSE [1], and since I worked on adding support for
> dracut in osbuilder (based on the initial work of Stefan Hajnoczi).
> 
> Cheers
> Marco
> 
> [1]:
> https://build.opensuse.org/package/show/openSUSE:Factory/katacontainers
> https://build.opensuse.org/package/show/openSUSE:Factory/katacontainers-image-initrd
> 

Thank you for the dracut work! It will simplify things on the Fedora
side quite a bit :)

> 
> On 9/8/19 3:10 AM, Cole Robinson wrote:
>> Hi kata folks. I want to give an update on the state of kata in Fedora,
>> discuss some of the remaining pieces, and ask some questions
>>
>> There's already some pieces of kata in Fedora: kata-ksm-throttler,
>> kata-proxy, kata-shim, kata-osbuilder. They were added around June 2018
>> by Lokesh Mandvekar. But they are quite out of date at this point, and
>> obviously not that useful because kata-runtime never made it into the
>> distro. The kata-osbuilder package does generate a Fedora initrd and
>> image using the distro specific scripts (not dracut), but I suspect they
>> never worked with the Fedora kernel at least.
>>
>> I spent the past two days getting the osbuilder dracut support to work
>> with the Fedora kernel. I now have the latest OBS kata packages working
>> on Fedora 30 with a stock Fedora kernel, dracut built initrd, and stock
>> Fedora qemu 4.1 build (from the virt-preview repo). Here's the changes I
>> added, see the attached files
>>
>> * configuration.qemu.diff: Changes I made to
>> /etc/kata-containers/configuration.toml compared to
>> /usr/share/defaults/kata-containers/configuration.toml. Besides changing
>> the qemu, kernel, and initrd path as mentioned above, I turned on q35
>> and vsock too; pc and virtio-serial were working as well, but this is
>> likely the config we will target for Fedora packaging
> 
> Probably you can keep this changes downstream, and have it done by the
> spec file.
> 

Make sense

>> * agent-0001-*.patch: Make kata-agent.service run after
>> systemd-udev-settle.service. At one point in my experimenting I needed
>> this: kata would communicate with the agent, but networking in the guest
>> didn't seem to be set up yet, and configuring eth0 was failing. However
>> if I revert this now it doesn't seem to cause any failures. This may be
>> due to /etc/modules-load.d/kata-modules.conf loading the necessary bits
>> early enough, but that may still be race-y. I figured I'd mention it for
>> posterity.
> 
> I recon there may be a race between the two services, logically
> kata-agent should be started after the kernel modules are loaded. I have
> opened a new issue here (contributions are welcome :D ):
> https://github.com/kata-containers/osbuilder/issues/353
> 

Thanks, I added a comment there

> 
>> * osbuilder-0001-*: This is a hack. If you use
>> AGENT_SOURCE_BIN=SOMEDIR/kata-agent, rootfs.sh will now use SOMEDIR as
>> the source for installing kata-agent.service and kata-containers.target.
>> We will want something like this in Fedora where we will likely have a
>> standalone kata-agent package shipping a prebuilt kata-agent. We
>> shouldn't have to depend on git and go compilation to build the
>> kata-agent at image generation time. Suggestions welcome on how to
>> implement this in an upstream suitable way
> 
> Why you don't want to build the kata-agent at image generation time?
> kata-agent does not need to be a standalone package, since it should not
> be installed on the host.
> 

Requiring content from a non-distro network source during a package
build process is definitely a distro no no, at least for Fedora.
Generally builds aren't expected to require network access for one thing

Right now in the existing fedora kata-osbuilder package, the initrd and
image are actually built at package install time, so that we can use the
host installed kernel and modules, and not have to ship a separate
kernel (which is also against Fedora policies). If we pull kata-agent
from git at package install time, this means the package has a runtime
dep on git and golang, which is not very friendly.

So while indeed kata-agent doesn't really have any usage on the host,
the most distro friendly way I can think to handle this is to make
kata-agent its own package, install the binary and systemd files in
/usr/libexec or somewhere else out of the way, and have the initrd build
process access those on the host FS.

>> * osbuilder-0002-*: This fixes the initrd /sbin/init symlinking for me.
>> The current code leaves /init, /sbin/init, and /usr/lib/systemd/systemd
>> all as symlinks pointing to each other. Maybe this works on another
>> distro but it didn't work on Fedora. Or maybe this is just a bug
> 
> This is not what is happening for me:
> init -> usr/lib/systemd/systemd
> /sbin/init -> ../usr/lib/systemd/systemd
> /usr/lib/systemd/systemd
> 

With current git, the overlay has

  /init -> ./usr/lib/systemd/systemd
  /lib/systemd/systemd -> ../../init
  /sbin/init -> ../init

But when this is serialized into the initrd, on Fedora at least, /lib is
a symlink to /usr/lib. So in practice the second symlink overwrites the
systemd binary at /usr/lib/systemd/system.

For Fedora it seems like /init doesn't matter for the boot process at
all and nothing seems to set it by default. Are those strictly required
for SUSE? There's probably some set of commands that will work for both

>> * osbuilder-0003-*: This extends the dracut config with the list of
>> kernel driver module names we need in the initrd. Not all virtio devices
>> are listed here because in Fedora some of them are already compiled in
>> (virtio_pci, virtio_rng at least). The list is: 9p 9pnet_virtio
>> vmw_vsock_virtio_transport virtio_console virtio_blk virtio_scsi
>> virtio_net. I presume this will stay downstream only
> 
> Nice, I think it makes sense to have a predefined list of virtualization
> specific kernel modules to add. You can create a PR for that.
> I am wondering however what could happen if some of these modules are
> already built-in in the kernel file.
> 

>From dracut perspective it doesn't seem to have any direct impact. I
just tested adding virtio_pci which is built in on Fedora, no
complaints. Adding an invalid module name though will make
systemd-module-load service report failure, but that's just noise

>> After those bits are applied, I build osbuilder with:
>>
>> # cat b.sh
>> IMAGEOUT="$(PWD)/kata-containers-initrd.img"
>> AGENTSRC="/root/go/src/github.com/kata-containers/agent"
>> KVERSION="5.0.9-301.fc30.x86_64"
>> export AGENT_SOURCE_BIN=$AGENTSRC/kata-agent
>> rm -rf dracut_overlay $IMAGEOUT
>> make BUILD_METHOD=dracut DRACUT_KVERSION=$KVERSION initrd
>>
>>
>> Understandably that KVERSION needs to match a kernel installed on the
>> host, and it needs to match the kernel referenced in configuration.toml.
>> IMAGEOUT is the location of the generated initrd, which is also
>> referenced in /etc/kata-containers/configuration.toml. Then I can test with:
>>
>> # podman run -it --runtime=/usr/bin/kata-runtime alpine sh
>>
>> (with selinux disabled on the host though, kata seems to require it for
>> podman usage last I checked).
>>
>>
>> So with that working, the major pieces that need to be finished so that
>> kata can at least be tested with native Fedora packages are:
>>
>> * Update the kata-osbuilder package to use the above info
>> * Update all the existing kata-* packages to latest versions
>> * Package kata-agent and get it accepted into Fedora
>> * Finish the kata-runtime package submission and get it accepted into Fedora
>>
>> After that, the important pieces IMO are:
>>
>> * Get non-x86 support working in Fedora
>>
>> * Find a way to package some kind of docker config snippet so 'docker
>> --runtime=kata-runtime' works out of the box. On Fedora the simplest way
>> I found was to add '--add-runtime kata-runtime=path/to/kata-runtime' in
>> /etc/sysconfig/docker OPTIONS=, but that's not really a programmatic
>> operation
>>
>> * Find a way to get podman to not require selinux=disabled which is a
>> really big hammer. Hopefully it's not hard to make it not error if
>> selinux=permissive which can be changed at runtime.
>>
>> * Get a minimal qemu build into fedora. Christophe and I have some work
>> on this already: https://github.com/c3d/fedora-qemu-mini
>> https://github.com/crobinso/fedora-qemu-mini
>>
>>
>> Now some misc questions that I have after poking around at this:
>>
>> * Is initrd preferred vs image boot? I didn't really try to get image
>> boot working. Should I?
> 
> Nope, it's not preferred. But it is packaging friendly.
> 

Yeah thinking about this more it's definitely not memory friendly to use
an initrd. I tried the image building and got the fs generating, but the
Fedora kernel needs some more drivers built in to make it work without
an initrd. I just filed a Fedora kernel bug:

https://bugzilla.redhat.com/show_bug.cgi?id=1750581

>> * AGENT_INIT=no is the default, meaning /sbin/init == systemd, not
>> /sbin/init == kata-agent. Is that generally the preferred config? When I
>> was flailing around trying to get anything to work I tried
>> AGENT_INIT=yes but didn't get anywhere
> 
> Agent init may give you a smaller image size and a faster boot time.
> However, you will not be able to load initrd kernel modules when using
> dracut-initrds, so you'll need all the features built into the kernel.
> 

Makes sense

>> * At first I spent a lot of time trying to get the old style rootfs
>> scripts to work with the Fedora kernel. But I was hitting some confusing
>> situations. The generated initrd fs would have kata agent and the
>> services in them, but nothing was adding it to basic.target.wants;
>> current code in rootfs.sh this only happens if using the dracut method.
>> So understandably kernel+initrd wouldn't work in that case, because
>> nothing was starting the kata-agent... except when I combined that
>> initrd with the kernel from the kata-linux-container package, the kata
>> container would start up fine! Which really perplexes me... does that
>> kernel have some particular knowledge about starting kata-agent or one
>> of the associated services? Or does it trigger some different systemd
>> behavior somehow?
> 
> 
> - When the sandbox is started using a regular image, the kernel cmdline
> includes "systemd.unit=kata-containers.target"
> See here:
> https://github.com/kata-containers/runtime/blob/master/pkg/katautils/create.go
> 
> - When using initrd instead, it is supposed that the init == agent.
> However for the dracut generated initrd, the kata-containers.target is
> added to basic.target.wants.
> See here:
> https://github.com/kata-containers/osbuilder/blob/master/rootfs-builder/rootfs.sh#L396
> 

Great, thanks for the pointers

Thanks,
Cole