Open Stack

Hi kata folks. I want to give an update on the state of kata in Fedora,
discuss some of the remaining pieces, and ask some questions

There's already some pieces of kata in Fedora: kata-ksm-throttler,
kata-proxy, kata-shim, kata-osbuilder. They were added around June 2018
by Lokesh Mandvekar. But they are quite out of date at this point, and
obviously not that useful because kata-runtime never made it into the
distro. The kata-osbuilder package does generate a Fedora initrd and
image using the distro specific scripts (not dracut), but I suspect they
never worked with the Fedora kernel at least.

I spent the past two days getting the osbuilder dracut support to work
with the Fedora kernel. I now have the latest OBS kata packages working
on Fedora 30 with a stock Fedora kernel, dracut built initrd, and stock
Fedora qemu 4.1 build (from the virt-preview repo). Here's the changes I
added, see the attached files

* configuration.qemu.diff: Changes I made to
/etc/kata-containers/configuration.toml compared to
/usr/share/defaults/kata-containers/configuration.toml. Besides changing
the qemu, kernel, and initrd path as mentioned above, I turned on q35
and vsock too; pc and virtio-serial were working as well, but this is
likely the config we will target for Fedora packaging

* agent-0001-*.patch: Make kata-agent.service run after
systemd-udev-settle.service. At one point in my experimenting I needed
this: kata would communicate with the agent, but networking in the guest
didn't seem to be set up yet, and configuring eth0 was failing. However
if I revert this now it doesn't seem to cause any failures. This may be
due to /etc/modules-load.d/kata-modules.conf loading the necessary bits
early enough, but that may still be race-y. I figured I'd mention it for
posterity.

* osbuilder-0001-*: This is a hack. If you use
AGENT_SOURCE_BIN=SOMEDIR/kata-agent, rootfs.sh will now use SOMEDIR as
the source for installing kata-agent.service and kata-containers.target.
We will want something like this in Fedora where we will likely have a
standalone kata-agent package shipping a prebuilt kata-agent. We
shouldn't have to depend on git and go compilation to build the
kata-agent at image generation time. Suggestions welcome on how to
implement this in an upstream suitable way

* osbuilder-0002-*: This fixes the initrd /sbin/init symlinking for me.
The current code leaves /init, /sbin/init, and /usr/lib/systemd/systemd
all as symlinks pointing to each other. Maybe this works on another
distro but it didn't work on Fedora. Or maybe this is just a bug

* osbuilder-0003-*: This extends the dracut config with the list of
kernel driver module names we need in the initrd. Not all virtio devices
are listed here because in Fedora some of them are already compiled in
(virtio_pci, virtio_rng at least). The list is: 9p 9pnet_virtio
vmw_vsock_virtio_transport virtio_console virtio_blk virtio_scsi
virtio_net. I presume this will stay downstream only

After those bits are applied, I build osbuilder with:

# cat b.sh
IMAGEOUT="$(PWD)/kata-containers-initrd.img"
AGENTSRC="/root/go/src/github.com/kata-containers/agent"
KVERSION="5.0.9-301.fc30.x86_64"
export AGENT_SOURCE_BIN=$AGENTSRC/kata-agent
rm -rf dracut_overlay $IMAGEOUT
make BUILD_METHOD=dracut DRACUT_KVERSION=$KVERSION initrd


Understandably that KVERSION needs to match a kernel installed on the
host, and it needs to match the kernel referenced in configuration.toml.
IMAGEOUT is the location of the generated initrd, which is also
referenced in /etc/kata-containers/configuration.toml. Then I can test with:

# podman run -it --runtime=/usr/bin/kata-runtime alpine sh

(with selinux disabled on the host though, kata seems to require it for
podman usage last I checked).


So with that working, the major pieces that need to be finished so that
kata can at least be tested with native Fedora packages are:

* Update the kata-osbuilder package to use the above info
* Update all the existing kata-* packages to latest versions
* Package kata-agent and get it accepted into Fedora
* Finish the kata-runtime package submission and get it accepted into Fedora

After that, the important pieces IMO are:

* Get non-x86 support working in Fedora

* Find a way to package some kind of docker config snippet so 'docker
--runtime=kata-runtime' works out of the box. On Fedora the simplest way
I found was to add '--add-runtime kata-runtime=path/to/kata-runtime' in
/etc/sysconfig/docker OPTIONS=, but that's not really a programmatic
operation

* Find a way to get podman to not require selinux=disabled which is a
really big hammer. Hopefully it's not hard to make it not error if
selinux=permissive which can be changed at runtime.

* Get a minimal qemu build into fedora. Christophe and I have some work
on this already: https://github.com/c3d/fedora-qemu-mini
https://github.com/crobinso/fedora-qemu-mini


Now some misc questions that I have after poking around at this:

* Is initrd preferred vs image boot? I didn't really try to get image
boot working. Should I?

* AGENT_INIT=no is the default, meaning /sbin/init == systemd, not
/sbin/init == kata-agent. Is that generally the preferred config? When I
was flailing around trying to get anything to work I tried
AGENT_INIT=yes but didn't get anywhere

* At first I spent a lot of time trying to get the old style rootfs
scripts to work with the Fedora kernel. But I was hitting some confusing
situations. The generated initrd fs would have kata agent and the
services in them, but nothing was adding it to basic.target.wants;
current code in rootfs.sh this only happens if using the dracut method.
So understandably kernel+initrd wouldn't work in that case, because
nothing was starting the kata-agent... except when I combined that
initrd with the kernel from the kata-linux-container package, the kata
container would start up fine! Which really perplexes me... does that
kernel have some particular knowledge about starting kata-agent or one
of the associated services? Or does it trigger some different systemd
behavior somehow?


Thanks,
Cole
-------------- next part --------------
A non-text attachment was scrubbed...
Name: osbuilder-0003-dracut-Add-Fedora-virtio-kernel-modules-to-the-initr.patch
Type: text/x-patch
Size: 1213 bytes
Desc: not available
URL: <http://lists.katacontainers.io/pipermail/kata-dev/attachments/20190907/f45027d2/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: osbuilder-0002-rootfs-Fix-systemd-sbin-init-symlinking.patch
Type: text/x-patch
Size: 1273 bytes
Desc: not available
URL: <http://lists.katacontainers.io/pipermail/kata-dev/attachments/20190907/f45027d2/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: osbuilder-0001-rootfs-allow-using-systemd-units-from-AGENT_SOURCE_B.patch
Type: text/x-patch
Size: 1156 bytes
Desc: not available
URL: <http://lists.katacontainers.io/pipermail/kata-dev/attachments/20190907/f45027d2/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configuration.toml.diff
Type: text/x-patch
Size: 1207 bytes
Desc: not available
URL: <http://lists.katacontainers.io/pipermail/kata-dev/attachments/20190907/f45027d2/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: agent-0001-service-Run-kata-agent-After-systemd-udev-settle.ser.patch
Type: text/x-patch
Size: 936 bytes
Desc: not available
URL: <http://lists.katacontainers.io/pipermail/kata-dev/attachments/20190907/f45027d2/attachment-0004.bin>