Open Stack

Hi Cole,
see inline for more clarifications.

Thanks
Marco

(PS: I'm using another address, @suse.com doesn't deliver to
kata-dev anymore)


On Tuesday, September 10, 2019 1:45 AM, Cole Robinson wrote:
> Requiring content from a non-distro network source during a package
> build process is definitely a distro no no, at least for Fedora.
> Generally builds aren't expected to require network access for one thing
> 
> Right now in the existing fedora kata-osbuilder package, the initrd and
> image are actually built at package install time, so that we can use the
> host installed kernel and modules, and not have to ship a separate
> kernel (which is also against Fedora policies). If we pull kata-agent
> from git at package install time, this means the package has a runtime
> dep on git and golang, which is not very friendly.
> 
> So while indeed kata-agent doesn't really have any usage on the host,
> the most distro friendly way I can think to handle this is to make
> kata-agent its own package, install the binary and systemd files in
> /usr/libexec or somewhere else out of the way, and have the initrd build
> process access those on the host FS.


This is what I am doing for openSUSE (package link in my first email):

- The image package sources include tarballs of both the agent and of
the osbuilder
source code.

- The spec file contains a `BuildRequires: go` line.

- The %build step runs the `make initrd` build target, to build the
agent and
to generate an initrd using dracut.

- The %install steps copies the initrd inside the final rpm.

- There's not need to run any complex %post step on the host.


> For Fedora it seems like /init doesn't matter for the boot process at
> all and nothing seems to set it by default. Are those strictly required
> for SUSE? There's probably some set of commands that will work for both

When using the dracut build method, these sets of links are created here:

https://github.com/kata-containers/osbuilder/blob/master/rootfs-builder/rootfs.sh#L387

If I recall correctly, I just tried to replicate the set of links
created using the non-dracut build method.

Feel free to open a PR to improve this and add support for Fedora.
BTW, a.t.m. dracut is unit tested with a openSUSE container only, adding
Fedora and other would be nice.


>>> * osbuilder-0003-*: This extends the dracut config with the list of
>>> kernel driver module names we need in the initrd. Not all virtio devices
>>> are listed here because in Fedora some of them are already compiled in
>>> (virtio_pci, virtio_rng at least). The list is: 9p 9pnet_virtio
>>> vmw_vsock_virtio_transport virtio_console virtio_blk virtio_scsi
>>> virtio_net. I presume this will stay downstream only
>>
>> Nice, I think it makes sense to have a predefined list of virtualization
>> specific kernel modules to add. You can create a PR for that.
>> I am wondering however what could happen if some of these modules are
>> already built-in in the kernel file.
>>

Cool!


> 
>>From dracut perspective it doesn't seem to have any direct impact. I
> just tested adding virtio_pci which is built in on Fedora, no
> complaints. Adding an invalid module name though will make
> systemd-module-load service report failure, but that's just noise
> 
>>> After those bits are applied, I build osbuilder with:
>>>
>>> # cat b.sh
>>> IMAGEOUT="$(PWD)/kata-containers-initrd.img"
>>> AGENTSRC="/root/go/src/github.com/kata-containers/agent"
>>> KVERSION="5.0.9-301.fc30.x86_64"
>>> export AGENT_SOURCE_BIN=$AGENTSRC/kata-agent
>>> rm -rf dracut_overlay $IMAGEOUT
>>> make BUILD_METHOD=dracut DRACUT_KVERSION=$KVERSION initrd
>>>
>>>
>>> Understandably that KVERSION needs to match a kernel installed on the
>>> host, and it needs to match the kernel referenced in configuration.toml.
>>> IMAGEOUT is the location of the generated initrd, which is also
>>> referenced in /etc/kata-containers/configuration.toml. Then I can
> test with:
>>>
>>> # podman run -it --runtime=/usr/bin/kata-runtime alpine sh
>>>
>>> (with selinux disabled on the host though, kata seems to require it for
>>> podman usage last I checked).
>>>
>>>
>>> So with that working, the major pieces that need to be finished so that
>>> kata can at least be tested with native Fedora packages are:
>>>
>>> * Update the kata-osbuilder package to use the above info
>>> * Update all the existing kata-* packages to latest versions
>>> * Package kata-agent and get it accepted into Fedora
>>> * Finish the kata-runtime package submission and get it accepted into
> Fedora
>>>
>>> After that, the important pieces IMO are:
>>>
>>> * Get non-x86 support working in Fedora
>>>
>>> * Find a way to package some kind of docker config snippet so 'docker
>>> --runtime=kata-runtime' works out of the box. On Fedora the simplest way
>>> I found was to add '--add-runtime kata-runtime=path/to/kata-runtime' in
>>> /etc/sysconfig/docker OPTIONS=, but that's not really a programmatic
>>> operation
>>>
>>> * Find a way to get podman to not require selinux=disabled which is a
>>> really big hammer. Hopefully it's not hard to make it not error if
>>> selinux=permissive which can be changed at runtime.
>>>
>>> * Get a minimal qemu build into fedora. Christophe and I have some work
>>> on this already: https://github.com/c3d/fedora-qemu-mini
>>> https://github.com/crobinso/fedora-qemu-mini
>>>
>>>
>>> Now some misc questions that I have after poking around at this:
>>>
>>> * Is initrd preferred vs image boot? I didn't really try to get image
>>> boot working. Should I?
>>
>> Nope, it's not preferred. But it is packaging friendly.
>>
> 
> Yeah thinking about this more it's definitely not memory friendly to use
> an initrd. I tried the image building and got the fs generating, but the
> Fedora kernel needs some more drivers built in to make it work without
> an initrd. I just filed a Fedora kernel bug:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1750581
> 
>>> * AGENT_INIT=no is the default, meaning /sbin/init == systemd, not
>>> /sbin/init == kata-agent. Is that generally the preferred config? When I
>>> was flailing around trying to get anything to work I tried
>>> AGENT_INIT=yes but didn't get anywhere
>>
>> Agent init may give you a smaller image size and a faster boot time.
>> However, you will not be able to load initrd kernel modules when using
>> dracut-initrds, so you'll need all the features built into the kernel.
>>
> 
> Makes sense
> 
>>> * At first I spent a lot of time trying to get the old style rootfs
>>> scripts to work with the Fedora kernel. But I was hitting some confusing
>>> situations. The generated initrd fs would have kata agent and the
>>> services in them, but nothing was adding it to basic.target.wants;
>>> current code in rootfs.sh this only happens if using the dracut method.
>>> So understandably kernel+initrd wouldn't work in that case, because
>>> nothing was starting the kata-agent... except when I combined that
>>> initrd with the kernel from the kata-linux-container package, the kata
>>> container would start up fine! Which really perplexes me... does that
>>> kernel have some particular knowledge about starting kata-agent or one
>>> of the associated services? Or does it trigger some different systemd
>>> behavior somehow?
>>
>>
>> - When the sandbox is started using a regular image, the kernel cmdline
>> includes "systemd.unit=kata-containers.target"
>> See here:
>>
> https://github.com/kata-containers/runtime/blob/master/pkg/katautils/create.go
>>
>> - When using initrd instead, it is supposed that the init == agent.
>> However for the dracut generated initrd, the kata-containers.target is
>> added to basic.target.wants.
>> See here:
>>
> https://github.com/kata-containers/osbuilder/blob/master/rootfs-builder/rootfs.sh#L396
>>
> 
> Great, thanks for the pointers
> 
> Thanks,
> Cole
>