Dracut initramfs for distro packaging
Hi, Paolo Bonzini, Xu Wang, and I discussed using dracut (https://dracut.wiki.kernel.org/) to build a distro-friendly initramfs at the OpenStack Denver PTG. I'm sharing a proof-of-concept for Fedora. Other distros can probably follow a similar approach. This process creates an initramfs from the host system using dracut. It's fast and produces a relatively small initramfs. On my Fedora 30 system the file size is 19 MB including kata-agent and it could probably be optimized further. osbuilder is replaced with dracut, a tool that distro kernel packagers are familiar with. I hope that this approach will make it easy and acceptable to package Kata Containers. The scripts below build a kata-containers-initrd.img with systemd and the kernel modules from the host system. The generic distro kernel is used in the sandbox VM, something that distro kernel packagers will be happy about since it keeps their test matrix unchanged. The proof-of-concept code is available here: https://github.com/stefanha/agent/tree/fedora-dracut To build the Kata sandbox VM initramfs: # make # mkdir initramfs-overlay # make DESTDIR=initramfs-overlay install # make initrd Edit /etc/kata-containers/configuration.toml so that hypervisor.qemu.kernel points to the host's vmlinuz and initrd points to the generated kata-containers-initrd.img file. hypervisor.qemu.kernel_params must contain systemd.unit=kata-containers.target so that kata-agent is launched upon sandbox VM boot. Also set hypervisor.qemu.use_vsock to true. Stefan
Hi Stefan, Thanks to bring this to the ML. This is an interesting approach I have not play with Dracut a lot in the past but it definitely is a good thing look at. When we started osbuilder the idea was to show adopters that Kata was able to boot with any OS as base (as long follow some requirements) and probably they adapt as needed or create their own tooling get Kata working for them. After all this time I see that it is more value to have just one tool that anyone can use and is flexible for developers and linux distributions adoption. So I think would need more feedback from all the community and also from linux distributions in general so we can have a solution that works better for all. Here some question related with dracut usage: 1. Is it possible to build a disk image? I think the answer is not. This is a feature that is still difficult to be compatible with packaging but helps in memory usage if used with DAX. So probably a nice feature to keep 2. Is it possible to have a custom rootfs with packages I want? Today we require system + agent + iptables + chrony. But also we support not having system but the agent as init. Also, users may want to add additional binaries in the future (to be part of the official image or just for hacking). On Tue, May 07, 2019 at 11:41:12AM -0400, Stefan Hajnoczi wrote:
Hi, Paolo Bonzini, Xu Wang, and I discussed using dracut (https://dracut.wiki.kernel.org/) to build a distro-friendly initramfs at the OpenStack Denver PTG.
I'm sharing a proof-of-concept for Fedora. Other distros can probably follow a similar approach.
This process creates an initramfs from the host system using dracut. It's fast and produces a relatively small initramfs. On my Fedora 30 system the file size is 19 MB including kata-agent and it could probably be optimized further.
osbuilder is replaced with dracut, a tool that distro kernel packagers are familiar with. I hope that this approach will make it easy and acceptable to package Kata Containers.
This is the major pro I can find of using this. The adoption will be easier for linux distributions, we had a workaround Fedora to build the image at post install but still got errors as for fedora the way to go is to use the host kernel. For SUSE Marco used a custom feature of OBS that is called kiwi. For Ubuntu we dont have any effort to get packages but we have snaps that have a bit more of freedom that traditional packaging tools. So I would like to get more feedback from differed linux distributions.
The scripts below build a kata-containers-initrd.img with systemd and the kernel modules from the host system. The generic distro kernel is used in the sandbox VM, something that distro kernel packagers will be happy about since it keeps their test matrix unchanged.
The proof-of-concept code is available here:
https://github.com/stefanha/agent/tree/fedora-dracut
To build the Kata sandbox VM initramfs:
# make # mkdir initramfs-overlay # make DESTDIR=initramfs-overlay install # make initrd
Edit /etc/kata-containers/configuration.toml so that hypervisor.qemu.kernel points to the host's vmlinuz and initrd points to the generated kata-containers-initrd.img file. hypervisor.qemu.kernel_params must contain systemd.unit=kata-containers.target so that kata-agent is launched upon sandbox VM boot. Also set hypervisor.qemu.use_vsock to true.
Stefan
On Tue, May 07, 2019 at 03:38:48PM -0500, Jose Carlos Venegas Munoz wrote:
Thanks to bring this to the ML. This is an interesting approach I have not play with Dracut a lot in the past but it definitely is a good thing look at.
When we started osbuilder the idea was to show adopters that Kata was able to boot with any OS as base (as long follow some requirements) and probably they adapt as needed or create their own tooling get Kata working for them. After all this time I see that it is more value to have just one tool that anyone can use and is flexible for developers and linux distributions adoption.
So I think would need more feedback from all the community and also from linux distributions in general so we can have a solution that works better for all.
This dracut approach is only aimed at distro package maintainers wishing to ship official Kata Containers packages. Users building Kata from source or customizing should continue to use osbuilder. dracut does not cover all cases that osbuilder covers. It's also not used by all Linux distros, so they will have to use their equivalent tool. However, I think dracut or equivalent distro tools are the right fit for official Kata Containers packages in Fedora, Debian, etc because: 1. The same tool is already used to build the host initramfs and is familiar to package maintainers. 2. It produces a smaller initramfs than osbuilder. 3. It is fast because files are copied from the host instead of installed using a package manager. 4. It eliminates packaging policy headaches because all files come from the host and therefore don't raise questions like "can we redistribute a ClearLinux rootfs and where do we mirror/publish the sources?". Due to dracut's speed it would be appropriate to rebuild kata-containers-initrd.img every time the host kernel or kata-agent package is updated. Previously this problem was discussed because building a rootfs with osbuilder is slow and shouldn't hold up software updates, but I think it becomes a non-issue with dracut.
Here some question related with dracut usage:
1. Is it possible to build a disk image? I think the answer is not. This is a feature that is still difficult to be compatible with packaging but helps in memory usage if used with DAX. So probably a nice feature to keep
AFAIK dracut does not build disk images. I've been booting from virtio-fs (with DAX) recently and am now wondering about skipping the cpio + compression step in dracut. That could achieve the same thing although virtiofsd has more memory overhead than a disk image NVDIMM.
2. Is it possible to have a custom rootfs with packages I want? Today we require system + agent + iptables + chrony. But also we support not having system but the agent as init. Also, users may want to add additional binaries in the future (to be part of the official image or just for hacking).
Additional files and binaries can be added from the host system. dracut does not use a package manager to install them, instead it copies them and library dependencies. In the example I posted there is an initramfs-overlay/ directory. Files dropped in there will be built into the initramfs. Stefan
To summarize: Use dracut to improve linux distribution adoption. - This can work for Feodra/RHEL - Allow to use the host kernel (required by fedora/Red Hat) - Waiting if some other distro provide feedback - If we implement it should we remove fedora support from osbuilder? as this was done to allow Fedora/Red Hat/CentOS adoption but are going with the dracut approach Keep using osbuilder for the rest of use cases dracut not cover. Spacially for DAX images And not systemd based images (sorry I did not get if possible to not ship systemd). Based in this summary, this lgtm. On Thu, May 09, 2019 at 05:23:58PM +0100, Stefan Hajnoczi wrote:
On Tue, May 07, 2019 at 03:38:48PM -0500, Jose Carlos Venegas Munoz wrote:
Thanks to bring this to the ML. This is an interesting approach I have not play with Dracut a lot in the past but it definitely is a good thing look at.
When we started osbuilder the idea was to show adopters that Kata was able to boot with any OS as base (as long follow some requirements) and probably they adapt as needed or create their own tooling get Kata working for them. After all this time I see that it is more value to have just one tool that anyone can use and is flexible for developers and linux distributions adoption.
So I think would need more feedback from all the community and also from linux distributions in general so we can have a solution that works better for all.
This dracut approach is only aimed at distro package maintainers wishing to ship official Kata Containers packages. Users building Kata from source or customizing should continue to use osbuilder.
dracut does not cover all cases that osbuilder covers. It's also not used by all Linux distros, so they will have to use their equivalent tool.
However, I think dracut or equivalent distro tools are the right fit for official Kata Containers packages in Fedora, Debian, etc because:
1. The same tool is already used to build the host initramfs and is familiar to package maintainers. 2. It produces a smaller initramfs than osbuilder. 3. It is fast because files are copied from the host instead of installed using a package manager. 4. It eliminates packaging policy headaches because all files come from the host and therefore don't raise questions like "can we redistribute a ClearLinux rootfs and where do we mirror/publish the sources?".
Due to dracut's speed it would be appropriate to rebuild kata-containers-initrd.img every time the host kernel or kata-agent package is updated. Previously this problem was discussed because building a rootfs with osbuilder is slow and shouldn't hold up software updates, but I think it becomes a non-issue with dracut.
It make sense your points, and if this works for your cases (Fedora/Red Hat) I am happy to go in this path. Some things to consider are : Where this could leave? - agent code -osbuilder code The best of this proposed case could be something that works well for your policy. Also from Fedora/Redhat perspective, do you think that keep Fedora in osbuilder supported rootfs has some value or we should remove it and go with this path. As starting point we could have a job that uses this proposal using Fedora. As extra note for the rest of the community: If someone is part of canonical, debian, or other distro let us know if this could simplify your work.
Here some question related with dracut usage:
1. Is it possible to build a disk image? I think the answer is not. This is a feature that is still difficult to be compatible with packaging but helps in memory usage if used with DAX. So probably a nice feature to keep
AFAIK dracut does not build disk images.
I've been booting from virtio-fs (with DAX) recently and am now wondering about skipping the cpio + compression step in dracut. That could achieve the same thing although virtiofsd has more memory overhead than a disk image NVDIMM.
2. Is it possible to have a custom rootfs with packages I want? Today we require system + agent + iptables + chrony. But also we support not having system but the agent as init. Also, users may want to add additional binaries in the future (to be part of the official image or just for hacking).
Additional files and binaries can be added from the host system. dracut does not use a package manager to install them, instead it copies them and library dependencies.
Nice I was open an issue to implement something similar for osbuilder https://github.com/kata-containers/osbuilder/issues/259 but if this work out of the box lets use it.
In the example I posted there is an initramfs-overlay/ directory. Files dropped in there will be built into the initramfs.
Stefan
On Thu, May 09, 2019 at 03:16:26PM -0500, Jose Carlos Venegas Munoz wrote:
To summarize:
Sounds good.
Use dracut to improve linux distribution adoption. - This can work for Feodra/RHEL - Allow to use the host kernel (required by fedora/Red Hat) - Waiting if some other distro provide feedback - If we implement it should we remove fedora support from osbuilder? as this was done to allow Fedora/Red Hat/CentOS adoption but are going with the dracut approach
The osbuilder Fedora support doesn't work with distro kernels. It also doesn't use the latest Fedora release (30). It's probably not used much and removing it would be fine.
Keep using osbuilder for the rest of use cases dracut not cover. Spacially for DAX images And not systemd based images (sorry I did not get if possible to not ship systemd).
I haven't tried. In theory systemd is just a dracut module that can be disabled. Maybe it can be replaced with a kata-agent module that acts a /sbin/init instead. This would result in a smaller initramfs. Stefan
On 5/7/19 10:38 PM, Jose Carlos Venegas Munoz wrote:
So I think would need more feedback from all the community and also from linux distributions in general so we can have a solution that works better for all.
Hi Jose Carlos, Stefan, I am providing a follow-up, since I also faced packaging problems trying to provide Kata image and kernel packages in openSUSE: - w.r.t. kernel: providing a "third party" kernel package along with the official distro kernels is something not usually possible for various reasons, such as the need to maintain a strict control over how the kernels are built/patched/maintained. - w.r.t. image: lack of standardized/simple ways of generating a image and to provide it inside a package. I've evaluated the use of dracut generated initramfs starting from Stefan PoC, and in the few tests I did it seems a viable solution, keeping in mind that at the moment the main use case would be to provide something that is packaging friendly. With respect to the distro kernels compatibility, the goal would be to reuse a vanilla or a generic kvm kernel that is already provided by the distribution, and have dracut load in the initramfs the kernel modules required by Kata.
Here some question related with dracut usage:
1. Is it possible to build a disk image?
You can build a disk image with DAX using as base the rootfs included in the initramfs.
2. Is it possible to have a custom rootfs with packages I want? Today we require system + agent + iptables + chrony. But also we support not having system but the agent as init. Also, users may want to add additional binaries in the future (to be part of the official image or just for hacking).
Support for agent + iptables + chrony is something that could be provided with a custom dracut modules, or using the dracut overlay (as Stefan PoC). I have spent some time polishing the tests I did, and I've submitted a "RFC" PR to osbuilder to add support for dracut-based initrd / images: https://github.com/kata-containers/osbuilder/pull/312 Hopefully this is a good starting point. Cheers Marco
On Wed, Jun 12, 2019 at 07:06:16PM +0200, Marco Vedovati wrote:
On 5/7/19 10:38 PM, Jose Carlos Venegas Munoz wrote:
So I think would need more feedback from all the community and also from linux distributions in general so we can have a solution that works better for all.
Hi Jose Carlos, Stefan,
I am providing a follow-up, since I also faced packaging problems trying to provide Kata image and kernel packages in openSUSE:
- w.r.t. kernel: providing a "third party" kernel package along with the official distro kernels is something not usually possible for various reasons, such as the need to maintain a strict control over how the kernels are built/patched/maintained.
- w.r.t. image: lack of standardized/simple ways of generating a image and to provide it inside a package.
I've evaluated the use of dracut generated initramfs starting from Stefan PoC, and in the few tests I did it seems a viable solution, keeping in mind that at the moment the main use case would be to provide something that is packaging friendly.
With respect to the distro kernels compatibility, the goal would be to reuse a vanilla or a generic kvm kernel that is already provided by the distribution, and have dracut load in the initramfs the kernel modules required by Kata.
Yes, couldn't agree more.
Here some question related with dracut usage:
1. Is it possible to build a disk image?
You can build a disk image with DAX using as base the rootfs included in the initramfs.
2. Is it possible to have a custom rootfs with packages I want? Today we require system + agent + iptables + chrony. But also we support not having system but the agent as init. Also, users may want to add additional binaries in the future (to be part of the official image or just for hacking).
Support for agent + iptables + chrony is something that could be provided with a custom dracut modules, or using the dracut overlay (as Stefan PoC).
I have spent some time polishing the tests I did, and I've submitted a "RFC" PR to osbuilder to add support for dracut-based initrd / images:
https://github.com/kata-containers/osbuilder/pull/312
Hopefully this is a good starting point.
Cool, thanks for doing this! Adding dracut support to osbuilder is nice because it may allow distro packagers to share code instead of everyone inventing their own dracut scripts. I'm not working on this as a Fedora packager but I hope Fedora folks will try this approach and contribute. Stefan
participants (3)
-
Jose Carlos Venegas Munoz
-
Marco Vedovati
-
Stefan Hajnoczi