Hi All, As you probably know, it is possible to create a Kata Container using either a rootfs image ("image=") or an initrd image ("initrd=") [1][2]. The former uses systemd to launch the agent whilst the latter uses the agent as the init daemon. Deciding which option is best is up to the user as they both have pros and cons. We are aware of a few changes recently that are making these two options diverge slightly in terms of features (however we do have plans to "pull that back" and minimise the differences). There is also ongoing debate about the use of systemd for the rootfs images. As these topics are related, we've created a wiki page to try to summarise the above including advantages and disadvantages of each approach (image compared with initrd, agent as init and systemd as init): https://github.com/kata-containers/documentation/wiki/Kata-images Please contribute to this page to allow us all to get the best possible picture of use-cases, features, limitations and benefits of each option. Cheers, James [1] - https://github.com/kata-containers/documentation/blob/master/architecture.md... [2] - https://github.com/kata-containers/osbuilder <https://github.com/kata-containers/osbuilder#usage> --- https://katacontainers.io/ | https://github.com/kata-containers <https://github.com/clearcontainers> Open Source Technology Center Intel Corporation (UK) Ltd. - Co. Reg. #1134945 - Pipers Way, Swindon SN3 1RJ.
On Fri, Mar 22, 2019 at 09:30:18AM +0000, Hunt, James O wrote:
As these topics are related, we've created a wiki page to try to summarise the above including advantages and disadvantages of each approach (image compared with initrd, agent as init and systemd as init):
https://github.com/kata-containers/documentation/wiki/Kata-images
Please contribute to this page to allow us all to get the best possible picture of use-cases, features, limitations and benefits of each option.
A missing use case is packaging Kata Containers for Linux distributions. Kata isn't available in the Debian or Fedora package collections yet. The image is one of the tricky issues that packagers need to solve. I think that the current solutions are developer-friendly but not packaging-friendly. As Kata matures, the packaging use case will become more important so that end-users can "apt install kata-containers" or "dnf install kata-containers". Although some users will continue to install the latest and greatest directly from kata-containers.io, having packages supported by distributions ensures that they integrate into the rest of the system (e.g. Docker, CRI-O) easily and with fewer steps than assembling a stack from multiple third-parties. Each distro has kernel update policies, licensing considerations, etc that affect how Kata Containers can be packaged in a supportable way. Taking at Debian and Fedora requirements on board when comparing image options would help. Discussing this use case and optimizing for it would be important if there is general agreement that getting Kata Containers into Linux distributions is a priority. Stefan
That is a good point. Working with current kata packages (in OBS) and with Lokesh from Fedora this not a very distro friendly process. In my mind remove systemd could simplify the rootfs creation, ideally features needed by the agent would be implemented there ( I know sometimes we just prefer to use an existing binary for that). I see the following requirements for rootfs creation point + packaging limitations. - rootfs creation should work without network access - Not need to rely on a package manager (like populate a directory with dnf) To handle this we probably have two options: - Create roots with a tool like Dracut ( If somebody has some experience with it feel free to open an issue to talk about it) - Remove systemd depencies, just have as rootfs dependency: agent, ipables <name other random binaries we need in the future). - The binaries need to be static as requirement: with this rootfs creation is just copy binaries to a directory. - If binaries are not static we can parse ldd output and copy the right libs to the rootfs directory. - See https://github.com/kata-containers/osbuilder/issues/259 for more information about this. - Carlos -----Original Message----- From: Stefan Hajnoczi [mailto:stefanha@redhat.com] Sent: Monday, March 25, 2019 10:28 AM To: Hunt, James O <james.o.hunt@intel.com> Cc: kata-dev <kata-dev@lists.katacontainers.io> Subject: Re: [kata-dev] kata image types compared On Fri, Mar 22, 2019 at 09:30:18AM +0000, Hunt, James O wrote:
As these topics are related, we've created a wiki page to try to summarise the above including advantages and disadvantages of each approach (image compared with initrd, agent as init and systemd as init):
https://github.com/kata-containers/documentation/wiki/Kata-images
Please contribute to this page to allow us all to get the best possible picture of use-cases, features, limitations and benefits of each option.
A missing use case is packaging Kata Containers for Linux distributions. Kata isn't available in the Debian or Fedora package collections yet. The image is one of the tricky issues that packagers need to solve. I think that the current solutions are developer-friendly but not packaging-friendly. As Kata matures, the packaging use case will become more important so that end-users can "apt install kata-containers" or "dnf install kata-containers". Although some users will continue to install the latest and greatest directly from kata-containers.io, having packages supported by distributions ensures that they integrate into the rest of the system (e.g. Docker, CRI-O) easily and with fewer steps than assembling a stack from multiple third-parties. Each distro has kernel update policies, licensing considerations, etc that affect how Kata Containers can be packaged in a supportable way. Taking at Debian and Fedora requirements on board when comparing image options would help. Discussing this use case and optimizing for it would be important if there is general agreement that getting Kata Containers into Linux distributions is a priority. Stefan
On Mon, Mar 25, 2019 at 04:42:17PM +0000, Venegas Munoz, Jose Carlos wrote:
That is a good point.
Working with current kata packages (in OBS) and with Lokesh from Fedora this not a very distro friendly process.
In my mind remove systemd could simplify the rootfs creation, ideally features needed by the agent would be implemented there ( I know sometimes we just prefer to use an existing binary for that).
I see the following requirements for rootfs creation point + packaging limitations.
- rootfs creation should work without network access - Not need to rely on a package manager (like populate a directory with dnf)
To handle this we probably have two options:
- Create roots with a tool like Dracut ( If somebody has some experience with it feel free to open an issue to talk about it) - Remove systemd depencies, just have as rootfs dependency: agent, ipables <name other random binaries we need in the future). - The binaries need to be static as requirement: with this rootfs creation is just copy binaries to a directory. - If binaries are not static we can parse ldd output and copy the right libs to the rootfs directory. - See https://github.com/kata-containers/osbuilder/issues/259 for more information about this.
I have CCed Lokesh for Fedora-specific requirements for Kata images: https://github.com/kata-containers/documentation/wiki/Kata-images Lokesh: Out of the various options for building Kata images (initramfs vs disk image, systemd vs agent), which ones are friendliest for Fedora packaging? Is anyone in touch with Debian packagers who could provide input? Stefan
Hi Stefan, Le mar. 26 mars 2019 à 08:02, Stefan Hajnoczi <stefanha@redhat.com> a écrit :
That is a good point.
Working with current kata packages (in OBS) and with Lokesh from Fedora
On Mon, Mar 25, 2019 at 04:42:17PM +0000, Venegas Munoz, Jose Carlos wrote: this not a very distro friendly process.
In my mind remove systemd could simplify the rootfs creation, ideally
features needed by the agent would be implemented there ( I know sometimes we just prefer to use an existing binary for that).
I see the following requirements for rootfs creation point +
packaging limitations.
- rootfs creation should work without network access - Not need to rely on a package manager (like populate a directory with
dnf)
To handle this we probably have two options:
- Create roots with a tool like Dracut ( If somebody has some experience
with it feel free to open an issue to talk about it)
- Remove systemd depencies, just have as rootfs dependency: agent, ipables <name other random binaries we need in the future). - The binaries need to be static as requirement: with this rootfs creation is just copy binaries to a directory. - If binaries are not static we can parse ldd output and copy the right libs to the rootfs directory. - See https://github.com/kata-containers/osbuilder/issues/259 for more information about this.
I have CCed Lokesh for Fedora-specific requirements for Kata images: https://github.com/kata-containers/documentation/wiki/Kata-images
This page was originally intended simply to summarise the differences between image options. However, I'm really glad you've brought up the important and ongoing issue of distro packaging. I've tried to summarise that in [1], but maybe that too needs to be surfaced by putting it into a wiki page of its own.
Lokesh: Out of the various options for building Kata images (initramfs vs disk image, systemd vs agent), which ones are friendliest for Fedora packaging?
There may be an alternative solution to the ongoing issue of packaging binaries images... - Kata needs a binary rootfs image and a binary kernel image to boot the hypervisor. Once booted, the user's chosen (binary) disk image will be made available for the workload (you might be able to see where this is going... :) - Our OBS packaging currently packages images and kernels as a convenience to users. - Container managers like Docker also require binary images. But those images are not packaged - they are downloaded on demand ("docker pull"). - This leads to a potential solution to the packaging blocker: Kata could be packaged without any binary assets. - If the binaries were not packaged, an additional step would be required before a Kata container could be created, something like [2]: $ sudo kata-runtime pull [all-assets|image|initrd|kernel] There are complications with this concept though, including: - Where would we host such assets? - How would the kernel be handled? - How would the system determine which type of image to download? - How would updates to these assets be handled? We need to avoid / minimise the "stale container" scenario [3].
Is anyone in touch with Debian packagers who could provide input?
As shown on [1], we're still looking for Debian volunteers, so if anyone is keen...? :)
Stefan
Cheers, James [1] - https://github.com/kata-containers/packaging/issues/6#issuecomment-453940053 [2] - If that wasn't done, the runtime would select suitable defaults when "kata-runtime create" was invoked. That would of course delay the creation of the initial Kata container but would be similar to how, say, Docker works if you "docker run" without a previous "docker pull". [3] - https://github.com/clearcontainers/runtime/wiki/Stale-containers --- https://katacontainers.io/ | https://github.com/kata-containers <https://github.com/clearcontainers> Open Source Technology Center Intel Corporation (UK) Ltd. - Co. Reg. #1134945 - Pipers Way, Swindon SN3 1RJ.
On Tue, Mar 26, 2019 at 09:21:06AM +0000, Hunt, James O wrote:
Le mar. 26 mars 2019 à 08:02, Stefan Hajnoczi <stefanha@redhat.com> a écrit :
On Mon, Mar 25, 2019 at 04:42:17PM +0000, Venegas Munoz, Jose Carlos wrote: Lokesh: Out of the various options for building Kata images (initramfs vs disk image, systemd vs agent), which ones are friendliest for Fedora packaging?
There may be an alternative solution to the ongoing issue of packaging binaries images...
- Kata needs a binary rootfs image and a binary kernel image to boot the hypervisor. Once booted, the user's chosen (binary) disk image will be made available for the workload (you might be able to see where this is going... :) - Our OBS packaging currently packages images and kernels as a convenience to users. - Container managers like Docker also require binary images. But those images are not packaged - they are downloaded on demand ("docker pull"). - This leads to a potential solution to the packaging blocker: Kata could be packaged without any binary assets. - If the binaries were not packaged, an additional step would be required before a Kata container could be created, something like [2]:
$ sudo kata-runtime pull [all-assets|image|initrd|kernel]
There are complications with this concept though, including:
- Where would we host such assets? - How would the kernel be handled? - How would the system determine which type of image to download? - How would updates to these assets be handled? We need to avoid / minimise the "stale container" scenario [3].
I am not familiar enough with Fedora and Debian packaging policies to know whether this approach is acceptable under those policies. I like it though because it gives the Kata Community control over producing and updating images without complicating package maintainers' lives. From a user perspective it would be nice not to know or worry about the sandbox kernel & rootfs. That's an implementation detail of Kata Containers. Stefan
I played around with a prototype a long time ago with getting clear containers deployable by container. Then you could use a kubernetes daemonset to deploy/manage additional runtimes. If I remember correctly, I failed at the time because crio was not reenterent (may be now). But since then I've also learned of nsenter and think that could be used to solve the problem, and do it in a much lighter weight way. Has anyone considered this use case recently? Thanks, Kevin ________________________________________ From: Stefan Hajnoczi [stefanha@redhat.com] Sent: Wednesday, March 27, 2019 9:05 AM To: Hunt, James O Cc: Lokesh Mandvekar; kata-dev Subject: Re: [kata-dev] kata image types compared On Tue, Mar 26, 2019 at 09:21:06AM +0000, Hunt, James O wrote:
Le mar. 26 mars 2019 à 08:02, Stefan Hajnoczi <stefanha@redhat.com> a écrit :
On Mon, Mar 25, 2019 at 04:42:17PM +0000, Venegas Munoz, Jose Carlos wrote: Lokesh: Out of the various options for building Kata images (initramfs vs disk image, systemd vs agent), which ones are friendliest for Fedora packaging?
There may be an alternative solution to the ongoing issue of packaging binaries images...
- Kata needs a binary rootfs image and a binary kernel image to boot the hypervisor. Once booted, the user's chosen (binary) disk image will be made available for the workload (you might be able to see where this is going... :) - Our OBS packaging currently packages images and kernels as a convenience to users. - Container managers like Docker also require binary images. But those images are not packaged - they are downloaded on demand ("docker pull"). - This leads to a potential solution to the packaging blocker: Kata could be packaged without any binary assets. - If the binaries were not packaged, an additional step would be required before a Kata container could be created, something like [2]:
$ sudo kata-runtime pull [all-assets|image|initrd|kernel]
There are complications with this concept though, including:
- Where would we host such assets? - How would the kernel be handled? - How would the system determine which type of image to download? - How would updates to these assets be handled? We need to avoid / minimise the "stale container" scenario [3].
I am not familiar enough with Fedora and Debian packaging policies to know whether this approach is acceptable under those policies. I like it though because it gives the Kata Community control over producing and updating images without complicating package maintainers' lives.
From a user perspective it would be nice not to know or worry about the sandbox kernel & rootfs. That's an implementation detail of Kata Containers.
Stefan
participants (4)
-
Fox, Kevin M
-
Hunt, James O
-
Stefan Hajnoczi
-
Venegas Munoz, Jose Carlos