kata in non-baremetal environments
Hi, it seems that nowadays a large chunk of k8s deployments being run in virtualized enviroments (i.e. some form of public/private IaaS system). So using kata-containers in such an enviroment means that the VMs spawned by kata are running nested inside the VMs of the k8s cluster. Given that nested virtualization adds quite a bit of overhead (in certain cases I believe) and that it still seems to be considered experimental [0]. Is that really the way to go or is kata-containers really more just aiming for baremetal k8s deployments? regards, Ralf [0] https://www.linux-kvm.org/page/Nested_Guests
* Ralf Haferkamp (rhafer@suse.com) wrote:
Hi,
it seems that nowadays a large chunk of k8s deployments being run in virtualized enviroments (i.e. some form of public/private IaaS system). So using kata-containers in such an enviroment means that the VMs spawned by kata are running nested inside the VMs of the k8s cluster.
Given that nested virtualization adds quite a bit of overhead (in certain cases I believe) and that it still seems to be considered experimental [0]. Is that really the way to go or is kata-containers really more just aiming for baremetal k8s deployments?
That page is a bit out of date; I'd say that nesting is finally getting usable. Indeed if you follow the bugzilla links on there, the RH bz 1076294 got duped in December to bz 1738741 which was marked verified a few weeks back. While I'd still suggest some caution with nesting, if you get all your hypervisors in a line it does work now. Testing of it varies a bit; e.g. there's quite a bit of kvm-on-kvm nest testing; there's probably less testing on the various KVM under other things nesting. Dave
regards, Ralf
[0] https://www.linux-kvm.org/page/Nested_Guests
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
Testing of it varies a bit; e.g. there's quite a bit of kvm-on-kvm nest testing; there's probably less testing on the various KVM under other things nesting.
Just to note then, and to add a touch of assurance, the majority of the Kata CI runs under nested mode afaik. The metrics do not (as we need the stability of bare metal for reproducible numbers in a reasonable timeframe), and a couple of the arch's run bare metal CIs. But, all the x86 CIs run nested VMs in the cloud. Graham --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
On Tue, Feb 04, 2020 at 10:38:27AM +0000, Dr. David Alan Gilbert wrote:
* Ralf Haferkamp (rhafer@suse.com) wrote:
[...]
That page is a bit out of date; I'd say that nesting is finally getting usable. Indeed if you follow the bugzilla links on there, the RH bz 1076294 got duped in December to bz 1738741 which was marked verified a few weeks back.
While I'd still suggest some caution with nesting, if you get all your hypervisors in a line it does work now.
Testing of it varies a bit; e.g. there's quite a bit of kvm-on-kvm nest testing; there's probably less testing on the various KVM under other things nesting.
[...]
As an addendum to what Dave wrote, yes, the above page will be retired in favor of this in-progress draft (once I address all the feedback, then it'll be ready for merge in the upstream kernel documentation) about nested virt: https://marc.info/?l=kvm&m=158108941605311&w=2 https://marc.info/?t=158108958800002&r=1&w=2 [...] -- /kashyap
participants (4)
-
Dr. David Alan Gilbert
-
Kashyap Chamarthy
-
Ralf Haferkamp
-
Whaley, Graham