Improving isolation in Kata
Kata currently launches QEMU as user 'root' and I haven't found isolation mechanisms that would confine the QEMU process in the event of a guest escape. Inevitably there will be a security bug that allows a malicious guest to escape to QEMU, so it's important that the QEMU process is unprivileged and isolated. Running QEMU as 'root' simplifies 9p configuration by allowing the QEMU process to access files with any uid/gid. QEMU does support other 9p configurations where the QEMU process does not run as root, including the virtfs-proxy-helper privilege separation approach. Are there any other reasons for running the QEMU process as root? For comparison, libvirt launches QEMU as the unprivileged 'qemu' user with a unique SELinux context. This makes it much harder to access other guests or escape to the host if the guest compromises QEMU. Is there any work underway to improve isolation in Kata? Two directions that come to mind are: 1. Design isolation from scratch for Kata, including unprivileged uid/gid, namespaces, and SELinux. This requires changes to Kata because the QEMU process will not be able to open arbitrary files (e.g. device hotplug, 9p configuration). FWIW, relying on namespaces doesn't add a lot of security because if the guest is malicious, then that may well be because of a Linux namespaces security hole - the attacker could just use the same exploit again to escape from QEMU to the host. 2. Use libvirt-go (API) or libvirt-go-xml (XML) instead of govmm to get strong isolation. This would also slim down virtcontainers and remove the dependency on govmm (no more command-line munging and defining object models for QEMU command-line concepts). Stefan
Is there any work underway to improve isolation in Kata?
Two directions that come to mind are:
1. Design isolation from scratch for Kata, including unprivileged uid/gid, namespaces, and SELinux. This requires changes to Kata because the QEMU process will not be able to open arbitrary files (e.g. device hotplug, 9p configuration).
FWIW, relying on namespaces doesn't add a lot of security because if the guest is malicious, then that may well be because of a Linux namespaces security hole - the attacker could just use the same exploit again to escape from QEMU to the host.
2. Use libvirt-go (API) or libvirt-go-xml (XML) instead of govmm to get strong isolation. This would also slim down virtcontainers and remove the dependency on govmm (no more command-line munging and defining object models for QEMU command-line concepts).
Stefan
Hi Stefan, I did some research previously into filesystem encryption for the container guest. This was intended as an additional control for SEV containers. (Memory encryption is more valuable if the guest container filesystem isn't mounted unencrypted in the host.) Of the two options that you outline above, option #2 (libvirt integration) seems like the right choice architecturally. (I'll have to update my SEV patches, but libvirt integration will likely make them smaller overall.) This may also address my filesystem concerns as well. Does the SELinux policy used by libvirt also protect the mount directory for the container filesystem? Sincerely, Jesse Larrew MTS Software Security Architect AMD Security Architecture R&D jesse.larrew@amd.com O: +(1) 512-602-0092 (x50092) M: +(1) 512-791-4852
Hi Stefan, Thanks for bringing up this topic. About better isolation, we have 2 issues opened here: https://github.com/kata-containers/runtime/issues/344 https://github.com/kata-containers/runtime/issues/345 They're about cgroups and namespaces for the qemu process itself. You've mentioned it's not worth isolating the process through namespaces but I think it is since the kernel running inside the VM is very likely to be different from the kernel running on the host, meaning the exploit might be harder on the host itself even if it happened inside the guest. Now, I agree that running the VM as root is also a real breach regarding security and we also have an issue to address this concern here: https://github.com/kata-containers/runtime/issues/389 I think you summarized the blockers pretty well right now. If qemu is not root, we have some issues accessing the files on the host through 9p, but you also seem to provide a solution here and it'd be great if you could comment on the Github issue. One small thing that needs to happen too is to make sure we run kata-runtime with a user that at least can have access to /dev/kvm, but other than that, nothing else I can think of. Last thing, about libvirt-go, is there all the support needed by Kata (hotplug of a bunch of different device types) ? Thanks, Sebastien ________________________________________ From: Stefan Hajnoczi [stefanha@redhat.com] Sent: Monday, July 16, 2018 5:31 AM To: kata-dev@lists.katacontainers.io Cc: Daniel Berrange Subject: [kata-dev] Improving isolation in Kata Kata currently launches QEMU as user 'root' and I haven't found isolation mechanisms that would confine the QEMU process in the event of a guest escape. Inevitably there will be a security bug that allows a malicious guest to escape to QEMU, so it's important that the QEMU process is unprivileged and isolated. Running QEMU as 'root' simplifies 9p configuration by allowing the QEMU process to access files with any uid/gid. QEMU does support other 9p configurations where the QEMU process does not run as root, including the virtfs-proxy-helper privilege separation approach. Are there any other reasons for running the QEMU process as root? For comparison, libvirt launches QEMU as the unprivileged 'qemu' user with a unique SELinux context. This makes it much harder to access other guests or escape to the host if the guest compromises QEMU. Is there any work underway to improve isolation in Kata? Two directions that come to mind are: 1. Design isolation from scratch for Kata, including unprivileged uid/gid, namespaces, and SELinux. This requires changes to Kata because the QEMU process will not be able to open arbitrary files (e.g. device hotplug, 9p configuration). FWIW, relying on namespaces doesn't add a lot of security because if the guest is malicious, then that may well be because of a Linux namespaces security hole - the attacker could just use the same exploit again to escape from QEMU to the host. 2. Use libvirt-go (API) or libvirt-go-xml (XML) instead of govmm to get strong isolation. This would also slim down virtcontainers and remove the dependency on govmm (no more command-line munging and defining object models for QEMU command-line concepts). Stefan
On Tue, Jul 17, 2018 at 6:32 AM, Boeuf, Sebastien <sebastien.boeuf@intel.com> wrote:
Hi Stefan,
Thanks for bringing up this topic.
About better isolation, we have 2 issues opened here: https://github.com/kata-containers/runtime/issues/344 https://github.com/kata-containers/runtime/issues/345 They're about cgroups and namespaces for the qemu process itself. You've mentioned it's not worth isolating the process through namespaces but I think it is since the kernel running inside the VM is very likely to be different from the kernel running on the host, meaning the exploit might be harder on the host itself even if it happened inside the guest.
Now, I agree that running the VM as root is also a real breach regarding security and we also have an issue to address this concern here: https://github.com/kata-containers/runtime/issues/389 I think you summarized the blockers pretty well right now. If qemu is not root, we have some issues accessing the files on the host through 9p, but you also seem to provide a solution here and it'd be great if you could comment on the Github issue. One small thing that needs to happen too is to make sure we run kata-runtime with a user that at least can have access to /dev/kvm, but other than that, nothing else I can think of.
Last thing, about libvirt-go, is there all the support needed by Kata (hotplug of a bunch of different device types) ?
libvirt-go does have that and runv is using libvirt-go (https://github.com/hyperhq/runv/tree/master/hypervisor/libvirt) for a long time. Last I checked, there are still a few missing pieces of libvirt though: 1. vsock -- runv only supports vsock through the qemu hypervisor 2. vm templating -- runv works around it by using a customized qemu wrapper IIUC, libvirt is set to be hypervisor agnostic so I'm not sure if these qemu specific features can get in there. However, if we only consider basic usage of Kata, libvirt-go should work w/o issues.
Thanks, Sebastien
________________________________________ From: Stefan Hajnoczi [stefanha@redhat.com] Sent: Monday, July 16, 2018 5:31 AM To: kata-dev@lists.katacontainers.io Cc: Daniel Berrange Subject: [kata-dev] Improving isolation in Kata
Kata currently launches QEMU as user 'root' and I haven't found isolation mechanisms that would confine the QEMU process in the event of a guest escape.
Inevitably there will be a security bug that allows a malicious guest to escape to QEMU, so it's important that the QEMU process is unprivileged and isolated.
Running QEMU as 'root' simplifies 9p configuration by allowing the QEMU process to access files with any uid/gid. QEMU does support other 9p configurations where the QEMU process does not run as root, including the virtfs-proxy-helper privilege separation approach.
Are there any other reasons for running the QEMU process as root?
Creation of new namespaces, and handling of process cgroups on the host side, would require CAP_SYS_ADMIN. Though we do not have them right now (only netns is enforced), we need to be aware of the limitation since there are people working on adding these abilities at host side.
For comparison, libvirt launches QEMU as the unprivileged 'qemu' user with a unique SELinux context. This makes it much harder to access other guests or escape to the host if the guest compromises QEMU.
Is there any work underway to improve isolation in Kata?
Two directions that come to mind are:
1. Design isolation from scratch for Kata, including unprivileged uid/gid, namespaces, and SELinux. This requires changes to Kata because the QEMU process will not be able to open arbitrary files (e.g. device hotplug, 9p configuration).
Does it prevent general device hotplug, or just hotplugging of local devices (e.g., `/dev/sdb` etc.)? I'm mostly considering two cases: 1. remote devices like rbd/iscsi/nbd 2. CPU and memory devices
FWIW, relying on namespaces doesn't add a lot of security because if the guest is malicious, then that may well be because of a Linux namespaces security hole - the attacker could just use the same exploit again to escape from QEMU to the host.
2. Use libvirt-go (API) or libvirt-go-xml (XML) instead of govmm to get strong isolation. This would also slim down virtcontainers and remove the dependency on govmm (no more command-line munging and defining object models for QEMU command-line concepts).
Even if we support libvirt, I would think it still reasonable to keep both qemu and libvirt-go implementations due to the feature differences. OTOH, libvirt-go can be a good place to support other hypervisors like xen. Cheers, Tao -- bergwolf@hyper.sh
On Mon, Jul 16, 2018 at 10:04:05PM +0000, Larrew, Jesse wrote:
Is there any work underway to improve isolation in Kata?
Two directions that come to mind are:
1. Design isolation from scratch for Kata, including unprivileged uid/gid, namespaces, and SELinux. This requires changes to Kata because the QEMU process will not be able to open arbitrary files (e.g. device hotplug, 9p configuration).
FWIW, relying on namespaces doesn't add a lot of security because if the guest is malicious, then that may well be because of a Linux namespaces security hole - the attacker could just use the same exploit again to escape from QEMU to the host.
2. Use libvirt-go (API) or libvirt-go-xml (XML) instead of govmm to get strong isolation. This would also slim down virtcontainers and remove the dependency on govmm (no more command-line munging and defining object models for QEMU command-line concepts).
Stefan
Hi Stefan,
I did some research previously into filesystem encryption for the container guest. This was intended as an additional control for SEV containers. (Memory encryption is more valuable if the guest container filesystem isn't mounted unencrypted in the host.)
Yeah, with SEV you would not want the container filesystem mounted in the host, as that puts a hole in the security SEV offers.
Of the two options that you outline above, option #2 (libvirt integration) seems like the right choice architecturally. (I'll have to update my SEV patches, but libvirt integration will likely make them smaller overall.) This may also address my filesystem concerns as well. Does the SELinux policy used by libvirt also protect the mount directory for the container filesystem?
Yes, the SELinux (sVirt) policy applied by libvirt confines every resource used by the guest. Each guest VM gets a common base label svirt_t, combined with a unique category (eg c123,c643). Any files/resources that are to be accessible by the guest must be labelled with a label svirt_image_t combined with the matching category (c123,c643). Normally libvirt will automagically assign the right svirt_image_t label to all files that the guest is configured to access (in its XML config). With 9p filesystem passthrough there is a caveat that libvirt won't (currently) recursively label an entire tree itself. The app using libvirt, however, still has the ability todoo this labelling upfront to grant access as needed. FYI, libvirt gained support for the SEV feature in its recent release. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, Jul 16, 2018 at 10:32:22PM +0000, Boeuf, Sebastien wrote:
Hi Stefan,
Thanks for bringing up this topic.
About better isolation, we have 2 issues opened here: https://github.com/kata-containers/runtime/issues/344 https://github.com/kata-containers/runtime/issues/345 They're about cgroups and namespaces for the qemu process itself. You've mentioned it's not worth isolating the process through namespaces but I think it is since the kernel running inside the VM is very likely to be different from the kernel running on the host, meaning the exploit might be harder on the host itself even if it happened inside the guest.
I think perhaps stefan oversimplified. The better point is that namespaces alone are not sufficient security protection against a malicious attacker, especially if the guest is relying on the same namespace separation for the containers it is running. When libvirt runs QEMU it will use multiple layers of security protection. The most important is sVirt/SELinux which applies mandatory access control, whitelisting resources that the guest is permitted to touch. Second, it will apply regular discretionary access control by running QEMU as an unprivileged user ID, with the privileged libvirtd setting file permissions on the few resources it is allowed to use. Third it will use namespaces, setting up a custom mount namespace which hides most of /dev, only allowing the few device nodes the guest is permitted to use. Fourth it will use cgroups to apply resource controls via a number of the controllers the kernel exposes. Finally it can use iptables+ebtables to setup filtering of guest traffic to provide protection against MAC & IP address spooofing and other arbitrary rules. All of these layers are configurable so you can decide how much protection you get, but by default you'll get all of them except the firewalling.
Now, I agree that running the VM as root is also a real breach regarding security and we also have an issue to address this concern here: https://github.com/kata-containers/runtime/issues/389 I think you summarized the blockers pretty well right now. If qemu is not root, we have some issues accessing the files on the host through 9p, but you also seem to provide a solution here and it'd be great if you could comment on the Github issue.
Yes, 9p is the biggest problem with running non-root, and would require use of some kind of proxied solution. You could perhaps use capabilities to grant the CAP_DAC_OVERRIDE bit but this does reduce the level of protection you have quite dramatically, so proxy is probably better if 9p is a must have.
One small thing that needs to happen too is to make sure we run kata-runtime with a user that at least can have access to /dev/kvm, but other than that, nothing else I can think of.
Last thing, about libvirt-go, is there all the support needed by Kata (hotplug of a bunch of different device types) ?
libvirt-go is just a shim to the main native libvirt.so library API, which in turn talks to libvirtd. The libvirtd QEMU driver supports hotplug for a wide array of device types (disks, disk controllers, leases, PCI & USB host devices, USB redirection devices, character devices, RNG devices, memory DIMMS, shmem devices, watchdogs, input devices, and vsock devices). Libvirt would be happy to see more devices support as people raise a need for it. Given the wide & diverse usage of libvirt I'd expect we would support the vast majority of functionality you currently need, and any feature gaps would be fairly small & straightforward to address. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Jul 17, 2018 at 10:50:35AM +0800, Tao Peng wrote:
On Tue, Jul 17, 2018 at 6:32 AM, Boeuf, Sebastien <sebastien.boeuf@intel.com> wrote:
Hi Stefan,
Thanks for bringing up this topic.
About better isolation, we have 2 issues opened here: https://github.com/kata-containers/runtime/issues/344 https://github.com/kata-containers/runtime/issues/345 They're about cgroups and namespaces for the qemu process itself. You've mentioned it's not worth isolating the process through namespaces but I think it is since the kernel running inside the VM is very likely to be different from the kernel running on the host, meaning the exploit might be harder on the host itself even if it happened inside the guest.
Now, I agree that running the VM as root is also a real breach regarding security and we also have an issue to address this concern here: https://github.com/kata-containers/runtime/issues/389 I think you summarized the blockers pretty well right now. If qemu is not root, we have some issues accessing the files on the host through 9p, but you also seem to provide a solution here and it'd be great if you could comment on the Github issue. One small thing that needs to happen too is to make sure we run kata-runtime with a user that at least can have access to /dev/kvm, but other than that, nothing else I can think of.
Last thing, about libvirt-go, is there all the support needed by Kata (hotplug of a bunch of different device types) ?
libvirt-go does have that and runv is using libvirt-go (https://github.com/hyperhq/runv/tree/master/hypervisor/libvirt) for a long time. Last I checked, there are still a few missing pieces of libvirt though: 1. vsock -- runv only supports vsock through the qemu hypervisor
Libvirt does support vsock now
2. vm templating -- runv works around it by using a customized qemu wrapper
I'm not sure what you're meaning by vm templating here ? As a conceptual design policy, libvirt tends to focus on providing the mechanism, and leaves policy upto the mgmt application to decide. What most people describe as "templating" is a usage a policy decision around disk images. Libvirt has APIs for managing and manipulating storage volumes, which can be used to implement templating concepts. Offhand I can't think of a reason why it would require use of a custom qemu wrapper.
IIUC, libvirt is set to be hypervisor agnostic so I'm not sure if these qemu specific features can get in there. However, if we only consider basic usage of Kata, libvirt-go should work w/o issues.
Hypervisor agnostic doesn't imply that we must stick to the lowest common denominator. It simply means that if we have a concept implemented for one hypervisor, we'll aim to implement it with the same APIs & configuration syntax when we add to for a second hypervisor. A large portion of the functionality in libvirt's QEMU driver is found only in QEMU and it is perfectly ok that no other hypervisor will ever support it. As a failsafe fallback, we do provide a way to pass arbitrary command line arguments to QEMU, however, we tend to discourage its usage, as we prefer to officially support any features that mgmt apps have a need for. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Jul 17, 2018 at 5:05 PM, Daniel P. Berrangé <berrange@redhat.com> wrote:
On Tue, Jul 17, 2018 at 10:50:35AM +0800, Tao Peng wrote:
On Tue, Jul 17, 2018 at 6:32 AM, Boeuf, Sebastien <sebastien.boeuf@intel.com> wrote:
Hi Stefan,
Thanks for bringing up this topic.
About better isolation, we have 2 issues opened here: https://github.com/kata-containers/runtime/issues/344 https://github.com/kata-containers/runtime/issues/345 They're about cgroups and namespaces for the qemu process itself. You've mentioned it's not worth isolating the process through namespaces but I think it is since the kernel running inside the VM is very likely to be different from the kernel running on the host, meaning the exploit might be harder on the host itself even if it happened inside the guest.
Now, I agree that running the VM as root is also a real breach regarding security and we also have an issue to address this concern here: https://github.com/kata-containers/runtime/issues/389 I think you summarized the blockers pretty well right now. If qemu is not root, we have some issues accessing the files on the host through 9p, but you also seem to provide a solution here and it'd be great if you could comment on the Github issue. One small thing that needs to happen too is to make sure we run kata-runtime with a user that at least can have access to /dev/kvm, but other than that, nothing else I can think of.
Last thing, about libvirt-go, is there all the support needed by Kata (hotplug of a bunch of different device types) ?
libvirt-go does have that and runv is using libvirt-go (https://github.com/hyperhq/runv/tree/master/hypervisor/libvirt) for a long time. Last I checked, there are still a few missing pieces of libvirt though: 1. vsock -- runv only supports vsock through the qemu hypervisor
Libvirt does support vsock now
Sorry I meant to say libvirt-go -- I only checked libvirt-go -- and vsock does not show up there yet, https://github.com/libvirt/libvirt-go/search?q=vsock&unscoped_q=vsock. Now that libvirt does support vsock, we only need to add the missing piece to libvirt-go.
2. vm templating -- runv works around it by using a customized qemu wrapper
I'm not sure what you're meaning by vm templating here ? As a conceptual design policy, libvirt tends to focus on providing the mechanism, and leaves policy upto the mgmt application to decide. What most people describe as "templating" is a usage a policy decision around disk images. Libvirt has APIs for managing and manipulating storage volumes, which can be used to implement templating concepts. Offhand I can't think of a reason why it would require use of a custom qemu wrapper.
This one is still WiP to get it merged by qemu upstream -- we want to allow qemu to do migration without copying shared memory. Basically a new migration capability "bypass-share-memory" is added before qemu starts migration. We use this capability to implement so-called vm templating feature in both runv and kata.
IIUC, libvirt is set to be hypervisor agnostic so I'm not sure if these qemu specific features can get in there. However, if we only consider basic usage of Kata, libvirt-go should work w/o issues.
Hypervisor agnostic doesn't imply that we must stick to the lowest common denominator. It simply means that if we have a concept implemented for one hypervisor, we'll aim to implement it with the same APIs & configuration syntax when we add to for a second hypervisor. A large portion of the functionality in libvirt's QEMU driver is found only in QEMU and it is perfectly ok that no other hypervisor will ever support it.
Thanks for the explanation! Now I understand libvirt policy better and it looks straight-forward to add the "bypass-share-memory" migration capability to libvirt when the qemu community accepts the patch. So the two points I listed are not hard blockers for libvirt support in kata. Cheers, Tao
On Tue, Jul 17, 2018 at 05:39:36PM +0800, Tao Peng wrote:
On Tue, Jul 17, 2018 at 5:05 PM, Daniel P. Berrangé <berrange@redhat.com> wrote:
On Tue, Jul 17, 2018 at 10:50:35AM +0800, Tao Peng wrote:
On Tue, Jul 17, 2018 at 6:32 AM, Boeuf, Sebastien <sebastien.boeuf@intel.com> wrote:
Hi Stefan,
Thanks for bringing up this topic.
About better isolation, we have 2 issues opened here: https://github.com/kata-containers/runtime/issues/344 https://github.com/kata-containers/runtime/issues/345 They're about cgroups and namespaces for the qemu process itself. You've mentioned it's not worth isolating the process through namespaces but I think it is since the kernel running inside the VM is very likely to be different from the kernel running on the host, meaning the exploit might be harder on the host itself even if it happened inside the guest.
Now, I agree that running the VM as root is also a real breach regarding security and we also have an issue to address this concern here: https://github.com/kata-containers/runtime/issues/389 I think you summarized the blockers pretty well right now. If qemu is not root, we have some issues accessing the files on the host through 9p, but you also seem to provide a solution here and it'd be great if you could comment on the Github issue. One small thing that needs to happen too is to make sure we run kata-runtime with a user that at least can have access to /dev/kvm, but other than that, nothing else I can think of.
Last thing, about libvirt-go, is there all the support needed by Kata (hotplug of a bunch of different device types) ?
libvirt-go does have that and runv is using libvirt-go (https://github.com/hyperhq/runv/tree/master/hypervisor/libvirt) for a long time. Last I checked, there are still a few missing pieces of libvirt though: 1. vsock -- runv only supports vsock through the qemu hypervisor
Libvirt does support vsock now
Sorry I meant to say libvirt-go -- I only checked libvirt-go -- and vsock does not show up there yet, https://github.com/libvirt/libvirt-go/search?q=vsock&unscoped_q=vsock. Now that libvirt does support vsock, we only need to add the missing piece to libvirt-go.
There is nothing special needed by libvirt-go - it is simply an API binding to libvirt and there are no custom APIs involved in vsock - it was simply a new part of the XML schema which doesn't involve API changes. There is a separate libvirt-go-xml package which maps the XML schemas into Go structs, and that does have support for vsock
2. vm templating -- runv works around it by using a customized qemu wrapper
I'm not sure what you're meaning by vm templating here ? As a conceptual design policy, libvirt tends to focus on providing the mechanism, and leaves policy upto the mgmt application to decide. What most people describe as "templating" is a usage a policy decision around disk images. Libvirt has APIs for managing and manipulating storage volumes, which can be used to implement templating concepts. Offhand I can't think of a reason why it would require use of a custom qemu wrapper.
This one is still WiP to get it merged by qemu upstream -- we want to allow qemu to do migration without copying shared memory. Basically a new migration capability "bypass-share-memory" is added before qemu starts migration. We use this capability to implement so-called vm templating feature in both runv and kata.
Ok, as & when QEMU merges that, it is something we could expose via libvirt in some manner. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Hi Stefan, Daniel, Thanks for the feedback here. I have just tried the virtfs-proxy-helper and it works like a charm, allowing me to run Qemu from a non root user. Now, I haven't tried this yet, but I wanted to know if you thought about any reason using virtio-net-pci would not work by using the vhost backend like this: -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,mq=on,vectors=18 Same question about block devices. We're doing heavy uses of virtio-scsi in order to share a block device (our container rootfs) with the guest. Is it possible to run Qemu non root in this case ? Should I put the user running Qemu in the disk group so that Qemu can read/write to this block device ? Last use case is VFIO for device passthrough. After we unbind the device from its driver on the host, should we change the ownership of the device created (something like /dev/vfio/X) after it has been bound to vfio-pci driver ? Thanks Sebastien ________________________________________ From: Daniel P. Berrangé [berrange@redhat.com] Sent: Tuesday, July 17, 2018 1:59 AM To: Boeuf, Sebastien Cc: Stefan Hajnoczi; kata-dev@lists.katacontainers.io Subject: Re: [kata-dev] Improving isolation in Kata On Mon, Jul 16, 2018 at 10:32:22PM +0000, Boeuf, Sebastien wrote:
Hi Stefan,
Thanks for bringing up this topic.
About better isolation, we have 2 issues opened here: https://github.com/kata-containers/runtime/issues/344 https://github.com/kata-containers/runtime/issues/345 They're about cgroups and namespaces for the qemu process itself. You've mentioned it's not worth isolating the process through namespaces but I think it is since the kernel running inside the VM is very likely to be different from the kernel running on the host, meaning the exploit might be harder on the host itself even if it happened inside the guest.
I think perhaps stefan oversimplified. The better point is that namespaces alone are not sufficient security protection against a malicious attacker, especially if the guest is relying on the same namespace separation for the containers it is running. When libvirt runs QEMU it will use multiple layers of security protection. The most important is sVirt/SELinux which applies mandatory access control, whitelisting resources that the guest is permitted to touch. Second, it will apply regular discretionary access control by running QEMU as an unprivileged user ID, with the privileged libvirtd setting file permissions on the few resources it is allowed to use. Third it will use namespaces, setting up a custom mount namespace which hides most of /dev, only allowing the few device nodes the guest is permitted to use. Fourth it will use cgroups to apply resource controls via a number of the controllers the kernel exposes. Finally it can use iptables+ebtables to setup filtering of guest traffic to provide protection against MAC & IP address spooofing and other arbitrary rules. All of these layers are configurable so you can decide how much protection you get, but by default you'll get all of them except the firewalling.
Now, I agree that running the VM as root is also a real breach regarding security and we also have an issue to address this concern here: https://github.com/kata-containers/runtime/issues/389 I think you summarized the blockers pretty well right now. If qemu is not root, we have some issues accessing the files on the host through 9p, but you also seem to provide a solution here and it'd be great if you could comment on the Github issue.
Yes, 9p is the biggest problem with running non-root, and would require use of some kind of proxied solution. You could perhaps use capabilities to grant the CAP_DAC_OVERRIDE bit but this does reduce the level of protection you have quite dramatically, so proxy is probably better if 9p is a must have.
One small thing that needs to happen too is to make sure we run kata-runtime with a user that at least can have access to /dev/kvm, but other than that, nothing else I can think of.
Last thing, about libvirt-go, is there all the support needed by Kata (hotplug of a bunch of different device types) ?
libvirt-go is just a shim to the main native libvirt.so library API, which in turn talks to libvirtd. The libvirtd QEMU driver supports hotplug for a wide array of device types (disks, disk controllers, leases, PCI & USB host devices, USB redirection devices, character devices, RNG devices, memory DIMMS, shmem devices, watchdogs, input devices, and vsock devices). Libvirt would be happy to see more devices support as people raise a need for it. Given the wide & diverse usage of libvirt I'd expect we would support the vast majority of functionality you currently need, and any feature gaps would be fairly small & straightforward to address. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Jul 17, 2018 at 10:25:53PM +0000, Boeuf, Sebastien wrote:
Hi Stefan, Daniel,
Thanks for the feedback here. I have just tried the virtfs-proxy-helper and it works like a charm, allowing me to run Qemu from a non root user. Now, I haven't tried this yet, but I wanted to know if you thought about any reason using virtio-net-pci would not work by using the vhost backend like this: -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,mq=on,vectors=18
Yes, libvirt supports passing multiple FDs for vhost, and setting mq/vectors properties.
Same question about block devices. We're doing heavy uses of virtio-scsi in order to share a block device (our container rootfs) with the guest. Is it possible to run Qemu non root in this case ? Should I put the user running Qemu in the disk group so that Qemu can read/write to this block device ?
What happens is that the libvirtd daemon is running as root. When launching a guest, it thus has privileges to change ownership of the block device so that QEMU can access it while running as non-root. So there's no need to change block device permissiosn or set group membership - libvirt will take care of changing it for you.
Last use case is VFIO for device passthrough. After we unbind the device from its driver on the host, should we change the ownership of the device created (something like /dev/vfio/X) after it has been bound to vfio-pci driver ?
Again, libvirt would take care of any permissions change that is required. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Hi Daniel, Ok, got it about libvirt being root and being able to change any ownership on files and devices. As we run our kata-runtime as root, it can exactly do the same thing as libvirt does, and I'd like to understand the details about the way changing ownership works. Does libvirt change the ownership of the device back to its initial ownership once it gets "released" by Qemu (when Qemu terminates) ? Also, the fies created will be owned by the non-root user being chosen, which modifies the security associated with those files. When Qemu runs root, only root can access/modify them on the host, but by running Qemu as non-root, they will be more accessible since they won't be owned by root. Thanks, Sebastien ________________________________________ From: Daniel P. Berrangé [berrange@redhat.com] Sent: Wednesday, July 18, 2018 12:14 AM To: Boeuf, Sebastien Cc: Stefan Hajnoczi; kata-dev@lists.katacontainers.io Subject: Re: [kata-dev] Improving isolation in Kata On Tue, Jul 17, 2018 at 10:25:53PM +0000, Boeuf, Sebastien wrote:
Hi Stefan, Daniel,
Thanks for the feedback here. I have just tried the virtfs-proxy-helper and it works like a charm, allowing me to run Qemu from a non root user. Now, I haven't tried this yet, but I wanted to know if you thought about any reason using virtio-net-pci would not work by using the vhost backend like this: -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,mq=on,vectors=18
Yes, libvirt supports passing multiple FDs for vhost, and setting mq/vectors properties.
Same question about block devices. We're doing heavy uses of virtio-scsi in order to share a block device (our container rootfs) with the guest. Is it possible to run Qemu non root in this case ? Should I put the user running Qemu in the disk group so that Qemu can read/write to this block device ?
What happens is that the libvirtd daemon is running as root. When launching a guest, it thus has privileges to change ownership of the block device so that QEMU can access it while running as non-root. So there's no need to change block device permissiosn or set group membership - libvirt will take care of changing it for you.
Last use case is VFIO for device passthrough. After we unbind the device from its driver on the host, should we change the ownership of the device created (something like /dev/vfio/X) after it has been bound to vfio-pci driver ?
Again, libvirt would take care of any permissions change that is required. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, Jul 23, 2018 at 10:34:11PM +0000, Boeuf, Sebastien wrote:
Hi Daniel,
Ok, got it about libvirt being root and being able to change any ownership on files and devices. As we run our kata-runtime as root, it can exactly do the same thing as libvirt does, and I'd like to understand the details about the way changing ownership works.
Does libvirt change the ownership of the device back to its initial ownership once it gets "released" by Qemu (when Qemu terminates) ?
It will reset ownership back to root when QEMU is quit, except for files that are designated sharable by multiple VMs.
Also, the fies created will be owned by the non-root user being chosen, which modifies the security associated with those files. When Qemu runs root, only root can access/modify them on the host, but by running Qemu as non-root, they will be more accessible since they won't be owned by root.
File ownership doesn't influence how accessible files are - file permissions bits do that. If the file permissions are rwx-------- then only the file owner can access it, regardless of whether the file owner is root or a non-root account. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Jul 24, 2018 at 09:19:47AM +0100, Daniel P. Berrangé wrote:
On Mon, Jul 23, 2018 at 10:34:11PM +0000, Boeuf, Sebastien wrote:
Also, the fies created will be owned by the non-root user being chosen, which modifies the security associated with those files. When Qemu runs root, only root can access/modify them on the host, but by running Qemu as non-root, they will be more accessible since they won't be owned by root.
File ownership doesn't influence how accessible files are - file permissions bits do that. If the file permissions are rwx-------- then only the file owner can access it, regardless of whether the file owner is root or a non-root account.
On top of this libvirt uses SELinux to further restrict access to the files. My understanding is each QEMU process runs with its own SELinux context so that 2 VMs cannot access each other's disk images, even if the QEMU processes have the same qemu:qemu uid/gid pair. In other words, the access is even more tightly controlled than just the UNIX file permissions, it's done on a per-VM basis. Stefan
On Tue, Jul 24, 2018 at 11:40:14AM +0100, Stefan Hajnoczi wrote:
On Tue, Jul 24, 2018 at 09:19:47AM +0100, Daniel P. Berrangé wrote:
On Mon, Jul 23, 2018 at 10:34:11PM +0000, Boeuf, Sebastien wrote:
Also, the fies created will be owned by the non-root user being chosen, which modifies the security associated with those files. When Qemu runs root, only root can access/modify them on the host, but by running Qemu as non-root, they will be more accessible since they won't be owned by root.
File ownership doesn't influence how accessible files are - file permissions bits do that. If the file permissions are rwx-------- then only the file owner can access it, regardless of whether the file owner is root or a non-root account.
On top of this libvirt uses SELinux to further restrict access to the files. My understanding is each QEMU process runs with its own SELinux context so that 2 VMs cannot access each other's disk images, even if the QEMU processes have the same qemu:qemu uid/gid pair.
In other words, the access is even more tightly controlled than just the UNIX file permissions, it's done on a per-VM basis.
Yes, out of the box, each VM will get a distinct SELinux context (using random MCS tag), while sharing the same qemu:qemu user/group. The app can opt-in to using a separate uid:gid per VM too, if they have a range of uid:gids reserved for use by VMs. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Thanks for your answers Daniel and Stefan, This helps :) Sebastien ________________________________________ From: Daniel P. Berrangé [berrange@redhat.com] Sent: Tuesday, July 24, 2018 3:42 AM To: Stefan Hajnoczi Cc: Boeuf, Sebastien; kata-dev@lists.katacontainers.io Subject: Re: [kata-dev] Improving isolation in Kata On Tue, Jul 24, 2018 at 11:40:14AM +0100, Stefan Hajnoczi wrote:
On Tue, Jul 24, 2018 at 09:19:47AM +0100, Daniel P. Berrangé wrote:
On Mon, Jul 23, 2018 at 10:34:11PM +0000, Boeuf, Sebastien wrote:
Also, the fies created will be owned by the non-root user being chosen, which modifies the security associated with those files. When Qemu runs root, only root can access/modify them on the host, but by running Qemu as non-root, they will be more accessible since they won't be owned by root.
File ownership doesn't influence how accessible files are - file permissions bits do that. If the file permissions are rwx-------- then only the file owner can access it, regardless of whether the file owner is root or a non-root account.
On top of this libvirt uses SELinux to further restrict access to the files. My understanding is each QEMU process runs with its own SELinux context so that 2 VMs cannot access each other's disk images, even if the QEMU processes have the same qemu:qemu uid/gid pair.
In other words, the access is even more tightly controlled than just the UNIX file permissions, it's done on a per-VM basis.
Yes, out of the box, each VM will get a distinct SELinux context (using random MCS tag), while sharing the same qemu:qemu user/group. The app can opt-in to using a separate uid:gid per VM too, if they have a range of uid:gids reserved for use by VMs. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (5)
-
Boeuf, Sebastien
-
Daniel P. Berrangé
-
Larrew, Jesse
-
Stefan Hajnoczi
-
Tao Peng