Hey Allison,

Our goal in Kata is to be configurable with respect to hypervisors, so your project sounds quite interesting. In our experience, Kata ends up exercising many more features than are typical, and thus may be a good test vehicle.

Features include: -hotplug for CPU, memory, vfio-pci, virtio-blk/scsi -virtio: 9pFS, -net (with vhost backend), blk/scsi, serial, vsock -DAX, nvdimm (used for our rootfs today) -vhostuser (blk,net,scsi)

Is your work on github, or available in the open today?

Thanks, Eric

On Wed, Jun 06, 2018 at 03:24:48PM +0000, Xu Wang wrote:

Hi Allison, The library hypervisor sounds very interesting, do you have any publication or code released? For kata, when you talk about image format, do you mean those like "qcow2" or "raw"? We currently mainly use raw format devices for qemu. And if we insert devices into VM, we may insert loop devices, other block devices, or ceph rbds, etc. Xu

On Wed, Jun 6, 2018 11:17 PM, Allison Randal allison@lohutok.net wrote: I wasn't at the session where this started (because I was speaking on a

panel at the same time about edge computing, containers/VMs, and hybrid

approaches), but several people pointed me at this list, because I'm

currently working on a lightweight hypervisor written in Rust (on top of

KVM), as PhD research at the University of Cambridge.

I'm interested in getting more details on what features of QEMU are used

by Kata Containers. Specifically, what image formats and devices do you

need? This information would be equally valuable to a stripped-down

QEMU, so it wouldn't be a distraction from that work, just a side

benefit of informing other approaches to lightweight hypervisors.

The fundamental concept of my work is building a "library hypervisor",

somewhat analogous to the way unikernels are a "library operating

system". So, instead of a monolithic body of code, it is a collection of

Rust libraries that provide devices, support for various image formats,

an interface to KVM, etc. A final compiled hypervisor imports a minimal

set of libraries to get the minimal set of features that it needs.

I had considered Kata Containers as a possible example target, though I

was prioritizing it behind libvirt, minikube, LinuxKit, and a few other

potential targets. I wouldn't mind bumping Kata Containers up to a

higher priority, if people here would be interested in a prototype to try.

Allison

---

Kata-hypervisor mailing list

Kata-hypervisor@lists.katacontainers.io

http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-hypervisor

-- Xu WangCTO & Cofounder, Hypergithub/twitter/wechat: @gnawuxhttp://hyper.sh Hyper_: Make VM run like container

---

Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-hypervisor

On 6/6/18, 12:00 PM, "Allison Randal" allison@lohutok.net wrote:

Hi Xu and Eric,

Yes, by image formats I mean raw/qcow2/etc. Basically, I'm just looking to see which libraries Kata would need.

That's a healthy list of features. I don't have libraries for all of those yet, but let's see how far I can get and how fast.

The repos are on gitlab, but still private (so far I've been working alone). Now that others are potentially interested, I'll work on flipping the repos to public, since it's always nice to get more perspectives.

Allison

On 06/06/2018 11:37 AM, Ernst, Eric wrote: > Hey Allison, > > Our goal in Kata is to be configurable with respect to hypervisors, so > your project sounds quite interesting. In our experience, Kata ends up > exercising many more features than are typical, and thus may be a good > test vehicle. > > Features include: > -hotplug for CPU, memory, vfio-pci, virtio-blk/scsi > -virtio: 9pFS, -net (with vhost backend), blk/scsi, serial, vsock > -DAX, nvdimm (used for our rootfs today) > -vhostuser (blk,net,scsi) > > Is your work on github, or available in the open today? > > Thanks, > Eric > > > On Wed, Jun 06, 2018 at 03:24:48PM +0000, Xu Wang wrote: >> Hi Allison, >> The library hypervisor sounds very interesting, do you have any publication or >> code released? >> For kata, when you talk about image format, do you mean those like "qcow2" or >> "raw"? We currently mainly use raw format devices for qemu. >> And if we insert devices into VM, we may insert loop devices, other block >> devices, or ceph rbds, etc. >> Xu >> >> >> >> >> >> On Wed, Jun 6, 2018 11:17 PM, Allison Randal allison@lohutok.net wrote: >> I wasn't at the session where this started (because I was speaking on a >> >> panel at the same time about edge computing, containers/VMs, and hybrid >> >> approaches), but several people pointed me at this list, because I'm >> >> currently working on a lightweight hypervisor written in Rust (on top of >> >> KVM), as PhD research at the University of Cambridge. >> >> >> >> >> I'm interested in getting more details on what features of QEMU are used >> >> by Kata Containers. Specifically, what image formats and devices do you >> >> need? This information would be equally valuable to a stripped-down >> >> QEMU, so it wouldn't be a distraction from that work, just a side >> >> benefit of informing other approaches to lightweight hypervisors. >> >> >> >> >> >> >> >> The fundamental concept of my work is building a "library hypervisor", >> >> somewhat analogous to the way unikernels are a "library operating >> >> system". So, instead of a monolithic body of code, it is a collection of >> >> Rust libraries that provide devices, support for various image formats, >> >> an interface to KVM, etc. A final compiled hypervisor imports a minimal >> >> set of libraries to get the minimal set of features that it needs. >> >> >> >> >> I had considered Kata Containers as a possible example target, though I >> >> was prioritizing it behind libvirt, minikube, LinuxKit, and a few other >> >> potential targets. I wouldn't mind bumping Kata Containers up to a >> >> higher priority, if people here would be interested in a prototype to try. >> >> >> >> >> Allison >> >> >> >> >> _______________________________________________ >> >> Kata-hypervisor mailing list >> >> Kata-hypervisor@lists.katacontainers.io >> >> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-hypervisor >> >> >> >> >> >> >> -- >> Xu WangCTO & Cofounder, Hypergithub/twitter/wechat: @gnawuxhttp://hyper.sh >> Hyper_: Make VM run like container > >> _______________________________________________ >> Kata-hypervisor mailing list >> Kata-hypervisor@lists.katacontainers.io >> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-hypervisor >

On 06/09/2018 02:13 PM, Allison Randal wrote:> I've carved out one small piece, because there seem to be several groups

independently evaluating whether Rust makes sense in this context. This is the beginnings of a Rust library to interface with KVM, and could be used by any Rust hypervisor:

https://gitlab.com/preon/libkvm

By way of an update, my research has led me to the conclusion that virtual machines aren't the best approach for secure container isolation. The problem isn't only that the current implementations are slow and bloated (which is what I was working on), it's also that virtual machines aren't really very secure.

So, I've mostly stopped work on libkvm, though I may use it for low-level security and performance comparisons to other approaches. From what I understand, though, someone else has picked up the code I released as open source and continued working on it, so you may get the benefit of it anyway.

I'm still working on secure container isolation, as part of the broader concept of "composable infrastructure", so I'll let you know if some parts of my work end up being relevant to what you're doing. But, it might not be relevant, since it is so different.

Allison

On 08/06/2018 11:01 AM, Dr. David Alan Gilbert wrote:

Can I ask you what your main security worries with them are?

Virtual machines were a pretty neat idea back in the day when security was just getting started on "Hey, now that we can run more than one process on the machine at the same time, maybe we shouldn't give every process complete and unrestricted control over the entire machine" (for example http://doi.acm.org/10.1145/800122.803961, from 1973).

But, computer security has evolved in the past 60 years, and virtual machines haven't really kept up. There's plenty of research on how to make virtual machines more secure, but it isn't making its way into stable implementations. This is partly for the practical reason that it's time-consuming and expensive to update a large codebase, but it's also partly because the "improvements" often come with performance trade-offs that make virtual machines unusable. Just a few examples out of thousands:

https://doi.org/10.1145/3193111.3193118 https://doi.org/10.1145/3134600.3134648 https://doi.org/10.1109/TDSC.2015.2429132 https://doi.org/10.1109/TIFS.2015.2505680 https://doi.org/10.1145/2517488.2517489 https://www.usenix.org/conference/atc17/technical-sessions/presentation/gopa...

While containers aren't currently considered very secure either (e.g. https://doi.org/10.1109/i-Society.2016.7854163, https://doi.org/10.1145/3058060.3058085), the isolation that containers do provide is a collection of features added to more modern kernels (namespaces, cgroups, capabilities, seccomp, etc), in a modular approach that has allowed containers to more readily adopt existing advances in security, and also means containers can more easily pivot to incorporate new low-level security features that become available in the future. Which makes it interesting to explore low-level isolation features exposed directly to libcontainer (and similar).

Doing secure isolation without virtualization isn't a radical new idea, the earliest work on capabilities for secure process isolation started several years before the first work on virtual machines (for some of the earliest history, see https://homes.cs.washington.edu/~levy/capabook/). There are a number of independent threads of research heading this direction in recent years, here are a few examples:

https://www.usenix.org/conference/hotcloud18/presentation/williams https://doi.org/10.1145/3078861.3078882 https://doi.org/10.1145/3102980.3103008 https://doi.org/10.1145/3086567.3086572 https://doi.org/10.1109/DSN.2017.49

This isn't a criticism of the Kata project. In terms of what you can download and install in production today, virtual machines wrapped around containers are a reasonable option, and certainly fairly common practice. And, since Kata is an abstraction layer over hypervisor isolation, it could potentially evolve to use other forms of isolation in the future.

Allison

On 08/06/2018 05:53 PM, Paolo Bonzini wrote:

Nice. This is a bit more tricky to assess; it's an (important) QoI issue but it's not a fundamental problem with virtual machines in general or KVM in particular. In a perfect world, the device emulator would be confined using OS-level security features such as cgroups, capabilities, seccomp, etc. However, this is not always the case and in particular it's not for Kata containers (though its predecessor Clear Containers did some OS-level isolation of the device emulator).

Nod, I'm not convinced that either virtual machines or containers are fundamentally and universally a more secure approach. So much depends on implementation, and if you get really, really fundamental both are just a way of running an OS image as an isolated process on a kernel, so a case could be made that the differences between the two are relatively superficial.

Some of the interesting future directions are hybrid approaches that look at secure isolation features and how to combine them, without getting too caught up in whether those features are commonly associated with virtual machines or containers.

Note that these same modular features can be applied to virtual machines simply by running the device emulation in a container. This provides isolation for the parts that are most complex and deal the most with untrusted data from the guest.

Nod, more hybrid approaches.

I think it would be disingenuous, and it somewhat reminds me to Theo de Raadt's famous virtualization quote[1], to say that OS-level containers can be more secure than virtual machines.

I started with the preconception that virtual machines were more secure, and didn't give it up lightly. Which doesn't mean that all arguments in favor of container security are compelling. :) But, some are.

Capabilities and the like are nice, but software has bugs and that's why you defend in depth. Virtual machines are just an additional layer in the defend-in-depth onion.

That would be more compelling if the additional layer was substantially more secure. Two weak layers of security is better than one weak layer of security, but a single layer that combined the security features of those two layers would be equivalent to them (and likely perform better).

No hacker ever has broken one weak layer of security and decided not to go any farther because it wasn't worth the bother of breaking a second weak layer of security.

Allison