[Kata-hypervisor] Link to the Hypervisor Prototype

Ben Armstrong Ben.Armstrong at microsoft.com
Thu Jun 7 04:52:58 UTC 2018


In Hyper-V we have virtual machines with no PCI and a synthetic interrupt controller - which also has no problems taking advantage of posted interrupts.  I do not know what Xen implemented here - however emulation of all these devices is not necessary.

Cheers,
Ben

-----Original Message-----
From: Anthony Liguori <aliguori at amazon.com> 
Sent: Wednesday, June 6, 2018 5:51 PM
To: Jon Olson <jonolson at google.com>
Cc: kata-hypervisor at lists.katacontainers.io
Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype

Jon Olson <jonolson at google.com> writes:

> (note: currently at an off-site: responses may be delayed or worse)
>
> The interest that I had (and at the risk of speaking for him and being 
> corrected, I think Ben as well) was in a VM *shape* focussed on 
> something suitable for containers. In particular something based on a 
> modern set of hardware features and minimizing the need for emulation 
> of anything (PITs, PICs, PCI controllers, etc.)

I get what you're after but I've grown to believe it's not the right thing.

You really want to present an APIC because then you get APIC-V and Posted Interrupts.  This is the path Xen went with PV and it's boxed them into a corner where not only are event channels far worse than APIC-based delivery (they are inherently unfair) but now you don't get the performance boost that comes from Posted Interrupts due to that feature.

PCI is really not that bad at all to emulate and you have to do discovery in some way.  Again, lots of bad examples here (Xenstore for
instance) but I don't mean to pick on Xen here.

The complexity of emulation is not so bad.  Part of what explodes QEMU in terms of complexity is the backend implementations.  Once you need to support VMDK and QCOW2, you've got a massive code base.

I know why folks want v9fs too but man that's a complex protocol and mapping it to POSIX is very complex.  The legacy device emulation is the least of your concerns IMHO.

> I would agree that for what qemu does that it's hard to do better, but 
> we're actively uninterested in many of the things qemu does being 
> supported by our offering. They don't help workloads we care about and 
> they present "challenging" surface areas from a security perspective.
>
> Note that Google does actually have such a VM, aimed at containers and 
> open source, implemented in Rust for Chrome OS. We are not using it in 
> Cloud currently (and to be honest I haven't dived into it beyond being 
> highly amused at the horrifying kernel command line that explicitly 
> MMIO maps app of their virtio devices). This was on a production Pixel 
> book, so nothing fancy or proprietary going on there.

I like crosvm and have looked at the code pretty deeply.  It actually has a bit of legacy emulation but it's naive in approach to it.  I'm not sure splitting the device model into a separate process is the right approach but I like the fact that it's at least doing something different.

I think if you are going to do legacy emulation, you need to do it right and completely.  You can get stuff booting with a UART that turns a THR write into a putchar() call but ultimately, you need to implement FIFO and flow control properly.

Almost every device model in QEMU started out as a bare bones minimum function model but grew over time because of need into what it is today.

Regards,

Anthony Liguori

>
> Jon
>
> On Wed, Jun 6, 2018, 4:14 PM Anthony Liguori <aliguori at amazon.com> wrote:
>
>> "Jessie Frazelle (TUPPERWARE)" <jessfraz at microsoft.com> writes:
>>
>> > Sorry outlook cut off the first line... windows...
>> >
>> > so rust :D
>>
>> I was able to join the prep call at least and I raised this during 
>> that discussion.  I'm not sure what problem this group is trying to tackle.
>>
>> What was brought up on the call is a "cloud VMM" as an alternative to 
>> QEMU but I don't really agree with that premise.  For the problem 
>> space that QEMU tackles, it's hard to do better than it does and it 
>> tackles a huge space.
>>
>> Cloud is an extremely ambiguous term too.  In another part of this 
>> thread, qcow2 was mentioned as a target which I don't think at all is 
>> something that is interesting if you are an infrastructure provider.
>>
>> There's lot of stuff that I am interested in (particularly around 
>> super fast start up time and super low overhead to get high density) 
>> but that's only possible to achieve if you constrain the problem 
>> space tremendously.
>>
>> Regards,
>>
>> Anthony Liguori
>>
>> >
>> > ________________________________________
>> > From: Jessie Frazelle (TUPPERWARE) <jessfraz at microsoft.com>
>> > Sent: Wednesday, June 6, 2018 6:37 PM
>> > To: Anthony Liguori; Ernst, Eric; Sasha Levin; Manohar Castelino
>> > Cc: kata-hypervisor at lists.katacontainers.io
>> > Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
>> >
>> > Rust?
>> >
>> > What would you do?
>> >
>> > ________________________________________
>> > From: Anthony Liguori <aliguori at amazon.com>
>> > Sent: Wednesday, June 6, 2018 6:36 PM
>> > To: Jessie Frazelle (TUPPERWARE); Ernst, Eric; Sasha Levin; Manohar
>> Castelino
>> > Cc: kata-hypervisor at lists.katacontainers.io
>> > Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
>> >
>> > "Jessie Frazelle (TUPPERWARE)" <jessfraz at microsoft.com> writes:
>> >
>> >> Anthony,
>> >>
>> >> Would you be willing to dedicate resources to a _hypothetical_ 
>> >> rewrite of that layer that we could replace qemu with in the form 
>> >> of a go library?
>> >>
>> >> 0:)
>> >
>> > Go is so 2015.  Rust is where it's at now :-)
>> >
>> > In all seriousness, I've always felt strongly that a device model 
>> > needs to avoid garbage collection.  While it's not strictly a real 
>> > time system, OSes have an expectation that PIO or MMIO operations 
>> > have an upper bound of latency.
>> >
>> > The other challenge with Go is unpredictable memory usage due to 
>> > automatic heap growth.  If you want to do planning around density, 
>> > being able to rationalize about the memory overhead from the VMM 
>> > layer is pretty important.
>> >
>> > Regards,
>> >
>> > Anthony Liguori
>> >
>> >>
>> >>
>> >>
>> >> ________________________________________
>> >> From: Anthony Liguori <aliguori at amazon.com>
>> >> Sent: Wednesday, June 6, 2018 5:00 PM
>> >> To: Ernst, Eric; Sasha Levin; Manohar Castelino
>> >> Cc: kata-hypervisor at lists.katacontainers.io
>> >> Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
>> >>
>> >> "Ernst, Eric" <eric.ernst at intel.com> writes:
>> >>
>> >>> Hey Sasha, Anthony –
>> >>>
>> >>> I want to set a bit of context here.  At the face to face 
>> >>> discussion in Vancouver, we had agreed to share the prototyping 
>> >>> we have put in place to better understand how far we could go to 
>> >>> remove legacy, and minimize the number of device models, while 
>> >>> still booting key workloads.  Manohar’s response is to address 
>> >>> the action that everyone in the meeting requested.
>> >>
>> >> Yup, appreciate the sharing and am disappointed I wasn't able to 
>> >> attend in person.  I'm sure Paolo gave you lots of great feedback 
>> >> but let me expand a bit on why I think this is the wrong approach.  
>> >> Ultimately, you are certainly free to continue down this path 
>> >> though but just thought you may find this useful.
>> >>
>> >> The pieces of QEMU that are easy to remove are largely already 
>> >> configurable at build time.  From a practical perspective, if you 
>> >> build your "prototype" and measure the resulting executable size 
>> >> compared to a minimalistic configuration, you will find very 
>> >> little actual reduction in code.
>> >>
>> >> The stuff that's hard to remove is deeply ingrained (like QMP and 
>> >> QOM) or parts that are just hairy (like vl.c).  Years of effort 
>> >> have gone into decoupling these bits.
>> >>
>> >> So you get very little actual benefit (in code size reduction) by 
>> >> forking but get to bare the burden of independent maintenance.
>> >>
>> >> Regards,
>> >>
>> >> Anthony Liguori
>> >>
>> >>
>> >>>
>> >>> Cheers,
>> >>> Eric
>> >>>
>> >>> On 6/6/18, 1:46 PM, "Sasha Levin" <Alexander.Levin at microsoft.com>
>> wrote:
>> >>>
>> >>>     On Wed, Jun 06, 2018 at 08:08:46PM +0000, Castelino, Manohar 
>> >>> R
>> wrote:
>> >>>     >Folks,
>> >>>     >
>> >>>     >Please find below the link to the initial prototype code 
>> >>> that
>> tries to reduce the number of emulated devices in QEMU that can still 
>> boot most cloud workloads.
>> >>>     [snip]
>> >>>
>> >>>     Out of curiousity, have you looked at kvmtool
>> >>>     (
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.
>> kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Fwill%2Fkvmtool.git%2F
>> &data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cb
>> f07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63663915615626289
>> 0&sdata=xEWVp9B0rIFl5Ez1cGTMz9ClFs5Ril%2FTJN2SgVta824%3D&reserved=0
>> )?
>> >>>
>> >>>     Forking qemu rather than building on top of it (or using 
>> >>> something
>> else)
>> >>>     isn't going to end well IMO.
>> >>>     _______________________________________________
>> >>>     Kata-hypervisor mailing list
>> >>>     Kata-hypervisor at lists.katacontainers.io
>> >>>
>> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists
>> .katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&d
>> ata=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf0
>> 7f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&
>> sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> Kata-hypervisor mailing list
>> >>> Kata-hypervisor at lists.katacontainers.io
>> >>>
>> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists
>> .katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&d
>> ata=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf0
>> 7f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&
>> sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
>> >>
>> >> _______________________________________________
>> >> Kata-hypervisor mailing list
>> >> Kata-hypervisor at lists.katacontainers.io
>> >>
>> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists
>> .katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&d
>> ata=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf0
>> 7f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&
>> sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
>> >
>> > _______________________________________________
>> > Kata-hypervisor mailing list
>> > Kata-hypervisor at lists.katacontainers.io
>> >
>> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists
>> .katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&d
>> ata=02%7C01%7Cjessfraz%40microsoft.com%7Cbd25c45767ee40588bd708d5cbfe
>> 1c8d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639214632902936&
>> sdata=5otQKy9Z928r%2Bto0bnOCEKrX%2BvxmJgxHyox06Aa2t00%3D&reserved=0
>>
>> _______________________________________________
>> Kata-hypervisor mailing list
>> Kata-hypervisor at lists.katacontainers.io
>> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists
>> .katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&d
>> ata=02%7C01%7CBen.Armstrong%40microsoft.com%7C07a3b40e9b184ffe07f108d
>> 5cc10c0fb%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63663929469983
>> 0057&sdata=3Ak87F%2F3o%2FZ5iyKcpExNxSMlsJ%2FPqX1LFnvuH0S%2FWnY%3D&res
>> erved=0
>>

_______________________________________________
Kata-hypervisor mailing list
Kata-hypervisor at lists.katacontainers.io
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7CBen.Armstrong%40microsoft.com%7C07a3b40e9b184ffe07f108d5cc10c0fb%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639294699840065&sdata=%2BIBQ4oOdn3cslTPKw%2Fa8XROAghO2jBMaJ8TLe0HL9lg%3D&reserved=0


More information about the Kata-hypervisor mailing list