<div dir="auto"><div dir="auto">(note: currently at an off-site: responses may be delayed or worse)</div><div dir="auto"><br></div><div dir="auto">The interest that I had (and at the risk of speaking for him and being corrected, I think Ben as well) was in a VM *shape* focussed on something suitable for containers. In particular something based on a modern set of hardware features and minimizing the need for emulation of anything (PITs, PICs, PCI controllers, etc.)<div dir="auto"><br></div><div dir="auto">I would agree that for what qemu does that it's hard to do better, but we're actively uninterested in many of the things qemu does being supported by our offering. They don't help workloads we care about and they present "challenging" surface areas from a security perspective.</div><div dir="auto"><br></div><div dir="auto">Note that Google does actually have such a VM, aimed at containers and open source, implemented in Rust for Chrome OS. We are not using it in Cloud currently (and to be honest I haven't dived into it beyond being highly amused at the horrifying kernel command line that explicitly MMIO maps app of their virtio devices). This was on a production Pixel book, so nothing fancy or proprietary going on there.</div><div dir="auto"><div dir="auto"><div dir="auto"><br></div><div dir="auto">Jon</div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Jun 6, 2018, 4:14 PM Anthony Liguori <<a href="mailto:aliguori@amazon.com" target="_blank" rel="noreferrer">aliguori@amazon.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">"Jessie Frazelle (TUPPERWARE)" <<a href="mailto:jessfraz@microsoft.com" rel="noreferrer noreferrer" target="_blank">jessfraz@microsoft.com</a>> writes:<br>
<br>
> Sorry outlook cut off the first line... windows...<br>
><br>
> so rust :D<br>
<br>
I was able to join the prep call at least and I raised this during that<br>
discussion.  I'm not sure what problem this group is trying to tackle.<br>
<br>
What was brought up on the call is a "cloud VMM" as an alternative to<br>
QEMU but I don't really agree with that premise.  For the problem space<br>
that QEMU tackles, it's hard to do better than it does and it tackles a<br>
huge space.<br>
<br>
Cloud is an extremely ambiguous term too.  In another part of this<br>
thread, qcow2 was mentioned as a target which I don't think at all is<br>
something that is interesting if you are an infrastructure provider.<br>
<br>
There's lot of stuff that I am interested in (particularly around super<br>
fast start up time and super low overhead to get high density) but<br>
that's only possible to achieve if you constrain the problem space<br>
tremendously.<br>
<br>
Regards,<br>
<br>
Anthony Liguori<br>
<br>
><br>
> ________________________________________<br>
> From: Jessie Frazelle (TUPPERWARE) <<a href="mailto:jessfraz@microsoft.com" rel="noreferrer noreferrer" target="_blank">jessfraz@microsoft.com</a>><br>
> Sent: Wednesday, June 6, 2018 6:37 PM<br>
> To: Anthony Liguori; Ernst, Eric; Sasha Levin; Manohar Castelino<br>
> Cc: <a href="mailto:kata-hypervisor@lists.katacontainers.io" rel="noreferrer noreferrer" target="_blank">kata-hypervisor@lists.katacontainers.io</a><br>
> Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype<br>
><br>
> Rust?<br>
><br>
> What would you do?<br>
><br>
> ________________________________________<br>
> From: Anthony Liguori <<a href="mailto:aliguori@amazon.com" rel="noreferrer noreferrer" target="_blank">aliguori@amazon.com</a>><br>
> Sent: Wednesday, June 6, 2018 6:36 PM<br>
> To: Jessie Frazelle (TUPPERWARE); Ernst, Eric; Sasha Levin; Manohar Castelino<br>
> Cc: <a href="mailto:kata-hypervisor@lists.katacontainers.io" rel="noreferrer noreferrer" target="_blank">kata-hypervisor@lists.katacontainers.io</a><br>
> Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype<br>
><br>
> "Jessie Frazelle (TUPPERWARE)" <<a href="mailto:jessfraz@microsoft.com" rel="noreferrer noreferrer" target="_blank">jessfraz@microsoft.com</a>> writes:<br>
><br>
>> Anthony,<br>
>><br>
>> Would you be willing to dedicate resources to a _hypothetical_ rewrite<br>
>> of that layer that we could replace qemu with in the form of a go<br>
>> library?<br>
>><br>
>> 0:)<br>
><br>
> Go is so 2015.  Rust is where it's at now :-)<br>
><br>
> In all seriousness, I've always felt strongly that a device model needs<br>
> to avoid garbage collection.  While it's not strictly a real time<br>
> system, OSes have an expectation that PIO or MMIO operations have an<br>
> upper bound of latency.<br>
><br>
> The other challenge with Go is unpredictable memory usage due to<br>
> automatic heap growth.  If you want to do planning around density, being<br>
> able to rationalize about the memory overhead from the VMM layer is<br>
> pretty important.<br>
><br>
> Regards,<br>
><br>
> Anthony Liguori<br>
><br>
>><br>
>><br>
>><br>
>> ________________________________________<br>
>> From: Anthony Liguori <<a href="mailto:aliguori@amazon.com" rel="noreferrer noreferrer" target="_blank">aliguori@amazon.com</a>><br>
>> Sent: Wednesday, June 6, 2018 5:00 PM<br>
>> To: Ernst, Eric; Sasha Levin; Manohar Castelino<br>
>> Cc: <a href="mailto:kata-hypervisor@lists.katacontainers.io" rel="noreferrer noreferrer" target="_blank">kata-hypervisor@lists.katacontainers.io</a><br>
>> Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype<br>
>><br>
>> "Ernst, Eric" <<a href="mailto:eric.ernst@intel.com" rel="noreferrer noreferrer" target="_blank">eric.ernst@intel.com</a>> writes:<br>
>><br>
>>> Hey Sasha, Anthony –<br>
>>><br>
>>> I want to set a bit of context here.  At the face to face discussion<br>
>>> in Vancouver, we had agreed to share the prototyping we have put in<br>
>>> place to better understand how far we could go to remove legacy, and<br>
>>> minimize the number of device models, while still booting key<br>
>>> workloads.  Manohar’s response is to address the action that everyone<br>
>>> in the meeting requested.<br>
>><br>
>> Yup, appreciate the sharing and am disappointed I wasn't able to attend<br>
>> in person.  I'm sure Paolo gave you lots of great feedback but let me<br>
>> expand a bit on why I think this is the wrong approach.  Ultimately, you<br>
>> are certainly free to continue down this path though but just thought<br>
>> you may find this useful.<br>
>><br>
>> The pieces of QEMU that are easy to remove are largely already<br>
>> configurable at build time.  From a practical perspective, if you build<br>
>> your "prototype" and measure the resulting executable size compared to a<br>
>> minimalistic configuration, you will find very little actual reduction<br>
>> in code.<br>
>><br>
>> The stuff that's hard to remove is deeply ingrained (like QMP and QOM)<br>
>> or parts that are just hairy (like vl.c).  Years of effort have gone<br>
>> into decoupling these bits.<br>
>><br>
>> So you get very little actual benefit (in code size reduction) by<br>
>> forking but get to bare the burden of independent maintenance.<br>
>><br>
>> Regards,<br>
>><br>
>> Anthony Liguori<br>
>><br>
>><br>
>>><br>
>>> Cheers,<br>
>>> Eric<br>
>>><br>
>>> On 6/6/18, 1:46 PM, "Sasha Levin" <<a href="mailto:Alexander.Levin@microsoft.com" rel="noreferrer noreferrer" target="_blank">Alexander.Levin@microsoft.com</a>> wrote:<br>
>>><br>
>>>     On Wed, Jun 06, 2018 at 08:08:46PM +0000, Castelino, Manohar R wrote:<br>
>>>     >Folks,<br>
>>>     ><br>
>>>     >Please find below the link to the initial prototype code that tries to reduce the number of emulated devices in QEMU that can still boot most cloud workloads.<br>
>>>     [snip]<br>
>>><br>
>>>     Out of curiousity, have you looked at kvmtool<br>
>>>     (<a href="https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Fwill%2Fkvmtool.git%2F&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=xEWVp9B0rIFl5Ez1cGTMz9ClFs5Ril%2FTJN2SgVta824%3D&reserved=0" rel="noreferrer noreferrer noreferrer" target="_blank">https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Fwill%2Fkvmtool.git%2F&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=xEWVp9B0rIFl5Ez1cGTMz9ClFs5Ril%2FTJN2SgVta824%3D&reserved=0</a>)?<br>
>>><br>
>>>     Forking qemu rather than building on top of it (or using something else)<br>
>>>     isn't going to end well IMO.<br>
>>>     _______________________________________________<br>
>>>     Kata-hypervisor mailing list<br>
>>>     <a href="mailto:Kata-hypervisor@lists.katacontainers.io" rel="noreferrer noreferrer" target="_blank">Kata-hypervisor@lists.katacontainers.io</a><br>
>>>     <a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0" rel="noreferrer noreferrer noreferrer" target="_blank">https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0</a><br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> Kata-hypervisor mailing list<br>
>>> <a href="mailto:Kata-hypervisor@lists.katacontainers.io" rel="noreferrer noreferrer" target="_blank">Kata-hypervisor@lists.katacontainers.io</a><br>
>>> <a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0" rel="noreferrer noreferrer noreferrer" target="_blank">https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0</a><br>
>><br>
>> _______________________________________________<br>
>> Kata-hypervisor mailing list<br>
>> <a href="mailto:Kata-hypervisor@lists.katacontainers.io" rel="noreferrer noreferrer" target="_blank">Kata-hypervisor@lists.katacontainers.io</a><br>
>> <a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0" rel="noreferrer noreferrer noreferrer" target="_blank">https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0</a><br>
><br>
> _______________________________________________<br>
> Kata-hypervisor mailing list<br>
> <a href="mailto:Kata-hypervisor@lists.katacontainers.io" rel="noreferrer noreferrer" target="_blank">Kata-hypervisor@lists.katacontainers.io</a><br>
> <a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7Cbd25c45767ee40588bd708d5cbfe1c8d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639214632902936&sdata=5otQKy9Z928r%2Bto0bnOCEKrX%2BvxmJgxHyox06Aa2t00%3D&reserved=0" rel="noreferrer noreferrer noreferrer" target="_blank">https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7Cbd25c45767ee40588bd708d5cbfe1c8d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639214632902936&sdata=5otQKy9Z928r%2Bto0bnOCEKrX%2BvxmJgxHyox06Aa2t00%3D&reserved=0</a><br>
<br>
_______________________________________________<br>
Kata-hypervisor mailing list<br>
<a href="mailto:Kata-hypervisor@lists.katacontainers.io" rel="noreferrer noreferrer" target="_blank">Kata-hypervisor@lists.katacontainers.io</a><br>
<a href="http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-hypervisor" rel="noreferrer noreferrer noreferrer" target="_blank">http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-hypervisor</a><br>
</blockquote></div></div>