Link to the Hypervisor Prototype
Folks, Please find below the link to the initial prototype code that tries to reduce the number of emulated devices in QEMU that can still boot most cloud workloads. https://github.com/intel/nemu We created some tooling around automated code reduction and a minimal CI to ensure that this code reduction still retained features that Linux expects to find when booting cloud images. https://github.com/intel/nemu/tree/master/tools https://github.com/intel/nemu/tree/master/tools/CI -manohar
"Castelino, Manohar R" <manohar.r.castelino@intel.com> writes:
Folks,
Please find below the link to the initial prototype code that tries to reduce the number of emulated devices in QEMU that can still boot most cloud workloads. https://github.com/intel/nemu
We created some tooling around automated code reduction and a minimal CI to ensure that this code reduction still retained features that Linux expects to find when booting cloud images. https://github.com/intel/nemu/tree/master/tools https://github.com/intel/nemu/tree/master/tools/CI
Such a long history of failed QEMU forks. I'd really advise against going that route. Regards, Anthony Liguori
-manohar
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-hypervisor
On Wed, Jun 06, 2018 at 08:08:46PM +0000, Castelino, Manohar R wrote:
Folks,
Please find below the link to the initial prototype code that tries to reduce the number of emulated devices in QEMU that can still boot most cloud workloads. [snip]
Out of curiousity, have you looked at kvmtool (https://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git/)? Forking qemu rather than building on top of it (or using something else) isn't going to end well IMO.
Hey Sasha, Anthony – I want to set a bit of context here. At the face to face discussion in Vancouver, we had agreed to share the prototyping we have put in place to better understand how far we could go to remove legacy, and minimize the number of device models, while still booting key workloads. Manohar’s response is to address the action that everyone in the meeting requested. Cheers, Eric On 6/6/18, 1:46 PM, "Sasha Levin" <Alexander.Levin@microsoft.com> wrote: On Wed, Jun 06, 2018 at 08:08:46PM +0000, Castelino, Manohar R wrote: >Folks, > >Please find below the link to the initial prototype code that tries to reduce the number of emulated devices in QEMU that can still boot most cloud workloads. [snip] Out of curiousity, have you looked at kvmtool (https://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git/)? Forking qemu rather than building on top of it (or using something else) isn't going to end well IMO. _______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-hypervisor
"Ernst, Eric" <eric.ernst@intel.com> writes:
Hey Sasha, Anthony –
I want to set a bit of context here. At the face to face discussion in Vancouver, we had agreed to share the prototyping we have put in place to better understand how far we could go to remove legacy, and minimize the number of device models, while still booting key workloads. Manohar’s response is to address the action that everyone in the meeting requested.
Yup, appreciate the sharing and am disappointed I wasn't able to attend in person. I'm sure Paolo gave you lots of great feedback but let me expand a bit on why I think this is the wrong approach. Ultimately, you are certainly free to continue down this path though but just thought you may find this useful. The pieces of QEMU that are easy to remove are largely already configurable at build time. From a practical perspective, if you build your "prototype" and measure the resulting executable size compared to a minimalistic configuration, you will find very little actual reduction in code. The stuff that's hard to remove is deeply ingrained (like QMP and QOM) or parts that are just hairy (like vl.c). Years of effort have gone into decoupling these bits. So you get very little actual benefit (in code size reduction) by forking but get to bare the burden of independent maintenance. Regards, Anthony Liguori
Cheers, Eric
On 6/6/18, 1:46 PM, "Sasha Levin" <Alexander.Levin@microsoft.com> wrote:
On Wed, Jun 06, 2018 at 08:08:46PM +0000, Castelino, Manohar R wrote: >Folks, > >Please find below the link to the initial prototype code that tries to reduce the number of emulated devices in QEMU that can still boot most cloud workloads. [snip]
Out of curiousity, have you looked at kvmtool (https://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git/)?
Forking qemu rather than building on top of it (or using something else) isn't going to end well IMO. _______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-hypervisor
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-hypervisor
Anthony, Would you be willing to dedicate resources to a _hypothetical_ rewrite of that layer that we could replace qemu with in the form of a go library? 0:) ________________________________________ From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 5:00 PM To: Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype "Ernst, Eric" <eric.ernst@intel.com> writes:
Hey Sasha, Anthony –
I want to set a bit of context here. At the face to face discussion in Vancouver, we had agreed to share the prototyping we have put in place to better understand how far we could go to remove legacy, and minimize the number of device models, while still booting key workloads. Manohar’s response is to address the action that everyone in the meeting requested.
Yup, appreciate the sharing and am disappointed I wasn't able to attend in person. I'm sure Paolo gave you lots of great feedback but let me expand a bit on why I think this is the wrong approach. Ultimately, you are certainly free to continue down this path though but just thought you may find this useful. The pieces of QEMU that are easy to remove are largely already configurable at build time. From a practical perspective, if you build your "prototype" and measure the resulting executable size compared to a minimalistic configuration, you will find very little actual reduction in code. The stuff that's hard to remove is deeply ingrained (like QMP and QOM) or parts that are just hairy (like vl.c). Years of effort have gone into decoupling these bits. So you get very little actual benefit (in code size reduction) by forking but get to bare the burden of independent maintenance. Regards, Anthony Liguori
Cheers, Eric
On 6/6/18, 1:46 PM, "Sasha Levin" <Alexander.Levin@microsoft.com> wrote:
On Wed, Jun 06, 2018 at 08:08:46PM +0000, Castelino, Manohar R wrote: >Folks, > >Please find below the link to the initial prototype code that tries to reduce the number of emulated devices in QEMU that can still boot most cloud workloads. [snip]
Out of curiousity, have you looked at kvmtool (https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Fwill%2Fkvmtool.git%2F&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=xEWVp9B0rIFl5Ez1cGTMz9ClFs5Ril%2FTJN2SgVta824%3D&reserved=0)?
Forking qemu rather than building on top of it (or using something else) isn't going to end well IMO. _______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
"Jessie Frazelle (TUPPERWARE)" <jessfraz@microsoft.com> writes:
Anthony,
Would you be willing to dedicate resources to a _hypothetical_ rewrite of that layer that we could replace qemu with in the form of a go library?
0:)
Go is so 2015. Rust is where it's at now :-) In all seriousness, I've always felt strongly that a device model needs to avoid garbage collection. While it's not strictly a real time system, OSes have an expectation that PIO or MMIO operations have an upper bound of latency. The other challenge with Go is unpredictable memory usage due to automatic heap growth. If you want to do planning around density, being able to rationalize about the memory overhead from the VMM layer is pretty important. Regards, Anthony Liguori
________________________________________ From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 5:00 PM To: Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
"Ernst, Eric" <eric.ernst@intel.com> writes:
Hey Sasha, Anthony –
I want to set a bit of context here. At the face to face discussion in Vancouver, we had agreed to share the prototyping we have put in place to better understand how far we could go to remove legacy, and minimize the number of device models, while still booting key workloads. Manohar’s response is to address the action that everyone in the meeting requested.
Yup, appreciate the sharing and am disappointed I wasn't able to attend in person. I'm sure Paolo gave you lots of great feedback but let me expand a bit on why I think this is the wrong approach. Ultimately, you are certainly free to continue down this path though but just thought you may find this useful.
The pieces of QEMU that are easy to remove are largely already configurable at build time. From a practical perspective, if you build your "prototype" and measure the resulting executable size compared to a minimalistic configuration, you will find very little actual reduction in code.
The stuff that's hard to remove is deeply ingrained (like QMP and QOM) or parts that are just hairy (like vl.c). Years of effort have gone into decoupling these bits.
So you get very little actual benefit (in code size reduction) by forking but get to bare the burden of independent maintenance.
Regards,
Anthony Liguori
Cheers, Eric
On 6/6/18, 1:46 PM, "Sasha Levin" <Alexander.Levin@microsoft.com> wrote:
On Wed, Jun 06, 2018 at 08:08:46PM +0000, Castelino, Manohar R wrote: >Folks, > >Please find below the link to the initial prototype code that tries to reduce the number of emulated devices in QEMU that can still boot most cloud workloads. [snip]
Out of curiousity, have you looked at kvmtool (https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Fwill%2Fkvmtool.git%2F&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=xEWVp9B0rIFl5Ez1cGTMz9ClFs5Ril%2FTJN2SgVta824%3D&reserved=0)?
Forking qemu rather than building on top of it (or using something else) isn't going to end well IMO. _______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
Rust? What would you do? ________________________________________ From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 6:36 PM To: Jessie Frazelle (TUPPERWARE); Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype "Jessie Frazelle (TUPPERWARE)" <jessfraz@microsoft.com> writes:
Anthony,
Would you be willing to dedicate resources to a _hypothetical_ rewrite of that layer that we could replace qemu with in the form of a go library?
0:)
Go is so 2015. Rust is where it's at now :-) In all seriousness, I've always felt strongly that a device model needs to avoid garbage collection. While it's not strictly a real time system, OSes have an expectation that PIO or MMIO operations have an upper bound of latency. The other challenge with Go is unpredictable memory usage due to automatic heap growth. If you want to do planning around density, being able to rationalize about the memory overhead from the VMM layer is pretty important. Regards, Anthony Liguori
________________________________________ From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 5:00 PM To: Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
"Ernst, Eric" <eric.ernst@intel.com> writes:
Hey Sasha, Anthony –
I want to set a bit of context here. At the face to face discussion in Vancouver, we had agreed to share the prototyping we have put in place to better understand how far we could go to remove legacy, and minimize the number of device models, while still booting key workloads. Manohar’s response is to address the action that everyone in the meeting requested.
Yup, appreciate the sharing and am disappointed I wasn't able to attend in person. I'm sure Paolo gave you lots of great feedback but let me expand a bit on why I think this is the wrong approach. Ultimately, you are certainly free to continue down this path though but just thought you may find this useful.
The pieces of QEMU that are easy to remove are largely already configurable at build time. From a practical perspective, if you build your "prototype" and measure the resulting executable size compared to a minimalistic configuration, you will find very little actual reduction in code.
The stuff that's hard to remove is deeply ingrained (like QMP and QOM) or parts that are just hairy (like vl.c). Years of effort have gone into decoupling these bits.
So you get very little actual benefit (in code size reduction) by forking but get to bare the burden of independent maintenance.
Regards,
Anthony Liguori
Cheers, Eric
On 6/6/18, 1:46 PM, "Sasha Levin" <Alexander.Levin@microsoft.com> wrote:
On Wed, Jun 06, 2018 at 08:08:46PM +0000, Castelino, Manohar R wrote: >Folks, > >Please find below the link to the initial prototype code that tries to reduce the number of emulated devices in QEMU that can still boot most cloud workloads. [snip]
Out of curiousity, have you looked at kvmtool (https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Fwill%2Fkvmtool.git%2F&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=xEWVp9B0rIFl5Ez1cGTMz9ClFs5Ril%2FTJN2SgVta824%3D&reserved=0)?
Forking qemu rather than building on top of it (or using something else) isn't going to end well IMO. _______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
Sorry outlook cut off the first line... windows... so rust :D ________________________________________ From: Jessie Frazelle (TUPPERWARE) <jessfraz@microsoft.com> Sent: Wednesday, June 6, 2018 6:37 PM To: Anthony Liguori; Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype Rust? What would you do? ________________________________________ From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 6:36 PM To: Jessie Frazelle (TUPPERWARE); Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype "Jessie Frazelle (TUPPERWARE)" <jessfraz@microsoft.com> writes:
Anthony,
Would you be willing to dedicate resources to a _hypothetical_ rewrite of that layer that we could replace qemu with in the form of a go library?
0:)
Go is so 2015. Rust is where it's at now :-) In all seriousness, I've always felt strongly that a device model needs to avoid garbage collection. While it's not strictly a real time system, OSes have an expectation that PIO or MMIO operations have an upper bound of latency. The other challenge with Go is unpredictable memory usage due to automatic heap growth. If you want to do planning around density, being able to rationalize about the memory overhead from the VMM layer is pretty important. Regards, Anthony Liguori
________________________________________ From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 5:00 PM To: Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
"Ernst, Eric" <eric.ernst@intel.com> writes:
Hey Sasha, Anthony –
I want to set a bit of context here. At the face to face discussion in Vancouver, we had agreed to share the prototyping we have put in place to better understand how far we could go to remove legacy, and minimize the number of device models, while still booting key workloads. Manohar’s response is to address the action that everyone in the meeting requested.
Yup, appreciate the sharing and am disappointed I wasn't able to attend in person. I'm sure Paolo gave you lots of great feedback but let me expand a bit on why I think this is the wrong approach. Ultimately, you are certainly free to continue down this path though but just thought you may find this useful.
The pieces of QEMU that are easy to remove are largely already configurable at build time. From a practical perspective, if you build your "prototype" and measure the resulting executable size compared to a minimalistic configuration, you will find very little actual reduction in code.
The stuff that's hard to remove is deeply ingrained (like QMP and QOM) or parts that are just hairy (like vl.c). Years of effort have gone into decoupling these bits.
So you get very little actual benefit (in code size reduction) by forking but get to bare the burden of independent maintenance.
Regards,
Anthony Liguori
Cheers, Eric
On 6/6/18, 1:46 PM, "Sasha Levin" <Alexander.Levin@microsoft.com> wrote:
On Wed, Jun 06, 2018 at 08:08:46PM +0000, Castelino, Manohar R wrote: >Folks, > >Please find below the link to the initial prototype code that tries to reduce the number of emulated devices in QEMU that can still boot most cloud workloads. [snip]
Out of curiousity, have you looked at kvmtool (https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Fwill%2Fkvmtool.git%2F&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=xEWVp9B0rIFl5Ez1cGTMz9ClFs5Ril%2FTJN2SgVta824%3D&reserved=0)?
Forking qemu rather than building on top of it (or using something else) isn't going to end well IMO. _______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7Cbd25c45767ee40588bd708d5cbfe1c8d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639214632902936&sdata=5otQKy9Z928r%2Bto0bnOCEKrX%2BvxmJgxHyox06Aa2t00%3D&reserved=0
"Jessie Frazelle (TUPPERWARE)" <jessfraz@microsoft.com> writes:
Sorry outlook cut off the first line... windows...
so rust :D
I was able to join the prep call at least and I raised this during that discussion. I'm not sure what problem this group is trying to tackle. What was brought up on the call is a "cloud VMM" as an alternative to QEMU but I don't really agree with that premise. For the problem space that QEMU tackles, it's hard to do better than it does and it tackles a huge space. Cloud is an extremely ambiguous term too. In another part of this thread, qcow2 was mentioned as a target which I don't think at all is something that is interesting if you are an infrastructure provider. There's lot of stuff that I am interested in (particularly around super fast start up time and super low overhead to get high density) but that's only possible to achieve if you constrain the problem space tremendously. Regards, Anthony Liguori
________________________________________ From: Jessie Frazelle (TUPPERWARE) <jessfraz@microsoft.com> Sent: Wednesday, June 6, 2018 6:37 PM To: Anthony Liguori; Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
Rust?
What would you do?
________________________________________ From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 6:36 PM To: Jessie Frazelle (TUPPERWARE); Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
"Jessie Frazelle (TUPPERWARE)" <jessfraz@microsoft.com> writes:
Anthony,
Would you be willing to dedicate resources to a _hypothetical_ rewrite of that layer that we could replace qemu with in the form of a go library?
0:)
Go is so 2015. Rust is where it's at now :-)
In all seriousness, I've always felt strongly that a device model needs to avoid garbage collection. While it's not strictly a real time system, OSes have an expectation that PIO or MMIO operations have an upper bound of latency.
The other challenge with Go is unpredictable memory usage due to automatic heap growth. If you want to do planning around density, being able to rationalize about the memory overhead from the VMM layer is pretty important.
Regards,
Anthony Liguori
________________________________________ From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 5:00 PM To: Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
"Ernst, Eric" <eric.ernst@intel.com> writes:
Hey Sasha, Anthony –
I want to set a bit of context here. At the face to face discussion in Vancouver, we had agreed to share the prototyping we have put in place to better understand how far we could go to remove legacy, and minimize the number of device models, while still booting key workloads. Manohar’s response is to address the action that everyone in the meeting requested.
Yup, appreciate the sharing and am disappointed I wasn't able to attend in person. I'm sure Paolo gave you lots of great feedback but let me expand a bit on why I think this is the wrong approach. Ultimately, you are certainly free to continue down this path though but just thought you may find this useful.
The pieces of QEMU that are easy to remove are largely already configurable at build time. From a practical perspective, if you build your "prototype" and measure the resulting executable size compared to a minimalistic configuration, you will find very little actual reduction in code.
The stuff that's hard to remove is deeply ingrained (like QMP and QOM) or parts that are just hairy (like vl.c). Years of effort have gone into decoupling these bits.
So you get very little actual benefit (in code size reduction) by forking but get to bare the burden of independent maintenance.
Regards,
Anthony Liguori
Cheers, Eric
On 6/6/18, 1:46 PM, "Sasha Levin" <Alexander.Levin@microsoft.com> wrote:
On Wed, Jun 06, 2018 at 08:08:46PM +0000, Castelino, Manohar R wrote: >Folks, > >Please find below the link to the initial prototype code that tries to reduce the number of emulated devices in QEMU that can still boot most cloud workloads. [snip]
Out of curiousity, have you looked at kvmtool (https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Fwill%2Fkvmtool.git%2F&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=xEWVp9B0rIFl5Ez1cGTMz9ClFs5Ril%2FTJN2SgVta824%3D&reserved=0)?
Forking qemu rather than building on top of it (or using something else) isn't going to end well IMO. _______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7Cbd25c45767ee40588bd708d5cbfe1c8d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639214632902936&sdata=5otQKy9Z928r%2Bto0bnOCEKrX%2BvxmJgxHyox06Aa2t00%3D&reserved=0
Anthony Liguori <aliguori@amazon.com> writes:
"Jessie Frazelle (TUPPERWARE)" <jessfraz@microsoft.com> writes:
Sorry outlook cut off the first line... windows...
so rust :D
I think I missed the actual question and just ranted. I tend to do that... I would use Rust. I love Rust :-) Regards, Anthony Liguori
I was able to join the prep call at least and I raised this during that discussion. I'm not sure what problem this group is trying to tackle.
What was brought up on the call is a "cloud VMM" as an alternative to QEMU but I don't really agree with that premise. For the problem space that QEMU tackles, it's hard to do better than it does and it tackles a huge space.
Cloud is an extremely ambiguous term too. In another part of this thread, qcow2 was mentioned as a target which I don't think at all is something that is interesting if you are an infrastructure provider.
There's lot of stuff that I am interested in (particularly around super fast start up time and super low overhead to get high density) but that's only possible to achieve if you constrain the problem space tremendously.
Regards,
Anthony Liguori
________________________________________ From: Jessie Frazelle (TUPPERWARE) <jessfraz@microsoft.com> Sent: Wednesday, June 6, 2018 6:37 PM To: Anthony Liguori; Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
Rust?
What would you do?
________________________________________ From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 6:36 PM To: Jessie Frazelle (TUPPERWARE); Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
"Jessie Frazelle (TUPPERWARE)" <jessfraz@microsoft.com> writes:
Anthony,
Would you be willing to dedicate resources to a _hypothetical_ rewrite of that layer that we could replace qemu with in the form of a go library?
0:)
Go is so 2015. Rust is where it's at now :-)
In all seriousness, I've always felt strongly that a device model needs to avoid garbage collection. While it's not strictly a real time system, OSes have an expectation that PIO or MMIO operations have an upper bound of latency.
The other challenge with Go is unpredictable memory usage due to automatic heap growth. If you want to do planning around density, being able to rationalize about the memory overhead from the VMM layer is pretty important.
Regards,
Anthony Liguori
________________________________________ From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 5:00 PM To: Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
"Ernst, Eric" <eric.ernst@intel.com> writes:
Hey Sasha, Anthony –
I want to set a bit of context here. At the face to face discussion in Vancouver, we had agreed to share the prototyping we have put in place to better understand how far we could go to remove legacy, and minimize the number of device models, while still booting key workloads. Manohar’s response is to address the action that everyone in the meeting requested.
Yup, appreciate the sharing and am disappointed I wasn't able to attend in person. I'm sure Paolo gave you lots of great feedback but let me expand a bit on why I think this is the wrong approach. Ultimately, you are certainly free to continue down this path though but just thought you may find this useful.
The pieces of QEMU that are easy to remove are largely already configurable at build time. From a practical perspective, if you build your "prototype" and measure the resulting executable size compared to a minimalistic configuration, you will find very little actual reduction in code.
The stuff that's hard to remove is deeply ingrained (like QMP and QOM) or parts that are just hairy (like vl.c). Years of effort have gone into decoupling these bits.
So you get very little actual benefit (in code size reduction) by forking but get to bare the burden of independent maintenance.
Regards,
Anthony Liguori
Cheers, Eric
On 6/6/18, 1:46 PM, "Sasha Levin" <Alexander.Levin@microsoft.com> wrote:
On Wed, Jun 06, 2018 at 08:08:46PM +0000, Castelino, Manohar R wrote: >Folks, > >Please find below the link to the initial prototype code that tries to reduce the number of emulated devices in QEMU that can still boot most cloud workloads. [snip]
Out of curiousity, have you looked at kvmtool (https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Fwill%2Fkvmtool.git%2F&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=xEWVp9B0rIFl5Ez1cGTMz9ClFs5Ril%2FTJN2SgVta824%3D&reserved=0)?
Forking qemu rather than building on top of it (or using something else) isn't going to end well IMO. _______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890&sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7Cbd25c45767ee40588bd708d5cbfe1c8d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639214632902936&sdata=5otQKy9Z928r%2Bto0bnOCEKrX%2BvxmJgxHyox06Aa2t00%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-hypervisor
(note: currently at an off-site: responses may be delayed or worse) The interest that I had (and at the risk of speaking for him and being corrected, I think Ben as well) was in a VM *shape* focussed on something suitable for containers. In particular something based on a modern set of hardware features and minimizing the need for emulation of anything (PITs, PICs, PCI controllers, etc.) I would agree that for what qemu does that it's hard to do better, but we're actively uninterested in many of the things qemu does being supported by our offering. They don't help workloads we care about and they present "challenging" surface areas from a security perspective. Note that Google does actually have such a VM, aimed at containers and open source, implemented in Rust for Chrome OS. We are not using it in Cloud currently (and to be honest I haven't dived into it beyond being highly amused at the horrifying kernel command line that explicitly MMIO maps app of their virtio devices). This was on a production Pixel book, so nothing fancy or proprietary going on there. Jon On Wed, Jun 6, 2018, 4:14 PM Anthony Liguori <aliguori@amazon.com> wrote:
"Jessie Frazelle (TUPPERWARE)" <jessfraz@microsoft.com> writes:
Sorry outlook cut off the first line... windows...
so rust :D
I was able to join the prep call at least and I raised this during that discussion. I'm not sure what problem this group is trying to tackle.
What was brought up on the call is a "cloud VMM" as an alternative to QEMU but I don't really agree with that premise. For the problem space that QEMU tackles, it's hard to do better than it does and it tackles a huge space.
Cloud is an extremely ambiguous term too. In another part of this thread, qcow2 was mentioned as a target which I don't think at all is something that is interesting if you are an infrastructure provider.
There's lot of stuff that I am interested in (particularly around super fast start up time and super low overhead to get high density) but that's only possible to achieve if you constrain the problem space tremendously.
Regards,
Anthony Liguori
________________________________________ From: Jessie Frazelle (TUPPERWARE) <jessfraz@microsoft.com> Sent: Wednesday, June 6, 2018 6:37 PM To: Anthony Liguori; Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
Rust?
What would you do?
________________________________________ From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 6:36 PM To: Jessie Frazelle (TUPPERWARE); Ernst, Eric; Sasha Levin; Manohar
Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
"Jessie Frazelle (TUPPERWARE)" <jessfraz@microsoft.com> writes:
Anthony,
Would you be willing to dedicate resources to a _hypothetical_ rewrite of that layer that we could replace qemu with in the form of a go library?
0:)
Go is so 2015. Rust is where it's at now :-)
In all seriousness, I've always felt strongly that a device model needs to avoid garbage collection. While it's not strictly a real time system, OSes have an expectation that PIO or MMIO operations have an upper bound of latency.
The other challenge with Go is unpredictable memory usage due to automatic heap growth. If you want to do planning around density, being able to rationalize about the memory overhead from the VMM layer is pretty important.
Regards,
Anthony Liguori
________________________________________ From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 5:00 PM To: Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
"Ernst, Eric" <eric.ernst@intel.com> writes:
Hey Sasha, Anthony –
I want to set a bit of context here. At the face to face discussion in Vancouver, we had agreed to share the prototyping we have put in place to better understand how far we could go to remove legacy, and minimize the number of device models, while still booting key workloads. Manohar’s response is to address the action that everyone in the meeting requested.
Yup, appreciate the sharing and am disappointed I wasn't able to attend in person. I'm sure Paolo gave you lots of great feedback but let me expand a bit on why I think this is the wrong approach. Ultimately, you are certainly free to continue down this path though but just thought you may find this useful.
The pieces of QEMU that are easy to remove are largely already configurable at build time. From a practical perspective, if you build your "prototype" and measure the resulting executable size compared to a minimalistic configuration, you will find very little actual reduction in code.
The stuff that's hard to remove is deeply ingrained (like QMP and QOM) or parts that are just hairy (like vl.c). Years of effort have gone into decoupling these bits.
So you get very little actual benefit (in code size reduction) by forking but get to bare the burden of independent maintenance.
Regards,
Anthony Liguori
Cheers, Eric
On 6/6/18, 1:46 PM, "Sasha Levin" <Alexander.Levin@microsoft.com>
wrote:
On Wed, Jun 06, 2018 at 08:08:46PM +0000, Castelino, Manohar R
wrote:
>Folks, > >Please find below the link to the initial prototype code that
Castelino tries to reduce the number of emulated devices in QEMU that can still boot most cloud workloads.
[snip]
Out of curiousity, have you looked at kvmtool (
Forking qemu rather than building on top of it (or using something
else)
isn't going to end well IMO. _______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-hypervisor
Jon Olson <jonolson@google.com> writes:
(note: currently at an off-site: responses may be delayed or worse)
The interest that I had (and at the risk of speaking for him and being corrected, I think Ben as well) was in a VM *shape* focussed on something suitable for containers. In particular something based on a modern set of hardware features and minimizing the need for emulation of anything (PITs, PICs, PCI controllers, etc.)
I get what you're after but I've grown to believe it's not the right thing. You really want to present an APIC because then you get APIC-V and Posted Interrupts. This is the path Xen went with PV and it's boxed them into a corner where not only are event channels far worse than APIC-based delivery (they are inherently unfair) but now you don't get the performance boost that comes from Posted Interrupts due to that feature. PCI is really not that bad at all to emulate and you have to do discovery in some way. Again, lots of bad examples here (Xenstore for instance) but I don't mean to pick on Xen here. The complexity of emulation is not so bad. Part of what explodes QEMU in terms of complexity is the backend implementations. Once you need to support VMDK and QCOW2, you've got a massive code base. I know why folks want v9fs too but man that's a complex protocol and mapping it to POSIX is very complex. The legacy device emulation is the least of your concerns IMHO.
I would agree that for what qemu does that it's hard to do better, but we're actively uninterested in many of the things qemu does being supported by our offering. They don't help workloads we care about and they present "challenging" surface areas from a security perspective.
Note that Google does actually have such a VM, aimed at containers and open source, implemented in Rust for Chrome OS. We are not using it in Cloud currently (and to be honest I haven't dived into it beyond being highly amused at the horrifying kernel command line that explicitly MMIO maps app of their virtio devices). This was on a production Pixel book, so nothing fancy or proprietary going on there.
I like crosvm and have looked at the code pretty deeply. It actually has a bit of legacy emulation but it's naive in approach to it. I'm not sure splitting the device model into a separate process is the right approach but I like the fact that it's at least doing something different. I think if you are going to do legacy emulation, you need to do it right and completely. You can get stuff booting with a UART that turns a THR write into a putchar() call but ultimately, you need to implement FIFO and flow control properly. Almost every device model in QEMU started out as a bare bones minimum function model but grew over time because of need into what it is today. Regards, Anthony Liguori
Jon
On Wed, Jun 6, 2018, 4:14 PM Anthony Liguori <aliguori@amazon.com> wrote:
"Jessie Frazelle (TUPPERWARE)" <jessfraz@microsoft.com> writes:
Sorry outlook cut off the first line... windows...
so rust :D
I was able to join the prep call at least and I raised this during that discussion. I'm not sure what problem this group is trying to tackle.
What was brought up on the call is a "cloud VMM" as an alternative to QEMU but I don't really agree with that premise. For the problem space that QEMU tackles, it's hard to do better than it does and it tackles a huge space.
Cloud is an extremely ambiguous term too. In another part of this thread, qcow2 was mentioned as a target which I don't think at all is something that is interesting if you are an infrastructure provider.
There's lot of stuff that I am interested in (particularly around super fast start up time and super low overhead to get high density) but that's only possible to achieve if you constrain the problem space tremendously.
Regards,
Anthony Liguori
________________________________________ From: Jessie Frazelle (TUPPERWARE) <jessfraz@microsoft.com> Sent: Wednesday, June 6, 2018 6:37 PM To: Anthony Liguori; Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
Rust?
What would you do?
________________________________________ From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 6:36 PM To: Jessie Frazelle (TUPPERWARE); Ernst, Eric; Sasha Levin; Manohar
Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
"Jessie Frazelle (TUPPERWARE)" <jessfraz@microsoft.com> writes:
Anthony,
Would you be willing to dedicate resources to a _hypothetical_ rewrite of that layer that we could replace qemu with in the form of a go library?
0:)
Go is so 2015. Rust is where it's at now :-)
In all seriousness, I've always felt strongly that a device model needs to avoid garbage collection. While it's not strictly a real time system, OSes have an expectation that PIO or MMIO operations have an upper bound of latency.
The other challenge with Go is unpredictable memory usage due to automatic heap growth. If you want to do planning around density, being able to rationalize about the memory overhead from the VMM layer is pretty important.
Regards,
Anthony Liguori
________________________________________ From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 5:00 PM To: Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
"Ernst, Eric" <eric.ernst@intel.com> writes:
Hey Sasha, Anthony –
I want to set a bit of context here. At the face to face discussion in Vancouver, we had agreed to share the prototyping we have put in place to better understand how far we could go to remove legacy, and minimize the number of device models, while still booting key workloads. Manohar’s response is to address the action that everyone in the meeting requested.
Yup, appreciate the sharing and am disappointed I wasn't able to attend in person. I'm sure Paolo gave you lots of great feedback but let me expand a bit on why I think this is the wrong approach. Ultimately, you are certainly free to continue down this path though but just thought you may find this useful.
The pieces of QEMU that are easy to remove are largely already configurable at build time. From a practical perspective, if you build your "prototype" and measure the resulting executable size compared to a minimalistic configuration, you will find very little actual reduction in code.
The stuff that's hard to remove is deeply ingrained (like QMP and QOM) or parts that are just hairy (like vl.c). Years of effort have gone into decoupling these bits.
So you get very little actual benefit (in code size reduction) by forking but get to bare the burden of independent maintenance.
Regards,
Anthony Liguori
Cheers, Eric
On 6/6/18, 1:46 PM, "Sasha Levin" <Alexander.Levin@microsoft.com>
wrote:
On Wed, Jun 06, 2018 at 08:08:46PM +0000, Castelino, Manohar R
wrote:
>Folks, > >Please find below the link to the initial prototype code that
Castelino tries to reduce the number of emulated devices in QEMU that can still boot most cloud workloads.
[snip]
Out of curiousity, have you looked at kvmtool (
Forking qemu rather than building on top of it (or using something
else)
isn't going to end well IMO. _______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-hypervisor
I'm not convinced by 9p Also I don't think a cloud native VMM or anything like that is the answer... But I'm a huge fan of being able to run things on bare metal or my desktop :P In my ideal scenario it would be the bare minimum needed and also not have anything that required backwards compat to old kernels.. because why.. Anyways if you all are interested and/or have time or resources... I'd love to collect ideas :) you all have a great deal of experience in this area and I think it's a cool opportunity to build something minimal, secure and fast. ________________________________________ From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 8:50:43 PM To: Jon Olson Cc: Jessie Frazelle (TUPPERWARE); Ernst, Eric; Sasha Levin; Manohar Castelino; kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype Jon Olson <jonolson@google.com> writes:
(note: currently at an off-site: responses may be delayed or worse)
The interest that I had (and at the risk of speaking for him and being corrected, I think Ben as well) was in a VM *shape* focussed on something suitable for containers. In particular something based on a modern set of hardware features and minimizing the need for emulation of anything (PITs, PICs, PCI controllers, etc.)
I get what you're after but I've grown to believe it's not the right thing. You really want to present an APIC because then you get APIC-V and Posted Interrupts. This is the path Xen went with PV and it's boxed them into a corner where not only are event channels far worse than APIC-based delivery (they are inherently unfair) but now you don't get the performance boost that comes from Posted Interrupts due to that feature. PCI is really not that bad at all to emulate and you have to do discovery in some way. Again, lots of bad examples here (Xenstore for instance) but I don't mean to pick on Xen here. The complexity of emulation is not so bad. Part of what explodes QEMU in terms of complexity is the backend implementations. Once you need to support VMDK and QCOW2, you've got a massive code base. I know why folks want v9fs too but man that's a complex protocol and mapping it to POSIX is very complex. The legacy device emulation is the least of your concerns IMHO.
I would agree that for what qemu does that it's hard to do better, but we're actively uninterested in many of the things qemu does being supported by our offering. They don't help workloads we care about and they present "challenging" surface areas from a security perspective.
Note that Google does actually have such a VM, aimed at containers and open source, implemented in Rust for Chrome OS. We are not using it in Cloud currently (and to be honest I haven't dived into it beyond being highly amused at the horrifying kernel command line that explicitly MMIO maps app of their virtio devices). This was on a production Pixel book, so nothing fancy or proprietary going on there.
I like crosvm and have looked at the code pretty deeply. It actually has a bit of legacy emulation but it's naive in approach to it. I'm not sure splitting the device model into a separate process is the right approach but I like the fact that it's at least doing something different. I think if you are going to do legacy emulation, you need to do it right and completely. You can get stuff booting with a UART that turns a THR write into a putchar() call but ultimately, you need to implement FIFO and flow control properly. Almost every device model in QEMU started out as a bare bones minimum function model but grew over time because of need into what it is today. Regards, Anthony Liguori
Jon
On Wed, Jun 6, 2018, 4:14 PM Anthony Liguori <aliguori@amazon.com> wrote:
"Jessie Frazelle (TUPPERWARE)" <jessfraz@microsoft.com> writes:
Sorry outlook cut off the first line... windows...
so rust :D
I was able to join the prep call at least and I raised this during that discussion. I'm not sure what problem this group is trying to tackle.
What was brought up on the call is a "cloud VMM" as an alternative to QEMU but I don't really agree with that premise. For the problem space that QEMU tackles, it's hard to do better than it does and it tackles a huge space.
Cloud is an extremely ambiguous term too. In another part of this thread, qcow2 was mentioned as a target which I don't think at all is something that is interesting if you are an infrastructure provider.
There's lot of stuff that I am interested in (particularly around super fast start up time and super low overhead to get high density) but that's only possible to achieve if you constrain the problem space tremendously.
Regards,
Anthony Liguori
________________________________________ From: Jessie Frazelle (TUPPERWARE) <jessfraz@microsoft.com> Sent: Wednesday, June 6, 2018 6:37 PM To: Anthony Liguori; Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
Rust?
What would you do?
________________________________________ From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 6:36 PM To: Jessie Frazelle (TUPPERWARE); Ernst, Eric; Sasha Levin; Manohar
Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
"Jessie Frazelle (TUPPERWARE)" <jessfraz@microsoft.com> writes:
Anthony,
Would you be willing to dedicate resources to a _hypothetical_ rewrite of that layer that we could replace qemu with in the form of a go library?
0:)
Go is so 2015. Rust is where it's at now :-)
In all seriousness, I've always felt strongly that a device model needs to avoid garbage collection. While it's not strictly a real time system, OSes have an expectation that PIO or MMIO operations have an upper bound of latency.
The other challenge with Go is unpredictable memory usage due to automatic heap growth. If you want to do planning around density, being able to rationalize about the memory overhead from the VMM layer is pretty important.
Regards,
Anthony Liguori
________________________________________ From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 5:00 PM To: Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
"Ernst, Eric" <eric.ernst@intel.com> writes:
Hey Sasha, Anthony –
I want to set a bit of context here. At the face to face discussion in Vancouver, we had agreed to share the prototyping we have put in place to better understand how far we could go to remove legacy, and minimize the number of device models, while still booting key workloads. Manohar’s response is to address the action that everyone in the meeting requested.
Yup, appreciate the sharing and am disappointed I wasn't able to attend in person. I'm sure Paolo gave you lots of great feedback but let me expand a bit on why I think this is the wrong approach. Ultimately, you are certainly free to continue down this path though but just thought you may find this useful.
The pieces of QEMU that are easy to remove are largely already configurable at build time. From a practical perspective, if you build your "prototype" and measure the resulting executable size compared to a minimalistic configuration, you will find very little actual reduction in code.
The stuff that's hard to remove is deeply ingrained (like QMP and QOM) or parts that are just hairy (like vl.c). Years of effort have gone into decoupling these bits.
So you get very little actual benefit (in code size reduction) by forking but get to bare the burden of independent maintenance.
Regards,
Anthony Liguori
Cheers, Eric
On 6/6/18, 1:46 PM, "Sasha Levin" <Alexander.Levin@microsoft.com>
wrote:
On Wed, Jun 06, 2018 at 08:08:46PM +0000, Castelino, Manohar R
wrote:
>Folks, > >Please find below the link to the initial prototype code that
Castelino tries to reduce the number of emulated devices in QEMU that can still boot most cloud workloads.
[snip]
Out of curiousity, have you looked at kvmtool (
Forking qemu rather than building on top of it (or using something
else)
isn't going to end well IMO. _______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7Cjessfraz%40microsoft.com%7C4f7a37b1c4314dc0b45208d5cc10b87f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639294556758752&sdata=0ATbtRKEWFux6e7kAYygc7pI0kJzv8PP2KuN81PMt0g%3D&reserved=0
On Wed, Jun 6, 2018 at 10:04 PM, Jessie Frazelle (TUPPERWARE) < jessfraz@microsoft.com> wrote:
I'm not convinced by 9p
Also I don't think a cloud native VMM or anything like that is the answer... But I'm a huge fan of being able to run things on bare metal or my desktop :P
In my ideal scenario it would be the bare minimum needed and also not have anything that required backwards compat to old kernels.. because why..
It depends how we define old :). In the context of the Vancouver discussion I had said, selfishly from a KubeVirt perspective (admittedly a quite different use case to Katacontainers), that we would be interested in a thinner, more modular, solution than we have today but that we still have a foreseeable need to run RHEL 6+ (so 2.6.32+GIANTDIFF as someone put it ;)) and W2K12+ on our current timelines. So I think the reason older kernels were in the discussion was my fault, but at the time it seemed like other folks were also interested in at least some variations along that spectrum and this was roughly in line with what Intel have been able to get working with their cut down qemu builds - notwithstanding that additional issues may likely appear when it comes to running any existing "real" workload VM with all its knobs on such a platform. Thanks, Steve
Ya I guess I was wondering what those of us who don't have to support rhel 6 or even 7 and can run close to mainline could do :) ________________________________________ From: Steve Gordon <sgordon@redhat.com> Sent: Thursday, June 7, 2018 9:05:36 AM To: Jessie Frazelle (TUPPERWARE) Cc: Anthony Liguori; Jon Olson; kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype On Wed, Jun 6, 2018 at 10:04 PM, Jessie Frazelle (TUPPERWARE) <jessfraz@microsoft.com<mailto:jessfraz@microsoft.com>> wrote: I'm not convinced by 9p Also I don't think a cloud native VMM or anything like that is the answer... But I'm a huge fan of being able to run things on bare metal or my desktop :P In my ideal scenario it would be the bare minimum needed and also not have anything that required backwards compat to old kernels.. because why.. It depends how we define old :). In the context of the Vancouver discussion I had said, selfishly from a KubeVirt perspective (admittedly a quite different use case to Katacontainers), that we would be interested in a thinner, more modular, solution than we have today but that we still have a foreseeable need to run RHEL 6+ (so 2.6.32+GIANTDIFF as someone put it ;)) and W2K12+ on our current timelines. So I think the reason older kernels were in the discussion was my fault, but at the time it seemed like other folks were also interested in at least some variations along that spectrum and this was roughly in line with what Intel have been able to get working with their cut down qemu builds - notwithstanding that additional issues may likely appear when it comes to running any existing "real" workload VM with all its knobs on such a platform. Thanks, Steve
"Jessie Frazelle (TUPPERWARE)" <jessfraz@microsoft.com> writes:
Ya I guess I was wondering what those of us who don't have to support rhel 6 or even 7 and can run close to mainline could do :)
Old kernels usually aren't the challenge. It's supporting distro kernels where there is an insistence on having a single kernel for all use-caess. As an example, you generally don't need a PC keyboard controller to boot Linux. It's been that way for a long time. However, the typical distro kernel is configured in a way in that the absence of a keyboard controller will result in a 1-second startup pause while probing fails. You can Kconfig it out or you can just emulate the controller to avoid having to it. Turns out that it's not so bad to emulate (less than 1,000 SLOCs) and is mostly just ifs and switches. Regards, Anthony Liguori
________________________________________ From: Steve Gordon <sgordon@redhat.com> Sent: Thursday, June 7, 2018 9:05:36 AM To: Jessie Frazelle (TUPPERWARE) Cc: Anthony Liguori; Jon Olson; kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
On Wed, Jun 6, 2018 at 10:04 PM, Jessie Frazelle (TUPPERWARE) <jessfraz@microsoft.com<mailto:jessfraz@microsoft.com>> wrote: I'm not convinced by 9p
Also I don't think a cloud native VMM or anything like that is the answer... But I'm a huge fan of being able to run things on bare metal or my desktop :P
In my ideal scenario it would be the bare minimum needed and also not have anything that required backwards compat to old kernels.. because why..
It depends how we define old :). In the context of the Vancouver discussion I had said, selfishly from a KubeVirt perspective (admittedly a quite different use case to Katacontainers), that we would be interested in a thinner, more modular, solution than we have today but that we still have a foreseeable need to run RHEL 6+ (so 2.6.32+GIANTDIFF as someone put it ;)) and W2K12+ on our current timelines.
So I think the reason older kernels were in the discussion was my fault, but at the time it seemed like other folks were also interested in at least some variations along that spectrum and this was roughly in line with what Intel have been able to get working with their cut down qemu builds - notwithstanding that additional issues may likely appear when it comes to running any existing "real" workload VM with all its knobs on such a platform.
Thanks,
Steve
In Hyper-V we have virtual machines with no PCI and a synthetic interrupt controller - which also has no problems taking advantage of posted interrupts. I do not know what Xen implemented here - however emulation of all these devices is not necessary. Cheers, Ben -----Original Message----- From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 5:51 PM To: Jon Olson <jonolson@google.com> Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype Jon Olson <jonolson@google.com> writes:
(note: currently at an off-site: responses may be delayed or worse)
The interest that I had (and at the risk of speaking for him and being corrected, I think Ben as well) was in a VM *shape* focussed on something suitable for containers. In particular something based on a modern set of hardware features and minimizing the need for emulation of anything (PITs, PICs, PCI controllers, etc.)
I get what you're after but I've grown to believe it's not the right thing. You really want to present an APIC because then you get APIC-V and Posted Interrupts. This is the path Xen went with PV and it's boxed them into a corner where not only are event channels far worse than APIC-based delivery (they are inherently unfair) but now you don't get the performance boost that comes from Posted Interrupts due to that feature. PCI is really not that bad at all to emulate and you have to do discovery in some way. Again, lots of bad examples here (Xenstore for instance) but I don't mean to pick on Xen here. The complexity of emulation is not so bad. Part of what explodes QEMU in terms of complexity is the backend implementations. Once you need to support VMDK and QCOW2, you've got a massive code base. I know why folks want v9fs too but man that's a complex protocol and mapping it to POSIX is very complex. The legacy device emulation is the least of your concerns IMHO.
I would agree that for what qemu does that it's hard to do better, but we're actively uninterested in many of the things qemu does being supported by our offering. They don't help workloads we care about and they present "challenging" surface areas from a security perspective.
Note that Google does actually have such a VM, aimed at containers and open source, implemented in Rust for Chrome OS. We are not using it in Cloud currently (and to be honest I haven't dived into it beyond being highly amused at the horrifying kernel command line that explicitly MMIO maps app of their virtio devices). This was on a production Pixel book, so nothing fancy or proprietary going on there.
I like crosvm and have looked at the code pretty deeply. It actually has a bit of legacy emulation but it's naive in approach to it. I'm not sure splitting the device model into a separate process is the right approach but I like the fact that it's at least doing something different. I think if you are going to do legacy emulation, you need to do it right and completely. You can get stuff booting with a UART that turns a THR write into a putchar() call but ultimately, you need to implement FIFO and flow control properly. Almost every device model in QEMU started out as a bare bones minimum function model but grew over time because of need into what it is today. Regards, Anthony Liguori
Jon
On Wed, Jun 6, 2018, 4:14 PM Anthony Liguori <aliguori@amazon.com> wrote:
"Jessie Frazelle (TUPPERWARE)" <jessfraz@microsoft.com> writes:
Sorry outlook cut off the first line... windows...
so rust :D
I was able to join the prep call at least and I raised this during that discussion. I'm not sure what problem this group is trying to tackle.
What was brought up on the call is a "cloud VMM" as an alternative to QEMU but I don't really agree with that premise. For the problem space that QEMU tackles, it's hard to do better than it does and it tackles a huge space.
Cloud is an extremely ambiguous term too. In another part of this thread, qcow2 was mentioned as a target which I don't think at all is something that is interesting if you are an infrastructure provider.
There's lot of stuff that I am interested in (particularly around super fast start up time and super low overhead to get high density) but that's only possible to achieve if you constrain the problem space tremendously.
Regards,
Anthony Liguori
________________________________________ From: Jessie Frazelle (TUPPERWARE) <jessfraz@microsoft.com> Sent: Wednesday, June 6, 2018 6:37 PM To: Anthony Liguori; Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
Rust?
What would you do?
________________________________________ From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 6:36 PM To: Jessie Frazelle (TUPPERWARE); Ernst, Eric; Sasha Levin; Manohar
Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
"Jessie Frazelle (TUPPERWARE)" <jessfraz@microsoft.com> writes:
Anthony,
Would you be willing to dedicate resources to a _hypothetical_ rewrite of that layer that we could replace qemu with in the form of a go library?
0:)
Go is so 2015. Rust is where it's at now :-)
In all seriousness, I've always felt strongly that a device model needs to avoid garbage collection. While it's not strictly a real time system, OSes have an expectation that PIO or MMIO operations have an upper bound of latency.
The other challenge with Go is unpredictable memory usage due to automatic heap growth. If you want to do planning around density, being able to rationalize about the memory overhead from the VMM layer is pretty important.
Regards,
Anthony Liguori
________________________________________ From: Anthony Liguori <aliguori@amazon.com> Sent: Wednesday, June 6, 2018 5:00 PM To: Ernst, Eric; Sasha Levin; Manohar Castelino Cc: kata-hypervisor@lists.katacontainers.io Subject: Re: [Kata-hypervisor] Link to the Hypervisor Prototype
"Ernst, Eric" <eric.ernst@intel.com> writes:
Hey Sasha, Anthony –
I want to set a bit of context here. At the face to face discussion in Vancouver, we had agreed to share the prototyping we have put in place to better understand how far we could go to remove legacy, and minimize the number of device models, while still booting key workloads. Manohar’s response is to address the action that everyone in the meeting requested.
Yup, appreciate the sharing and am disappointed I wasn't able to attend in person. I'm sure Paolo gave you lots of great feedback but let me expand a bit on why I think this is the wrong approach. Ultimately, you are certainly free to continue down this path though but just thought you may find this useful.
The pieces of QEMU that are easy to remove are largely already configurable at build time. From a practical perspective, if you build your "prototype" and measure the resulting executable size compared to a minimalistic configuration, you will find very little actual reduction in code.
The stuff that's hard to remove is deeply ingrained (like QMP and QOM) or parts that are just hairy (like vl.c). Years of effort have gone into decoupling these bits.
So you get very little actual benefit (in code size reduction) by forking but get to bare the burden of independent maintenance.
Regards,
Anthony Liguori
Cheers, Eric
On 6/6/18, 1:46 PM, "Sasha Levin" <Alexander.Levin@microsoft.com>
wrote:
On Wed, Jun 06, 2018 at 08:08:46PM +0000, Castelino, Manohar R
wrote:
>Folks, > >Please find below the link to the initial prototype code that
Castelino tries to reduce the number of emulated devices in QEMU that can still boot most cloud workloads.
[snip]
Out of curiousity, have you looked at kvmtool (
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit. kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Fwill%2Fkvmtool.git%2F &data=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cb f07f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63663915615626289 0&sdata=xEWVp9B0rIFl5Ez1cGTMz9ClFs5Ril%2FTJN2SgVta824%3D&reserved=0 )?
Forking qemu rather than building on top of it (or using something
else)
isn't going to end well IMO. _______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists .katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&d ata=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf0 7f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890& sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists .katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&d ata=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf0 7f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890& sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists .katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&d ata=02%7C01%7Cjessfraz%40microsoft.com%7C604dbe2ada034e62d2f908d5cbf0 7f6b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639156156262890& sdata=kJsXnzaf7xjwtTB6HPpG4nXVVV7YdXlGnp4Eu3Mn2gA%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists .katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&d ata=02%7C01%7Cjessfraz%40microsoft.com%7Cbd25c45767ee40588bd708d5cbfe 1c8d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639214632902936& sdata=5otQKy9Z928r%2Bto0bnOCEKrX%2BvxmJgxHyox06Aa2t00%3D&reserved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists .katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&d ata=02%7C01%7CBen.Armstrong%40microsoft.com%7C07a3b40e9b184ffe07f108d 5cc10c0fb%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63663929469983 0057&sdata=3Ak87F%2F3o%2FZ5iyKcpExNxSMlsJ%2FPqX1LFnvuH0S%2FWnY%3D&res erved=0
_______________________________________________ Kata-hypervisor mailing list Kata-hypervisor@lists.katacontainers.io https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-hypervisor&data=02%7C01%7CBen.Armstrong%40microsoft.com%7C07a3b40e9b184ffe07f108d5cc10c0fb%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636639294699840065&sdata=%2BIBQ4oOdn3cslTPKw%2Fa8XROAghO2jBMaJ8TLe0HL9lg%3D&reserved=0
On 07/06/2018 06:52, Ben Armstrong wrote:
In Hyper-V we have virtual machines with no PCI and a synthetic interrupt controller - which also has no problems taking advantage of posted interrupts. I do not know what Xen implemented here - however emulation of all these devices is not necessary.
The synthetic interrupt controller is basically "APIC plus fancy stuff for events and timers". It's more emulation, not less, and it's in kernel space. I'm not sure how it's an advantage, even though it's certainly better than Xen. It can perform better than doing the fancy stuff in the guest kernel, but it adds to the attack surface. Devices such as PIC and PIT have complicated logic because of the legacy from 1980s electronics, but they are changed very little and they hardly even have arrays in their code. We can remove them, but they don't pose an appreciable security risk. The big exception would be wrong locking, but QEMU doesn't have that issue because non-performance-critical devices (which most devices are) are placed under a "big QEMU lock". Perhaps it's exactly the simple stuff like the PIC that is a candidate for playing with Rust, showing where we want to go in terms of writing test cases, etc, since they have well-known and unchanging data sheets. Paolo
On 06/06/2018 23:00, Anthony Liguori wrote:
"Ernst, Eric" <eric.ernst@intel.com> writes:
Hey Sasha, Anthony –
I want to set a bit of context here. At the face to face discussion in Vancouver, we had agreed to share the prototyping we have put in place to better understand how far we could go to remove legacy, and minimize the number of device models, while still booting key workloads. Manohar’s response is to address the action that everyone in the meeting requested.
Yup, appreciate the sharing and am disappointed I wasn't able to attend in person. I'm sure Paolo gave you lots of great feedback but let me expand a bit on why I think this is the wrong approach. Ultimately, you are certainly free to continue down this path though but just thought you may find this useful.
Indeed similar points were "vociferously" raised by yours truly. ;) I think Manohar agreed that QEMU is already _somewhat_ modular and there is lots of room for improvement. If we approach the problem this way, it's okay to have Intel's code as a guideline for what could be done, similar to how qboot vs. SeaBIOS played out. Another important point that we agreed, was that total repository SLOC doesn't really count, what matters is a clear understanding of which files are being compiled. Also Google expressed a preference for avoiding #ifdefs, and QEMU isn't that bad in this respect. My suggestion was to handle the diff in a similar way as the realtime kernel patch, working upstream on both tooling and modularity so that the size can be reduced. Paolo
The pieces of QEMU that are easy to remove are largely already configurable at build time. From a practical perspective, if you build your "prototype" and measure the resulting executable size compared to a minimalistic configuration, you will find very little actual reduction in code.
The stuff that's hard to remove is deeply ingrained (like QMP and QOM) or parts that are just hairy (like vl.c). Years of effort have gone into decoupling these bits.
So you get very little actual benefit (in code size reduction) by forking but get to bare the burden of independent maintenance.
On Wed, Jun 06, 2018 at 10:08:46PM +0200, Castelino, Manohar R wrote:
Folks,
Please find below the link to the initial prototype code that tries to reduce the number of emulated devices in QEMU that can still boot most cloud workloads. https://github.com/intel/nemu
Some updates on the project from our side: The work we've done on reducing the 2.12 emulated devices number, code size, features set and supported platforms, and that currently lives in the NEMU master branch, has been very educative. We feel that the next natural step is about defining and implementing a new x86 machine type, based on a virtual platform rather than an actual chipset. This is going to be a simplified, emulation free and virtual hardware based machine type for running modern cloud and container workloads. We believe that this new machine type will allow us to reduce even further the number of devices that QEMU needs to support in order to run this kind of workloads. We will start by implementing this new machine type on top of an upstream QEMU code base, as we hope that this could become upstream acceptable material. Then we will re-run our automatic code reduction tooling on top of a QEMU code base that would only support this machine type and the ARM virt one. This will become the new NEMU master branch. We created a github issue with a few more details about this new machine type for x86: https://github.com/intel/nemu/issues/11 Cheers, Samuel. --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
On 13/06/2018 18:54, Samuel Ortiz wrote:
This is going to be a simplified, emulation free and virtual hardware based machine type for running modern cloud and container workloads. We believe that this new machine type will allow us to reduce even further the number of devices that QEMU needs to support in order to run this kind of workloads.
We will start by implementing this new machine type on top of an upstream QEMU code base, as we hope that this could become upstream acceptable material. Then we will re-run our automatic code reduction tooling on top of a QEMU code base that would only support this machine type and the ARM virt one. This will become the new NEMU master branch.
Some notes: - the ARM virt machine type is not a "virtual hardware platform". It has PL011 serial, PL031 RTC, PL061 GPIO controller, a generic PCIe bridge, fw_cfg for the firmware to get data from QEMU, and emulated flash. The idea behind the ARM virt machine type is simply to make everything discoverable via device tree; the "virtualness" of it is simply that the device tree is provided by QEMU and not embedded into something like U-Boot or Linux. - support for legacy BIOS is irrelevant to QEMU; if any developer wants to port SeaBIOS to this new machine, why not? SeaBIOS is already very modular (and qboot is not particularly faster than a properly configured SeaBIOS). - UEFI runtime services need to provide an RTC, so you need to emulate one. - without MCH, you'll have to find a different way to handle SMRAM. - no ISA means no serial port for early boot messages, no debug port for firmware logs, and more painful debugging in general. I'm not sure that's a good idea. Of course you don't want any IDE or floppy crap, but that's orthogonal to having ISA. - no LPC also means no hotplug (on q35; on PIIX hotplug was part of the SMBus controller, which you probably don't want anyway). So you'd have to reinvent something very similar to a PCH anyway in order to provide hotplug. IMNSHO, a more interesting starting point would be to upstream the patches to remove legacy devices from the binary (e.g. the infamous floppy). Paolo
On 13/06/2018 18:54, Samuel Ortiz wrote:
This is going to be a simplified, emulation free and virtual hardware based machine type for running modern cloud and container workloads. We believe that this new machine type will allow us to reduce even further the number of devices that QEMU needs to support in order to run this kind of workloads.
We will start by implementing this new machine type on top of an upstream QEMU code base, as we hope that this could become upstream acceptable material. Then we will re-run our automatic code reduction tooling on top of a QEMU code base that would only support this machine type and the ARM virt one. This will become the new NEMU master branch.
Some notes:
- the ARM virt machine type is not a "virtual hardware platform". It has PL011 serial, PL031 RTC, PL061 GPIO controller, a generic PCIe bridge, fw_cfg for the firmware to get data from QEMU, and emulated flash. The idea behind the ARM virt machine type is simply to make everything discoverable via device tree; the "virtualness" of it is simply that the device tree is provided by QEMU and not embedded into something like U-Boot or Linux. Yes, we're aware of that. It's also one of the very few ARM machine type
On Wed, Jun 13, 2018 at 07:25:10PM +0200, Paolo Bonzini wrote: that does not map to an existing SoC. On that topic, one question: Would you be accepting patches for optionally making the ARM virt machine discoverable through ACPI only? Booting an ARM64 kernel on top of that should work in theory at least.
- support for legacy BIOS is irrelevant to QEMU; if any developer wants to port SeaBIOS to this new machine, why not? Oh absolutely. It's just that this is not something we'll work on initially but obviously everyone is free to add BIOS support for this new machine.
- UEFI runtime services need to provide an RTC, so you need to emulate one. Ok. I was not awatre of that, thanks.
- without MCH, you'll have to find a different way to handle SMRAM.
- no ISA means no serial port for early boot messages, no debug port for firmware logs, and more painful debugging in general. I'm not sure that's a good idea. Of course you don't want any IDE or floppy crap, but that's orthogonal to having ISA. Yes, I forgot to add it to the github issue (I'll fix that), but we do want to add basic support for a PIO based serial port for early debugging. Output only.
IMNSHO, a more interesting starting point would be to upstream the patches to remove legacy devices from the binary (e.g. the infamous floppy). I think this is not exclusive, we can do both.
Cheers, Samuel. --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
On 14/06/2018 12:53, Samuel Ortiz wrote:
- the ARM virt machine type is not a "virtual hardware platform". It has PL011 serial, PL031 RTC, PL061 GPIO controller, a generic PCIe bridge, fw_cfg for the firmware to get data from QEMU, and emulated flash. The idea behind the ARM virt machine type is simply to make everything discoverable via device tree; the "virtualness" of it is simply that the device tree is provided by QEMU and not embedded into something like U-Boot or Linux.
Yes, we're aware of that. It's also one of the very few ARM machine type that does not map to an existing SoC. On that topic, one question: Would you be accepting patches for optionally making the ARM virt machine discoverable through ACPI only? Booting an ARM64 kernel on top of that should work in theory at least.
I'm not the maintainer of the ARM virt machine, so I don't know even what the hurdles are. :) As you probably know, the ARM virt machine is already providing ACPI tables built by QEMU, in addition to the device tree. It includes the UART, flash, virtio-mmio devices, GPIO, but for some reason not the RTC which is only included in the device tree. Probably because it's only meant to be used via the UEFI runtime services, but that's just a guess. Paolo
On Thu, Jun 14, 2018 at 11:15:04AM +0200, Paolo Bonzini wrote:
On 14/06/2018 12:53, Samuel Ortiz wrote:
- the ARM virt machine type is not a "virtual hardware platform". It has PL011 serial, PL031 RTC, PL061 GPIO controller, a generic PCIe bridge, fw_cfg for the firmware to get data from QEMU, and emulated flash. The idea behind the ARM virt machine type is simply to make everything discoverable via device tree; the "virtualness" of it is simply that the device tree is provided by QEMU and not embedded into something like U-Boot or Linux.
Yes, we're aware of that. It's also one of the very few ARM machine type that does not map to an existing SoC. On that topic, one question: Would you be accepting patches for optionally making the ARM virt machine discoverable through ACPI only? Booting an ARM64 kernel on top of that should work in theory at least.
I'm not the maintainer of the ARM virt machine, so I don't know even what the hurdles are. :) Fair enough :)
As you probably know, the ARM virt machine is already providing ACPI tables built by QEMU, in addition to the device tree. It includes the UART, flash, virtio-mmio devices, GPIO, but for some reason not the RTC which is only included in the device tree. Probably because it's only meant to be used via the UEFI runtime services, but that's just a guess. Yes, I quickly hacked it to not add any DT at all, and force an ACPI boot but that did not work. Will try to investigate a little further.
Cheers, Samuel. --------------------------------------------------------------------- Intel Corporation SAS (French simplified joint stock company) Registered headquarters: "Les Montalets"- 2, rue de Paris, 92196 Meudon Cedex, France Registration Number: 302 456 199 R.C.S. NANTERRE Capital: 4,572,000 Euros This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
participants (10)
-
Anthony Liguori
-
Ben Armstrong
-
Castelino, Manohar R
-
Ernst, Eric
-
Jessie Frazelle (TUPPERWARE)
-
Jon Olson
-
Paolo Bonzini
-
Samuel Ortiz
-
Sasha Levin
-
Steve Gordon