govmm/Kata integration, revisited
So.. I just discovered I had a mail misconfiguration which meant at least one recent message I sent to the list on the govmm stuff didn't get through. Which probably explains why I had a different idea of what was going on that other people seem to have had. I proposed the integration of govmm into the Kata repository for two reasons: 1) To avoid the back and forth of synchronized govmm & Kata changes 2) To avoid maintaining another external interface When I first pitched it to Fabiano I had the impression that Kata was the only user of govmm making it an internal interface in practice, so (1) was the main motivation. Having discovered that's not the case, though, (2) is really the more important factor. The Kata team really doesn't have the resources to maintain a general VMM interface - which is in fact an extremely difficult problem (witness the complexity of libvirt). It's made more difficult by the fact that there's not really a natural API boundary in between qemu itself and Kata's hypervisor interface, so what is and isn't in govmm is already pretty arbitrary and based on Kata's needs. AIUI, at the last AC meeting, the decision was made to move govmm into the kata-containers repo, encouraging external users to consume it from the new location. That addresses (1), but not (2), since it's still effectively an external interface. It makes the dilineation of that interface even less clear than it already is. So, I really think this is a bad approach. Instead, what I believe we should do is leave the govmm repository as is, but orphaned, for external users. They can take up maintenance of it, if they have the interest. For Kata, I think we should simply fork the current govmm state, removing any parts we don't need, and in future make no attempt to keep it as anything but an internal interface tailored for Kata's needs. I believe this will let us make some significant simplifications. My issue and pull request (#2393 and #2395) were made with the latter approach in mind, and I'm presently holding off on merging until we're clear on how we're proceding here. -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
David, On Tue, Aug 24, 2021 at 6:35 AM David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
So.. I just discovered I had a mail misconfiguration which meant at least one recent message I sent to the list on the govmm stuff didn't get through. Which probably explains why I had a different idea of what was going on that other people seem to have had.
I proposed the integration of govmm into the Kata repository for two reasons:
1) To avoid the back and forth of synchronized govmm & Kata changes 2) To avoid maintaining another external interface
When I first pitched it to Fabiano I had the impression that Kata was the only user of govmm making it an internal interface in practice, so (1) was the main motivation. Having discovered that's not the case, though, (2) is really the more important factor. The Kata team really doesn't have the resources to maintain a general VMM interface - which is in fact an extremely difficult problem (witness the complexity of libvirt). It's made more difficult by the fact that there's not really a natural API boundary in between qemu itself and Kata's hypervisor interface, so what is and isn't in govmm is already pretty arbitrary and based on Kata's needs.
AIUI, at the last AC meeting, the decision was made to move govmm into the kata-containers repo, encouraging external users to consume it from the new location. That addresses (1), but not (2), since it's still effectively an external interface. It makes the dilineation of that interface even less clear than it already is.
So, I really think this is a bad approach. Instead, what I believe we should do is leave the govmm repository as is, but orphaned, for external users. They can take up maintenance of it, if they have the interest. For Kata, I think we should simply fork the current govmm state, removing any parts we don't need, and in future make no attempt to keep it as anything but an internal interface tailored for Kata's needs. I believe this will let us make some significant simplifications.
My issue and pull request (#2393 and #2395) were made with the latter approach in mind, and I'm presently holding off on merging until we're clear on how we're proceding here.
Ok, thanks. My personal suggestion then would be: 1. Do not import govmm inside kata-containers, at all. 2. Write a thin layer with what you want to have from scratch. 3. Propose that. 4. Be cool if that's not accepted. I don't see this as a high priority effort, though, and I think that for the time being we could keep using govmm, as that has not been a blocker so far. Opinions from others? Best Regards, -- Fabiano Fidêncio
On Mon, Aug 30, 2021 at 01:24:20PM +0200, Fabiano Fidêncio wrote:
David,
On Tue, Aug 24, 2021 at 6:35 AM David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
So.. I just discovered I had a mail misconfiguration which meant at least one recent message I sent to the list on the govmm stuff didn't get through. Which probably explains why I had a different idea of what was going on that other people seem to have had.
I proposed the integration of govmm into the Kata repository for two reasons:
1) To avoid the back and forth of synchronized govmm & Kata changes 2) To avoid maintaining another external interface
When I first pitched it to Fabiano I had the impression that Kata was the only user of govmm making it an internal interface in practice, so (1) was the main motivation. Having discovered that's not the case, though, (2) is really the more important factor. The Kata team really doesn't have the resources to maintain a general VMM interface - which is in fact an extremely difficult problem (witness the complexity of libvirt). It's made more difficult by the fact that there's not really a natural API boundary in between qemu itself and Kata's hypervisor interface, so what is and isn't in govmm is already pretty arbitrary and based on Kata's needs.
AIUI, at the last AC meeting, the decision was made to move govmm into the kata-containers repo, encouraging external users to consume it from the new location. That addresses (1), but not (2), since it's still effectively an external interface. It makes the dilineation of that interface even less clear than it already is.
So, I really think this is a bad approach. Instead, what I believe we should do is leave the govmm repository as is, but orphaned, for external users. They can take up maintenance of it, if they have the interest. For Kata, I think we should simply fork the current govmm state, removing any parts we don't need, and in future make no attempt to keep it as anything but an internal interface tailored for Kata's needs. I believe this will let us make some significant simplifications.
My issue and pull request (#2393 and #2395) were made with the latter approach in mind, and I'm presently holding off on merging until we're clear on how we're proceding here.
Ok, thanks.
My personal suggestion then would be: 1. Do not import govmm inside kata-containers, at all. 2. Write a thin layer with what you want to have from scratch. 3. Propose that. 4. Be cool if that's not accepted.
I don't see this as a high priority effort, though, and I think that for the time being we could keep using govmm, as that has not been a blocker so far.
This really seems like making things more difficult for no benefit. govmm isn't pretty but it works now. By forking and integrating we can strip away the unnecessary bits and clean things up as and when we have time / motivation. By insisting we create a new layer upfront we force doing this as a big, difficult and disruptive mega-patch. -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
On Tue, Aug 31, 2021 at 5:16 AM David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
On Mon, Aug 30, 2021 at 01:24:20PM +0200, Fabiano Fidêncio wrote:
David,
On Tue, Aug 24, 2021 at 6:35 AM David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
So.. I just discovered I had a mail misconfiguration which meant at least one recent message I sent to the list on the govmm stuff didn't get through. Which probably explains why I had a different idea of what was going on that other people seem to have had.
I proposed the integration of govmm into the Kata repository for two reasons:
1) To avoid the back and forth of synchronized govmm & Kata changes 2) To avoid maintaining another external interface
When I first pitched it to Fabiano I had the impression that Kata was the only user of govmm making it an internal interface in practice, so (1) was the main motivation. Having discovered that's not the case, though, (2) is really the more important factor. The Kata team really doesn't have the resources to maintain a general VMM interface - which is in fact an extremely difficult problem (witness the complexity of libvirt). It's made more difficult by the fact that there's not really a natural API boundary in between qemu itself and Kata's hypervisor interface, so what is and isn't in govmm is already pretty arbitrary and based on Kata's needs.
AIUI, at the last AC meeting, the decision was made to move govmm into the kata-containers repo, encouraging external users to consume it from the new location. That addresses (1), but not (2), since it's still effectively an external interface. It makes the dilineation of that interface even less clear than it already is.
So, I really think this is a bad approach. Instead, what I believe we should do is leave the govmm repository as is, but orphaned, for external users. They can take up maintenance of it, if they have the interest. For Kata, I think we should simply fork the current govmm state, removing any parts we don't need, and in future make no attempt to keep it as anything but an internal interface tailored for Kata's needs. I believe this will let us make some significant simplifications.
My issue and pull request (#2393 and #2395) were made with the latter approach in mind, and I'm presently holding off on merging until we're clear on how we're proceding here.
Ok, thanks.
My personal suggestion then would be: 1. Do not import govmm inside kata-containers, at all. 2. Write a thin layer with what you want to have from scratch. 3. Propose that. 4. Be cool if that's not accepted.
I don't see this as a high priority effort, though, and I think that for the time being we could keep using govmm, as that has not been a blocker so far.
This really seems like making things more difficult for no benefit. govmm isn't pretty but it works now. By forking and integrating we can strip away the unnecessary bits and clean things up as and when we have time / motivation.
"when we have time / motivation" is the part that worries me the most, David.
By insisting we create a new layer upfront we force doing this as a big, difficult and disruptive mega-patch.
Actually, the opposite. We force this to be as small and surgical as possible, because we're only including the parts that actually matter for us. Then we will have an one not small patch removing the govmm vendor. Best Regards, -- Fabiano Fidêncio
On Tue, Aug 31, 2021 at 09:23:04AM +0200, Fabiano Fidêncio wrote:
On Tue, Aug 31, 2021 at 5:16 AM David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
On Mon, Aug 30, 2021 at 01:24:20PM +0200, Fabiano Fidêncio wrote:
David,
On Tue, Aug 24, 2021 at 6:35 AM David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
So.. I just discovered I had a mail misconfiguration which meant at least one recent message I sent to the list on the govmm stuff didn't get through. Which probably explains why I had a different idea of what was going on that other people seem to have had.
I proposed the integration of govmm into the Kata repository for two reasons:
1) To avoid the back and forth of synchronized govmm & Kata changes 2) To avoid maintaining another external interface
When I first pitched it to Fabiano I had the impression that Kata was the only user of govmm making it an internal interface in practice, so (1) was the main motivation. Having discovered that's not the case, though, (2) is really the more important factor. The Kata team really doesn't have the resources to maintain a general VMM interface - which is in fact an extremely difficult problem (witness the complexity of libvirt). It's made more difficult by the fact that there's not really a natural API boundary in between qemu itself and Kata's hypervisor interface, so what is and isn't in govmm is already pretty arbitrary and based on Kata's needs.
AIUI, at the last AC meeting, the decision was made to move govmm into the kata-containers repo, encouraging external users to consume it from the new location. That addresses (1), but not (2), since it's still effectively an external interface. It makes the dilineation of that interface even less clear than it already is.
So, I really think this is a bad approach. Instead, what I believe we should do is leave the govmm repository as is, but orphaned, for external users. They can take up maintenance of it, if they have the interest. For Kata, I think we should simply fork the current govmm state, removing any parts we don't need, and in future make no attempt to keep it as anything but an internal interface tailored for Kata's needs. I believe this will let us make some significant simplifications.
My issue and pull request (#2393 and #2395) were made with the latter approach in mind, and I'm presently holding off on merging until we're clear on how we're proceding here.
Ok, thanks.
My personal suggestion then would be: 1. Do not import govmm inside kata-containers, at all. 2. Write a thin layer with what you want to have from scratch. 3. Propose that. 4. Be cool if that's not accepted.
I don't see this as a high priority effort, though, and I think that for the time being we could keep using govmm, as that has not been a blocker so far.
This really seems like making things more difficult for no benefit. govmm isn't pretty but it works now. By forking and integrating we can strip away the unnecessary bits and clean things up as and when we have time / motivation.
"when we have time / motivation" is the part that worries me the most, David.
That's not an unreasonable concern. But what's really that bad about having a half converted thing sitting in there for a while, as long as it's better than what we have now? In addition, there's a natural motivation to fix things further in the integrated govmm code. Basically every time we need to extend or fix something in the ex-govmm code it's an opportunity to clean up that code path.
By insisting we create a new layer upfront we force doing this as a big, difficult and disruptive mega-patch.
Actually, the opposite. We force this to be as small and surgical as possible, because we're only including the parts that actually matter for us.
So, at least at a coarse level, we do actually use most of govmm. It's not that govmm doesn't have code that's useful and/or neccessary - it's just that there's no particular rationale for what sits on the govmm side and what sits on the kata side. I already made a first-pass removal patch, which you've seen, and it's only able to remove one or two functions and structures More stuff can be removed is at a finer-grained level. Options from device configs that Kata doesn't need is the obvious one. Further on it's probably reasonable to start removing the govmm specific device representations entirely, in favour of a more general properties map, for example. There doesn't seem to be much benefit to having an extra data structure as an intermediate between the Kata structures and qemu's representation.
Then we will have an one not small patch removing the govmm vendor.
So, I'd be happy enough with this approach if we could incrementally introduce the new interface then drop govmm once the new interface covers everything. Unfortunately, that doesn't really work, because govmm owns the qmp interface and the qemu command line, so we can't build a new interface "bottom up". -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
On 31 Aug 2021, at 12:31, David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
On Tue, Aug 31, 2021 at 09:23:04AM +0200, Fabiano Fidêncio wrote:
On Tue, Aug 31, 2021 at 5:16 AM David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
On Mon, Aug 30, 2021 at 01:24:20PM +0200, Fabiano Fidêncio wrote:
David,
On Tue, Aug 24, 2021 at 6:35 AM David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
So.. I just discovered I had a mail misconfiguration which meant at least one recent message I sent to the list on the govmm stuff didn't get through. Which probably explains why I had a different idea of what was going on that other people seem to have had.
I proposed the integration of govmm into the Kata repository for two reasons:
1) To avoid the back and forth of synchronized govmm & Kata changes 2) To avoid maintaining another external interface
When I first pitched it to Fabiano I had the impression that Kata was the only user of govmm making it an internal interface in practice, so (1) was the main motivation. Having discovered that's not the case, though, (2) is really the more important factor. The Kata team really doesn't have the resources to maintain a general VMM interface - which is in fact an extremely difficult problem (witness the complexity of libvirt). It's made more difficult by the fact that there's not really a natural API boundary in between qemu itself and Kata's hypervisor interface, so what is and isn't in govmm is already pretty arbitrary and based on Kata's needs.
AIUI, at the last AC meeting, the decision was made to move govmm into the kata-containers repo, encouraging external users to consume it from the new location. That addresses (1), but not (2), since it's still effectively an external interface. It makes the dilineation of that interface even less clear than it already is.
So, I really think this is a bad approach. Instead, what I believe we should do is leave the govmm repository as is, but orphaned, for external users. They can take up maintenance of it, if they have the interest. For Kata, I think we should simply fork the current govmm state, removing any parts we don't need, and in future make no attempt to keep it as anything but an internal interface tailored for Kata's needs. I believe this will let us make some significant simplifications.
My issue and pull request (#2393 and #2395) were made with the latter approach in mind, and I'm presently holding off on merging until we're clear on how we're proceding here.
Ok, thanks.
My personal suggestion then would be: 1. Do not import govmm inside kata-containers, at all. 2. Write a thin layer with what you want to have from scratch. 3. Propose that. 4. Be cool if that's not accepted.
I don't see this as a high priority effort, though, and I think that for the time being we could keep using govmm, as that has not been a blocker so far.
This really seems like making things more difficult for no benefit. govmm isn't pretty but it works now. By forking and integrating we can strip away the unnecessary bits and clean things up as and when we have time / motivation.
"when we have time / motivation" is the part that worries me the most, David.
That's not an unreasonable concern. But what's really that bad about having a half converted thing sitting in there for a while, as long as it's better than what we have now?
In addition, there's a natural motivation to fix things further in the integrated govmm code. Basically every time we need to extend or fix something in the ex-govmm code it's an opportunity to clean up that code path.
By insisting we create a new layer upfront we force doing this as a big, difficult and disruptive mega-patch.
Actually, the opposite. We force this to be as small and surgical as possible, because we're only including the parts that actually matter for us.
So, at least at a coarse level, we do actually use most of govmm. It's not that govmm doesn't have code that's useful and/or neccessary - it's just that there's no particular rationale for what sits on the govmm side and what sits on the kata side. I already made a first-pass removal patch, which you've seen, and it's only able to remove one or two functions and structures
More stuff can be removed is at a finer-grained level. Options from device configs that Kata doesn't need is the obvious one. Further on it's probably reasonable to start removing the govmm specific device representations entirely, in favour of a more general properties map, for example. There doesn't seem to be much benefit to having an extra data structure as an intermediate between the Kata structures and qemu's representation.
Then we will have an one not small patch removing the govmm vendor.
So, I'd be happy enough with this approach if we could incrementally introduce the new interface then drop govmm once the new interface covers everything.
Unfortunately, that doesn't really work, because govmm owns the qmp interface and the qemu command line, so we can't build a new interface "bottom up".
While we are at it, should we rename "govmm" to something that indicates it's really specific to qemu? Or do we want to take the opposite route (which was evoked yesterday) to move to VMM the "hypervisor" abstraction that we have in Kata?
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au <http://gibson.dropbear.id.au/> | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson <http://www.ozlabs.org/~dgibson> _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io <mailto:kata-dev@lists.katacontainers.io> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev <http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev>
[resend with cut quote, original was held in moderation for being > 40K]
On 31 Aug 2021, at 12:31, David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
On Tue, Aug 31, 2021 at 09:23:04AM +0200, Fabiano Fidêncio wrote:
On Tue, Aug 31, 2021 at 5:16 AM David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
On Mon, Aug 30, 2021 at 01:24:20PM +0200, Fabiano Fidêncio wrote:
David,
Then we will have an one not small patch removing the govmm vendor.
So, I'd be happy enough with this approach if we could incrementally introduce the new interface then drop govmm once the new interface covers everything.
Unfortunately, that doesn't really work, because govmm owns the qmp interface and the qemu command line, so we can't build a new interface "bottom up".
While we are at it, should we rename "govmm" to something that indicates it's really specific to qemu? Or do we want to take the opposite route (which was evoked yesterday) to move to VMM the "hypervisor" abstraction that we have in Kata?
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au <http://gibson.dropbear.id.au/> | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson <http://www.ozlabs.org/~dgibson> _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io <mailto:kata-dev@lists.katacontainers.io> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev <http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev>
On Wed, Sep 01, 2021 at 11:43:59AM +0200, Christophe de Dinechin wrote:
[resend with cut quote, original was held in moderation for being > 40K]
On 31 Aug 2021, at 12:31, David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
On Tue, Aug 31, 2021 at 09:23:04AM +0200, Fabiano Fidêncio wrote:
On Tue, Aug 31, 2021 at 5:16 AM David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
On Mon, Aug 30, 2021 at 01:24:20PM +0200, Fabiano Fidêncio wrote:
David,
Then we will have an one not small patch removing the govmm vendor.
So, I'd be happy enough with this approach if we could incrementally introduce the new interface then drop govmm once the new interface covers everything.
Unfortunately, that doesn't really work, because govmm owns the qmp interface and the qemu command line, so we can't build a new interface "bottom up".
While we are at it, should we rename "govmm" to something that indicates it's really specific to qemu?
Fwiw, my draft PR simply calls it the "qemu" package in the imported copy.
Or do we want to take the opposite route (which was evoked yesterday) to move to VMM the "hypervisor" abstraction that we have in Kata?
I'm not really sure what you're suggesting here. -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
On 1 Sep 2021, at 12:05, David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
On Wed, Sep 01, 2021 at 11:43:59AM +0200, Christophe de Dinechin wrote:
[resend with cut quote, original was held in moderation for being > 40K]
On 31 Aug 2021, at 12:31, David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
On Tue, Aug 31, 2021 at 09:23:04AM +0200, Fabiano Fidêncio wrote:
On Tue, Aug 31, 2021 at 5:16 AM David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
On Mon, Aug 30, 2021 at 01:24:20PM +0200, Fabiano Fidêncio wrote:
David,
Then we will have an one not small patch removing the govmm vendor.
So, I'd be happy enough with this approach if we could incrementally introduce the new interface then drop govmm once the new interface covers everything.
Unfortunately, that doesn't really work, because govmm owns the qmp interface and the qemu command line, so we can't build a new interface "bottom up".
While we are at it, should we rename "govmm" to something that indicates it's really specific to qemu?
Fwiw, my draft PR simply calls it the "qemu" package in the imported copy.
Or do we want to take the opposite route (which was evoked yesterday) to move to VMM the "hypervisor" abstraction that we have in Kata?
I'm not really sure what you're suggesting here.
If govmm is a bad hypervisor abstraction, but we have a slightly better one in Kata (i.e. hypervisor.go and the various hypervisor back-ends), then maybe we should rather turn govmm into the hypervisor abstraction it was supposed to be. But that's clearly more complicated.
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au <http://gibson.dropbear.id.au/> | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson <http://www.ozlabs.org/~dgibson>
On Thu, Sep 02, 2021 at 05:52:38PM +0200, Christophe de Dinechin wrote:
On 1 Sep 2021, at 12:05, David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
On Wed, Sep 01, 2021 at 11:43:59AM +0200, Christophe de Dinechin wrote:
[resend with cut quote, original was held in moderation for being > 40K]
On 31 Aug 2021, at 12:31, David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
On Tue, Aug 31, 2021 at 09:23:04AM +0200, Fabiano Fidêncio wrote:
On Tue, Aug 31, 2021 at 5:16 AM David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
On Mon, Aug 30, 2021 at 01:24:20PM +0200, Fabiano Fidêncio wrote: > David, >
Then we will have an one not small patch removing the govmm vendor.
So, I'd be happy enough with this approach if we could incrementally introduce the new interface then drop govmm once the new interface covers everything.
Unfortunately, that doesn't really work, because govmm owns the qmp interface and the qemu command line, so we can't build a new interface "bottom up".
While we are at it, should we rename "govmm" to something that indicates it's really specific to qemu?
Fwiw, my draft PR simply calls it the "qemu" package in the imported copy.
Or do we want to take the opposite route (which was evoked yesterday) to move to VMM the "hypervisor" abstraction that we have in Kata?
I'm not really sure what you're suggesting here.
If govmm is a bad hypervisor abstraction, but we have a slightly better one in Kata (i.e. hypervisor.go and the various hypervisor back-ends), then maybe we should rather turn govmm into the hypervisor abstraction it was supposed to be.
hypervisor.go is a better hypervisor abstraction *for the purpose*. Which is specifically to suit Kata's needs. It's not a viable hypervisor abstraction for anyone else.
But that's clearly more complicated.
I think that's underselling it. Making a general purpose hypervisor abstraction is an immensely difficult problem. I know I sound like a broken record here, but that's been libvirt's entire thing for a decade, and they've at best partially succeeded. Making an general purpose interface purely for qemu is much more practical.. but I still think it's more than the Kata project has spare capacity to take on. And, in this case, it's not very clear exactly what you're abstracting. -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
On 3 Sep 2021, at 07:04, David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
On Thu, Sep 02, 2021 at 05:52:38PM +0200, Christophe de Dinechin wrote:
On 1 Sep 2021, at 12:05, David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
On Wed, Sep 01, 2021 at 11:43:59AM +0200, Christophe de Dinechin wrote:
[resend with cut quote, original was held in moderation for being > 40K]
On 31 Aug 2021, at 12:31, David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
On Tue, Aug 31, 2021 at 09:23:04AM +0200, Fabiano Fidêncio wrote:
On Tue, Aug 31, 2021 at 5:16 AM David Gibson <kata-dev@gibson.dropbear.id.au> wrote: > > On Mon, Aug 30, 2021 at 01:24:20PM +0200, Fabiano Fidêncio wrote: >> David, >>
Then we will have an one not small patch removing the govmm vendor.
So, I'd be happy enough with this approach if we could incrementally introduce the new interface then drop govmm once the new interface covers everything.
Unfortunately, that doesn't really work, because govmm owns the qmp interface and the qemu command line, so we can't build a new interface "bottom up".
While we are at it, should we rename "govmm" to something that indicates it's really specific to qemu?
Fwiw, my draft PR simply calls it the "qemu" package in the imported copy.
Or do we want to take the opposite route (which was evoked yesterday) to move to VMM the "hypervisor" abstraction that we have in Kata?
I'm not really sure what you're suggesting here.
If govmm is a bad hypervisor abstraction, but we have a slightly better one in Kata (i.e. hypervisor.go and the various hypervisor back-ends), then maybe we should rather turn govmm into the hypervisor abstraction it was supposed to be.
hypervisor.go is a better hypervisor abstraction *for the purpose*. Which is specifically to suit Kata's needs. It's not a viable hypervisor abstraction for anyone else.
Most likely true.
But that's clearly more complicated.
I think that's underselling it. Making a general purpose hypervisor abstraction is an immensely difficult problem. I know I sound like a broken record here, but that's been libvirt's entire thing for a decade, and they've at best partially succeeded.
Making an general purpose interface purely for qemu is much more practical.. but I still think it's more than the Kata project has spare capacity to take on. And, in this case, it's not very clear exactly what you're abstracting.
I agree. I was not advocating this approach, but pointing out that the possibility exists. "Clearly more complicated" was intended to implicitly convey the points you made (and others).
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au <http://gibson.dropbear.id.au/> | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson <http://www.ozlabs.org/~dgibson>
On 24 Aug 2021, at 06:34, David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
So.. I just discovered I had a mail misconfiguration which meant at least one recent message I sent to the list on the govmm stuff didn't get through. Which probably explains why I had a different idea of what was going on that other people seem to have had.
I proposed the integration of govmm into the Kata repository for two reasons:
1) To avoid the back and forth of synchronized govmm & Kata changes 2) To avoid maintaining another external interface
When I first pitched it to Fabiano I had the impression that Kata was the only user of govmm making it an internal interface in practice, so (1) was the main motivation. Having discovered that's not the case, though, (2) is really the more important factor. The Kata team really doesn't have the resources to maintain a general VMM interface - which is in fact an extremely difficult problem (witness the complexity of libvirt). It's made more difficult by the fact that there's not really a natural API boundary in between qemu itself and Kata's hypervisor interface, so what is and isn't in govmm is already pretty arbitrary and based on Kata's needs.
AIUI, at the last AC meeting, the decision was made to move govmm into the kata-containers repo, encouraging external users to consume it from the new location. That addresses (1), but not (2), since it's still effectively an external interface. It makes the dilineation of that interface even less clear than it already is.
Yes and no. By putting it inside the kata repo, we make it clear that it is not a public interface. Anyone using it is at risk of breakage. What we should do, however, is mark that with a tag or branch so that if we create incompatible changes, external users can still return to a compatible version. Also maybe add a README pointing out that there is no intention to keep the API stable.
So, I really think this is a bad approach. Instead, what I believe we should do is leave the govmm repository as is, but orphaned, for external users. They can take up maintenance of it, if they have the interest. For Kata, I think we should simply fork the current govmm state, removing any parts we don't need, and in future make no attempt to keep it as anything but an internal interface tailored for Kata's needs. I believe this will let us make some significant simplifications.
My issue and pull request (#2393 and #2395) were made with the latter approach in mind, and I'm presently holding off on merging until we're clear on how we're proceding here.
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
On Mon, Aug 30, 2021 at 02:20:54PM +0200, Christophe de Dinechin wrote:
On 24 Aug 2021, at 06:34, David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
So.. I just discovered I had a mail misconfiguration which meant at least one recent message I sent to the list on the govmm stuff didn't get through. Which probably explains why I had a different idea of what was going on that other people seem to have had.
I proposed the integration of govmm into the Kata repository for two reasons:
1) To avoid the back and forth of synchronized govmm & Kata changes 2) To avoid maintaining another external interface
When I first pitched it to Fabiano I had the impression that Kata was the only user of govmm making it an internal interface in practice, so (1) was the main motivation. Having discovered that's not the case, though, (2) is really the more important factor. The Kata team really doesn't have the resources to maintain a general VMM interface - which is in fact an extremely difficult problem (witness the complexity of libvirt). It's made more difficult by the fact that there's not really a natural API boundary in between qemu itself and Kata's hypervisor interface, so what is and isn't in govmm is already pretty arbitrary and based on Kata's needs.
AIUI, at the last AC meeting, the decision was made to move govmm into the kata-containers repo, encouraging external users to consume it from the new location. That addresses (1), but not (2), since it's still effectively an external interface. It makes the dilineation of that interface even less clear than it already is.
Yes and no. By putting it inside the kata repo, we make it clear that it is not a public interface. Anyone using it is at risk of breakage.
Except that that appears to be explicitly *not* the intention. AIUI, Salvatore (representing the one known external user) was at the AC meeting, and the decision was to merge the repo but that Salavatore would consume govmm from the kata-containers repository. Salvatore's use was cited as a possible argument against my draft patch to remove some things that are not used in Kata [0]. In other words, still an external interface in practice.
What we should do, however, is mark that with a tag or branch so that if we create incompatible changes, external users can still return to a compatible version. Also maybe add a README pointing out that there is no intention to keep the API stable.
So, I really think this is a bad approach. Instead, what I believe we should do is leave the govmm repository as is, but orphaned, for external users. They can take up maintenance of it, if they have the interest. For Kata, I think we should simply fork the current govmm state, removing any parts we don't need, and in future make no attempt to keep it as anything but an internal interface tailored for Kata's needs. I believe this will let us make some significant simplifications.
My issue and pull request (#2393 and #2395) were made with the latter approach in mind, and I'm presently holding off on merging until we're clear on how we're proceding here.
[0] https://github.com/kata-containers/kata-containers/pull/2395#issuecomment-90... https://github.com/kata-containers/kata-containers/pull/2395#issuecomment-90... -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
On Tue, Aug 31, 2021 at 11:15:06AM +1000, David Gibson wrote:
On Mon, Aug 30, 2021 at 02:20:54PM +0200, Christophe de Dinechin wrote:
On 24 Aug 2021, at 06:34, David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
So.. I just discovered I had a mail misconfiguration which meant at least one recent message I sent to the list on the govmm stuff didn't get through. Which probably explains why I had a different idea of what was going on that other people seem to have had.
I proposed the integration of govmm into the Kata repository for two reasons:
1) To avoid the back and forth of synchronized govmm & Kata changes 2) To avoid maintaining another external interface
When I first pitched it to Fabiano I had the impression that Kata was the only user of govmm making it an internal interface in practice, so (1) was the main motivation. Having discovered that's not the case, though, (2) is really the more important factor. The Kata team really doesn't have the resources to maintain a general VMM interface - which is in fact an extremely difficult problem (witness the complexity of libvirt). It's made more difficult by the fact that there's not really a natural API boundary in between qemu itself and Kata's hypervisor interface, so what is and isn't in govmm is already pretty arbitrary and based on Kata's needs.
AIUI, at the last AC meeting, the decision was made to move govmm into the kata-containers repo, encouraging external users to consume it from the new location. That addresses (1), but not (2), since it's still effectively an external interface. It makes the dilineation of that interface even less clear than it already is.
Yes and no. By putting it inside the kata repo, we make it clear that it is not a public interface. Anyone using it is at risk of breakage.
Except that that appears to be explicitly *not* the intention.
AIUI, Salvatore (representing the one known external user) was at the AC meeting, and the decision was to merge the repo but that Salavatore would consume govmm from the kata-containers repository. Salvatore's use was cited as a possible argument against my draft patch to remove some things that are not used in Kata [0].
In other words, still an external interface in practice.
I finally spotted the links to the recordings of the AC meetings, so I've watched those and have a better idea of the details that were discussed. Salvatore using govmm from within the kata repository and being kept in the loop for upcoming changes was explicitly discussed. So, an implicit promise was essentially made to maintain this as an external interface if integrated (at least for now). But as discussed above, I really don't think we have anyone lined up to actually fulfil that promise. It's a little vexing: the discussion started off with essentially the arguments I've made above, however, when it came to actually making a decision focus shifted to the proposed "interim step" of combining the repository but having it used externalling. My fundamental point is that as an interim step this is counter-productive: - It makes things harder on the Kata side, because we still have to treat this as a maintained external interface that we can't break. - It makes things harder on the external user side, because now you need to import a different (and much larger) repository, and the risk of an interface being broken from the Kata side is increased (because it's less obviously an external interface). So to re-iterate, I really think the next step ought to be instead to *fork* govmm into the kata containers repository. Once there it becomes a burely Kata-local copy, which can be modified (and/or eliminated) as per Kata needs. As @sameo said on the call, we already have a hypervisor abstraction layer, we don't need a second one. External users keep using the external govmm, and its up to those users to fix bugs or make extensions there. -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
On 2021/8/31 13:21, David Gibson wrote:
On Tue, Aug 31, 2021 at 11:15:06AM +1000, David Gibson wrote:
On Mon, Aug 30, 2021 at 02:20:54PM +0200, Christophe de Dinechin wrote:
On 24 Aug 2021, at 06:34, David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
So.. I just discovered I had a mail misconfiguration which meant at least one recent message I sent to the list on the govmm stuff didn't get through. Which probably explains why I had a different idea of what was going on that other people seem to have had.
I proposed the integration of govmm into the Kata repository for two reasons:
1) To avoid the back and forth of synchronized govmm & Kata changes 2) To avoid maintaining another external interface
When I first pitched it to Fabiano I had the impression that Kata was the only user of govmm making it an internal interface in practice, so (1) was the main motivation. Having discovered that's not the case, though, (2) is really the more important factor. The Kata team really doesn't have the resources to maintain a general VMM interface - which is in fact an extremely difficult problem (witness the complexity of libvirt). It's made more difficult by the fact that there's not really a natural API boundary in between qemu itself and Kata's hypervisor interface, so what is and isn't in govmm is already pretty arbitrary and based on Kata's needs.
AIUI, at the last AC meeting, the decision was made to move govmm into the kata-containers repo, encouraging external users to consume it from the new location. That addresses (1), but not (2), since it's still effectively an external interface. It makes the dilineation of that interface even less clear than it already is.
Yes and no. By putting it inside the kata repo, we make it clear that it is not a public interface. Anyone using it is at risk of breakage.
Except that that appears to be explicitly *not* the intention.
AIUI, Salvatore (representing the one known external user) was at the AC meeting, and the decision was to merge the repo but that Salavatore would consume govmm from the kata-containers repository. Salvatore's use was cited as a possible argument against my draft patch to remove some things that are not used in Kata [0].
In other words, still an external interface in practice.
I finally spotted the links to the recordings of the AC meetings, so I've watched those and have a better idea of the details that were discussed. Salvatore using govmm from within the kata repository and being kept in the loop for upcoming changes was explicitly discussed.
So, an implicit promise was essentially made to maintain this as an external interface if integrated (at least for now). But as discussed above, I really don't think we have anyone lined up to actually fulfil that promise.
It's a little vexing: the discussion started off with essentially the arguments I've made above, however, when it came to actually making a decision focus shifted to the proposed "interim step" of combining the repository but having it used externalling. My fundamental point is that as an interim step this is counter-productive:
- It makes things harder on the Kata side, because we still have to treat this as a maintained external interface that we can't break.
- It makes things harder on the external user side, because now you need to import a different (and much larger) repository, and the risk of an interface being broken from the Kata side is increased (because it's less obviously an external interface).
So to re-iterate, I really think the next step ought to be instead to *fork* govmm into the kata containers repository. Once there it becomes a burely Kata-local copy, which can be modified (and/or eliminated) as per Kata needs. As @sameo said on the call, we already have a hypervisor abstraction layer, we don't need a second one. External users keep using the external govmm, and its up to those users to fix bugs or make extensions there.
I suggested on the AC call to fork govmm because I thought we'd want to keep its abstraction. If the long term goal is to drop govmm layer, we should make a **copy** of it instead of moving it to kata repo IMO. Then as David outlined, we can tailor it and eventually replace it with a kata specific direct integration with qemu. OTOH, I think what's important for now is to reach agreements on these questions: 1. Do we want to keep the govmm abstraction layer for kata? 2. If not, what is the replacement? We can take it to today's AC meeting IMO. Cheers, Tao
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
On 31 Aug 2021, at 11:50, Peng Tao <bergwolf@hyper.sh> wrote:
On 2021/8/31 13:21, David Gibson wrote:
On Mon, Aug 30, 2021 at 02:20:54PM +0200, Christophe de Dinechin wrote:
On 24 Aug 2021, at 06:34, David Gibson <kata-dev@gibson.dropbear.id.au> wrote:
So.. I just discovered I had a mail misconfiguration which meant at least one recent message I sent to the list on the govmm stuff didn't get through. Which probably explains why I had a different idea of what was going on that other people seem to have had.
I proposed the integration of govmm into the Kata repository for two reasons:
1) To avoid the back and forth of synchronized govmm & Kata changes 2) To avoid maintaining another external interface
When I first pitched it to Fabiano I had the impression that Kata was the only user of govmm making it an internal interface in practice, so (1) was the main motivation. Having discovered that's not the case, though, (2) is really the more important factor. The Kata team really doesn't have the resources to maintain a general VMM interface - which is in fact an extremely difficult problem (witness the complexity of libvirt). It's made more difficult by the fact that there's not really a natural API boundary in between qemu itself and Kata's hypervisor interface, so what is and isn't in govmm is already pretty arbitrary and based on Kata's needs.
AIUI, at the last AC meeting, the decision was made to move govmm into the kata-containers repo, encouraging external users to consume it from the new location. That addresses (1), but not (2), since it's still effectively an external interface. It makes the dilineation of that interface even less clear than it already is.
Yes and no. By putting it inside the kata repo, we make it clear that it is not a public interface. Anyone using it is at risk of breakage.
Except that that appears to be explicitly *not* the intention.
AIUI, Salvatore (representing the one known external user) was at the AC meeting, and the decision was to merge the repo but that Salavatore would consume govmm from the kata-containers repository. Salvatore's use was cited as a possible argument against my draft patch to remove some things that are not used in Kata [0].
In other words, still an external interface in practice. I finally spotted the links to the recordings of the AC meetings, so I've watched those and have a better idea of the details that were discussed. Salvatore using govmm from within the kata repository and being kept in the loop for upcoming changes was explicitly discussed. So, an implicit promise was essentially made to maintain this as an external interface if integrated (at least for now). But as discussed above, I really don't think we have anyone lined up to actually fulfil that promise. It's a little vexing: the discussion started off with essentially the arguments I've made above, however, when it came to actually making a decision focus shifted to the proposed "interim step" of combining the repository but having it used externalling. My fundamental point is
On Tue, Aug 31, 2021 at 11:15:06AM +1000, David Gibson wrote: that as an interim step this is counter-productive: - It makes things harder on the Kata side, because we still have to treat this as a maintained external interface that we can't break. - It makes things harder on the external user side, because now you need to import a different (and much larger) repository, and the risk of an interface being broken from the Kata side is increased (because it's less obviously an external interface). So to re-iterate, I really think the next step ought to be instead to *fork* govmm into the kata containers repository. Once there it becomes a burely Kata-local copy, which can be modified (and/or eliminated) as per Kata needs. As @sameo said on the call, we already have a hypervisor abstraction layer, we don't need a second one. External users keep using the external govmm, and its up to those users to fix bugs or make extensions there. I suggested on the AC call to fork govmm because I thought we'd want to keep its abstraction. If the long term goal is to drop govmm layer, we should make a **copy** of it instead of moving it to kata repo IMO. Then as David outlined, we can tailor it and eventually replace it with a kata specific direct integration with qemu.
OTOH, I think what's important for now is to reach agreements on these questions: 1. Do we want to keep the govmm abstraction layer for kata?
I think one of the points David and others have made is that we already have a hypervisor abstraction in Kata, so we don't need a second one, and that govmm is not a particularly good one either.
2. If not, what is the replacement?
We can take it to today's AC meeting IMO.
Cheers, Tao
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io <mailto:kata-dev@lists.katacontainers.io> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev <http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev>
On 2021/8/31 17:52, Christophe de Dinechin wrote:
On 31 Aug 2021, at 11:50, Peng Tao <bergwolf@hyper.sh <mailto:bergwolf@hyper.sh>> wrote:
On 2021/8/31 13:21, David Gibson wrote:
On Mon, Aug 30, 2021 at 02:20:54PM +0200, Christophe de Dinechin wrote:
On 24 Aug 2021, at 06:34, David Gibson <kata-dev@gibson.dropbear.id.au <mailto:kata-dev@gibson.dropbear.id.au>> wrote:
So.. I just discovered I had a mail misconfiguration which meant at least one recent message I sent to the list on the govmm stuff didn't get through. Which probably explains why I had a different idea of what was going on that other people seem to have had.
I proposed the integration of govmm into the Kata repository for two reasons:
1) To avoid the back and forth of synchronized govmm & Kata changes 2) To avoid maintaining another external interface
When I first pitched it to Fabiano I had the impression that Kata was the only user of govmm making it an internal interface in practice, so (1) was the main motivation. Having discovered that's not the case, though, (2) is really the more important factor. The Kata team really doesn't have the resources to maintain a general VMM interface - which is in fact an extremely difficult problem (witness the complexity of libvirt). It's made more difficult by the fact that there's not really a natural API boundary in between qemu itself and Kata's hypervisor interface, so what is and isn't in govmm is already pretty arbitrary and based on Kata's needs.
AIUI, at the last AC meeting, the decision was made to move govmm into the kata-containers repo, encouraging external users to consume it from the new location. That addresses (1), but not (2), since it's still effectively an external interface. It makes the dilineation of that interface even less clear than it already is.
Yes and no. By putting it inside the kata repo, we make it clear that it is not a public interface. Anyone using it is at risk of breakage.
Except that that appears to be explicitly *not* the intention.
AIUI, Salvatore (representing the one known external user) was at the AC meeting, and the decision was to merge the repo but that Salavatore would consume govmm from the kata-containers repository. Salvatore's use was cited as a possible argument against my draft patch to remove some things that are not used in Kata [0].
In other words, still an external interface in practice. I finally spotted the links to the recordings of the AC meetings, so I've watched those and have a better idea of the details that were discussed. Salvatore using govmm from within the kata repository and being kept in the loop for upcoming changes was explicitly discussed. So, an implicit promise was essentially made to maintain this as an external interface if integrated (at least for now). But as discussed above, I really don't think we have anyone lined up to actually fulfil that promise. It's a little vexing: the discussion started off with essentially the arguments I've made above, however, when it came to actually making a decision focus shifted to the proposed "interim step" of combining the repository but having it used externalling. My fundamental point is
On Tue, Aug 31, 2021 at 11:15:06AM +1000, David Gibson wrote: that as an interim step this is counter-productive: - It makes things harder on the Kata side, because we still have to treat this as a maintained external interface that we can't break. - It makes things harder on the external user side, because now you need to import a different (and much larger) repository, and the risk of an interface being broken from the Kata side is increased (because it's less obviously an external interface). So to re-iterate, I really think the next step ought to be instead to *fork* govmm into the kata containers repository. Once there it becomes a burely Kata-local copy, which can be modified (and/or eliminated) as per Kata needs. As @sameo said on the call, we already have a hypervisor abstraction layer, we don't need a second one. External users keep using the external govmm, and its up to those users to fix bugs or make extensions there. I suggested on the AC call to fork govmm because I thought we'd want to keep its abstraction. If the long term goal is to drop govmm layer, we should make a **copy** of it instead of moving it to kata repo IMO. Then as David outlined, we can tailor it and eventually replace it with a kata specific direct integration with qemu.
OTOH, I think what's important for now is to reach agreements on these questions: 1. Do we want to keep the govmm abstraction layer for kata?
I think one of the points David and others have made is that we already have a hypervisor abstraction in Kata, so we don't need a second one, and that govmm is not a particularly good one either.
I intentionally asked the second one because IMO we need a design for the replacement instead of just saying that we need to drop govmm ;-) I share the same concern that Fabiano expressed earlier:
"when we have time / motivation" is the part that worries me the most, David.
Cheers, Tao
2. If not, what is the replacement?
We can take it to today's AC meeting IMO.
Cheers, Tao
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io <mailto:kata-dev@lists.katacontainers.io> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
participants (4)
-
Christophe de Dinechin
-
David Gibson
-
Fabiano Fidêncio
-
Peng Tao