What to do about release versioning ?
Hi folks, This morning we wanted to release 1.0.1 but held on because of some discussions about the semver semantic not being followed with this 1.0.1 version. The problem here, we have some "sort of" backward compatibility breakage between the runtime and the agent. And for any non compatible release, semver suggest that the new release should bump to the next X version (based on X.Y.Z notation). But before we can answer this, we have to figure out what is that we consider a "breakage". Let me try to explain the different possibility here, starting from an environment where you installed Kata 1.0.0 from packages: First, let's say we introduced a new PauseContainer() command to our agent protocol, this command will have one commit on the agent repo, updating the gRPC protocol and implementing the new feature. This will be followed by a new patch on the runtime to use this new feature, which means we'll have to revendor the new protocol and implement the runtime to rely on it. Now, let's say we bump both runtime and agent repos including those patches to the version 1.0.1, here is the issue we will run into: - The user with Kata 1.0.0 already installed on his system might update the kata-runtime package without updating the VM image containing the agent. And from his perspective, this is fine because he knows about semver semantics and thinks that a kata-runtime 1.0.1 might still work with the VM image container a kata-agent 1.0.0. Unfortunately, this won't work because the kata-runtime will try to call into the new command PauseContainer() that had been introduced, but the agent will have no idea about what this is, and the gRPC will return an error saying this is not a supported/known command. I have a second example, sorry this email is getting long but we have to talk about all cases here... Let's get back to 1.0.0 and say that some changes on both kata-runtime and kata-agent are modifying an existing command CreateContainer(). In this case, this is a more logical breakage since moving one or the other component to the newest version without bumping the other (either kata-runtime 1.0.0 and kata-agent 1.0.1, or kata-runtime 1.0.1 and kata-agent 1.0.0), will end up in functional issues. The second example is what I would call an obvious backward compatibility breakage, but the first one looks still valid to be taken into account. So now, to summarize, we have to agree on what we call a non compatible version that will be translated into a bump to the next X version. And I can see happening a lot of breakage in the future if we include both cases in what is considered as a "breakage", meaning we might end up on version 100.y.z pretty quickly. Numbers are cheap, but here is the status and I/we need some feedback before we can push the pending release (as we don't want to introduce a wrong version number in the history). Thanks, Sebastien
This came up in Vancouver, but I know not everyone had the opportunity to be there. Here were/are my thoughts drawing from my experience working on similar problems for GCE VMs: currently iiuc (until in-place-upgrade and downgrade of releases is supported) the runtime and the guest image (including agent) are all part of the "release artifacts". I would argue that the protocol between the runtime and the agent is not (yet) part of the external contract for Kata. It probably should be, but until that protocol is itself formally documented and versioned I would say you have only changed internal contracts between components that are part of the same atomic set of release artifacts. I do think it's valuable to make the agent protocol part of the external contract for Kata, but I think that requires the protocol itself be explicitly versioned[0]. Jon [0]: I also still have an AI to put together a slide deck on gRPC API evolution best practices to minimize breaking changes generally. I should probably go do that. On Thu, Jun 21, 2018 at 11:54 AM Boeuf, Sebastien <sebastien.boeuf@intel.com> wrote:
Hi folks,
This morning we wanted to release 1.0.1 but held on because of some discussions about the semver semantic not being followed with this 1.0.1 version. The problem here, we have some "sort of" backward compatibility breakage between the runtime and the agent. And for any non compatible release, semver suggest that the new release should bump to the next X version (based on X.Y.Z notation).
But before we can answer this, we have to figure out what is that we consider a "breakage".
Let me try to explain the different possibility here, starting from an environment where you installed Kata 1.0.0 from packages:
First, let's say we introduced a new PauseContainer() command to our agent protocol, this command will have one commit on the agent repo, updating the gRPC protocol and implementing the new feature. This will be followed by a new patch on the runtime to use this new feature, which means we'll have to revendor the new protocol and implement the runtime to rely on it.
Now, let's say we bump both runtime and agent repos including those patches to the version 1.0.1, here is the issue we will run into: - The user with Kata 1.0.0 already installed on his system might update the kata-runtime package without updating the VM image containing the agent. And from his perspective, this is fine because he knows about semver semantics and thinks that a kata-runtime 1.0.1 might still work with the VM image container a kata-agent 1.0.0. Unfortunately, this won't work because the kata-runtime will try to call into the new command PauseContainer() that had been introduced, but the agent will have no idea about what this is, and the gRPC will return an error saying this is not a supported/known command.
I have a second example, sorry this email is getting long but we have to talk about all cases here... Let's get back to 1.0.0 and say that some changes on both kata-runtime and kata-agent are modifying an existing command CreateContainer(). In this case, this is a more logical breakage since moving one or the other component to the newest version without bumping the other (either kata-runtime 1.0.0 and kata-agent 1.0.1, or kata-runtime 1.0.1 and kata-agent 1.0.0), will end up in functional issues.
The second example is what I would call an obvious backward compatibility breakage, but the first one looks still valid to be taken into account.
So now, to summarize, we have to agree on what we call a non compatible version that will be translated into a bump to the next X version. And I can see happening a lot of breakage in the future if we include both cases in what is considered as a "breakage", meaning we might end up on version 100.y.z pretty quickly.
Numbers are cheap, but here is the status and I/we need some feedback before we can push the pending release (as we don't want to introduce a wrong version number in the history).
Thanks, Sebastien _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
I also think that the first change that you described Sebastien, regarding the addition of PauseContainer could be considered backward compatible change requiring a change in the minor number since we are just introducing new functionality here without breaking previous interfaces like CreateContainer etc. -Archana ________________________________ From: Jon Olson via kata-dev [kata-dev@lists.katacontainers.io] Sent: Thursday, June 21, 2018 12:05 PM To: Boeuf, Sebastien Cc: kata-dev@lists.katacontainers.io Subject: Re: [kata-dev] What to do about release versioning ? This came up in Vancouver, but I know not everyone had the opportunity to be there. Here were/are my thoughts drawing from my experience working on similar problems for GCE VMs: currently iiuc (until in-place-upgrade and downgrade of releases is supported) the runtime and the guest image (including agent) are all part of the "release artifacts". I would argue that the protocol between the runtime and the agent is not (yet) part of the external contract for Kata. It probably should be, but until that protocol is itself formally documented and versioned I would say you have only changed internal contracts between components that are part of the same atomic set of release artifacts. I do think it's valuable to make the agent protocol part of the external contract for Kata, but I think that requires the protocol itself be explicitly versioned[0]. Jon [0]: I also still have an AI to put together a slide deck on gRPC API evolution best practices to minimize breaking changes generally. I should probably go do that. On Thu, Jun 21, 2018 at 11:54 AM Boeuf, Sebastien <sebastien.boeuf@intel.com<mailto:sebastien.boeuf@intel.com>> wrote: Hi folks, This morning we wanted to release 1.0.1 but held on because of some discussions about the semver semantic not being followed with this 1.0.1 version. The problem here, we have some "sort of" backward compatibility breakage between the runtime and the agent. And for any non compatible release, semver suggest that the new release should bump to the next X version (based on X.Y.Z notation). But before we can answer this, we have to figure out what is that we consider a "breakage". Let me try to explain the different possibility here, starting from an environment where you installed Kata 1.0.0 from packages: First, let's say we introduced a new PauseContainer() command to our agent protocol, this command will have one commit on the agent repo, updating the gRPC protocol and implementing the new feature. This will be followed by a new patch on the runtime to use this new feature, which means we'll have to revendor the new protocol and implement the runtime to rely on it. Now, let's say we bump both runtime and agent repos including those patches to the version 1.0.1, here is the issue we will run into: - The user with Kata 1.0.0 already installed on his system might update the kata-runtime package without updating the VM image containing the agent. And from his perspective, this is fine because he knows about semver semantics and thinks that a kata-runtime 1.0.1 might still work with the VM image container a kata-agent 1.0.0. Unfortunately, this won't work because the kata-runtime will try to call into the new command PauseContainer() that had been introduced, but the agent will have no idea about what this is, and the gRPC will return an error saying this is not a supported/known command. I have a second example, sorry this email is getting long but we have to talk about all cases here... Let's get back to 1.0.0 and say that some changes on both kata-runtime and kata-agent are modifying an existing command CreateContainer(). In this case, this is a more logical breakage since moving one or the other component to the newest version without bumping the other (either kata-runtime 1.0.0 and kata-agent 1.0.1, or kata-runtime 1.0.1 and kata-agent 1.0.0), will end up in functional issues. The second example is what I would call an obvious backward compatibility breakage, but the first one looks still valid to be taken into account. So now, to summarize, we have to agree on what we call a non compatible version that will be translated into a bump to the next X version. And I can see happening a lot of breakage in the future if we include both cases in what is considered as a "breakage", meaning we might end up on version 100.y.z pretty quickly. Numbers are cheap, but here is the status and I/we need some feedback before we can push the pending release (as we don't want to introduce a wrong version number in the history). Thanks, Sebastien _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io<mailto:kata-dev@lists.katacontainers.io> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
I agree on this stance, Jon. Perhaps until we have the protocol/API defined and versioned this shouldn’t be necessary. Stepping back, I think it makes more sense to fix the bugs existing (which I think are quite troubling) and documenting expectations of tags/packages rather than holding off releases. I do acknowledge I often like to break the rules, and think this is such a scenario. Bottom line, for many reasons we need to very well define the protocol and version it (this is not done yet), and I expect there will be changes that continue in this space until we have the major use cases of Frakti/Huawei figured out. Eric From: Jon Olson via kata-dev <kata-dev@lists.katacontainers.io> Reply-To: Jon Olson <jonolson@google.com> Date: Thursday, June 21, 2018 at 12:06 PM To: "Boeuf, Sebastien" <sebastien.boeuf@intel.com> Cc: "kata-dev@lists.katacontainers.io" <kata-dev@lists.katacontainers.io> Subject: Re: [kata-dev] What to do about release versioning ? This came up in Vancouver, but I know not everyone had the opportunity to be there. Here were/are my thoughts drawing from my experience working on similar problems for GCE VMs: currently iiuc (until in-place-upgrade and downgrade of releases is supported) the runtime and the guest image (including agent) are all part of the "release artifacts". I would argue that the protocol between the runtime and the agent is not (yet) part of the external contract for Kata. It probably should be, but until that protocol is itself formally documented and versioned I would say you have only changed internal contracts between components that are part of the same atomic set of release artifacts. I do think it's valuable to make the agent protocol part of the external contract for Kata, but I think that requires the protocol itself be explicitly versioned[0]. Jon [0]: I also still have an AI to put together a slide deck on gRPC API evolution best practices to minimize breaking changes generally. I should probably go do that. On Thu, Jun 21, 2018 at 11:54 AM Boeuf, Sebastien <sebastien.boeuf@intel.com<mailto:sebastien.boeuf@intel.com>> wrote: Hi folks, This morning we wanted to release 1.0.1 but held on because of some discussions about the semver semantic not being followed with this 1.0.1 version. The problem here, we have some "sort of" backward compatibility breakage between the runtime and the agent. And for any non compatible release, semver suggest that the new release should bump to the next X version (based on X.Y.Z notation). But before we can answer this, we have to figure out what is that we consider a "breakage". Let me try to explain the different possibility here, starting from an environment where you installed Kata 1.0.0 from packages: First, let's say we introduced a new PauseContainer() command to our agent protocol, this command will have one commit on the agent repo, updating the gRPC protocol and implementing the new feature. This will be followed by a new patch on the runtime to use this new feature, which means we'll have to revendor the new protocol and implement the runtime to rely on it. Now, let's say we bump both runtime and agent repos including those patches to the version 1.0.1, here is the issue we will run into: - The user with Kata 1.0.0 already installed on his system might update the kata-runtime package without updating the VM image containing the agent. And from his perspective, this is fine because he knows about semver semantics and thinks that a kata-runtime 1.0.1 might still work with the VM image container a kata-agent 1.0.0. Unfortunately, this won't work because the kata-runtime will try to call into the new command PauseContainer() that had been introduced, but the agent will have no idea about what this is, and the gRPC will return an error saying this is not a supported/known command. I have a second example, sorry this email is getting long but we have to talk about all cases here... Let's get back to 1.0.0 and say that some changes on both kata-runtime and kata-agent are modifying an existing command CreateContainer(). In this case, this is a more logical breakage since moving one or the other component to the newest version without bumping the other (either kata-runtime 1.0.0 and kata-agent 1.0.1, or kata-runtime 1.0.1 and kata-agent 1.0.0), will end up in functional issues. The second example is what I would call an obvious backward compatibility breakage, but the first one looks still valid to be taken into account. So now, to summarize, we have to agree on what we call a non compatible version that will be translated into a bump to the next X version. And I can see happening a lot of breakage in the future if we include both cases in what is considered as a "breakage", meaning we might end up on version 100.y.z pretty quickly. Numbers are cheap, but here is the status and I/we need some feedback before we can push the pending release (as we don't want to introduce a wrong version number in the history). Thanks, Sebastien _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io<mailto:kata-dev@lists.katacontainers.io> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
Fine with me, but if we go this way, we should indicate here: https://github.com/kata-containers/documentation/blob/master/Releases.md#ver... that a user can never expect to use components (runtime, proxy, agent,...) from different versions, even if the bump is simply regarding Z (based on X.Y.Z), because we cannot guarantee they'll work together. Thanks, Sebastien ________________________________ From: Ernst, Eric Sent: Thursday, June 21, 2018 3:59 PM To: Jon Olson; Boeuf, Sebastien Cc: kata-dev@lists.katacontainers.io Subject: Re: [kata-dev] What to do about release versioning ? I agree on this stance, Jon. Perhaps until we have the protocol/API defined and versioned this shouldn’t be necessary. Stepping back, I think it makes more sense to fix the bugs existing (which I think are quite troubling) and documenting expectations of tags/packages rather than holding off releases. I do acknowledge I often like to break the rules, and think this is such a scenario. Bottom line, for many reasons we need to very well define the protocol and version it (this is not done yet), and I expect there will be changes that continue in this space until we have the major use cases of Frakti/Huawei figured out. Eric From: Jon Olson via kata-dev <kata-dev@lists.katacontainers.io> Reply-To: Jon Olson <jonolson@google.com> Date: Thursday, June 21, 2018 at 12:06 PM To: "Boeuf, Sebastien" <sebastien.boeuf@intel.com> Cc: "kata-dev@lists.katacontainers.io" <kata-dev@lists.katacontainers.io> Subject: Re: [kata-dev] What to do about release versioning ? This came up in Vancouver, but I know not everyone had the opportunity to be there. Here were/are my thoughts drawing from my experience working on similar problems for GCE VMs: currently iiuc (until in-place-upgrade and downgrade of releases is supported) the runtime and the guest image (including agent) are all part of the "release artifacts". I would argue that the protocol between the runtime and the agent is not (yet) part of the external contract for Kata. It probably should be, but until that protocol is itself formally documented and versioned I would say you have only changed internal contracts between components that are part of the same atomic set of release artifacts. I do think it's valuable to make the agent protocol part of the external contract for Kata, but I think that requires the protocol itself be explicitly versioned[0]. Jon [0]: I also still have an AI to put together a slide deck on gRPC API evolution best practices to minimize breaking changes generally. I should probably go do that. On Thu, Jun 21, 2018 at 11:54 AM Boeuf, Sebastien <sebastien.boeuf@intel.com<mailto:sebastien.boeuf@intel.com>> wrote: Hi folks, This morning we wanted to release 1.0.1 but held on because of some discussions about the semver semantic not being followed with this 1.0.1 version. The problem here, we have some "sort of" backward compatibility breakage between the runtime and the agent. And for any non compatible release, semver suggest that the new release should bump to the next X version (based on X.Y.Z notation). But before we can answer this, we have to figure out what is that we consider a "breakage". Let me try to explain the different possibility here, starting from an environment where you installed Kata 1.0.0 from packages: First, let's say we introduced a new PauseContainer() command to our agent protocol, this command will have one commit on the agent repo, updating the gRPC protocol and implementing the new feature. This will be followed by a new patch on the runtime to use this new feature, which means we'll have to revendor the new protocol and implement the runtime to rely on it. Now, let's say we bump both runtime and agent repos including those patches to the version 1.0.1, here is the issue we will run into: - The user with Kata 1.0.0 already installed on his system might update the kata-runtime package without updating the VM image containing the agent. And from his perspective, this is fine because he knows about semver semantics and thinks that a kata-runtime 1.0.1 might still work with the VM image container a kata-agent 1.0.0. Unfortunately, this won't work because the kata-runtime will try to call into the new command PauseContainer() that had been introduced, but the agent will have no idea about what this is, and the gRPC will return an error saying this is not a supported/known command. I have a second example, sorry this email is getting long but we have to talk about all cases here... Let's get back to 1.0.0 and say that some changes on both kata-runtime and kata-agent are modifying an existing command CreateContainer(). In this case, this is a more logical breakage since moving one or the other component to the newest version without bumping the other (either kata-runtime 1.0.0 and kata-agent 1.0.1, or kata-runtime 1.0.1 and kata-agent 1.0.0), will end up in functional issues. The second example is what I would call an obvious backward compatibility breakage, but the first one looks still valid to be taken into account. So now, to summarize, we have to agree on what we call a non compatible version that will be translated into a bump to the next X version. And I can see happening a lot of breakage in the future if we include both cases in what is considered as a "breakage", meaning we might end up on version 100.y.z pretty quickly. Numbers are cheap, but here is the status and I/we need some feedback before we can push the pending release (as we don't want to introduce a wrong version number in the history). Thanks, Sebastien _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io<mailto:kata-dev@lists.katacontainers.io> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
My two cents: this resembles k8s's kubelet / apiserver versioning. a change that can potentially be incompatable is a minor version bump. so say, 1.10 to 1.11. upgrading kubelet 1.10.1 to 1.10.2 usually doesn't break existing things. at times they have said you must evacuate a node going from 1.x to 1.x+1 though. Which is similar to the use case below. no way to maintain state. They grantee kubelet 1.x will work with up to apiserver 1.x+2 So, kata-runtime is similar to the kube-apiserver in this case and the rest is kubelet. the vm/agenty things should be able to lag a few versions but until implemented, must match. It should probably bump the minor version, as it really is an incompatible version change and not just a bug fix release as 1.0.1 would imply. so I'd recommend calling the next version 1.1.0 instead of 1.0.1. And note in the upgrade notes that the node must be drained before upgrading. People might not look for a release note if it was called 1.0.1 as they won't really expect a breaking change in that case. Thanks, Kevin ________________________________ From: Ernst, Eric [eric.ernst@intel.com] Sent: Thursday, June 21, 2018 3:59 PM To: Jon Olson; Boeuf, Sebastien Cc: kata-dev@lists.katacontainers.io Subject: Re: [kata-dev] What to do about release versioning ? I agree on this stance, Jon. Perhaps until we have the protocol/API defined and versioned this shouldn’t be necessary. Stepping back, I think it makes more sense to fix the bugs existing (which I think are quite troubling) and documenting expectations of tags/packages rather than holding off releases. I do acknowledge I often like to break the rules, and think this is such a scenario. Bottom line, for many reasons we need to very well define the protocol and version it (this is not done yet), and I expect there will be changes that continue in this space until we have the major use cases of Frakti/Huawei figured out. Eric From: Jon Olson via kata-dev <kata-dev@lists.katacontainers.io> Reply-To: Jon Olson <jonolson@google.com> Date: Thursday, June 21, 2018 at 12:06 PM To: "Boeuf, Sebastien" <sebastien.boeuf@intel.com> Cc: "kata-dev@lists.katacontainers.io" <kata-dev@lists.katacontainers.io> Subject: Re: [kata-dev] What to do about release versioning ? This came up in Vancouver, but I know not everyone had the opportunity to be there. Here were/are my thoughts drawing from my experience working on similar problems for GCE VMs: currently iiuc (until in-place-upgrade and downgrade of releases is supported) the runtime and the guest image (including agent) are all part of the "release artifacts". I would argue that the protocol between the runtime and the agent is not (yet) part of the external contract for Kata. It probably should be, but until that protocol is itself formally documented and versioned I would say you have only changed internal contracts between components that are part of the same atomic set of release artifacts. I do think it's valuable to make the agent protocol part of the external contract for Kata, but I think that requires the protocol itself be explicitly versioned[0]. Jon [0]: I also still have an AI to put together a slide deck on gRPC API evolution best practices to minimize breaking changes generally. I should probably go do that. On Thu, Jun 21, 2018 at 11:54 AM Boeuf, Sebastien <sebastien.boeuf@intel.com<mailto:sebastien.boeuf@intel.com>> wrote: Hi folks, This morning we wanted to release 1.0.1 but held on because of some discussions about the semver semantic not being followed with this 1.0.1 version. The problem here, we have some "sort of" backward compatibility breakage between the runtime and the agent. And for any non compatible release, semver suggest that the new release should bump to the next X version (based on X.Y.Z notation). But before we can answer this, we have to figure out what is that we consider a "breakage". Let me try to explain the different possibility here, starting from an environment where you installed Kata 1.0.0 from packages: First, let's say we introduced a new PauseContainer() command to our agent protocol, this command will have one commit on the agent repo, updating the gRPC protocol and implementing the new feature. This will be followed by a new patch on the runtime to use this new feature, which means we'll have to revendor the new protocol and implement the runtime to rely on it. Now, let's say we bump both runtime and agent repos including those patches to the version 1.0.1, here is the issue we will run into: - The user with Kata 1.0.0 already installed on his system might update the kata-runtime package without updating the VM image containing the agent. And from his perspective, this is fine because he knows about semver semantics and thinks that a kata-runtime 1.0.1 might still work with the VM image container a kata-agent 1.0.0. Unfortunately, this won't work because the kata-runtime will try to call into the new command PauseContainer() that had been introduced, but the agent will have no idea about what this is, and the gRPC will return an error saying this is not a supported/known command. I have a second example, sorry this email is getting long but we have to talk about all cases here... Let's get back to 1.0.0 and say that some changes on both kata-runtime and kata-agent are modifying an existing command CreateContainer(). In this case, this is a more logical breakage since moving one or the other component to the newest version without bumping the other (either kata-runtime 1.0.0 and kata-agent 1.0.1, or kata-runtime 1.0.1 and kata-agent 1.0.0), will end up in functional issues. The second example is what I would call an obvious backward compatibility breakage, but the first one looks still valid to be taken into account. So now, to summarize, we have to agree on what we call a non compatible version that will be translated into a bump to the next X version. And I can see happening a lot of breakage in the future if we include both cases in what is considered as a "breakage", meaning we might end up on version 100.y.z pretty quickly. Numbers are cheap, but here is the status and I/we need some feedback before we can push the pending release (as we don't want to introduce a wrong version number in the history). Thanks, Sebastien _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io<mailto:kata-dev@lists.katacontainers.io> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
On Fri, Jun 22, 2018 at 2:53 AM, Boeuf, Sebastien <sebastien.boeuf@intel.com> wrote:
Hi folks,
This morning we wanted to release 1.0.1 but held on because of some discussions about the semver semantic not being followed with this 1.0.1 version. The problem here, we have some "sort of" backward compatibility breakage between the runtime and the agent. And for any non compatible release, semver suggest that the new release should bump to the next X version (based on X.Y.Z notation).
But before we can answer this, we have to figure out what is that we consider a "breakage".
Let me try to explain the different possibility here, starting from an environment where you installed Kata 1.0.0 from packages:
First, let's say we introduced a new PauseContainer() command to our agent protocol, this command will have one commit on the agent repo, updating the gRPC protocol and implementing the new feature. This will be followed by a new patch on the runtime to use this new feature, which means we'll have to revendor the new protocol and implement the runtime to rely on it.
Now, let's say we bump both runtime and agent repos including those patches to the version 1.0.1, here is the issue we will run into: - The user with Kata 1.0.0 already installed on his system might update the kata-runtime package without updating the VM image containing the agent. And from his perspective, this is fine because he knows about semver semantics and thinks that a kata-runtime 1.0.1 might still work with the VM image container a kata-agent 1.0.0. Unfortunately, this won't work because the kata-runtime will try to call into the new command PauseContainer() that had been introduced, but the agent will have no idea about what this is, and the gRPC will return an error saying this is not a supported/known command.
I have a second example, sorry this email is getting long but we have to talk about all cases here... Let's get back to 1.0.0 and say that some changes on both kata-runtime and kata-agent are modifying an existing command CreateContainer(). In this case, this is a more logical breakage since moving one or the other component to the newest version without bumping the other (either kata-runtime 1.0.0 and kata-agent 1.0.1, or kata-runtime 1.0.1 and kata-agent 1.0.0), will end up in functional issues.
The second example is what I would call an obvious backward compatibility breakage, but the first one looks still valid to be taken into account.
So now, to summarize, we have to agree on what we call a non compatible version that will be translated into a bump to the next X version. And I can see happening a lot of breakage in the future if we include both cases in what is considered as a "breakage", meaning we might end up on version 100.y.z pretty quickly.
If we follow the general guide of adding new grpc fields but do not change or delete existing ones [1], we can be sure that v1.0.0 runtime works with the new v1.0.x agent. OTOH, with any agent protocol extension (new fields or new RPC requests), we cannot guarantee that the new v1.0.x runtime works with v1.0.0 agent, because the runtime will send RPC requests containing newly added fields/RPCs. So the question is what kind of compatibility we are looking at here? 1). old runtime + new agent 2). new runtime + old agent If we want to keep both 1 and 2, I think we will have to bump up the major release version number every time we change the agent protocol. If we just want to keep 1, most agent protocol change will not break it as long as we do not change or delete existing fields. I am fine with either one but we need to get consensus about the kind of compatibility we want to maintain and stick to it. ref: [1] https://developers.google.com/protocol-buffers/docs/overview Cheers, Tao -- bergwolf@hyper.sh
Actually I also mentioned this in Vancouver, in my opinion, a breakage between kata-agent and kata-runtime should always be considered as a backward compatibility breakage. This breakage is a "gap" between "project" and "product" for kata-containers, I'll elaborate why here. Starting from Huawei's requirement for a mature cloud product in use of kata, we have SLA with our customers, which means we can't shutdown customers' service while we are updating Kata components, this feature is named as "live-update", so running kata-runtime and agent of different versions will very likely happen: 1). New runtime + old agent: updating kata-runtime when VM+old agent is running, kata-runtime shouldn't issue a command which will crash the agent. 2). old runtime + new agent: rollback when new kata version has issues, in this case, some service could be started already, new agent should always handle commands from old runtime. So what will happen if we miss 1) and 2)? We need to shutdown user's running workload whenever we want to upgrade/downgrade the kata-components, that will make our SLA a joke. (Of course we can also choose to send them notification and let users shutdown their workload by themselves, but we definitely hope to do better and go further. ) So to guarantee the "live-upgrade" ability of kata-components(meas install kata rpm packages while workloads are still running), what we need to do for these components are: 1) kata-runtime: A. issue "versioned" command to kata-agent, can always communicate correctly with old kata-agent. (MUST) B. disk persist data should be "versioned", kata-runtime can always handle old "version" of persist data to restore sandbox/container struct from disk to memory. (MUST) 2) kata-agent: protocol needs to be versioned, can always handle commands from old kata-runtime. ("versioned" may be achieved by leveraging protobuf) (MUST) 3) kata-shim/kata-proxy: daemon process, no need to shut down while updating kata rpm package. So I don't see a problem currently, need to guarantee interact between kata-runtime and shim/proxy. (MUST) 4) qemu: A. current status: NO WAY to upgrade now. running workload must be shutdown before installing newer version of qemu rpm package. (IMPOSSIBLE) B. In future: qemu live-migration, live-replacement, live-patch etc. (BETTER HAVE) 5) guest kernel: A. current status: after install kata rpm package with newer VM image, old workloads can keep running with old kernel, newly started workload will use new VM kernel. It's fine. (ALREADY HAVE) B. in future: live patch. (BETTER HAVE) To summarize: 1) We already break the backward compatibility, and we will break a lot more in near future definitely. Actually in Vancouver, the participants all agree that we can't promise the API won't be broken and current API isn't a stable version. 2) Before we claim that kata can support "live ugrade" and kata is real production ready, I'm fine with the breakage and also fine with 1.0.1 or 1.1.0, maybe latter one looks better. 3) After we claim that kata can support "live upgrade" in future, we should reject any modifications which will break the running workloads, unless this is really inevitable, by then, we need to upgrade kata version from x.0.0 to y.0.0. But I hope our kata developers can understand what a disaster this could be to a cloud provider like us :-(, and I hope this will never happen. 4) Better document that we don't support "live upgrade" yet, and tell users that if you want to upgrade to this new kata-containers version, you must stop all you running kata containers, or there will be anticipated issues. BR, Wei -----邮件原件----- 发件人: Tao Peng [mailto:bergwolf@hyper.sh] 发送时间: 2018年6月22日 8:26 收件人: Boeuf, Sebastien <sebastien.boeuf@intel.com> 抄送: kata-dev@lists.katacontainers.io 主题: Re: [kata-dev] What to do about release versioning ? On Fri, Jun 22, 2018 at 2:53 AM, Boeuf, Sebastien <sebastien.boeuf@intel.com> wrote:
Hi folks,
This morning we wanted to release 1.0.1 but held on because of some discussions about the semver semantic not being followed with this 1.0.1 version. The problem here, we have some "sort of" backward compatibility breakage between the runtime and the agent. And for any non compatible release, semver suggest that the new release should bump to the next X version (based on X.Y.Z notation).
But before we can answer this, we have to figure out what is that we consider a "breakage".
Let me try to explain the different possibility here, starting from an environment where you installed Kata 1.0.0 from packages:
First, let's say we introduced a new PauseContainer() command to our agent protocol, this command will have one commit on the agent repo, updating the gRPC protocol and implementing the new feature. This will be followed by a new patch on the runtime to use this new feature, which means we'll have to revendor the new protocol and implement the runtime to rely on it.
Now, let's say we bump both runtime and agent repos including those patches to the version 1.0.1, here is the issue we will run into: - The user with Kata 1.0.0 already installed on his system might update the kata-runtime package without updating the VM image containing the agent. And from his perspective, this is fine because he knows about semver semantics and thinks that a kata-runtime 1.0.1 might still work with the VM image container a kata-agent 1.0.0. Unfortunately, this won't work because the kata-runtime will try to call into the new command PauseContainer() that had been introduced, but the agent will have no idea about what this is, and the gRPC will return an error saying this is not a supported/known command.
I have a second example, sorry this email is getting long but we have to talk about all cases here... Let's get back to 1.0.0 and say that some changes on both kata-runtime and kata-agent are modifying an existing command CreateContainer(). In this case, this is a more logical breakage since moving one or the other component to the newest version without bumping the other (either kata-runtime 1.0.0 and kata-agent 1.0.1, or kata-runtime 1.0.1 and kata-agent 1.0.0), will end up in functional issues.
The second example is what I would call an obvious backward compatibility breakage, but the first one looks still valid to be taken into account.
So now, to summarize, we have to agree on what we call a non compatible version that will be translated into a bump to the next X version. And I can see happening a lot of breakage in the future if we include both cases in what is considered as a "breakage", meaning we might end up on version 100.y.z pretty quickly.
If we follow the general guide of adding new grpc fields but do not change or delete existing ones [1], we can be sure that v1.0.0 runtime works with the new v1.0.x agent. OTOH, with any agent protocol extension (new fields or new RPC requests), we cannot guarantee that the new v1.0.x runtime works with v1.0.0 agent, because the runtime will send RPC requests containing newly added fields/RPCs. So the question is what kind of compatibility we are looking at here? 1). old runtime + new agent 2). new runtime + old agent If we want to keep both 1 and 2, I think we will have to bump up the major release version number every time we change the agent protocol. If we just want to keep 1, most agent protocol change will not break it as long as we do not change or delete existing fields. I am fine with either one but we need to get consensus about the kind of compatibility we want to maintain and stick to it. ref: [1] https://developers.google.com/protocol-buffers/docs/overview Cheers, Tao -- bergwolf@hyper.sh _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
zhangwei (CR) wrote:
To summarize: 1) We already break the backward compatibility, and we will break a lot more in near future definitely. Actually in Vancouver, the participants all agree that we can't promise the API won't be broken and current API isn't a stable version. 2) Before we claim that kata can support "live ugrade" and kata is real production ready, I'm fine with the breakage and also fine with 1.0.1 or 1.1.0, maybe latter one looks better. 3) After we claim that kata can support "live upgrade" in future, we should reject any modifications which will break the running workloads, unless this is really inevitable, by then, we need to upgrade kata version from x.0.0 to y.0.0. But I hope our kata developers can understand what a disaster this could be to a cloud provider like us :-(, and I hope this will never happen. 4) Better document that we don't support "live upgrade" yet, and tell users that if you want to upgrade to this new kata-containers version, you must stop all you running kata containers, or there will be anticipated issues.
That summary sounds in line with what Jon said... the protocol between the runtime and the agent is not (yet) part of the external contract for Kata. Once it is (be it by supporting live upgrade or other explicit documentation that you support mixing versions between agent and runtime) then you should avoid breaking that altogether (and bump X number in the case you really need to). In example 1 from Sebastien (PauseContainer()), I would still recommend you bump Y though, and make it 1.1.0. You're adding a feature and modifying the protocol. I would keep .Z bumps for basic bugfixes that do not introduce incompatible protocol changes at all, if only to develop user confidence that those .Z bumps can be deployed with limited risk. -- Thierry Carrez (ttx)
Hi Sebastien, Thanks very much for kicking off what has turned into a great thread from the humble beginnings of [1] ;) I've attempted to summarise it to distill the essence as there is a lot of detail: - There seems to be general consensus that: - 1.1.0 is better than 2.0.0. - 1.1.0 is also better than 1.0.1. By changing the minor number users can see it is more than a bugfix release, but we're arguably abusing the naming slightly by not bumping the major number. That seeming abuse of semver is handled by a "get out of jail free card", namely that we haven't documented the gRPC protocol and hence implicitly haven't agreed (and documented) precisely what API breakage means yet. We also need to remember to include details of the breakage in the release notes. - Plan B: It's worth mentioned that another thought we had yesterday was that we could revert the breaking change [2] to allow us to release a true 1.0.1. That would give us breathing space to investigate the breakage more fully and find a way to avoid it happening in the future (or atleast detect and minimise). == TODO list from this thread == I've raised a bunch of issues here and referenced some existing ones. Please dive in by commenting, emojifying, assigning to yourself (please! ;), etc: - Jon plans to write up a slide deck on gRPC API evolution best practices (I'd love to see this! ;) - We need to document the gRPC protocol: https://github.com/kata-containers/agent/issues/150 - We need to version the gRPC protocol: https://github.com/kata-containers/kata-containers/issues/17 https://github.com/kata-containers/agent/issues/272 - We need to guarantee persistent state files are consumable by newer component versions: https://github.com/kata-containers/kata-containers/issues/25 - We need to update our documentation to explain that we cannot handle "live upgrade": https://github.com/kata-containers/documentation/issues/178 - We need to update our documentation to explain that all component versions must (currently) match for correct operation: https://github.com/kata-containers/documentation/issues/177 Cheers, James [1] - https://github.com/kata-containers/proxy/pull/76 [2] - https://github.com/kata-containers/runtime/pull/357 2018-06-22 10:55 GMT+01:00 Thierry Carrez <thierry@openstack.org>:
zhangwei (CR) wrote:
To summarize: 1) We already break the backward compatibility, and we will break a lot more in near future definitely. Actually in Vancouver, the participants all agree that we can't promise the API won't be broken and current API isn't a stable version. 2) Before we claim that kata can support "live ugrade" and kata is real production ready, I'm fine with the breakage and also fine with 1.0.1 or 1.1.0, maybe latter one looks better. 3) After we claim that kata can support "live upgrade" in future, we should reject any modifications which will break the running workloads, unless this is really inevitable, by then, we need to upgrade kata version from x.0.0 to y.0.0. But I hope our kata developers can understand what a disaster this could be to a cloud provider like us :-(, and I hope this will never happen. 4) Better document that we don't support "live upgrade" yet, and tell users that if you want to upgrade to this new kata-containers version, you must stop all you running kata containers, or there will be anticipated issues.
That summary sounds in line with what Jon said... the protocol between the runtime and the agent is not (yet) part of the external contract for Kata. Once it is (be it by supporting live upgrade or other explicit documentation that you support mixing versions between agent and runtime) then you should avoid breaking that altogether (and bump X number in the case you really need to).
In example 1 from Sebastien (PauseContainer()), I would still recommend you bump Y though, and make it 1.1.0. You're adding a feature and modifying the protocol. I would keep .Z bumps for basic bugfixes that do not introduce incompatible protocol changes at all, if only to develop user confidence that those .Z bumps can be deployed with limited risk.
-- Thierry Carrez (ttx)
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
-- James --- https://katacontainers.io/ | https://github.com/kata-containers <https://github.com/clearcontainers> Open Source Technology Center Intel Corporation (UK) Ltd. - Co. Reg. #1134945 - Pipers Way, Swindon SN3 1RJ.
+1 on 1.1.0. Live-upgrade is nice, but lots of people are starting to do more immutable infrastructure. Bring up new workloads on a brand new server/cluster with a newer version. Just need to document that 1.1.0 is not compatible with 1.0.0. We see something similar with K8s and Mesos. It's pretty common with projects with a fast release cycle. We are running K8s 1.4.x and the latest is 1.11.x. We are not even thinking about upgrading from 1.4.x to 1.11.x, we are just saying let's create a new cluster and move all the workloads gradually. Also, 1.11.x comes a brand new set of container runtimes. With Mesos, we are doing the same, upgrading from 1.3.0 to 1.6.0 by creating a new cluster. Cheers. On Fri, Jun 22, 2018 at 6:13 AM, Hunt, James O <james.o.hunt@intel.com> wrote:
Hi Sebastien,
Thanks very much for kicking off what has turned into a great thread from the humble beginnings of [1] ;)
I've attempted to summarise it to distill the essence as there is a lot of detail:
- There seems to be general consensus that:
- 1.1.0 is better than 2.0.0. - 1.1.0 is also better than 1.0.1.
By changing the minor number users can see it is more than a bugfix release, but we're arguably abusing the naming slightly by not bumping the major number.
That seeming abuse of semver is handled by a "get out of jail free card", namely that we haven't documented the gRPC protocol and hence implicitly haven't agreed (and documented) precisely what API breakage means yet.
We also need to remember to include details of the breakage in the release notes.
- Plan B:
It's worth mentioned that another thought we had yesterday was that we could revert the breaking change [2] to allow us to release a true 1.0.1. That would give us breathing space to investigate the breakage more fully and find a way to avoid it happening in the future (or atleast detect and minimise).
== TODO list from this thread ==
I've raised a bunch of issues here and referenced some existing ones. Please dive in by commenting, emojifying, assigning to yourself (please! ;), etc:
- Jon plans to write up a slide deck on gRPC API evolution best practices (I'd love to see this! ;)
- We need to document the gRPC protocol:
https://github.com/kata-containers/agent/issues/150
- We need to version the gRPC protocol:
https://github.com/kata-containers/kata-containers/issues/17 https://github.com/kata-containers/agent/issues/272
- We need to guarantee persistent state files are consumable by newer component versions:
https://github.com/kata-containers/kata-containers/issues/25
- We need to update our documentation to explain that we cannot handle "live upgrade":
https://github.com/kata-containers/documentation/issues/178
- We need to update our documentation to explain that all component versions must (currently) match for correct operation:
https://github.com/kata-containers/documentation/issues/177
Cheers,
James
[1] - https://github.com/kata-containers/proxy/pull/76 [2] - https://github.com/kata-containers/runtime/pull/357
2018-06-22 10:55 GMT+01:00 Thierry Carrez <thierry@openstack.org>:
zhangwei (CR) wrote:
To summarize: 1) We already break the backward compatibility, and we will break a lot more in near future definitely. Actually in Vancouver, the participants all agree that we can't promise the API won't be broken and current API isn't a stable version. 2) Before we claim that kata can support "live ugrade" and kata is real production ready, I'm fine with the breakage and also fine with 1.0.1 or 1.1.0, maybe latter one looks better. 3) After we claim that kata can support "live upgrade" in future, we should reject any modifications which will break the running workloads, unless this is really inevitable, by then, we need to upgrade kata version from x.0.0 to y.0.0. But I hope our kata developers can understand what a disaster this could be to a cloud provider like us :-(, and I hope this will never happen. 4) Better document that we don't support "live upgrade" yet, and tell users that if you want to upgrade to this new kata-containers version, you must stop all you running kata containers, or there will be anticipated issues.
That summary sounds in line with what Jon said... the protocol between the runtime and the agent is not (yet) part of the external contract for Kata. Once it is (be it by supporting live upgrade or other explicit documentation that you support mixing versions between agent and runtime) then you should avoid breaking that altogether (and bump X number in the case you really need to).
In example 1 from Sebastien (PauseContainer()), I would still recommend you bump Y though, and make it 1.1.0. You're adding a feature and modifying the protocol. I would keep .Z bumps for basic bugfixes that do not introduce incompatible protocol changes at all, if only to develop user confidence that those .Z bumps can be deployed with limited risk.
-- Thierry Carrez (ttx)
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
-- James --- https://katacontainers.io/ | https://github.com/kata-containers <https://github.com/clearcontainers> Open Source Technology Center Intel Corporation (UK) Ltd. - Co. Reg. #1134945 - Pipers Way, Swindon SN3 1RJ.
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
In short, +1 to the above ideas. I think 1.1.0 looks a better choice now. And we'd better have a branch for the bug fixes releases of each feature release, such as branch 1.0.y. I offline discussed with @bergwolf and @WeiZhang555. To make things simple: - The major version change, 1.x to 2.x should include significant feature update, or introduce significant changes for the users. The current situation is not such a big change definitely.- The minor version change, let me talk it later because it is the current debating point.- The bug fix number, such as 1.0.x to 1.0.y, should only include bug fix or security fix, which should not break main data structures, interfaces, and wire protocol. As the protobuf definition had been changed, bump version to 1.1.0 is almost what we have to do. Then we should could define things like the following: - Any feature update that introduce interface or protobuf definition change - should not be put into "stable release branch vM.n.y", and should be merged to master. - should be labeled as "feature update" or similar label, and be listed in the release note later.- We checkpoint the master to make a new stable release tag 'vM.(n+1).0' and branch 'vM.(n+1).y' at the release date, and make packages. - Xu On Sat, Jun 23, 2018 12:59 AM, Ricardo Aravena raravena80@gmail.com wrote: +1 on 1.1.0. Live-upgrade is nice, but lots of people are starting to do more immutable infrastructure. Bring up new workloads on a brand new server/cluster with a newer version. Just need to document that 1.1.0 is not compatible with 1.0.0. We see something similar with K8s and Mesos. It's pretty common with projects with a fast release cycle. We are running K8s 1.4.x and the latest is 1.11.x. We are not even thinking about upgrading from 1.4.x to 1.11.x, we are just saying let's create a new cluster and move all the workloads gradually. Also, 1.11.x comes a brand new set of container runtimes. With Mesos, we are doing the same, upgrading from 1.3.0 to 1.6.0 by creating a new cluster. Cheers. On Fri, Jun 22, 2018 at 6:13 AM, Hunt, James O <james.o.hunt@intel.com> wrote: Hi Sebastien, Thanks very much for kicking off what has turned into a great thread from the humble beginnings of [1] ;) I've attempted to summarise it to distill the essence as there is a lot of detail: - There seems to be general consensus that: - 1.1.0 is better than 2.0.0. - 1.1.0 is also better than 1.0.1. By changing the minor number users can see it is more than a bugfix release, but we're arguably abusing the naming slightly by not bumping the major number. That seeming abuse of semver is handled by a "get out of jail free card", namely that we haven't documented the gRPC protocol and hence implicitly haven't agreed (and documented) precisely what API breakage means yet. We also need to remember to include details of the breakage in the release notes. - Plan B: It's worth mentioned that another thought we had yesterday was that we could revert the breaking change [2] to allow us to release a true 1.0.1. That would give us breathing space to investigate the breakage more fully and find a way to avoid it happening in the future (or atleast detect and minimise). == TODO list from this thread == I've raised a bunch of issues here and referenced some existing ones. Please dive in by commenting, emojifying, assigning to yourself (please! ;), etc: - Jon plans to write up a slide deck on gRPC API evolution best practices (I'd love to see this! ;) - We need to document the gRPC protocol: https://github.com/kata-containers/agent/issues/150 - We need to version the gRPC protocol: https://github.com/kata-containers/kata-containers/issues/17 https://github.com/kata-containers/agent/issues/272 - We need to guarantee persistent state files are consumable by newer component versions: https://github.com/kata-containers/kata-containers/issues/25 - We need to update our documentation to explain that we cannot handle "live upgrade": https://github.com/kata-containers/documentation/issues/178 - We need to update our documentation to explain that all component versions must (currently) match for correct operation: https://github.com/kata-containers/documentation/issues/177 Cheers, James [1] - https://github.com/kata-containers/proxy/pull/76 [2] -https://github.com/kata-containers/runtime/pull/357 2018-06-22 10:55 GMT+01:00 Thierry Carrez <thierry@openstack.org>: zhangwei (CR) wrote: To summarize: 1) We already break the backward compatibility, and we will break a lot more in near future definitely. Actually in Vancouver, the participants all agree that we can't promise the API won't be broken and current API isn't a stable version. 2) Before we claim that kata can support "live ugrade" and kata is real production ready, I'm fine with the breakage and also fine with 1.0.1 or 1.1.0, maybe latter one looks better. 3) After we claim that kata can support "live upgrade" in future, we should reject any modifications which will break the running workloads, unless this is really inevitable, by then, we need to upgrade kata version from x.0.0 to y.0.0. But I hope our kata developers can understand what a disaster this could be to a cloud provider like us :-(, and I hope this will never happen. 4) Better document that we don't support "live upgrade" yet, and tell users that if you want to upgrade to this new kata-containers version, you must stop all you running kata containers, or there will be anticipated issues. That summary sounds in line with what Jon said... the protocol between the runtime and the agent is not (yet) part of the external contract for Kata. Once it is (be it by supporting live upgrade or other explicit documentation that you support mixing versions between agent and runtime) then you should avoid breaking that altogether (and bump X number in the case you really need to). In example 1 from Sebastien (PauseContainer()), I would still recommend you bump Y though, and make it 1.1.0. You're adding a feature and modifying the protocol. I would keep .Z bumps for basic bugfixes that do not introduce incompatible protocol changes at all, if only to develop user confidence that those .Z bumps can be deployed with limited risk. -- Thierry Carrez (ttx) _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev -- James ---https://katacontainers.io/ | https://github.com/kata-containers Open Source Technology Center Intel Corporation (UK) Ltd. - Co. Reg. #1134945 - Pipers Way, Swindon SN3 1RJ. _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev -- Xu WangCTO & Cofounder, Hypergithub/twitter/wechat: @gnawuxhttp://hyper.sh Hyper_: Make VM run like container
participants (11)
-
Boeuf, Sebastien
-
Ernst, Eric
-
Fox, Kevin M
-
Hunt, James O
-
Jon Olson
-
Ricardo Aravena
-
Shinde, Archana M
-
Tao Peng
-
Thierry Carrez
-
Xu Wang
-
zhangwei (CR)