Open Stack

Actually I also mentioned this in Vancouver, in my opinion, a breakage between kata-agent and kata-runtime should always be considered as a backward compatibility breakage.
This breakage is a "gap" between "project" and "product" for kata-containers,  I'll elaborate why here.

Starting from Huawei's requirement for a mature cloud product in use of kata, we have SLA with our customers, which means we can't shutdown customers' service while we are updating 
Kata components, this feature is named as "live-update", so running kata-runtime and agent of different versions will very likely happen:

1). New runtime + old agent: updating kata-runtime when VM+old agent is running, kata-runtime shouldn't issue a command which will crash the agent.
2). old runtime + new agent: rollback when new kata version has issues, in this case, some service could be started already, new agent should always handle commands from old runtime.

So what will happen if we miss 1) and 2)? We need to shutdown user's running workload whenever we want to upgrade/downgrade the kata-components, that will make our SLA a joke.
(Of course we can also choose to send them notification and let users shutdown their workload by themselves, but we definitely hope to do better and go further. )

So to guarantee the "live-upgrade" ability of kata-components(meas install kata rpm packages while workloads are still running), what we need to do for these components are:
1) kata-runtime: 
	A. issue "versioned" command to kata-agent, can always communicate correctly with old kata-agent.   (MUST)
	B. disk persist data should be "versioned", kata-runtime can always handle old "version" of persist data to restore sandbox/container struct from disk to memory. (MUST)
2) kata-agent:  protocol needs to be versioned, can always handle commands from old  kata-runtime. ("versioned" may be achieved by leveraging protobuf) (MUST)
3) kata-shim/kata-proxy:  daemon process, no need to shut down while updating  kata rpm package. So I don't see a problem currently, need to guarantee interact between kata-runtime and shim/proxy. (MUST)
4) qemu: 
	A. current status: NO WAY to upgrade now. running workload must be shutdown before installing newer version of qemu rpm package. (IMPOSSIBLE)
	B. In future: qemu live-migration, live-replacement, live-patch etc. (BETTER HAVE)
5) guest kernel:  
	A. current status: after install kata rpm package with newer VM image, old workloads can keep running with old kernel, newly started workload will use new VM kernel. It's fine. (ALREADY HAVE)
	B. in future: live patch. (BETTER HAVE)

To summarize:
1) We  already break the backward compatibility, and we will break a lot more in near future definitely.  Actually in Vancouver, the participants all agree that we can't promise the API won't be broken and current API isn't a stable version.
2) Before we claim that kata can support "live ugrade" and kata is real production ready, I'm fine with the breakage and also fine with 1.0.1 or 1.1.0, maybe latter one looks better.
3) After we claim that kata can support "live upgrade" in future, we should reject any modifications which will break the running workloads, unless this is really inevitable, by then, we need to upgrade kata version from x.0.0 to y.0.0.
But I hope our kata developers can understand what a disaster this could be to a cloud provider like us :-(, and I hope this will never happen.
4) Better document that we don't support "live upgrade" yet, and tell users that if you want to upgrade to this new kata-containers version, you must stop all you running kata containers, or there will be anticipated issues.


BR,
Wei
-----邮件原件-----
发件人: Tao Peng [mailto:bergwolf at hyper.sh] 
发送时间: 2018年6月22日 8:26
收件人: Boeuf, Sebastien <sebastien.boeuf at intel.com>
抄送: kata-dev at lists.katacontainers.io
主题: Re: [kata-dev] What to do about release versioning ?

On Fri, Jun 22, 2018 at 2:53 AM, Boeuf, Sebastien <sebastien.boeuf at intel.com> wrote:
> Hi folks,
>
> This morning we wanted to release 1.0.1 but held on because of some 
> discussions about the semver semantic not being followed with this 
> 1.0.1 version.
> The problem here, we have some "sort of" backward compatibility 
> breakage between the runtime and the agent. And for any non compatible 
> release, semver suggest that the new release should bump to the next X 
> version (based on X.Y.Z notation).
>
> But before we can answer this, we have to figure out what is that we 
> consider a "breakage".
>
> Let me try to explain the different possibility here, starting from an 
> environment where you installed Kata 1.0.0 from packages:
>
> First, let's say we introduced a new PauseContainer() command to our 
> agent protocol, this command will have one commit on the agent repo, 
> updating the gRPC protocol and implementing the new feature. This will 
> be followed by a new patch on the runtime to use this new feature, 
> which means we'll have to revendor the new protocol and implement the runtime to rely on it.
>
> Now, let's say we bump both runtime and agent repos including those 
> patches to the version 1.0.1, here is the issue we will run into:
> - The user with Kata 1.0.0 already installed on his system might 
> update the kata-runtime package without updating the VM image 
> containing the agent. And from his perspective, this is fine because 
> he knows about semver semantics and thinks that a kata-runtime 1.0.1 
> might still work with the VM image container a kata-agent 1.0.0. 
> Unfortunately, this won't work because the kata-runtime will try to 
> call into the new command PauseContainer() that had been introduced, 
> but the agent will have no idea about what this is, and the gRPC will return an error saying this is not a supported/known command.
>
> I have a second example, sorry this email is getting long but we have 
> to talk about all cases here... Let's get back to 1.0.0 and say that 
> some changes on both kata-runtime and kata-agent are modifying an 
> existing command CreateContainer(). In this case, this is a more 
> logical breakage since moving one or the other component to the newest 
> version without bumping the other (either kata-runtime 1.0.0 and 
> kata-agent 1.0.1, or kata-runtime 1.0.1 and kata-agent 1.0.0), will end up in functional issues.
>
> The second example is what I would call an obvious backward 
> compatibility breakage, but the first one looks still valid to be taken into account.
>
> So now, to summarize, we have to agree on what we call a non 
> compatible version that will be translated into a bump to the next X version.
> And I can see happening a lot of breakage in the future if we include 
> both cases in what is considered as a "breakage", meaning we might end 
> up on version 100.y.z pretty quickly.
>
If we follow the general guide of adding new grpc fields but do not change or delete existing ones [1], we can be sure that v1.0.0 runtime works with the new v1.0.x agent.

OTOH, with any agent protocol extension (new fields or new RPC requests), we cannot guarantee that the new v1.0.x runtime works with
v1.0.0 agent, because the runtime will send RPC requests containing newly added fields/RPCs.

So the question is what kind of compatibility we are looking at here?
1). old runtime + new agent
2). new runtime + old agent

If we want to keep both 1 and 2, I think we will have to bump up the major release version number every time we change the agent protocol.

If we just want to keep 1, most agent protocol change will not break it as long as we do not change or delete existing fields.

I am fine with either one but we need to get consensus about the kind of compatibility we want to maintain and stick to it.

ref:
[1] https://developers.google.com/protocol-buffers/docs/overview

Cheers,
Tao

--
bergwolf at hyper.sh

_______________________________________________
kata-dev mailing list
kata-dev at lists.katacontainers.io
http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev