Open Stack

Chris,

> By the way, if my understanding is correct, the correct response to my question is G.

Yes in your example I would pick G. However, having a VM always hotplug 4 vCPU's by default seems kind of high but in your example G is what I would do.

> I would say, to the contrary, that the expectation in that case is that you get "as much as you can", since this is the runc behaviour. In other words, if I compile my Linux kernel and request 5 CPUs, I'm happy if I get 5 CPUs, and I'm happier if I get 96. However, I cannot be satisfied to get only one.

I tried to look at this from different points of view.  If you don't own the cluster and request 5 CPU's and set no limit you hope you can get more but no guarantee. As a developer I probably would never set a limit and only set my minimum request always hoping for more. The sys admin might enforce limits. I wonder how this really works in practice. With Kata I personally feel it is ok if the request becomes the limit even if runc can run unbounded. In the runc case you get at least 5 CPU's and perhaps the entire system at a slow time but with Kata you would get at least 5 vCPU + some fragment of a vCPU unused by the VM or agent and never have the opportunity to burst higher than your request. I am assuming the default vCPU is 1 and the behavior is that Kata would hotplug at least the requested number of vCPU's and the original 1 vCPU would have a little spare processing power.  I should note that runc running unbounded in multi socketed systems can sometimes be a lot slower than if you just ran on a limited amount of CPUs confined to a NUMA region. I thought I once found a workload where Kata with 1/4 the CPU's ran 2x faster than and equivalent runc. Once I reduced the amount of processors to stay within a NUMA boundary the runc ran much much faster.   That was a lot words for why I think having different behavior between kata and runc in the unbounded "no limit" case is ok as long as the minimum expectation of the request is met.

Thanks
Eric


-----Original Message-----
From: Christophe de Dinechin <dinechin at redhat.com> 
Sent: Wednesday, July 7, 2021 3:34 AM
To: Adams, Eric <eric.adams at intel.com>
Cc: Eric Ernst <eric.g.ernst at gmail.com>; kata-dev at lists.katacontainers.io
Subject: Re: [kata-dev] Performance isolation: expectations for number of CPUs


On 2021-07-07 at 01:31 UTC, "Adams, Eric" <eric.adams at intel.com> wrote...
> Christophe,
>
> I re-read through the limits, requests, resource quotas, and pod 
> overhead Kubernetes page and imagined what a dev running container 
> workloads would expect and also considered what a sys admin would 
> worry about.  I had some time this afternoon so I tried some 
> experiments to better understand how the requests/limits actually 
> works. I've been meaning to dig into this for my own understanding 
> from the user perspective.  After thinking through these different 
> scenarios, one thing I would consider changing is making the 
> configuration.toml default vCPU setting a podman or docker only 
> setting. For Kubernetes I would just ignore that field and hotplug 
> CPU's based on requests and limits set in the Kubernetes yaml files 
> with a default of 1vCPU at minimum. There might be a good reason for Kubernetes to allow someone to set the default base vCPU higher than 1, but I can't think of a scenario now.

The default vCPU size is orthogonal, and somewhat related to pod overhead.

Additional host CPUs could be beneficial to handle I/Os (they cost more in a VM typically) or to give more scheduling freedom. We discussed such a scenario in the performance isolation use case meeting when the agent itself starts using a lot of CPU e.g. to process container console output. That is probably not the only case.

>
> Here is what I observe. I was going back and forth on this email all day so I hope I didn't make a typo mistake on what I observed.
>
> For normal Kubernetes with no Kata.
> 1) Limit is the max amount of CPUs that you will get, and the max 
> performance your workload can achieve. Request is the minimum amount 
> of CPU's that you are guaranteed to receive. The Request and Limits 
> are the sum of all container requests/limits in a pod.
> 2) If you set a request so big (say 100 cpus) then the pod gets stuck in pending which is expected.
> 3) If you set a request but no limit and no LimitRange is set for the 
> namespace you are in then you get all the CPU resource of the cluster. 
> I tried this with compiling the linux kernel and indeed I did max out 
> all 96 cores when using a request of just 5 with no limit.

This is consistent with my understanding. For a normal runtime, the CPU request is only used to schedule on a sufficiently large node (and presumably one where there is enough leftover CPU capacity, i.e. I assume that if you can't fit four cpu requests of 32 on a 96-CPU host.

By the way, if my understanding is correct, the correct response to my question is G.


>
> For Kata 2.1.1 this is what I observe
> 1) If you set no requests and no limits than no extra vCPU's is hot 
> plugged. I set a request of 5 with no limit with runc and it used all 
> 96 cores. With Kata I only got 1 total vCPU for the pod.  I don't 
> think this is the same as issue 2130 below but is what the user 
> reported in 2071.  Since the request is a minimum I think it should add up all the requests and hotplug that as a minimum.

Yes, this is 2071.


> 2) I also observed that Pod Overhead isn't used in the calculation for 
> hotplugged CPU's.

That seems correct. The pod overhead accounts for the overhead "outside" the VM, i.e. virtiofsd, qemu's own memory needs, the extra cost of doing I/Os, etc. So this is additional resources the host needs, not the VM.


> It seems that in the scenario where you have a Request and no limit 
> you would expect to get at least enough vCPU to ensure the workload 
> meets the request.  If you don't specify any request or limit you get 
> one total vCPU. If you were to compare this to a runc pod it would get 
> at least 1 CPU but more than likely get a lot more CPU. I doubt this 
> happens much because probably most people enforce a limit if one is 
> not set which would cover the Kata case.

The problem is that we get a really bad output if the limit is not set. And having to set a limit to improve performance is counter-intuitive.


> 3) In my cluster, the Pod Overhead was 250m CPU for the kata-qemu 
> namespace.  If I request/limit 9 CPU for one container then I get 10 
> in the pod.  If I request/limit 9.25 CPU for one container then I get 
> 11 in the pod. If I request/limit 9.75 CPU for one container I also 
> get 11 in the pod. Finally, if I request/limit 9.90 CPU for one 
> container then I get 11 in the pod.  It seems like in the case where 
> you request 9.25 vCPU and Pod Overhead is 0.25 vCPU then it seems you 
> could get away with only having 10vCPU in the pod. That probably isn't a huge overhead to have one extra vCPU hot plugged.

All this is correct if you consider the interpretation of overhead I gave you above. We simply round up to the next higher number of VCPUs, and add one for the agent.


>
> I should probably do the below in a table format. Right now I don't 
> see that pod overhead is used in the hot plug calculation. I don't 
> know that it matters since you get 1 vCPU by default and any 
> additional limits are hotplugged in. However, in the case where no 
> requests or limits are set then 1vCPU is used for the pod. In that 
> scenario your workload would have an estimated max of 750m CPU
>
> Different scenarios to consider
> 1) Pod with multiple containers with request/limit set for everything
>
> In this scenario you would add up all the limits and hotplug at least 
> that many vCPU's. That seems to work already for Kata. If I start two 
> containers each with a limit of 3 then I end up with 7 vCPU's in the pod.

That is indeed correct, except that it takes the value of "limit" and not the value of "request" to set the number of VCPUs. In a sense, it is a good thing, since it means if you have a request of 5 and a limit of 7, you get 7 CPUs.

>
> 2) Pod with multiple containers with only request set
>
> In this case I would hotplug the sum of all CPU requests. In the runc 
> scenario it would use the entire system, but with Kata the request 
> would become the limit. I feel that if someone set a request with no 
> limit then there is no expectation for it to go higher than the 
> request. This warrants further discussion though.

I would say, to the contrary, that the expectation in that case is that you get "as much as you can", since this is the runc behaviour. In other words, if I compile my Linux kernel and request 5 CPUs, I'm happy if I get 5 CPUs, and I'm happier if I get 96. However, I cannot be satisfied to get only one.

The problem is that as far as understand today, we don't get the required information to be able to do the right thing here (except as a side effect through some annotation for CRI-O). Apparently, k8s assumes that if it does not need to pass the information to runc, then it does not need to pass the information at all.

>
> 3) Pod with multiple containers with only limit set
>
> This works already how I would expect. Kata adds all the limits and 
> hotplugs that in.

Yes.

>
> 4) Pod with multiple containers with some limits set in one container 
> but requests set in another container
>
> This one is tricky. I think inside the pod there are cgroups where you 
> could limit one container to a slice of CPU/memory but have not looked 
> into that. Logically I would expect that you would take all the 
> containers with limits and add that number of vCPU's to all the 
> containers that only have requests.  Ex: Container1 2 
> requests/unspecified limit Container2 unspecified
> request/3 limit For this I would hotplug 5 vCPU's and ensure container 
> 1 gets at least 2 vCPU in the pod. Container 2 would get whatever it 
> gets when container 1 isn't busy.
>
> 5) Pod with multiple container with limits on one container but 
> requests/limits not set on another
>
> Another tricky one. For this one what makes sense to me is to add up 
> the limits and hotplug that number of vCPUs. That seems to be the case 
> now. The container with nothing set would get whatever it gets and 
> that would likely be less than a vCPU when container1 is busy.
>
> 6) Pod with multiple containers with no request/limits set
>
> In this case nothing is hot plugged.
>
> At the very least not having enough vCPU's for a container that 
> requested a certain minimum amount is a bug.

Yes, at least relative to runc.

>
> Thanks
> Eric
>
> -----Original Message-----
> From: Christophe de Dinechin <dinechin at redhat.com>
> Sent: Thursday, July 1, 2021 8:02 AM
> To: Eric Ernst <eric.g.ernst at gmail.com>
> Cc: kata-dev <kata-dev at lists.katacontainers.io>
> Subject: Re: [kata-dev] Performance isolation: expectations for number 
> of CPUs
>
>
>
>> On 1 Jul 2021, at 16:55, Eric Ernst <eric.g.ernst at gmail.com> wrote:
>>
>> In this example, you’ll want to clarify what the CPUs requested means. I’ll assume limits=request, and that you’re referring to a kubernetes pod.
>
> Indeed, see the linked issues for examples. Also, ideally, we would like this to work with request but no limit.
>
>> Based on that I’d expect 12. I would not recommend four default vCPUs though.
>
> The 3, 4 and 5 were just examples to get different numbers as an 
> output. I chose
>> four for the VM initial VCPUs to illustrate that we may have a 
>> possible workaround for request.vcpu=4 not doing anything.
>
>>
>> Eric
>>
>> Sent from my iPhone
>>
>>> On Jul 1, 2021, at 3:54 AM, Christophe de Dinechin <dinechin at redhat.com> wrote:
>>>
>>> An interesting question arose about the number of CPUs we want to 
>>> get in the VM, notably in the context of
>>> https://github.com/kata-containers/kata-containers/issues/2071 as 
>>> well as regarding 
>>> https://github.com/kata-containers/kata-containers/pull/2131, a fix for https://github.com/kata-containers/kata-containers/issues/2130.
>>>
>>> Let's say that we have two containers A and B requesting 5 and 3 CPUs respectively. How many CPUs should we get in the VM? Let us assume that the default number of VCPUs is 4.
>>>
>>> Possible answers:
>>>
>>> A) 4 (default number of VCPUs), because the current OCI spec does 
>>> not give us information about the number of CPUs. That's how I 
>>> interpret Julio's answer, https://github.com/kata-containers/kata-containers/issues/2071#issuecomment-865034753.
>>>
>>> B) 5 (maximum request). This seems to be more or less how the Rust agent behaves today, making sure that each time there is a request, we online at least that many CPUs.
>>>
>>> C) 8 (sum of requests for all containers). If the two containers 
>>> request CPUs, they have good reasons to, so we should honor both 
>>> requests independently. This seems to be what the runtime has in 
>>> mind, since it hotplugs the new CPUs and the struct VM field is called "cpusDelta"
>>>
>>> D) 6 (maximum request, plus one for the agent). During the performance isolation meetings, we seem to have shifted towards the idea that the agent should get a dedicated CPU.
>>>
>>> E) 9 (maximum request, plus one for the agent)
>>>
>>> F) 9 (maximum request plus the four original)
>>>
>>> G) 12 (sum of requests plus the four original)
>>>
>>> H) 42 (the correct answer in most cases)
>>>
>>> What do you think?
>>>
>>>
>>> _______________________________________________
>>> kata-dev mailing list
>>> kata-dev at lists.katacontainers.io
>>> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
>>
>
>
> _______________________________________________
> kata-dev mailing list
> kata-dev at lists.katacontainers.io
> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
> _______________________________________________
> kata-dev mailing list
> kata-dev at lists.katacontainers.io
> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev


--
Cheers,
Christophe de Dinechin (IRC c3d)