Open Stack

On 2021-07-07 at 01:31 UTC, "Adams, Eric" <eric.adams at intel.com> wrote...
> Christophe,
>
> I re-read through the limits, requests, resource quotas, and pod overhead
> Kubernetes page and imagined what a dev running container workloads would expect
> and also considered what a sys admin would worry about.  I had some time this
> afternoon so I tried some experiments to better understand how the
> requests/limits actually works. I've been meaning to dig into this for my own
> understanding from the user perspective.  After thinking through these different
> scenarios, one thing I would consider changing is making the configuration.toml
> default vCPU setting a podman or docker only setting. For Kubernetes I would
> just ignore that field and hotplug CPU's based on requests and limits set in the
> Kubernetes yaml files with a default of 1vCPU at minimum. There might be a good
> reason for Kubernetes to allow someone to set the default base vCPU higher than
> 1, but I can't think of a scenario now.

The default vCPU size is orthogonal, and somewhat related to pod overhead.

Additional host CPUs could be beneficial to handle I/Os (they cost more in a
VM typically) or to give more scheduling freedom. We discussed such a
scenario in the performance isolation use case meeting when the agent itself
starts using a lot of CPU e.g. to process container console output. That is
probably not the only case.

>
> Here is what I observe. I was going back and forth on this email all day so I hope I didn't make a typo mistake on what I observed.
>
> For normal Kubernetes with no Kata.
> 1) Limit is the max amount of CPUs that you will get, and the max performance
> your workload can achieve. Request is the minimum amount of CPU's that you are
> guaranteed to receive. The Request and Limits are the sum of all container
> requests/limits in a pod.
> 2) If you set a request so big (say 100 cpus) then the pod gets stuck in pending which is expected.
> 3) If you set a request but no limit and no LimitRange is set for the namespace
> you are in then you get all the CPU resource of the cluster. I tried this with
> compiling the linux kernel and indeed I did max out all 96 cores when using a
> request of just 5 with no limit.

This is consistent with my understanding. For a normal runtime, the CPU
request is only used to schedule on a sufficiently large node (and
presumably one where there is enough leftover CPU capacity, i.e. I assume
that if you can't fit four cpu requests of 32 on a 96-CPU host.

By the way, if my understanding is correct, the correct response to my
question is G.


>
> For Kata 2.1.1 this is what I observe
> 1) If you set no requests and no limits than no extra vCPU's is hot plugged. I
> set a request of 5 with no limit with runc and it used all 96 cores. With Kata I
> only got 1 total vCPU for the pod.  I don't think this is the same as issue 2130
> below but is what the user reported in 2071.  Since the request is a minimum I
> think it should add up all the requests and hotplug that as a minimum.

Yes, this is 2071.


> 2) I also observed that Pod Overhead isn't used in the calculation for
> hotplugged CPU's.

That seems correct. The pod overhead accounts for the overhead "outside" the
VM, i.e. virtiofsd, qemu's own memory needs, the extra cost of doing I/Os,
etc. So this is additional resources the host needs, not the VM.


> It seems that in the scenario where you have a Request and no
> limit you would expect to get at least enough vCPU to ensure the workload meets
> the request.  If you don't specify any request or limit you get one total
> vCPU. If you were to compare this to a runc pod it would get at least 1 CPU but
> more than likely get a lot more CPU. I doubt this happens much because probably
> most people enforce a limit if one is not set which would cover the Kata
> case.

The problem is that we get a really bad output if the limit is not set. And
having to set a limit to improve performance is counter-intuitive.


> 3) In my cluster, the Pod Overhead was 250m CPU for the kata-qemu namespace.  If
> I request/limit 9 CPU for one container then I get 10 in the pod.  If I
> request/limit 9.25 CPU for one container then I get 11 in the pod. If I
> request/limit 9.75 CPU for one container I also get 11 in the pod. Finally, if I
> request/limit 9.90 CPU for one container then I get 11 in the pod.  It seems
> like in the case where you request 9.25 vCPU and Pod Overhead is 0.25 vCPU then
> it seems you could get away with only having 10vCPU in the pod. That probably
> isn't a huge overhead to have one extra vCPU hot plugged.

All this is correct if you consider the interpretation of overhead I gave
you above. We simply round up to the next higher number of VCPUs, and add
one for the agent.


>
> I should probably do the below in a table format. Right now I don't see that pod
> overhead is used in the hot plug calculation. I don't know that it matters since
> you get 1 vCPU by default and any additional limits are hotplugged in. However,
> in the case where no requests or limits are set then 1vCPU is used for the
> pod. In that scenario your workload would have an estimated max of 750m CPU
>
> Different scenarios to consider
> 1) Pod with multiple containers with request/limit set for everything
>
> In this scenario you would add up all the limits and hotplug at least that many
> vCPU's. That seems to work already for Kata. If I start two containers each with
> a limit of 3 then I end up with 7 vCPU's in the pod.

That is indeed correct, except that it takes the value of "limit" and not
the value of "request" to set the number of VCPUs. In a sense, it is a good
thing, since it means if you have a request of 5 and a limit of 7, you get 7
CPUs.

>
> 2) Pod with multiple containers with only request set
>
> In this case I would hotplug the sum of all CPU requests. In the runc scenario
> it would use the entire system, but with Kata the request would become the
> limit. I feel that if someone set a request with no limit then there is no
> expectation for it to go higher than the request. This warrants further
> discussion though.

I would say, to the contrary, that the expectation in that case is that you
get "as much as you can", since this is the runc behaviour. In other words,
if I compile my Linux kernel and request 5 CPUs, I'm happy if I get 5 CPUs,
and I'm happier if I get 96. However, I cannot be satisfied to get only one.

The problem is that as far as understand today, we don't get the required
information to be able to do the right thing here (except as a side effect
through some annotation for CRI-O). Apparently, k8s assumes that if it does
not need to pass the information to runc, then it does not need to pass the
information at all.

>
> 3) Pod with multiple containers with only limit set
>
> This works already how I would expect. Kata adds all the limits and
> hotplugs that in.

Yes.

>
> 4) Pod with multiple containers with some limits set in one container but requests set in another container
>
> This one is tricky. I think inside the pod there are cgroups where you could
> limit one container to a slice of CPU/memory but have not looked into
> that. Logically I would expect that you would take all the containers with
> limits and add that number of vCPU's to all the containers that only have
> requests.  Ex: Container1 2 requests/unspecified limit Container2 unspecified
> request/3 limit For this I would hotplug 5 vCPU's and ensure container 1 gets at
> least 2 vCPU in the pod. Container 2 would get whatever it gets when container 1
> isn't busy.
>
> 5) Pod with multiple container with limits on one container but requests/limits not set on another
>
> Another tricky one. For this one what makes sense to me is to add up the limits
> and hotplug that number of vCPUs. That seems to be the case now. The container
> with nothing set would get whatever it gets and that would likely be less than a
> vCPU when container1 is busy.
>
> 6) Pod with multiple containers with no request/limits set
>
> In this case nothing is hot plugged.
>
> At the very least not having enough vCPU's for a container that requested
> a certain minimum amount is a bug.

Yes, at least relative to runc.

>
> Thanks
> Eric
>
> -----Original Message-----
> From: Christophe de Dinechin <dinechin at redhat.com>
> Sent: Thursday, July 1, 2021 8:02 AM
> To: Eric Ernst <eric.g.ernst at gmail.com>
> Cc: kata-dev <kata-dev at lists.katacontainers.io>
> Subject: Re: [kata-dev] Performance isolation: expectations for number of CPUs
>
>
>
>> On 1 Jul 2021, at 16:55, Eric Ernst <eric.g.ernst at gmail.com> wrote:
>>
>> In this example, you’ll want to clarify what the CPUs requested means. I’ll assume limits=request, and that you’re referring to a kubernetes pod.
>
> Indeed, see the linked issues for examples. Also, ideally, we would like this to work with request but no limit.
>
>> Based on that I’d expect 12. I would not recommend four default vCPUs though.
>
> The 3, 4 and 5 were just examples to get different numbers as an output. I chose
>> four for the VM initial VCPUs to illustrate that we may have a possible
>> workaround for request.vcpu=4 not doing anything.
>
>>
>> Eric
>>
>> Sent from my iPhone
>>
>>> On Jul 1, 2021, at 3:54 AM, Christophe de Dinechin <dinechin at redhat.com> wrote:
>>>
>>> An interesting question arose about the number of CPUs we want to get in the
>>> VM, notably in the context of
>>> https://github.com/kata-containers/kata-containers/issues/2071 as well as
>>> regarding https://github.com/kata-containers/kata-containers/pull/2131, a fix
>>> for https://github.com/kata-containers/kata-containers/issues/2130.
>>>
>>> Let's say that we have two containers A and B requesting 5 and 3 CPUs respectively. How many CPUs should we get in the VM? Let us assume that the default number of VCPUs is 4.
>>>
>>> Possible answers:
>>>
>>> A) 4 (default number of VCPUs), because the current OCI spec does not give us
>>> information about the number of CPUs. That's how I interpret Julio's answer,
>>> https://github.com/kata-containers/kata-containers/issues/2071#issuecomment-865034753.
>>>
>>> B) 5 (maximum request). This seems to be more or less how the Rust agent behaves today, making sure that each time there is a request, we online at least that many CPUs.
>>>
>>> C) 8 (sum of requests for all containers). If the two containers request
>>> CPUs, they have good reasons to, so we should honor both requests
>>> independently. This seems to be what the runtime has in mind, since it
>>> hotplugs the new CPUs and the struct VM field is called "cpusDelta"
>>>
>>> D) 6 (maximum request, plus one for the agent). During the performance isolation meetings, we seem to have shifted towards the idea that the agent should get a dedicated CPU.
>>>
>>> E) 9 (maximum request, plus one for the agent)
>>>
>>> F) 9 (maximum request plus the four original)
>>>
>>> G) 12 (sum of requests plus the four original)
>>>
>>> H) 42 (the correct answer in most cases)
>>>
>>> What do you think?
>>>
>>>
>>> _______________________________________________
>>> kata-dev mailing list
>>> kata-dev at lists.katacontainers.io
>>> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
>>
>
>
> _______________________________________________
> kata-dev mailing list
> kata-dev at lists.katacontainers.io
> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
> _______________________________________________
> kata-dev mailing list
> kata-dev at lists.katacontainers.io
> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev


--
Cheers,
Christophe de Dinechin (IRC c3d)