Open Stack

On 2021-07-06 at 19:04 -07, Eric Ernst <eric.g.ernst at gmail.com> wrote...  >
Thanks for the explanation Eric.
>
> The high level summary on why you're seeing this: Kata works well, and
>  correctly imo, when a pod is 'guaranteed', as well as 'burstable' if a
>  limit is set. We do not 'properly' support bestEffort or unbound
>  burstable (today).
>
> In the configurations I have used of Kubernetes, a default limit is
> usually applied (or we validate that a limit is set) at pod admission
> time, so all is well behaved. This is what I'd recommend for other users
> as well.
>
> Having said that, I *think* if we can sanely and safely deduce that a
> limit was not set, we can and should size the VM accordingly (# vCPU
> matching that of the host).

I was about to suggest we do that, with a minor additional suggestion, that
we add a configuration option indicating the maximum number of CPUs we
allocate in that case, possibly as a fraction of host CPUs. For example, we
could limit the number of CPUs to 25% of the host. I'm trying to come up
with an idea that would work well with mixed nodes of different sizes.


> AFAIU we should be able to identify the
> scenario if, for a given container in the pod: unbound := cfs quota == -1
> && cpusets are not specified
>
> This function would need some refactoring, and a lot of unit tests:
> https://github.com/kata-containers/kata-containers/blob/7d37fbfdfba05be660b11f2b4d6545dfdfbecd63/src/runtime/virtcontainers/sandbox.go#L1884
>
> --Eric
>
> On Tue, Jul 6, 2021 at 6:31 PM Adams, Eric <eric.adams at intel.com> wrote:
>
>  Christophe,
>
>  I re-read through the limits, requests, resource quotas, and pod overhead Kubernetes page and imagined what a dev running container workloads would expect and also considered what a sys admin would worry about.  I had some
>  time this afternoon so I tried some experiments to better understand how the requests/limits actually works. I've been meaning to dig into this for my own understanding from the user perspective.  After thinking through these different
>  scenarios, one thing I would consider changing  is making the configuration.toml default vCPU setting a podman or docker only setting. For Kubernetes I would just ignore that field and hotplug CPU's based on requests and limits set in
>  the Kubernetes yaml files with a default of 1vCPU at minimum. There might be a good reason for Kubernetes to allow someone to set the default base vCPU higher than 1, but I can't think of a scenario now.
>
>  Here is what I observe. I was going back and forth on this email all day so I hope I didn't make a typo mistake on what I observed.
>
>  For normal Kubernetes with no Kata.
>  1) Limit is the max amount of CPUs that you will get, and the max performance your workload can achieve. Request is the minimum amount of CPU's that you are guaranteed to receive. The Request and Limits are the sum of all
>  container requests/limits in a pod.
>  2) If you set a request so big (say 100 cpus) then the pod gets stuck in pending which is expected.
>  3) If you set a request but no limit and no LimitRange is set for the namespace you are in then you get all the CPU resource of the cluster. I tried this with compiling the linux kernel and indeed I did max out all 96 cores when using a
>  request of just 5 with no limit.
>
>  For Kata 2.1.1 this is what I observe
>  1) If you set no requests and no limits than no extra vCPU's is hot plugged. I set a request of 5 with no limit with runc and it used all 96 cores. With Kata I only got 1 total vCPU for the pod.  I don't think this is the same as issue 2130 below
>  but is what the user reported in 2071.   Since the request is a minimum I think it should add up all the requests and hotplug that as a minimum.
>  2) I also observed that Pod Overhead isn't used in the calculation for hotplugged CPU's.  It seems that in the scenario where you have a Request and no limit you would expect to get at least enough vCPU to ensure the workload meets
>  the request.  If you don't specify any request or limit you get one total vCPU. If you were to compare this to a runc pod it would get at least 1 CPU but more than likely get a lot more CPU. I doubt this happens much because probably
>  most people enforce a limit if one is not set which would cover the Kata case.
>  3)  In my cluster, the Pod Overhead was 250m CPU for the kata-qemu namespace.  If I request/limit 9 CPU for one container then I get 10 in the pod.  If I request/limit 9.25 CPU for one container then I get 11 in the pod. If I request/limit
>  9.75 CPU for one container I also get 11 in the pod. Finally, if I request/limit 9.90 CPU for one container then I get 11 in the pod.    It seems like in the case where you request 9.25 vCPU and Pod Overhead is 0.25 vCPU then it seems
>  you could get away with only having 10vCPU in the pod. That probably isn't a huge overhead to have one extra vCPU hot plugged.
>
>  I should probably do the below in a table format. Right now I don't see that pod overhead is used in the hot plug calculation. I don't know that it matters since you get 1 vCPU by default and any additional limits are hotplugged in.
>  However, in the case where no requests or limits are set then 1vCPU is used for the pod. In that scenario your workload would have an estimated max of 750m CPU
>
>  Different scenarios to consider
>  1) Pod with multiple containers with request/limit set for everything
>
>  In this scenario you would add up all the limits and hotplug at least that many vCPU's. That seems to work already for Kata. If I start two containers each with a limit of 3 then I end up with 7 vCPU's in the pod.
>
>  2) Pod with multiple containers with only request set
>
>  In this case I would hotplug the sum of all CPU requests. In the runc scenario it would use the entire system, but with Kata the request would become the limit. I feel that if someone set a request with no limit then there is no expectation
>  for it to go higher than the request. This warrants further discussion though.
>
>  3) Pod with multiple containers with only limit set
>
>  This works already how I would expect. Kata adds all the limits and hotplugs that in.
>
>  4) Pod with multiple containers with some limits set in one container but requests set in another container
>
>  This one is tricky. I think inside the pod there are cgroups where you could limit one container to a slice of CPU/memory but have not looked into that. Logically I would expect that you would take all the containers with limits and add
>  that number of vCPU's to all the containers that only have requests.  Ex: Container1  2 requests/unspecified limit   Container2 unspecified request/3 limit   For this I would hotplug 5 vCPU's and ensure container 1 gets at least 2 vCPU in
>  the pod. Container 2 would get whatever it gets when container 1 isn't busy.
>
>  5) Pod with multiple container with limits on one container but requests/limits not set on another
>
>  Another tricky one. For this one what makes sense to me is to add up the limits and hotplug that number of vCPUs. That seems to be the case now. The container with nothing set would get whatever it gets and that would likely be less
>  than a vCPU when container1 is busy.
>
>  6) Pod with multiple containers with no request/limits set
>
>  In this case nothing is hot plugged.
>
>  At the very least not having enough vCPU's for a container that requested a certain minimum amount is a bug.
>
>  Thanks
>  Eric
>
>  -----Original Message-----
>  From: Christophe de Dinechin <dinechin at redhat.com>
>  Sent: Thursday, July 1, 2021 8:02 AM
>  To: Eric Ernst <eric.g.ernst at gmail.com>
>  Cc: kata-dev <kata-dev at lists.katacontainers.io>
>  Subject: Re: [kata-dev] Performance isolation: expectations for number of CPUs
>
>  > On 1 Jul 2021, at 16:55, Eric Ernst <eric.g.ernst at gmail.com> wrote:
>  >
>  > In this example, you’ll want to clarify what the CPUs requested means. I’ll assume limits=request, and that you’re referring to a kubernetes pod.
>
>  Indeed, see the linked issues for examples. Also, ideally, we would like this to work with request but no limit.
>
>  > Based on that I’d expect 12. I would not recommend four default vCPUs though.
>
>  The 3, 4 and 5 were just examples to get different numbers as an output. I chose four for the VM initial VCPUs to illustrate that we may have a possible workaround for request.vcpu=4 not doing anything.
>
>  >
>  > Eric
>  >
>  > Sent from my iPhone
>  >
>  >> On Jul 1, 2021, at 3:54 AM, Christophe de Dinechin <dinechin at redhat.com> wrote:
>  >>
>  >> An interesting question arose about the number of CPUs we want to get in the VM, notably in the context of https://github.com/kata-containers/kata-containers/issues/2071 as well as regarding
>  https://github.com/kata-containers/kata-containers/pull/2131, a fix for https://github.com/kata-containers/kata-containers/issues/2130.
>  >>
>  >> Let's say that we have two containers A and B requesting 5 and 3 CPUs respectively. How many CPUs should we get in the VM? Let us assume that the default number of VCPUs is 4.
>  >>
>  >> Possible answers:
>  >>
>  >> A) 4 (default number of VCPUs), because the current OCI spec does not give us information about the number of CPUs. That's how I interpret Julio's answer,
>  https://github.com/kata-containers/kata-containers/issues/2071#issuecomment-865034753.
>  >>
>  >> B) 5 (maximum request). This seems to be more or less how the Rust agent behaves today, making sure that each time there is a request, we online at least that many CPUs.
>  >>
>  >> C) 8 (sum of requests for all containers). If the two containers request CPUs, they have good reasons to, so we should honor both requests independently. This seems to be what the runtime has in mind, since it hotplugs the new
>  CPUs and the struct VM field is called "cpusDelta"
>  >>
>  >> D) 6 (maximum request, plus one for the agent). During the performance isolation meetings, we seem to have shifted towards the idea that the agent should get a dedicated CPU.
>  >>
>  >> E) 9 (maximum request, plus one for the agent)
>  >>
>  >> F) 9 (maximum request plus the four original)
>  >>
>  >> G) 12 (sum of requests plus the four original)
>  >>
>  >> H) 42 (the correct answer in most cases)
>  >>
>  >> What do you think?
>  >>
>  >>
>  >> _______________________________________________
>  >> kata-dev mailing list
>  >> kata-dev at lists.katacontainers.io
>  >> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
>  >
>
>  _______________________________________________
>  kata-dev mailing list
>  kata-dev at lists.katacontainers.io
>  http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev


--
Cheers,
Christophe de Dinechin (IRC c3d)