On 2021-07-07 at 22:46 +02, Jens Freimann <jfreiman@redhat.com> wrote...
On Wed, Jul 7, 2021 at 12:48 PM Christophe de Dinechin <dinechin@redhat.com> wrote:
On 2021-07-07 at 01:31 UTC, "Adams, Eric" <eric.adams@intel.com> wrote...
Christophe, 2) I also observed that Pod Overhead isn't used in the calculation for hotplugged CPU's.
That seems correct. The pod overhead accounts for the overhead "outside" the VM, i.e. virtiofsd, qemu's own memory needs, the extra cost of doing I/Os, etc. So this is additional resources the host needs, not the VM.
This is very counter-intuitive and also doesn't fit the 'Motivation' section in the original design[1]. Wouldn't this lead to wrong results for the resource quota and scheduler calculations?
In addition to Eric's answer, let me answer your specific question: on the contrary, I believe that this fits the motivation section and ensures we get the right results from resource scheduling. Consider a container where you request 5 CPUs and 17G of memory. We need 5VCPUs to run it, and 17G of memory in the guest. Since the guest starts with 1CPU and its default 2G, and adding a limit to workaround the #2071 issue, we now get 6 VCPUs and 19G in the guest. So the "VM overhead" is already covered by the default values and the hotplugging mechanism. Now, in practice, when we measure things, we observe that the VM actually consumes something like 110M (minimal kernel) - 300M (standard kernel with file buffers after an install). In any case, that's way below the 2G allocated by default. On top of this, we need memory for virtiofs, qemu or the shum (~50M-80M). So when we tell kubernetes "how much extra memory do you need", we need to report that amount that k8s does not know about from the container requests, which is the 160M-350M I just described above. That is what will allow the scheduler to do its job properly. Same with CPU. Hope this makes sense.
[1] https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/688-pod...
regards Jens
-- Cheers, Christophe de Dinechin (IRC c3d)