Open Stack

On 2020/5/6 19:53, Stefano Brivio wrote:
> Hi Tao,
> 
> On Wed, 6 May 2020 10:36:02 +0800
> Peng Tao <tao.peng at linux.alibaba.com> wrote:
> 
>> On 2020/5/5 21:27, Stefano Brivio wrote:
>>> Hi,
>>>
>>> I mentioned in the architecture call last week two examples of dynamic
>>> kernel configuration appearing by default with kata-runtime that came
>>> somewhat as a surprise to me, in particular compared to what we'd get
>>> with a regular container (e.g. crun). Sorry for the delay in sharing
>>> details, here they come.
>>>
>>> --
>>> Examples:
>>> - fq_codel instead of noqueue for default virtio-net interface:
>>>
>>>     - crun on Fedora 32:
>>>
>>> [root at 300cd72baa94 /]# ip li sh eth0
>>> 3: eth0 at if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
>>>       link/ether de:87:26:08:c3:25 brd ff:ff:ff:ff:ff:ff link-netnsid 0
>>>
>>>     - kata-runtime:
>>>
>>> [root at 420e660f3870 /]# ip li sh eth0
>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
>>>       link/ether 86:ab:39:73:7c:b8 brd ff:ff:ff:ff:ff:ff
>>>
>>> This seems to come from
>>> runtime/vendor/github.com/vishvananda/netlink/qdisc.go. I don't know the
>>> exact reason, fq_codel might be a sane default choice for almost any
>>> environment, and I didn't observe any breakage due to this.
>>>
>>> - nodad on IPv6 addresses:
>>>
>>>     - crun:
>>>
>>> [root at 300cd72baa94 /]# ip ad sh eth0
>>> 3: eth0 at if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
>>>       link/ether de:87:26:08:c3:25 brd ff:ff:ff:ff:ff:ff link-netnsid 0
>>>       inet 10.88.0.27/16 brd 10.88.255.255 scope global eth0
>>>          valid_lft forever preferred_lft forever
>>>       inet6 fe80::dc87:26ff:fe08:c325/64 scope link
>>>          valid_lft forever preferred_lft forever
>>>
>>>     - kata-runtime:
>>>
>>> [root at 420e660f3870 /]# ip ad sh eth0
>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
>>>       link/ether 86:ab:39:73:7c:b8 brd ff:ff:ff:ff:ff:ff
>>>       inet 10.88.0.28/16 brd 10.88.255.255 scope global eth0
>>>          valid_lft forever preferred_lft forever
>>>       inet6 fe80::84ab:39ff:fe73:7cb8/64 scope link nodad
>>>          valid_lft forever preferred_lft forever
>>>
>>> This was introduced by:
>>> 	https://github.com/kata-containers/agent/pull/722/commits/c66b9279cc8ee273973878240c255e9d8fe8e552
>>> the reason is clearly explained in comments and was also reiterated by
>>> Archana last week, and totally makes sense to me in the general case. I
>>> wonder, in this particular case, if we can really assume that any upper
>>> stack or any environment can actually guarantee DAD is not needed, but
>>> I also couldn't observe any issue due to this, so far.
>>> --
>>>
>>> My concern is not so much related to networking components themselves
>>> at this stage, but rather, more in general, to what might happen with
>>> peculiar choices for other subsystems and userspace expectations. I
>>> don't have concrete examples about breakages and, again, this is not
>>> about build-time kernel configuration.
>>>
>>> If I understood correctly, bergwolf's proposal from last week goes in
>>> the direction of having some kind of facility or approach that allows
>>> us to track divergences introduced in kata-runtime compared to regular
>>> containers (or to the current configuration of the host kernel), to
>>> configure those behaviours and to systematically document particular
>>> behaviours.
>> Hi Stefano,
>>
>> My main concern about making guest kernel behave like the host kernel
> 
> Wait, that's not what I'm proposing or suggesting in any way, I'm just
> concerned that we might, more or less knowingly, over time, grow a set
> of rather specific *runtime* divergences in kernel configuration with
> what one might reasonably expect from a container environment, and end
> up breaking reasonable user or userspace expectations without keeping
> track and without an easy way to fix those expectations later.
> 
> Going back to my (harmless) examples in this perspective:
> 
> - fq_codel as default qdisc: I don't know why it's there, git log
>    didn't help, I'm not sure anybody can even remember or explain
> 
Maybe Manohar can answer that? But it is quite possible that he just 
picked one and didn't think a container workload should care that much 
dependency. So there is chance to enhance it to be configurable if 
necessary.

> - nodad on IPv6 addresses: makes sense in a general kata-runtime case,
>    it's well documented in the agent code, but will it be visible enough
>    to others playing with this ecosystem? Should it be configurable, in
>    case one can't ensure it's not needed from "outside"?
>
Same here. We can make it configurable but we need a use case to prove 
the need first.

>> is that we might lose the ability to have a customized/optimized kernel
>> just for container use case.
> 
> Sure, I'm just trying to find out if there's a way to consistently
> document (and perhaps make configurable) these customisations or
> optimisations, at least the most significant ones, also to make them
> more sustainable in some sense.
> 
In that case, I'm all with you! Better documentation always helps!

>> There are a lot of kernel config options
>> that are not going to be useful for container workload. So instead of
>> just using the host kernel (for kata containers), I would suggest just
>> using a minimal guest kernel as a basis and start adding new config
>> options/modules as we identify new needs. And that is what we have been
>> doing for Kata Containers in the past years.
> 
> Actually, I see this as a separate topic, and it wasn't my intention to
> raise this here.

Sorry I missed last AC meeting and thought you were continuing that 
topic from the previous AC meeting ;)

> 
> As a side note, I understand the reasoning behind your suggestion, and
> I also think Ariel's and your further points are valid in specific
> situations, that are currently covered by specific Linux distributions.
> 
> I think configuration fragments are useful, and so is the current
> script, for testing, integration, and as guidance. It's also something
> a distribution might decide to ship and integrate, I guess.
> 
> However, attempts to somehow force that model as the True and Only one
> looks severely limiting to me, and I don't really see the benefit (more
> on that in my other reply on this thread).
> 
>> As far as dynamic kernel options, is it enough to use kernel boot
>> options + sysctls + kernel modules to meet the requirement? I asked it
>> in the AC meeting but there was no conclusion yet.
> 
> Yes, I didn't present a requirement in this sense, because I don't have
> any specific requirement "to be supported" in mind. I was referring to
> a more general topic of traceability of specific behaviours.
> 
> Now, if I reuse your input as something that we should consider to
> include in those specific behaviours, yes, ideally I think that:
> 
> - boot options that are not explicitly appended via toml configuration,
>    and that are obviously not functional for the specific container
>    scenario (kata-runtime) should be somewhat tracked
> 
> - sysctl writes (I guess by the agent), same
> 
> - modules, would the agent really need to "manually" load specific
>    modules? Will it ever need to?
> 
Not just the agent. I was thinking that apps might want to use modules 
too. We do allow container with SYS_ADMIN privilege.

> Both examples that I presented are about configurations that happen via
> netlink, so I'd consider that in a tentative list of the kind of
> configuration that needs to be tracked (and, I'm not sure, perhaps made
> configurable in any case?).
> 
I'm not sure if we want to make every option configurable in any case. 
Right now we are looking at it case by case based on real world use cases.

Cheers,
Tao

-- 
Into something rich and strange.