On 2020/5/5 21:27, Stefano Brivio wrote:
> Hi,
>
> I mentioned in the architecture call last week two examples of dynamic
> kernel configuration appearing by default with kata-runtime that came
> somewhat as a surprise to me, in particular compared to what we'd get
> with a regular container (e.g. crun). Sorry for the delay in sharing
> details, here they come.
>
> --
> Examples:
> - fq_codel instead of noqueue for default virtio-net interface:
>
> - crun on Fedora 32:
>
> [root@300cd72baa94 /]# ip li sh eth0
> 3: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
> link/ether de:87:26:08:c3:25 brd ff:ff:ff:ff:ff:ff link-netnsid 0
>
> - kata-runtime:
>
> [root@420e660f3870 /]# ip li sh eth0
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
> link/ether 86:ab:39:73:7c:b8 brd ff:ff:ff:ff:ff:ff
>
> This seems to come from
> runtime/vendor/github.com/vishvananda/netlink/qdisc.go. I don't know the
> exact reason, fq_codel might be a sane default choice for almost any
> environment, and I didn't observe any breakage due to this.
>
> - nodad on IPv6 addresses:
>
> - crun:
>
> [root@300cd72baa94 /]# ip ad sh eth0
> 3: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
> link/ether de:87:26:08:c3:25 brd ff:ff:ff:ff:ff:ff link-netnsid 0
> inet 10.88.0.27/16 brd 10.88.255.255 scope global eth0
> valid_lft forever preferred_lft forever
> inet6 fe80::dc87:26ff:fe08:c325/64 scope link
> valid_lft forever preferred_lft forever
>
> - kata-runtime:
>
> [root@420e660f3870 /]# ip ad sh eth0
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
> link/ether 86:ab:39:73:7c:b8 brd ff:ff:ff:ff:ff:ff
> inet 10.88.0.28/16 brd 10.88.255.255 scope global eth0
> valid_lft forever preferred_lft forever
> inet6 fe80::84ab:39ff:fe73:7cb8/64 scope link nodad
> valid_lft forever preferred_lft forever
>
> This was introduced by:
> https://github.com/kata-containers/agent/pull/722/commits/c66b9279cc8ee273973878240c255e9d8fe8e552
> the reason is clearly explained in comments and was also reiterated by
> Archana last week, and totally makes sense to me in the general case. I
> wonder, in this particular case, if we can really assume that any upper
> stack or any environment can actually guarantee DAD is not needed, but
> I also couldn't observe any issue due to this, so far.
> --
>
> My concern is not so much related to networking components themselves
> at this stage, but rather, more in general, to what might happen with
> peculiar choices for other subsystems and userspace expectations. I
> don't have concrete examples about breakages and, again, this is not
> about build-time kernel configuration.
>
> If I understood correctly, bergwolf's proposal from last week goes in
> the direction of having some kind of facility or approach that allows
> us to track divergences introduced in kata-runtime compared to regular
> containers (or to the current configuration of the host kernel), to
> configure those behaviours and to systematically document particular
> behaviours.
Hi Stefano,
My main concern about making guest kernel behave like the host kernel is
that we might lose the ability to have a customized/optimized kernel
just for container use case. There are a lot of kernel config options
that are not going to be useful for container workload. So instead of
just using the host kernel (for kata containers), I would suggest just
using a minimal guest kernel as a basis and start adding new config
options/modules as we identify new needs. And that is what we have been
doing for Kata Containers in the past years.
Production wise there is a lot of value in having the same kernel on the host and the guest.
For example, taking a workload that has been run as a vanila container and then running it on a kata container could require a testing/certification process from scratch if the host/guest kernels are different.
Kernel CVEs would also be better managed if the host/guest kernels are the same.
As far as dynamic kernel options, is it enough to use kernel boot
options + sysctls + kernel modules to meet the requirement? I asked it
in the AC meeting but there was no conclusion yet. Most containers
shouldn't care about specific kernel options. If kernel boot options +
sysctls + kernel modules still cannot satisfy it, that should be a very
specialized workload and IMO such users (who would likely be very
professional at customizing things) can use a customized guest kernel
instead of the standard one shipped with Kata Containers.
Just my 2 cents...
Cheers,
Tao
>
> I'm rather new to this, so I don't have a clear picture of how this
> might apply, in practice, should we choose to start from those two
> cases as examples. Feedback and further input are warmly welcome!
>
--
Into something rich and strange.
_______________________________________________
kata-dev mailing list
kata-dev@lists.katacontainers.io
http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev