Open Stack

Hi,

I mentioned in the architecture call last week two examples of dynamic
kernel configuration appearing by default with kata-runtime that came
somewhat as a surprise to me, in particular compared to what we'd get
with a regular container (e.g. crun). Sorry for the delay in sharing
details, here they come.

--
Examples:
- fq_codel instead of noqueue for default virtio-net interface:

  - crun on Fedora 32:

[root at 300cd72baa94 /]# ip li sh eth0
3: eth0 at if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether de:87:26:08:c3:25 brd ff:ff:ff:ff:ff:ff link-netnsid 0

  - kata-runtime:

[root at 420e660f3870 /]# ip li sh eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 86:ab:39:73:7c:b8 brd ff:ff:ff:ff:ff:ff

This seems to come from
runtime/vendor/github.com/vishvananda/netlink/qdisc.go. I don't know the
exact reason, fq_codel might be a sane default choice for almost any
environment, and I didn't observe any breakage due to this.

- nodad on IPv6 addresses:

  - crun:

[root at 300cd72baa94 /]# ip ad sh eth0
3: eth0 at if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether de:87:26:08:c3:25 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.88.0.27/16 brd 10.88.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::dc87:26ff:fe08:c325/64 scope link 
       valid_lft forever preferred_lft forever

  - kata-runtime:

[root at 420e660f3870 /]# ip ad sh eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 86:ab:39:73:7c:b8 brd ff:ff:ff:ff:ff:ff
    inet 10.88.0.28/16 brd 10.88.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::84ab:39ff:fe73:7cb8/64 scope link nodad 
       valid_lft forever preferred_lft forever

This was introduced by:
	https://github.com/kata-containers/agent/pull/722/commits/c66b9279cc8ee273973878240c255e9d8fe8e552
the reason is clearly explained in comments and was also reiterated by
Archana last week, and totally makes sense to me in the general case. I
wonder, in this particular case, if we can really assume that any upper
stack or any environment can actually guarantee DAD is not needed, but
I also couldn't observe any issue due to this, so far.
--

My concern is not so much related to networking components themselves
at this stage, but rather, more in general, to what might happen with
peculiar choices for other subsystems and userspace expectations. I
don't have concrete examples about breakages and, again, this is not
about build-time kernel configuration.

If I understood correctly, bergwolf's proposal from last week goes in
the direction of having some kind of facility or approach that allows
us to track divergences introduced in kata-runtime compared to regular
containers (or to the current configuration of the host kernel), to
configure those behaviours and to systematically document particular
behaviours.

I'm rather new to this, so I don't have a clear picture of how this
might apply, in practice, should we choose to start from those two
cases as examples. Feedback and further input are warmly welcome!

-- 
Stefano