Open Stack

Hi Stefan,

Thanks for sharing your experiments. Your approach to integrate Kata shows some
points improve to support this use case.

I did not consider in the past use the host kernel as the kata kernel but
definitely is a use case we could support.

A couple of questions related on how to mange the kernel and image when you
add Fedora RPMs.

Are you going to add a symlink to allow the kata runtime always find the latest
installed kata kernel ?

How are you planning to update the modules if a kernel changes ? The image will
by generated after a new kernel update? Or every time the kernel changes you
are going to update the kernel from package?

On Fri, Jun 29, 2018 at 06:42:27PM +0100, Stefan Hajnoczi wrote:
> Hi,
> I wanted to share some thoughts after running Kata on Fedora.  I built
> Kata from source and used the Fedora rootfs together with a vanilla
> modular kernel.  This configuration is similar to what I imagine
> official Fedora RPMs would eventually use, because they probably want to
> use a stock Fedora kernel instead of building a special monolithic
> kernel for Kata.
> 
> The Fedora rootfs was built like this:
> 
>   # KERNEL_MODULES_DIR=/lib/modules/$(uname -r) OS_VERSION=28 ./rootfs.sh fedora
> 
> My understanding is Kata wants *either* an initramfs or a disk image.
> Modular kernels typically need to load drivers before they can access
> the disk, so I opted for the initramfs even though it is several hundred
> megabytes.  The initramfs could probably be shrunk significantly with
> some work on the Fedora rootfs script.
> 
> This configuration required several tweaks to get Kata running.  I'm not
> going to create a patch series or send a pull request, because at this
> stage these are dirty hacks that cannot be merged.  But it's likely that
> other people will encounter these issues, so I'm including them below.
> 
> Stefan
> 
> ---
> 
>  * kata-agent assumes initramfs is equivalent to running the agent as
>    init.  In our case kata-agent *is* in an initramfs but *not* init, so
>    it calculates the wrong value for noPivotRoot:
> 
> diff --git a/agent.go b/agent.go
> index ec01250..182c2de 100644
> --- a/agent.go
> +++ b/agent.go
> @@ -830,7 +832,7 @@ func realMain() {
>                 running:    false,
>                 // pivot_root won't work for init, see
>                 // Documention/filesystem/ramfs-rootfs-initramfs.txt
> -               noPivotRoot:    os.Getpid() == 1,
> +               noPivotRoot:    true, // TODO check initramfs, not pid=1!  kata-agent might have been started by systemd form an initramfs os.Getpid() == 1,
>                 subreaper:      r,
>                 pciDeviceMap:   make(map[string]string),
>                 deviceWatchers: make(map[string](chan string)),
> 
>  * kata-agent assumes AF_VSOCK should be used if the guest kernel has
>    vsock support.  A generic kernel like the Fedora kernel will have
>    AF_VSOCK but that doesn't mean the virtio-serial device should be
>    ignored.  Perhaps change the check to see if
>    /dev/virtio-ports/agent.channel.0 is present?
> 
> diff --git a/channel.go b/channel.go
> index b4f8977..34ef292 100644
> --- a/channel.go
> +++ b/channel.go
> @@ -29,25 +29,29 @@ type channel interface {
>  }
>  
>  func newChannel() (channel, error) {
> +       // This is broken, it ignores the virtio-serial agent.channel.0 device
> +       // in favor of AF_VSOCK support, even on systems that are not going to
> +       // use AF_VSOCK but happen to have the drivers installed.
> +
>         // Check for vsock support.
> -       vSockSupported, err := isAFVSockSupported()
> -       if err != nil {
> -               return nil, err
> -       }
> -
> -       if vSockSupported {
> -               // Check if vsock socket exists. We want to cover the case
> -               // where the guest OS can support vsock, but the runtime is
> -               // still using virtio serial connection.
> -               exist, err := vSockPathExist()
> -               if err != nil {
> -                       return nil, err
> -               }
> -
> -               if exist {
> -                       return &vSockChannel{}, nil
> -               }
> -       }
> +//     vSockSupported, err := isAFVSockSupported()
> +//     if err != nil {
> +//             return nil, err
> +//     }
> +//
> +//     if vSockSupported {
> +//             // Check if vsock socket exists. We want to cover the case
> +//             // where the guest OS can support vsock, but the runtime is
> +//             // still using virtio serial connection.
> +//             exist, err := vSockPathExist()
> +//             if err != nil {
> +//                     return nil, err
> +//             }
> +//
> +//             if exist {
> +//                     return &vSockChannel{}, nil
> +//             }
> +//     }
>  
>         return &serialChannel{}, nil
>  }
> 
>  * The Fedora initramfs and modular kernel rely on udev to load the
>    virtio_console.ko driver.  Make sure to start kata-agent *after* the
>    agent device appears.  Note this change will break AF_VSOCK if you
>    choose to use that instead of virtio-serial!
> 
> diff --git a/kata-agent.service.in b/kata-agent.service.in
> index 861757d..5395047 100644
> --- a/kata-agent.service.in
> +++ b/kata-agent.service.in
> @@ -2,6 +2,8 @@
>  Description=Kata Containers Agent
>  Documentation=https://github.com/kata-containers/agent
>  Wants=kata-containers.target
> +BindsTo=dev-virtio\x2dports-agent.channel.0.device
> +After=dev-virtio\x2dports-agent.channel.0.device
>  
>  [Service]
>  # Send agent output to tty to allow capture debug logs
> 
>  * Make sure the rootfs has modprobe and udev.  Needed by the modular
>    kernel.
> 
> diff --git a/rootfs-builder/fedora/config.sh b/rootfs-builder/fedora/config.sh
> index a0ca15a..19fbc7b 100644
> --- a/rootfs-builder/fedora/config.sh
> +++ b/rootfs-builder/fedora/config.sh
> @@ -14,4 +14,4 @@ PACKAGES="iptables"
>  #Optional packages:
>  # systemd: An init system that will start kata-agent if kata-agent
>  #          itself is not configured as init process.
> -[ "$AGENT_INIT" == "no" ] && PACKAGES+=" systemd" || true
> +[ "$AGENT_INIT" == "no" ] && PACKAGES+=" systemd systemd-udev" || true



> _______________________________________________
> kata-dev mailing list
> kata-dev at lists.katacontainers.io
> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev