Regarding, RPC Error with kata-runtime, kata 2.x (kata-containers-stable-2.0.0) with mem_encryption (AMD SEV) enabled

10 Feb 2021

      [AMD Official Use Only - Internal Distribution Only]

Hi Kata Developers,

Any suggestions for the following please -

(1) Using containerd version 1.3.3 on Ubuntu "20.04.1 LTS (Focal Fossa)" Host:
	# containerd --version
	containerd github.com/containerd/containerd 1.3.3-0ubuntu2.2

(2) Using kata-runtime 2.0.0 (with AMD SEV mem_encryption changes):
	# /usr/local/bin/kata-runtime --version
  	: 2.0.0
   	commit   : fed5a81d373c5cb4d4a3961b5731407729572ea2-dirty
   	OCI specs: 1.0.1-dev

(3) /etc/containerd/config.toml :

# cat /etc/containerd/config.toml

[debug]
        level = "debug"

[plugins.cri.containerd]
      no_pivot = false
    [plugins.cri.containerd.runtimes]
      [plugins.cri.containerd.runtimes.runc]
         runtime_type = "io.containerd.runc.v1"
         [plugins.cri.containerd.runtimes.runc.options]
           NoPivotRoot = false
           NoNewKeyring = false
           ShimCgroup = ""
           IoUid = 0
           IoGid = 0
           BinaryName = "runc"
           Root = ""
           CriuPath = ""
           SystemdCgroup = false
      [plugins.cri.containerd.runtimes.kata]
         runtime_type = "io.containerd.kata.v2"
          [plugins.cri.containerd.runtimes.kata.options]
           ConfigPath = "/etc/kata-containers/configuration.toml"
           BinaryName = "/usr/local/bin/kata-runtime"
      [plugins.cri.containerd.runtimes.katacli]
         runtime_type = "io.containerd.runc.v1"
         [plugins.cri.containerd.runtimes.katacli.options]
           NoPivotRoot = false
           NoNewKeyring = false
           ShimCgroup = ""
           IoUid = 0
           IoGid = 0
           BinaryName = "/usr/local/bin/kata-runtime"
           Root = ""
           CriuPath = ""
           SystemdCgroup = false

(4) See RPC Error: ctr: Failed to check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 823476853:1024: unknown

# ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest csh

ctr: Failed to check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 823476853:1024: unknown

(5) "containerd" DEBUG Log:

DEBU[2021-02-10T13:54:19.789109935Z] stat snapshot                                 key="sha256:de168d3b8ec4765250376be559836953e27ff721fff3bc1124dd758bb153ce51"
DEBU[2021-02-10T13:54:19.798103373Z] prepare snapshot                              key=csh parent="sha256:de168d3b8ec4765250376be559836953e27ff721fff3bc1124dd758bb153ce51"
DEBU[2021-02-10T13:54:19.803804854Z] event published                               ns=default topic=/snapshot/prepare type=containerd.events.SnapshotPrepare
DEBU[2021-02-10T13:54:19.810185332Z] get snapshot mounts                           key=csh
DEBU[2021-02-10T13:54:19.819678316Z] event published                               ns=default topic=/containers/create type=containerd.events.ContainerCreate
DEBU[2021-02-10T13:54:19.823584296Z] get snapshot mounts                           key=csh
time="2021-02-10T13:54:19.902979570Z" level=debug msg="registering ttrpc server"
time="2021-02-10T13:54:19.903105529Z" level=debug msg="serving api on abstract socket" socket="[inherited from parent]"
time="2021-02-10T13:54:19.903130236Z" level=info msg="starting signal loop" namespace=default path=/run/containerd/io.containerd.runtime.v2.task/default/csh pid=5845
time="2021-02-10T13:54:19.903697390Z" level=debug msg="converting /run/containerd/io.containerd.runtime.v2.task/default/csh/config.json" pid=5845 sandbox=csh source=virtcontainers subsystem=compatoci
time="2021-02-10T13:54:19.908139666Z" level=info msg="loaded configuration" file=/etc/kata-containers/configuration.toml format=TOML pid=5845 sandbox=csh source=katautils
time="2021-02-10T13:54:19.908320729Z" level=info msg="IOMMUPlatform is disabled by default." pid=5845 sandbox=csh source=katautils
time="2021-02-10T13:54:19.909315704Z" level=info msg="shm-size detected: 67108864" source=virtcontainers subsystem=oci
time="2021-02-10T13:54:19.912364771Z" level=info msg="create netns" netns=/var/run/netns/cnitest-98821389-a527-2806-7bb4-8eb3f00d5cee pid=5845 sandbox=csh source=katautils
time="2021-02-10T13:54:19.941668156Z" level=info msg="adding volume" pid=5845 sandbox=csh source=virtcontainers subsystem=qemu volume-type=virtio-9p
time="2021-02-10T13:54:19.942304291Z" level=info msg="Endpoints found after scan" endpoints="[]" pid=5845 sandbox=csh source=virtcontainers subsystem=network
time="2021-02-10T13:54:19.942425240Z" level=info msg="Starting VM" pid=5845 sandbox=csh source=virtcontainers subsystem=sandbox
time="2021-02-10T13:54:19.942779692Z" level=info msg="Adding extra file [0xc0000b2d90]" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp
time="2021-02-10T13:54:19.942888969Z" level=info msg="launching /usr/local/bin/qemu-system-x86_64 with: [-name sandbox-csh -uuid 5e3826f7-9544-49fa-aa07-ed92460f061a -machine q35,accel=kvm,kernel_irqchip -cpu host,pmu=off -qmp unix:/run/vc/vm/csh/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=129755M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,disable-legacy=on,id=serial0,romfile=,iommu_platform=true -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/csh/console.sock,server,nowait -device virtio-scsi-pci,id=scsi0,disable-modern=false,disable-legacy=on,romfile=,iommu_platform=true -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=5 -machine memory-encryption=sev0 -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0,romfile= -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-823476853,guest-cid=823476853,romfile= -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile=,disable-legacy=on -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/csh/shared,security_model=none,multidevs=remap -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object memory-backend-ram,id=dimm1,size=2048M -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug panic=1 nr_cpus=240 scsi_mod.scan=none root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -drive if=pflash,format=raw,unit=0,file=/usr/local/share/qemu/OVMF_CODE.fd.kata,readonly -pidfile /run/vc/vm/csh/pid -D /run/vc/vm/csh/qemu.log -smp 1,cores=1,threads=1,sockets=240,maxcpus=240]" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp
DEBU[2021-02-10T13:54:19.981283337Z] garbage collected                             d=4.578314ms
time="2021-02-10T13:54:21.633862888Z" level=info msg="{\"QMP\": {\"version\": {\"qemu\": {\"micro\": 0, \"minor\": 0, \"major\": 5}, \"package\": \"v5.0.0-dirty\"}, \"capabilities\": [\"oob\"]}}" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp
time="2021-02-10T13:54:21.634304685Z" level=info msg="QMP details" pid=5845 qmp-capabilities=oob qmp-major-version=5 qmp-micro-version=0 qmp-minor-version=0 sandbox=csh source=virtcontainers subsystem=qemu
time="2021-02-10T13:54:21.634487351Z" level=info msg="{\"execute\":\"qmp_capabilities\"}" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp
time="2021-02-10T13:54:21.635071618Z" level=info msg="{\"return\": {}}" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp
time="2021-02-10T13:54:21.635296183Z" level=info msg="scanner return error: read unix @->/run/vc/vm/csh/qmp.sock: use of closed network connection" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp
time="2021-02-10T13:54:21.635464933Z" level=info msg="VM started" pid=5845 sandbox=csh source=virtcontainers subsystem=sandbox
time="2021-02-10T13:54:21.635653630Z" level=info msg="New client" pid=5845 sandbox=csh source=virtcontainers subsystem=kata_agent url="vsock://823476853:1024"
time="2021-02-10T13:54:37.761624843Z" level=info msg="Stopping Sandbox" pid=5845 sandbox=csh source=virtcontainers subsystem=qemu
time="2021-02-10T13:54:37.762194181Z" level=info msg="{\"QMP\": {\"version\": {\"qemu\": {\"micro\": 0, \"minor\": 0, \"major\": 5}, \"package\": \"v5.0.0-dirty\"}, \"capabilities\": [\"oob\"]}}" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp
time="2021-02-10T13:54:37.762398849Z" level=info msg="{\"execute\":\"qmp_capabilities\"}" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp
time="2021-02-10T13:54:37.763609992Z" level=info msg="{\"return\": {}}" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp
time="2021-02-10T13:54:37.763795964Z" level=info msg="{\"execute\":\"quit\"}" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp
time="2021-02-10T13:54:37.764363579Z" level=info msg="{\"return\": {}}" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp
time="2021-02-10T13:54:37.764444993Z" level=info msg="{\"timestamp\": {\"seconds\": 1612965277, \"microseconds\": 764351}, \"event\": \"SHUTDOWN\", \"data\": {\"guest\": false, \"reason\": \"host-qmp-quit\"}}" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp
time="2021-02-10T13:54:37.764556475Z" level=info msg="cleanup vm path" dir=/run/vc/vm/csh link=/run/vc/vm/csh pid=5845 sandbox=csh source=virtcontainers subsystem=qemu
time="2021-02-10T13:54:37.764797291Z" level=info msg="Network namespace \"/var/run/netns/cnitest-98821389-a527-2806-7bb4-8eb3f00d5cee\" deleted" pid=5845 sandbox=csh source=virtcontainers subsystem=network
time="2021-02-10T13:54:37.765695852Z" level=warning msg="sandbox cgroups path is empty" pid=5845 sandbox=csh source=virtcontainers subsystem=sandbox
time="2021-02-10T13:54:37.765758611Z" level=info msg="cleanup agent" path=/run/kata-containers/shared/sandboxes/csh/shared pid=5845 sandbox=csh source=virtcontainers subsystem=kata_agent
time="2021-02-10T13:54:37.766279267Z" level=warning msg="failed to cleanup netns" error="failed to get netns /var/run/netns/cnitest-98821389-a527-2806-7bb4-8eb3f00d5cee: failed to Statfs \"/var/run/netns/cnitest-98821389-a527-2806-7bb4-8eb3f00d5cee\": no such file or directory" path=/var/run/netns/cnitest-98821389-a527-2806-7bb4-8eb3f00d5cee pid=5845 sandbox=csh source=katautils
time="2021-02-10T13:54:37.771597117Z" level=info msg="scanner return error: <nil>" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp
INFO[2021-02-10T13:54:37.776331001Z] shim disconnected                             id=csh
DEBU[2021-02-10T13:54:37.778567937Z] remove snapshot                               key=csh
DEBU[2021-02-10T13:54:37.782134712Z] event published                               ns=default topic=/snapshot/remove type=containerd.events.SnapshotRemove
DEBU[2021-02-10T13:54:37.785163547Z] event published                               ns=default topic=/containers/delete type=containerd.events.ContainerDelete
DEBU[2021-02-10T13:54:37.960836857Z] schedule snapshotter cleanup                  snapshotter=overlayfs
DEBU[2021-02-10T13:54:37.963802433Z] removed snapshot                              key=default/163/csh snapshotter=overlayfs
DEBU[2021-02-10T13:54:37.964395075Z] snapshot garbage collected                    d=3.468218ms snapshotter=overlayfs
DEBU[2021-02-10T13:54:37.964440331Z] garbage collected                             d=4.447562ms

(6) With mem_encryption OFF  (bring-up is proper) :

# ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest csh
/ #
/ #
/ # ps
PID   USER     TIME  COMMAND
    1 root      0:00 sh
    2 root      0:00 ps
/ # uname -a
Linux sandeepg-ubuntu 5.6.0-sev-sev-es-5.6-v3.container #3 SMP Tue Nov 3 06:38:40 UTC 2020 x86_64 GNU/Linux
/ # exit

Any suggestions please,

--Regards
--Sandeep Gupta

Gupta, Sandeep

tags

participants (1)