-----Original Message----- From: Gupta, Sandeep <Sandeep.Gupta@amd.com> Sent: Wednesday, February 10, 2021 12:02 PM To: Venegas Munoz, Jose Carlos <jose.carlos.venegas.munoz@intel.com> Cc: Gupta, Sandeep <Sandeep.Gupta@amd.com> Subject: FW: Regarding, RPC Error with kata-runtime, kata 2.x (kata-containers-stable-2.0.0) with mem_encryption (AMD SEV) enabled [AMD Official Use Only - Internal Distribution Only] Hi Carlos, Forwarding, Not sure if this email made it to the kata-dev list, If not could you please forward to the kata-dev list ? --Best Regards, --Sandeep Gupta -----Original Message----- From: Gupta, Sandeep <Sandeep.Gupta@amd.com> Sent: Wednesday, February 10, 2021 6:19 AM To: kata-dev <kata-dev@lists.katacontainers.io> Cc: Gupta, Sandeep <Sandeep.Gupta@amd.com>; Hollingsworth, Brent <brent.hollingsworth@amd.com> Subject: Regarding, RPC Error with kata-runtime, kata 2.x (kata-containers-stable-2.0.0) with mem_encryption (AMD SEV) enabled [AMD Official Use Only - Internal Distribution Only] Hi Kata Developers, Any suggestions for the following please - (1) Using containerd version 1.3.3 on Ubuntu "20.04.1 LTS (Focal Fossa)" Host: # containerd --version containerd github.com/containerd/containerd 1.3.3-0ubuntu2.2 (2) Using kata-runtime 2.0.0 (with AMD SEV mem_encryption changes): # /usr/local/bin/kata-runtime --version : 2.0.0 commit : fed5a81d373c5cb4d4a3961b5731407729572ea2-dirty OCI specs: 1.0.1-dev (3) /etc/containerd/config.toml : # cat /etc/containerd/config.toml [debug] level = "debug" [plugins.cri.containerd] no_pivot = false [plugins.cri.containerd.runtimes] [plugins.cri.containerd.runtimes.runc] runtime_type = "io.containerd.runc.v1" [plugins.cri.containerd.runtimes.runc.options] NoPivotRoot = false NoNewKeyring = false ShimCgroup = "" IoUid = 0 IoGid = 0 BinaryName = "runc" Root = "" CriuPath = "" SystemdCgroup = false [plugins.cri.containerd.runtimes.kata] runtime_type = "io.containerd.kata.v2" [plugins.cri.containerd.runtimes.kata.options] ConfigPath = "/etc/kata-containers/configuration.toml" BinaryName = "/usr/local/bin/kata-runtime" [plugins.cri.containerd.runtimes.katacli] runtime_type = "io.containerd.runc.v1" [plugins.cri.containerd.runtimes.katacli.options] NoPivotRoot = false NoNewKeyring = false ShimCgroup = "" IoUid = 0 IoGid = 0 BinaryName = "/usr/local/bin/kata-runtime" Root = "" CriuPath = "" SystemdCgroup = false (4) See RPC Error: ctr: Failed to check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 823476853:1024: unknown # ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest csh ctr: Failed to check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 823476853:1024: unknown (5) "containerd" DEBUG Log: DEBU[2021-02-10T13:54:19.789109935Z] stat snapshot key="sha256:de168d3b8ec4765250376be559836953e27ff721fff3bc1124dd758bb153ce51" DEBU[2021-02-10T13:54:19.798103373Z] prepare snapshot key=csh parent="sha256:de168d3b8ec4765250376be559836953e27ff721fff3bc1124dd758bb153ce51" DEBU[2021-02-10T13:54:19.803804854Z] event published ns=default topic=/snapshot/prepare type=containerd.events.SnapshotPrepare DEBU[2021-02-10T13:54:19.810185332Z] get snapshot mounts key=csh DEBU[2021-02-10T13:54:19.819678316Z] event published ns=default topic=/containers/create type=containerd.events.ContainerCreate DEBU[2021-02-10T13:54:19.823584296Z] get snapshot mounts key=csh time="2021-02-10T13:54:19.902979570Z" level=debug msg="registering ttrpc server" time="2021-02-10T13:54:19.903105529Z" level=debug msg="serving api on abstract socket" socket="[inherited from parent]" time="2021-02-10T13:54:19.903130236Z" level=info msg="starting signal loop" namespace=default path=/run/containerd/io.containerd.runtime.v2.task/default/csh pid=5845 time="2021-02-10T13:54:19.903697390Z" level=debug msg="converting /run/containerd/io.containerd.runtime.v2.task/default/csh/config.json" pid=5845 sandbox=csh source=virtcontainers subsystem=compatoci time="2021-02-10T13:54:19.908139666Z" level=info msg="loaded configuration" file=/etc/kata-containers/configuration.toml format=TOML pid=5845 sandbox=csh source=katautils time="2021-02-10T13:54:19.908320729Z" level=info msg="IOMMUPlatform is disabled by default." pid=5845 sandbox=csh source=katautils time="2021-02-10T13:54:19.909315704Z" level=info msg="shm-size detected: 67108864" source=virtcontainers subsystem=oci time="2021-02-10T13:54:19.912364771Z" level=info msg="create netns" netns=/var/run/netns/cnitest-98821389-a527-2806-7bb4-8eb3f00d5cee pid=5845 sandbox=csh source=katautils time="2021-02-10T13:54:19.941668156Z" level=info msg="adding volume" pid=5845 sandbox=csh source=virtcontainers subsystem=qemu volume-type=virtio-9p time="2021-02-10T13:54:19.942304291Z" level=info msg="Endpoints found after scan" endpoints="[]" pid=5845 sandbox=csh source=virtcontainers subsystem=network time="2021-02-10T13:54:19.942425240Z" level=info msg="Starting VM" pid=5845 sandbox=csh source=virtcontainers subsystem=sandbox time="2021-02-10T13:54:19.942779692Z" level=info msg="Adding extra file [0xc0000b2d90]" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp time="2021-02-10T13:54:19.942888969Z" level=info msg="launching /usr/local/bin/qemu-system-x86_64 with: [-name sandbox-csh -uuid 5e3826f7-9544-49fa-aa07-ed92460f061a -machine q35,accel=kvm,kernel_irqchip -cpu host,pmu=off -qmp unix:/run/vc/vm/csh/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=129755M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,disable-legacy=on,id=serial0,romfile=,iommu_platform=true -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/csh/console.sock,server,nowait -device virtio-scsi-pci,id=scsi0,disable-modern=false,disable-legacy=on,romfile=,iommu_platform=true -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=5 -machine memory-encryption=sev0 -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0,romfile= -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-823476853,guest-cid=823476853,romfile= -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile=,disable-legacy=on -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/csh/shared,security_model=none,multidevs=remap -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object memory-backend-ram,id=dimm1,size=2048M -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug panic=1 nr_cpus=240 scsi_mod.scan=none root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -drive if=pflash,format=raw,unit=0,file=/usr/local/share/qemu/OVMF_CODE.fd.kata,readonly -pidfile /run/vc/vm/csh/pid -D /run/vc/vm/csh/qemu.log -smp 1,cores=1,threads=1,sockets=240,maxcpus=240]" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp DEBU[2021-02-10T13:54:19.981283337Z] garbage collected d=4.578314ms time="2021-02-10T13:54:21.633862888Z" level=info msg="{\"QMP\": {\"version\": {\"qemu\": {\"micro\": 0, \"minor\": 0, \"major\": 5}, \"package\": \"v5.0.0-dirty\"}, \"capabilities\": [\"oob\"]}}" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp time="2021-02-10T13:54:21.634304685Z" level=info msg="QMP details" pid=5845 qmp-capabilities=oob qmp-major-version=5 qmp-micro-version=0 qmp-minor-version=0 sandbox=csh source=virtcontainers subsystem=qemu time="2021-02-10T13:54:21.634487351Z" level=info msg="{\"execute\":\"qmp_capabilities\"}" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp time="2021-02-10T13:54:21.635071618Z" level=info msg="{\"return\": {}}" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp time="2021-02-10T13:54:21.635296183Z" level=info msg="scanner return error: read unix @->/run/vc/vm/csh/qmp.sock: use of closed network connection" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp time="2021-02-10T13:54:21.635464933Z" level=info msg="VM started" pid=5845 sandbox=csh source=virtcontainers subsystem=sandbox time="2021-02-10T13:54:21.635653630Z" level=info msg="New client" pid=5845 sandbox=csh source=virtcontainers subsystem=kata_agent url="vsock://823476853:1024" time="2021-02-10T13:54:37.761624843Z" level=info msg="Stopping Sandbox" pid=5845 sandbox=csh source=virtcontainers subsystem=qemu time="2021-02-10T13:54:37.762194181Z" level=info msg="{\"QMP\": {\"version\": {\"qemu\": {\"micro\": 0, \"minor\": 0, \"major\": 5}, \"package\": \"v5.0.0-dirty\"}, \"capabilities\": [\"oob\"]}}" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp time="2021-02-10T13:54:37.762398849Z" level=info msg="{\"execute\":\"qmp_capabilities\"}" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp time="2021-02-10T13:54:37.763609992Z" level=info msg="{\"return\": {}}" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp time="2021-02-10T13:54:37.763795964Z" level=info msg="{\"execute\":\"quit\"}" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp time="2021-02-10T13:54:37.764363579Z" level=info msg="{\"return\": {}}" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp time="2021-02-10T13:54:37.764444993Z" level=info msg="{\"timestamp\": {\"seconds\": 1612965277, \"microseconds\": 764351}, \"event\": \"SHUTDOWN\", \"data\": {\"guest\": false, \"reason\": \"host-qmp-quit\"}}" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp time="2021-02-10T13:54:37.764556475Z" level=info msg="cleanup vm path" dir=/run/vc/vm/csh link=/run/vc/vm/csh pid=5845 sandbox=csh source=virtcontainers subsystem=qemu time="2021-02-10T13:54:37.764797291Z" level=info msg="Network namespace \"/var/run/netns/cnitest-98821389-a527-2806-7bb4-8eb3f00d5cee\" deleted" pid=5845 sandbox=csh source=virtcontainers subsystem=network time="2021-02-10T13:54:37.765695852Z" level=warning msg="sandbox cgroups path is empty" pid=5845 sandbox=csh source=virtcontainers subsystem=sandbox time="2021-02-10T13:54:37.765758611Z" level=info msg="cleanup agent" path=/run/kata-containers/shared/sandboxes/csh/shared pid=5845 sandbox=csh source=virtcontainers subsystem=kata_agent time="2021-02-10T13:54:37.766279267Z" level=warning msg="failed to cleanup netns" error="failed to get netns /var/run/netns/cnitest-98821389-a527-2806-7bb4-8eb3f00d5cee: failed to Statfs \"/var/run/netns/cnitest-98821389-a527-2806-7bb4-8eb3f00d5cee\": no such file or directory" path=/var/run/netns/cnitest-98821389-a527-2806-7bb4-8eb3f00d5cee pid=5845 sandbox=csh source=katautils time="2021-02-10T13:54:37.771597117Z" level=info msg="scanner return error: <nil>" pid=5845 sandbox=csh source=virtcontainers subsystem=qmp INFO[2021-02-10T13:54:37.776331001Z] shim disconnected id=csh DEBU[2021-02-10T13:54:37.778567937Z] remove snapshot key=csh DEBU[2021-02-10T13:54:37.782134712Z] event published ns=default topic=/snapshot/remove type=containerd.events.SnapshotRemove DEBU[2021-02-10T13:54:37.785163547Z] event published ns=default topic=/containers/delete type=containerd.events.ContainerDelete DEBU[2021-02-10T13:54:37.960836857Z] schedule snapshotter cleanup snapshotter=overlayfs DEBU[2021-02-10T13:54:37.963802433Z] removed snapshot key=default/163/csh snapshotter=overlayfs DEBU[2021-02-10T13:54:37.964395075Z] snapshot garbage collected d=3.468218ms snapshotter=overlayfs DEBU[2021-02-10T13:54:37.964440331Z] garbage collected d=4.447562ms (6) With mem_encryption OFF (bring-up is proper) : # ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest csh / # / # / # ps PID USER TIME COMMAND 1 root 0:00 sh 2 root 0:00 ps / # uname -a Linux sandeepg-ubuntu 5.6.0-sev-sev-es-5.6-v3.container #3 SMP Tue Nov 3 06:38:40 UTC 2020 x86_64 GNU/Linux / # exit Any suggestions please, --Regards --Sandeep Gupta