<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 4 Aug 2021, at 09:42, Christophe de Dinechin <<a href="mailto:cdupontd@redhat.com" class="">cdupontd@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">On 2 Aug 2021, at 23:07, Gupta, Sandeep <<a href="mailto:Sandeep.Gupta@amd.com" class="">Sandeep.Gupta@amd.com</a>> wrote:<br class=""><br class="">[AMD Official Use Only]<br class=""><br class="">Hi,<br class=""><br class="">I switched to “virtio-fs”,<br class="">But now I see –<br class=""><br class="">time="2021-08-02T20:54:02.621074148Z" level=info msg="virtiofsd quits" name=containerd-shim-v2 pid=180910 sandbox=ksh source=virtcontainers subsystem=virtiofsd<br class="">time="2021-08-02T20:54:02.621252859Z" level=info msg="Stopping Sandbox" name=containerd-shim-v2 pid=180910 sandbox=ksh source=virtcontainers subsystem=qemu<br class="">time="2021-08-02T20:54:02.621410880Z" level=debug msg="qemu-system-x86_64: -device vhost-user-fs-pci,chardev=char-2478d93684006754,tag=kataShared: iommu_platform=true is not supported by the device" name=containerd-shim-v2 pid=180910 sandbox=ksh source=virtcontainers subsystem=qemu<br class=""></blockquote><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">What version of qemu are you using? You may want to rebase it on something more recent I believe.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""></div></blockquote><div><br class=""></div><div>Ah, strike that. I just saw James Cadden's response. For SEV setup, please defer to him, he has hands on experience and I do not. I did not know that SEV and virtiofs were incompatible.</div><br class=""><blockquote type="cite" class=""><div class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br class=""><br class="">qemu-system-x86_64:<br class="">============================<br class="">time="2021-08-02T21:01:49.140554897Z" level=info msg="launching /usr/local/bin/qemu-system-x86_64 with: [-name sandbox-ksh -uuid d7bc6db6-03a6-4900-9aff-0cf40d0f27ea -machine q35,accel=kvm,kernel_irqchip=split,confidential-guest-support=sev -cpu host,pmu=off -qmp unix:/run/vc/vm/ksh/qmp.sock,server=on,wait=off -m 2048M,slots=10,maxmem=129755M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2 -device virtio-serial-pci,disable-modern=false,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/ksh/console.sock,server=on,wait=off -device virtio-scsi-pci,id=scsi0,disable-modern=false -object sev-guest,id=sev,cbitpos=51,reduced-phys-bits=5 -drive if=pflash,format=raw,readonly=on,file=/usr/local/share/qemu/OVMF_CODE.fd.kata -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0 -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1358627234,guest-cid=1358627234 -chardev socket,id=char-5492eaf755dce6c5,path=/run/vc/vm/ksh/vhost-fs.sock -device vhost-user-fs-pci,chardev=char-5492eaf755dce6c5,tag=kataShared -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object memory-backend-file,id=dimm1,size=2048M,mem-path=/dev/shm,share=on -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinux-sev-es-5.9-v1.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug panic=1 nr_cpus=240 scsi_mod.scan=none agent.log=debug root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -pidfile /run/vc/vm/ksh/pid -D /run/vc/vm/ksh/qemu.log -smp 1,cores=1,threads=1,sockets=240,maxcpus=240]" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=qmp<br class=""><br class=""><br class="">FULL LOG:<br class="">============<br class="">DEBU[2021-08-02T21:01:48.966766727Z] stat snapshot key="sha256:7444ea29e45e927abea1f923bf24cac20deaddea603c4bb1c7f2f5819773d453"<br class="">DEBU[2021-08-02T21:01:48.972845851Z] prepare snapshot key=ksh parent="sha256:7444ea29e45e927abea1f923bf24cac20deaddea603c4bb1c7f2f5819773d453"<br class="">DEBU[2021-08-02T21:01:48.978950864Z] event published ns=default topic=/snapshot/prepare type=containerd.events.SnapshotPrepare<br class="">DEBU[2021-08-02T21:01:48.985793529Z] get snapshot mounts key=ksh<br class="">DEBU[2021-08-02T21:01:49.002191295Z] event published ns=default topic=/containers/create type=containerd.events.ContainerCreate<br class="">DEBU[2021-08-02T21:01:49.006646863Z] get snapshot mounts key=ksh<br class="">time="2021-08-02T21:01:49.094225293Z" level=debug msg="registering ttrpc server"<br class="">time="2021-08-02T21:01:49.094330423Z" level=debug msg="serving api on socket" socket="[inherited from parent]"<br class="">time="2021-08-02T21:01:49.094348687Z" level=info msg="starting signal loop" namespace=default path=/run/containerd/io.containerd.runtime.v2.task/default/ksh pid=181864<br class="">time="2021-08-02T21:01:49.094796228Z" level=debug msg="converting /run/containerd/io.containerd.runtime.v2.task/default/ksh/config.json" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=compatoci<br class="">time="2021-08-02T21:01:49.098307832Z" level=info msg="loaded configuration" file=/etc/kata-containers/configuration.toml format=TOML name=containerd-shim-v2 pid=181864 sandbox=ksh source=katautils<br class="">time="2021-08-02T21:01:49.098468848Z" level=info msg="IOMMUPlatform is enabled by default." name=containerd-shim-v2 pid=181864 sandbox=ksh source=katautils<br class="">time="2021-08-02T21:01:49.098516779Z" level=debug default-kernel-parameters= name=containerd-shim-v2 pid=181864 sandbox=ksh source=katautils<br class="">time="2021-08-02T21:01:49.099545003Z" level=debug msg="container rootfs: /run/containerd/io.containerd.runtime.v2.task/default/ksh/rootfs" source=virtcontainers subsystem=oci<br class="">time="2021-08-02T21:01:49.099611590Z" level=info msg="shm-size detected: 67108864" source=virtcontainers subsystem=oci<br class="">time="2021-08-02T21:01:49.104516813Z" level=info msg="create netns" name=containerd-shim-v2 netns=/var/run/netns/cnitest-8cca1b82-4956-c050-40ff-9d300bb4e199 pid=181864 sandbox=ksh source=katautils<br class="">time="2021-08-02T21:01:49.104849866Z" level=debug msg="restore sandbox failed" error="open /run/vc/sbs/ksh/persist.json: no such file or directory" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=sandbox<br class="">time="2021-08-02T21:01:49.129147606Z" level=info msg="Enabling SEV guest protection" kernel-params="[{tsc reliable} {no_timer_check } {rcupdate.rcu_expedited 1} {i8042.direct 1} {i8042.dumbkbd 1} {i8042.nopnp 1} {i8042.noaux 1} {noreplace-smp } {reboot k} {console hvc0} {console hvc1} {cryptomgr.notests } {net.ifnames 0} {pci lastbus=0}]" kernel-params-debug="[{debug }]" kernel-params-non-debug="[{quiet }]" machine="{q35 accel=kvm,kernel_irqchip=split}" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=qemuAmd64<br class="">time="2021-08-02T21:01:49.129350371Z" level=debug msg="Creating bridges" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=qemu<br class="">time="2021-08-02T21:01:49.129395657Z" level=debug msg="Creating UUID" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=qemu<br class="">time="2021-08-02T21:01:49.136527482Z" level=debug msg="Disable nesting environment checks" inside-vm=false name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=qemu<br class="">time="2021-08-02T21:01:49.137406503Z" level=info msg="adding volume" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=qemu volume-type=virtio-fs<br class="">time="2021-08-02T21:01:49.138303958Z" level=info msg="Endpoints found after scan" endpoints="[]" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=network<br class="">time="2021-08-02T21:01:49.138475655Z" level=debug msg="Network added" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=network<br class="">time="2021-08-02T21:01:49.138523075Z" level=info msg="Starting VM" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=sandbox<br class="">time="2021-08-02T21:01:49.138628215Z" level=debug default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=qemu<br class="">time="2021-08-02T21:01:49.139581246Z" level=info name=containerd-shim-v2 path=/usr/local/libexec/qemu/virtiofsd pid=181864 sandbox=ksh source=virtcontainers subsystem=virtiofsd<br class="">time="2021-08-02T21:01:49.139680605Z" level=info args="--syslog -o cache=auto -o no_posix_lock -o source=/run/kata-containers/shared/sandboxes/ksh/shared --fd=3 -d --thread-pool-size=1" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=virtiofsd<br class="">time="2021-08-02T21:01:49.140476457Z" level=info msg="Adding extra file [0xc000011468]" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=qmp<br class="">time="2021-08-02T21:01:49.140554897Z" level=info msg="launching /usr/local/bin/qemu-system-x86_64 with: [-name sandbox-ksh -uuid d7bc6db6-03a6-4900-9aff-0cf40d0f27ea -machine q35,accel=kvm,kernel_irqchip=split,confidential-guest-support=sev -cpu host,pmu=off -qmp unix:/run/vc/vm/ksh/qmp.sock,server=on,wait=off -m 2048M,slots=10,maxmem=129755M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2 -device virtio-serial-pci,disable-modern=false,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/ksh/console.sock,server=on,wait=off -device virtio-scsi-pci,id=scsi0,disable-modern=false -object sev-guest,id=sev,cbitpos=51,reduced-phys-bits=5 -drive if=pflash,format=raw,readonly=on,file=/usr/local/share/qemu/OVMF_CODE.fd.kata -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0 -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1358627234,guest-cid=1358627234 -chardev socket,id=char-5492eaf755dce6c5,path=/run/vc/vm/ksh/vhost-fs.sock -device vhost-user-fs-pci,chardev=char-5492eaf755dce6c5,tag=kataShared -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object memory-backend-file,id=dimm1,size=2048M,mem-path=/dev/shm,share=on -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinux-sev-es-5.9-v1.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug panic=1 nr_cpus=240 scsi_mod.scan=none agent.log=debug root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -pidfile /run/vc/vm/ksh/pid -D /run/vc/vm/ksh/qemu.log -smp 1,cores=1,threads=1,sockets=240,maxcpus=240]" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=qmp<br class="">DEBU[2021-08-02T21:01:49.177999243Z] garbage collected d=3.573111ms<br class="">time="2021-08-02T21:01:50.609796180Z" level=info msg="virtiofsd quits" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=virtiofsd<br class="">time="2021-08-02T21:01:50.609974409Z" level=info msg="Stopping Sandbox" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=qemu<br class="">time="2021-08-02T21:01:50.610074960Z" level=debug msg="qemu-system-x86_64: -device vhost-user-fs-pci,chardev=char-5492eaf755dce6c5,tag=kataShared: iommu_platform=true is not supported by the device" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=qemu<br class="">time="2021-08-02T21:01:50.610218803Z" level=warning msg="Unable to connect to unix socket (/run/vc/vm/ksh/qmp.sock): dial unix /run/vc/vm/ksh/qmp.sock: connect: connection refused" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=qmp<br class="">time="2021-08-02T21:01:50.610258178Z" level=error msg="Failed to connect to QEMU instance" error="dial unix /run/vc/vm/ksh/qmp.sock: connect: connection refused" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=qemu<br class="">time="2021-08-02T21:01:50.610313303Z" level=info msg="cleanup vm path" dir=/run/vc/vm/ksh link=/run/vc/vm/ksh name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=qemu<br class="">time="2021-08-02T21:01:50.611649432Z" level=error msg="Unable to launch /usr/local/bin/qemu-system-x86_64: exit status 1" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=qmp<br class="">time="2021-08-02T21:01:50.611754672Z" level=error name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=qmp<br class="">time="2021-08-02T21:01:50.611826689Z" level=error msg="failed to launch qemu: " error="exit status 1" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=qemu<br class="">time="2021-08-02T21:01:50.936215508Z" level=info msg="Stopping Sandbox" name=containerd-shim-v2 pid=181864 sandbox=ksh source=virtcontainers subsystem=qemu<br class=""><br class=""><br class=""><br class="">From: Christophe de Dinechin <<a href="mailto:cdupontd@redhat.com" class="">cdupontd@redhat.com</a>><span class="Apple-converted-space"> </span><br class="">Sent: Monday, August 2, 2021 1:53 AM<br class="">To: Gupta, Sandeep <<a href="mailto:Sandeep.Gupta@amd.com" class="">Sandeep.Gupta@amd.com</a>><br class="">Cc: kata-dev <<a href="mailto:kata-dev@lists.katacontainers.io" class="">kata-dev@lists.katacontainers.io</a>>; James Cadden <<a href="mailto:jcadden@ibm.com" class="">jcadden@ibm.com</a>>; Hollingsworth, Brent <<a href="mailto:brent.hollingsworth@amd.com" class="">brent.hollingsworth@amd.com</a>><br class="">Subject: Re: [kata-dev] , "kata-containers-2.2.0-alpha1" with confidential_guest = true (SEV Guest)<br class=""><br class="">[CAUTION: External Email]<span class="Apple-converted-space"> </span><br class=""><br class=""><br class=""><br class="">On 31 Jul 2021, at 19:42, Gupta, Sandeep <<a href="mailto:Sandeep.Gupta@amd.com" class="">Sandeep.Gupta@amd.com</a>> wrote:<br class=""><br class="">[AMD Official Use Only]<br class=""><br class="">Hi Christophe,<br class=""><br class="">The 'qemu logs' are empty.<br class=""><br class="">root@sandeepg-ethanolx-milan:/var/run/vc/vm/ksh# ls -al<br class="">total 4<br class="">drwxr-x--- 2 root root 120 Jul 31 17:37 .<br class="">drwxr-x--- 3 root root 60 Jul 31 17:37 ..<br class="">srwxr-x--- 1 root root 0 Jul 31 17:37 console.sock<br class="">-rw------- 1 root root 8 Jul 31 17:37 pid<br class="">-rw-r--r-- 1 root root 0 Jul 31 17:37 qemu.log<br class="">srwxr-x--- 1 root root 0 Jul 31 17:37 qmp.sock<br class=""><br class="">time="2021-07-31T17:37:05.007627568Z" level=info msg="VM started" name=containerd-shim-v2 pid=1897831 sandbox=ksh source=virtcontainers subsystem=sandbox<br class="">time="2021-07-31T17:37:05.007708140Z" level=debug msg="console watcher starts" name=containerd-shim-v2 pid=1897831 sandbox=ksh source=virtcontainers subsystem=sandbox<br class="">time="2021-07-31T17:37:05.007841804Z" level=debug msg="DNS file not present in ociMounts. Sandbox DNS will not be set." name=containerd-shim-v2 pid=1897831 sandbox=ksh source=virtcontainers subsystem=kata_agent<br class="">time="2021-07-31T17:37:05.007915363Z" level=info msg="New client" name=containerd-shim-v2 pid=1897831 sandbox=ksh source=virtcontainers subsystem=kata_agent url="<a href="vsock://3018373783:1024" class="">vsock://3018373783:1024</a>"<br class="">time="2021-07-31T17:37:35.251396370Z" level=info msg="Stopping Sandbox" name=containerd-shim-v2 pid=1897831 sandbox=ksh source=virtcontainers subsystem=qemu<br class="">time="2021-07-31T17:37:35.253486712Z" level=warning msg="The virtiofsd had stopped" name=containerd-shim-v2 pid=1897831 sandbox=ksh source=virtcontainers subsystem=qemu<br class="">time="2021-07-31T17:37:35.253569129Z" level=debug msg="got QMP event" event="{SHUTDOWN map[guest:false reason:host-qmp-quit] 2021-07-31 17:37:35.000253386 +0000 UTC}" name=containerd-shim-v2 pid=1897831 sandbox=ksh source=virtcontainers subsystem=qemu<br class=""><br class=""><br class="">I am using "virtio-9p" --<br class=""><br class=""># Shared file system type:<br class=""># - virtio-fs (default)<br class=""># - virtio-9p<br class="">shared_fs = "virtio-9p"<br class=""><br class="">Then why this warning message - warning msg="The virtiofsd had stopped" name=containerd-shim-v2 pid=1897831 sandbox=ksh source=virtcontainers subsystem=qemu ??<br class=""><br class="">Oh, I had not seen that you had configured 9pfs.<br class=""><br class="">Is that still supported?<br class=""><br class="">In my limited experience, you can get the "virtiofsd has stopped" message when there is some abrupt crash of one of the components, and the socket is closed.<br class=""><br class=""><br class=""><br class=""><br class="">--Thanking You<br class="">--Best<br class="">--Sandeep<br class=""><br class=""><br class=""><br class="">-----Original Message-----<br class="">From: Christophe de Dinechin <<a href="mailto:cdupontd@redhat.com" class="">cdupontd@redhat.com</a>><span class="Apple-converted-space"> </span><br class="">Sent: Friday, July 30, 2021 1:54 AM<br class="">To: Gupta, Sandeep <<a href="mailto:Sandeep.Gupta@amd.com" class="">Sandeep.Gupta@amd.com</a>><br class="">Cc: kata-dev <<a href="mailto:kata-dev@lists.katacontainers.io" class="">kata-dev@lists.katacontainers.io</a>><br class="">Subject: Re: [kata-dev] , "kata-containers-2.2.0-alpha1" with confidential_guest = true (SEV Guest)<br class=""><br class="">[CAUTION: External Email]<br class=""><br class=""><br class="">On 30 Jul 2021, at 10:21, Gupta, Sandeep via kata-dev <<a href="mailto:kata-dev@lists.katacontainers.io" class="">kata-dev@lists.katacontainers.io</a>> wrote:<br class=""><br class="">[AMD Official Use Only]<br class=""><br class="">Hi All,<br class=""><br class="">(1) Using kata-containers-2.2.0-alpha1 - configuration.toml --<br class=""><br class=""># Copyright (c) 2017-2019 Intel Corporation # # SPDX-License-Identifier: Apache-2.0 #<br class=""><br class=""># XXX: WARNING: this file is auto-generated.<br class=""># XXX:<br class=""># XXX: Source file: "cli/config/configuration-qemu.toml.in"<br class=""># XXX: Project:<br class=""># XXX: Name: Kata Containers<br class=""># XXX: Type: kata<br class=""><br class="">[hypervisor.qemu]<br class="">path = "/usr/local/bin/qemu-system-x86_64"<br class="">#kernel = "/usr/share/kata-containers/vmlinux.container"<br class="">kernel = "/usr/share/kata-containers/vmlinux-sev-es-5.9-v1.container"<br class="">#image = "/usr/share/kata-containers/kata-containers.img"<br class="">initrd = "/usr/share/kata-containers/kata-containers-initrd.img"<br class="">machine_type = "q35"<br class=""><br class=""># Enable confidential guest support.<br class=""># Toggling that setting may trigger different hardware features, ranging # from memory encryption to both memory and CPU-state encryption and integrity.<br class=""># The Kata Containers runtime dynamically detects the available feature set and # aims at enabling the largest possible one.<br class=""># Default false<br class="">confidential_guest = true<br class=""><br class=""># List of valid annotation names for the hypervisor # Each member of the list is a regular expression, which is the base name # of the annotation, e.g. "path" for io.katacontainers.config.hypervisor.path"<br class="">enable_annotations = []<br class=""><br class=""># List of valid annotations values for the hypervisor # Each member of the list is a path pattern as described by glob(3).<br class=""># The default if not set is empty (all annotations rejected.) # Your distribution recommends: ["/usr/bin/qemu-system-x86_64"] valid_hypervisor_paths = ["/usr/local/bin/qemu-system-x86_64"]<br class=""><br class=""># Optional space-separated list of options to pass to the guest kernel.<br class=""># For example, use `kernel_params = "vsyscall=emulate"` if you are having # trouble running pre-2.15 glibc.<br class="">#<br class=""># WARNING: - any parameter specified here will take priority over the default # parameter value of the same name used to start the virtual machine.<br class=""># Do not set values here unless you understand the impact of doing so as you # may stop the virtual machine from booting.<br class=""># To see the list of default parameters, enable hypervisor debug, create a # container and look for 'default-kernel-parameters' log entries.<br class="">#kernel_params = ""<br class="">kernel_params = " root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug"<br class=""><br class=""># Path to the firmware.<br class=""># If you want that qemu uses the default firmware leave this option empty #firmware = ""<br class="">firmware = "/usr/local/share/qemu/OVMF_CODE.fd.kata"<br class="">...........<br class="">..............<br class=""># Shared file system type:<br class=""># - virtio-fs (default)<br class=""># - virtio-9p<br class="">shared_fs = "virtio-9p"<br class="">..............<br class="">................<br class=""><br class=""><br class="">(2) containerd version 1.5.2<br class="">/usr/local/bin/containerd --version<br class="">containerd <a href="http://github.com/containerd/containerd" class="">github.com/containerd/containerd</a> 1.5.2-0ubuntu1~20.04.2<br class=""><br class="">/usr/local/bin/containerd --config /etc/containerd/config.toml<br class=""><br class=""># cat /etc/containerd/config.toml<br class=""># Copyright 2018-2020 Docker Inc.<br class=""><br class=""># Licensed under the Apache License, Version 2.0 (the "License");<br class=""># you may not use this file except in compliance with the License.<br class=""># You may obtain a copy of the License at<br class=""><br class=""># <a href="https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=04%7C01%7Csandeep.gupta%40amd.com%7Cb5c1b6fad29a45a8c2bd08d953378884%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637632320258587100%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=M8I15ylVkd6wnaUe54JbPrXl%2BpACVBEHAIjwBiQw16Q%3D&reserved=0" class="">https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=04%7C01%7Csandeep.gupta%40amd.com%7Cb5c1b6fad29a45a8c2bd08d953378884%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637632320258587100%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=M8I15ylVkd6wnaUe54JbPrXl%2BpACVBEHAIjwBiQw16Q%3D&reserved=0</a><br class=""><br class=""># Unless required by applicable law or agreed to in writing, software<br class=""># distributed under the License is distributed on an "AS IS" BASIS,<br class=""># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.<br class=""># See the License for the specific language governing permissions and<br class=""># limitations under the License.<br class="">[debug]<br class=""> level = "debug"<br class=""><br class="">[plugins.cri.containerd]<br class=""> no_pivot = false<br class=""> [plugins.cri.containerd.runtimes]<br class=""> [plugins.cri.containerd.runtimes.runc]<br class=""> runtime_type = "io.containerd.runc.v1"<br class=""> [plugins.cri.containerd.runtimes.runc.options]<br class=""> NoPivotRoot = false<br class=""> NoNewKeyring = false<br class=""> ShimCgroup = ""<br class=""> IoUid = 0<br class=""> IoGid = 0<br class=""> BinaryName = "runc"<br class=""> Root = ""<br class=""> CriuPath = ""<br class=""> SystemdCgroup = false<br class=""> [plugins.cri.containerd.runtimes.kata]<br class=""> runtime_type = "io.containerd.kata.v2"<br class=""> [plugins.cri.containerd.runtimes.kata.options]<br class=""> ConfigPath = "/etc/kata-containers/configuration.toml"<br class=""> NoPivotRoot = false<br class=""> NoNewKeyring = false<br class=""> ShimCgroup = ""<br class=""> IoUid = 0<br class=""> IoGid = 0<br class=""> BinaryName = "/usr/local/bin/kata-runtime"<br class=""> Root = ""<br class=""> CriuPath = ""<br class=""> SystemdCgroup = false<br class=""> [plugins.cri.containerd.runtimes.katacli]<br class=""> runtime_type = "io.containerd.runc.v1"<br class=""> [plugins.cri.containerd.runtimes.katacli.options]<br class=""> NoPivotRoot = false<br class=""> NoNewKeyring = false<br class=""> ShimCgroup = ""<br class=""> IoUid = 0<br class=""> IoGid = 0<br class=""> BinaryName = "/usr/local/bin/kata-runtime"<br class=""> Root = ""<br class=""> CriuPath = ""<br class=""> SystemdCgroup = false<br class=""><br class=""><br class="">(3) # ctr run --runtime io.containerd.kata.v2 -t --rm <a href="http://docker.io/library/alpine_linux:latest" class="">docker.io/library/alpine_linux:latest</a> bash<br class="">ctr: failed to create shim: Failed to check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 1025935578:1024: unknown<br class=""><br class=""><br class="">(4) containerd LOG:<br class=""><br class="">DEBU[2021-07-30T07:53:27.915940335Z] stat snapshot key="sha256:7444ea29e45e927abea1f923bf24cac20deaddea603c4bb1c7f2f5819773d453"<br class="">DEBU[2021-07-30T07:53:27.921636473Z] prepare snapshot key=bash parent="sha256:7444ea29e45e927abea1f923bf24cac20deaddea603c4bb1c7f2f5819773d453"<br class="">DEBU[2021-07-30T07:53:27.930411184Z] event published ns=default topic=/snapshot/prepare type=containerd.events.SnapshotPrepare<br class="">DEBU[2021-07-30T07:53:27.939032985Z] get snapshot mounts key=bash<br class="">DEBU[2021-07-30T07:53:27.955103540Z] event published ns=default topic=/containers/create type=containerd.events.ContainerCreate<br class="">DEBU[2021-07-30T07:53:27.962015595Z] get snapshot mounts key=bash<br class="">DEBU[2021-07-30T07:53:28.039779478Z] garbage collected d=2.588852ms<br class="">time="2021-07-30T07:53:28.043806590Z" level=debug msg="registering ttrpc server"<br class="">time="2021-07-30T07:53:28.043934212Z" level=debug msg="serving api on socket" socket="[inherited from parent]"<br class="">time="2021-07-30T07:53:28.043956084Z" level=info msg="starting signal loop" namespace=default path=/run/containerd/io.containerd.runtime.v2.task/default/bash pid=1590317 time="2021-07-30T07:53:28.044534952Z" level=debug msg="converting /run/containerd/io.containerd.runtime.v2.task/default/bash/config.json" name=containerd-shim-v2 pid=1590317 sandbox=bash source=virtcontainers subsystem=compatoci time="2021-07-30T07:53:28.049788100Z" level=info msg="loaded configuration" file=/etc/kata-containers/configuration.toml format=TOML name=containerd-shim-v2 pid=1590317 sandbox=bash source=katautils time="2021-07-30T07:53:28.049991937Z" level=info msg="IOMMUPlatform is enabled by default." name=containerd-shim-v2 pid=1590317 sandbox=bash source=katautils time="2021-07-30T07:53:28.051163250Z" level=info msg="shm-size detected: 67108864" source=virtcontainers subsystem=oci time="2021-07-30T07:53:28.054669132Z" level=info msg="create netns" name=containerd-shim-v2 netns=/var/run/netns/cnitest-6747183b-4b3c-3ec6-81c4-d517d32b120b pid=1590317 sandbox=bash source=katautils time="2021-07-30T07:53:28.071513285Z" level=info msg="Enabling SEV guest protection" kernel-params="[{tsc reliable} {no_timer_check } {rcupdate.rcu_expedited 1} {i8042.direct 1} {i8042.dumbkbd 1} {i8042.nopnp 1} {i8042.noaux 1} {noreplace-smp } {reboot k} {console hvc0} {console hvc1} {cryptomgr.notests } {net.ifnames 0} {pci lastbus=0}]" kernel-params-debug="[{debug }]" kernel-params-non-debug="[{quiet }]" machine="{q35 accel=kvm,kernel_irqchip=split}" name=containerd-shim-v2 pid=1590317 sandbox=bash source=virtcontainers subsystem=qemuAmd64 time="2021-07-30T07:53:28.092119026Z" level=info msg="adding volume" name=containerd-shim-v2 pid=1590317 sandbox=bash source=virtcontainers subsystem=qemu volume-type=virtio-9p time="2021-07-30T07:53:28.093015598Z" level=info msg="Endpoints found after scan" endpoints="[]" name=containerd-shim-v2 pid=1590317 sandbox=bash source=virtcontainers subsystem=network time="2021-07-30T07:53:28.093157978Z" level=info msg="Starting VM" name=containerd-shim-v2 pid=1590317 sandbox=bash source=virtcontainers subsystem=sandbox time="2021-07-30T07:53:28.093562065Z" level=info msg="Adding extra file [0xc000010058]" name=containerd-shim-v2 pid=1590317 sandbox=bash source=virtcontainers subsystem=qmp time="2021-07-30T07:53:28.093652406Z" level=info msg="launching /usr/local/bin/qemu-system-x86_64 with: [-name sandbox-bash -uuid b1ff0af6-de06-4025-ab63-9aad959900d5 -machine q35,accel=kvm,kernel_irqchip=split,confidential-guest-support=sev -cpu host,pmu=off -qmp unix:/run/vc/vm/bash/qmp.sock,server=on,wait=off -m 2048M,slots=10,maxmem=129746M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2 -device virtio-serial-pci,disable-modern=false,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/bash/console.sock,server=on,wait=off -device virtio-scsi-pci,id=scsi0,disable-modern=false -object sev-guest,id=sev,cbitpos=51,reduced-phys-bits=5 -drive if=pflash,format=raw,readonly=on,file=/usr/local/share/qemu/OVMF_CODE.fd.kata -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0 -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1025935578,guest-cid=1025935578 -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/bash/shared,security_model=none,multidevs=remap -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object memory-backend-ram,id=dimm1,size=2048M -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinux-sev-es-5.9-v1.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug panic=1 nr_cpus=240 scsi_mod.scan=none root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -pidfile /run/vc/vm/bash/pid -D /run/vc/vm/bash/qemu.log -smp 1,cores=1,threads=1,sockets=240,maxcpus=240]"<br class=""><br class=""><br class=""><br class="">name=containerd-shim-v2 pid=1590317 sandbox=bash source=virtcontainers subsystem=qmp time="2021-07-30T07:53:29.231939187Z" level=info msg="QMP details" name=containerd-shim-v2 pid=1590317 qmp-capabilities=oob qmp-major-version=6 qmp-micro-version=91 qmp-minor-version=0 sandbox=bash source=virtcontainers subsystem=qemu time="2021-07-30T07:53:29.232726561Z" level=info msg="scanner return error: read unix @->/run/vc/vm/bash/qmp.sock: use of closed network connection" name=containerd-shim-v2 pid=1590317 sandbox=bash<br class=""><br class="">It looks like your qemu stopped right after being started (T07:53:29.231939187Z vs T07:53:28.093652406Z). I think that the qemu logs would help us.<br class=""><br class=""><br class="">source=virtcontainers subsystem=qmp time="2021-07-30T07:53:29.232914117Z" level=info msg="VM started" name=containerd-shim-v2 pid=1590317 sandbox=bash source=virtcontainers subsystem=sandbox time="2021-07-30T07:53:29.233044113Z" level=info msg="New client" name=containerd-shim-v2 pid=1590317 sandbox=bash source=virtcontainers subsystem=kata_agent url="<a href="vsock://1025935578:1024" class="">vsock://1025935578:1024</a>"<br class="">time="2021-07-30T07:53:59.475362081Z" level=info msg="Stopping Sandbox" name=containerd-shim-v2 pid=1590317 sandbox=bash source=virtcontainers subsystem=qemu time="2021-07-30T07:53:59.477063059Z" level=warning msg="The virtiofsd had stopped" name=containerd-shim-v2 pid=1590317 sandbox=bash source=virtcontainers subsystem=qemu time="2021-07-30T07:53:59.477246988Z" level=info msg="cleanup vm path" dir=/run/vc/vm/bash link=/run/vc/vm/bash name=containerd-shim-v2 pid=1590317 sandbox=bash source=virtcontainers subsystem=qemu time="2021-07-30T07:53:59.477447608Z" level=info msg="Network namespace \"/var/run/netns/cnitest-6747183b-4b3c-3ec6-81c4-d517d32b120b\" deleted" name=containerd-shim-v2 pid=1590317 sandbox=bash source=virtcontainers subsystem=network time="2021-07-30T07:53:59.478504534Z" level=warning msg="sandbox cgroups path is empty" name=containerd-shim-v2 pid=1590317 sandbox=bash source=virtcontainers subsystem=sandbox time="2021-07-30T07:53:59.478574457Z" level=info msg="cleanup agent" name=containerd-shim-v2 path=/run/kata-containers/shared/sandboxes/bash/shared pid=1590317 sandbox=bash source=virtcontainers subsystem=kata_agent time="2021-07-30T07:53:59.479262623Z" level=warning msg="failed to cleanup netns" error="failed to get netns /var/run/netns/cnitest-6747183b-4b3c-3ec6-81c4-d517d32b120b: failed to Statfs \"/var/run/netns/cnitest-6747183b-4b3c-3ec6-81c4-d517d32b120b\": no such file or directory" name=containerd-shim-v2 path=/var/run/netns/cnitest-6747183b-4b3c-3ec6-81c4-d517d32b120b pid=1590317 sandbox=bash source=katautils time="2021-07-30T07:53:59.479464936Z" level=info msg="scanner return error: <nil>" name=containerd-shim-v2 pid=1590317 sandbox=bash source=virtcontainers subsystem=qmp time="2021-07-30T07:53:59.479683231Z" level=info msg="QMP event channel closed" name=containerd-shim-v2 pid=1590317 sandbox=bash source=virtcontainers subsystem=qemu<br class="">DEBU[2021-07-30T07:53:59.480447321Z] failed to delete task error="rpc error: code = NotFound desc = container does not exist bash: not found" id=bash<br class="">INFO[2021-07-30T07:53:59.484752059Z] shim disconnected id=bash<br class="">WARN[2021-07-30T07:53:59.484860645Z] cleaning up after shim disconnected id=bash namespace=default<br class="">INFO[2021-07-30T07:53:59.484882296Z] cleaning up dead shim<br class="">ERRO[2021-07-30T07:53:59.545695272Z] failed to delete cmd="/usr/local/bin/containerd-shim-kata-v2 -namespace default -address /run/containerd/containerd.sock -publish-binary /usr/local/bin/containerd -id bash -bundle /run/containerd/io.containerd.runtime.v2.task/default/bash delete" error="exit status 1"<br class="">WARN[2021-07-30T07:53:59.545862980Z] failed to clean up after shim disconnected error="time=\"2021-07-30T07:53:59Z\" level=warning msg=\"failed to cleanup container\" container=bash error=\"open /run/vc/sbs/bash: no such file or directory\" name=containerd-shim-v2 pid=1590410 sandbox=bash source=containerd-kata-shim-v2\nio.containerd.kata.v2: open /run/vc/sbs/bash: no such file or directory\n: exit status 1" id=bash namespace=default<br class="">ERRO[2021-07-30T07:53:59.545951698Z] copy shim log error="read /proc/self/fd/15: file already closed"<br class="">DEBU[2021-07-30T07:53:59.557980504Z] remove snapshot key=bash<br class="">DEBU[2021-07-30T07:53:59.560442896Z] event published ns=default topic=/snapshot/remove type=containerd.events.SnapshotRemove<br class="">DEBU[2021-07-30T07:53:59.563483766Z] event published ns=default topic=/containers/delete type=containerd.events.ContainerDelete<br class="">DEBU[2021-07-30T07:53:59.610327214Z] schedule snapshotter cleanup snapshotter=overlayfs<br class="">DEBU[2021-07-30T07:53:59.612570831Z] removed snapshot key=default/23/bash snapshotter=overlayfs<br class="">DEBU[2021-07-30T07:53:59.613058306Z] snapshot garbage collected d=2.633938ms snapshotter=overlayfs<br class="">DEBU[2021-07-30T07:53:59.613107950Z] garbage collected d=2.265409ms<br class=""><br class=""><br class="">Any suggestions please.<br class=""><br class=""><br class="">--Thanking You<br class="">--Sandeep<br class=""><br class="">_______________________________________________<br class="">kata-dev mailing list<br class=""><a href="mailto:kata-dev@lists.katacontainers.io" class="">kata-dev@lists.katacontainers.io</a><br class="">https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-dev&data=04%7C01%7Csandeep.gupta%40amd.com%7Cb5c1b6fad29a45a8c2bd08d953378884%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637632320258597093%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=iX91RIGQ3MdA9A%2BQPc60RByH06oEAyLyEkqPO%2F%2FyLgM%3D&reserved=0</blockquote></div></blockquote></div><br class=""></body></html>