[AMD Official Use Only - Internal Distribution Only] Thanks again Christophe,

From my kata-runtime 1.x, setup :

root 2972741 2845780 0 20:13 pts/2 00:00:00 docker run --runtime kata-sev-runtime --cap-add=NET_ADMIN -it apache2-sev root 2018662 2018416 0 15:01 ? 00:00:16 /usr/local/bin/qemu-system-x86_64 -name sandbox-4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc -uuid fce8220b-780b-4da6-8f5d-77ace1e06ed6 -machine q35,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/vc/sbs/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc/qmp.sock,server,nowait -m 512M,slots=2,maxmem=258779M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2 -device virtio-serial-pci,disable-legacy=on,id=serial0,iommu_platform=true -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/sbs/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc/console.sock,server,nowait -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=5 -machine memory-encryption=sev0 -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/sbs/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc/kata.sock,server,nowait -device virtio-9p-pci,disable-legacy=on,iommu_platform=true,fsdev=extra-9p-kataShared,mount_tag=kataShared -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc,security_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,disable-legacy=on,iommu_platform=true,mq=on,vectors=18 -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug panic=1 initcall_debug nr_cpus=240 ip=::::::4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc::off:: root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -drive if=pflash,format=raw,unit=0,file=/usr/local/share/qemu/OVMF_CODE.fd.kata,readonly -smp 1,cores=1,threads=1,sockets=1,maxcpus=240 root 2972841 2970388 0 20:13 ? 00:00:00 containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8 -address /var/run/docker/containerd/containerd.sock -containerd-binary /snap/docker/471/bin/containerd -runtime-root /var/run/docker/runtime-kata-sev-runtime -debug root 2973736 2972841 0 20:13 ? 00:00:00 /usr/libexec/kata-containers/kata-proxy -listen-socket unix:///run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/proxy.sock -mux-socket /run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/kata.sock -log debug -agent-logs-socket /run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/console.sock root 2973856 2972841 0 20:13 pts/4 00:00:00 /usr/libexec/kata-containers/kata-shim -agent unix:///run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/proxy.sock -container 54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8 -exec-id 54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8 -terminal -log debug --Best Regards, --Sandeep Gupta -----Original Message----- From: Christophe de Dinechin <cdupontd@redhat.com> Sent: Tuesday, January 5, 2021 11:25 AM To: Gupta, Sandeep <Sandeep.Gupta@amd.com> Cc: Fabiano Fidêncio <fidencio@redhat.com>; kata-dev@lists.katacontainers.io; Hollingsworth, Brent <brent.hollingsworth@amd.com> Subject: Re: [kata-dev] Regarding, Kata 2.0.0, kata-runtime, qemu OVMF [CAUTION: External Email]

On 5 Jan 2021, at 17:21, Gupta, Sandeep <Sandeep.Gupta@amd.com> wrote:

...

Hi Developers, One more please - for Kata 2.0.0, kata-runtime, qemu OVMF

In /etc/kata-containers/configuration.toml --

====== # WARNING: - any parameter specified here will take priority over the default # parameter value of the same name used to start the virtual machine. # Do not set values here unless you understand the impact of doing so as you # may stop the virtual machine from booting. # To see the list of default parameters, enable hypervisor debug, create a # container and look for 'default-kernel-parameters' log entries. kernel_params = " root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug"

# Path to the firmware. # If you want that qemu uses the default firmware leave this option empty #firmware = "/usr/local/share/qemu/OVMF_CODE.fd.kata" firmware = "" ==========

If I use the default firmware - I get correct results :

# ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest sh / # / # uname -a Linux sandeepg-ubuntu 5.6.0-sev-sev-es-5.6-v3.container #3 SMP Tue Nov 3 06:38:40 UTC 2020 x86_64 GNU/Linux / #

# ps -eaf | grep -i qemu

root 360867 1 0 15:56 ? 00:00:01 /usr/local/bin/qemu-system-x86_64 -name sandbox-sh -uuid 72820914-dcce-4cea-a01f-3cf9851d136d -machine q35,accel=kvm,kernel_irqchip -cpu host,pmu=off -qmp unix:/run/vc/vm/sh/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=129755M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/sh/console.sock,server,nowait -device virtio-scsi-pci,id=scsi0,disable-modern=false,romfile= -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0,romfile= -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1189071429,guest-cid=1189071429,romfile= -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/sh/shared,security_model=none,multidevs=remap -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object memory-backend-ram,id=dimm1,size=2048M -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 pci=lastbus=0 quiet panic=1 nr_cpus=240 scsi_mod.scan=none root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -pidfile /run/vc/vm/sh/pid -smp 1,cores=1,threads=1,sockets=240,maxcpus=240 root 360966 360958 0 16:05 pts/2 00:00:00 grep --color=auto -i qemu

But if I use : firmware = "/usr/local/share/qemu/OVMF_CODE.fd.kata" I get Error:

# ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest s ctr: failed to launch qemu: exit status 1, error messages from qemu log: qemu: could not load PC BIOS '/usr/local/share/qemu/OVMF_CODE.fd.kata' : unknown

Any suggestions please.

I did a quick look. The runtime apparently uses the -bios option when the firmware option is passed, not the -drive option as I thought it would. I'm not entirely sure you can boot OVMF that way. I don't have any idea at the moment why it worked with version 1.x. If you could share with me the 1.x qemu command line option with the firmware option, I'm curious to see what it is. Thanks Christophe

--Thanking You --Sandeep

_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists .katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-dev&data= 04%7C01%7Csandeep.gupta%40amd.com%7C5764f6fb89fd4809aafc08d8b1af94df%7 C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637454714974608155%7CUnkno wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL CJXVCI6Mn0%3D%7C1000&sdata=U85zx8%2FDKniPytZoOsm%2F7eR%2FNq5e8aHFm fJKMKlX%2B84%3D&reserved=0

[AMD Official Use Only - Internal Distribution Only] Hi Tao, Attaching full log for Kata 2.0.0, kata-runtime, qemu OVMF -- including 'qemu-system-x86_64' # ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest csh ctr: failed to launch qemu: exit status 1, error messages from qemu log: qemu: could not load PC BIOS '/usr/local/share/qemu/OVMF_CODE.fd.kata' : unknown qemu-system-x86_64 : ===================== msg="launching /usr/local/bin/qemu-system-x86_64 with: [-name sandbox-csh -uuid a9ecf610-ebb5-42b4-bd8c-d72445f60601 -machine q35,accel=kvm,kernel_irqchip -cpu host,pmu=off -qmp unix:/run/vc/vm/csh/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=129755M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/csh/console.sock,server,nowait -device virtio-scsi-pci,id=scsi0,disable-modern=false,romfile= -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0,romfile= -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-3113278101,guest-cid=3113278101,romfile= -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/csh/shared,security_model=none,multidevs=remap -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object memory-backend-ram,id=dimm1,size=2048M -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 pci=lastbus=0 quiet panic=1 nr_cpus=240 scsi_mod.scan=none root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -bios /usr/local/share/qemu/OVMF_CODE.fd.kata -pidfile /run/vc/vm/csh/pid -smp 1,cores=1,threads=1,sockets=240,maxcpus=240]" pid=361802 sandbox=csh source=virtcontainers subsystem=qmp time="2021-01-06T01:09:08.598025317Z" level=error msg="Unable to launch /usr/local/bin/qemu-system-x86_64: exit status 1" pid=361802 sandbox=csh source=virtcontainers subsystem=qmp time="2021-01-06T01:09:08.598108225Z" level=error msg="qemu: could not load PC BIOS '/usr/local/share/qemu/OVMF_CODE.fd.kata'\n" pid=361802 sandbox=csh source=virtcontainers subsystem=qmp time="2021-01-06T01:09:08.598155344Z" level=error msg="failed to launch qemu: qemu: could not load PC BIOS '/usr/local/share/qemu/OVMF_CODE.fd.kata'\n" error="exit status 1" pid=361802 sandbox=csh source=virtcontainers subsystem=qemu ===================== More complete Log : =================================== DEBU[2021-01-06T01:08:29.615317241Z] stat snapshot key="sha256:de168d3b8ec4765250376be559836953e27ff721fff3bc1124dd758bb153ce51" DEBU[2021-01-06T01:08:29.622956797Z] prepare snapshot key=sh parent="sha256:de168d3b8ec4765250376be559836953e27ff721fff3bc1124dd758bb153ce51" DEBU[2021-01-06T01:08:56.304661933Z] stat snapshot key="sha256:de168d3b8ec4765250376be559836953e27ff721fff3bc1124dd758bb153ce51" DEBU[2021-01-06T01:08:56.317821748Z] prepare snapshot key=bash parent="sha256:de168d3b8ec4765250376be559836953e27ff721fff3bc1124dd758bb153ce51" DEBU[2021-01-06T01:09:08.368128323Z] stat snapshot key="sha256:de168d3b8ec4765250376be559836953e27ff721fff3bc1124dd758bb153ce51" DEBU[2021-01-06T01:09:08.380135161Z] prepare snapshot key=csh parent="sha256:de168d3b8ec4765250376be559836953e27ff721fff3bc1124dd758bb153ce51" DEBU[2021-01-06T01:09:08.385519872Z] event published ns=default topic="/snapshot/prepare" type=containerd.events.SnapshotPrepare DEBU[2021-01-06T01:09:08.390393343Z] get snapshot mounts key=csh DEBU[2021-01-06T01:09:08.400461926Z] event published ns=default topic="/containers/create" type=containerd.events.ContainerCreate DEBU[2021-01-06T01:09:08.404528738Z] get snapshot mounts key=csh time="2021-01-06T01:09:08.503529148Z" level=debug msg="registering ttrpc server" time="2021-01-06T01:09:08.503667060Z" level=debug msg="serving api on abstract socket" socket="[inherited from parent]" time="2021-01-06T01:09:08.503701054Z" level=info msg="starting signal loop" namespace=default path=/run/containerd/io.containerd.runtime.v2.task/default/csh pid=361802 time="2021-01-06T01:09:08.504550545Z" level=debug msg="converting /run/containerd/io.containerd.runtime.v2.task/default/csh/config.json" pid=361802 sandbox=csh source=virtcontainers subsystem=compatoci time="2021-01-06T01:09:08.508812677Z" level=info msg="loaded configuration" file=/etc/kata-containers/configuration.toml format=TOML pid=361802 sandbox=csh source=katautils time="2021-01-06T01:09:08.509043996Z" level=info msg="IOMMUPlatform is disabled by default." pid=361802 sandbox=csh source=katautils time="2021-01-06T01:09:08.510648468Z" level=info msg="shm-size detected: 67108864" source=virtcontainers subsystem=oci time="2021-01-06T01:09:08.513613030Z" level=info msg="create netns" netns=/var/run/netns/cnitest-96e33c0e-136c-b90e-a5c3-a7ff1a47067e pid=361802 sandbox=csh source=katautils time="2021-01-06T01:09:08.543947472Z" level=info msg="adding volume" pid=361802 sandbox=csh source=virtcontainers subsystem=qemu volume-type=virtio-9p time="2021-01-06T01:09:08.544942889Z" level=info msg="Endpoints found after scan" endpoints="[]" pid=361802 sandbox=csh source=virtcontainers subsystem=network time="2021-01-06T01:09:08.545072174Z" level=info msg="Starting VM" pid=361802 sandbox=csh source=virtcontainers subsystem=sandbox time="2021-01-06T01:09:08.545428179Z" level=info msg="Adding extra file [0xc000496c70]" pid=361802 sandbox=csh source=virtcontainers subsystem=qmp time="2021-01-06T01:09:08.545532948Z" level=info msg="launching /usr/local/bin/qemu-system-x86_64 with: [-name sandbox-csh -uuid a9ecf610-ebb5-42b4-bd8c-d72445f60601 -machine q35,accel=kvm,kernel_irqchip -cpu host,pmu=off -qmp unix:/run/vc/vm/csh/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=129755M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/csh/console.sock,server,nowait -device virtio-scsi-pci,id=scsi0,disable-modern=false,romfile= -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0,romfile= -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-3113278101,guest-cid=3113278101,romfile= -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/csh/shared,security_model=none,multidevs=remap -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object memory-backend-ram,id=dimm1,size=2048M -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 pci=lastbus=0 quiet panic=1 nr_cpus=240 scsi_mod.scan=none root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -bios /usr/local/share/qemu/OVMF_CODE.fd.kata -pidfile /run/vc/vm/csh/pid -smp 1,cores=1,threads=1,sockets=240,maxcpus=240]" pid=361802 sandbox=csh source=virtcontainers subsystem=qmp time="2021-01-06T01:09:08.598025317Z" level=error msg="Unable to launch /usr/local/bin/qemu-system-x86_64: exit status 1" pid=361802 sandbox=csh source=virtcontainers subsystem=qmp time="2021-01-06T01:09:08.598108225Z" level=error msg="qemu: could not load PC BIOS '/usr/local/share/qemu/OVMF_CODE.fd.kata'\n" pid=361802 sandbox=csh source=virtcontainers subsystem=qmp time="2021-01-06T01:09:08.598155344Z" level=error msg="failed to launch qemu: qemu: could not load PC BIOS '/usr/local/share/qemu/OVMF_CODE.fd.kata'\n" error="exit status 1" pid=361802 sandbox=csh source=virtcontainers subsystem=qemu time="2021-01-06T01:09:08.623018731Z" level=info msg="Network namespace \"/var/run/netns/cnitest-96e33c0e-136c-b90e-a5c3-a7ff1a47067e\" deleted" pid=361802 sandbox=csh source=virtcontainers subsystem=network time="2021-01-06T01:09:08.623889152Z" level=warning msg="sandbox cgroups path is empty" pid=361802 sandbox=csh source=virtcontainers subsystem=sandbox time="2021-01-06T01:09:08.623991075Z" level=info msg="cleanup agent" path=/run/kata-containers/shared/sandboxes/csh/shared pid=361802 sandbox=csh source=virtcontainers subsystem=kata_agent time="2021-01-06T01:09:08.625160482Z" level=warning msg="failed to cleanup netns" error="failed to get netns /var/run/netns/cnitest-96e33c0e-136c-b90e-a5c3-a7ff1a47067e: failed to Statfs \"/var/run/netns/cnitest-96e33c0e-136c-b90e-a5c3-a7ff1a47067e\": no such file or directory" path=/var/run/netns/cnitest-96e33c0e-136c-b90e-a5c3-a7ff1a47067e pid=361802 sandbox=csh source=katautils DEBU[2021-01-06T01:09:08.636712678Z] remove snapshot key=csh DEBU[2021-01-06T01:09:08.639720111Z] event published ns=default topic="/snapshot/remove" type=containerd.events.SnapshotRemove DEBU[2021-01-06T01:09:08.642734608Z] event published ns=default topic="/containers/delete" type=containerd.events.ContainerDelete DEBU[2021-01-06T01:09:08.765685424Z] schedule snapshotter cleanup snapshotter=overlayfs DEBU[2021-01-06T01:09:08.768590533Z] removed snapshot key="default/148/csh" snapshotter=overlayfs DEBU[2021-01-06T01:09:08.769013054Z] snapshot garbage collected d=3.245765ms snapshotter=overlayfs DEBU[2021-01-06T01:09:08.769056436Z] garbage collected d=4.672521ms ======================================================== --Thanking You --Best --Sandeep -----Original Message----- From: Peng Tao <tao.peng@linux.alibaba.com> Sent: Tuesday, January 5, 2021 7:01 PM To: Gupta, Sandeep <Sandeep.Gupta@amd.com>; Christophe de Dinechin <cdupontd@redhat.com> Cc: Hollingsworth, Brent <brent.hollingsworth@amd.com>; kata-dev@lists.katacontainers.io Subject: Re: [kata-dev] Regarding, Kata 2.0.0, kata-runtime, qemu OVMF [CAUTION: External Email] On 2021/1/6 04:19, Gupta, Sandeep wrote:

[AMD Official Use Only - Internal Distribution Only]

Thanks again Christophe,

From my kata-runtime 1.x, setup :

root 2972741 2845780 0 20:13 pts/2 00:00:00 docker run --runtime kata-sev-runtime --cap-add=NET_ADMIN -it apache2-sev

root 2018662 2018416 0 15:01 ? 00:00:16 /usr/local/bin/qemu-system-x86_64 -name sandbox-4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc -uuid fce8220b-780b-4da6-8f5d-77ace1e06ed6 -machine q35,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/vc/sbs/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc/qmp.sock,server,nowait -m 512M,slots=2,maxmem=258779M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2 -device virtio-serial-pci,disable-legacy=on,id=serial0,iommu_platform=true -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/sbs/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc/console.sock,server,nowait -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=5 -machine memory-encryption=sev0 -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/sbs/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc/kata.sock,server,nowait -device virtio-9p-pci,disable-legacy=on,iommu_platform=true,fsdev=extra-9p-kataShared,mount_tag=kataShared -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc,security_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,disable-legacy=on,iommu_platform=true,mq=on,vectors=18 -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug panic=1 initcall_debug nr_cpus=240 ip=::::::4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc::off:: root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -drive if=pflash,format=raw,unit=0,file=/usr/local/share/qemu/OVMF_CODE.fd.kata,readonly -smp 1,cores=1,threads=1,sockets=1,maxcpus=240

Please also share your kata 2.0 qemu commandline. You can get it by: 1. start containerd with `--log-level debug` 2. look for it in containerd's output With both 1.x and 2.x qemu commandline, it would be more easier to check that is changed and why 2.0x doesn't boot with custom ovmf. Cheers, Tao -- Into something rich and strange.

Hi Sandeep,

On 6 Jan 2021, at 09:51, Christophe de Dinechin <cdupontd@redhat.com> wrote:

Hi Sandeep,

...

On 5 Jan 2021, at 21:19, Gupta, Sandeep <Sandeep.Gupta@amd.com> wrote:

[AMD Official Use Only - Internal Distribution Only]

Thanks again Christophe,

From my kata-runtime 1.x, setup :

root 2972741 2845780 0 20:13 pts/2 00:00:00 docker run --runtime kata-sev-runtime --cap-add=NET_ADMIN -it apache2-sev

root 2018662 2018416 0 15:01 ? 00:00:16 /usr/local/bin/qemu-system-x86_64 -name sandbox-4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc -uuid fce8220b-780b-4da6-8f5d-77ace1e06ed6 -machine q35,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/vc/sbs/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc/qmp.sock,server,nowait -m 512M,slots=2,maxmem=258779M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2 -device virtio-serial-pci,disable-legacy=on,id=serial0,iommu_platform=true -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/sbs/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc/console.sock,server,nowait -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=5 -machine memory-encryption=sev0 -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/sbs/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc/kata.sock,server,nowait -device virtio-9p-pci,disable-legacy=on,iommu_platform=true,fsdev=extra-9p-kataShared,mount_tag=kataShared -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc,security_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,disable-legacy=on,iommu_platform=true,mq=on,vectors=18 -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug panic=1 initcall_debug nr_cpus=240 ip=::::::4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc::off:: root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -drive if=pflash,format=raw,unit=0,file=/usr/local/share/qemu/OVMF_CODE.fd.kata,readonly -smp 1,cores=1,threads=1,sockets=1,maxcpus=240

As Fabiano pointed out (indirectly), there is no "-bios" option in the command line above. So either

I'm sorry, Fabiano pointed out that I somehow had managed to miss the -drive if=pflash,format=raw,unit=0,file=/usr/local/share/qemu/OVMF_CODE.fd.kata,readonly option. Not enough coffee, I guess. Disregard what is below. Instead, now we have an observable command-line difference between 1.x and 2.0. My guess is that this explains why you have trouble with 2.0. Even if my own quick experiments today showed that you can boot an OVMF with -bios, I'm not sure you can get SecureBoot that way, because of the missing vars partition. I've opened https://github.com/kata-containers/kata-containers/issues/1231. I will add a comment highlighting the difference.

1) You did that run without having the "firmware = " option. If that was the case, please send the command-line with the firmware option

(Hypothesis: maybe you are looking at the wrong configuration file, e.g. you have a local /etc/ override; that might explain why it "worked" with 1.x, although it would then require investigation to understand why SEV was enabled without your firmware)

2) Something else removes the firmware setting or causes qemu to not receive the -bios option

When "firmware" is specified, qemu should be invoked with the -bios option, and the corresponding commit dates back to early 2018, so I don't expect any difference between 1.x and 2.0 there.

Cheers, Christophe

...

root 2972841 2970388 0 20:13 ? 00:00:00 containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8 -address /var/run/docker/containerd/containerd.sock -containerd-binary /snap/docker/471/bin/containerd -runtime-root /var/run/docker/runtime-kata-sev-runtime -debug

root 2973736 2972841 0 20:13 ? 00:00:00 /usr/libexec/kata-containers/kata-proxy -listen-socket unix:///run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/proxy.sock -mux-socket /run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/kata.sock -log debug -agent-logs-socket /run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/console.sock

root 2973856 2972841 0 20:13 pts/4 00:00:00 /usr/libexec/kata-containers/kata-shim -agent unix:///run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/proxy.sock -container 54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8 -exec-id 54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8 -terminal -log debug

--Best Regards, --Sandeep Gupta

-----Original Message----- From: Christophe de Dinechin <cdupontd@redhat.com> Sent: Tuesday, January 5, 2021 11:25 AM To: Gupta, Sandeep <Sandeep.Gupta@amd.com> Cc: Fabiano Fidêncio <fidencio@redhat.com>; kata-dev@lists.katacontainers.io; Hollingsworth, Brent <brent.hollingsworth@amd.com> Subject: Re: [kata-dev] Regarding, Kata 2.0.0, kata-runtime, qemu OVMF

[CAUTION: External Email]

...

On 5 Jan 2021, at 17:21, Gupta, Sandeep <Sandeep.Gupta@amd.com> wrote:

...

Hi Developers, One more please - for Kata 2.0.0, kata-runtime, qemu OVMF

In /etc/kata-containers/configuration.toml --

====== # WARNING: - any parameter specified here will take priority over the default # parameter value of the same name used to start the virtual machine. # Do not set values here unless you understand the impact of doing so as you # may stop the virtual machine from booting. # To see the list of default parameters, enable hypervisor debug, create a # container and look for 'default-kernel-parameters' log entries. kernel_params = " root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug"

# Path to the firmware. # If you want that qemu uses the default firmware leave this option empty #firmware = "/usr/local/share/qemu/OVMF_CODE.fd.kata" firmware = "" ==========

If I use the default firmware - I get correct results :

# ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest sh / # / # uname -a Linux sandeepg-ubuntu 5.6.0-sev-sev-es-5.6-v3.container #3 SMP Tue Nov 3 06:38:40 UTC 2020 x86_64 GNU/Linux / #

# ps -eaf | grep -i qemu

root 360867 1 0 15:56 ? 00:00:01 /usr/local/bin/qemu-system-x86_64 -name sandbox-sh -uuid 72820914-dcce-4cea-a01f-3cf9851d136d -machine q35,accel=kvm,kernel_irqchip -cpu host,pmu=off -qmp unix:/run/vc/vm/sh/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=129755M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/sh/console.sock,server,nowait -device virtio-scsi-pci,id=scsi0,disable-modern=false,romfile= -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0,romfile= -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1189071429,guest-cid=1189071429,romfile= -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/sh/shared,security_model=none,multidevs=remap -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object memory-backend-ram,id=dimm1,size=2048M -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 pci=lastbus=0 quiet panic=1 nr_cpus=240 scsi_mod.scan=none root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -pidfile /run/vc/vm/sh/pid -smp 1,cores=1,threads=1,sockets=240,maxcpus=240 root 360966 360958 0 16:05 pts/2 00:00:00 grep --color=auto -i qemu

But if I use : firmware = "/usr/local/share/qemu/OVMF_CODE.fd.kata" I get Error:

# ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest s ctr: failed to launch qemu: exit status 1, error messages from qemu log: qemu: could not load PC BIOS '/usr/local/share/qemu/OVMF_CODE.fd.kata' : unknown

Any suggestions please.

I did a quick look. The runtime apparently uses the -bios option when the firmware option is passed, not the -drive option as I thought it would. I'm not entirely sure you can boot OVMF that way. I don't have any idea at the moment why it worked with version 1.x. If you could share with me the 1.x qemu command line option with the firmware option, I'm curious to see what it is.

Thanks Christophe

...

--Thanking You --Sandeep

_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists .katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-dev&data= 04%7C01%7Csandeep.gupta%40amd.com%7C5764f6fb89fd4809aafc08d8b1af94df%7 C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637454714974608155%7CUnkno wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL CJXVCI6Mn0%3D%7C1000&sdata=U85zx8%2FDKniPytZoOsm%2F7eR%2FNq5e8aHFm fJKMKlX%2B84%3D&reserved=0

[AMD Official Use Only - Internal Distribution Only] Hi All, I may have mixed up my working and not working cases - sorry about this, I have finally isolated my working and not-working cases, (for sure) as follows : 1. Not working : msg="launching /usr/local/bin/qemu-system-x86_64 with: [-name sandbox-csh -uuid 8b5369f5-1439-48d7-9b67-a32f6ef8ff7f -machine q35,accel=kvm,kernel_irqchip -cpu host, -qmp unix:/run/vc/vm/csh/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=129755M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/csh/console.sock,server,nowait -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0,romfile= -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/vm/csh/kata.sock,server,nowait -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/csh/shared,security_model=none,multidevs=remap -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object memory-backend-ram,id=dimm1,size=2048M -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 pci=lastbus=0 iommu=off debug panic=1 nr_cpus=240 agent.use_vsock=false root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -bios /usr/local/share/qemu/OVMF_CODE.fd.kata -pidfile /run/vc/vm/csh/pid -D /run/vc/vm/csh/qemu.log -smp 1,cores=1,threads=1,sockets=240,maxcpus=240]" 1. Working : /usr/local/bin/qemu-system-x86_64 -name sandbox-91d9f660fe1338d724c0ccd5c755a618d44100e697c82f8753d1051d2fdb30c1 -uuid e7888d8f-2e65-43d5-bc5a-3a3e853b3e13 -machine q35,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/vc/sbs/91d9f660fe1338d724c0ccd5c755a618d44100e697c82f8753d1051d2fdb30c1/qmp.sock,server,nowait -m 512M,slots=2,maxmem=258779M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2 -device virtio-serial-pci,disable-legacy=on,id=serial0,iommu_platform=true -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/sbs/91d9f660fe1338d724c0ccd5c755a618d44100e697c82f8753d1051d2fdb30c1/console.sock,server,nowait -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=5 -machine memory-encryption=sev0 -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/sbs/91d9f660fe1338d724c0ccd5c755a618d44100e697c82f8753d1051d2fdb30c1/kata.sock,server,nowait -device virtio-9p-pci,disable-legacy=on,iommu_platform=true,fsdev=extra-9p-kataShared,mount_tag=kataShared -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/91d9f660fe1338d724c0ccd5c755a618d44100e697c82f8753d1051d2fdb30c1,security_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,disable-legacy=on,iommu_platform=true,mq=on,vectors=18 -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug panic=1 initcall_debug nr_cpus=240 ip=::::::91d9f660fe1338d724c0ccd5c755a618d44100e697c82f8753d1051d2fdb30c1::off:: root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -drive if=pflash,format=raw,unit=0,file=/usr/local/share/qemu/OVMF_CODE.fd.kata,readonly -smp 1,cores=1,threads=1,sockets=1,maxcpus=240 --Best Regards, --Sandeep Gupta From: Christophe de Dinechin <cdupontd@redhat.com> Sent: Wednesday, January 6, 2021 7:00 AM To: Gupta, Sandeep <Sandeep.Gupta@amd.com> Cc: Fabiano Fidêncio <fidencio@redhat.com>; kata-dev@lists.katacontainers.io; Hollingsworth, Brent <brent.hollingsworth@amd.com> Subject: Re: [kata-dev] Regarding, Kata 2.0.0, kata-runtime, qemu OVMF [CAUTION: External Email] Hi Sandeep, On 6 Jan 2021, at 15:19, Christophe de Dinechin <cdupontd@redhat.com<mailto:cdupontd@redhat.com>> wrote: Hi Sandeep, On 6 Jan 2021, at 09:51, Christophe de Dinechin <cdupontd@redhat.com<mailto:cdupontd@redhat.com>> wrote: Hi Sandeep, On 5 Jan 2021, at 21:19, Gupta, Sandeep <Sandeep.Gupta@amd.com<mailto:Sandeep.Gupta@amd.com>> wrote: [AMD Official Use Only - Internal Distribution Only] Thanks again Christophe,

From my kata-runtime 1.x, setup :

root 2972741 2845780 0 20:13 pts/2 00:00:00 docker run --runtime kata-sev-runtime --cap-add=NET_ADMIN -it apache2-sev root 2018662 2018416 0 15:01 ? 00:00:16 /usr/local/bin/qemu-system-x86_64 -name sandbox-4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc -uuid fce8220b-780b-4da6-8f5d-77ace1e06ed6 -machine q35,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/vc/sbs/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc/qmp.sock,server,nowait -m 512M,slots=2,maxmem=258779M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2 -device virtio-serial-pci,disable-legacy=on,id=serial0,iommu_platform=true -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/sbs/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc/console.sock,server,nowait -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=5 -machine memory-encryption=sev0 -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/sbs/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc/kata.sock,server,nowait -device virtio-9p-pci,disable-legacy=on,iommu_platform=true,fsdev=extra-9p-kataShared,mount_tag=kataShared -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc,security_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,disable-legacy=on,iommu_platform=true,mq=on,vectors=18 -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug panic=1 initcall_debug nr_cpus=240 ip=::::::4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc::off:: root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -drive if=pflash,format=raw,unit=0,file=/usr/local/share/qemu/OVMF_CODE.fd.kata,readonly -smp 1,cores=1,threads=1,sockets=1,maxcpus=240 As Fabiano pointed out (indirectly), there is no "-bios" option in the command line above. So either I'm sorry, Fabiano pointed out that I somehow had managed to miss the -drive if=pflash,format=raw,unit=0,file=/usr/local/share/qemu/OVMF_CODE.fd.kata,readonly option. Not enough coffee, I guess. There is something mysterious about this, as I pointed out in https://github.com/kata-containers/kata-containers/issues/1231#issuecomment-755333403<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkata-containers%2Fkata-containers%2Fissues%2F1231%23issuecomment-755333403&data=04%7C01%7Csandeep.gupta%40amd.com%7Cad9f58a692c841c2823508d8b253c2b1%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637455420113626564%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=mjbC4X7djDhpehe3PwNyz2ECrW7F5FeMMeMA4aYiLxQ%3D&reserved=0>. Basically, govmm does not seem to emit that option. func (config *Config) appendBios() { if config.Bios != "" { config.qemuParams = append(config.qemuParams, "-bios") config.qemuParams = append(config.qemuParams, config.Bios) } } It apparently has always been like this. Do you remember either patching that file, or maybe finding some other way to pass the OVMF arguments to qemu (like using a qemu wrapper)? Disregard what is below. Instead, now we have an observable command-line difference between 1.x and 2.0. My guess is that this explains why you have trouble with 2.0. Even if my own quick experiments today showed that you can boot an OVMF with -bios, I'm not sure you can get SecureBoot that way, because of the missing vars partition. I've opened https://github.com/kata-containers/kata-containers/issues/1231<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkata-containers%2Fkata-containers%2Fissues%2F1231&data=04%7C01%7Csandeep.gupta%40amd.com%7Cad9f58a692c841c2823508d8b253c2b1%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637455420113636556%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=bmWstpZedr0v1yYOjCXcOH37Z1KOTR3zxy2F%2FnCXmF0%3D&reserved=0>. I will add a comment highlighting the difference. 1) You did that run without having the "firmware = " option. If that was the case, please send the command-line with the firmware option (Hypothesis: maybe you are looking at the wrong configuration file, e.g. you have a local /etc/ override; that might explain why it "worked" with 1.x, although it would then require investigation to understand why SEV was enabled without your firmware) 2) Something else removes the firmware setting or causes qemu to not receive the -bios option When "firmware" is specified, qemu should be invoked with the -bios option, and the corresponding commit dates back to early 2018, so I don't expect any difference between 1.x and 2.0 there. Cheers, Christophe root 2972841 2970388 0 20:13 ? 00:00:00 containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8 -address /var/run/docker/containerd/containerd.sock -containerd-binary /snap/docker/471/bin/containerd -runtime-root /var/run/docker/runtime-kata-sev-runtime -debug root 2973736 2972841 0 20:13 ? 00:00:00 /usr/libexec/kata-containers/kata-proxy -listen-socket unix:///run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/proxy.sock<unix://run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/proxy.sock> -mux-socket /run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/kata.sock -log debug -agent-logs-socket /run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/console.sock root 2973856 2972841 0 20:13 pts/4 00:00:00 /usr/libexec/kata-containers/kata-shim -agent unix:///run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/proxy.sock<unix://run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/proxy.sock> -container 54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8 -exec-id 54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8 -terminal -log debug --Best Regards, --Sandeep Gupta -----Original Message----- From: Christophe de Dinechin <cdupontd@redhat.com<mailto:cdupontd@redhat.com>> Sent: Tuesday, January 5, 2021 11:25 AM To: Gupta, Sandeep <Sandeep.Gupta@amd.com<mailto:Sandeep.Gupta@amd.com>> Cc: Fabiano Fidêncio <fidencio@redhat.com<mailto:fidencio@redhat.com>>; kata-dev@lists.katacontainers.io<mailto:kata-dev@lists.katacontainers.io>; Hollingsworth, Brent <brent.hollingsworth@amd.com<mailto:brent.hollingsworth@amd.com>> Subject: Re: [kata-dev] Regarding, Kata 2.0.0, kata-runtime, qemu OVMF [CAUTION: External Email] On 5 Jan 2021, at 17:21, Gupta, Sandeep <Sandeep.Gupta@amd.com<mailto:Sandeep.Gupta@amd.com>> wrote: Hi Developers, One more please - for Kata 2.0.0, kata-runtime, qemu OVMF In /etc/kata-containers/configuration.toml -- ====== # WARNING: - any parameter specified here will take priority over the default # parameter value of the same name used to start the virtual machine. # Do not set values here unless you understand the impact of doing so as you # may stop the virtual machine from booting. # To see the list of default parameters, enable hypervisor debug, create a # container and look for 'default-kernel-parameters' log entries. kernel_params = " root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug" # Path to the firmware. # If you want that qemu uses the default firmware leave this option empty #firmware = "/usr/local/share/qemu/OVMF_CODE.fd.kata" firmware = "" ========== If I use the default firmware - I get correct results : # ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdocker.io%2Flibrary%2Fbusybox%3Alatest&data=04%7C01%7Csandeep.gupta%40amd.com%7Cad9f58a692c841c2823508d8b253c2b1%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637455420113636556%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ydvwfcetd0OtZk8JegCBHp88E3FaEkJ8wss86Q8HlGs%3D&reserved=0> sh / # / # uname -a Linux sandeepg-ubuntu 5.6.0-sev-sev-es-5.6-v3.container #3 SMP Tue Nov 3 06:38:40 UTC 2020 x86_64 GNU/Linux / # # ps -eaf | grep -i qemu root 360867 1 0 15:56 ? 00:00:01 /usr/local/bin/qemu-system-x86_64 -name sandbox-sh -uuid 72820914-dcce-4cea-a01f-3cf9851d136d -machine q35,accel=kvm,kernel_irqchip -cpu host,pmu=off -qmp unix:/run/vc/vm/sh/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=129755M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/sh/console.sock,server,nowait -device virtio-scsi-pci,id=scsi0,disable-modern=false,romfile= -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0,romfile= -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1189071429,guest-cid=1189071429,romfile= -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/sh/shared,security_model=none,multidevs=remap -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object memory-backend-ram,id=dimm1,size=2048M -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 pci=lastbus=0 quiet panic=1 nr_cpus=240 scsi_mod.scan=none root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -pidfile /run/vc/vm/sh/pid -smp 1,cores=1,threads=1,sockets=240,maxcpus=240 root 360966 360958 0 16:05 pts/2 00:00:00 grep --color=auto -i qemu But if I use : firmware = "/usr/local/share/qemu/OVMF_CODE.fd.kata" I get Error: # ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdocker.io%2Flibrary%2Fbusybox%3Alatest&data=04%7C01%7Csandeep.gupta%40amd.com%7Cad9f58a692c841c2823508d8b253c2b1%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637455420113646549%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=1RG1Xo7OIM%2FiffFhzJKRbahB%2Bd9bP7%2BrTijGuPwDjvs%3D&reserved=0> s ctr: failed to launch qemu: exit status 1, error messages from qemu log: qemu: could not load PC BIOS '/usr/local/share/qemu/OVMF_CODE.fd.kata' : unknown Any suggestions please. I did a quick look. The runtime apparently uses the -bios option when the firmware option is passed, not the -drive option as I thought it would. I'm not entirely sure you can boot OVMF that way. I don't have any idea at the moment why it worked with version 1.x. If you could share with me the 1.x qemu command line option with the firmware option, I'm curious to see what it is. Thanks Christophe --Thanking You --Sandeep _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io<mailto:kata-dev@lists.katacontainers.io> https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists .katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-dev&data= 04%7C01%7Csandeep.gupta%40amd.com%7C5764f6fb89fd4809aafc08d8b1af94df%7 C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637454714974608155%7CUnkno wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL CJXVCI6Mn0%3D%7C1000&sdata=U85zx8%2FDKniPytZoOsm%2F7eR%2FNq5e8aHFm fJKMKlX%2B84%3D&reserved=0

[AMD Official Use Only - Internal Distribution Only] Thank you Christophe, Yes - We change .../vendor/src/github.com/intel/govmm/qemu/qemu.go, "appendBios()" func (config *Config) appendBios() { if config.Bios != "" { if config.Knobs.MemEncrypt == true { config.qemuParams = append(config.qemuParams, "-drive") config.qemuParams = append(config.qemuParams, fmt.Sprintf("if=pflash,format=raw,unit=0,file=%s,readonly", config.Bios)) } else { config.qemuParams = append(config.qemuParams, "-bios") config.qemuParams = append(config.qemuParams, config.Bios) } } } But then "containerd" does not use this method ? --Thanking You --Sandeep -----Original Message----- From: Christophe de Dinechin <cdupontd@redhat.com> Sent: Thursday, January 7, 2021 12:43 AM To: Gupta, Sandeep <Sandeep.Gupta@amd.com> Cc: Fabiano Fidêncio <fidencio@redhat.com>; kata-dev@lists.katacontainers.io; Hollingsworth, Brent <brent.hollingsworth@amd.com> Subject: Re: [kata-dev] Regarding, Kata 2.0.0, kata-runtime, qemu OVMF Importance: High [CAUTION: External Email]

On 7 Jan 2021, at 09:01, Gupta, Sandeep <Sandeep.Gupta@amd.com> wrote:

[AMD Official Use Only - Internal Distribution Only]

Hi All,

I may have mixed up my working and not working cases - sorry about this, I have finally isolated my working and not-working cases, (for sure) as follows :

This seems consistent with what we deduced from the earlier exchanges. In short:

. Not working :

msg="launching /usr/local/bin/qemu-system-x86_64 with: [-name sandbox-csh -uuid 8b5369f5-1439-48d7-9b67-a32f6ef8ff7f -machine q35,accel=kvm,kernel_irqchip -cpu host, -qmp unix:/run/vc/vm/csh/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=129755M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/csh/console.sock,server,nowait -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0,romfile= -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/vm/csh/kata.sock,server,nowait -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/csh/shared,security_model=none,multidevs=remap -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object memory-backend-ram,id=dimm1,size=2048M -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 pci=lastbus=0 iommu=off debug panic=1 nr_cpus=240 agent.use_vsock=false root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -bios /usr/local/share/qemu/OVMF_CODE.fd.kata -pidfile /run/vc/vm/csh/pid -D /run/vc/vm/csh/qemu.log -smp 1,cores=1,threads=1,sockets=240,maxcpus=240]"

So this one uses the -bios option. -bios /usr/local/share/qemu/OVMF_CODE.fd.kata I believe you told us this is what 2.0 does, right?

. Working :

/usr/local/bin/qemu-system-x86_64 -name sandbox-91d9f660fe1338d724c0ccd5c755a618d44100e697c82f8753d1051d2fdb30 c1 -uuid e7888d8f-2e65-43d5-bc5a-3a3e853b3e13 -machine q35,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/vc/sbs/91d9f660fe1338d724c0ccd5c755a618d44100e697c82f8753d10 51d2fdb30c1/qmp.sock,server,nowait -m 512M,slots=2,maxmem=258779M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2 -device virtio-serial-pci,disable-legacy=on,id=serial0,iommu_platform=true -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/sbs/91d9f660fe1338d724c0ccd5c755a6 18d44100e697c82f8753d1051d2fdb30c1/console.sock,server,nowait -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=5 -machine memory-encryption=sev0 -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/sbs/91d9f660fe1338d724c0ccd5c755a618d44 100e697c82f8753d1051d2fdb30c1/kata.sock,server,nowait -device virtio-9p-pci,disable-legacy=on,iommu_platform=true,fsdev=extra-9p-kat aShared,mount_tag=kataShared -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxe s/91d9f660fe1338d724c0ccd5c755a618d44100e697c82f8753d1051d2fdb30c1,sec urity_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15 :16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,disable-l egacy=on,iommu_platform=true,mq=on,vectors=18 -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug panic=1 initcall_debug nr_cpus=240 ip=::::::91d9f660fe1338d724c0ccd5c755a618d44100e697c82f8753d1051d2fdb3 0c1::off:: root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -drive if=pflash,format=raw,unit=0,file=/usr/local/share/qemu/OVMF_CODE.fd.ka ta,readonly -smp 1,cores=1,threads=1,sockets=1,maxcpus=240

This one uses -drive -drive if=pflash,format=raw,unit=0,file=/usr/local/share/qemu/OVMF_CODE.fd.kata,readonly I believe this was your 1.x demo, right? Now, this -drive is good, but it is unexpected, because I see nowhere in the normal Kata code that would emit that. This suggests you use a version I don't know about, or maybe that there is for example a patch that was sent to a release but dropped from the 2.0 branch. Further, please note that Laszlo Ersek seems to imply in a parallel thread that -drive is no longer the right way to do it either. We may need to implement qemu-version dependent options.

--Best Regards, --Sandeep Gupta

From: Christophe de Dinechin <cdupontd@redhat.com> Sent: Wednesday, January 6, 2021 7:00 AM To: Gupta, Sandeep <Sandeep.Gupta@amd.com> Cc: Fabiano Fidêncio <fidencio@redhat.com>; kata-dev@lists.katacontainers.io; Hollingsworth, Brent <brent.hollingsworth@amd.com> Subject: Re: [kata-dev] Regarding, Kata 2.0.0, kata-runtime, qemu OVMF

[CAUTION: External Email] Hi Sandeep,

On 6 Jan 2021, at 15:19, Christophe de Dinechin <cdupontd@redhat.com> wrote:

Hi Sandeep,

On 6 Jan 2021, at 09:51, Christophe de Dinechin <cdupontd@redhat.com> wrote:

Hi Sandeep,

On 5 Jan 2021, at 21:19, Gupta, Sandeep <Sandeep.Gupta@amd.com> wrote:

[AMD Official Use Only - Internal Distribution Only]

Thanks again Christophe,

From my kata-runtime 1.x, setup :

root 2972741 2845780 0 20:13 pts/2 00:00:00 docker run --runtime kata-sev-runtime --cap-add=NET_ADMIN -it apache2-sev

root 2018662 2018416 0 15:01 ? 00:00:16 /usr/local/bin/qemu-system-x86_64 -name sandbox-4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc -uuid fce8220b-780b-4da6-8f5d-77ace1e06ed6 -machine q35,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/vc/sbs/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc/qmp.sock,server,nowait -m 512M,slots=2,maxmem=258779M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2 -device virtio-serial-pci,disable-legacy=on,id=serial0,iommu_platform=true -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/sbs/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc/console.sock,server,nowait -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=5 -machine memory-encryption=sev0 -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/sbs/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc/kata.sock,server,nowait -device virtio-9p-pci,disable-legacy=on,iommu_platform=true,fsdev=extra-9p-kataShared,mount_tag=kataShared -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc,security_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,disable-legacy=on,iommu_platform=true,mq=on,vectors=18 -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug panic=1 initcall_debug nr_cpus=240 ip=::::::4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc::off:: root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -drive if=pflash,format=raw,unit=0,file=/usr/local/share/qemu/OVMF_CODE.fd.kata,readonly -smp 1,cores=1,threads=1,sockets=1,maxcpus=240

As Fabiano pointed out (indirectly), there is no "-bios" option in the command line above. So either

I'm sorry, Fabiano pointed out that I somehow had managed to miss the -drive if=pflash,format=raw,unit=0,file=/usr/local/share/qemu/OVMF_CODE.fd.kata,readonly option. Not enough coffee, I guess.

There is something mysterious about this, as I pointed out in https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com.... Basically, govmm does not seem to emit that option.

func (config *Config) appendBios() { if config.Bios != "" { config.qemuParams = append(config.qemuParams, "-bios") config.qemuParams = append(config.qemuParams, config.Bios) } }

It apparently has always been like this. Do you remember either patching that file, or maybe finding some other way to pass the OVMF arguments to qemu (like using a qemu wrapper)?

Disregard what is below. Instead, now we have an observable command-line difference between 1.x and 2.0. My guess is that this explains why you have trouble with 2.0. Even if my own quick experiments today showed that you can boot an OVMF with -bios, I'm not sure you can get SecureBoot that way, because of the missing vars partition.

I've opened https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com.... I will add a comment highlighting the difference.

1) You did that run without having the "firmware = " option. If that was the case, please send the command-line with the firmware option

(Hypothesis: maybe you are looking at the wrong configuration file, e.g. you have a local /etc/ override; that might explain why it "worked" with 1.x, although it would then require investigation to understand why SEV was enabled without your firmware)

2) Something else removes the firmware setting or causes qemu to not receive the -bios option

When "firmware" is specified, qemu should be invoked with the -bios option, and the corresponding commit dates back to early 2018, so I don't expect any difference between 1.x and 2.0 there.

Cheers, Christophe

root 2972841 2970388 0 20:13 ? 00:00:00 containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8 -address /var/run/docker/containerd/containerd.sock -containerd-binary /snap/docker/471/bin/containerd -runtime-root /var/run/docker/runtime-kata-sev-runtime -debug

root 2973736 2972841 0 20:13 ? 00:00:00 /usr/libexec/kata-containers/kata-proxy -listen-socket unix:///run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/proxy.sock -mux-socket /run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/kata.sock -log debug -agent-logs-socket /run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/console.sock

root 2973856 2972841 0 20:13 pts/4 00:00:00 /usr/libexec/kata-containers/kata-shim -agent unix:///run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/proxy.sock -container 54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8 -exec-id 54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8 -terminal -log debug

--Best Regards, --Sandeep Gupta

-----Original Message----- From: Christophe de Dinechin <cdupontd@redhat.com> Sent: Tuesday, January 5, 2021 11:25 AM To: Gupta, Sandeep <Sandeep.Gupta@amd.com> Cc: Fabiano Fidêncio <fidencio@redhat.com>; kata-dev@lists.katacontainers.io; Hollingsworth, Brent <brent.hollingsworth@amd.com> Subject: Re: [kata-dev] Regarding, Kata 2.0.0, kata-runtime, qemu OVMF

[CAUTION: External Email]

On 5 Jan 2021, at 17:21, Gupta, Sandeep <Sandeep.Gupta@amd.com> wrote:

Hi Developers, One more please - for Kata 2.0.0, kata-runtime, qemu OVMF

In /etc/kata-containers/configuration.toml --

====== # WARNING: - any parameter specified here will take priority over the default # parameter value of the same name used to start the virtual machine. # Do not set values here unless you understand the impact of doing so as you # may stop the virtual machine from booting. # To see the list of default parameters, enable hypervisor debug, create a # container and look for 'default-kernel-parameters' log entries. kernel_params = " root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug"

# Path to the firmware. # If you want that qemu uses the default firmware leave this option empty #firmware = "/usr/local/share/qemu/OVMF_CODE.fd.kata" firmware = "" ==========

If I use the default firmware - I get correct results :

# ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest sh / # / # uname -a Linux sandeepg-ubuntu 5.6.0-sev-sev-es-5.6-v3.container #3 SMP Tue Nov 3 06:38:40 UTC 2020 x86_64 GNU/Linux / #

# ps -eaf | grep -i qemu

root 360867 1 0 15:56 ? 00:00:01 /usr/local/bin/qemu-system-x86_64 -name sandbox-sh -uuid 72820914-dcce-4cea-a01f-3cf9851d136d -machine q35,accel=kvm,kernel_irqchip -cpu host,pmu=off -qmp unix:/run/vc/vm/sh/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=129755M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/sh/console.sock,server,nowait -device virtio-scsi-pci,id=scsi0,disable-modern=false,romfile= -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0,romfile= -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1189071429,guest-cid=1189071429,romfile= -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/sh/shared,security_model=none,multidevs=remap -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object memory-backend-ram,id=dimm1,size=2048M -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 pci=lastbus=0 quiet panic=1 nr_cpus=240 scsi_mod.scan=none root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -pidfile /run/vc/vm/sh/pid -smp 1,cores=1,threads=1,sockets=240,maxcpus=240 root 360966 360958 0 16:05 pts/2 00:00:00 grep --color=auto -i qemu

But if I use : firmware = "/usr/local/share/qemu/OVMF_CODE.fd.kata" I get Error:

# ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest s ctr: failed to launch qemu: exit status 1, error messages from qemu log: qemu: could not load PC BIOS '/usr/local/share/qemu/OVMF_CODE.fd.kata' : unknown

Any suggestions please.

I did a quick look. The runtime apparently uses the -bios option when the firmware option is passed, not the -drive option as I thought it would. I'm not entirely sure you can boot OVMF that way. I don't have any idea at the moment why it worked with version 1.x. If you could share with me the 1.x qemu command line option with the firmware option, I'm curious to see what it is.

Thanks Christophe

--Thanking You --Sandeep

_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists .katacontainers.io%2Fcgi-bin%2Fmailman%2Flistinfo%2Fkata-dev&data= 04%7C01%7Csandeep.gupta%40amd.com%7C5764f6fb89fd4809aafc08d8b1af94df%7 C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637454714974608155%7CUnkno wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL CJXVCI6Mn0%3D%7C1000&sdata=U85zx8%2FDKniPytZoOsm%2F7eR%2FNq5e8aHFm fJKMKlX%2B84%3D&reserved=0

On 8 Jan 2021, at 21:16, Gupta, Sandeep <Sandeep.Gupta@amd.com <mailto:Sandeep.Gupta@amd.com>> wrote:

[AMD Official Use Only - Internal Distribution Only]

Thank you Christophe,

Yes - We change .../vendor/src/github.com/intel/govmm/qemu/qemu.go <http://github.com/intel/govmm/qemu/qemu.go>, "appendBios()"

func (config *Config) appendBios() { if config.Bios != "" { if config.Knobs.MemEncrypt == true { config.qemuParams = append(config.qemuParams, "-drive") config.qemuParams = append(config.qemuParams, fmt.Sprintf("if=pflash,format=raw,unit=0,file=%s,readonly", config.Bios)) } else { config.qemuParams = append(config.qemuParams, "-bios") config.qemuParams = append(config.qemuParams, config.Bios) } } }

So I see two problems with that change: 1/ I think it depends on using OVMF, not on encrypted memory. 2/ You need the VARS file as discussed earlier. I believe that a more robust solution would be: 1/ Add an OVMF option alongside "firmware", as well as OVMF_VARS 2/ When OVMF is specified, create a copy of the VARS file, and pass two -drive options, one for OVMF_CODE and one for the copied OVMF_VARS. 3/ When MemEncrypt is true, check that you ahve an OVMF given and no bios.

But then "containerd" does not use this method ?

--Thanking You --Sandeep

-----Original Message----- From: Christophe de Dinechin <cdupontd@redhat.com <mailto:cdupontd@redhat.com>> Sent: Thursday, January 7, 2021 12:43 AM To: Gupta, Sandeep <Sandeep.Gupta@amd.com <mailto:Sandeep.Gupta@amd.com>> Cc: Fabiano Fidêncio <fidencio@redhat.com <mailto:fidencio@redhat.com>>; kata-dev@lists.katacontainers.io <mailto:kata-dev@lists.katacontainers.io>; Hollingsworth, Brent <brent.hollingsworth@amd.com <mailto:brent.hollingsworth@amd.com>> Subject: Re: [kata-dev] Regarding, Kata 2.0.0, kata-runtime, qemu OVMF Importance: High

[CAUTION: External Email]

...

On 7 Jan 2021, at 09:01, Gupta, Sandeep <Sandeep.Gupta@amd.com <mailto:Sandeep.Gupta@amd.com>> wrote:

[AMD Official Use Only - Internal Distribution Only]

Hi All,

I may have mixed up my working and not working cases - sorry about this, I have finally isolated my working and not-working cases, (for sure) as follows :

This seems consistent with what we deduced from the earlier exchanges.

In short:

...

. Not working :

msg="launching /usr/local/bin/qemu-system-x86_64 with: [-name sandbox-csh -uuid 8b5369f5-1439-48d7-9b67-a32f6ef8ff7f -machine q35,accel=kvm,kernel_irqchip -cpu host, -qmp unix:/run/vc/vm/csh/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=129755M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/csh/console.sock,server,nowait -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0,romfile= -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/vm/csh/kata.sock,server,nowait -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/csh/shared,security_model=none,multidevs=remap -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object memory-backend-ram,id=dimm1,size=2048M -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 pci=lastbus=0 iommu=off debug panic=1 nr_cpus=240 agent.use_vsock=false root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -bios /usr/local/share/qemu/OVMF_CODE.fd.kata -pidfile /run/vc/vm/csh/pid -D /run/vc/vm/csh/qemu.log -smp 1,cores=1,threads=1,sockets=240,maxcpus=240]"

So this one uses the -bios option. -bios /usr/local/share/qemu/OVMF_CODE.fd.kata

I believe you told us this is what 2.0 does, right?

...

. Working :

/usr/local/bin/qemu-system-x86_64 -name sandbox-91d9f660fe1338d724c0ccd5c755a618d44100e697c82f8753d1051d2fdb30 c1 -uuid e7888d8f-2e65-43d5-bc5a-3a3e853b3e13 -machine q35,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/vc/sbs/91d9f660fe1338d724c0ccd5c755a618d44100e697c82f8753d10 51d2fdb30c1/qmp.sock,server,nowait -m 512M,slots=2,maxmem=258779M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2 -device virtio-serial-pci,disable-legacy=on,id=serial0,iommu_platform=true -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/sbs/91d9f660fe1338d724c0ccd5c755a6 18d44100e697c82f8753d1051d2fdb30c1/console.sock,server,nowait -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=5 -machine memory-encryption=sev0 -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/sbs/91d9f660fe1338d724c0ccd5c755a618d44 100e697c82f8753d1051d2fdb30c1/kata.sock,server,nowait -device virtio-9p-pci,disable-legacy=on,iommu_platform=true,fsdev=extra-9p-kat aShared,mount_tag=kataShared -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxe s/91d9f660fe1338d724c0ccd5c755a618d44100e697c82f8753d1051d2fdb30c1,sec urity_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15 :16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,disable-l egacy=on,iommu_platform=true,mq=on,vectors=18 -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug panic=1 initcall_debug nr_cpus=240 ip=::::::91d9f660fe1338d724c0ccd5c755a618d44100e697c82f8753d1051d2fdb3 0c1::off:: root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -drive if=pflash,format=raw,unit=0,file=/usr/local/share/qemu/OVMF_CODE.fd.ka ta,readonly -smp 1,cores=1,threads=1,sockets=1,maxcpus=240

This one uses -drive -drive if=pflash,format=raw,unit=0,file=/usr/local/share/qemu/OVMF_CODE.fd.kata,readonly

I believe this was your 1.x demo, right?

Now, this -drive is good, but it is unexpected, because I see nowhere in the normal Kata code that would emit that. This suggests you use a version I don't know about, or maybe that there is for example a patch that was sent to a release but dropped from the 2.0 branch.

Further, please note that Laszlo Ersek seems to imply in a parallel thread that -drive is no longer the right way to do it either. We may need to implement qemu-version dependent options.

...

--Best Regards, --Sandeep Gupta

From: Christophe de Dinechin <cdupontd@redhat.com <mailto:cdupontd@redhat.com>> Sent: Wednesday, January 6, 2021 7:00 AM To: Gupta, Sandeep <Sandeep.Gupta@amd.com <mailto:Sandeep.Gupta@amd.com>> Cc: Fabiano Fidêncio <fidencio@redhat.com <mailto:fidencio@redhat.com>>; kata-dev@lists.katacontainers.io <mailto:kata-dev@lists.katacontainers.io>; Hollingsworth, Brent <brent.hollingsworth@amd.com <mailto:brent.hollingsworth@amd.com>> Subject: Re: [kata-dev] Regarding, Kata 2.0.0, kata-runtime, qemu OVMF

[CAUTION: External Email] Hi Sandeep,

On 6 Jan 2021, at 15:19, Christophe de Dinechin <cdupontd@redhat.com <mailto:cdupontd@redhat.com>> wrote:

Hi Sandeep,

On 6 Jan 2021, at 09:51, Christophe de Dinechin <cdupontd@redhat.com <mailto:cdupontd@redhat.com>> wrote:

Hi Sandeep,

On 5 Jan 2021, at 21:19, Gupta, Sandeep <Sandeep.Gupta@amd.com <mailto:Sandeep.Gupta@amd.com>> wrote:

[AMD Official Use Only - Internal Distribution Only]

Thanks again Christophe,

From my kata-runtime 1.x, setup :

root 2972741 2845780 0 20:13 pts/2 00:00:00 docker run --runtime kata-sev-runtime --cap-add=NET_ADMIN -it apache2-sev

root 2018662 2018416 0 15:01 ? 00:00:16 /usr/local/bin/qemu-system-x86_64 -name sandbox-4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc -uuid fce8220b-780b-4da6-8f5d-77ace1e06ed6 -machine q35,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/vc/sbs/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc/qmp.sock,server,nowait -m 512M,slots=2,maxmem=258779M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2 -device virtio-serial-pci,disable-legacy=on,id=serial0,iommu_platform=true -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/sbs/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc/console.sock,server,nowait -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=5 -machine memory-encryption=sev0 -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/sbs/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc/kata.sock,server,nowait -device virtio-9p-pci,disable-legacy=on,iommu_platform=true,fsdev=extra-9p-kataShared,mount_tag=kataShared -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc,security_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3:4:5:6:7:8:9:10,fds=11:12:13:14:15:16:17:18 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,disable-legacy=on,iommu_platform=true,mq=on,vectors=18 -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug panic=1 initcall_debug nr_cpus=240 ip=::::::4a28f6d66e95a08be407b10e1fbfd6d0bd41d4048b5823029f02a17b7200adbc::off:: root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -drive if=pflash,format=raw,unit=0,file=/usr/local/share/qemu/OVMF_CODE.fd.kata,readonly -smp 1,cores=1,threads=1,sockets=1,maxcpus=240

As Fabiano pointed out (indirectly), there is no "-bios" option in the command line above. So either

I'm sorry, Fabiano pointed out that I somehow had managed to miss the -drive if=pflash,format=raw,unit=0,file=/usr/local/share/qemu/OVMF_CODE.fd.kata,readonly option. Not enough coffee, I guess.

There is something mysterious about this, as I pointed out in https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com... <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkata-containers%2Fkata-containers%2Fissues%2F1231%23issuecomment-755333403&data=04%7C01%7Csandeep.gupta%40amd.com>%7C3d884bd4e4904f858e6308d8b2e83de9%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637456057849018749%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=6Gm0u0gxkCbpYwD6eSnpi99BtnGszhvyWVV%2Bzfs2dMc%3D&reserved=0. Basically, govmm does not seem to emit that option.

func (config *Config) appendBios() { if config.Bios != "" { config.qemuParams = append(config.qemuParams, "-bios") config.qemuParams = append(config.qemuParams, config.Bios) } }

It apparently has always been like this. Do you remember either patching that file, or maybe finding some other way to pass the OVMF arguments to qemu (like using a qemu wrapper)?

Disregard what is below. Instead, now we have an observable command-line difference between 1.x and 2.0. My guess is that this explains why you have trouble with 2.0. Even if my own quick experiments today showed that you can boot an OVMF with -bios, I'm not sure you can get SecureBoot that way, because of the missing vars partition.

I've opened https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com... <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkata-containers%2Fkata-containers%2Fissues%2F1231&data=04%7C01%7Csandeep.gupta%40amd.com>%7C3d884bd4e4904f858e6308d8b2e83de9%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637456057849018749%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=tUAcbzpVNvm8iiSX1UuOcLAjwcLTNBNQY1MUEAu2%2Bf0%3D&reserved=0. I will add a comment highlighting the difference.

1) You did that run without having the "firmware = " option. If that was the case, please send the command-line with the firmware option

(Hypothesis: maybe you are looking at the wrong configuration file, e.g. you have a local /etc/ override; that might explain why it "worked" with 1.x, although it would then require investigation to understand why SEV was enabled without your firmware)

2) Something else removes the firmware setting or causes qemu to not receive the -bios option

When "firmware" is specified, qemu should be invoked with the -bios option, and the corresponding commit dates back to early 2018, so I don't expect any difference between 1.x and 2.0 there.

Cheers, Christophe

root 2972841 2970388 0 20:13 ? 00:00:00 containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8 -address /var/run/docker/containerd/containerd.sock -containerd-binary /snap/docker/471/bin/containerd -runtime-root /var/run/docker/runtime-kata-sev-runtime -debug

root 2973736 2972841 0 20:13 ? 00:00:00 /usr/libexec/kata-containers/kata-proxy -listen-socket unix:///run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/proxy.sock <unix:///run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/proxy.sock> -mux-socket /run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/kata.sock -log debug -agent-logs-socket /run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/console.sock

root 2973856 2972841 0 20:13 pts/4 00:00:00 /usr/libexec/kata-containers/kata-shim -agent unix:///run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/proxy.sock <unix:///run/vc/sbs/54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8/proxy.sock> -container 54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8 -exec-id 54db6810ae4ed8ab3e7a378c4e95d73e334797a5a5234978b4f976750f4f64d8 -terminal -log debug

--Best Regards, --Sandeep Gupta

-----Original Message----- From: Christophe de Dinechin <cdupontd@redhat.com <mailto:cdupontd@redhat.com>> Sent: Tuesday, January 5, 2021 11:25 AM To: Gupta, Sandeep <Sandeep.Gupta@amd.com <mailto:Sandeep.Gupta@amd.com>> Cc: Fabiano Fidêncio <fidencio@redhat.com <mailto:fidencio@redhat.com>>; kata-dev@lists.katacontainers.io <mailto:kata-dev@lists.katacontainers.io>; Hollingsworth, Brent <brent.hollingsworth@amd.com <mailto:brent.hollingsworth@amd.com>> Subject: Re: [kata-dev] Regarding, Kata 2.0.0, kata-runtime, qemu OVMF

[CAUTION: External Email]

On 5 Jan 2021, at 17:21, Gupta, Sandeep <Sandeep.Gupta@amd.com <mailto:Sandeep.Gupta@amd.com>> wrote:

Hi Developers, One more please - for Kata 2.0.0, kata-runtime, qemu OVMF

In /etc/kata-containers/configuration.toml --

====== # WARNING: - any parameter specified here will take priority over the default # parameter value of the same name used to start the virtual machine. # Do not set values here unless you understand the impact of doing so as you # may stop the virtual machine from booting. # To see the list of default parameters, enable hypervisor debug, create a # container and look for 'default-kernel-parameters' log entries. kernel_params = " root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug"

# Path to the firmware. # If you want that qemu uses the default firmware leave this option empty #firmware = "/usr/local/share/qemu/OVMF_CODE.fd.kata" firmware = "" ==========

If I use the default firmware - I get correct results :

# ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest <http://docker.io/library/busybox:latest> sh / # / # uname -a Linux sandeepg-ubuntu 5.6.0-sev-sev-es-5.6-v3.container #3 SMP Tue Nov 3 06:38:40 UTC 2020 x86_64 GNU/Linux / #

# ps -eaf | grep -i qemu

root 360867 1 0 15:56 ? 00:00:01 /usr/local/bin/qemu-system-x86_64 -name sandbox-sh -uuid 72820914-dcce-4cea-a01f-3cf9851d136d -machine q35,accel=kvm,kernel_irqchip -cpu host,pmu=off -qmp unix:/run/vc/vm/sh/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=129755M -device pci-bridge,bus=pcie.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/sh/console.sock,server,nowait -device virtio-scsi-pci,id=scsi0,disable-modern=false,romfile= -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0,romfile= -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-1189071429,guest-cid=1189071429,romfile= -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/sh/shared,security_model=none,multidevs=remap -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object memory-backend-ram,id=dimm1,size=2048M -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinuz-sev-es-5.6-v3.container -initrd /usr/share/kata-containers/kata-containers-initrd.img -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 pci=lastbus=0 quiet panic=1 nr_cpus=240 scsi_mod.scan=none root=/dev/vda1 rootflags=data=ordered,errors=remount-ro agent.log=debug initcall_debug -pidfile /run/vc/vm/sh/pid -smp 1,cores=1,threads=1,sockets=240,maxcpus=240 root 360966 360958 0 16:05 pts/2 00:00:00 grep --color=auto -i qemu

But if I use : firmware = "/usr/local/share/qemu/OVMF_CODE.fd.kata" I get Error:

# ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest <http://docker.io/library/busybox:latest> s ctr: failed to launch qemu: exit status 1, error messages from qemu log: qemu: could not load PC BIOS '/usr/local/share/qemu/OVMF_CODE.fd.kata' : unknown

Any suggestions please.

I did a quick look. The runtime apparently uses the -bios option when the firmware option is passed, not the -drive option as I thought it would. I'm not entirely sure you can boot OVMF that way. I don't have any idea at the moment why it worked with version 1.x. If you could share with me the 1.x qemu command line option with the firmware option, I'm curious to see what it is.

Thanks Christophe

--Thanking You --Sandeep