Open Stack

Hi Jesse,

Thanks for the detailed explanation, glad to see you got that working
with Clear Containers.
Per container/VM memory encryption is an exciting feature that we'll
have to support as, as you said, it fits really well into the Kata
Containers goals and architecture.

A few comments/questions:

On Wed, Feb 21, 2018 at 10:06:25PM +0000, Larrew, Jesse wrote:
> Hi all,
> 
> The virtualization instructions in the latest AMD EPYC server processors have been enhanced with a memory encryption feature that could provide projects like Kata Containers with unique security capabilities compared to their non-virtualized counterparts. We call this feature Secure Encrypted Virtualization (or SEV) and would be interested in collaborating with the Kata Container community to enable support for containers with encrypted memory.
> 
> In short, we've added an inline AES engine to our memory controller that encrypts data written to system DRAM and decrypts data read from DRAM. The encryption keys are generated from a TRNG in the onboard AMD Secure Processor (SP) and programmed into the memory controller as needed in a manner that is never visible to software. Additionally, our virtualization instructions have been enhanced to be able to associate a VM ASID with a unique encryption key, so each VM (or container) can keep the contents of its memory confidential from the host and/or other tenant VMs/containers. The guest kernel can choose which pages to encrypt and which to share with the host by setting a bit in the guest page tables, which puts the guest in complete control of the visibility of their data in the cloud. More information can be found in our Memory Encryption whitepaper [1] and in the Architecture Programmer's Manual [2].
> 
> Linux kernel support for SEV has been merged into the 4.15 and upcoming 4.16 kernels. OVMF BIOS support has been merged as well. The qemu changes are still being upstreamed, but the patches are available for testing on github [3].
> 
> With the above support in place, we have developed a proof-of-concept demo that is based on Clear Containers. Since the Clear Containers project had already done the heavy lifting to run container workloads inside of a VM, it was rather straightforward to add support to encrypt those VMs using SEV. The required changes are summarized below:
> 	* Container kernel:
> 		- Add SEV support patches from the Linux kernel repo in [3].
> 		- Force virtio to use the DMA API (and hence SWIOTLB) when adding/removing buffers to/from the virtio ring buffer.
>
Ah, now I understand where the iommu question on the cc-devel mailing
list was coming from :) Would you mind explaining why you need SWIOTLB
when SEV is enabled? Also, I assume you need to force all virtio devices
to do DMA, right? Did you just hack vring_use_dma_api() at the moment?


> 		- SEV requires a memory copy in order to perform the encryption, so zero-copy solutions using DAX for the container initial user space will not work.
> 			+ Build in a small initramfs to use as the guest kernel initial user space.
> 			+ Include the updated container agent binary and supporting libs (~14MB total).
So not using nvdimm from the QEMU command line and switching to a virtio
block would have been sufficient here, right?

> As a check, dumping the contents of a page from the qemu heap reveals plaintext data:
> 
> amd at pecanporter:~/src/git$ sudo dd if=/proc/$(pgrep qemu)/mem bs=4096 count=1 skip=23058854513 | xxd | tail
> dd: /proc/38572/mem: cannot skip to specified offset
> 1+0 records in
> 1+0 records out
> 4096 bytes (4.1 kB, 4.0 KiB) copied, 8.8437e-05 s, 46.3 MB/s
> 00000f60: 	0000 0000 0000 0000 0000 0000 0000 0000	................
> 00000f70: 	0000 0000 0000 0000 0000 0000 0000 0000	................
> 00000f80: 	0000 0000 0000 0000 7100 0000 0000 0000	........q.......
> 00000f90: 	2f72 756e 2f76 6972 7463 6f6e 7461 696e	/run/virtcontain
> 00000fa0: 	6572 732f 706f 6473 2f33 3565 3233 6565	ers/pods/35e23ee
> 00000fb0: 	3330 6466 6237 3266 3135 3730 6265 3432	30dfb72f1570be42
> 00000fc0: 	6665 3165 6331 3366 3331 3332 6138 6133	fe1ec13f3132a8a3
> 00000fd0: 	6463 3336 6463 3131 6235 6365 3837 6236	dc36dc11b5ce87b6
> 00000fe0: 	3437 3930 3736 6339 612f 636f 6e73 6f6c	479076c9a/consol
> 00000ff0: 	652e 736f 636b 0000 0104 0000 0000 0000	e.sock..........
> 
> However, any attempt to read the container memory from the host produces only ciphertext:
> 
> amd at pecanporter:~/src/git$ sudo dd if=/proc/$(pgrep qemu)/mem bs=4096 count=1 skip=34165702144 | xxd | head
> dd: /proc/38572/mem: cannot skip to specified offset
> 1+0 records in
> 1+0 records out
> 4096 bytes (4.1 kB, 4.0 KiB) copied, 8.9039e-05 s, 46.0 MB/s
> 00000000: 	e9b8 e14d c063 ee18 fd85 5ecc 4d1f c1a2	...M.c....^.M...
> 00000010: 	d681 cdf2 259b a97e c43b 5cde bf9e 695b	....%..~.;\...i[
> 00000020: 	db3c 778b 8e77 89f4 f795 e5a6 9ebb 765b	.<w..w........v[
> 00000030: 	0905 e1d3 c7ec 6f2b bada ed15 b2e0 db7f	......o+........
> 00000040: 	d5e9 6d15 cf28 0ca1 4a45 3b9a 1779 e3ff	..m..(..JE;..y..
> 00000050: 	9ee0 b562 2311 6e5a e972 4c06 3f6a 6ebf	...b#.nZ.rL.?jn.
> 00000060: 	909a 88ea 737a 6226 5d87 8968 b31b d096	....szb&]..h....
> 00000070: 	9360 cbb0 4f34 d811 89a7 048f 01e8 d19e	.`..O4..........
> 00000080: 	5429 995a 4de0 6fba 3360 8bb4 a2dc 17e4	T).ZM.o.3`......
> 00000090: 	80f5 6657 9fd7 0347 e78d 4d13 6b6c c649	..fW...G..M.kl.I
>
Sweet!


> Our threat model is to allow container workloads to reduce their risk exposure to security vulnerabilities in the hosting environment, which seems to overlap nicely with the threat model of Kata Containers. Is this a feature that the Kata community would find useful? If so, we would be very interested to work with the community to enable SEV memory encryption for Kata Containers. Any and all feedback is welcome!
>
So I guess we'll gather the kernel, qemu and firmware patches through
upstream at some point. Or we can backport them once they're in if we
don't want to move to the latest versions for those.
I guess Kata Containers main task to support this would be at the
hypervisor level, specifically at being able to pass the right options
to QEMU. In my mind we should make our qemu hypervisor implementation
detect SEV/MK-TME support dynamically and set the right qemu options
(+ memory-encryption, - nvdimm) by default when the host CPU supports
it. I believe we should also provide an opt-out runtime option for those
who don't want to pay the performance penalty of memory encryption.

Cheers,
Samuel.