Open Stack

Hi Jesse,

This is very exciting that you have been able to get Clear Containers working with this new AMD technology. I am sure this is something Kata Containers will need since the goal of this project is to run virtualized and secure containers for any architecture able to do so. There are currently some discussions about using virtcontainers as the core API/library for Kata Containers runtime, and as an example of the way you could contribute to this, here is the link to the recent PR that has been raised to bring simple support for ARM architecture (through Qemu): https://github.com/containers/virtcontainers/pull/614

If it gets confirmed that virtcontainers will become part of the Kata Containers runtime, then I would suggest that you could raise a similar PR in order to support AMD.

Don't forget to add some documentation about constraints and limitations for users that might want to use your AMD support, especially regarding the host kernel version. As long as you manage to get the Qemu patches merged into upstream, everything will be fine since the goal is to rely on a vanilla version of Qemu for Kata.

Thanks,
Sebastien
________________________________________
From: Larrew, Jesse [Jesse.Larrew at amd.com]
Sent: Wednesday, February 21, 2018 2:06 PM
To: kata-dev at lists.katacontainers.io
Cc: Hollingsworth, Brent; Kaplan, David; Woller, Thomas
Subject: [kata-dev] Kata with AMD Secure Encrypted Virtualization (SEV)

Hi all,

The virtualization instructions in the latest AMD EPYC server processors have been enhanced with a memory encryption feature that could provide projects like Kata Containers with unique security capabilities compared to their non-virtualized counterparts. We call this feature Secure Encrypted Virtualization (or SEV) and would be interested in collaborating with the Kata Container community to enable support for containers with encrypted memory.

In short, we've added an inline AES engine to our memory controller that encrypts data written to system DRAM and decrypts data read from DRAM. The encryption keys are generated from a TRNG in the onboard AMD Secure Processor (SP) and programmed into the memory controller as needed in a manner that is never visible to software. Additionally, our virtualization instructions have been enhanced to be able to associate a VM ASID with a unique encryption key, so each VM (or container) can keep the contents of its memory confidential from the host and/or other tenant VMs/containers. The guest kernel can choose which pages to encrypt and which to share with the host by setting a bit in the guest page tables, which puts the guest in complete control of the visibility of their data in the cloud. More information can be found in our Memory Encryption whitepaper [1] and in the Architecture Programmer's Manual [2].

Linux kernel support for SEV has been merged into the 4.15 and upcoming 4.16 kernels. OVMF BIOS support has been merged as well. The qemu changes are still being upstreamed, but the patches are available for testing on github [3].

With the above support in place, we have developed a proof-of-concept demo that is based on Clear Containers. Since the Clear Containers project had already done the heavy lifting to run container workloads inside of a VM, it was rather straightforward to add support to encrypt those VMs using SEV. The required changes are summarized below:
        * Container kernel:
                - Add SEV support patches from the Linux kernel repo in [3].
                - Force virtio to use the DMA API (and hence SWIOTLB) when adding/removing buffers to/from the virtio ring buffer.
                - SEV requires a memory copy in order to perform the encryption, so zero-copy solutions using DAX for the container initial user space will not work.
                        + Build in a small initramfs to use as the guest kernel initial user space.
                        + Include the updated container agent binary and supporting libs (~14MB total).
        * Container agent:
                - Update the agent not to use the pivot_root() method from the initramfs environment, and perform the pivot to the container workload filesystem manually instead.
        * Container runtime:
                - Add the new qemu command line options for starting an SEV guest.
        * Qemu-lite:
                - Add the SEV support patches from the qemu repo in [3].

With the above changes, we are able to start docker containers inside of SEV-protected VMs:

amd at pecanporter:~/src/git$ sudo docker run -ti --runtime sev-runtime busybox sh
/ # whoami
root
/ # dmesg | grep SEV
[    0.001000] AMD Secure Encrypted Virtualization (SEV) active
[    0.219196] SEV is active and system is using DMA bounce buffers

As a check, dumping the contents of a page from the qemu heap reveals plaintext data:

amd at pecanporter:~/src/git$ sudo dd if=/proc/$(pgrep qemu)/mem bs=4096 count=1 skip=23058854513 | xxd | tail
dd: /proc/38572/mem: cannot skip to specified offset
1+0 records in
1+0 records out
4096 bytes (4.1 kB, 4.0 KiB) copied, 8.8437e-05 s, 46.3 MB/s
00000f60:       0000 0000 0000 0000 0000 0000 0000 0000 ................
00000f70:       0000 0000 0000 0000 0000 0000 0000 0000 ................
00000f80:       0000 0000 0000 0000 7100 0000 0000 0000 ........q.......
00000f90:       2f72 756e 2f76 6972 7463 6f6e 7461 696e /run/virtcontain
00000fa0:       6572 732f 706f 6473 2f33 3565 3233 6565 ers/pods/35e23ee
00000fb0:       3330 6466 6237 3266 3135 3730 6265 3432 30dfb72f1570be42
00000fc0:       6665 3165 6331 3366 3331 3332 6138 6133 fe1ec13f3132a8a3
00000fd0:       6463 3336 6463 3131 6235 6365 3837 6236 dc36dc11b5ce87b6
00000fe0:       3437 3930 3736 6339 612f 636f 6e73 6f6c 479076c9a/consol
00000ff0:       652e 736f 636b 0000 0104 0000 0000 0000 e.sock..........

However, any attempt to read the container memory from the host produces only ciphertext:

amd at pecanporter:~/src/git$ sudo dd if=/proc/$(pgrep qemu)/mem bs=4096 count=1 skip=34165702144 | xxd | head
dd: /proc/38572/mem: cannot skip to specified offset
1+0 records in
1+0 records out
4096 bytes (4.1 kB, 4.0 KiB) copied, 8.9039e-05 s, 46.0 MB/s
00000000:       e9b8 e14d c063 ee18 fd85 5ecc 4d1f c1a2 ...M.c....^.M...
00000010:       d681 cdf2 259b a97e c43b 5cde bf9e 695b ....%..~.;\...i[
00000020:       db3c 778b 8e77 89f4 f795 e5a6 9ebb 765b .<w..w........v[
00000030:       0905 e1d3 c7ec 6f2b bada ed15 b2e0 db7f ......o+........
00000040:       d5e9 6d15 cf28 0ca1 4a45 3b9a 1779 e3ff ..m..(..JE;..y..
00000050:       9ee0 b562 2311 6e5a e972 4c06 3f6a 6ebf ...b#.nZ.rL.?jn.
00000060:       909a 88ea 737a 6226 5d87 8968 b31b d096 ....szb&]..h....
00000070:       9360 cbb0 4f34 d811 89a7 048f 01e8 d19e .`..O4..........
00000080:       5429 995a 4de0 6fba 3360 8bb4 a2dc 17e4 T).ZM.o.3`......
00000090:       80f5 6657 9fd7 0347 e78d 4d13 6b6c c649 ..fW...G..M.kl.I

Our threat model is to allow container workloads to reduce their risk exposure to security vulnerabilities in the hosting environment, which seems to overlap nicely with the threat model of Kata Containers. Is this a feature that the Kata community would find useful? If so, we would be very interested to work with the community to enable SEV memory encryption for Kata Containers. Any and all feedback is welcome!

Thanks!

References:
[1] AMD Memory Encryption Whitepaper:
http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_Memory_Encryption_Whitepaper_v7-Public.pdf
[2] AMD64 Architecture Programmer's Manual Volume 2: System Programming:
http://developer.amd.com/wordpress/media/2012/10/24593_APM_v21.pdf
[3] AMD SEV github repo:
https://github.com/AMDESE/AMDSEV

Sincerely,

Jesse Larrew
MTS Software Security Architect
AMD Security Architecture R&D
jesse.larrew at amd.com
O: +(1) 512-602-0092 (x50092)
M: +(1) 512-791-4852


_______________________________________________
kata-dev mailing list
kata-dev at lists.katacontainers.io
http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev