Open Stack

Hi Jesse,
As Sebastien said, the memory encrypt is an exciting feature for VMs.
Based on the discuss of runtime merging proposal [1], the kata-runtime 1.0 will
based on the runVhypervisor drivers [2] and vm factory [3]. And we should
consider if this should be enable on all qemubased drivers if CPU supports this
feature, or implemented as a new hypervisor which let usersdecide.
Looks like, by enabling the memory encryption, we will get
- stronger isolation we want- bigger memory footprint (vm clone and dax won't
work, and ksm won't work as well?)
Will read the references in detail after today's meeting. Will you attend the
online meeting[4], Jesse?
[1]https://github.com/kata-containers/runtime/issues/33[2]
https://github.com/hyperhq/runv/tree/master/hypervisor[3]
https://github.com/hyperhq/runv/tree/master/factory[4]
https://etherpad.openstack.org/p/katacontainers-2018-architecture-committee-mtgs
-Xu  





On Thu, Feb 22, 2018 6:48 AM, Larrew, Jesse Jesse.Larrew at amd.com  wrote:
Hi Sebastien,




Thanks for the pointer to the virtcontainers PR. I'll keep an eye on it as you
suggested.




That's interesting that you plan to use vanilla qemu for Kata Containers. Do you
intend to upstream the "nofw" and "static-prt" accelerators from qemu-lite? Or
will those optimizations be abandoned? They seem like a clever solution for
reducing the guest boot time.




Sincerely,

Jesse




> -----Original Message-----

> From: Boeuf, Sebastien [mailto:sebastien.boeuf at intel.com]

> Sent: Wednesday, February 21, 2018 4:31 PM

> To: Larrew, Jesse <Jesse.Larrew at amd.com>; kata-

> dev at lists.katacontainers.io

> Cc: Hollingsworth, Brent <brent.hollingsworth at amd.com>; Kaplan, David

> <David.Kaplan at amd.com>; Woller, Thomas <thomas.woller at amd.com>

> Subject: RE: Kata with AMD Secure Encrypted Virtualization (SEV)

> 

> Hi Jesse,

> 

> This is very exciting that you have been able to get Clear Containers working

> with this new AMD technology. I am sure this is something Kata Containers

> will need since the goal of this project is to run virtualized and secure

> containers for any architecture able to do so. There are currently some

> discussions about using virtcontainers as the core API/library for Kata

> Containers runtime, and as an example of the way you could contribute to

> this, here is the link to the recent PR that has been raised to bring simple

> support for ARM architecture (through Qemu):

> https://github.com/containers/virtcontainers/pull/614

> 

> If it gets confirmed that virtcontainers will become part of the Kata

> Containers runtime, then I would suggest that you could raise a similar PR in

> order to support AMD.

> 

> Don't forget to add some documentation about constraints and limitations

> for users that might want to use your AMD support, especially regarding the

> host kernel version. As long as you manage to get the Qemu patches merged

> into upstream, everything will be fine since the goal is to rely on a vanilla

> version of Qemu for Kata.

> 

> Thanks,

> Sebastien

> ________________________________________

> From: Larrew, Jesse [Jesse.Larrew at amd.com]

> Sent: Wednesday, February 21, 2018 2:06 PM

> To: kata-dev at lists.katacontainers.io

> Cc: Hollingsworth, Brent; Kaplan, David; Woller, Thomas

> Subject: [kata-dev] Kata with AMD Secure Encrypted Virtualization (SEV)

> 

> Hi all,

> 

> The virtualization instructions in the latest AMD EPYC server processors have

> been enhanced with a memory encryption feature that could provide

> projects like Kata Containers with unique security capabilities compared to

> their non-virtualized counterparts. We call this feature Secure Encrypted

> Virtualization (or SEV) and would be interested in collaborating with the Kata

> Container community to enable support for containers with encrypted

> memory.

> 

> In short, we've added an inline AES engine to our memory controller that

> encrypts data written to system DRAM and decrypts data read from DRAM.

> The encryption keys are generated from a TRNG in the onboard AMD Secure

> Processor (SP) and programmed into the memory controller as needed in a

> manner that is never visible to software. Additionally, our virtualization

> instructions have been enhanced to be able to associate a VM ASID with a

> unique encryption key, so each VM (or container) can keep the contents of

> its memory confidential from the host and/or other tenant VMs/containers.

> The guest kernel can choose which pages to encrypt and which to share with

> the host by setting a bit in the guest page tables, which puts the guest in

> complete control of the visibility of their data in the cloud. More
information

> can be found in our Memory Encryption whitepaper [1] and in the

> Architecture Programmer's Manual [2].

> 

> Linux kernel support for SEV has been merged into the 4.15 and upcoming

> 4.16 kernels. OVMF BIOS support has been merged as well. The qemu

> changes are still being upstreamed, but the patches are available for testing

> on github [3].

> 

> With the above support in place, we have developed a proof-of-concept

> demo that is based on Clear Containers. Since the Clear Containers project

> had already done the heavy lifting to run container workloads inside of a VM,

> it was rather straightforward to add support to encrypt those VMs using SEV.

> The required changes are summarized below:

> * Container kernel:

> - Add SEV support patches from the Linux kernel repo in [3].

> - Force virtio to use the DMA API (and hence SWIOTLB) when

> adding/removing buffers to/from the virtio ring buffer.

> - SEV requires a memory copy in order to perform the encryption, so

> zero-copy solutions using DAX for the container initial user space will not

> work.

> + Build in a small initramfs to use as the guest kernel initial user

> space.

> + Include the updated container agent binary and supporting libs

> (~14MB total).

> * Container agent:

> - Update the agent not to use the pivot_root() method from the

> initramfs environment, and perform the pivot to the container workload

> filesystem manually instead.

> * Container runtime:

> - Add the new qemu command line options for starting an SEV guest.

> * Qemu-lite:

> - Add the SEV support patches from the qemu repo in [3].

> 

> With the above changes, we are able to start docker containers inside of SEV-

> protected VMs:

> 

> amd at pecanporter:~/src/git$ sudo docker run -ti --runtime sev-runtime

> busybox sh / # whoami root / # dmesg | grep SEV

> [ 0.001000] AMD Secure Encrypted Virtualization (SEV) active

> [ 0.219196] SEV is active and system is using DMA bounce buffers

> 

> As a check, dumping the contents of a page from the qemu heap reveals

> plaintext data:

> 

> amd at pecanporter:~/src/git$ sudo dd if=/proc/$(pgrep qemu)/mem

> bs=4096 count=1 skip=23058854513 | xxd | tail

> dd: /proc/38572/mem: cannot skip to specified offset

> 1+0 records in

> 1+0 records out

> 4096 bytes (4.1 kB, 4.0 KiB) copied, 8.8437e-05 s, 46.3 MB/s

> 00000f60: 0000 0000 0000 0000 0000 0000 0000 0000 ................

> 00000f70: 0000 0000 0000 0000 0000 0000 0000 0000 ................

> 00000f80: 0000 0000 0000 0000 7100 0000 0000 0000 ........q.......

> 00000f90: 2f72 756e 2f76 6972 7463 6f6e 7461 696e /run/virtcontain

> 00000fa0: 6572 732f 706f 6473 2f33 3565 3233 6565 ers/pods/35e23ee

> 00000fb0: 3330 6466 6237 3266 3135 3730 6265 3432 30dfb72f1570be42

> 00000fc0: 6665 3165 6331 3366 3331 3332 6138 6133 fe1ec13f3132a8a3

> 00000fd0: 6463 3336 6463 3131 6235 6365 3837 6236 dc36dc11b5ce87b6

> 00000fe0: 3437 3930 3736 6339 612f 636f 6e73 6f6c 479076c9a/consol

> 00000ff0: 652e 736f 636b 0000 0104 0000 0000 0000 e.sock..........

> 

> However, any attempt to read the container memory from the host

> produces only ciphertext:

> 

> amd at pecanporter:~/src/git$ sudo dd if=/proc/$(pgrep qemu)/mem

> bs=4096 count=1 skip=34165702144 | xxd | head

> dd: /proc/38572/mem: cannot skip to specified offset

> 1+0 records in

> 1+0 records out

> 4096 bytes (4.1 kB, 4.0 KiB) copied, 8.9039e-05 s, 46.0 MB/s

> 00000000: e9b8 e14d c063 ee18 fd85 5ecc 4d1f c1a2 ...M.c....^.M...

> 00000010: d681 cdf2 259b a97e c43b 5cde bf9e 695b ....%..~.;\...i[

> 00000020: db3c 778b 8e77 89f4 f795 e5a6 9ebb 765b .<w..w........v[

> 00000030: 0905 e1d3 c7ec 6f2b bada ed15 b2e0 db7f ......o+........

> 00000040: d5e9 6d15 cf28 0ca1 4a45 3b9a 1779 e3ff ..m..(..JE;..y..

> 00000050: 9ee0 b562 2311 6e5a e972 4c06 3f6a 6ebf ...b#.nZ.rL.?jn.

> 00000060: 909a 88ea 737a 6226 5d87 8968 b31b d096 ....szb&]..h....

> 00000070: 9360 cbb0 4f34 d811 89a7 048f 01e8 d19e .`..O4..........

> 00000080: 5429 995a 4de0 6fba 3360 8bb4 a2dc 17e4 T).ZM.o.3`......

> 00000090: 80f5 6657 9fd7 0347 e78d 4d13 6b6c c649 ..fW...G..M.kl.I

> 

> Our threat model is to allow container workloads to reduce their risk

> exposure to security vulnerabilities in the hosting environment, which seems

> to overlap nicely with the threat model of Kata Containers. Is this a feature

> that the Kata community would find useful? If so, we would be very

> interested to work with the community to enable SEV memory encryption

> for Kata Containers. Any and all feedback is welcome!

> 

> Thanks!

> 

> References:

> [1] AMD Memory Encryption Whitepaper:

> http://amd-dev.wpengine.netdna-

> cdn.com/wordpress/media/2013/12/AMD_Memory_Encryption_Whitepape

> r_v7-Public.pdf

> [2] AMD64 Architecture Programmer's Manual Volume 2: System

> Programming:

> http://developer.amd.com/wordpress/media/2012/10/24593_APM_v21.pdf

> [3] AMD SEV github repo:

> https://github.com/AMDESE/AMDSEV

> 

> Sincerely,

> 

> Jesse Larrew

> MTS Software Security Architect

> AMD Security Architecture R&D

> jesse.larrew at amd.com

> O: +(1) 512-602-0092 (x50092)

> M: +(1) 512-791-4852

> 

> 

> _______________________________________________

> kata-dev mailing list

> kata-dev at lists.katacontainers.io

> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev




_______________________________________________

kata-dev mailing list

kata-dev at lists.katacontainers.io

http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev






--
Xu WangCTO & Cofounder, Hypergithub/twitter/wechat: @gnawuxhttp://hyper.sh
Hyper_: Make VM run like container
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.katacontainers.io/pipermail/kata-dev/attachments/20180222/3ea642e1/attachment-0001.html>