We had seen something similar to thison another project when we switched from go 1.8 to 1.9. On Linux, Go now calls the getrandom system call without the GRND_NONBLOCK flag; it will now block until the kernel has sufficient randomness. On kernels predating the getrandom system call, Go continues to read from /dev/urandom. We had to implement something along the lines of https://github.com/ciao-project/ciao/commit/30ddabb9e201a7985100750e64172ae4... to work around this issue. Is something like this happening within the VM in the agent which is written in go? We had to modify our go code to sample On Mon, 2018-09-24 at 16:40 -0700, Jon Olson via kata-dev wrote:

+tytso@mit.edu -- Ted, I know you had some thoughts on seeding virtio-rng from /dev/urandom (not sure the listserv will let you post, but it should catch at least Sebastien and I). Jon

On Mon, Sep 24, 2018 at 3:21 PM Boeuf, Sebastien < sebastien.boeuf@intel.com> wrote:

...

Hi folks,

Following the discussion from this morning during the Arch committee meeting, I have investigated the sporadic issue https://github.com/kata-containers/runtime/issues/702 preventing from starting some Kata containers.

I have been able to reproduce it pretty easily and I have identified it is related to the entropy of the host being almost entirely consumed by the first containers, leaving no time for the host to regenerate new entropy for the next containers.

Currently, the virtio-rng device exposed by Qemu relies on /dev/random on the host, and because this device will block any access to it until some more entropy is ready, that's why we end up getting the timeout from the gRPC client as the agent is not ready, hence the gRPC server does not run yet (the guest is blocked on getting new entropy from /dev/random). One way to workaround this issue is to tweak the parameters of the virtio-rng device such as max-bytes=10, limiting the amount of entropy that can be consumed by the guest each period. This means that starting one container will not consume all host's entropy, but eventually, if we run a lot of containers, we'll be very likely to hit this same issue.

The long term solution seems to rely on /dev/urandom device as this one will not block if no entropy is ready yet. But from what we can read online, it seems that some people have some security concerns about it. I'd like to understand if those worries are valid or not, and if we should keep thinking about another way to fix this issue.

Thanks, Sebastien

_______________________________________________

kata-dev mailing list

kata-dev@lists.katacontainers.io

http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev

_______________________________________________kata-dev mailing listkata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev

Manohar, the agent does not have any call to generate a random number (we remove the only call it had long time ago, before add virio-rng, for the a similar issue you described with the kernel + Go). See: https://github.com/kata-containers/agent/pull/279 In the past I've seen that what is getting blocked is the kernel at at early boot (probably getting a random number with virio-rng). We were able to reproduce something similar vexxhost VMs. After the adding a daemon to generate entropy in vexxhost this was fixed. See: https://github.com/kata-containers/runtime/pull/676#issuecomment-418812957 I could not reproduce this locally but probably is the same issue. The dmesg logs from the VM could give us more information. Second, the VMs are pulling a lot of entropy at startup. Given that the agent is not trying to get random numbers, I am not sure if this is the kernel or systemd reducing the amount of entropy in the host. For now we are a adding a configuration option to decide what host entropy source use. Also from administrator perspective allow limit the amount of entropy is a good option. - Carlos On Mon, Sep 24, 2018 at 11:59:51PM +0000, Castelino, Manohar R wrote:

We had seen something similar to thison another project when we switched from go 1.8 to 1.9.

On Linux, Go now calls the getrandom system call without the GRND_NONBLOCK flag; it will now block until the kernel has sufficient randomness. On kernels predating the getrandom system call, Go continues to read from /dev/urandom.

We had to implement something along the lines of https://github.com/ ciao-project/ciao/commit/30ddabb9e201a7985100750e64172ae4b518d1e6 to work around this issue.

Is something like this happening within the VM in the agent which is written in go?

We had to modify our go code to sample On Mon, 2018-09-24 at 16:40 -0700, Jon Olson via kata-dev wrote:

+tytso@mit.edu -- Ted, I know you had some thoughts on seeding virtio-rng from /dev/urandom (not sure the listserv will let you post, but it should catch at least Sebastien and I).

Jon

On Mon, Sep 24, 2018 at 3:21 PM Boeuf, Sebastien <sebastien.boeuf@intel.com > wrote:

Hi folks,

Following the discussion from this morning during the Arch committee meeting, I have investigated the sporadic issue https://github.com/ kata-containers/runtime/issues/702 preventing from starting some Kata containers.

I have been able to reproduce it pretty easily and I have identified it is related to the entropy of the host being almost entirely consumed by the first containers, leaving no time for the host to regenerate new entropy for the next containers.

Currently, the virtio-rng device exposed by Qemu relies on /dev/random on the host, and because this device will block any access to it until some more entropy is ready, that's why we end up getting the timeout from the gRPC client as the agent is not ready, hence the gRPC server does not run yet (the guest is blocked on getting new entropy from /dev /random). One way to workaround this issue is to tweak the parameters of the virtio-rng device such as max-bytes=10, limiting the amount of entropy that can be consumed by the guest each period. This means that starting one container will not consume all host's entropy, but eventually, if we run a lot of containers, we'll be very likely to hit this same issue.

The long term solution seems to rely on /dev/urandom device as this one will not block if no entropy is ready yet. But from what we can read online, it seems that some people have some security concerns about it. I'd like to understand if those worries are valid or not, and if we should keep thinking about another way to fix this issue.

Thanks, Sebastien _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev

_______________________________________________

kata-dev mailing list

kata-dev@lists.katacontainers.io

http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev

_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev

Thanks Tao, That's a great summary IMO. I'll take care of testing Ted's patch, and will proceed depending on the result. Thanks, Sebastien ________________________________________ From: Tao Peng [bergwolf@hyper.sh] Sent: Tuesday, September 25, 2018 8:15 AM To: Castelino, Manohar R Cc: jonolson@google.com; tytso@mit.edu; Boeuf, Sebastien; kata-dev@lists.katacontainers.io Subject: Re: [kata-dev] /dev/urandom or /dev/random On Tue, Sep 25, 2018 at 7:59 AM, Castelino, Manohar R <manohar.r.castelino@intel.com> wrote:

We had seen something similar to thison another project when we switched from go 1.8 to 1.9.

On Linux, Go now calls the getrandom system call without the GRND_NONBLOCK flag; it will now block until the kernel has sufficient randomness. On kernels predating the getrandom system call, Go continues to read from /dev/urandom.

We had to implement something along the lines of https://github.com/ciao-project/ciao/commit/30ddabb9e201a7985100750e64172ae4... to work around this issue.

Is something like this happening within the VM in the agent which is written in go?

I agree we should understand why every guest needs to read so many entropies first. IMO, it does so because it needs to initialize the CRNG. So I wonder if Ted's patch [1] can help us on the guest side. It relies on CPU to initialize the guest's CRNG, which (IMO) is the main reason we read a lot from virtio-rng for every new guest. If we can apply Ted's patch and enable it, we can possibly get rid of the container startup timeout problem. OTOH, if users wants to run some entropy hungry containers, they should make sure the host have enough entropies, -- the same situation as if they were running with runc containers. And let's re-visit why we have virtio-rng in the first place. IIRC we added it because the guest runs out of entropy on its own. IIUC, it doesn't help security to use virtio-rng unless user passes a hwrng to virtio-rng. If we use /dev/urandom as a random number source for virtio-rng, we kind of map both guest /dev/random to host /dev/urandom. If we can trust host CPU that much, it is really convenient to do so. But if we don't, we need to get container app the level of randomness it asks for (by reading /dev/random). So my suggestions are: 1. see if Ted's above patch can help keeping /dev/random as virtio-rng source by default 2. If it does help, let's keep the virtio-rng configurable source implementation, but set it to /dev/random by default 3. If it doesn't help, let's default to /dev/urandom and let user decide if they want /dev/random or some hwrng as virtio-rng source [1] http://lkml.iu.edu/hypermail/linux/kernel/1807.2/02498.html Cheers, Tao

We had to modify our go code to sample On Mon, 2018-09-24 at 16:40 -0700, Jon Olson via kata-dev wrote:

+tytso@mit.edu -- Ted, I know you had some thoughts on seeding virtio-rng from /dev/urandom (not sure the listserv will let you post, but it should catch at least Sebastien and I).

Jon

On Mon, Sep 24, 2018 at 3:21 PM Boeuf, Sebastien <sebastien.boeuf@intel.com> wrote:

Hi folks,

Following the discussion from this morning during the Arch committee meeting, I have investigated the sporadic issue https://github.com/kata-containers/runtime/issues/702 preventing from starting some Kata containers.

I have been able to reproduce it pretty easily and I have identified it is related to the entropy of the host being almost entirely consumed by the first containers, leaving no time for the host to regenerate new entropy for the next containers.

Currently, the virtio-rng device exposed by Qemu relies on /dev/random on the host, and because this device will block any access to it until some more entropy is ready, that's why we end up getting the timeout from the gRPC client as the agent is not ready, hence the gRPC server does not run yet (the guest is blocked on getting new entropy from /dev/random). One way to workaround this issue is to tweak the parameters of the virtio-rng device such as max-bytes=10, limiting the amount of entropy that can be consumed by the guest each period. This means that starting one container will not consume all host's entropy, but eventually, if we run a lot of containers, we'll be very likely to hit this same issue.

The long term solution seems to rely on /dev/urandom device as this one will not block if no entropy is ready yet. But from what we can read online, it seems that some people have some security concerns about it. I'd like to understand if those worries are valid or not, and if we should keep thinking about another way to fix this issue.

Thanks, Sebastien _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev

_______________________________________________

kata-dev mailing list

kata-dev@lists.katacontainers.io

http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev

_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev

-- bergwolf@hyper.sh

Thanks for the feedback Ted. I am glad I've asked as it feels safer to go with /dev/urandom solution now :) Thanks, Sebastien ________________________________________ From: Theodore Y. Ts'o [tytso@mit.edu] Sent: Tuesday, September 25, 2018 3:15 PM To: Jon Olson Cc: Boeuf, Sebastien; kata-dev@lists.katacontainers.io Subject: Re: [kata-dev] /dev/urandom or /dev/random On Mon, Sep 24, 2018 at 04:40:13PM -0700, Jon Olson wrote:

+tytso@mit.edu -- Ted, I know you had some thoughts on seeding virtio-rng from /dev/urandom (not sure the listserv will let you post, but it should catch at least Sebastien and I).

From a guest perspective, you have to trust the Host system anyway. The host as the ability to read or modify arbitrary guest memory, including that used by the kernel. The host can mess with the guest kernel before it's booted, or change the contents of the emulated hard drive. So trusting that the host CPU has implemented RDRAND correctly isn't a stretch. So even if the customer running the VM is a tin-foil-hatter who believes the NSA may have backdoored RDRAND and not have it be discovered or leaked out, in most cases, they aren't

My recommendation is to just seed virtio-rng from /dev/urandom. Here's why. /dev/urandom provides a cryptographic random number generator, and assuming it is adequately initialized, that should be *fine* for pretty much all practical purposes. And on the sort of systems you would be using as a Host, there are enough hardware devices that /dev/urandom will be initialized very quickly --- in practice, well before any VM's will be started. Furthermore, all modern x86 systems have RDRAND by now, and on those systems, the host kernel will mix RDRAND output with CRNG before sending the random numbers out via getrandom(2) or /dev/urandom. the sort of people who would want to use a cloud infrastructure. They're much more likely to want to use hardware which always stays under their personal control. And even if you *do* believe the NSA has somehow managed to sneak a backdoor into Intel CPU designs without it become public, that still means that you have to break the CRNG. And practically speaking, assuming it's been initialized correctly, if you can break the CRNG, we've got much bigger problems, since that implies you can't trust the basic crypto primitives we use to secure our entire internet. Cheers, - Ted

This was an interesting discussion for me to follow and I learned some new things. I did my own research and found this https://www.2uo.de/myths-about-urandom/ which was a fairly thorough interpretation of how things work and parrots in more detail what Ted said. Tl;dr: Just use /dev/urandom! " The man page for /dev/random and /dev/urandom is pretty effective when it comes to instilling fear into the gullible programmer's mind:" Thanks Eric -----Original Message----- From: Venegas Munoz, Jose Carlos [mailto:jose.carlos.venegas.munoz@intel.com] Sent: Tuesday, September 25, 2018 3:33 PM To: Boeuf, Sebastien <sebastien.boeuf@intel.com>; Theodore Y. Ts'o <tytso@mit.edu>; Jon Olson <jonolson@google.com> Cc: kata-dev@lists.katacontainers.io Subject: Re: [kata-dev] /dev/urandom or /dev/random Thanks for the feedback! -----Original Message----- From: Boeuf, Sebastien [mailto:sebastien.boeuf@intel.com] Sent: Tuesday, September 25, 2018 5:26 PM To: Theodore Y. Ts'o <tytso@mit.edu>; Jon Olson <jonolson@google.com> Cc: kata-dev@lists.katacontainers.io Subject: Re: [kata-dev] /dev/urandom or /dev/random Thanks for the feedback Ted. I am glad I've asked as it feels safer to go with /dev/urandom solution now :) Thanks, Sebastien ________________________________________ From: Theodore Y. Ts'o [tytso@mit.edu] Sent: Tuesday, September 25, 2018 3:15 PM To: Jon Olson Cc: Boeuf, Sebastien; kata-dev@lists.katacontainers.io Subject: Re: [kata-dev] /dev/urandom or /dev/random On Mon, Sep 24, 2018 at 04:40:13PM -0700, Jon Olson wrote:

+tytso@mit.edu -- Ted, I know you had some thoughts on seeding +virtio-rng from /dev/urandom (not sure the listserv will let you post, but it should catch at least Sebastien and I).

My recommendation is to just seed virtio-rng from /dev/urandom. Here's why. /dev/urandom provides a cryptographic random number generator, and assuming it is adequately initialized, that should be *fine* for pretty much all practical purposes. And on the sort of systems you would be using as a Host, there are enough hardware devices that /dev/urandom will be initialized very quickly --- in practice, well before any VM's will be started. Furthermore, all modern x86 systems have RDRAND by now, and on those systems, the host kernel will mix RDRAND output with CRNG before sending the random numbers out via getrandom(2) or /dev/urandom. From a guest perspective, you have to trust the Host system anyway. The host as the ability to read or modify arbitrary guest memory, including that used by the kernel. The host can mess with the guest kernel before it's booted, or change the contents of the emulated hard drive. So trusting that the host CPU has implemented RDRAND correctly isn't a stretch. So even if the customer running the VM is a tin-foil-hatter who believes the NSA may have backdoored RDRAND and not have it be discovered or leaked out, in most cases, they aren't the sort of people who would want to use a cloud infrastructure. They're much more likely to want to use hardware which always stays under their personal control. And even if you *do* believe the NSA has somehow managed to sneak a backdoor into Intel CPU designs without it become public, that still means that you have to break the CRNG. And practically speaking, assuming it's been initialized correctly, if you can break the CRNG, we've got much bigger problems, since that implies you can't trust the basic crypto primitives we use to secure our entire internet. Cheers, - Ted _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev

Since you really never want your VMs blocking, I'd suggest just telling qemu to use urandom; you can pass it as a parameter to -object rng-random

I think that's probably part of the debate isn't it - if you don't have enough entropy, do you still want to run your VM/container, or do you want to block (or fail after some suitable time) if there is not enough entropy. I'm presuming not having enough entropy will result in some potential security implications such as weakening to brute force and side channel attacks etc. For the 'block until there is entropy' option - and I guess this depends what is blocking at current (is it the kernel or the agent) - could we get the agent to bring up the grpc interface and then stall launching the actual container workload until enough entropy is available? (and have that option and the limits set in the kata config file).

Dave

Graham --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

On Tue, Sep 25, 2018 at 09:11:59AM +0100, Dr. David Alan Gilbert wrote:

* Boeuf, Sebastien (sebastien.boeuf@intel.com) wrote:

[U]Randomly chiming in. [...]

...

Currently, the virtio-rng device exposed by Qemu relies on /dev/random on the host, and because this device will block any access to it until some more entropy is ready, that's why we end up getting the timeout from the gRPC client as the agent is not ready, hence the gRPC server does not run yet (the guest is blocked on getting new entropy from /dev/random). One way to workaround this issue is to tweak the parameters of the virtio-rng device such as max-bytes=10, limiting the amount of entropy that can be consumed by the guest each period. This means that starting one container will not consume all host's entropy, but eventually, if we run a lot of containers, we'll be very likely to hit this same issue.

The long term solution seems to rely on /dev/urandom device as this one will not block if no entropy is ready yet. But from what we can read online, it seems that some people have some security concerns about it. I'd like to understand if those worries are valid or not, and if we should keep thinking about another way to fix this issue.

I'm just curious what are those concerns. Because the official documentation[1] of random(4) says: The /dev/random device is a legacy interface which dates back to a time where the cryptographic primitives used in the implementation of /dev/urandom were not widely trusted. It will return random bytes only within the estimated number of bits of fresh noise in the entropy pool, blocking if necessary. /dev/random is suitable for applications that need high quality randomness, and can afford indeterminate delays. And that of urandom(4) says[2]: Usage The /dev/random interface is considered a legacy interface, and /dev/urandom is preferred and sufficient in all use cases, with the exception of applications which require randomness during early boot time; for these applications, getrandom(2) must be used instead, because it will block until the entropy pool is initialized As you noticed, QEMU still defaults the source of entropy for Virtio-RNG device to `/dev/random`. I don't see a reason why the default source for Virtio-RNG shouldn't be changed to `/dev/urandom`. FWIW, to that end, there are two related threads[3][4] on 'qemu-devel'. (No resolution yet, though.) [1] http://man7.org/linux/man-pages/man4/random.4.html [2] http://man7.org/linux/man-pages/man4/urandom.4.html [3] https://lists.nongnu.org/archive/html/qemu-devel/2018-09/msg02724.html -- [RFC] Virtio RNG: Consider changing the default entropy source to /dev/urandom? [4] https://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg07052.html -- Any reason QEMU doesn't default to `/dev/urandom`?

Those concerns seem mostly to be in older discussions.

That's also my observation, FWIW.

On the BSDs you find that they're actually the same device linked together (behaving as I understand it in a way that's similar to Linux's urandom).

Since you really never want your VMs blocking, I'd suggest just telling qemu to use urandom; you can pass it as a parameter to -object rng-random

-- /kashyap

From: Marco Vedovati [mailto:mvedovati@suse.com] [snip] In my view, given the ephemeral aspect of containers, /dev/urandom should be fine most of the time. As an option, user should be left the possibility to select /dev/random if that is needed for his specific application.

From a read around, I agree with the sentiment, that /dev/urandom is . I'm not sure how that ties in with QEMU/KVM though - if we direct QEMU to use the host /dev/urandom, does that mean that the guest /dev/random is really using the host /dev/urandom - so any strong /dev/random guarantees inside the guest are now a lie... I don't know how the virtio rng and qemu hang together etc.

BTW, it may be worth investigating on what is the state of the art solution chosen in virtualization systems with a high VM count.

+1 I doubt we are the first people to run into this scenario.

Marco

Graham --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.