Thanks Tao, That's a great summary IMO. I'll take care of testing Ted's patch, and will proceed depending on the result. Thanks, Sebastien ________________________________________ From: Tao Peng [bergwolf@hyper.sh] Sent: Tuesday, September 25, 2018 8:15 AM To: Castelino, Manohar R Cc: jonolson@google.com; tytso@mit.edu; Boeuf, Sebastien; kata-dev@lists.katacontainers.io Subject: Re: [kata-dev] /dev/urandom or /dev/random On Tue, Sep 25, 2018 at 7:59 AM, Castelino, Manohar R <manohar.r.castelino@intel.com> wrote:
We had seen something similar to thison another project when we switched from go 1.8 to 1.9.
On Linux, Go now calls the getrandom system call without the GRND_NONBLOCK flag; it will now block until the kernel has sufficient randomness. On kernels predating the getrandom system call, Go continues to read from /dev/urandom.
We had to implement something along the lines of https://github.com/ciao-project/ciao/commit/30ddabb9e201a7985100750e64172ae4... to work around this issue.
Is something like this happening within the VM in the agent which is written in go?
I agree we should understand why every guest needs to read so many entropies first. IMO, it does so because it needs to initialize the CRNG. So I wonder if Ted's patch [1] can help us on the guest side. It relies on CPU to initialize the guest's CRNG, which (IMO) is the main reason we read a lot from virtio-rng for every new guest. If we can apply Ted's patch and enable it, we can possibly get rid of the container startup timeout problem. OTOH, if users wants to run some entropy hungry containers, they should make sure the host have enough entropies, -- the same situation as if they were running with runc containers. And let's re-visit why we have virtio-rng in the first place. IIRC we added it because the guest runs out of entropy on its own. IIUC, it doesn't help security to use virtio-rng unless user passes a hwrng to virtio-rng. If we use /dev/urandom as a random number source for virtio-rng, we kind of map both guest /dev/random to host /dev/urandom. If we can trust host CPU that much, it is really convenient to do so. But if we don't, we need to get container app the level of randomness it asks for (by reading /dev/random). So my suggestions are: 1. see if Ted's above patch can help keeping /dev/random as virtio-rng source by default 2. If it does help, let's keep the virtio-rng configurable source implementation, but set it to /dev/random by default 3. If it doesn't help, let's default to /dev/urandom and let user decide if they want /dev/random or some hwrng as virtio-rng source [1] http://lkml.iu.edu/hypermail/linux/kernel/1807.2/02498.html Cheers, Tao
We had to modify our go code to sample On Mon, 2018-09-24 at 16:40 -0700, Jon Olson via kata-dev wrote:
+tytso@mit.edu -- Ted, I know you had some thoughts on seeding virtio-rng from /dev/urandom (not sure the listserv will let you post, but it should catch at least Sebastien and I).
Jon
On Mon, Sep 24, 2018 at 3:21 PM Boeuf, Sebastien <sebastien.boeuf@intel.com> wrote:
Hi folks,
Following the discussion from this morning during the Arch committee meeting, I have investigated the sporadic issue https://github.com/kata-containers/runtime/issues/702 preventing from starting some Kata containers.
I have been able to reproduce it pretty easily and I have identified it is related to the entropy of the host being almost entirely consumed by the first containers, leaving no time for the host to regenerate new entropy for the next containers.
Currently, the virtio-rng device exposed by Qemu relies on /dev/random on the host, and because this device will block any access to it until some more entropy is ready, that's why we end up getting the timeout from the gRPC client as the agent is not ready, hence the gRPC server does not run yet (the guest is blocked on getting new entropy from /dev/random). One way to workaround this issue is to tweak the parameters of the virtio-rng device such as max-bytes=10, limiting the amount of entropy that can be consumed by the guest each period. This means that starting one container will not consume all host's entropy, but eventually, if we run a lot of containers, we'll be very likely to hit this same issue.
The long term solution seems to rely on /dev/urandom device as this one will not block if no entropy is ready yet. But from what we can read online, it seems that some people have some security concerns about it. I'd like to understand if those worries are valid or not, and if we should keep thinking about another way to fix this issue.
Thanks, Sebastien _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
_______________________________________________
kata-dev mailing list
kata-dev@lists.katacontainers.io
http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
-- bergwolf@hyper.sh