Manohar, the agent does not have any call to generate a random number (we remove the only call it had long time ago, before add virio-rng, for the a similar issue you described with the kernel + Go). See: https://github.com/kata-containers/agent/pull/279 In the past I've seen that what is getting blocked is the kernel at at early boot (probably getting a random number with virio-rng). We were able to reproduce something similar vexxhost VMs. After the adding a daemon to generate entropy in vexxhost this was fixed. See: https://github.com/kata-containers/runtime/pull/676#issuecomment-418812957 I could not reproduce this locally but probably is the same issue. The dmesg logs from the VM could give us more information. Second, the VMs are pulling a lot of entropy at startup. Given that the agent is not trying to get random numbers, I am not sure if this is the kernel or systemd reducing the amount of entropy in the host. For now we are a adding a configuration option to decide what host entropy source use. Also from administrator perspective allow limit the amount of entropy is a good option. - Carlos On Mon, Sep 24, 2018 at 11:59:51PM +0000, Castelino, Manohar R wrote:
We had seen something similar to thison another project when we switched from go 1.8 to 1.9.
On Linux, Go now calls the getrandom system call without the GRND_NONBLOCK flag; it will now block until the kernel has sufficient randomness. On kernels predating the getrandom system call, Go continues to read from /dev/urandom.
We had to implement something along the lines of https://github.com/ ciao-project/ciao/commit/30ddabb9e201a7985100750e64172ae4b518d1e6 to work around this issue.
Is something like this happening within the VM in the agent which is written in go?
We had to modify our go code to sample On Mon, 2018-09-24 at 16:40 -0700, Jon Olson via kata-dev wrote:
+tytso@mit.edu -- Ted, I know you had some thoughts on seeding virtio-rng from /dev/urandom (not sure the listserv will let you post, but it should catch at least Sebastien and I).
Jon
On Mon, Sep 24, 2018 at 3:21 PM Boeuf, Sebastien <sebastien.boeuf@intel.com > wrote:
Hi folks,
Following the discussion from this morning during the Arch committee meeting, I have investigated the sporadic issue https://github.com/ kata-containers/runtime/issues/702 preventing from starting some Kata containers.
I have been able to reproduce it pretty easily and I have identified it is related to the entropy of the host being almost entirely consumed by the first containers, leaving no time for the host to regenerate new entropy for the next containers.
Currently, the virtio-rng device exposed by Qemu relies on /dev/random on the host, and because this device will block any access to it until some more entropy is ready, that's why we end up getting the timeout from the gRPC client as the agent is not ready, hence the gRPC server does not run yet (the guest is blocked on getting new entropy from /dev /random). One way to workaround this issue is to tweak the parameters of the virtio-rng device such as max-bytes=10, limiting the amount of entropy that can be consumed by the guest each period. This means that starting one container will not consume all host's entropy, but eventually, if we run a lot of containers, we'll be very likely to hit this same issue.
The long term solution seems to rely on /dev/urandom device as this one will not block if no entropy is ready yet. But from what we can read online, it seems that some people have some security concerns about it. I'd like to understand if those worries are valid or not, and if we should keep thinking about another way to fix this issue.
Thanks, Sebastien _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
_______________________________________________
kata-dev mailing list
kata-dev@lists.katacontainers.io
http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev