[kata-dev] kata-containers issues when running CRI-O
salvador.fuentes at intel.com
Thu Jun 21 19:20:22 UTC 2018
Recently we have seen issues with the CRI-O tests in our CI. We have recorded some of the failures on https://github.com/kata-containers/tests/issues/361
I have tried to reproduce the failures and these are my findings:
- I found that several kernel warnings are being displayed on the kata-proxy log (you can see the warnings on a comment in the issue above).
- There seems to be a race condition somewhere because when running cri-o (crictl) commands manually, I cannot reproduce the issue.
- The cri-o random failures seem to only appear in Ubuntu 16-04 host machines and the tests that mostly fail are: 6,7,8 and 9.
- I am not sure that the cri-o tests failures are 100% related to the kernel warnings. Sometimes the kernel warning occurs, but the tests pass.
- It may be related to the storage driver been used: On ubuntu we use devicemapper, and in the fedora and centos we use overlay.
If someone can help debugging this, it would be very helpful.
As for the CI failures, which are very annoying, an interim action could be to disable the tests 6-9. With this, we can have a stable CI, while we continue investigating the issue.
How to reproduce the issue:
1. Update your kata-containers components to the master branch
2. You'll need to install CRI-O 1.10 and the CNI plugins.
If you already have the kata-containers/tests repository, you can run:
3. Run the tests using kata-containers runtime:
In The kata-containers/tests repository, run:
sudo -E PATH=$PATH make crio
4. Check logs:
sudo journalctl -t kata-proxy
sudo journalctl -t kata-runtime
More information about the kata-dev