Open Stack

Hi all, 

Recently we have seen issues with the CRI-O tests in our CI. We have recorded some of the failures on https://github.com/kata-containers/tests/issues/361 
I have tried to reproduce the failures and these are my findings:
- I found that several kernel warnings are being displayed on the kata-proxy log (you can see the warnings on a comment in the issue above).
- There seems to be a race condition somewhere because when running cri-o (crictl) commands manually, I cannot reproduce the issue.
- The cri-o random failures seem to only appear in Ubuntu 16-04 host machines and the tests that mostly fail are: 6,7,8 and 9.
- I am not sure that the cri-o tests failures are 100% related to the kernel warnings. Sometimes the kernel warning occurs, but the tests pass. 
- It may be related to the storage driver been used: On ubuntu we use devicemapper, and in the fedora and centos we use overlay.

If someone can help debugging this, it would be very helpful. 

As for the CI failures, which are very annoying,  an interim action could be to disable the tests 6-9. With this, we can have a stable CI, while we continue investigating the issue. 

How to reproduce the issue:

1. Update your kata-containers components to the master branch
2. You'll need to install CRI-O 1.10 and the CNI plugins.
     If you already have the kata-containers/tests repository, you can run:
	.ci/install_crio.sh
	.ci/install_cni_plugins.sh 

3. Run the tests using kata-containers runtime:
	In The kata-containers/tests repository, run:
	export RUNTIME=kata-runtime 
	sudo -E PATH=$PATH make crio 

4. Check logs:
	sudo journalctl -t kata-proxy
	sudo journalctl -t kata-runtime

Thanks, 
Salvador Fuentes.