Open Stack

Hi all,

I'm a pretty recently started Kata contributor.  One of many things I
found frustrating when I started working on Kata was the way the code
was split into a whole batch of repositories.  I'm relieved that
they're now merged together for Kata 2.x.

However, the tests repository is still separate, and AIUI there are no
plans to integrate it.  I still find that really annoying, so I want
to make the case for integrating it into the kata-containers
repository as well.

1. It removes delays fixing test bugs

On several occasions when working on code changes, I've hit CI
failures due to what turn out to be bugs in the tests code.  These
have often been problems with static checks essentially unrelated to
what I'm actually trying to accomplish.

To deal with this, I've had to file a separate issue and PR against
the tests repo, wait for that to be tested and reviewed, and only then
can I resume work on what I was actually trying to do.  Especially
since I'm a different timezone from most of the Kata team, that can
easily be several days of delay.

If the tests were integrated, I could include a fix for the test as a
commit in the same PR, it would automatically fix the CI for that PR
and could be reviewed with the same batch.

2. Allows tests to be commited along with new features

Obviously it's desirable for new features to come with tests for those
features.  Integrating the tests repo means that can be done in the
same PR, reducing the amount of github busywork involved in doing so.
It also means the code and tests can be reviewed and discussed in the
same place.

In addition to being more convenient, that means we don't need to
rerun basically the same tests for both the code PR and the tests PR.
That means less demand on the CI infrastructure, which helps everyone.

3. Testing scripts can be simplified

AFAICT the current scripts in the tests code have a bunch of
conditionals for various different configurations: Kata1 vs. Kata 2.x,
Rust agent vs. Go agent amongst others.  If the tests repo is
integrated we can cut down the test scripts to just the cases that are
relevant for the current version of the code.

Understanding how the test scripts work is a real barrier to debugging
Kata at the moment, so simplifying them is valuable.

3. Removes a barrier to running CI tests locally

At the moment debugging CI failures is really painful, because
duplicating the problem in a debuggable environment is extremely
awkward.  One (certainly not the only) reason for that is that the
code repo and tests repo need to be both assembled together in a
suitable environment.  The CI scripts do that, but this often involves
downloading one or the other repo from a global URL.  If you want to
do frequently repeated tests on a local repo while debugging, that has
to be worked around.

If tests and code are in the same repo, it removes the need to
download either one from a global URL, making it easier to replicate
the CI test in a local environment and with an ad-hoc tree.  It also
allows some further simplifications of the CI scripts.


I could probably come up with more reasons, but those 3 seem like a
solid start.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.katacontainers.io/pipermail/kata-dev/attachments/20210115/4675381d/attachment.sig>