[kata-dev] The case to integrate the tests repository
Christophe de Dinechin
dinechin at redhat.com
Fri Jan 15 09:49:34 UTC 2021
> On 15 Jan 2021, at 04:29, David Gibson <david at gibson.dropbear.id.au> wrote:
> Hi all,
> I'm a pretty recently started Kata contributor. One of many things I
> found frustrating when I started working on Kata was the way the code
> was split into a whole batch of repositories. I'm relieved that
> they're now merged together for Kata 2.x.
> However, the tests repository is still separate, and AIUI there are no
> plans to integrate it. I still find that really annoying, so I want
> to make the case for integrating it into the kata-containers
> repository as well.
> 1. It removes delays fixing test bugs
> On several occasions when working on code changes, I've hit CI
> failures due to what turn out to be bugs in the tests code. These
> have often been problems with static checks essentially unrelated to
> what I'm actually trying to accomplish.
> To deal with this, I've had to file a separate issue and PR against
> the tests repo, wait for that to be tested and reviewed, and only then
> can I resume work on what I was actually trying to do. Especially
> since I'm a different timezone from most of the Kata team, that can
> easily be several days of delay.
> If the tests were integrated, I could include a fix for the test as a
> commit in the same PR, it would automatically fix the CI for that PR
> and could be reviewed with the same batch.
> 2. Allows tests to be commited along with new features
> Obviously it's desirable for new features to come with tests for those
> features. Integrating the tests repo means that can be done in the
> same PR, reducing the amount of github busywork involved in doing so.
> It also means the code and tests can be reviewed and discussed in the
> same place.
> In addition to being more convenient, that means we don't need to
> rerun basically the same tests for both the code PR and the tests PR.
> That means less demand on the CI infrastructure, which helps everyone.
> 3. Testing scripts can be simplified
> AFAICT the current scripts in the tests code have a bunch of
> conditionals for various different configurations: Kata1 vs. Kata 2.x,
> Rust agent vs. Go agent amongst others. If the tests repo is
> integrated we can cut down the test scripts to just the cases that are
> relevant for the current version of the code.
> Understanding how the test scripts work is a real barrier to debugging
> Kata at the moment, so simplifying them is valuable.
> 3. Removes a barrier to running CI tests locally
> At the moment debugging CI failures is really painful, because
> duplicating the problem in a debuggable environment is extremely
> awkward. One (certainly not the only) reason for that is that the
> code repo and tests repo need to be both assembled together in a
> suitable environment. The CI scripts do that, but this often involves
> downloading one or the other repo from a global URL. If you want to
> do frequently repeated tests on a local repo while debugging, that has
> to be worked around.
> If tests and code are in the same repo, it removes the need to
> download either one from a global URL, making it easier to replicate
> the CI test in a local environment and with an ad-hoc tree. It also
> allows some further simplifications of the CI scripts.
> I could probably come up with more reasons, but those 3 seem like a
> solid start.
Solid +1 from me ;-)
Now, in the interest of exhaustivity, can you think of reasons not to?
I can think of a couple, but I'm interested in your viewpoint here.
More information about the kata-dev