<div dir="ltr">David,<div><br></div><div>All the issues you have pointed out are definitely valid and need to be resolved. I think all t of these have been discussed in the ongoing effort to improve Upstream CI. The suggestion of merging the tests repo has been brought up before with some debate. I would like to point out that we also have a CI repo that is separate as well that is also part of testing and CI.</div><div><br></div><div>The current approach to improvements are described in <a href="https://docs.google.com/document/d/15YobCIUFMEvdNIGem0CJ_g30hJY4YH5Lr6ty7_8ASPY/edit">https://docs.google.com/document/d/15YobCIUFMEvdNIGem0CJ_g30hJY4YH5Lr6ty7_8ASPY/edit</a> It takes the approach of trying to understand and clean up what currently exists and solve he issues of test maintenance, debugging failures, and local execution of CI tests. I think all of these can be resolved to various degrees without merging repos. That is not an argument against the idea but a preference to do what we can that is not dependent on where the code/tests lives.</div><div><br></div><div>Your point 2 is the most compelling to me for merging repos, Having tests committed along with new features would be a great place be. Currently I don't see many tests being committed, unit, manual verification, or automation. Maybe I am missing them (can't review every PR). I would like to understand if the state of having separate test repos is a factor in not getting more tests with code changes, or if the other issues are really the cuplrit. I suspect the later because I have seen 100% code coverage through integration tests being committed separately along with code changes when integration tests were in a separate repo (unit tests in repo with code).</div><div><br></div><div>That's my two cents. Thank you for reaffirming that the issues we identified are still issues and that we really need to put more effort in to resolve them.</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 15, 2021 at 8:27 AM David Gibson <<a href="mailto:david@gibson.dropbear.id.au">david@gibson.dropbear.id.au</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Fri, Jan 15, 2021 at 10:49:34AM +0100, Christophe de Dinechin wrote:<br>
> <br>
> <br>
> > On 15 Jan 2021, at 04:29, David Gibson <<a href="mailto:david@gibson.dropbear.id.au" target="_blank">david@gibson.dropbear.id.au</a>> wrote:<br>
> > <br>
> > Hi all,<br>
> > <br>
> > I'm a pretty recently started Kata contributor. One of many things I<br>
> > found frustrating when I started working on Kata was the way the code<br>
> > was split into a whole batch of repositories. I'm relieved that<br>
> > they're now merged together for Kata 2.x.<br>
> > <br>
> > However, the tests repository is still separate, and AIUI there are no<br>
> > plans to integrate it. I still find that really annoying, so I want<br>
> > to make the case for integrating it into the kata-containers<br>
> > repository as well.<br>
> > <br>
> > 1. It removes delays fixing test bugs<br>
> > <br>
> > On several occasions when working on code changes, I've hit CI<br>
> > failures due to what turn out to be bugs in the tests code. These<br>
> > have often been problems with static checks essentially unrelated to<br>
> > what I'm actually trying to accomplish.<br>
> > <br>
> > To deal with this, I've had to file a separate issue and PR against<br>
> > the tests repo, wait for that to be tested and reviewed, and only then<br>
> > can I resume work on what I was actually trying to do. Especially<br>
> > since I'm a different timezone from most of the Kata team, that can<br>
> > easily be several days of delay.<br>
> > <br>
> > If the tests were integrated, I could include a fix for the test as a<br>
> > commit in the same PR, it would automatically fix the CI for that PR<br>
> > and could be reviewed with the same batch.<br>
> > <br>
> > 2. Allows tests to be commited along with new features<br>
> > <br>
> > Obviously it's desirable for new features to come with tests for those<br>
> > features. Integrating the tests repo means that can be done in the<br>
> > same PR, reducing the amount of github busywork involved in doing so.<br>
> > It also means the code and tests can be reviewed and discussed in the<br>
> > same place.<br>
> > <br>
> > In addition to being more convenient, that means we don't need to<br>
> > rerun basically the same tests for both the code PR and the tests PR.<br>
> > That means less demand on the CI infrastructure, which helps everyone.<br>
> > <br>
> > 3. Testing scripts can be simplified<br>
> > <br>
> > AFAICT the current scripts in the tests code have a bunch of<br>
> > conditionals for various different configurations: Kata1 vs. Kata 2.x,<br>
> > Rust agent vs. Go agent amongst others. If the tests repo is<br>
> > integrated we can cut down the test scripts to just the cases that are<br>
> > relevant for the current version of the code.<br>
> > <br>
> > Understanding how the test scripts work is a real barrier to debugging<br>
> > Kata at the moment, so simplifying them is valuable.<br>
> > <br>
> > 3. Removes a barrier to running CI tests locally<br>
> > <br>
> > At the moment debugging CI failures is really painful, because<br>
> > duplicating the problem in a debuggable environment is extremely<br>
> > awkward. One (certainly not the only) reason for that is that the<br>
> > code repo and tests repo need to be both assembled together in a<br>
> > suitable environment. The CI scripts do that, but this often involves<br>
> > downloading one or the other repo from a global URL. If you want to<br>
> > do frequently repeated tests on a local repo while debugging, that has<br>
> > to be worked around.<br>
> > <br>
> > If tests and code are in the same repo, it removes the need to<br>
> > download either one from a global URL, making it easier to replicate<br>
> > the CI test in a local environment and with an ad-hoc tree. It also<br>
> > allows some further simplifications of the CI scripts.<br>
> > <br>
> > <br>
> > I could probably come up with more reasons, but those 3 seem like a<br>
> > solid start.<br>
> <br>
> Solid +1 from me ;-)<br>
> <br>
> Now, in the interest of exhaustivity, can you think of reasons not to?<br>
<br>
I really can't, but I'm kind of assuming someone will tell me some.<br>
<br>
> I can think of a couple, but I'm interested in your viewpoint here.<br>
> <br>
<br>
-- <br>
David Gibson | I'll have my music baroque, and my code<br>
david AT <a href="http://gibson.dropbear.id.au" rel="noreferrer" target="_blank">gibson.dropbear.id.au</a> | minimalist, thank you. NOT _the_ _other_<br>
| _way_ _around_!<br>
<a href="http://www.ozlabs.org/~dgibson" rel="noreferrer" target="_blank">http://www.ozlabs.org/~dgibson</a><br>
_______________________________________________<br>
kata-dev mailing list<br>
<a href="mailto:kata-dev@lists.katacontainers.io" target="_blank">kata-dev@lists.katacontainers.io</a><br>
<a href="http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev" rel="noreferrer" target="_blank">http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev</a><br>
</blockquote></div>