Open Stack

> On 15 Jan 2021, at 16:07, Cameron Meadors <cmeadors at redhat.com> wrote:
> 
> David,
> 
> All the issues you have pointed out are definitely valid and need to be resolved.  I think all t of these have been discussed in the ongoing effort to improve Upstream CI.  The suggestion of merging the tests repo has been brought up before with some debate.  I would like to point out that we also have a CI repo that is separate as well that is also part of testing and CI.
> 
> The current approach to improvements are described in https://docs.google.com/document/d/15YobCIUFMEvdNIGem0CJ_g30hJY4YH5Lr6ty7_8ASPY/edit <https://docs.google.com/document/d/15YobCIUFMEvdNIGem0CJ_g30hJY4YH5Lr6ty7_8ASPY/edit>  It takes the approach of trying to understand and clean up what currently exists and solve he issues of test maintenance, debugging failures, and local execution of CI tests.  I think all of these can be resolved to various degrees without merging repos.  That is not an argument against the idea but a preference to do what we can that is not dependent on where the code/tests lives.
> 
> Your point 2 is the most compelling to me for merging repos,  Having tests committed along with new features would be a great place be.  Currently I don't see many tests being committed, unit, manual verification, or automation.  Maybe I am missing them (can't review every PR).  I would like to understand if the state of having separate test repos is a factor in not getting more tests with code changes, or if the other issues are really the cuplrit.  I suspect the later because I have seen 100% code coverage through integration tests being committed separately along with code changes when integration tests were in a separate repo (unit tests in repo with code).

It is definitely a factor.

- If you push the test first, then other commits will presumably fail.

- If you push the code first, chances are it does not pass the test in the CI.

So right now, pushing a test is hard (on a scale of 1 to 10, I'd say a solid 20).

> 
> That's my two cents.  Thank you for reaffirming that the issues we identified are still issues and that we really need to put more effort in to resolve them.
> 
> 
> 
> On Fri, Jan 15, 2021 at 8:27 AM David Gibson <david at gibson.dropbear.id.au <mailto:david at gibson.dropbear.id.au>> wrote:
> On Fri, Jan 15, 2021 at 10:49:34AM +0100, Christophe de Dinechin wrote:
> > 
> > 
> > > On 15 Jan 2021, at 04:29, David Gibson <david at gibson.dropbear.id.au <mailto:david at gibson.dropbear.id.au>> wrote:
> > > 
> > > Hi all,
> > > 
> > > I'm a pretty recently started Kata contributor.  One of many things I
> > > found frustrating when I started working on Kata was the way the code
> > > was split into a whole batch of repositories.  I'm relieved that
> > > they're now merged together for Kata 2.x.
> > > 
> > > However, the tests repository is still separate, and AIUI there are no
> > > plans to integrate it.  I still find that really annoying, so I want
> > > to make the case for integrating it into the kata-containers
> > > repository as well.
> > > 
> > > 1. It removes delays fixing test bugs
> > > 
> > > On several occasions when working on code changes, I've hit CI
> > > failures due to what turn out to be bugs in the tests code.  These
> > > have often been problems with static checks essentially unrelated to
> > > what I'm actually trying to accomplish.
> > > 
> > > To deal with this, I've had to file a separate issue and PR against
> > > the tests repo, wait for that to be tested and reviewed, and only then
> > > can I resume work on what I was actually trying to do.  Especially
> > > since I'm a different timezone from most of the Kata team, that can
> > > easily be several days of delay.
> > > 
> > > If the tests were integrated, I could include a fix for the test as a
> > > commit in the same PR, it would automatically fix the CI for that PR
> > > and could be reviewed with the same batch.
> > > 
> > > 2. Allows tests to be commited along with new features
> > > 
> > > Obviously it's desirable for new features to come with tests for those
> > > features.  Integrating the tests repo means that can be done in the
> > > same PR, reducing the amount of github busywork involved in doing so.
> > > It also means the code and tests can be reviewed and discussed in the
> > > same place.
> > > 
> > > In addition to being more convenient, that means we don't need to
> > > rerun basically the same tests for both the code PR and the tests PR.
> > > That means less demand on the CI infrastructure, which helps everyone.
> > > 
> > > 3. Testing scripts can be simplified
> > > 
> > > AFAICT the current scripts in the tests code have a bunch of
> > > conditionals for various different configurations: Kata1 vs. Kata 2.x,
> > > Rust agent vs. Go agent amongst others.  If the tests repo is
> > > integrated we can cut down the test scripts to just the cases that are
> > > relevant for the current version of the code.
> > > 
> > > Understanding how the test scripts work is a real barrier to debugging
> > > Kata at the moment, so simplifying them is valuable.
> > > 
> > > 3. Removes a barrier to running CI tests locally
> > > 
> > > At the moment debugging CI failures is really painful, because
> > > duplicating the problem in a debuggable environment is extremely
> > > awkward.  One (certainly not the only) reason for that is that the
> > > code repo and tests repo need to be both assembled together in a
> > > suitable environment.  The CI scripts do that, but this often involves
> > > downloading one or the other repo from a global URL.  If you want to
> > > do frequently repeated tests on a local repo while debugging, that has
> > > to be worked around.
> > > 
> > > If tests and code are in the same repo, it removes the need to
> > > download either one from a global URL, making it easier to replicate
> > > the CI test in a local environment and with an ad-hoc tree.  It also
> > > allows some further simplifications of the CI scripts.
> > > 
> > > 
> > > I could probably come up with more reasons, but those 3 seem like a
> > > solid start.
> > 
> > Solid +1 from me ;-)
> > 
> > Now, in the interest of exhaustivity, can you think of reasons not to?
> 
> I really can't, but I'm kind of assuming someone will tell me some.
> 
> > I can think of a couple, but I'm interested in your viewpoint here.
> > 
> 
> -- 
> David Gibson                    | I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au <http://gibson.dropbear.id.au/>  | minimalist, thank you.  NOT _the_ _other_
>                                 | _way_ _around_!
> http://www.ozlabs.org/~dgibson <http://www.ozlabs.org/~dgibson>
> _______________________________________________
> kata-dev mailing list
> kata-dev at lists.katacontainers.io <mailto:kata-dev at lists.katacontainers.io>
> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev <http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.katacontainers.io/pipermail/kata-dev/attachments/20210119/7dad4ffd/attachment.html>