The case to integrate the tests repository
Hi all, I'm a pretty recently started Kata contributor. One of many things I found frustrating when I started working on Kata was the way the code was split into a whole batch of repositories. I'm relieved that they're now merged together for Kata 2.x. However, the tests repository is still separate, and AIUI there are no plans to integrate it. I still find that really annoying, so I want to make the case for integrating it into the kata-containers repository as well. 1. It removes delays fixing test bugs On several occasions when working on code changes, I've hit CI failures due to what turn out to be bugs in the tests code. These have often been problems with static checks essentially unrelated to what I'm actually trying to accomplish. To deal with this, I've had to file a separate issue and PR against the tests repo, wait for that to be tested and reviewed, and only then can I resume work on what I was actually trying to do. Especially since I'm a different timezone from most of the Kata team, that can easily be several days of delay. If the tests were integrated, I could include a fix for the test as a commit in the same PR, it would automatically fix the CI for that PR and could be reviewed with the same batch. 2. Allows tests to be commited along with new features Obviously it's desirable for new features to come with tests for those features. Integrating the tests repo means that can be done in the same PR, reducing the amount of github busywork involved in doing so. It also means the code and tests can be reviewed and discussed in the same place. In addition to being more convenient, that means we don't need to rerun basically the same tests for both the code PR and the tests PR. That means less demand on the CI infrastructure, which helps everyone. 3. Testing scripts can be simplified AFAICT the current scripts in the tests code have a bunch of conditionals for various different configurations: Kata1 vs. Kata 2.x, Rust agent vs. Go agent amongst others. If the tests repo is integrated we can cut down the test scripts to just the cases that are relevant for the current version of the code. Understanding how the test scripts work is a real barrier to debugging Kata at the moment, so simplifying them is valuable. 3. Removes a barrier to running CI tests locally At the moment debugging CI failures is really painful, because duplicating the problem in a debuggable environment is extremely awkward. One (certainly not the only) reason for that is that the code repo and tests repo need to be both assembled together in a suitable environment. The CI scripts do that, but this often involves downloading one or the other repo from a global URL. If you want to do frequently repeated tests on a local repo while debugging, that has to be worked around. If tests and code are in the same repo, it removes the need to download either one from a global URL, making it easier to replicate the CI test in a local environment and with an ad-hoc tree. It also allows some further simplifications of the CI scripts. I could probably come up with more reasons, but those 3 seem like a solid start. -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
On 15 Jan 2021, at 04:29, David Gibson <david@gibson.dropbear.id.au> wrote:
Hi all,
I'm a pretty recently started Kata contributor. One of many things I found frustrating when I started working on Kata was the way the code was split into a whole batch of repositories. I'm relieved that they're now merged together for Kata 2.x.
However, the tests repository is still separate, and AIUI there are no plans to integrate it. I still find that really annoying, so I want to make the case for integrating it into the kata-containers repository as well.
1. It removes delays fixing test bugs
On several occasions when working on code changes, I've hit CI failures due to what turn out to be bugs in the tests code. These have often been problems with static checks essentially unrelated to what I'm actually trying to accomplish.
To deal with this, I've had to file a separate issue and PR against the tests repo, wait for that to be tested and reviewed, and only then can I resume work on what I was actually trying to do. Especially since I'm a different timezone from most of the Kata team, that can easily be several days of delay.
If the tests were integrated, I could include a fix for the test as a commit in the same PR, it would automatically fix the CI for that PR and could be reviewed with the same batch.
2. Allows tests to be commited along with new features
Obviously it's desirable for new features to come with tests for those features. Integrating the tests repo means that can be done in the same PR, reducing the amount of github busywork involved in doing so. It also means the code and tests can be reviewed and discussed in the same place.
In addition to being more convenient, that means we don't need to rerun basically the same tests for both the code PR and the tests PR. That means less demand on the CI infrastructure, which helps everyone.
3. Testing scripts can be simplified
AFAICT the current scripts in the tests code have a bunch of conditionals for various different configurations: Kata1 vs. Kata 2.x, Rust agent vs. Go agent amongst others. If the tests repo is integrated we can cut down the test scripts to just the cases that are relevant for the current version of the code.
Understanding how the test scripts work is a real barrier to debugging Kata at the moment, so simplifying them is valuable.
3. Removes a barrier to running CI tests locally
At the moment debugging CI failures is really painful, because duplicating the problem in a debuggable environment is extremely awkward. One (certainly not the only) reason for that is that the code repo and tests repo need to be both assembled together in a suitable environment. The CI scripts do that, but this often involves downloading one or the other repo from a global URL. If you want to do frequently repeated tests on a local repo while debugging, that has to be worked around.
If tests and code are in the same repo, it removes the need to download either one from a global URL, making it easier to replicate the CI test in a local environment and with an ad-hoc tree. It also allows some further simplifications of the CI scripts.
I could probably come up with more reasons, but those 3 seem like a solid start.
Solid +1 from me ;-) Now, in the interest of exhaustivity, can you think of reasons not to? I can think of a couple, but I'm interested in your viewpoint here.
On Fri, Jan 15, 2021 at 10:49:34AM +0100, Christophe de Dinechin wrote:
On 15 Jan 2021, at 04:29, David Gibson <david@gibson.dropbear.id.au> wrote:
Hi all,
I'm a pretty recently started Kata contributor. One of many things I found frustrating when I started working on Kata was the way the code was split into a whole batch of repositories. I'm relieved that they're now merged together for Kata 2.x.
However, the tests repository is still separate, and AIUI there are no plans to integrate it. I still find that really annoying, so I want to make the case for integrating it into the kata-containers repository as well.
1. It removes delays fixing test bugs
On several occasions when working on code changes, I've hit CI failures due to what turn out to be bugs in the tests code. These have often been problems with static checks essentially unrelated to what I'm actually trying to accomplish.
To deal with this, I've had to file a separate issue and PR against the tests repo, wait for that to be tested and reviewed, and only then can I resume work on what I was actually trying to do. Especially since I'm a different timezone from most of the Kata team, that can easily be several days of delay.
If the tests were integrated, I could include a fix for the test as a commit in the same PR, it would automatically fix the CI for that PR and could be reviewed with the same batch.
2. Allows tests to be commited along with new features
Obviously it's desirable for new features to come with tests for those features. Integrating the tests repo means that can be done in the same PR, reducing the amount of github busywork involved in doing so. It also means the code and tests can be reviewed and discussed in the same place.
In addition to being more convenient, that means we don't need to rerun basically the same tests for both the code PR and the tests PR. That means less demand on the CI infrastructure, which helps everyone.
3. Testing scripts can be simplified
AFAICT the current scripts in the tests code have a bunch of conditionals for various different configurations: Kata1 vs. Kata 2.x, Rust agent vs. Go agent amongst others. If the tests repo is integrated we can cut down the test scripts to just the cases that are relevant for the current version of the code.
Understanding how the test scripts work is a real barrier to debugging Kata at the moment, so simplifying them is valuable.
3. Removes a barrier to running CI tests locally
At the moment debugging CI failures is really painful, because duplicating the problem in a debuggable environment is extremely awkward. One (certainly not the only) reason for that is that the code repo and tests repo need to be both assembled together in a suitable environment. The CI scripts do that, but this often involves downloading one or the other repo from a global URL. If you want to do frequently repeated tests on a local repo while debugging, that has to be worked around.
If tests and code are in the same repo, it removes the need to download either one from a global URL, making it easier to replicate the CI test in a local environment and with an ad-hoc tree. It also allows some further simplifications of the CI scripts.
I could probably come up with more reasons, but those 3 seem like a solid start.
Solid +1 from me ;-)
Now, in the interest of exhaustivity, can you think of reasons not to?
I really can't, but I'm kind of assuming someone will tell me some.
I can think of a couple, but I'm interested in your viewpoint here.
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
David, All the issues you have pointed out are definitely valid and need to be resolved. I think all t of these have been discussed in the ongoing effort to improve Upstream CI. The suggestion of merging the tests repo has been brought up before with some debate. I would like to point out that we also have a CI repo that is separate as well that is also part of testing and CI. The current approach to improvements are described in https://docs.google.com/document/d/15YobCIUFMEvdNIGem0CJ_g30hJY4YH5Lr6ty7_8A... It takes the approach of trying to understand and clean up what currently exists and solve he issues of test maintenance, debugging failures, and local execution of CI tests. I think all of these can be resolved to various degrees without merging repos. That is not an argument against the idea but a preference to do what we can that is not dependent on where the code/tests lives. Your point 2 is the most compelling to me for merging repos, Having tests committed along with new features would be a great place be. Currently I don't see many tests being committed, unit, manual verification, or automation. Maybe I am missing them (can't review every PR). I would like to understand if the state of having separate test repos is a factor in not getting more tests with code changes, or if the other issues are really the cuplrit. I suspect the later because I have seen 100% code coverage through integration tests being committed separately along with code changes when integration tests were in a separate repo (unit tests in repo with code). That's my two cents. Thank you for reaffirming that the issues we identified are still issues and that we really need to put more effort in to resolve them. On Fri, Jan 15, 2021 at 8:27 AM David Gibson <david@gibson.dropbear.id.au> wrote:
On Fri, Jan 15, 2021 at 10:49:34AM +0100, Christophe de Dinechin wrote:
On 15 Jan 2021, at 04:29, David Gibson <david@gibson.dropbear.id.au>
wrote:
Hi all,
I'm a pretty recently started Kata contributor. One of many things I found frustrating when I started working on Kata was the way the code was split into a whole batch of repositories. I'm relieved that they're now merged together for Kata 2.x.
However, the tests repository is still separate, and AIUI there are no plans to integrate it. I still find that really annoying, so I want to make the case for integrating it into the kata-containers repository as well.
1. It removes delays fixing test bugs
On several occasions when working on code changes, I've hit CI failures due to what turn out to be bugs in the tests code. These have often been problems with static checks essentially unrelated to what I'm actually trying to accomplish.
To deal with this, I've had to file a separate issue and PR against the tests repo, wait for that to be tested and reviewed, and only then can I resume work on what I was actually trying to do. Especially since I'm a different timezone from most of the Kata team, that can easily be several days of delay.
If the tests were integrated, I could include a fix for the test as a commit in the same PR, it would automatically fix the CI for that PR and could be reviewed with the same batch.
2. Allows tests to be commited along with new features
Obviously it's desirable for new features to come with tests for those features. Integrating the tests repo means that can be done in the same PR, reducing the amount of github busywork involved in doing so. It also means the code and tests can be reviewed and discussed in the same place.
In addition to being more convenient, that means we don't need to rerun basically the same tests for both the code PR and the tests PR. That means less demand on the CI infrastructure, which helps everyone.
3. Testing scripts can be simplified
AFAICT the current scripts in the tests code have a bunch of conditionals for various different configurations: Kata1 vs. Kata 2.x, Rust agent vs. Go agent amongst others. If the tests repo is integrated we can cut down the test scripts to just the cases that are relevant for the current version of the code.
Understanding how the test scripts work is a real barrier to debugging Kata at the moment, so simplifying them is valuable.
3. Removes a barrier to running CI tests locally
At the moment debugging CI failures is really painful, because duplicating the problem in a debuggable environment is extremely awkward. One (certainly not the only) reason for that is that the code repo and tests repo need to be both assembled together in a suitable environment. The CI scripts do that, but this often involves downloading one or the other repo from a global URL. If you want to do frequently repeated tests on a local repo while debugging, that has to be worked around.
If tests and code are in the same repo, it removes the need to download either one from a global URL, making it easier to replicate the CI test in a local environment and with an ad-hoc tree. It also allows some further simplifications of the CI scripts.
I could probably come up with more reasons, but those 3 seem like a solid start.
Solid +1 from me ;-)
Now, in the interest of exhaustivity, can you think of reasons not to?
I really can't, but I'm kind of assuming someone will tell me some.
I can think of a couple, but I'm interested in your viewpoint here.
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
On 15 Jan 2021, at 16:07, Cameron Meadors <cmeadors@redhat.com> wrote:
David,
All the issues you have pointed out are definitely valid and need to be resolved. I think all t of these have been discussed in the ongoing effort to improve Upstream CI. The suggestion of merging the tests repo has been brought up before with some debate. I would like to point out that we also have a CI repo that is separate as well that is also part of testing and CI.
The current approach to improvements are described in https://docs.google.com/document/d/15YobCIUFMEvdNIGem0CJ_g30hJY4YH5Lr6ty7_8A... <https://docs.google.com/document/d/15YobCIUFMEvdNIGem0CJ_g30hJY4YH5Lr6ty7_8ASPY/edit> It takes the approach of trying to understand and clean up what currently exists and solve he issues of test maintenance, debugging failures, and local execution of CI tests. I think all of these can be resolved to various degrees without merging repos. That is not an argument against the idea but a preference to do what we can that is not dependent on where the code/tests lives.
Your point 2 is the most compelling to me for merging repos, Having tests committed along with new features would be a great place be. Currently I don't see many tests being committed, unit, manual verification, or automation. Maybe I am missing them (can't review every PR). I would like to understand if the state of having separate test repos is a factor in not getting more tests with code changes, or if the other issues are really the cuplrit. I suspect the later because I have seen 100% code coverage through integration tests being committed separately along with code changes when integration tests were in a separate repo (unit tests in repo with code).
It is definitely a factor. - If you push the test first, then other commits will presumably fail. - If you push the code first, chances are it does not pass the test in the CI. So right now, pushing a test is hard (on a scale of 1 to 10, I'd say a solid 20).
That's my two cents. Thank you for reaffirming that the issues we identified are still issues and that we really need to put more effort in to resolve them.
On Fri, Jan 15, 2021 at 8:27 AM David Gibson <david@gibson.dropbear.id.au <mailto:david@gibson.dropbear.id.au>> wrote: On Fri, Jan 15, 2021 at 10:49:34AM +0100, Christophe de Dinechin wrote:
On 15 Jan 2021, at 04:29, David Gibson <david@gibson.dropbear.id.au <mailto:david@gibson.dropbear.id.au>> wrote:
Hi all,
I'm a pretty recently started Kata contributor. One of many things I found frustrating when I started working on Kata was the way the code was split into a whole batch of repositories. I'm relieved that they're now merged together for Kata 2.x.
However, the tests repository is still separate, and AIUI there are no plans to integrate it. I still find that really annoying, so I want to make the case for integrating it into the kata-containers repository as well.
1. It removes delays fixing test bugs
On several occasions when working on code changes, I've hit CI failures due to what turn out to be bugs in the tests code. These have often been problems with static checks essentially unrelated to what I'm actually trying to accomplish.
To deal with this, I've had to file a separate issue and PR against the tests repo, wait for that to be tested and reviewed, and only then can I resume work on what I was actually trying to do. Especially since I'm a different timezone from most of the Kata team, that can easily be several days of delay.
If the tests were integrated, I could include a fix for the test as a commit in the same PR, it would automatically fix the CI for that PR and could be reviewed with the same batch.
2. Allows tests to be commited along with new features
Obviously it's desirable for new features to come with tests for those features. Integrating the tests repo means that can be done in the same PR, reducing the amount of github busywork involved in doing so. It also means the code and tests can be reviewed and discussed in the same place.
In addition to being more convenient, that means we don't need to rerun basically the same tests for both the code PR and the tests PR. That means less demand on the CI infrastructure, which helps everyone.
3. Testing scripts can be simplified
AFAICT the current scripts in the tests code have a bunch of conditionals for various different configurations: Kata1 vs. Kata 2.x, Rust agent vs. Go agent amongst others. If the tests repo is integrated we can cut down the test scripts to just the cases that are relevant for the current version of the code.
Understanding how the test scripts work is a real barrier to debugging Kata at the moment, so simplifying them is valuable.
3. Removes a barrier to running CI tests locally
At the moment debugging CI failures is really painful, because duplicating the problem in a debuggable environment is extremely awkward. One (certainly not the only) reason for that is that the code repo and tests repo need to be both assembled together in a suitable environment. The CI scripts do that, but this often involves downloading one or the other repo from a global URL. If you want to do frequently repeated tests on a local repo while debugging, that has to be worked around.
If tests and code are in the same repo, it removes the need to download either one from a global URL, making it easier to replicate the CI test in a local environment and with an ad-hoc tree. It also allows some further simplifications of the CI scripts.
I could probably come up with more reasons, but those 3 seem like a solid start.
Solid +1 from me ;-)
Now, in the interest of exhaustivity, can you think of reasons not to?
I really can't, but I'm kind of assuming someone will tell me some.
I can think of a couple, but I'm interested in your viewpoint here.
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au <http://gibson.dropbear.id.au/> | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson <http://www.ozlabs.org/~dgibson> _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io <mailto:kata-dev@lists.katacontainers.io> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev <http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev>
On Fri, Jan 15, 2021 at 10:07:23AM -0500, Cameron Meadors wrote:
David,
All the issues you have pointed out are definitely valid and need to be resolved. I think all t of these have been discussed in the ongoing effort to improve Upstream CI. The suggestion of merging the tests repo has been brought up before with some debate. I would like to point out that we also have a CI repo that is separate as well that is also part of testing and CI.
The current approach to improvements are described in https://docs.google.com/document/d/15YobCIUFMEvdNIGem0CJ_g30hJY4YH5Lr6ty7_8A... It takes the approach of trying to understand and clean up what currently exists and solve he issues of test maintenance, debugging failures, and local execution of CI tests. I think all of these can be resolved to various degrees without merging repos. That is not an argument against the idea but a preference to do what we can that is not dependent on where the code/tests lives.
Your point 2 is the most compelling to me for merging repos, Having tests
I can seem that being valuable long term, but actually point 2 is the least pressing from my point of view. (1) and things in its orbit is the real kicker for me. I'm assuming the theory in separate repos is that most of the time the tests should remain static and keep verifying code changes. However, my practical experience so far really doesn't match that. I'd say that something like half of the code change PRs I've worked on have ended up requiring tests changes indirectly. That's mostly not been a true requirement for synchronized changes between the tests and code, but instead things like needing a tests change to avoid ugly workarounds on the code side for overly-zealous static checks and the like. This includes with pretty simple PRs, so adding a few days of diversion into a tests fix isn't trivial. Neither is the hassle of filing an extra issue and PR in github. In fact stuff I was doing today made me think of a variant on (1) that's even harder to work around: (1A) Sometimes a code change will cause a CI regression that you're not able to reproduce locally. Right now that's *very* common because reproducing tests locally is a nightmare. Obviously we want to fix that, but it will take a while, and even then it's likely to happen occasionally. To debug that case, the obvious way forward is sometimes to add temporary instrumentation to the CI tests. With integrated tests, that's easy: just add the instrumentation to the PR while it's still a WiP, feed it into the CI and debug. With separate tests repo, it's essentially impossible - we don't want to merge that sort of temporary hack into the "real" tests repo.
committed along with new features would be a great place be. Currently I don't see many tests being committed, unit, manual verification, or automation. Maybe I am missing them (can't review every PR). I would like to understand if the state of having separate test repos is a factor in not getting more tests with code changes, or if the other issues are really the cuplrit. I suspect the later because I have seen 100% code coverage through integration tests being committed separately along with code changes when integration tests were in a separate repo (unit tests in repo with code).
That's my two cents. Thank you for reaffirming that the issues we identified are still issues and that we really need to put more effort in to resolve them.
On Fri, Jan 15, 2021 at 8:27 AM David Gibson <david@gibson.dropbear.id.au> wrote:
On Fri, Jan 15, 2021 at 10:49:34AM +0100, Christophe de Dinechin wrote:
On 15 Jan 2021, at 04:29, David Gibson <david@gibson.dropbear.id.au>
wrote:
Hi all,
I'm a pretty recently started Kata contributor. One of many things I found frustrating when I started working on Kata was the way the code was split into a whole batch of repositories. I'm relieved that they're now merged together for Kata 2.x.
However, the tests repository is still separate, and AIUI there are no plans to integrate it. I still find that really annoying, so I want to make the case for integrating it into the kata-containers repository as well.
1. It removes delays fixing test bugs
On several occasions when working on code changes, I've hit CI failures due to what turn out to be bugs in the tests code. These have often been problems with static checks essentially unrelated to what I'm actually trying to accomplish.
To deal with this, I've had to file a separate issue and PR against the tests repo, wait for that to be tested and reviewed, and only then can I resume work on what I was actually trying to do. Especially since I'm a different timezone from most of the Kata team, that can easily be several days of delay.
If the tests were integrated, I could include a fix for the test as a commit in the same PR, it would automatically fix the CI for that PR and could be reviewed with the same batch.
2. Allows tests to be commited along with new features
Obviously it's desirable for new features to come with tests for those features. Integrating the tests repo means that can be done in the same PR, reducing the amount of github busywork involved in doing so. It also means the code and tests can be reviewed and discussed in the same place.
In addition to being more convenient, that means we don't need to rerun basically the same tests for both the code PR and the tests PR. That means less demand on the CI infrastructure, which helps everyone.
3. Testing scripts can be simplified
AFAICT the current scripts in the tests code have a bunch of conditionals for various different configurations: Kata1 vs. Kata 2.x, Rust agent vs. Go agent amongst others. If the tests repo is integrated we can cut down the test scripts to just the cases that are relevant for the current version of the code.
Understanding how the test scripts work is a real barrier to debugging Kata at the moment, so simplifying them is valuable.
3. Removes a barrier to running CI tests locally
At the moment debugging CI failures is really painful, because duplicating the problem in a debuggable environment is extremely awkward. One (certainly not the only) reason for that is that the code repo and tests repo need to be both assembled together in a suitable environment. The CI scripts do that, but this often involves downloading one or the other repo from a global URL. If you want to do frequently repeated tests on a local repo while debugging, that has to be worked around.
If tests and code are in the same repo, it removes the need to download either one from a global URL, making it easier to replicate the CI test in a local environment and with an ad-hoc tree. It also allows some further simplifications of the CI scripts.
I could probably come up with more reasons, but those 3 seem like a solid start.
Solid +1 from me ;-)
Now, in the interest of exhaustivity, can you think of reasons not to?
I really can't, but I'm kind of assuming someone will tell me some.
I can think of a couple, but I'm interested in your viewpoint here.
-- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
participants (3)
-
Cameron Meadors
-
Christophe de Dinechin
-
David Gibson