About merging the tests repo into the kata-containers one
During the last architecture committee meeting (Aug 10th, 2021) this topic was brought up again and it's driven by simplifying the current workflow. Archana & Eric raised some concerns about the amount of churn / commit pollution in the kata-containers repository. This discussion is not new, we've had it on https://github.com/kata-containers/kata-containers/issues/187 and we even agreed in one of the meetings that merging the repo would be the way to go, although it'd generate a lot of work that no-one at that point would be willing to take. Regardless of what was discussed, we didn't get the input of people who are majoritarily working on that repo, being Gabriela & Salvador the main ones, and I know they may have some concerns about the topic. So, Gabriela, Salvador, would you mind sharing the concerns? One last thing from my side, while I'm in favour of doing the move, I'd like to ensure we take the decision based on the right motivations. Merging the repos will *not* solve the problems we have on maintaining the stable branches and, for that, please take a look at Gabriela's proposal https://github.com/kata-containers/kata-containers/issues/2429 Please, share your opinions, concerns, and ideas. Best Regards, -- Fabiano Fidêncio
I only have one comment that I have already made and it got some traction so worth bringing up again. The comment is that there is value in separating e2e/integration tests that are not version specific from component/unit tests that are. There should be much less churn in the non-version specific tests therefore reducing the overhead of having a separate repo. On the other hand, changes to version specific tests normally are made at the same time as their respective code changes. I am not sure if we know which tests are version specific or not and if we are ready to make a forward looking decisions like this now, but I think keeping the test repo but only having non-versions specific tests there would solve the problems that have been presented. I think it also might solve the problem that Gabriela's proposal would solve, although I don't fully understand that problem. Anyway, just wanted to bring that up again. One single repo or two separate repos will both work. Just a matter of the tradeoffs. On Mon, Aug 16, 2021 at 8:44 AM Fabiano Fidêncio <fabiano@fidencio.org> wrote:
During the last architecture committee meeting (Aug 10th, 2021) this topic was brought up again and it's driven by simplifying the current workflow.
Archana & Eric raised some concerns about the amount of churn / commit pollution in the kata-containers repository.
This discussion is not new, we've had it on https://github.com/kata-containers/kata-containers/issues/187 and we even agreed in one of the meetings that merging the repo would be the way to go, although it'd generate a lot of work that no-one at that point would be willing to take.
Regardless of what was discussed, we didn't get the input of people who are majoritarily working on that repo, being Gabriela & Salvador the main ones, and I know they may have some concerns about the topic. So, Gabriela, Salvador, would you mind sharing the concerns?
One last thing from my side, while I'm in favour of doing the move, I'd like to ensure we take the decision based on the right motivations. Merging the repos will *not* solve the problems we have on maintaining the stable branches and, for that, please take a look at Gabriela's proposal https://github.com/kata-containers/kata-containers/issues/2429
Please, share your opinions, concerns, and ideas.
Best Regards, -- Fabiano Fidêncio
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
Hi all, I am not sure that merging the test repository on the kata-containers repository will help to make easier the release process or if we will gain any benefit in doing it. I see a lot issues and a lot work that will need to be done if we merge the test repository into kata-containers. First, we will need to change the CI installation scripts, the tests scripts, the Jenkins jobs configurations, the scripts that are running on the Github Actions, etc. Second, when we raise the PR to merge tests in kata-containers, we will need to create a special CI to test this PR because if we only merge it without testing it well that will be a disaster. Third, do we have a strategy to merge PRs that are merged later on the test repository or what we are going to do? Now about the release, I strongly believe that even if we merge the repos the release process the problem will be still there. The main issue why we spend a lot of time in doing releases is because the lack of backporting. In my experience, I have face that people do not backport their PRs that are merge on main into the stable branches maybe because they forget, they do not know if a certain PR needs a backport or they just do not care about stable branches. If someone is doing a backport and a certain CI is failing (which is non familiar or it is not the owner) an issue needs to be created and ask the owners to start to take a look in order to have help and fix the issue before the release process begins. I noticed that most of the time PRs are merged and a certain CI is failing and no issues are created at all. Doing the merging of the repositories will require effort and time, even if a certain person only does the job this will require guidance and support from other members and I am not sure if this is a priority for the project. Regards, Gabriela From: Cameron Meadors <cmeadors@redhat.com> Sent: Tuesday, August 17, 2021 2:00 PM To: Fabiano Fidêncio <fabiano@fidencio.org> Cc: kata-dev <kata-dev@lists.katacontainers.io> Subject: Re: [kata-dev] About merging the tests repo into the kata-containers one I only have one comment that I have already made and it got some traction so worth bringing up again. The comment is that there is value in separating e2e/integration tests that are not version specific from component/unit tests that are. There should be much less churn in the non-version specific tests therefore reducing the overhead of having a separate repo. On the other hand, changes to version specific tests normally are made at the same time as their respective code changes. I am not sure if we know which tests are version specific or not and if we are ready to make a forward looking decisions like this now, but I think keeping the test repo but only having non-versions specific tests there would solve the problems that have been presented. I think it also might solve the problem that Gabriela's proposal would solve, although I don't fully understand that problem. Anyway, just wanted to bring that up again. One single repo or two separate repos will both work. Just a matter of the tradeoffs. On Mon, Aug 16, 2021 at 8:44 AM Fabiano Fidêncio <fabiano@fidencio.org<mailto:fabiano@fidencio.org>> wrote: During the last architecture committee meeting (Aug 10th, 2021) this topic was brought up again and it's driven by simplifying the current workflow. Archana & Eric raised some concerns about the amount of churn / commit pollution in the kata-containers repository. This discussion is not new, we've had it on https://github.com/kata-containers/kata-containers/issues/187 and we even agreed in one of the meetings that merging the repo would be the way to go, although it'd generate a lot of work that no-one at that point would be willing to take. Regardless of what was discussed, we didn't get the input of people who are majoritarily working on that repo, being Gabriela & Salvador the main ones, and I know they may have some concerns about the topic. So, Gabriela, Salvador, would you mind sharing the concerns? One last thing from my side, while I'm in favour of doing the move, I'd like to ensure we take the decision based on the right motivations. Merging the repos will *not* solve the problems we have on maintaining the stable branches and, for that, please take a look at Gabriela's proposal https://github.com/kata-containers/kata-containers/issues/2429 Please, share your opinions, concerns, and ideas. Best Regards, -- Fabiano Fidêncio _______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io<mailto:kata-dev@lists.katacontainers.io> http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
On 17.08.21 20:59, Cameron Meadors wrote:
The comment is that there is value in separating e2e/integration tests that are not version specific from component/unit tests that are.
That's valid, but...
I am not sure if we know which tests are version specific or not and if we are ready to make a forward looking decisions like this now, but I think keeping the test repo but only having non-versions specific tests there would solve the problems that have been presented.
...wouldn't this mean that we could never have E2E version-specific tests? I.e., to test a new feature E2E, it has to be released as stable first? I don't think that could work. -- Jakob Naucke IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Gregor Pillen Geschäftsführung: Dirk Wittkopp; Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
On Wed, Aug 18, 2021 at 5:20 AM Jakob Naucke <naucke@linux.vnet.ibm.com> wrote:
On 17.08.21 20:59, Cameron Meadors wrote:
The comment is that there is value in separating e2e/integration tests that are not version specific from component/unit tests that are.
That's valid, but...
I am not sure if we know which tests are version specific or not and if we are ready to make a forward looking decisions like this now, but I think keeping the test repo but only having non-versions specific tests there would solve the problems that have been presented.
...wouldn't this mean that we could never have E2E version-specific tests? I.e., to test a new feature E2E, it has to be released as stable first? I don't think that could work.
I see how this could sound confusing. In my mind there are two ways the tests in the tests repo can be used. There are CI run , which are regression and only include tests for released features. Then there are automated test runs for feature verification. The case you are talking about is the later case. There would need to be some very high level branching in the test repo that follows releases in the kata-container repo to manage this. Main or stable release branches in tests would only container regression tests. New tests would go in branches for a new release.
In other words. new features are version specific in that they only exist in release foo and newer, but once the feature is released. If the feature does not change with each new release then that would be considered a non-version specific feature. As i understand it, we only really support the latest release in anyway. The process to manage running the right tests on the right versions should be simple. One common way is to couple it with feature flags so tests only run for enabled features. But that requires a lot of work to put in place, easy once there. Not trying to trivialize this issue. I am just throwing out how I have managed testing for other projects. That may not work here. There is also the fact that the tests repo has a lot of stuff in it that may or may not work the way people want it to work. There might be some clever changes that take care of multiple issues.
-- Jakob Naucke IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Gregor Pillen Geschäftsführung: Dirk Wittkopp; Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
participants (4)
-
Cameron Meadors
-
Cervantes Tellez, Gabriela
-
Fabiano Fidêncio
-
Jakob Naucke