Open Stack

On Wed, Aug 18, 2021 at 5:20 AM Jakob Naucke <naucke at linux.vnet.ibm.com>
wrote:

> On 17.08.21 20:59, Cameron Meadors wrote:
> > The comment is that there is value in separating e2e/integration tests
> > that are not version specific from component/unit tests that are.
>
> That's valid, but...
>
> > I am not sure if we know which tests are version specific or not and
> > if we are ready to make a forward looking decisions like this now, but
> > I think keeping the test repo but only having non-versions specific
> > tests there would solve the problems that have been presented.
>
> ...wouldn't this mean that we could never have E2E version-specific
> tests? I.e., to test a new feature E2E, it has to be released as stable
> first? I don't think that could work.
>
> I see how this could sound confusing.  In my mind there are two ways the
tests in the tests repo can be used.  There are CI run , which are
regression and only include tests for released features.  Then there are
automated test runs for feature verification.  The case you are talking
about is the later case.  There would need to be some very high level
branching in the test repo that follows releases in the kata-container repo
to manage this. Main or stable release branches in tests would only
container regression tests.  New tests would go in branches for a new
release.

In other words.  new features are version specific in that they only exist
in release foo and newer, but once the feature is released.  If the feature
does not change with each new release then that would be considered a
non-version specific feature.  As i understand it, we only really support
the latest release in anyway.  The process to manage running the right
tests on the right versions should be simple.  One common way is to couple
it with feature flags so tests only run for enabled features.  But that
requires a lot of work to put in place, easy once there.

Not trying to trivialize this issue.  I am just throwing out how I have
managed testing for other projects.  That may not work here.  There is also
the fact that the tests repo has a lot of stuff in it that may or may not
work the way people want it to work.  There might be some clever changes
that take care of multiple issues.


> --
> Jakob Naucke
> IBM Deutschland Research & Development GmbH
> Vorsitzender des Aufsichtsrats: Gregor Pillen
> Geschäftsführung: Dirk Wittkopp; Sitz der Gesellschaft: Böblingen
> Registergericht: Amtsgericht Stuttgart, HRB 243294
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.katacontainers.io/pipermail/kata-dev/attachments/20210819/f844312c/attachment.html>