Automated test focus / skip

[fedepaol]

Is your feature request related to a problem?

A new contributor might find daunting that when we run e2e we run ALL The tests, whereas what we run depends on the bgp mode and on the ip family (and on having or not prometheus there).

Describe the solution you'd like

I envision a mode we run dev-env with that checks the bgp type, the ip family and what else and provides the right focuses.
This has to be an opt-in feature. We want the CI to run with specific focuses that are hardcoded and driven from outside to avoid starting to have volkswagen tests (https://github.com/auchenberg/volkswagen)

We should document this new mode and promote it as the default when developing locally.

Additional context

No response

I've read and agree with the following

• I've checked all open and closed issues and my request is not there.
• I've checked all open and closed pull requests and my request is not there.

[fedepaol]

An easier way would be to panic in inv e2etest if no focus is provided, and print out the table of focus-es we need depending on the environment

[sbshah97]

How do we go about this task?

[fedepaol]

Hey @sbshah97 , I changed my mind when writing this, and instead of automation, we could go simpler and do the following

• running inv e2etest with no focus - skip parameter fails and prints the set of skips that must be applied depending on the environment, as a nicely formatted table
• the content of the table must mirror what we do in CI https://github.com/metallb/metallb/blob/main/.github/workflows/ci.yaml#L256

Something along the line of:

• ipv4 - frr - no prometheus | SKIP="FRRK8S-MODE|IPV6|DUALSTACK"

and so on.

This should help first time contributors in understanding what should they do when they want to test MetalLB locally.

[fedepaol]

@sbshah97 are you interested in working on this?

Additionally, it would make sense to maintain / build the list of skips in a single configuration file that is used both by CI and by the aforemenitioned logic.

[roopeshsn]

Hi, @fedepaol! I am interested in working on this issue. I am a first-time contributor to MetalLB. As of my understanding after reading your above comments, running inv e2etest with the skip flag (inv e2etest --skip BGP) should fail, right? and a table of tests that should be skipped based on the environment should be outputted. In the CI for E2E, it seems that the tests are skipped based on bgp-type and ip-family flags. Could you clarify on this?

[fedepaol]

Hi, @fedepaol! I am interested in working on this issue. I am a first-time contributor to MetalLB. As of my understanding after reading your above comments, running inv e2etest with the skip flag (inv e2etest --skip BGP) should fail, right?

Nope, it should fail if you don't pass any skips

and a table of tests that should be skipped based on the environment should be outputted. In the CI for E2E, it seems that the tests are skipped based on bgp-type and ip-family flags. Could you clarify on this?

Basically, there are some tests that are relevant only for the bgp implementation (we have a native one, an FRR based one and soon an FRR-K8s based one). Some other are valid only for IPv4, or IPv6.
That's reflected in inv dev-env parameters, and we want to skip the tests that are not relevant for the current deployment (and make the user aware).

Hope this helps

[roopeshsn]

Hi, @fedepaol! I am interested in working on this issue. I am a first-time contributor to MetalLB. As of my understanding after reading your above comments, running inv e2etest with the skip flag (inv e2etest --skip BGP) should fail, right?

Nope, it should fail if you don't pass any skips

and a table of tests that should be skipped based on the environment should be outputted. In the CI for E2E, it seems that the tests are skipped based on bgp-type and ip-family flags. Could you clarify on this?

Basically, there are some tests that are relevant only for the bgp implementation (we have a native one, an FRR based one and soon an FRR-K8s based one). Some other are valid only for IPv4, or IPv6. That's reflected in inv dev-env parameters, and we want to skip the tests that are not relevant for the current deployment (and make the user aware).

Hope this helps

Yep, this helps! So if I am executing the command inv e2etest, the command should fail and print a table with relevant and skipped tests. By seeing that table, the user can be aware of how to execute the relevant tests using the focus and skip flags.

If I am not using BGP mode, then the tests native, frr, and frr-k8s should be printed under the skipped tests. Correct me if I am wrong.

My other doubt is: how can I check whether the user is using L2 or BGP mode? Should I check whether one of the custom resources BGPAdvertisement or L2Advertisement is applied? @fedepaol

[roopeshsn]

Hi, @fedepaol! I am interested in working on this issue. I am a first-time contributor to MetalLB. As of my understanding after reading your above comments, running inv e2etest with the skip flag (inv e2etest --skip BGP) should fail, right?

Nope, it should fail if you don't pass any skips

and a table of tests that should be skipped based on the environment should be outputted. In the CI for E2E, it seems that the tests are skipped based on bgp-type and ip-family flags. Could you clarify on this?

Basically, there are some tests that are relevant only for the bgp implementation (we have a native one, an FRR based one and soon an FRR-K8s based one). Some other are valid only for IPv4, or IPv6. That's reflected in inv dev-env parameters, and we want to skip the tests that are not relevant for the current deployment (and make the user aware).
Hope this helps

Yep, this helps! So if I am executing the command inv e2etest, the command should fail and print a table with relevant and skipped tests. By seeing that table, the user can be aware of how to execute the relevant tests using the focus and skip flags.

If I am not using BGP mode, then the tests native, frr, and frr-k8s should be printed under the skipped tests. Correct me if I am wrong.

My other doubt is: how can I check whether the user is using L2 or BGP mode? Should I check whether one of the custom resources BGPAdvertisement or L2Advertisement is applied? @fedepaol

Hi, @fedepaol! If this point is clear, I'll get started working on it.