AlertLogPkg: Make it possible so that on ReportAlerts, a vacuous pass results in failure.

[SkydiverTricky]

Currently, if you call ReportAlerts, if no affirmations were checked then a test always passes. In a testbench where the UUT has locked up and the testbench has timed out, this may give a false sense of security in a CI environment.

Currently I have the following check in my generic OSVVM control entity:

if not G_ALLOW_VACUOUS_PASS then
   AlertIf(G_ALERT_ID, GetAffirmCount = 0, "Test Ended with no affirmations checked", FAILURE);
end if;

ReportAlerts;

Would it be nicer to pass the required affirm count to the ReportAlerts method, or some machinism to turn a Vacuous pass into a failure within OSVVM itself?

[SkydiverTricky]

Ideally, it would be possible to set Vacuous Pass failure per AlertID/Scoreboard. I have Scoreboards that I expect to actually check things, and sometimes scoreboards exist in VIP modules that are not used in a specific test. (eg. I have a AXI4 VIP module that has checkers for AWADDR, WDATA and ARADDR, and a particular testbench may only be performing writes to the AXI4 i/f so ARADDR is unused).

My current implementation requires a finish signal to mark the end of a test:

action_end_proc : process
  begin

    wait until finish;

    AlertIf(sboard.GetAlertLogID,
            not sboard.empty,
            "Simulation Finished. Scoreboard still has expected values to check.",
            ERROR );

    if not G_ALLOW_VACUOUS_PASS then
      AlertIf(sboard.GetAlertLogID,
              sboard.getCheckCount = 0,
              "Simulation finished with 0 checks",
              FAILURE );
    end if;

    -- Allow tvalid to drop
    wait_for_clks(1, axi_clk);


    if G_CHECK_NOT_VALID_AT_END then
      AlertIf(sboard.GetAlertLogID,
              axi_valid = '1',
              "UUT still has data to output - this is not expected",
              FAILURE );
    end if;


    wait;
  end process;

[JimLewis]

@SkydiverTricky
Currently adding affirmation counts to each level (each level has its own count and the top a summation of the individual counts)

If we add G_ALLOW_VACUOUS_PASS for scoreboards, why would we not make it a general thing for each AlertLogID? If we invert the check, it then amounts to a coverage goal on an AlertLogID basis.

I have also been working on requirements tracking. It is interesting and challenging as
it is a little like Coverge and a little like self-checking as there can be failures and one
failure within a requirements check indicates the test failed. I have a experimental / toy
package that currently layers on top of CoveragePkg, however, if each AlertLogID had
a coverage goal that could be set either in the call to GetAlertLogID or in a call to
SetAlertLogCovGoal (to turn them off or on for a specific test), then it might be
interesting.

Still brainstorming that requirements tracking.

[SkydiverTricky]

Having affirmation goals per ID would certainly be useful. As I said here in a test where multiple checkers are involved, having the test report a pass when one channel locked up and didnt meet an Affirmation goal could currently lead a CI system to pass - potentially leading to confusion in the delivered system.

[SkydiverTricky]

If you are talking about goals - it might be useful to be able to specify specific numbers and ranges. I can see all of the following being useful goals:

• non 0
• specific number (including 0)
• A > GOAL > B

[JimLewis]

Given that you can do this, why add something inside.
AlertIf(G_ALERT_ID, GetAffirmCount = 0, "Test Ended with no affirmations checked", FAILURE);

If we add something internal, you would loose the insight as to what the value means.
Where as when you expose it like this, it is there for all to see.

Said another way, abstraction is cool - except when it obfuscates things.

[JimLewis]

Note to self: If the alert is configured to have a requirement goal or set up as a requirement, then there are settings that will allow a requirement that is not met to be treated as a test failure.

I expect to be doing another pass through requirement tracking later in 2022. I will look at this in more detail then.