Skip to content

[FEATURE] Added a "Never Worked" state to the tests#1071

Open
pulk17 wants to merge 1 commit intoCCExtractor:masterfrom
pulk17:feat/never-worked
Open

[FEATURE] Added a "Never Worked" state to the tests#1071
pulk17 wants to merge 1 commit intoCCExtractor:masterfrom
pulk17:feat/never-worked

Conversation

@pulk17
Copy link

@pulk17 pulk17 commented Mar 21, 2026

[FEATURE] Add "Never Worked" status for regression tests

In raising this pull request, I confirm the following (please check boxes):

  • I have read and understood the contributors guide.
  • I have checked that another pull request for this purpose does not exist.
  • I have considered, and confirmed that this submission will be valuable to others.
  • I accept that this submission may not be used, and the pull request closed at the will of the maintainer.
  • I give this submission freely, and claim no ownership to its content.

My familiarity with the project is as follows (check one):

  • I have never used the project.
  • I have used the project briefly.
  • I have used the project extensively, but have not contributed previously.
  • I am an active contributor to the project.

What's the problem?

Right now, every regression test is either Pass or Fail. The issue is that some samples have never worked with any version of CCExtractor — they're broken by nature, not because of anything a PR did. When these tests fail in CI, they show up as regressions in the PR comment alongside tests that the PR genuinely broke. This makes it harder to see what actually needs attention, and it's unfair to contributors whose PRs get flagged for failures that existed long before their changes.

There's currently no way to communicate "yes, this test is failing, but it has always failed and this is a known pre-existing issue" — the system just treats it the same as a real regression.

What this PR does

Adds a third classification: Never Worked.

image

Admins can flag any regression test as "never worked" from the test edit and add pages using a simple checkbox. Once flagged, the test still runs as normal, but its failure is interpreted differently across the whole system:

image
  • It gets its own clearly labeled section in the PR comment: "The following tests have NEVER passed on any version of CCExtractor — these are pre-existing issues and are NOT caused by this PR"
  • It does not count toward PR failure status — a PR only fails CI if it introduces new regressions (extra_failed_tests), never-worked tests are excluded from that decision entirely
  • The regression test list shows a "Never worked" badge next to the command so admins can see which tests are flagged at a glance
  • The test detail page shows the flag clearly with an explanation
  • On the test results page, individual failing rows show a "Never worked" warning label, and if every failing test in a category is never-worked, the category header turns amber instead of red — visually distinct from a real failure
image

Changes

Database:

  • New never_worked boolean column on the regression_test table, NOT NULL, defaults to false
  • Alembic migration included

Backend:

  • New TestResultStatus enum (passed / failed / never_worked) in mod_test/models.py — this is derived at query time from test output and the never_worked flag, it is not stored in the database
  • get_test_results() in mod_test/controllers.py now classifies each test into one of the three states and initializes status and error fields upfront for type safety. Also caches get_customized_regressiontests() into a set to avoid repeated calls inside the comprehension
  • get_info_for_pr_comment() in mod_ci/controllers.py buckets never-worked tests into their own list, separate from extra_failed and common_failed
  • comment_pr() now explicitly returns Status.SUCCESS or Status.FAILURE based solely on whether the PR introduced new regressions
  • PrCommentInfo dataclass updated with the new never_worked_tests field
  • RegressionTest model and constructor updated with the new field
  • Form updated with a BooleanField for the flag

Templates:

  • pr_comment.txt — new section for never-worked tests, rendered only when present
  • regression/index.html — "Never worked" badge on flagged tests in the list
  • regression/test_add.html and test_edit.html — checkbox for the flag with a short explanation
  • regression/test_view.html — status field showing Normal or Never Worked
  • test/by_id.html — per-row "Never worked" label in results table, amber category header when all failures in a category are never-worked

Tests:

  • Updated test_comments_successfuly_in_failed_pr_test to include status in the mock data and assert the return value is Status.FAILURE
  • New test test_get_info_for_pr_comment_buckets_tests_correctly covering all four cases: passed, new regression, pre-existing failure, and never-worked — verifying each ends up in the right list
@pulk17 pulk17 changed the title feat(ci): add never-worked status for test regression classification [FEATURE] Added a "Never Worked" state to the tests Mar 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant