Skip to content

Add test filtering to e2e tests#7707

Draft
eddyashton wants to merge 1 commit intomicrosoft:mainfrom
eddyashton:py_e2e_test_filter
Draft

Add test filtering to e2e tests#7707
eddyashton wants to merge 1 commit intomicrosoft:mainfrom
eddyashton:py_e2e_test_filter

Conversation

@eddyashton
Copy link
Member

Our current end-to-end Python test runner is hand-written, and invoked from ctest. This makes it hard to run single specific tests - they don't have standalone entry points, and we can't pass extra args to ctest for filtering.

The current solution is to comment out specific tests. That's manual and error-prone. I think we should make the runner richer, and able to apply this filtering.

There was a regex -R option for when you called python ../tests/... ... directly, but I don't think that's a common use case. This approach uses an envvar instead, which isn't ideal, but is a sensible way to plumb something past ctest. So it works whether you call python ... or ctest ....

Opening for discussion now. For full utility, we'd want to improve our @reqs.description coverage (this only filters the test functions that are correctly decorated!), but I think we should do that as a separate pass.

@achamayou
Copy link
Member

I think I would like to find a way to drop the tests.sh wrapper, by either using uv or some other auto-venv setup feature, and cut ctest out as well.

Now that we have one set of cross-platform binaries, I think the value of ctest is essentially zero or mildly negative. We may still want to create a test config with directories, but that's a one-off job that does not require a runner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants