Repository navigation
libregrtest: add option to run every test case in separate process #109817
Description
Activity
For now I can do this by using the following command:
PYTHON=./python; $PYTHON -m test -uall --list-tests 2>/dev/null | while read -r test do $PYTHON -m test -uall --list-cases $test 2>/dev/null | xargs -n1 $PYTHON -m test -vuall $test -m done
But it is slow (because it does not parallel different test modules), the output is difficult to analyze, and it cannot be run on Windows.
hii @serhiy-storchaka i'm planning to go with this, please let me know if you have any pointers for me. currently going through the mentioned PR.
Look at the implementation of
--list-testsand--list-casesoptions. What is common and what is different. Then look how running tests in multiple processes is implemented (option-j). Make it running tests one-case-per-process instead of one-module-per-process.This can be an easy task, but it can also be difficult. Only undertake this if you are confident in your abilities.
Reacted by Aniket@serhiy-storchaka Are we left with something in this or any improvement, please feel free to ping me . If this needs work
Running the whole test suite with
--single-process-per-casefound five pre-existing test bugs:- test_sysconfig leaves the sysconfig prefix cache modified #155726 -- test_sysconfig leaves the sysconfig prefix cache modified
- test_import leaves files in the current directory #155731 -- test_import leaves files in the current directory
- test_repr_lock leaves a dangling process #155732 -- test_repr_lock leaves a dangling process
- test_attribute_completion fails because of a lazy internal import #155727 -- test_attribute_completion fails because of a lazy internal import
- test_track_literals depends on a garbage collection #155728 -- test_track_literals depends on a garbage collection
The first three modify the environment, but this is not detected when the environment is checked once per test file, because other tests in the same file clean up after them.
It also made the lists of failed and skipped tests unreadable, because test case identifiers are longer than module names (GH-155714).
Reacted by AniketReacted by Victor StinnerReacted by AniketReacted by Victor Stinner
Feature or enhancement
Proposal:
libregrtest can run every test module in a separate process, and can even run several test modules in parallel. It has options to simply output all test modules or all test cases.
I propose to add an option to run every test case in a separate process. Test cases from the same module should still run sequentially, because they can use the same resource (for example create a server at fixed port).
This will slow down the tests several times, but it is not for regular running in the CI, it is for manual running once every months or for manual running selected test module while writing new tests or debugging test flakiness. It can help to solve the following issues:
cc @vstinner @gpshead
Has this already been discussed elsewhere?
No response given
Links to previous discussion of this feature:
No response
Linked PRs