Skip to content

unittest: execute tests in parallel #82054

Closed
Closed
unittest: execute tests in parallel#82054
Labels
stdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

Description

@donpellegrino
donpellegrino
mannequin
opened on Aug 16, 2019
BPO 37873
Nosy @terryjreedy, @giampaolo, @ezio-melotti, @voidspace, @bharel, @tirkarthi, @donpellegrino

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-08-16.14:15:27.971>
labels = ['type-feature', 'library', '3.9']
title = 'unittest: execute tests in parallel'
updated_at = <Date 2020-09-02.09:57:36.866>
user = 'https://github.com/donpellegrino'

bugs.python.org fields:

activity = <Date 2020-09-02.09:57:36.866>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-08-16.14:15:27.971>
creator = 'user93448'
dependencies = []
files = []
hgrepos = []
issue_num = 37873
keywords = []
message_count = 5.0
messages = ['349864', '349938', '350080', '375126', '375128']
nosy_count = 7.0
nosy_names = ['terry.reedy', 'giampaolo.rodola', 'ezio.melotti', 'michael.foord', 'bar.harel', 'xtreak', 'user93448']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue37873'
versions = ['Python 3.9']

Linked PRs

Activity

  1. donpellegrino commented on Aug 16, 2019

    @donpellegrino
    donpellegrinomannequin
    on Aug 16, 2019
    MannequinAuthor

    The unittest documentation makes reference to a potential parallelization feature:

    "Note that shared fixtures do not play well with [potential] features like test parallelization and they break test isolation. They should be used with care." (https://docs.python.org/3/library/unittest.html)

    However, it seems that executing tests in parallel is not yet a feature of unittest. This enhancement request is to add parallel execution of tests to unittest.

    A command line option may be a good interface. Ideally, it would be compatible with test discovery. Outside of the Python ecosystem, a common practice is to define test cases in a Makefile and then execute GNU Make with the '-j' flag (https://www.gnu.org/software/make/manual/html_node/Parallel.html#Parallel). Adding such an option to unittest would be a convenience and may save the effort of bringing in additional libraries or tools for parallel unit test execution.

  2. donpellegrino
    added
    3.9 (EOL)end of life
    stdlibStandard Library Python modules in the Lib/ directory
    type-featureA feature request or enhancement
    on Aug 16, 2019

  3. terryjreedy commented on Aug 19, 2019

    @terryjreedy
    terryjreedy
    on Aug 19, 2019
    Member

    test.regrtest has a -j option. Perhaps some of the Python coding for that could be used for unitest also.

  4. tirkarthi commented on Aug 21, 2019

    @tirkarthi
    tirkarthi
    on Aug 21, 2019
    Member

    See also https://mail.python.org/pipermail/python-ideas/2017-September/047100.html . One of the ideas in the thread was to move test.regrtest parallel execution functionality into unittest. I think this would be good to have it in unittest like support in pytest for -j.

  5. donpellegrino commented on Aug 10, 2020

    @donpellegrino
    donpellegrinomannequin
    on Aug 10, 2020
    MannequinAuthor

    Leveraging GNU Parallel (https://www.gnu.org/software/parallel/) might help simplify implementation. Perhaps that could be used as a subprocess call?

  6. vstinner commented on Aug 10, 2020

    @vstinner
    vstinner
    on Aug 10, 2020
    Member

    Leveraging GNU Parallel (https://www.gnu.org/software/parallel/) might help simplify implementation. Perhaps that could be used as a subprocess call?

    In general, we attempt to avoid depending on the availability of external tool. For example, I don't expect this tool to be available on Windows, whereas it would be better to support parallel execution on Windows as well.

  7. ezio-melotti
    transferred this issue fromon Apr 10, 2022
  8. serhiy-storchaka
    added this to Unittest & doctest issueson May 20, 2022

  9. gpshead commented on Nov 19, 2022

    @gpshead
    gpshead
    on Nov 19, 2022
    Member

    An example of implementing sharding within a single unittest so they can run in parallel across whatever number of processes and machines you want is what we use at work: https://github.com/abseil/abseil-py/blob/v1.3.0/absl/testing/absltest.py#L2359 using the Bazel sharding protocol.

    If test.regrtest supported using such a sharding protocol and unittest adopted that TestCase.getTestCaseNames implementation logic to compute the shards for each process, we'd really pull in our long tail test time in CPython's own unittest suite.

    One of the ideas in the thread was to move test.regrtest parallel execution functionality into unittest. I think this would be good to have it in unittest like support in pytest for -j.

    Yes. Though wider Python community wise you'll just find many people saying "just use pytest-xdist" in your project for -j support. I can't disagree with that.

    I still want parallelization for CPython's own unittest suite (launched via python -m test) regardless of that. It'd help save core dev & contributor time as well as increasing buildbot and CI thruput.

  10. gpshead
    added
    3.12only security fixes
    and removed
    3.9 (EOL)end of life
    on Nov 19, 2022
  11. gpshead
    moved this to Todo in Unittest & doctest issueson Nov 19, 2022
  12. zitterbewegung
    added 6 commits that reference this issue on Apr 25, 2023

    pythongh-82054: Implementing unittest sharding for asyncio

    70108e3

    Merge branch 'main' of github.com:zitterbewegung/cpython into pythong…

    a8305f4

    pythongh-82054: Executing tests in parallel by sharding.

    6901a03

    pythongh-82054 Removing news entry since end users won't be impacted.

    f4b476b

    Merge branch 'python:main' into pythongh-82054

    f5e24bf

    Merge branch 'python:main' into pythongh-82054

    b366f9f
  13. gpshead
    added a commit that references this issue on Apr 30, 2023

    gh-82054: allow test runner to split test_asyncio to execute in paral…

    9e011e7
  14. gpshead
    removed
    3.12only security fixes
    on Apr 30, 2023

  15. gpshead commented on Apr 30, 2023

    @gpshead
    gpshead
    on Apr 30, 2023
    Member

    I'm leaving this open as there's plenty more opportunity to do things here (up to and including potentially making my full parallelism PR work reliably). The just merged PR at least splits our longest test, test_asyncio, up into parallel sub-tests given it was already defined as a package of tests in separate files.

    Refactoring test_concurrent_futures.py and the entire _test_multiprocessing.py suite and its trio of start-method runners into similar package of sub-test.py files would allow it to use the already committed pieces upon make test or python -m test when no list of tests is specified to further pull in the long tail on default run-everything CI, buildbot, and developer runs.

  16. carljm
    added a commit that references this issue on May 1, 2023

    Merge branch 'main' into supermon

    159b2aa
  17. vstinner
    added 2 commits that reference this issue on Sep 2, 2023

    pythongh-82054: allow test runner to split test_asyncio to execute in…

    ec875d5

    [3.11] gh-108822: Backport libregrtest changes from the main branch (#…

    79f7a4c

  18. gpshead commented on Sep 23, 2023

    @gpshead
    gpshead
    on Sep 23, 2023
    Member

    With the refactorings @vstinner has done recently, we're close enough that I'm going to just close this issue. See my now closed draft PR #99637 for the full parallelism implementation I started with and reasoning for why it likely isn't worth us chasing at this point.

    Python users who want that parallelism in their own test suites can use absltest from https://pypi.org/project/absl-py/ or other frameworks (pytest-xdist?) that offer it.

  19. gpshead
    closed this as completedon Sep 23, 2023
  20. github-project-automation
    moved this from Todo to Done in Unittest & doctest issueson Sep 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions