Skip to content

[WASI] test cases with sys.path.insert(0, os.curdir) fail on WASI #93461

Description

@tiran

Bug report

Test cases that insert os.curdir ('.') into sys.path fail on WASI platforms. I think it is related how wasmtime maps directories into the runtime environment and how it sets up the current working directory. The test passes if I replace '.' with an empty string '' or remove the cached FileFinder instance for '.' with sys.path_importer_cache.pop('.').

 ./test-wasi.sh test_support -m test_forget
0:00:00 Run tests sequentially
0:00:00 [1/1] test_support
test test_support failed -- Traceback (most recent call last):
  File "/Lib/test/test_support.py", line 116, in test_forget
    mod = __import__(TESTFN)
          ^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '@test_66600_tmpæ'

test_support failed (1 error)

== Tests result: FAILURE ==

The culprit seems to be an outdated FileFinder instance: {'.': FileFinder('/.')}. The CWD is '/python-wasm/cpython/builddir/wasi/build/test_python_66600æ' when the test case is executed.

0:00:00 [1/1] test_support
> /Lib/test/test_support.py(114)test_forget()
-> sys.path.insert(0, os.curdir)
(Pdb) p sys.path
['.', '/usr/local/lib/python312.zip', '/Lib', '/build/lib.wasi-wasm32-3.12']
(Pdb) pp sys.path_importer_cache
{'.': FileFinder('/.'),
 '/Lib': FileFinder('/Lib'),
 '/Lib/collections': FileFinder('/Lib/collections'),
 '/Lib/encodings': FileFinder('/Lib/encodings'),
 '/Lib/importlib': FileFinder('/Lib/importlib'),
 '/Lib/re': FileFinder('/Lib/re'),
 '/Lib/test': FileFinder('/Lib/test'),
 '/Lib/test/libregrtest': FileFinder('/Lib/test/libregrtest'),
 '/Lib/test/support': FileFinder('/Lib/test/support'),
 '/Lib/unittest': FileFinder('/Lib/unittest'),
 '/Lib/urllib': FileFinder('/Lib/urllib'),
 '/build/lib.wasi-wasm32-3.12': FileFinder('/build/lib.wasi-wasm32-3.12'),
 '/usr/local/lib/python312.zip': None}
(Pdb) p os.getcwd()
'/python-wasm/cpython/builddir/wasi/build/test_python_66600æ'
(Pdb) n
> /Lib/test/test_support.py(115)test_forget()
-> importlib.invalidate_caches()
(Pdb) n
> /Lib/test/test_support.py(116)test_forget()
-> try:
(Pdb) p sys.path
['.', '.', '/usr/local/lib/python312.zip', '/Lib', '/build/lib.wasi-wasm32-3.12']
(Pdb) pp sys.path_importer_cache
{'.': FileFinder('/.'),
 '/Lib': FileFinder('/Lib'),
 '/Lib/collections': FileFinder('/Lib/collections'),
 '/Lib/encodings': FileFinder('/Lib/encodings'),
 '/Lib/importlib': FileFinder('/Lib/importlib'),
 '/Lib/re': FileFinder('/Lib/re'),
 '/Lib/test': FileFinder('/Lib/test'),
 '/Lib/test/libregrtest': FileFinder('/Lib/test/libregrtest'),
 '/Lib/test/support': FileFinder('/Lib/test/support'),
 '/Lib/unittest': FileFinder('/Lib/unittest'),
 '/Lib/urllib': FileFinder('/Lib/urllib'),
 '/build/lib.wasi-wasm32-3.12': FileFinder('/build/lib.wasi-wasm32-3.12'),
 '/usr/local/lib/python312.zip': None}

A build on Linux has an absolute path to build directory as first sys.path entry.

reproducer

$ podman run --rm -ti -v $(pwd):/python-wasm/cpython:Z quay.io/tiran/cpythonbuild:emsdk3
# ./build-python-build.sh
# ./build-python-wasi.sh
# ./test-wasi.sh test_support -m test_forget

Activity

  1. added
    type-bugAn unexpected behavior, bug, or error
    3.11only security fixes
    3.12only security fixes
    on Jun 3, 2022
  2. added a commit that references this issue on Jun 9, 2022
  3. tiran commented on Jun 9, 2022

    @tiran
    MemberAuthor

    GH-93461 is the simplest solution I could come up with. importlib.invalidate_caches now drops all FileFinder instances if the cache key is not an absolute path.

  4. brettcannon commented on Jun 9, 2022

    @brettcannon
    Member

    Seems like a reasonable solution to the problem.

  5. added a commit that references this issue on Jun 10, 2022
  6. added 2 commits that reference this issue on Jun 10, 2022
  7. brettcannon commented on Jun 17, 2022

    @brettcannon
    Member

    @tiran can this be closed?

  8. tiran commented on Jun 17, 2022

    @tiran
    MemberAuthor

    Yes, it is fixed now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

3.11only security fixes3.12only security fixestopic-importlibtype-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions