Isolate the _io extension module

Isolate the _io extension module by moving all global variables to module state, porting static types to heap types, and implementing multi-phase init. All global variables in the _io module are static types:

Modules/_io/bufferedio.c: PyBufferedIOBase_Type
Modules/_io/bufferedio.c: PyBufferedRWPair_Type
Modules/_io/bufferedio.c: PyBufferedRandom_Type
Modules/_io/bufferedio.c: PyBufferedReader_Type
Modules/_io/bufferedio.c: PyBufferedWriter_Type
Modules/_io/bytesio.c: PyBytesIO_Type
Modules/_io/bytesio.c: _PyBytesIOBuffer_Type
Modules/_io/fileio.c: PyFileIO_Type
Modules/_io/iobase.c: PyIOBase_Type
Modules/_io/iobase.c: PyRawIOBase_Type
Modules/_io/textio.c: PyIncrementalNewlineDecoder_Type
Modules/_io/textio.c: PyTextIOBase_Type
Modules/_io/textio.c: PyTextIOWrapper_Type
Modules/_io/winconsoleio.c: PyWindowsConsoleIO_Type

Converting the static types to heap types involves applying PEP-687 to _io.
Adapting multi-phase init involves applying PEP-489 to _io.

Linked PRs

@kumaraditya303

cc. @kumaraditya303

@zooba

I see that _PyWindowsConsoleIO_Type is marked with PyAPI_DATA, but I can't find it in Misc/stable_abi.toml. @zooba or @eryksun, do you know why this is so? I did a quick code search on GitHub, but I only got hits in various clones of the CPython code base.

cpython/Modules/_io/_iomodule.h

Lines 24 to 30 in 2db2c4b

    
           #ifndef Py_LIMITED_API 
        
           #ifdef MS_WINDOWS 
        
           extern PyTypeObject PyWindowsConsoleIO_Type; 
        
           PyAPI_DATA(PyObject *) _PyWindowsConsoleIO_Type; 
        
           #define PyWindowsConsoleIO_Check(op) (PyObject_TypeCheck((op), (PyTypeObject*)_PyWindowsConsoleIO_Type)) 
        
           #endif /* MS_WINDOWS */ 
        
           #endif /* Py_LIMITED_API */

UPDATE: I see it was added by Steve in 312cef7, probably because of the PyWindowsConsoleIO_Check checks in PC/_testconsole.c and Python/pylifecycle.c:

cpython/PC/_testconsole.c

Lines 54 to 57 in 2db2c4b

    
           if (!PyWindowsConsoleIO_Check(file)) { 
        
               PyErr_SetString(PyExc_TypeError, "expected raw console object"); 
        
               return NULL; 
        
           }

cpython/Python/pylifecycle.c

Lines 58 to 61 in 2db2c4b

    
              extern PyTypeObject PyWindowsConsoleIO_Type; 
        
           #  define PyWindowsConsoleIO_Check(op) \ 
        
                  (PyObject_TypeCheck((op), &PyWindowsConsoleIO_Type)) 
        
           #endif

cpython/Python/pylifecycle.c

Lines 2359 to 2363 in 2db2c4b

    
           #ifdef MS_WINDOWS 
        
               /* Windows console IO is always UTF-8 encoded */ 
        
               if (PyWindowsConsoleIO_Check(raw)) 
        
                   encoding = L"utf-8"; 
        
           #endif

Perhaps exposing _PyWindowsConsoleIO_Type through a CPython internal capsulated C API could be an option?

Other issues that must be resolved before continuing: _PyBytesIOBuffer_Type is used by _testcapimodule. Since that type is not exposed in _io, we cannot simply use PyImport_Import and look it up. One possibility is to expose it through a CPython internal capsule C API.

... or we could simply expose the types mentioned in #101819 (comment) and #101819 (comment) in the _io module.

Internal types are allowed to be exported without being stable API. On Windows, only explicitly exported names are available outside the DLL - it's different on other platforms, so please don't bring their assumptions over 😉

The type is explicitly not part of the limited API/stable ABI, see #ifndef Py_LIMITED_API.

Internal types are allowed to be exported without being stable API. On Windows, only explicitly exported names are available outside the DLL - it's different on other platforms, so please don't bring their assumptions over 😉

IIUC, we can remove this from the DLL and instead export it via _io and look it up from the module using the C API (PyImport_Import, etc.).

Yeah, if it's got a leading underscore then we can do whatever we need to make our tests work.

gh-101819: Fix inverted debug preprocessor check in winconsoleio.c (#…

Merge branch 'main' into supermon

gh-101819: Prepare _io._IOBase for module state (#104386)

pythongh-101819: Fix _io clinic input for unused base class method stubs

Merge branch 'main' into supermon

gh-101819: Isolate `_io` (#101948)

gh-101819: Fix _io clinic input for unused base class method stubs (#…

Merge branch 'main' into supermon

Merge branch 'main' into cached_property_docs

@erlend-aasland

@erlend-aasland erlend-aasland closed this as completed

Great achievement!

It causes a crash during garbage collection. See #111049, #111211.

pythongh-101819: Adapt _io types to heap types, batch 1 (pythonGH-101949

Navigation Menu

Repository navigation

Isolate the `_io` extension module #101819

Linked PRs

50 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	#ifndef Py_LIMITED_API
	#ifdef MS_WINDOWS
	extern PyTypeObject PyWindowsConsoleIO_Type;
	PyAPI_DATA(PyObject *) _PyWindowsConsoleIO_Type;
	#define PyWindowsConsoleIO_Check(op) (PyObject_TypeCheck((op), (PyTypeObject*)_PyWindowsConsoleIO_Type))
	#endif /* MS_WINDOWS */
	#endif /* Py_LIMITED_API */

	if (!PyWindowsConsoleIO_Check(file)) {
	PyErr_SetString(PyExc_TypeError, "expected raw console object");
	return NULL;
	}

	extern PyTypeObject PyWindowsConsoleIO_Type;
	# define PyWindowsConsoleIO_Check(op) \
	(PyObject_TypeCheck((op), &PyWindowsConsoleIO_Type))
	#endif

	#ifdef MS_WINDOWS
	/* Windows console IO is always UTF-8 encoded */
	if (PyWindowsConsoleIO_Check(raw))
	encoding = L"utf-8";
	#endif

Isolate the _io extension module #101819

Description

Linked PRs

Activity

erlend-aasland commented on Feb 11, 2023

erlend-aasland commented on Feb 13, 2023

erlend-aasland commented on Feb 13, 2023

erlend-aasland commented on Feb 13, 2023

zooba commented on Feb 13, 2023

encukou commented on Feb 13, 2023

erlend-aasland commented on Feb 13, 2023

zooba commented on Feb 13, 2023

50 remaining items

vstinner commented on May 15, 2023

serhiy-storchaka commented on Oct 23, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Isolate the `_io` extension module #101819