Repository navigation
Memory leak on executables embedded with 3.13 #113055
Description
Bug report
Bug description:
Since 67807cf, _testembed.exe (x64 Release) increases the memory usage in the cycle of Py_Initialize and Py_Finalize.
I monitored the Task Manager (taskmgr.exe) on Windows10/11, with a change against Programs/_testembed.c:
- #define INIT_LOOPS 4
+ #define INIT_LOOPS 200Commands:
_testembed.exe test_repeated_simple_initorpython -m test test_embed -m test_simple_initialization_api(no log)
With RADIX TREE (as-is)
| Loop | 67807cf | main | 3.11.7 |
|---|---|---|---|
| 100 | 67MB | 210MB | 4MB |
| 200 | 135MB | 406MB | 4MB |
No RADIX TREE (as-is)
| Loop | 67807cf | main | 3.11.7 |
|---|---|---|---|
| 100 | 10MB | 168MB | 4MB |
| 200 | 16MB | 336MB | 4MB |
Recent obmalloc.c looks ready for finalization. Just adding my rough (invalid?) experiment made the leaks even. So, I hope that they share the same issue. Otherwise, ea2c001 or 15d4c9f has another leak. #98359 (comment)
patch
void
_PyInterpreterState_FinalizeAllocatedBlocks(PyInterpreterState *interp)
{
if (has_own_state(interp)) {
Py_ssize_t leaked = _PyInterpreterState_GetAllocatedBlocks(interp);
assert(has_own_state(interp) || leaked == 0);
interp->runtime->obmalloc.interpreter_leaks += leaked;
+ OMState *state = &interp->obmalloc;
+ for (uint i = 0; i < maxarenas; ++i) {
+ if (allarenas[i].address == 0) {
+ continue;
+ }
+ _PyObject_Arena.free(_PyObject_Arena.ctx,
+ (void *)allarenas[i].address, ARENA_SIZE);
+ allarenas[i].address = 0;
+ --narenas_currently_allocated;
+ }
+ PyMem_RawFree(allarenas);
}
}With RADIX TREE (patched)
| Loop | 67807cf | main |
|---|---|---|
| 100 | 42MB | 40MB |
| 200 | 80MB | 80MB |
No RADIX TREE (patched)
| Loop | 67807cf | main |
|---|---|---|
| 100 | 3.5M | 3.3M |
| 200 | 3.7M | 3.4M |
cc @ericsnowcurrently @nascheme
CPython versions tested on:
3.12, 3.13, CPython main branch: 3aea6c4
Operating systems tested on:
Windows
Activity
nascheme commentedon Dec 13, 2023
That doesn't look so nice, I can investigate. How are you measuring memory usage and what's the OS? In the previous versions of Python the obmalloc radix tree data structures were in the BSS and so most of RAM used by them was actually virtual and not real. I'm not sure if the move to interpreter state structures lost that behaviour but it could be cause for an increase in (real) memory use. Maybe there is also a memory leak there, e.g. malloc without free.
neonene commentedon Dec 13, 2023
Only the Task Manager on Windows 10 and 11 for me.
The microsoft.com offers VMMap as a small alternative.
[-]Memory leak on embedded Windows executable with 3.12 and later[/-][+]Memory leak on Windows executable embedded with 3.12 and later[/+]nascheme commentedon Dec 16, 2023
@ericsnowcurrently
I spent some time looking into this today. I suspect the fix is not going to be simple, unfortunately. What was previously global variables in the obmalloc.c module has become part of the PyInterpreterState structure. When an interpreter is freed, memory allocated by obmalloc doesn't get freed. I think that's the leak you are seeing. Previous to
67807cfcit didn't get freed either but it was global across interpreters, got allocated only once and didn't grow.A possible way to fix this: when
free_interpreter()gets called, walk through obmalloc arenas and free them. Thepymalloc_print_stats()function is an example of how to walk over the arenas. The obmalloc radix tree nodes would also have to be walked and deallocated.If you free the arenas and there is allocated blocks inside of them, you can't call
pymalloc_free()on those memory blocks. It is maybe not a problem since the interpreter is being destroyed. If an object is passed from one interpreter to another, that would definitely break. If you want to share objects across interpreters, they would need to be allocated from a memory allocator that's global across interpreters.nascheme commentedon Dec 16, 2023
Building gh-113218 with the address sanitizer and then running
./Programs _testembed test_repeated_simple_initshows it doesn't leak anymore. I'm not sure my fix is totally correct but I think it's something close to what we need to fix this.[-]Memory leak on Windows executable embedded with 3.12 and later[/-][+]Memory leak on executables embedded with 3.12 and later[/+]neonene commentedon Dec 17, 2023
I can also personally confirm that on Windows, including the 32-bit version of Python.
nascheme commentedon Dec 22, 2023
I think I've come up with a better fix. See gh-113412.
neonene commentedon Dec 23, 2023
ea2c001 has a memory leak to be fixed, which is found at gh-113412.
I've seen a few reports about likely suspects: #110411 (comment), #113190 (comment)
nascheme commentedon Dec 23, 2023
Your
test_repeated_simple_initcase leaks on Linux as well. With gh-113412 the leak is slower but it leaks without bound, it seems. Using valgrind, I suspect it (at least partially) due to immortal interned strings. I generated the trace with:PYTHONMALLOC=malloc valgrind --leak-check=full --show-leak-kinds=all --log-file=valgrind.log --num-callers=20 ./Programs/_testembed test_repeated_simple_initTrace log:
https://gist.github.com/nascheme/fac2dd566f09fe3d75c98134f18c6170
E.g.
And
nascheme commentedon Dec 23, 2023
The following change seems to greatly reduce the leaks. The
_Py_hashtable_set(INTERNED_STRINGS, s, s)call looks like a problem to me.neonene commentedon Dec 24, 2023
My memory leak seems to come from
_PyUnicode_ClearInterned()when thePy_DEBUGflag is not specified. I'm not sure of the root cause, though.cpython/Objects/unicodeobject.c
Lines 14952 to 14973 in 0c57454
neonene commentedon Dec 29, 2023
For the record, my memory gain reduced from 2M bytes to 200K bytes per cycle with gh-113412 (non-debug).
Regarding the crash after re-initialization due to the full cleanup by
_PyUnicode_ClearInterned():(leaks)
A dangling pointer in a non-isolable (small) extension could be worked around by making the interned string statically allocated, if any.
Original issue: #100911
nascheme commentedon Dec 29, 2023
My previous comment about disabling the
INTERNED_STRINGSglobal hash table was incorrect. I must have built with Py_DEBUG enabled and so I erroneously thought it fixed the leak. You seem to be correct about_PyUnicode_ClearInterned(). There is this comment:If Py_DEBUG is enabled, then that function will free interned strings with the flag
SSTATE_INTERNED_IMMORTAL. That comment was from Eddie's immortal refcnt change (gh-19474, ea2c001).3 remaining items
gh-113055: Use pointer for interp->obmalloc state (gh-113412)
pythongh-113055: Use pointer for interp->obmalloc state (pythongh-113412
pythongh-113055: Use pointer for interp->obmalloc state (pythongh-113412
[-]Memory leak on executables embedded with 3.12 and later[/-][+]Memory leak on executables embedded with 3.13[/+]neonene commentedon Oct 15, 2024
I think this can be closed after 3203a74 is backported to 3.13. #113190 is opened for 3.12.
erlend-aasland commentedon Oct 15, 2024
What about #113218 then, @nascheme?
nascheme commentedon Oct 15, 2024
GH-118618 would fix the most serious memory leak in 3.12. For the last bugfix release of 3.12, @Yhg1s decided it was too large/disruptive to merge a backport and so that PR is closed. Assuming we are not changing our minds on merging that, I think the memory leak due to the obmalloc state will remain unfixed in 3.12. In theory there might be a less invasive and simpler fix for 3.12, I'd have to research that though.
It's getting confusing to keep track of these embedding bugs and what fixes have been backported. There are two classes of them: memory leaks and crashers. The crashers are the more serious ones. I believe we have fixes the known crashers (see GH-124865, GH-124536, GH-125314). For the memory leaks, the obmalloc state one is the worst. The leaks due to interned strings also depend on the mortal string change (49f6beb). Running the 3.12 head with ASAN enabled would show what's still leaking.
In the long term, i think it would be nice to have CPython running with no ASAN warnings. Then we have a build-bot to ensure we don't regress on that.
Yhg1s commentedon Oct 21, 2024
FYI, 3.12.7 was not the last bugfix release of 3.12. We have scheduled bugfix releases of 3.12 until April 2025.
vstinner commentedon Nov 27, 2024
I cannot reproduce this issue anymore on the main branch:
Output:
Since 3.12 branch is not going to be fixed, can we close the issue?