gzip raising exception when closing with buffer backed by BytesIO

Bug report

Bug description:

Hello,

the following snippet raises an exception in Python 3.13.2 while it's fine with Python 3.12.

import io
import gzip

def foo():
    buffer = gzip.GzipFile(fileobj=io.BytesIO(), mode="w")

foo()

Running this with python3.12 is silent, with Python 3.13.2 instead:

Exception ignored in: <gzip on 0x7fa4fd99c550>
Traceback (most recent call last):
  File "/usr/lib/python3.13/gzip.py", line 359, in close
    fileobj.write(self.compress.flush())

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

Explicit buffer.close() and del buffer in the method make the error go away, so I think this has to do with ordering of cleanup of references when buffer is actually deallocated. Moving to with gzip.GzipFile ... as buffer: also doesn't error. The issue seems to be that the io.BytesIO is cleaned up before the gzip.GzipFile.

The same behavior occurs with an explicit reference to the io.BytesIO:

def explicit_reference():
    bio = io.BytesIO()
    buffer = gzip.GzipFile(fileobj=bio, mode="w")

Can work around it so no longer error at least by updating GzipFile.close() to check if self.fileobj.closed and fast-path exiting (See: https://github.com/python/cpython/compare/main...cmaloney:cpython:exp/gzip?expand=0). Not sure that is the right approach though vs. getting the objects to destruct in the right order.

Explicit buffer.close() and del buffer in the method make the error go away, so I think this has to do with ordering of cleanup of references when buffer is actually deallocated. Moving to with gzip.GzipFile ... as buffer: also doesn't error.

Thanks for testing these, I cannot use the context manager but I may be able to close the buffer explictly.

The issue seems to be that the io.BytesIO is cleaned up before the gzip.GzipFile.

Yeah, that is my impression too

The same behavior occurs with an explicit reference to the io.BytesIO:

def explicit_reference():
bio = io.BytesIO()
buffer = gzip.GzipFile(fileobj=bio, mode="w")

Can work around it so no longer error at least by updating GzipFile.close() to check if self.fileobj.closed and fast-path exiting (See: https://github.com/python/cpython/compare/main...cmaloney:cpython:exp/gzip?expand=0). Not sure that is the right approach though vs. getting the objects to destruct in the right order.

Yeah would be nice to keep the code working instead of not just crashing :) The gzip code should already have references to the fileobj so to me is surprising it gets deallocated while we are in in GzipFile.close.

So my original code when getting the gzipped content is this so am not sure I can workaround this other than stop using GzipFile.

    fileobj = buffer.fileobj  # get a reference to the fileobj before closing the gzip file
    buffer.close()

    data = fileobj.getbuffer()

Irony of the comment 😅

The documentation explicitly says that calling GzipFile's close() does not close the fileobj, so I guess it also means it should not be freed:

Calling a GzipFile object’s close() method does not close fileobj, since you might wish to append more material after the compressed data.
This also allows you to pass an io.BytesIO object opened for writing as fileobj, and retrieve the resulting memory buffer using the io.BytesIO object’s getvalue() method.

I still haven't found what is causing the fileobj to be closed... The line which is causing the exception print was added to 3.12 in https://github.com/python/cpython/pull/105920/files it looks like to fix #105808

@cmaloney

@cmaloney the one you linked touches flush, here is crashing in close. AFAICS the only commit that touches close (and coincidentally the gc 😅 ) that is in 3.13 but not in 3.12 is b52fc70#diff-ad9b54ac8ef847cbb11fb0550f7e8cede55b1d92de15899e0a885a94a124838a

I think it's the move to using a BufferedWriter (gh-89550) changed the behavior around flush and close in some unintended ways (https://github.com/python/cpython/blob/main/Modules/_io/bufferedio.c#L543-L598 being one of the parts). That the stream is being marked as "closed", but later gets flushed to. Still investigating / don't have any verifiable answers yet, just hunches.

Bisected this issue to #105104

Ok so with 3.12 and development mode I see the same warning, don't see the warning in 3.10 and 3.11 though:

$ python3.13 gzipfoo.py 
Exception ignored in: <gzip on 0x7f39c7098580>
Traceback (most recent call last):
  File "/usr/lib/python3.13/gzip.py", line 359, in close
    fileobj.write(self.compress.flush())
ValueError: I/O operation on closed file.
$ python3.12 gzipfoo.py 
$ python3.12 -X dev gzipfoo.py 
Exception ignored in: <gzip on 0x7f1450669590>
Traceback (most recent call last):
  File "/usr/lib/python3.12/gzip.py", line 357, in close
    fileobj.write(self.compress.flush())
ValueError: I/O operation on closed file.
$ /usr/bin/python3.10 -X dev gzipfoo.py 
$ /usr/bin/python3.11 -X dev gzipfoo.py

So there is something that changed the behavior in 3.12.

@danifus

@danifus thanks for your bisection. Any chance you can bisect with the -X dev param? I think there are good chances the code was failing already, just not printing the exception.

pythongh-129726: Break `gzip.GzipFile` reference loop (pythonGH-130055)

Fixed by change 7f39137. Backports to 3.12 and 3.13 will follow.

Thanks a lot!

Maybe GzipFile should emit a ResourceWarning if the file is not closed explicitly to make sure that data are written on disk.

[3.12] gh-129726: Break `gzip.GzipFile` reference loop (GH-130055) (#…

[3.13] gh-129726: Break `gzip.GzipFile` reference loop (GH-130055) (#…

I implemented a ResourceWarning on BufferedWriter checking for unwritten data, and just the CPython I/O tests it fires a lot. Even with conditions (only if raw isn't closed, etc) many cases to work through. I think worthwhile to help track down and cleanup inconsistencies at least in CPython itself. Not sure if it's going to be too noisy in general Python ecosystem codebases though.

The test suite can be fixed. The risk of losing data justify emitting ResourceWarning.

pythongh-130806: emit ResourceWarning if GzipFile unclosed

docs: Add the reference of `pythonGH-129726` in tests

gh-130806: Emit ResourceWarning if GzipFile unclosed (#130905)

pythongh-130806: Emit ResourceWarning if GzipFile unclosed (python#13…

@xrmx

Hey @xrmx I see that a new CPython version was released, looks like it does include the fix of the issue you reported: https://github.com/python/cpython/releases/tag/v3.13.3

Navigation Menu

Repository navigation

gzip raising exception when closing with buffer backed by BytesIO #129726

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

4 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

gzip raising exception when closing with buffer backed by BytesIO #129726

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

cmaloney commented on Feb 9, 2025

xrmx commented on Feb 9, 2025

xrmx commented on Feb 10, 2025

cmaloney commented on Feb 10, 2025

xrmx commented on Feb 10, 2025

cmaloney commented on Feb 10, 2025

danifus commented on Feb 11, 2025

xrmx commented on Feb 11, 2025

xrmx commented on Feb 12, 2025

4 remaining items

vstinner commented on Feb 28, 2025

xrmx commented on Feb 28, 2025

vstinner commented on Feb 28, 2025

cmaloney commented on Feb 28, 2025

vstinner commented on Mar 2, 2025

josecsotomorales commented on Apr 9, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions