Support moving across filesystems in pathlib.Path, as shutil.move() does #73991
Description
| BPO | 29805 |
|---|---|
| Nosy | @brettcannon, @pfmoore, @ericvsmith, @tjguk, @zware, @eryksun, @zooba |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2017-03-13.21:03:44.694>
labels = ['3.8', 'type-feature', 'library', '3.9', '3.10']
title = 'Support moving across filesystems in pathlib.Path, as shutil.move() does'
updated_at = <Date 2021-03-15.21:26:30.922>
user = 'https://bugs.python.org/LaurentMazuel'bugs.python.org fields:
activity = <Date 2021-03-15.21:26:30.922>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-03-13.21:03:44.694>
creator = 'Laurent.Mazuel'
dependencies = []
files = []
hgrepos = []
issue_num = 29805
keywords = []
message_count = 6.0
messages = ['289549', '289552', '289559', '289630', '289687', '289688']
nosy_count = 8.0
nosy_names = ['brett.cannon', 'paul.moore', 'eric.smith', 'tim.golden', 'Laurent.Mazuel', 'zach.ware', 'eryksun', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue29805'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']Linked PRs
- GH-73991: Add pathlib.Path.move that can handle rename across FS #30650
- GH-73991: Add
pathlib.Path.copy()#119058 - GH-73991: Add
pathlib.Path.rmtree()#119060 - [WIP] GH-73991: Add
os.copy()and friends #119079 - GH-73991: Split "Directory and file operations" section in shutil docs #119159
- GH-73991: Use same signature for
shutil._rmtree_[un]safe(). #120517 - GH-73991: Add follow_symlinks argument to
pathlib.Path.copy()#120519 - GH-73991: pathlib ABC tests: add
DummyPath.unlink()andrmdir()#120715 - GH-73991: Add
pathlib.Path.copytree()#120718 - GH-73991: Support preserving metadata in
pathlib.Path.copy()#120806 - GH-73991: Support copying directory symlinks on older Windows #120807
- GH-73991: Support preserving metadata in
pathlib.Path.copytree()#121438 - GH-73991: Fix "Operation not supported" on Fedora buildbot. #121444
- GH-73991: Add
pathlib.Path.move()#122073 - GH-73991: Rework
pathlib.Path.rmtree()intodelete()#122368 - GH-73991: Rework
pathlib.Path.copytree()intocopy()#122369 - GH-73991: Disallow copying directory into itself via
pathlib.Path.copy()#122924 - GH-73991: Prune
pathlib.Path.delete()arguments #123158 - GH-73991: Add
pathlib.Path.copy_into()andmove_into()#123314 - GH-73991: Make
pathlib.Path.delete()private. #123315 - GH-73991: Prune
pathlib.Path.copy()andcopy_into()arguments #123337
Activity
LaurentMazuel commentedon Mar 13, 2017
Trying to use Pathlib and Path.replace on Windows if drive are different leads to an issue:
This is a known situation of os.rename, and workaround I found is to use shutil or to copy/delete manually in two steps (e.g. http://stackoverflow.com/questions/21116510/python-oserror-winerror-17-the-system-cannot-move-the-file-to-a-different-d)
When using Pathlib, it's not that easy to workaround using shutil (even if thanks to Brett Cannon now shutil accepts Path in Py3.6, not everybody has Py3.6). At least this should be documented with a recommendation for that situation. I love Pathlib and it's too bad my code becomes complicated when it was so simple :(
LaurentMazuel commentedon Mar 13, 2017
Just to confirm, I was able to workaround it with Py3.6:
It is reasonable to think about adding a similar feature to pathlib if it doesn't support it and just special-case the drive-to-drive scenario for Windows
eryksun commentedon Mar 14, 2017
Moving a file across volumes isn't atomic. Getting an exception in this case can be useful. There could be a "strict" keyword-only parameter that defaults to False. If it's true, then replace() won't try to move the file.
ericvsmith commentedon Mar 15, 2017
I agree this needs to be different from replace(), due to not being atomic. That makes it an enhancement, so I'm removing 3.5.
I'm +1 on Path supporting something like shutil.move().
brettcannon commentedon Mar 15, 2017
I also support the idea of getting something like shutil.move() into pathlib.
brettcannon commentedon Mar 15, 2017
I should also mention that rename() (https://docs.python.org/3/library/pathlib.html#pathlib.Path.rename) and replace() (https://docs.python.org/3/library/pathlib.html#pathlib.Path.replace) already do exist, so it might be best to add a keyword-only flag to one of those for this use-case.
145 remaining items
GH-73991: Add `pathlib.Path.copy_into()` and `move_into()` (#123314)
Merge branch 'main' into pythongh-73991-private-delete
pythonGH-73991: Prune `pathlib.Path.copy()` and `copy_into()` arguments
GH-73991: Make `pathlib.Path.delete()` private. (#123315)
GH-73991: Prune `pathlib.Path.copy()` and `copy_into()` arguments (#1…
barneygale commentedon Aug 26, 2024
It's done! Thank you everyone who helped formulate and review this feature.
We've added new
copy(),copy_into(),move()andmove_into()methods, which work on both files and directories. The interfaces are refined to pathlib standards, with certain nonessential functionality (like support for filtering incopy()) omitted for now. Users may wish to propose new capabilities on the ideas forum.The new methods expect exact targets (or target directories for the
_into()variants), whereasshutilchecks for existing directories and adjusts the target as necessary. The latter behaviour is helpful in an interactive shell with tab-complete, but less so in a stored program where state of the filesystem might not be known (for example,shutil.copy('a', 'b')could create eitherborb/a).The
copy()andcopy_into()methods usefcntl.FICLONEoros.copy_file_rangewhere available (see GH-81338, GH-81340). This work has stalled for years inshutilfor backwards-compatibility concerns, happily not present here.Feedback most welcome! We have ~9 months until 3.14 beta 1 where the interface is more locked down, so there's still plenty of time make changes. I'll close this issue, but I can re-open it (or log new issues) if problems are identified. Thanks again, all.
encukou commentedon Dec 2, 2024
For copying trees, what happens on error -- for example, when one of the files has a name that's incompatible with the target filesystem?
barneygale commentedon Dec 2, 2024
In that case you'll get an
OSError. Any copies already made will be left on disk, and in the case ofmove(), none of the original files will be deleted.agronholm commentedon Dec 6, 2024
It looks like, unlike
copy(), this method fails if I pass a path-like object that just happens to have a method namedstat(). It then fails atself._ensure_different_file(target)which already assumes aPathas its argument type. It really should do an isinstance check on the target. In my case,stat()is a coroutine method, so it fails withAttributeError: 'coroutine' object has no attribute 'st_ino'.zooba commentedon Dec 9, 2024
Passing an object that looks similar but isn't the same is the caller's error. An
AttributeErroris exactly what you should get in this case.That said, it probably wouldn't be unreasonable to convert any incoming paths to the same type as the source. We shouldn't type check here, we should try to convert and let the errors bubble out to the caller.
agronholm commentedon Dec 9, 2024
Such type checks are done everywhere else, including in the method in question! This would be solved simply by doing said type check earlier.
Is it reasonable to expect this to fail on a
PathLikeobject? How am I supposed to know what I can pass as the target then?zooba commentedon Dec 9, 2024
In general for pathlib, you should be able to pass any object that can be interpreted as the same kind of path as the one you're passing it to. I know that upsets the static typing people because it's hard to define, but being accepting of inputs is a key Python feature.
In this case, that means converting the input to the same type as the target before using it. Most of the functions that have type checks are essentially doing this - they're negative checks, followed by essentially
type(self)(other), which will try its hardest to parse the path out of the input. Butsamefileis not doing this and clearly ought to be. I suggest opening a new issue.