If PyDict_GetItemWithError is only used to check whether the key is in dict,
it is better to use PyDict_Contains instead.
And if it is used in combination with PyDict_SetItem, PyDict_SetDefault can
replace the combination.
This API is relatively lightweight and organizationally, given that it's
used by multiple modules, it makes sense to move it to fileutils.
Requires making sure that _posixsubprocess is compiled with the appropriate
Py_BUIILD_CORE_BUILTIN macro.
This suppression is no longer needed in os_closerange_impl, as it just
invokes the internal _Py_closerange implementation. On the other hand,
consumers of _Py_closerange may not have any other reason to suppress
invalid parameter issues, so narrow the scope to here.
close_range(2) should be preferred at all times if it's available, otherwise we'll use closefrom(2) if available with a fallback to fdwalk(3) or plain old loop over fd range in order of most efficient to least.
[note that this version does check for ENOSYS, but currently ignores all other errors]
Automerge-Triggered-By: @pablogsal
Such an API can be used both for os.closerange and subprocess. For the latter, this yields potential improvement for platforms that have fdwalk but wouldn't have used it there. This will prove even more beneficial later for platforms that have close_range(2), as the new API will prefer that over all else if it's available.
The new API is structured to look more like close_range(2), closing from [start, end] rather than the [low, high) of os.closerange().
Automerge-Triggered-By: @gpshead
I just realized that my recent PR with sendfile on Solaris ([PR 22040](https://github.com/python/cpython/pull/22040)) has broken error handling.
Sorry for that, this simple followup fixes that.
Automerge-Triggered-By: @1st1
There are a bunch of other fd: int uses in this file, I expect many if not
all of them would be better off using the fildes converter. This particular
one was flagged by Coverity as it presumably flags fpathconf as not accepting
negative fds. I'd expect the other fd's to have been flagged as well
otherwise.
I'm marking this one as skip news as it really is a no-op.
PyOS_AfterFork_Child() helper functions now return a PyStatus:
PyOS_AfterFork_Child() is now responsible to handle errors.
* Move _PySignal_AfterFork() to the internal C API
* Add #ifdef HAVE_FORK on _PyGILState_Reinit(), _PySignal_AfterFork()
and _PyInterpreterState_DeleteExceptMain().
Previously, the result could have been an instance of a subclass of int.
Also revert bpo-26202 and make attributes start, stop and step of the range
object having exact type int.
Add private function _PyNumber_Index() which preserves the old behavior
of PyNumber_Index() for performance to use it in the conversion functions
like PyLong_AsLong().
Pass PEP 573 defining_class to os.DirEntry methods. The module state
is now retrieve from defining_class rather than Py_TYPE(self), to
support subclasses (even if DirEntry doesn't support subclasses yet).
* Pass the module rather than defining_class to DirEntry_fetch_stat().
* Only get the module state once in _posix_clear(),
_posix_traverse() and _posixmodule_exec().
Convert posixmodule.c ("posix" or "nt" module) to the multiphase
initialization (PEP 489).
* Create the module using PyModuleDef_Init().
* Create ScandirIteratorType and DirEntryType with the new
PyType_FromModuleAndSpec() (PEP 573)
* Get the module state from ScandirIteratorType and DirEntryType with
the new PyType_GetModule() (PEP 573)
* Pass module to functions which access the module state.
* convert_sched_param() gets a new module parameter. It is now called
directly since Argument Clinic doesn't support passing the module
to an argument converter callback.
* Remove _posixstate_global macro.
An isolated subinterpreter cannot spawn threads, spawn a child
process or call os.fork().
* Add private _Py_NewInterpreter(isolated_subinterpreter) function.
* Add isolated=True keyword-only parameter to
_xxsubinterpreters.create().
* Allow again os.fork() in "non-isolated" subinterpreters.
On FreeBSD, os.closerange(fd_low, fd_high) now calls
closefrom(fd_low) if fd_high is greater than or equal to
sysconf(_SC_OPEN_MAX).
Initial patch by Ed Maste (emaste), Conrad Meyer (cem), Kyle Evans
(kevans) and Kubilay Kocak (koobs):
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274
Fix the Windows implementation of os.waitpid() for exit code
larger than "INT_MAX >> 8". The exit status is now interpreted as an
unsigned number.
os.waitstatus_to_exitcode() now accepts wait status larger than
INT_MAX.
Rename _PyInterpreterState_GET_UNSAFE() to _PyInterpreterState_GET()
for consistency with _PyThreadState_GET() and to have a shorter name
(help to fit into 80 columns).
Add also "assert(tstate != NULL);" to the function.
Add a private _at_fork_reinit() method to _thread.Lock,
_thread.RLock, threading.RLock and threading.Condition classes:
reinitialize the lock after fork in the child process; reset the lock
to the unlocked state.
Rename also the private _reset_internal_locks() method of
threading.Event to _at_fork_reinit().
* Add _PyThread_at_fork_reinit() private function. It is excluded
from the limited C API.
* threading.Thread._reset_internal_locks() now calls
_at_fork_reinit() on self._tstate_lock rather than creating a new
Python lock object.
Add os.waitstatus_to_exitcode() function to convert a wait status to an
exitcode.
Suggest waitstatus_to_exitcode() usage in the documentation when
appropriate.
Use waitstatus_to_exitcode() in:
* multiprocessing, os, subprocess and _bootsubprocess modules;
* test.support.wait_process();
* setup.py: run_command();
* and many tests.
Fix os.getgrouplist(): if getgrouplist() function fails because the
group list is too small, retry with a larger group list.
On failure, the glibc implementation of getgrouplist() sets ngroups
to the total number of groups. For other implementations, double the
group list size.
On macOS, getgrouplist() returns a non-zero value without setting
errno if the group list is too small. Double the list size and call
it again in this case.
Replace _PyInterpreterState_Get() function call with
_PyInterpreterState_GET_UNSAFE() macro which is more efficient but
don't check if tstate or interp is NULL.
_Py_GetConfigsAsDict() now uses _PyThreadState_GET().
The os.putenv() and os.unsetenv() functions are now always available.
On non-Windows platforms, Python now requires setenv() and unsetenv()
functions to build.
Remove putenv_dict from posixmodule.c: it's not longer needed.
If setenv() C function is available, os.putenv() is now implemented
with setenv() instead of putenv(), so Python doesn't have to handle
the environment variable memory.
On most platforms, the `environ` symbol is accessible everywhere.
In a dylib on OSX, it's not easily accessible, you need to find it with
_NSGetEnviron.
The code was caching the *value* of environ. But a setenv() can change the value,
leaving garbage at the old value. Fix: don't cache the value of environ, just
read it every time.
After #9665, this moves the remaining types in posixmodule to be heap-allocated to make it compatible with PEP384 as well as modifying all the type accessors to fully make the type opaque.
The original PR that got messed up a rebase: https://github.com/python/cpython/pull/10854. All the issues in that commit have now been addressed since https://github.com/python/cpython/pull/11661 got committed.
This change also removes any state from the data segment and onto the module state itself.
https://bugs.python.org/issue35381
Automerge-Triggered-By: @encukou
In ArgumentClinic, value "NULL" should now be used only for unrepresentable default values
(like in the optional third parameter of getattr). "None" should be used if None is accepted
as argument and passing None has the same effect as not passing the argument at all.
bpo-37834: Normalise handling of reparse points on Windows
* ntpath.realpath() and nt.stat() will traverse all supported reparse points (previously was mixed)
* nt.lstat() will let the OS traverse reparse points that are not name surrogates (previously would not traverse any reparse point)
* nt.[l]stat() will only set S_IFLNK for symlinks (previous behaviour)
* nt.readlink() will read destinations for symlinks and junction points only
bpo-1311: os.path.exists('nul') now returns True on Windows
* nt.stat('nul').st_mode is now S_IFCHR (previously was an error)
The os.getcwdb() function now uses the UTF-8 encoding on Windows,
rather than the ANSI code page: see PEP 529 for the rationale. The
function is no longer deprecated on Windows.
os.getcwd() and os.getcwdb() now detect integer overflow on memory
allocations. On Unix, these functions properly report MemoryError on
memory allocation failure.
Python now gets the absolute path of the script filename specified on
the command line (ex: "python3 script.py"): the __file__ attribute of
the __main__ module, sys.argv[0] and sys.path[0] become an absolute
path, rather than a relative path.
* Add _Py_isabs() and _Py_abspath() functions.
* _PyConfig_Read() now tries to get the absolute path of
run_filename, but keeps the relative path if _Py_abspath() fails.
* Reimplement os._getfullpathname() using _Py_abspath().
* Use _Py_isabs() in getpath.c.
* Add Include/cpython/import.h and Include/internal/pycore_import.h
header files.
* Move _PyImport_ReInitLock() to the internal C API. Don't export the
symbol anymore.
It is now allowed to add new fields at the end of the PyTypeObject struct without having to allocate a dedicated compatibility flag in tp_flags.
This will reduce the risk of running out of bits in the 32-bit tp_flags value.
* bpo-26836: Add os.memfd_create()
* Use the glibc wrapper for memfd_create()
Co-Authored-By: Christian Heimes <christian@python.org>
* Fix deletions caused by autoreconf.
* Use MFD_CLOEXEC as the default value for *flags*.
* Add memset_s to configure.ac.
* Revert memset_s changes.
* Apply the requested changes.
* Tweak the docs.
PyOS_AfterFork_Child(): _PyInterpreterState_DeleteExceptMain() must
be called after _PyRuntimeState_ReInitThreads().
_PyRuntimeState_ReInitThreads() resets interpreters mutex after fork,
mutex used by _PyInterpreterState_DeleteExceptMain().
Remove the PyEval_ReInitThreads() function from the Python C API.
It should not be called explicitly: use PyOS_AfterFork_Child()
instead.
Rename PyEval_ReInitThreads() to _PyEval_ReInitThreads() and add a
'runtime' parameter.
The PyOS_AfterFork_Child() function now pass a 'runtime' parameter to
subfunctions.
* Fix _PyRuntimeState_ReInitThreads(): use the correct memory allocator
* Add runtime parameter to _PyRuntimeState_ReInitThreads(),
_PyGILState_Reinit() and _PyInterpreterState_DeleteExceptMain()
* Move _PyGILState_Reinit() to the internal C API.
The error message emitted when returning invalid types from __fspath__ in interfaces that allow passing PathLike objects has been improved and now it does explain the origin of the error.
LoadLibrary, GetProcAddress, FreeLibrary and GetModuleHandle acquire the system loader lock. Calling these while holding the GIL will cause a deadlock on the rare occasion that another thread is detaching and needs to destroy its thread state at the same time.
Fix invalid function cast warnings with gcc 8
for method conventions different from METH_NOARGS, METH_O and
METH_VARARGS excluding Argument Clinic generated code.
os_read_impl() now also truncates the size to _PY_READ_MAX
on macOS, to avoid to allocate a larger buffer even if _Py_read() is
limited to _PY_READ_MAX bytes (ex: INT_MAX on macOS).
Some methods in the os module can accept path-like objects. This is documented in the general documentation but not in the function docstrings. To keep both in sync, the docstrings need to be updated to reflect that path-like objects are also accepted.
path_error() uses GetLastError() on Windows, but some os functions
are implemented via CRT APIs which report errors via errno.
This may result in raising OSError with invalid error code (such
as zero).
Introduce posix_path_error() function and use it where appropriate.
Such functions as os.path.exists(), os.path.lexists(), os.path.isdir(),
os.path.isfile(), os.path.islink(), and os.path.ismount() now return False
instead of raising ValueError or its subclasses UnicodeEncodeError
and UnicodeDecodeError for paths that contain characters or bytes
unrepresentative at the OS level.
When os.fork() is called (on platforms that support it) all threads but the current one are destroyed in the child process. Consequently we must ensure that all but the associated interpreter are likewise destroyed. The main interpreter is critical for runtime operation, so we must ensure that fork only happens in the main interpreter.
https://bugs.python.org/issue34651
os.readlink() now accepts path-like and bytes objects on Windows.
Previously, support for path-like and bytes objects was only
implemented on Unix.
This commit also merges Unix and Windows implementations of
os.readlink() in one function and adds basic unit tests to increase
test coverage of the function.
* Fix integer overflow in os.readv(), os.writev(), os.preadv()
and os.pwritev() and in os.sendfile() with headers or trailers
arguments (on BSD-based OSes and MacOS).
* Fix sending the part of the file in os.sendfile() on MacOS.
Using the trailers argument could cause sending more bytes from
the input file than was specified.
Thanks Ned Deily for testing on 32-bit MacOS.
bpo-33671
* use memoryview() with size == file size on Windows, see https://github.com/python/cpython/pull/7160#discussion_r195405230
* release intermediate (sliced) memoryview immediately
* replace "OSX" occurrences with "macOS"
* add some unittests for copyfileobj()
* have shutil.copyfileobj use sendfile() if possible
* refactoring: use ctx manager
* add test with non-regular file obj
* emulate case where file size can't be determined
* reference _copyfileobj_sendfile directly
* add test for offset() at certain position
* add test for empty file
* add test for non regular file dst
* small refactoring
* leave copyfileobj() alone in order to not introduce any incompatibility
* minor refactoring
* remove old test
* update docstring
* update docstring; rename exception class
* detect platforms which only support file to socket zero copy
* don't run test on platforms where file-to-file zero copy is not supported
* use tempfiles
* reset verbosity
* add test for smaller chunks
* add big file size test
* add comment
* update doc
* update whatsnew doc
* update doc
* catch Exception
* remove unused import
* add test case for error on second sendfile() call
* turn docstring into comment
* add one more test
* update comment
* add Misc/NEWS entry
* get rid of COPY_BUFSIZE; it belongs to another PR
* update doc
* expose posix._fcopyfile() for OSX
* merge from linux branch
* merge from linux branch
* expose fcopyfile
* arg clinic for the win implementation
* convert path type to path_t
* expose CopyFileW
* fix windows tests
* release GIL
* minor refactoring
* update doc
* update comment
* update docstrings
* rename functions
* rename test classes
* update doc
* update doc
* update docstrings and comments
* avoid do import nt|posix modules if unnecessary
* set nt|posix modules to None if not available
* micro speedup
* update description
* add doc note
* use better wording in doc
* rename function using 'fastcopy' prefix instead of 'zerocopy'
* use :ref: in rst doc
* change wording in doc
* add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies
* move CopyFileW in _winapi and actually expose CopyFileExW instead
* fix line endings
* add tests for mode bits
* add docstring
* remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies
* update doc to reflect new changes
* update doc
* adjust tests on win
* fix argument clinic error
* update doc
* OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python
* osx / copyfile: use path_t instead of char
* do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux)
* add same file test
* add test for same file
* have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted
* turn PermissionError into appropriate SameFileError
* expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError
* honour follow_symlinks arg when using CopyFileEx
* update Misc/NEWS
* expose CreateDirectoryEx mock
* change C type
* CreateDirectoryExW actual implementation
* provide specific makedirs() implementation for win
* fix typo
* skeleton for SetNamedSecurityInfo
* get security info for src path
* finally set security attrs
* add unit tests
* mimick os.makedirs() behavior and raise if dst dir exists
* set 2 paths for OSError object
* set 2 paths for OSError object
* expand windows test
* in case of exception on os.sendfile() set filename and filename2 exception attributes
* set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX
* update doc
* do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation
* use bytearray() and readinto()
* use memoryview() with bytearray()
* refactoring + introduce a new _fastcopy_binfileobj() fun
* remove CopyFileEx and other C wrappers
* remove code related to CopyFileEx
* Recognize binary files in copyfileobj()
...and use fastest _fastcopy_binfileobj() when possible
* set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable
* use ctx manager for memoryview()
* update doc
* remove outdated doc
* remove last CopyFileEx remnants
* OSX - use fcopyfile(3) instead of copyfile(3)
...as an extra safety measure: in case src/dst are "exotic" files (non
regular or living on a network fs etc.) we better fail on open() instead
of copyfile(3) as we're not quite sure what's gonna happen in that
case.
* update doc
This call became unneeded after the posix module was converted to the
Argument Clinic in 2f93635d34 and should
have been removed as part of that change.
* Expose the sigset_t converter via private API _Py_Sigset_Converter().
* Use Argument Clinic for parsing sigset_t in signalmodule.c.
* Raise ValueError instead OverflowError for integers out of
the C long range.
Based on patch by Pablo Galindo Salgado.
os.dup2() tests for dup3() system call availability at runtime,
but doesn't remember the result across calls, repeating
the test on each call with inheritable=False.
Since the caller of os.dup2() is expected to hold the GIL,
fix this by making the variable holding the test result static.
* Fix memory leaks and error handling in posix spawn
* Improve error handling when destroying the file_actions object
* Py_DECREF the result of PySequence_Fast on error
* Handle uninitialized pid
* Use OSError if file actions fails to initialize
* Move _file_actions to outer scope to avoid undefined behaviour
* Remove HAVE_POSIX_SPAWN define in Modules/posixmodule.c
* Unshadow exception and clean error message
Victor Stinner
Serhiy Storchaka
Kyle Evans
Jakub Kulík
pxinwr
Gregory P. Smith
YoSTEALTH
Minmin Gong
Batuhan Taşkaya
Hai Shi
Andy Lester
Saiyang Gou
Benoit Hudson
Eddie Elizondo
Benjamin Peterson
Dong-hee Na
Steve Dower
Antonio Gutierrez
Zackery Spytz
Min ho Kim
Jeroen Demeyer
Victor Stinner
Brandt Bucher
Jeffrey Kintscher
Pablo Galindo
Antoine Pitrou
Steve Dower
Anthony Shaw
Rémi Lapeyre
Vidar Tonaas Fauske
Eric Snow
Anthony Sottile
Mark Becwar
Tony Roberts
Joannah Nanjekye
Eddie Elizondo
BNMetrics
Alexey Izbyshev
William Orr
Joe Pamer
Oren Milman
Berker Peksag
Giampaolo Rodola
Alexey Izbyshev
ngie-eign
SSE4
oldk
Stéphane Wirtel
Chih-Hsuan Yen
jcea
Michael Felt
Giuseppe Scrivano