Commit Graph

1344 Commits

Author SHA1 Message Date
neonene d4e64cd4b0
bpo-46362: Ensure ntpath.abspath() uses the Windows API correctly (GH-30571)
This makes ntpath.abspath()/getpath_abspath() follow normpath(), since some WinAPIs such as PathCchSkipRoot() require backslashed paths.
2022-01-13 23:35:42 +00:00
Christian Heimes a6ca8eee22
bpo-46315: Add ifdef HAVE_ feature checks for WASI compatibility (GH-30507) 2022-01-13 09:46:04 +01:00
David CARLIER c960b191b8
bpo-46222: posixmodule sendfile FreeBSD's constants updates. (GH-30327)
* posixodule sendfile FreeBSD's constants updates.

* 📜🤖 Added by blurb_it.

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-01-03 15:01:04 +02:00
Kumar Aditya 41026c3155
bpo-45855: Replaced deprecated `PyImport_ImportModuleNoBlock` with PyImport_ImportModule (GH-30046) 2021-12-12 10:45:20 +02:00
Eric Snow 758b74e71e
bpo-46008: Add _PyInterpreterState_Main(). (gh-29978)
PyInterpreterState_Main() is a plain function exposed in the public C-API. For internal usage we can take the more efficient approach in this PR.

https://bugs.python.org/issue46008
2021-12-07 18:56:06 -07:00
Steve Dower 99fcf15052
bpo-45582: Port getpath[p].c to Python (GH-29041)
The getpath.py file is frozen at build time and executed as code over a namespace. It is never imported, nor is it meant to be importable or reusable. However, it should be easier to read, modify, and patch than the previous code.

This commit attempts to preserve every previously tested quirk, but these may be changed in the future to better align platforms.
2021-12-03 00:08:42 +00:00
Christian Heimes 4ebde73b8e
bpo-40280: Move hard-coded feature checks to configure (GH-29789)
Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@innova.no>
2021-11-26 19:36:48 +01:00
Christian Heimes dc19e86983
bpo-41498: Fix build on platforms without sigset_t (GH-29770) 2021-11-25 12:28:41 +01:00
Victor Stinner aac29af678
bpo-45434: pyport.h no longer includes <stdlib.h> (GH-28914)
Include <stdlib.h> explicitly in C files.

Python.h includes <wchar.h>.
2021-10-13 19:25:53 +02:00
Victor Stinner 7cdc2a0f4b
pycore_pystate.h no longer redefines PyThreadState_GET() (GH-28921)
Redefining the PyThreadState_GET() macro in pycore_pystate.h is
useless since it doesn't affect files not including it. Either use
_PyThreadState_GET() directly, or don't use pycore_pystate.h internal
C API. For example, the _testcapi extension don't use the internal C
API, but use the public PyThreadState_Get() function instead.

Replace PyThreadState_Get() with _PyThreadState_GET(). The
_PyThreadState_GET() macro is more efficient than PyThreadState_Get()
and PyThreadState_GET() function calls which call fail with a fatal
Python error.

posixmodule.c and _ctypes extension now include <windows.h> before
pycore header files (like pycore_call.h).

_PyTraceback_Add() now uses _PyErr_Fetch()/_PyErr_Restore() instead
of PyErr_Fetch()/PyErr_Restore().

The _decimal and _xxsubinterpreters extensions are now built with the
Py_BUILD_CORE_MODULE macro defined to get access to the internal C
API.
2021-10-13 14:09:13 +02:00
Victor Stinner d943d19172
bpo-45439: Move _PyObject_CallNoArgs() to pycore_call.h (GH-28895)
* Move _PyObject_CallNoArgs() to pycore_call.h (internal C API).
* _ssl, _sqlite and _testcapi extensions now call the public
  PyObject_CallNoArgs() function, rather than _PyObject_CallNoArgs().
* _lsprof extension is now built with Py_BUILD_CORE_MODULE macro
  defined to get access to internal _PyObject_CallNoArgs().
2021-10-12 08:38:19 +02:00
Victor Stinner ce3489cfdb
bpo-45439: Rename _PyObject_CallNoArg() to _PyObject_CallNoArgs() (GH-28891)
Fix typo in the private _PyObject_CallNoArg() function name: rename
it to _PyObject_CallNoArgs() to be consistent with the public
function PyObject_CallNoArgs().
2021-10-12 00:42:23 +02:00
Serhiy Storchaka f25f2e2e8c
Clean up initialization __class_getitem__ with Py_GenericAlias. (GH-28450)
The cast to PyCFunction is redundant. Overuse of redundant casts
can hide actual bugs.
2021-09-19 18:05:30 +03:00
Vincent Michel 06148b1870
bpo-44219: Release the GIL during isatty syscalls (GH-28250)
Release the GIL while performing isatty() system calls on arbitrary
file descriptors. In particular, this affects os.isatty(),
os.device_encoding() and io.TextIOWrapper. By extension,
io.open() in text mode is also affected.
2021-09-09 15:12:03 +02:00
Stanisław Skonieczny 9dc363ee7c
bpo-45012: Release GIL around stat in os.scandir (GH-28085)
Releasing GIL allows other threads to continue
its work when os.scandir is fetching DirEntry.stat
info from file system.
2021-09-07 19:55:20 +02:00
Samuel Marks 19459f8ce6
bpo-44321: Adds `os.EX_OK` for Windows (GH-26559) 2021-06-24 17:45:18 +01:00
Victor Stinner 0cad068ec1
bpo-43916: Remove _disabled_new() function (GH-25745)
posix and _hashlib use the new Py_TPFLAGS_DISALLOW_INSTANTIATION
flag on their heap types, rather than using a custom tp_new function
(_disabled_new).
2021-04-30 14:06:49 +02:00
Segev Finer 5e437fb872
bpo-30555: Fix WindowsConsoleIO fails in the presence of fd redirection (GH-1927)
This works by not caching the handle and instead getting the handle from
the file descriptor each time, so that if the actual handle changes by
fd redirection closing/opening the console handle beneath our feet, we
will keep working correctly.
2021-04-23 23:00:27 +01:00
Steve Dower 019e9e8168
bpo-43538: Add extra arguments to os.startfile (GH-25538) 2021-04-23 18:03:17 +01:00
Steve Dower e07d809889
bpo-38222: Check specifically for a drive, not just a colon (GH-25540) 2021-04-23 00:30:37 +01:00
Steve Dower fe63a401a9
bpo-38822: Fixed os.stat failing on inaccessible directories. (GH-25527)
It would just fail if the path was inaccessible and had a trailing slash. It should fall back to the parent directory's metadata.
2021-04-22 20:45:02 +01:00
Victor Stinner cdad2724e6
bpo-40137: Add pycore_moduleobject.h internal header (GH-25507)
Add pycore_moduleobject.h internal header file with static inline
functions to access module members:

* _PyModule_GetDict()
* _PyModule_GetDef()
* _PyModule_GetState()

These functions don't check at runtime if their argument has a valid
type and can be inlined even if Python is not built with LTO.

_PyType_GetModuleByDef() uses _PyModule_GetDef().

Replace PyModule_GetState() with _PyModule_GetState() in the
extension modules, considered as performance sensitive:

* _abc
* _functools
* _operator
* _pickle
* _queue
* _random
* _sre
* _struct
* _thread
* _winapi
* array
* posix

The following extensions are now built with the Py_BUILD_CORE_MODULE
macro defined, to be able to use the internal pycore_moduleobject.h
header: _abc, array, _operator, _queue, _sre, _struct.
2021-04-22 00:52:52 +02:00
Steve Dower 04732ca993
bpo-43105: Importlib now resolves relative paths when creating module spec objects from file locations (GH-25121) 2021-04-07 01:02:07 +01:00
Dong-hee Na f917c243c5
bpo-43106: Add os.O_EVTONLY/O_FSYNC/O_SYMLINK/O_NOFOLLOW_ANY (GH-24428) 2021-02-04 08:32:55 +09:00
Joshua Root df21f502fd
bpo-42692: fix __builtin_available check on older compilers (GH-23873)
A compiler that doesn't define `__has_builtin` will error out when it is
used on the same line as the check for it.

Automerge-Triggered-By: GH:ronaldoussoren
2021-01-04 02:36:58 -08:00
Jakub Kulík 0159e5efee
bpo-42655: Fix subprocess extra_groups gid conversion (GH-23762) 2020-12-29 14:58:27 +02:00
Pablo Galindo dedc2cd5f0
bpo-41625: Do not add os.splice on AIX due to compatibility issues (GH-23608) 2020-12-02 17:57:18 +00:00
Victor Stinner 00d7abd7ef
bpo-42519: Replace PyMem_MALLOC() with PyMem_Malloc() (GH-23586)
No longer use deprecated aliases to functions:

* Replace PyMem_MALLOC() with PyMem_Malloc()
* Replace PyMem_REALLOC() with PyMem_Realloc()
* Replace PyMem_FREE() with PyMem_Free()
* Replace PyMem_Del() with PyMem_Free()
* Replace PyMem_DEL() with PyMem_Free()

Modify also the PyMem_DEL() macro to use directly PyMem_Free().
2020-12-01 09:56:42 +01:00
Victor Stinner 6223071421
bpo-1635741: Convert _imp to multi-phase init (GH-23378)
Convert the _imp extension module to the multi-phase initialization
API (PEP 489).

* Add _PyImport_BootstrapImp() which fix a bootstrap issue: import
  the _imp module before importlib is initialized.
* Add create_builtin() sub-function, used by _imp_create_builtin().
* Initialize PyInterpreterState.import_func earlier, in
  pycore_init_builtins().
* Remove references to _PyImport_Cleanup(). This function has been
  renamed to finalize_modules() and moved to pylifecycle.c.
2020-11-18 23:18:29 +01:00
Pablo Galindo 2a9eddf070
bpo-41625: Add a guard for Linux for splice() constants in the os module (GH-23350) 2020-11-17 11:57:49 -08:00
Pablo Galindo a57b3d30f6
bpo-41625: Expose the splice() system call in the os module (GH-21947) 2020-11-17 00:00:38 +00:00
Christian Heimes cd9fed6afb
bpo-41001: Add os.eventfd() (#20930)
Co-authored-by: Kyle Stanley <aeros167@gmail.com>
2020-11-13 19:48:52 +01:00
Jakub Stasiak fd4ed57674
bpo-42237: Fix os.sendfile() on illumos (GH-23154) 2020-11-12 11:49:30 +02:00
Ronald Oussoren 41761933c1
bpo-41100: Support macOS 11 and Apple Silicon (GH-22855)
Co-authored-by:  Lawrence D’Anna <lawrence_danna@apple.com>

* Add support for macOS 11 and Apple Silicon (aka arm64)
   
  As a side effect of this work use the system copy of libffi on macOS, and remove the vendored copy

* Support building on recent versions of macOS while deploying to older versions

  This allows building installers on macOS 11 while still supporting macOS 10.9.
2020-11-08 10:05:27 +01:00
Victor Stinner 5776663675
bpo-42029: Remove IRIX code (GH-23023)
IRIX code was slowy removed in Python 2.4 (--with-sgi-dl), Python 3.3
(Irix threads), and Python 3.7.
2020-10-29 15:16:23 +01:00
Serhiy Storchaka b510e101f8
bpo-42152: Use PyDict_Contains and PyDict_SetDefault if appropriate. (GH-22986)
If PyDict_GetItemWithError is only used to check whether the key is in dict,
it is better to use PyDict_Contains instead.

And if it is used in combination with PyDict_SetItem, PyDict_SetDefault can
replace the combination.
2020-10-26 12:47:57 +02:00
Kyle Evans 7992579cd2
bpo-40422: Move _Py_closerange to fileutils.c (GH-22680)
This API is relatively lightweight and organizationally, given that it's
used by multiple modules, it makes sense to move it to fileutils.

Requires making sure that _posixsubprocess is compiled with the appropriate
Py_BUIILD_CORE_BUILTIN macro.
2020-10-13 22:04:44 +02:00
Kyle Evans 64eb259cc1
bpo-40422: Move _Py_*_SUPPRESS_IPH bits into _Py_closerange (GH-22672)
This suppression is no longer needed in os_closerange_impl, as it just
invokes the internal _Py_closerange implementation. On the other hand,
consumers of _Py_closerange may not have any other reason to suppress
invalid parameter issues, so narrow the scope to here.
2020-10-12 16:53:16 -07:00
Kyle Evans 1800c60080
bpo-40423: Optimization: use close_range(2) if available (GH-22651)
close_range(2) should be preferred at all times if it's available, otherwise we'll use closefrom(2) if available with a fallback to fdwalk(3) or plain old loop over fd range in order of most efficient to least.

[note that this version does check for ENOSYS, but currently ignores all other errors]

Automerge-Triggered-By: @pablogsal
2020-10-11 13:18:53 -07:00
Kyle Evans c230fde847
bpo-40422: create a common _Py_closerange API (GH-19754)
Such an API can be used both for os.closerange and subprocess. For the latter, this yields potential improvement for platforms that have fdwalk but wouldn't have used it there. This will prove even more beneficial later for platforms that have close_range(2), as the new API will prefer that over all else if it's available.

The new API is structured to look more like close_range(2), closing from [start, end] rather than the [low, high) of os.closerange().

Automerge-Triggered-By: @gpshead
2020-10-11 11:54:11 -07:00
Serhiy Storchaka 9975cc5008
bpo-41985: Add _PyLong_FileDescriptor_Converter and AC converter for "fildes". (GH-22620) 2020-10-09 23:00:45 +03:00
Jakub Kulík fa8c9e7010
bpo-41687: Fix error handling in Solaris sendfile implementation (GH-22128)
I just realized that my recent PR with sendfile on Solaris ([PR 22040](https://github.com/python/cpython/pull/22040)) has broken error handling.

Sorry for that, this simple followup fixes that.

Automerge-Triggered-By: @1st1
2020-09-09 12:29:42 -07:00
Jakub Kulík 8c0be6fd91
bpo-41687: Fix sendfile implementation to work with Solaris (#22040) 2020-09-05 12:10:01 -07:00
pxinwr 3405e05428
bpo-41440: add os.cpu_count() support for VxWorks RTOS (GH-21685) 2020-08-07 14:21:52 +09:00
Serhiy Storchaka 4c8f09d7ce
bpo-36346: Make using the legacy Unicode C API optional (GH-21437)
Add compile time option USE_UNICODE_WCHAR_CACHE. Setting it to 0
makes the interpreter not using the wchar_t cache and the legacy Unicode C API.
2020-07-10 23:26:06 +03:00
Gregory P. Smith 3ccb96c978
bpo-41056: Use the fildes converter for fd to please Coverity. (GH-21011)
There are a bunch of other fd: int uses in this file, I expect many if not
all of them would be better off using the fildes converter.  This particular
one was flagged by Coverity as it presumably flags fpathconf as not accepting
negative fds.  I'd expect the other fd's to have been flagged as well
otherwise.

I'm marking this one as skip news as it really is a no-op.
2020-06-20 15:06:48 -07:00
Victor Stinner 317bab0bf6
PyOS_AfterFork_Child() pass tstate to _PyEval_ReInitThreads() (GH-20598) 2020-06-02 18:44:54 +02:00
Victor Stinner 26881c8fae
PyOS_AfterFork_Child() uses PyStatus (GH-20596)
PyOS_AfterFork_Child() helper functions now return a PyStatus:
PyOS_AfterFork_Child() is now responsible to handle errors.

* Move _PySignal_AfterFork() to the internal C API
* Add #ifdef HAVE_FORK on _PyGILState_Reinit(), _PySignal_AfterFork()
  and _PyInterpreterState_DeleteExceptMain().
2020-06-02 15:51:37 +02:00
Serhiy Storchaka 5f4b229df7
bpo-40792: Make the result of PyNumber_Index() always having exact type int. (GH-20443)
Previously, the result could have been an instance of a subclass of int.

Also revert bpo-26202 and make attributes start, stop and step of the range
object having exact type int.

Add private function _PyNumber_Index() which preserves the old behavior
of PyNumber_Index() for performance to use it in the conversion functions
like PyLong_AsLong().
2020-05-28 10:33:45 +03:00
YoSTEALTH 76ef255bde
bpo-37129: Add os.RWF_APPEND flag for os.pwritev() (GH-20336) 2020-05-27 23:32:22 +02:00
Serhiy Storchaka 578c3955e0
bpo-37999: No longer use __int__ in implicit integer conversions. (GH-15636)
Only __index__ should be used to make integer conversions lossless.
2020-05-26 18:43:38 +03:00
Minmin Gong 7f21c9ac87
bpo-40653: Move _dirnameW out of #ifdef HAVE_SYMLINK/#endif (GH-20144) 2020-05-18 17:17:19 +01:00
Victor Stinner 97f33c3544
bpo-40549: posixmodule.c uses defining_class (GH-20075)
Pass PEP 573 defining_class to os.DirEntry methods. The module state
is now retrieve from defining_class rather than Py_TYPE(self), to
support subclasses (even if DirEntry doesn't support subclasses yet).

* Pass the module rather than defining_class to DirEntry_fetch_stat().
* Only get the module state once in _posix_clear(),
  _posix_traverse() and _posixmodule_exec().
2020-05-14 18:05:58 +02:00
Victor Stinner 1c2fa78156
bpo-40549: Convert posixmodule.c to multiphase init (GH-19982)
Convert posixmodule.c ("posix" or "nt" module) to the multiphase
initialization (PEP 489).

* Create the module using PyModuleDef_Init().
* Create ScandirIteratorType and DirEntryType with the new
  PyType_FromModuleAndSpec() (PEP 573)
* Get the module state from ScandirIteratorType and DirEntryType with
  the new PyType_GetModule() (PEP 573)
* Pass module to functions which access the module state.
* convert_sched_param() gets a new module parameter. It is now called
  directly since Argument Clinic doesn't support passing the module
  to an argument converter callback.
* Remove _posixstate_global macro.
2020-05-10 11:05:29 +02:00
Victor Stinner 252346acd9
bpo-40453: Add PyConfig._isolated_subinterpreter (GH-19820)
An isolated subinterpreter cannot spawn threads, spawn a child
process or call os.fork().

* Add private _Py_NewInterpreter(isolated_subinterpreter) function.
* Add isolated=True keyword-only parameter to
  _xxsubinterpreters.create().
* Allow again os.fork() in "non-isolated" subinterpreters.
2020-05-01 11:33:44 +02:00
Victor Stinner 162c567d16
bpo-38061: os.closerange() uses closefrom() on FreeBSD (GH-19696)
On FreeBSD, os.closerange(fd_low, fd_high) now calls
closefrom(fd_low) if fd_high is greater than or equal to
sysconf(_SC_OPEN_MAX).

Initial patch by Ed Maste (emaste), Conrad Meyer (cem), Kyle Evans
(kevans) and Kubilay Kocak (koobs):
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274
2020-04-24 12:00:51 +02:00
Victor Stinner 9bee32b34e
bpo-40138: Fix Windows os.waitpid() for large exit code (GH-19637)
Fix the Windows implementation of os.waitpid() for exit code
larger than "INT_MAX >> 8". The exit status is now interpreted as an
unsigned number.

os.waitstatus_to_exitcode() now accepts wait status larger than
INT_MAX.
2020-04-22 16:30:35 +02:00
Serhiy Storchaka 2b5603140c
bpo-40178: Convert the remaining os functions to Argument Clinic. (GH-19360)
Convert os.getgrouplist(), os.initgroups(), os.sendfile() and
os.get_terminal_size().
2020-04-18 19:14:10 +03:00
Victor Stinner 5eca75df03
bpo-40268: Reformat posixmodule.c includes (GH-19536) 2020-04-15 15:07:31 +02:00
Victor Stinner 361dcdcefc
bpo-40268: Remove unused osdefs.h includes (GH-19532)
When the include is needed, add required symbol in a comment.
2020-04-15 03:24:57 +02:00
Victor Stinner 4a21e57fe5
bpo-40268: Remove unused structmember.h includes (GH-19530)
If only offsetof() is needed: include stddef.h instead.

When structmember.h is used, add a comment explaining that
PyMemberDef is used.
2020-04-15 02:35:41 +02:00
Victor Stinner 62183b8d6d
bpo-40268: Remove explicit pythread.h includes (#19529)
Remove explicit pythread.h includes: it is always included
by Python.h.
2020-04-15 02:04:42 +02:00
Victor Stinner e5014be049
bpo-40268: Remove a few pycore_pystate.h includes (GH-19510) 2020-04-14 17:52:15 +02:00
Victor Stinner 81a7be3fa2
bpo-40268: Rename _PyInterpreterState_GET_UNSAFE() (GH-19509)
Rename _PyInterpreterState_GET_UNSAFE() to _PyInterpreterState_GET()
for consistency with _PyThreadState_GET() and to have a shorter name
(help to fit into 80 columns).

Add also "assert(tstate != NULL);" to the function.
2020-04-14 15:14:01 +02:00
Batuhan Taşkaya f9dd51e7db
bpo-39481: Make os.DirEntry generic (GH-19415) 2020-04-07 14:37:19 -07:00
Victor Stinner 87255be696
bpo-40089: Add _at_fork_reinit() method to locks (GH-19195)
Add a private _at_fork_reinit() method to _thread.Lock,
_thread.RLock, threading.RLock and threading.Condition classes:
reinitialize the lock after fork in the child process; reset the lock
to the unlocked state.

Rename also the private _reset_internal_locks() method of
threading.Event to _at_fork_reinit().

* Add _PyThread_at_fork_reinit() private function. It is excluded
  from the limited C API.
* threading.Thread._reset_internal_locks() now calls
  _at_fork_reinit() on self._tstate_lock rather than creating a new
  Python lock object.
2020-04-07 23:11:49 +02:00
Batuhan Taşkaya 909f4a3009
bpo-40190: Add support for _SC_AIX_REALMEM in sysconf (GH-19380) 2020-04-05 02:40:49 +02:00
Victor Stinner 65a796e527
bpo-40094: Add os.waitstatus_to_exitcode() (GH-19201)
Add os.waitstatus_to_exitcode() function to convert a wait status to an
exitcode.

Suggest waitstatus_to_exitcode() usage in the documentation when
appropriate.

Use waitstatus_to_exitcode() in:

* multiprocessing, os, subprocess and _bootsubprocess modules;
* test.support.wait_process();
* setup.py: run_command();
* and many tests.
2020-04-01 18:49:29 +02:00
Victor Stinner f5c7cabb2b
bpo-40014: Fix os.getgrouplist() (GH-19126)
Fix os.getgrouplist(): if getgrouplist() function fails because the
group list is too small, retry with a larger group list.

On failure, the glibc implementation of getgrouplist() sets ngroups
to the total number of groups. For other implementations, double the
group list size.
2020-03-24 18:22:10 +01:00
Victor Stinner 8ec7370c89
bpo-40014: Fix os.getgrouplist() on macOS (GH-19118)
On macOS, getgrouplist() returns a non-zero value without setting
errno if the group list is too small. Double the list size and call
it again in this case.
2020-03-23 20:00:57 +01:00
Hai Shi f707d94af6
bpo-39968: Convert extension modules' macros of get_module_state() to inline functions (GH-19017) 2020-03-16 14:15:01 +01:00
Victor Stinner ff4584caca
bpo-39947: Use _PyInterpreterState_GET_UNSAFE() (GH-18978)
Replace _PyInterpreterState_Get() function call with
_PyInterpreterState_GET_UNSAFE() macro which is more efficient but
don't check if tstate or interp is NULL.

_Py_GetConfigsAsDict() now uses _PyThreadState_GET().
2020-03-13 18:03:56 +01:00
Andy Lester 557287075c
bpo-39573: Use Py_IS_TYPE() macro to check for types (GH-18809)
Co-authored-by: Victor Stinner <vstinner@python.org>
2020-03-06 23:53:17 +01:00
Saiyang Gou 7514f4f625
bpo-39184: Add audit events to functions in `fcntl`, `msvcrt`, `os`, `resource`, `shutil`, `signal`, `syslog` (GH-18407) 2020-02-13 07:47:42 +00:00
Saiyang Gou 95f6001021
bpo-39184: Add audit events to command execution functions in os and pty modules (GH-17824) 2020-02-05 11:15:00 +11:00
Victor Stinner b8d1262e8a
bpo-39395: putenv() and unsetenv() always available (GH-18135)
The os.putenv() and os.unsetenv() functions are now always available.

On non-Windows platforms, Python now requires setenv() and unsetenv()
functions to build.

Remove putenv_dict from posixmodule.c: it's not longer needed.
2020-01-24 14:05:48 +01:00
Victor Stinner 161e7b36b1
bpo-39413: Implement os.unsetenv() on Windows (GH-18163)
The os.unsetenv() function is now also available on Windows.
2020-01-24 11:53:44 +01:00
Victor Stinner b477d19a6b
bpo-39406: Implement os.putenv() with setenv() if available (GH-18128)
If setenv() C function is available, os.putenv() is now implemented
with setenv() instead of putenv(), so Python doesn't have to handle
the environment variable memory.
2020-01-22 22:48:16 +01:00
Victor Stinner 0852c7dd52
bpo-39406: os.putenv() avoids putenv_dict on Windows (GH-18126)
Windows: _wputenv(env) copies the *env* string and doesn't require
the caller to manage the variable memory.
2020-01-22 21:53:26 +01:00
Victor Stinner b73dd02ea7
Revert "bpo-39413: Implement os.unsetenv() on Windows (GH-18104)" (GH-18124)
This reverts commit 56cd3710a1.
2020-01-22 21:11:17 +01:00
Victor Stinner 623ed6171e
bpo-39406: Add PY_PUTENV_DICT macro to posixmodule.c (GH-18106)
Rename posix_putenv_garbage to putenv_dict.
2020-01-21 19:25:32 +01:00
Victor Stinner 56cd3710a1
bpo-39413: Implement os.unsetenv() on Windows (GH-18104)
The os.unsetenv() function is now also available on Windows.

It is implemented with SetEnvironmentVariableW(name, NULL).
2020-01-21 16:13:09 +01:00
Benoit Hudson 723f71abf7 bpo-37931: Fix crash on OSX re-initializing os.environ (GH-15428)
On most platforms, the `environ` symbol is accessible everywhere.

In a dylib on OSX, it's not easily accessible, you need to find it with
_NSGetEnviron.

The code was caching the *value* of environ. But a setenv() can change the value,
leaving garbage at the old value. Fix: don't cache the value of environ, just
read it every time.
2019-12-06 20:15:03 +01:00
Eddie Elizondo e4db1f05e9 closes bpo-38803: Fix leak in posixmodule. (GH-17373) 2019-11-25 19:07:37 -08:00
Benjamin Peterson 5c0c325453 closes bpo-38713: Expose P_PIDFD in os if it's defined. (GH-17071)
https://bugs.python.org/issue38713
2019-11-05 21:58:31 -08:00
Benjamin Peterson 6c4c45efae
bpo-38692: Add os.pidfd_open. (GH-17063) 2019-11-05 19:21:29 -08:00
Eddie Elizondo b3966639d2 bpo-35381 Remove all static state from posixmodule (GH-15892)
After #9665, this moves the remaining types in posixmodule to be heap-allocated to make it compatible with PEP384 as well as modifying all the type accessors to fully make the type opaque.

The original PR that got messed up a rebase: https://github.com/python/cpython/pull/10854. All the issues in that commit have now been addressed since https://github.com/python/cpython/pull/11661 got committed.

This change also removes any state from the data segment and onto the module state itself.


https://bugs.python.org/issue35381



Automerge-Triggered-By: @encukou
2019-11-05 07:16:14 -08:00
Dong-hee Na 2eba6ad7bf bpo-38493: Add os.CLD_KILLED and os.CLD_STOPPED. (GH-16821) 2019-10-21 10:01:05 +03:00
Steve Dower fbe3c76c7c bpo-38418: Fixes audit event for os.system to be named 'os.system' (GH-16670)
https://bugs.python.org/issue38418



Automerge-Triggered-By: @zooba
2019-10-18 00:52:15 -07:00
Serhiy Storchaka 140a7d1f35
bpo-38378: Rename parameters "out" and "in" of os.sendfile(). (GH-16742)
They conflicted with keyword "in".

Also rename positional-only parameters of private os._fcopyfile()
for consistency.
2019-10-13 11:59:31 +03:00
Antonio Gutierrez 594e2edfb5 closes bpo-36161: Use thread-safe ttyname_r instead of ttyname. (GH-14868)
Signed-off-by: Antonio Gutierrez <chibby0ne@gmail.com>
2019-10-08 19:19:48 -07:00
Dong-hee Na ad7736faf5 bpo-38265: Update os.pread to accept the length type as Py_ssize_t. (GH-16359) 2019-09-25 08:47:04 +03:00
Serhiy Storchaka 279f44678c
bpo-37206: Unrepresentable default values no longer represented as None. (GH-13933)
In ArgumentClinic, value "NULL" should now be used only for unrepresentable default values
(like in the optional third parameter of getattr). "None" should be used if None is accepted
as argument and passing None has the same effect as not passing the argument at all.
2019-09-14 12:24:05 +03:00
Steve Dower 0b72ccff56 bpo-32592: Set Windows 8 as the minimum required version for API support (GH-15951) 2019-09-11 17:03:37 +01:00
Steve Dower aa929273ca bpo-33166: Change os.cpu_count to return active (real) processors (GH-15949) 2019-09-11 16:15:39 +01:00
Jakub Kulík e20134f889 bpo-38110: Use fdwalk for os.closerange() when available. (GH-15224)
Use fdwalk() on platforms that support it to implement os.closerange().
2019-09-11 10:11:57 -05:00
Zackery Spytz 682107cf45 bpo-36279: Ensure os.wait3() rusage is initialized (GH-15111)
Co-Authored-By: David Wilson <dw@botanicus.net>
2019-09-09 08:48:32 -07:00
Steve Dower 772ec0fad5
bpo-38030: Fix os.stat failures on block devices on Windows (GH-15681) 2019-09-04 14:42:54 -07:00
Steve Dower 993ac92418
bpo-38020: Fixes crash in os.readlink() on Windows (GH-15663) 2019-09-03 12:50:51 -07:00
Steve Dower df2d4a6f3d
bpo-37834: Normalise handling of reparse points on Windows (GH-15231)
bpo-37834: Normalise handling of reparse points on Windows
* ntpath.realpath() and nt.stat() will traverse all supported reparse points (previously was mixed)
* nt.lstat() will let the OS traverse reparse points that are not name surrogates (previously would not traverse any reparse point)
* nt.[l]stat() will only set S_IFLNK for symlinks (previous behaviour)
* nt.readlink() will read destinations for symlinks and junction points only

bpo-1311: os.path.exists('nul') now returns True on Windows
* nt.stat('nul').st_mode is now S_IFCHR (previously was an error)
2019-08-21 15:27:33 -07:00
Min ho Kim c4cacc8c5e Fix typos in comments, docs and test names (#15018)
* Fix typos in comments, docs and test names

* Update test_pyparse.py

account for change in string length

* Apply suggestion: splitable -> splittable

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Apply suggestion: splitable -> splittable

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Apply suggestion: Dealloccte -> Deallocate

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Update posixmodule checksum.

* Reverse idlelib changes.
2019-07-30 18:16:13 -04:00
Jeroen Demeyer 7f41c8e0dd bpo-37493: use _PyObject_CallNoArg in more places (GH-14575) 2019-07-04 19:35:31 +09:00
Victor Stinner ec3e20a2d1
bpo-37412: Fix os.getcwd() for long path on Windows (GH-14424)
* Fix test for integer overflow.
* Add an unit test.
2019-06-28 18:01:59 +02:00
Brandt Bucher 45a30af109 closes bpo-37420: Handle errors during iteration in os.sched_setaffinity. (GH-14414) 2019-06-27 09:10:57 -07:00
Zackery Spytz d52a83a3d4 bpo-37419: Fix possible segfaults when passing large sequences to os.posix_spawn() (GH-14409)
Use Py_ssize_t instead of int for i.
2019-06-26 21:54:19 +01:00
Victor Stinner 689830ee62
bpo-37412: os.getcwdb() now uses UTF-8 on Windows (GH-14396)
The os.getcwdb() function now uses the UTF-8 encoding on Windows,
rather than the ANSI code page: see PEP 529 for the rationale. The
function is no longer deprecated on Windows.

os.getcwd() and os.getcwdb() now detect integer overflow on memory
allocations. On Unix, these functions properly report MemoryError on
memory allocation failure.
2019-06-26 17:31:12 +02:00
Victor Stinner 3939c321c9
bpo-20443: _PyConfig_Read() gets the absolute path of run_filename (GH-14053)
Python now gets the absolute path of the script filename specified on
the command line (ex: "python3 script.py"): the __file__ attribute of
the __main__ module, sys.argv[0] and sys.path[0] become an absolute
path, rather than a relative path.

* Add _Py_isabs() and _Py_abspath() functions.
* _PyConfig_Read() now tries to get the absolute path of
  run_filename, but keeps the relative path if _Py_abspath() fails.
* Reimplement os._getfullpathname() using _Py_abspath().
* Use _Py_isabs() in getpath.c.
2019-06-25 15:02:43 +02:00
Steve Dower 60419a7e96
bpo-37363: Add audit events for a range of modules (GH-14301) 2019-06-24 08:42:54 -07:00
Victor Stinner 01b63ecac6
bpo-35134: Add Include/cpython/import.h header file (GH-14213)
* Add Include/cpython/import.h and Include/internal/pycore_import.h
  header files.
* Move _PyImport_ReInitLock() to the internal C API. Don't export the
  symbol anymore.
2019-06-19 00:48:09 +02:00
Jeffrey Kintscher 8725c83ed5 bpo-35070: test_getgrouplist may fail on macOS if too many groups (GH-13071) 2019-06-13 03:01:29 -04:00
Pablo Galindo aac4d0342c
bpo-26826: Expose copy_file_range in the os module (GH-7255) 2019-05-31 19:39:47 +01:00
Jeroen Demeyer 530f506ac9 bpo-36974: tp_print -> tp_vectorcall_offset and tp_reserved -> tp_as_async (GH-13464)
Automatically replace
tp_print -> tp_vectorcall_offset
tp_compare -> tp_as_async
tp_reserved -> tp_as_async
2019-05-30 19:13:39 -07:00
Zackery Spytz e70bfa95e6 bpo-26836: Add ifdefs for all MFD_HUGE* constants (GH-13666)
https://bugs.python.org/issue26836
2019-05-29 13:43:49 -07:00
Antoine Pitrou ada319bb6d
bpo-32388: Remove cross-version binary compatibility requirement in tp_flags (GH-4944)
It is now allowed to add new fields at the end of the PyTypeObject struct without having to allocate a dedicated compatibility flag in tp_flags.

This will reduce the risk of running out of bits in the 32-bit tp_flags value.
2019-05-29 22:12:38 +02:00
Zackery Spytz 43fdbd2729 bpo-26836: Add os.memfd_create() (#13567)
* bpo-26836: Add os.memfd_create()

* Use the glibc wrapper for memfd_create()

Co-Authored-By: Christian Heimes <christian@python.org>

* Fix deletions caused by autoreconf.

* Use MFD_CLOEXEC as the default value for *flags*.

* Add memset_s to configure.ac.

* Revert memset_s changes.

* Apply the requested changes.

* Tweak the docs.
2019-05-29 21:57:03 +02:00
Victor Stinner b49858b4b7
bpo-37031: Fix PyOS_AfterFork_Child() (GH-13537)
PyOS_AfterFork_Child(): _PyInterpreterState_DeleteExceptMain() must
be called after _PyRuntimeState_ReInitThreads().

_PyRuntimeState_ReInitThreads() resets interpreters mutex after fork,
mutex used by _PyInterpreterState_DeleteExceptMain().
2019-05-24 15:20:23 +02:00
Steve Dower b82e17e626
bpo-36842: Implement PEP 578 (GH-12613)
Adds sys.audit, sys.addaudithook, io.open_code, and associated C APIs.
2019-05-23 08:45:22 -07:00
pxinwr f2d7ac7e5b bpo-31904: Add posix module support for VxWorks (GH-12118) 2019-05-21 12:46:37 +02:00
Victor Stinner d5d9e81ce9
bpo-36728: Remove PyEval_ReInitThreads() from C API (GH-13241)
Remove the PyEval_ReInitThreads() function from the Python C API.
It should not be called explicitly: use PyOS_AfterFork_Child()
instead.

Rename PyEval_ReInitThreads() to _PyEval_ReInitThreads() and add a
'runtime' parameter.
2019-05-13 12:35:37 +02:00
Anthony Shaw 948ed8c96b bpo-36814: ensure os.posix_spawn() handles None (GH-13144)
Fix an issue where os.posix_spawn() would incorrectly raise a TypeError
when file_actions is None.
2019-05-10 04:00:06 +02:00
Rémi Lapeyre f0900199d5 Suppress clang warning (GH-12384) 2019-05-04 08:30:53 +09:00
Victor Stinner b930a2d2b1
bpo-36710: PyOS_AfterFork_Child() pass runtime parameter (GH-12936)
The PyOS_AfterFork_Child() function now pass a 'runtime' parameter to
subfunctions.

* Fix _PyRuntimeState_ReInitThreads(): use the correct memory allocator
* Add runtime parameter to _PyRuntimeState_ReInitThreads(),
  _PyGILState_Reinit() and _PyInterpreterState_DeleteExceptMain()
* Move _PyGILState_Reinit() to the internal C API.
2019-04-24 17:14:33 +02:00
Vidar Tonaas Fauske 0e10766574 bpo-31512: Add non-elevated symlink support for Windows (GH-3652) 2019-04-09 11:19:46 -07:00
Steve Dower 2438cdf0e9
bpo-36085: Enable better DLL resolution on Windows (GH-12302) 2019-03-29 16:37:16 -07:00
Eric Snow 8479a3426e
bpo-33608: Make sure locks in the runtime are properly re-created. (gh-12245) 2019-03-08 23:44:33 -07:00
Serhiy Storchaka a24107b04c
bpo-35459: Use PyDict_GetItemWithError() instead of PyDict_GetItem(). (GH-11112) 2019-02-25 17:59:46 +02:00
Pablo Galindo 09fbcd6085
bpo-35942: Improve the error message if __fspath__ returns invalid types in path_converter (GH-11831)
The error message emitted when returning invalid types from __fspath__ in interfaces that allow passing PathLike objects has been improved and now it does explain the origin of the error.
2019-02-18 10:46:34 +00:00
Anthony Sottile 73d600239b Remove stray quote in os.replace docstring. (GH-11556) 2019-02-12 20:15:54 -08:00
Mark Becwar b82bfac436 bpo-29734: nt._getfinalpathname handle leak (GH-740)
Make sure that failure paths call CloseHandle outside of the function that failed
2019-02-02 13:08:23 -08:00
Tony Roberts 4860f01ac0 bpo-33895: Relase GIL while calling functions that acquire Windows loader lock (GH-7789)
LoadLibrary, GetProcAddress, FreeLibrary and GetModuleHandle acquire the system loader lock. Calling these while holding the GIL will cause a deadlock on the rare occasion that another thread is detaching and needs to destroy its thread state at the same time.
2019-02-02 09:16:42 -08:00
Victor Stinner 325e4bac5a
bpo-35537: Fix function name in os.posix_spawnp() errors (GH-11719) 2019-02-01 15:47:24 +01:00
Joannah Nanjekye 80c5dfe74b bpo-35537: Add setsid parameter to os.posix_spawn() and os.posix_spawnp() (GH-11608) 2019-02-01 11:05:22 +01:00
Joannah Nanjekye 92b8322e7e bpo-35674: Add os.posix_spawnp() (GH-11554)
Add a new os.posix_spawnp() function.
2019-01-16 14:29:26 +01:00
Gregory P. Smith 1d300ce1d8
bpo-35214: Annotate posix calls for clang MSan. (#11389)
It doesn't know the details of a few less common libc functions.
2018-12-30 21:13:02 -08:00
Jakub Kulík 6f9bc72c79 bpo-35550: Fix incorrect Solaris define guards (GH-11275)
Python source code uses on several places ifdef sun or defined(sun) without the underscores, which is not standard compliant and shouldn't be used.

Defines should check for __sun instead. Reference: http://nadeausoftware.com/articles/2012/01/c_c_tip_how_use_compiler_predefined_macros_detect_operating_system#Solaris

https://bugs.python.org/issue35550
2018-12-30 18:16:40 -08:00
Serhiy Storchaka 4db62e1158
bpo-35490: Remove the DecodeFSDefault return converter in AC. (#11152) 2018-12-17 16:47:45 +02:00
Serhiy Storchaka afb3e71a17
bpo-35489: Use "const Py_UNICODE *" for the Py_UNICODE converter in AC. (GH-11150) 2018-12-14 11:19:51 +02:00
Serhiy Storchaka 6fef0f1a81
bpo-35445: Do not ignore memory errors when create posix.environ. (GH-11049) 2018-12-10 12:10:56 +02:00
Zackery Spytz 4c49da0cb7 bpo-35436: Add missing PyErr_NoMemory() calls and other minor bug fixes. (GH-11015)
Set MemoryError when appropriate, add missing failure checks,
and fix some potential leaks.
2018-12-07 12:11:30 +02:00
Serhiy Storchaka 32bc11c33c
bpo-35371: Fix possible crash in os.utime() on Windows. (GH-10844) 2018-12-01 14:30:20 +02:00
Serhiy Storchaka 62be74290a
bpo-33012: Fix invalid function cast warnings with gcc 8. (GH-6749)
Fix invalid function cast warnings with gcc 8
for method conventions different from METH_NOARGS, METH_O and
METH_VARARGS excluding Argument Clinic generated code.
2018-11-27 13:27:31 +02:00
Victor Stinner 9a0d7a7648
bpo-24658: os.read() reuses _PY_READ_MAX (GH-10657)
os_read_impl() now also truncates the size to _PY_READ_MAX
on macOS, to avoid to allocate a larger buffer even if _Py_read() is
limited to _PY_READ_MAX bytes (ex: INT_MAX on macOS).
2018-11-22 15:03:40 +01:00
Eddie Elizondo 474eedfb3d bpo-34784: Fix PyStructSequence_NewType with heap-allocated StructSequence (GH-9665) 2018-11-13 13:09:31 +01:00
Victor Stinner 621cebe81b
bpo-35081: Rename internal headers (GH-10275)
Rename Include/internal/ headers:

* pycore_hash.h -> pycore_pyhash.h
* pycore_lifecycle.h -> pycore_pylifecycle.h
* pycore_mem.h -> pycore_pymem.h
* pycore_state.h -> pycore_pystate.h

Add missing headers to Makefile.pre.in and PCbuild:

* pycore_condvar.h.
* pycore_hamt.h
* pycore_pyhash.h
2018-11-12 16:53:38 +01:00
BNMetrics b942707fc2 bpo-29341: Clarify that path-like objects are accepted in some os methods (GH-10101)
Some methods in the os module can accept path-like objects. This is documented in the general documentation but not in the function docstrings. To keep both in sync, the docstrings need to be updated to reflect that path-like objects are also accepted.
2018-11-02 11:20:19 -04:00
Victor Stinner 27e2d1f219
bpo-35081: Add pycore_ prefix to internal header files (GH-10263)
* Rename Include/internal/ header files:

  * pyatomic.h -> pycore_atomic.h
  * ceval.h -> pycore_ceval.h
  * condvar.h -> pycore_condvar.h
  * context.h -> pycore_context.h
  * pygetopt.h -> pycore_getopt.h
  * gil.h -> pycore_gil.h
  * hamt.h -> pycore_hamt.h
  * hash.h -> pycore_hash.h
  * mem.h -> pycore_mem.h
  * pystate.h -> pycore_state.h
  * warnings.h -> pycore_warnings.h

* PCbuild project, Makefile.pre.in, Modules/Setup: add the
  Include/internal/ directory to the search paths of header files.
* Update includes. For example, replace #include "internal/mem.h"
  with #include "pycore_mem.h".
2018-11-01 00:52:28 +01:00
Alexey Izbyshev 834603112e bpo-32890, os: Use errno instead of GetLastError() in execve() and truncate() (GH-5784)
path_error() uses GetLastError() on Windows, but some os functions
are implemented via CRT APIs which report errors via errno.
This may result in raising OSError with invalid error code (such
as zero).

Introduce posix_path_error() function and use it where appropriate.
2018-10-20 02:28:22 +02:00
William Orr 81574b80e9 closes bpo-34862: Guard definition of convert_sched_param with POSIX_SPAWN_SETSCHEDULER. (GH-9658)
Fixes broken build on OpenBSD-current.
2018-10-01 22:19:56 -07:00
Joe Pamer c8c0249c9e bpo-32557: allow shutil.disk_usage to take a file path on Windows also (GH-9372)
https://bugs.python.org/issue32557
2018-09-25 07:57:36 -07:00
Serhiy Storchaka 0185f34ddc
bpo-33721: Make some os.path functions and pathlib.Path methods be tolerant to invalid paths. (#7695)
Such functions as os.path.exists(), os.path.lexists(), os.path.isdir(),
os.path.isfile(), os.path.islink(), and os.path.ismount() now return False
instead of raising ValueError or its subclasses UnicodeEncodeError
and UnicodeDecodeError for paths that contain characters or bytes
unrepresentative at the OS level.
2018-09-18 11:28:51 +03:00
Serhiy Storchaka 12a69db908
Convert os.readlink() to Argument Clinic. (GH-8778)
Also convert os.get_blocking() and os.set_blocking().
2018-09-17 15:38:27 +03:00
Eric Snow 5903296045
bpo-34651: Only allow the main interpreter to fork. (gh-9279)
When os.fork() is called (on platforms that support it) all threads but the current one are destroyed in the child process. Consequently we must ensure that all but the associated interpreter are likewise destroyed. The main interpreter is critical for runtime operation, so we must ensure that fork only happens in the main interpreter.

https://bugs.python.org/issue34651
2018-09-14 14:17:20 -07:00
Benjamin Peterson ea13740a37
bpo-34674: Assume unistd.h exists on Unix. (GH-9290) 2018-09-13 21:57:31 -07:00
Oren Milman 0bd1a2dcfd bpo-31577: Fix a crash in os.utime() in case of a bad ns argument. (GH-3752) 2018-09-12 22:14:35 +03:00
Benjamin Peterson e502451781
closes bpo-34646: Remove PyAPI_* macros from declarations. (GH-9218) 2018-09-12 12:06:42 -07:00
Serhiy Storchaka d700f97b62
bpo-20104: Change the file_actions parameter of os.posix_spawn(). (GH-6725)
* Make its default value an empty tuple instead of None.
* Make it a keyword-only parameter.
2018-09-08 14:48:18 +03:00
Pablo Galindo 254a4663d8
bpo-20104: Add flag capabilities to posix_spawn (GH-6693)
Implement the "attributes objects" parameter of `os.posix_spawn` to complete the implementation and fully cover the underlying API.
2018-09-07 16:44:24 +01:00
Berker Peksag e0b5b2096e
bpo-34384: Fix os.readlink() on Windows (GH-8740)
os.readlink() now accepts path-like and bytes objects on Windows.
Previously, support for path-like and bytes objects was only
implemented on Unix.

This commit also merges Unix and Windows implementations of
os.readlink() in one function and adds basic unit tests to increase
test coverage of the function.
2018-08-15 13:03:41 +03:00
Victor Stinner caba55b3b7
bpo-34301: Add _PyInterpreterState_Get() helper function (GH-8592)
sys_setcheckinterval() now uses a local variable to parse arguments,
before writing into interp->check_interval.
2018-08-03 15:33:52 +02:00
Serhiy Storchaka 9d5727326a
bpo-33871: Fix os.sendfile(), os.writev(), os.readv(), etc. (GH-7931)
* Fix integer overflow in os.readv(), os.writev(), os.preadv()
  and os.pwritev() and in os.sendfile() with headers or trailers
  arguments (on BSD-based OSes and MacOS).

* Fix sending the part of the file in os.sendfile() on MacOS.
  Using the trailers argument could cause sending more bytes from
  the input file than was specified.

Thanks Ned Deily for testing on 32-bit MacOS.
2018-07-31 10:24:54 +03:00
Giampaolo Rodola c7f02a9659
bpo-33671 / shutil.copyfile: use memoryview() with dynamic size on Windows (#7681)
bpo-33671
* use memoryview() with size == file size on Windows, see https://github.com/python/cpython/pull/7160#discussion_r195405230
* release intermediate (sliced) memoryview immediately
* replace "OSX" occurrences with "macOS"
* add some unittests for copyfileobj()
2018-06-19 08:27:29 -07:00
Pablo Galindo cb970730e3 bpo-33630: Fix using of freed memory in old versions of glicb for posix_spawn(). (GH-7685) 2018-06-19 11:19:50 +03:00
Giampaolo Rodola 4a172ccc73
bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) (#7160)
* have shutil.copyfileobj use sendfile() if possible

* refactoring: use ctx manager

* add test with non-regular file obj

* emulate case where file size can't be determined

* reference _copyfileobj_sendfile directly

* add test for offset() at certain position

* add test for empty file

* add test for non regular file dst

* small refactoring

* leave copyfileobj() alone in order to not introduce any incompatibility

* minor refactoring

* remove old test

* update docstring

* update docstring; rename exception class

* detect platforms which only support file to socket zero copy

* don't run test on platforms where file-to-file zero copy is not supported

* use tempfiles

* reset verbosity

* add test for smaller chunks

* add big file size test

* add comment

* update doc

* update whatsnew doc

* update doc

* catch Exception

* remove unused import

* add test case for error on second sendfile() call

* turn docstring into comment

* add one more test

* update comment

* add Misc/NEWS entry

* get rid of COPY_BUFSIZE; it belongs to another PR

* update doc

* expose posix._fcopyfile() for OSX

* merge from linux branch

* merge from linux branch

* expose fcopyfile

* arg clinic for the win implementation

* convert path type to path_t

* expose CopyFileW

* fix windows tests

* release GIL

* minor refactoring

* update doc

* update comment

* update docstrings

* rename functions

* rename test classes

* update doc

* update doc

* update docstrings and comments

* avoid do import nt|posix modules if unnecessary

* set nt|posix modules to None if not available

* micro speedup

* update description

* add doc note

* use better wording in doc

* rename function using 'fastcopy' prefix instead of 'zerocopy'

* use :ref: in rst doc

* change wording in doc

* add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies

* move CopyFileW in _winapi and actually expose CopyFileExW instead

* fix line endings

* add tests for mode bits

* add docstring

* remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies

* update doc to reflect new changes

* update doc

* adjust tests on win

* fix argument clinic error

* update doc

* OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python

* osx / copyfile: use path_t instead of char

* do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux)

* add same file test

* add test for same file

* have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted

* turn PermissionError into appropriate SameFileError

* expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError

* honour follow_symlinks arg when using CopyFileEx

* update Misc/NEWS

* expose CreateDirectoryEx mock

* change C type

* CreateDirectoryExW actual implementation

* provide specific makedirs() implementation for win

* fix typo

* skeleton for SetNamedSecurityInfo

* get security info for src path

* finally set security attrs

* add unit tests

* mimick os.makedirs() behavior and raise if dst dir exists

* set 2 paths for OSError object

* set 2 paths for OSError object

* expand windows test

* in case of exception on os.sendfile() set filename and filename2 exception attributes

* set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX

* update doc

* do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation

* use bytearray() and readinto()

* use memoryview() with bytearray()

* refactoring + introduce a new _fastcopy_binfileobj() fun

* remove CopyFileEx and other C wrappers

* remove code related to CopyFileEx

* Recognize binary files in copyfileobj()
...and use fastest _fastcopy_binfileobj() when possible

* set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable

* use ctx manager for memoryview()

* update doc

* remove outdated doc

* remove last CopyFileEx remnants

* OSX - use fcopyfile(3) instead of copyfile(3)

...as an extra safety measure: in case src/dst are "exotic" files (non
regular or living on a network fs etc.) we better fail on open() instead
of copyfile(3) as we're not quite sure what's gonna happen in that
case.

* update doc
2018-06-12 23:04:50 +02:00
Zackery Spytz 69dccc397a Remove unneeded PyErr_Occurred() check in os_lseek_impl() (GH-7557)
This call became unneeded after the posix module was converted to the
Argument Clinic in 2f93635d34 and should
have been removed as part of that change.
2018-06-10 10:28:01 +03:00
Serhiy Storchaka d54cfb160c
bpo-33441: Make the sigset_t converter available in other modules. (GH-6720)
* Expose the sigset_t converter via private API _Py_Sigset_Converter().
* Use Argument Clinic for parsing sigset_t in signalmodule.c.
* Raise ValueError instead OverflowError for integers out of
  the C long range.

Based on patch by Pablo Galindo Salgado.
2018-05-08 07:48:50 +03:00
Serhiy Storchaka ef347535f2
bpo-20104: Improve error handling and fix a reference leak in os.posix_spawn(). (#6332) 2018-05-01 16:45:04 +03:00
Alexey Izbyshev 3b20d3454e bpo-33016: Fix potential use of uninitialized memory in nt._getfinalpathname (#6010) 2018-03-08 08:03:25 -08:00
Steve Dower 6921e73e33
bpo-33001: Prevent buffer overrun in os.symlink (GH-5989) 2018-03-05 14:26:08 -08:00
Alexey Izbyshev 3e197c7a67 bpo-32903: Fix a memory leak in os.chdir() on Windows (GH-5801) 2018-03-01 17:13:56 +08:00
Steve Dower 23ad6d0d1a
bpo-32556: nt._getfinalpathname, nt._getvolumepathname and nt._getdiskusage now correctly convert from bytes. (GH-5761) 2018-02-22 10:39:10 -08:00
Alexey Izbyshev b3caf388a0 closes bpo-32859: Don't retry dup3() if it is not available at runtime (GH-5708)
os.dup2() tests for dup3() system call availability at runtime,
but doesn't remember the result across calls, repeating
the test on each call with inheritable=False.

Since the caller of os.dup2() is expected to hold the GIL,
fix this by making the variable holding the test result static.
2018-02-19 23:25:46 -08:00
ngie-eign 7745ec4e35 Update comment in posixmodule.c (GH-5681)
A closing parentheses was missing.

Signed-off-by: Ngie Cooper <yaneurabeya@gmail.com>
2018-02-14 11:54:28 -08:00
SSE4 3c34aad4e7 bpo-29248: Fix os.readlink() on Windows (GH-5577)
The PrintNameOffset field of the reparse data buffer
was treated as a number of characters instead of bytes.
2018-02-12 20:10:35 +03:00
oldk aa0735f597 bpo-32747: Remove trailing spaces in docstrings. (GH-5491) 2018-02-02 10:52:55 +02:00
Stéphane Wirtel 3d86e484de bpo-32681: Fix an uninitialized variable in the C implementation of os.dup2 (GH-5346)
See https://bugs.python.org/issue32441 for where this was introduced.
2018-01-29 22:04:36 -08:00
Pablo Galindo 0cd6bca655 bpo-20104: Fix leaks and errors in new os.posix_spawn (GH-5418)
* Fix memory leaks and error handling in posix spawn
* Improve error handling when destroying the file_actions object
* Py_DECREF the result of PySequence_Fast on error
* Handle uninitialized pid
* Use OSError if file actions fails to initialize
* Move _file_actions to outer scope to avoid undefined behaviour
* Remove HAVE_POSIX_SPAWN define in Modules/posixmodule.c
* Unshadow exception and clean error message
2018-01-29 12:34:42 -08:00
Chih-Hsuan Yen 8997f9cd1a bpo-32705: Current Android does not have posix_spawn (#5413) 2018-01-29 10:50:35 +01:00
Pablo Galindo 6c6ddf97c4 bpo-20104: Expose `posix_spawn` in the os module (GH-5109)
Add os.posix_spawn to wrap the low level POSIX API of the same name.

Contributed by Pablo Galindo.
2018-01-28 17:56:10 -08:00
jcea 6c51d51880
bpo-32659: Solaris "stat" should support "st_fstype" (#5307)
* bpo-32659: Solaris "stat" should support "st_fstype"

* Add 'versionadded'
2018-01-28 14:00:08 +01:00
Pablo Galindo 4defba3b95 bpo-31368: Expose preadv and pwritev in the os module (#5239) 2018-01-27 17:16:37 +01:00
Michael Felt 502d551c6d bpo-32390: Fix compilation failure on AIX after f_fsid was added to os.statvfs() (#4972) 2018-01-05 13:01:58 +01:00
Benjamin Peterson bbdb17d19b
return the new file descriptor from os.dup2 (closes bpo-32441) (#5041) 2017-12-29 13:13:06 -08:00
Michael Felt c5ae169e1b bpo-26439 Fix ctypes.util.find_library failure on AIX (#4507)
Implement find_library() support in ctypes/util for AIX.

Add some AIX specific tests.
2017-12-19 13:58:49 +01:00
Giuseppe Scrivano 96a5e50a5d bpo-32143: add f_fsid to os.statvfs() (#4571)
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2017-12-14 17:46:46 -05:00
Anthony Sottile 233ef249cc bpo-32277: Fix exception raised from chmod(..., follow_symlinks=False) (#4797) 2017-12-14 18:57:55 +02:00
Erik Bray 03eb11f0b3 bpo-16135: Cleanup: Code rot left over from OS/2 support (GH-4147)
Remove dangling references to PYCC_VACPP that are not relelvant
since removal of OS/2 support.
2017-10-27 05:27:06 -07:00
Victor Stinner 850a18e03e bpo-30768: Recompute timeout on interrupted lock (GH-4103)
Fix the pthread+semaphore implementation of
PyThread_acquire_lock_timed() when called with timeout > 0 and
intr_flag=0: recompute the timeout if sem_timedwait() is interrupted
by a signal (EINTR).

See also the PEP 475.

The pthread implementation of PyThread_acquire_lock() now fails with
a fatal error if the timeout is larger than PY_TIMEOUT_MAX, as done
in the Windows implementation.

The check prevents any risk of overflow in PyThread_acquire_lock().

Add also PY_DWORD_MAX constant.
2017-10-24 16:53:32 -07:00
Victor Stinner 01b5aab7bf bpo-31827: Remove os.stat_float_times() (GH-4061) 2017-10-24 02:02:00 -07:00
Victor Stinner a99121526a os.startfile(): add a C comment on security (#3877)
LoadLibrary("SHELL32") is not vulnerable to DLL hijacking.
2017-10-13 13:46:57 -07:00
Masayuki Yamamoto 731e189014 bpo-25658: Implement PEP 539 for Thread Specific Storage (TSS) API (GH-1362)
See PEP 539 for details.

Highlights of changes:

- Add Thread Specific Storage (TSS) API
- Document the Thread Local Storage (TLS) API as deprecated
- Update code that used TLS API to use TSS API
2017-10-06 20:41:34 +10:00
Benjamin Peterson 288d1daada remove support for BSD/OS (closes bpo-31624) (#3812) 2017-09-28 22:44:27 -07:00
Antoine Pitrou a6a4dc816d bpo-31370: Remove support for threads-less builds (#3385)
* Remove Setup.config
* Always define WITH_THREAD for compatibility.
2017-09-07 18:56:24 +02:00
Christian Heimes 75b961869a bpo-31343: Include sys/sysmacros.h (#3318)
Include sys/sysmacros.h for major(), minor(), and makedev(). GNU C libray
plans to remove the functions from sys/types.h.

Signed-off-by: Christian Heimes <christian@python.org>
2017-09-05 15:53:09 +02:00
Christopher Wilcox c67bae0478 bpo-30581: Windows: os.cpu_count() returns wrong number of processors (#2934)
* Fixes #30581 by adding a path to use newer GetMaximumProcessorCount API on Windows calls to os.cpu_count()

* Add NEWS.d entry for bpo-30581, os.cpu_count on Windows.

* Tweak NEWS entry
2017-08-30 11:01:08 +02:00
Коренберг Марк d4b93e21c2 bpo-31106: Fix handling of erros in posix_fallocate() and posix_fadvise() (#3000) (#3000) 2017-08-14 06:55:16 -07:00
Ville Skyttä 49b2734bf1 Spelling fixes (#2902) 2017-08-03 09:00:59 +03:00
Serhiy Storchaka 1180e5a518 bpo-30879: os.listdir() and os.scandir() now emit bytes names when (#2634)
called with bytes-like argument.
2017-07-11 06:36:46 +03:00
Serhiy Storchaka f7eae0adfc [security] bpo-13617: Reject embedded null characters in wchar* strings. (#2302)
Based on patch by Victor Stinner.

Add private C API function _PyUnicode_AsUnicode() which is similar to
PyUnicode_AsUnicode(), but checks for null characters.
2017-06-28 08:30:06 +03:00
Eric N. Vander Weele a7874c73c0 bpo-30769: Fix reference leak introduced in 77703942c5 (#2416)
New error condition paths were introduced, which did not decrement
`key2` and `val2` objects.  Therefore, decrement references before
jumping to the error label.

Signed-off-by: Eric N. Vander Weele <ericvw@gmail.com>
2017-06-27 03:35:20 +02:00
Serhiy Storchaka 77703942c5 bpo-30746: Prohibited the '=' character in environment variable names (#2382)
in `os.putenv()` and `os.spawn*()`.
2017-06-25 07:33:01 +03:00