bpo-37834: Normalise handling of reparse points on Windows
* ntpath.realpath() and nt.stat() will traverse all supported reparse points (previously was mixed)
* nt.lstat() will let the OS traverse reparse points that are not name surrogates (previously would not traverse any reparse point)
* nt.[l]stat() will only set S_IFLNK for symlinks (previous behaviour)
* nt.readlink() will read destinations for symlinks and junction points only
bpo-1311: os.path.exists('nul') now returns True on Windows
* nt.stat('nul').st_mode is now S_IFCHR (previously was an error)
In a subinterpreter, spawning a daemon thread now raises an
exception. Daemon threads were never supported in subinterpreters.
Previously, the subinterpreter finalization crashed with a Pyton
fatal error if a daemon thread was still running.
* Add _thread._is_main_interpreter()
* threading.Thread.start() now raises RuntimeError if the thread is a
daemon thread and the method is called from a subinterpreter.
* The _thread module now uses Argument Clinic for the new function.
* Use textwrap.dedent() in test_threading.SubinterpThreadingTests
* bpo-26836: Add os.memfd_create()
* Use the glibc wrapper for memfd_create()
Co-Authored-By: Christian Heimes <christian@python.org>
* Fix deletions caused by autoreconf.
* Use MFD_CLOEXEC as the default value for *flags*.
* Add memset_s to configure.ac.
* Revert memset_s changes.
* Apply the requested changes.
* Tweak the docs.
* bpo-22385: Support output separators in hex methods.
Also in binascii.hexlify aka b2a_hex.
The underlying implementation behind all hex generation in CPython uses the
same pystrhex.c implementation. This adds support to bytes, bytearray,
and memoryview objects.
The binascii module functions exist rather than being slated for deprecation
because they return bytes rather than requiring an intermediate step through a
str object.
This change was inspired by MicroPython which supports sep in its binascii
implementation (and does not yet support the .hex methods).
https://bugs.python.org/issue22385
* Add math.isqrt function computing the integer square root.
* Code cleanup: remove redundant comments, rename some variables.
* Tighten up code a bit more; use Py_XDECREF to simplify error handling.
* Update Modules/mathmodule.c
Co-Authored-By: Serhiy Storchaka <storchaka@gmail.com>
* Update Modules/mathmodule.c
Use real argument clinic type instead of an alias
Co-Authored-By: Serhiy Storchaka <storchaka@gmail.com>
* Add proof sketch
* Updates from review.
* Correct and expand documentation.
* Fix bad reference handling on error; make some variables block-local; other tidying.
* Style and consistency fixes.
* Add missing error check; don't try to DECREF a NULL a
* Simplify some error returns.
* Another two test cases:
- clarify that floats are rejected even if they happen to be
squares of small integers
- TypeError beats ValueError for a negative float
* Documentation and markup improvements; thanks Serhiy for the suggestions!
* Cleaner Misc/NEWS entry wording.
* Clean up (with one fix) to the algorithm explanation and proof.
* bpo-36673: Implement comment/PI parsing support for the TreeBuilder in ElementTree.
* bpo-36673: Rewrite the comment/PI factory handling for the TreeBuilder in "_elementtree" to make it use the same factories as the ElementTree module, and to make it explicit when the comments/PIs are inserted into the tree and when they are not (which is the default).
* Added tests for shared_memory submodule.
* Added tests for ShareableList.
* Fix bug in allocationn size during creation of empty ShareableList illuminated by existing test run on Linux.
* Initial set of docs for shared_memory module.
* Added docs for ShareableList, added doctree entry for shared_memory submodule, name refactoring for greater clarity.
* Added examples to SharedMemoryManager docs, for ease of documentation switched away from exclusively registered functions to some explicit methods on SharedMemoryManager.
* Wording tweaks to docs.
* Fix test failures on Windows.
* Added tests around SharedMemoryManager.
* Documentation tweaks.
* Fix inappropriate test on Windows.
* Further documentation tweaks.
* Fix bare exception.
* Removed __copyright__.
* Fixed typo in doc, removed comment.
* Updated SharedMemoryManager preliminary tests to reflect change of not supporting all registered functions on SyncManager.
* Added Sphinx doctest run controls.
* CloseHandle should be in a finally block in case MapViewOfFile fails.
* Missed opportunity to use with statement.
* Switch to self.addCleanup to spare long try/finally blocks and save one indentation, change to use decorator to skip test instead.
* Simplify the posixshmem extension module.
Provide shm_open() and shm_unlink() functions. Move other
functionality into the shared_memory.py module.
* Added to doc around size parameter of SharedMemory.
* Changed PosixSharedMemory.size to use os.fstat.
* Change SharedMemory.buf to a read-only property as well as NamedSharedMemory.size.
* Marked as provisional per PEP411 in docstring.
* Changed SharedMemoryTracker to be private.
* Removed registered Proxy Objects from SharedMemoryManager.
* Removed shareable_wrap().
* Removed shareable_wrap() and dangling references to it.
* For consistency added __reduce__ to key classes.
* Fix for potential race condition on Windows for O_CREX.
* Remove unused imports.
* Update access to kernel32 on Windows per feedback from eryksun.
* Moved kernel32 calls to _winapi.
* Removed ShareableList.copy as redundant.
* Changes to _winapi use from eryksun feedback.
* Adopt simpler SharedMemory API, collapsing PosixSharedMemory and WindowsNamedSharedMemory into one.
* Fix missing docstring on class, add test for ignoring size when attaching.
* Moved SharedMemoryManager to managers module, tweak to fragile test.
* Tweak to exception in OpenFileMapping suggested by eryksun.
* Mark a few dangling bits as private as suggested by Giampaolo.
Use _PyArg_CheckPositional() and inlined code instead of
PyArg_UnpackTuple() and _PyArg_UnpackStack() if all parameters
are positional and use the "object" converter.
As in title, expose C `raise` function as `raise_function` in `signal` module. Also drop existing `raise_signal` in `_testcapi` module and replace all usages with new function.
https://bugs.python.org/issue35568
* bpo-32492: 2.5x speed up in namedtuple attribute access using C fast path
* Add News entry
* fixup! bpo-32492: 2.5x speed up in namedtuple attribute access using C fast path
* Check for tuple in the __get__ of the new descriptor and don't cache the descriptor itself
* Don't inherit from property. Implement GC methods to handle __doc__
* Add a test for the docstring substitution in descriptors
* Update NEWS entry to reflect time against 3.7 branch
* Simplify implementation with argument clinic, better error messages, only __new__
* Use positional-only parameters for the __new__
* Use PyTuple_GET_SIZE and PyTuple_GET_ITEM to tighter the implementation of tuplegetterdescr_get
* Implement __set__ to make tuplegetter a data descriptor
* Use Py_INCREF now that we inline PyTuple_GetItem
* Apply the valid_index() function, saving one test
* Move Py_None test out of the critical path.
Fix invalid function cast warnings with gcc 8
for method conventions different from METH_NOARGS, METH_O and
METH_VARARGS in Argument Clinic generated code.
Some methods in the os module can accept path-like objects. This is documented in the general documentation but not in the function docstrings. To keep both in sync, the docstrings need to be updated to reflect that path-like objects are also accepted.
Add SSLContext.post_handshake_auth and
SSLSocket.verify_client_post_handshake for TLS 1.3 post-handshake
authentication.
Signed-off-by: Christian Heimes <christian@python.org>q
https://bugs.python.org/issue34670
Such functions as os.path.exists(), os.path.lexists(), os.path.isdir(),
os.path.isfile(), os.path.islink(), and os.path.ismount() now return False
instead of raising ValueError or its subclasses UnicodeEncodeError
and UnicodeDecodeError for paths that contain characters or bytes
unrepresentative at the OS level.
* help(hashlib) didn't work because of incorrect module name in blake2b and
blake2s classes.
* Constructors blake2*(), sha3_*(), shake_*() and keccak_*() incorrectly
accepted keyword argument "string" for binary data, but documented as
accepting the "data" keyword argument. Now this parameter is positional-only.
* Keyword-only parameters in blake2b() and blake2s() were not documented as
keyword-only.
* Default value for some parameters of blake2b() and blake2s() was None,
which is not acceptable value.
* The length argument for shake_*.digest() was wrapped out to 32 bits.
* The argument for shake_128.digest() and shake_128.hexdigest() was not
positional-only as intended.
* TypeError messages for incorrect arguments in all constructors sha3_*(),
shake_*() and keccak_*() incorrectly referred to sha3_224.
Also made the following enhancements:
* More accurately specified input and result types for strings, bytes and
bytes-like objects.
* Unified positional parameter names for update() and constructors.
* Improved formatting.
bpo-33671
* use memoryview() with size == file size on Windows, see https://github.com/python/cpython/pull/7160#discussion_r195405230
* release intermediate (sliced) memoryview immediately
* replace "OSX" occurrences with "macOS"
* add some unittests for copyfileobj()
* have shutil.copyfileobj use sendfile() if possible
* refactoring: use ctx manager
* add test with non-regular file obj
* emulate case where file size can't be determined
* reference _copyfileobj_sendfile directly
* add test for offset() at certain position
* add test for empty file
* add test for non regular file dst
* small refactoring
* leave copyfileobj() alone in order to not introduce any incompatibility
* minor refactoring
* remove old test
* update docstring
* update docstring; rename exception class
* detect platforms which only support file to socket zero copy
* don't run test on platforms where file-to-file zero copy is not supported
* use tempfiles
* reset verbosity
* add test for smaller chunks
* add big file size test
* add comment
* update doc
* update whatsnew doc
* update doc
* catch Exception
* remove unused import
* add test case for error on second sendfile() call
* turn docstring into comment
* add one more test
* update comment
* add Misc/NEWS entry
* get rid of COPY_BUFSIZE; it belongs to another PR
* update doc
* expose posix._fcopyfile() for OSX
* merge from linux branch
* merge from linux branch
* expose fcopyfile
* arg clinic for the win implementation
* convert path type to path_t
* expose CopyFileW
* fix windows tests
* release GIL
* minor refactoring
* update doc
* update comment
* update docstrings
* rename functions
* rename test classes
* update doc
* update doc
* update docstrings and comments
* avoid do import nt|posix modules if unnecessary
* set nt|posix modules to None if not available
* micro speedup
* update description
* add doc note
* use better wording in doc
* rename function using 'fastcopy' prefix instead of 'zerocopy'
* use :ref: in rst doc
* change wording in doc
* add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies
* move CopyFileW in _winapi and actually expose CopyFileExW instead
* fix line endings
* add tests for mode bits
* add docstring
* remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies
* update doc to reflect new changes
* update doc
* adjust tests on win
* fix argument clinic error
* update doc
* OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python
* osx / copyfile: use path_t instead of char
* do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux)
* add same file test
* add test for same file
* have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted
* turn PermissionError into appropriate SameFileError
* expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError
* honour follow_symlinks arg when using CopyFileEx
* update Misc/NEWS
* expose CreateDirectoryEx mock
* change C type
* CreateDirectoryExW actual implementation
* provide specific makedirs() implementation for win
* fix typo
* skeleton for SetNamedSecurityInfo
* get security info for src path
* finally set security attrs
* add unit tests
* mimick os.makedirs() behavior and raise if dst dir exists
* set 2 paths for OSError object
* set 2 paths for OSError object
* expand windows test
* in case of exception on os.sendfile() set filename and filename2 exception attributes
* set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX
* update doc
* do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation
* use bytearray() and readinto()
* use memoryview() with bytearray()
* refactoring + introduce a new _fastcopy_binfileobj() fun
* remove CopyFileEx and other C wrappers
* remove code related to CopyFileEx
* Recognize binary files in copyfileobj()
...and use fastest _fastcopy_binfileobj() when possible
* set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable
* use ctx manager for memoryview()
* update doc
* remove outdated doc
* remove last CopyFileEx remnants
* OSX - use fcopyfile(3) instead of copyfile(3)
...as an extra safety measure: in case src/dst are "exotic" files (non
regular or living on a network fs etc.) we better fail on open() instead
of copyfile(3) as we're not quite sure what's gonna happen in that
case.
* update doc
* Expose the sigset_t converter via private API _Py_Sigset_Converter().
* Use Argument Clinic for parsing sigset_t in signalmodule.c.
* Raise ValueError instead OverflowError for integers out of
the C long range.
Based on patch by Pablo Galindo Salgado.
This makes performance better and produces shorter pickles. This change is backwards compatible up to the oldest currently supported version of Python (3.4).
The ssl module now detects missing NPN support in LibreSSL.
Co-Authored-By: Bernard Spil <brnrd@FreeBSD.org>
Signed-off-by: Christian Heimes <christian@python.org>
* The SSLSocket is no longer implemented on top of SSLObject to
avoid an extra level of indirection.
* Owner and session are now handled in the internal constructor.
* _ssl._SSLSocket now uses the same method names as SSLSocket and
SSLObject.
* Channel binding type check is now handled in C code. Channel binding
is always available.
The patch also changes the signature of SSLObject.__init__(). In my
opinion it's fine. A SSLObject is not a user-constructable object.
SSLContext.wrap_bio() is the only valid factory.
Previously, the ssl module stored international domain names (IDNs)
as U-labels. This is problematic for a number of reasons -- for
example, it made it impossible for users to use a different version
of IDNA than the one built into Python.
After this change, we always convert to A-labels as soon as possible,
and use them for all internal processing. In particular, server_hostname
attribute is now an A-label, and on the server side there's a new
sni_callback that receives the SNI servername as an A-label rather than
a U-label.
The hmac module now has hmac.digest(), which provides an optimized HMAC
digest for short messages. hmac.digest() is up to three times faster
than hmac.HMAC().digest().
Signed-off-by: Christian Heimes <christian@python.org>
Specifically, it's not possible to subclass Task/Future classes
and override the following methods:
* Future._schedule_callbacks
* Task._step
* Task._wakeup
* Use wider types (int => Py_ssize_t) to avoid integer overflows.
* Fix gc.get_freeze_count(): use Py_ssize_t type rather than int, since gc_list_size() returns a Py_ssize_t.
Even though Python marks any handles it opens as non-inheritable there
is still a race when using `subprocess.Popen` since creating a process
with redirected stdio requires temporarily creating inheritable handles.
By implementing support for `subprocess.Popen(close_fds=True)` we fix
this race.
In order to implement this we use PROC_THREAD_ATTRIBUTE_HANDLE_LIST
which is available since Windows Vista. Which allows to pass an explicit
list of handles to inherit when creating a process.
This commit also adds `STARTUPINFO.lpAttributeList["handle_list"]`
which can be used to control PROC_THREAD_ATTRIBUTE_HANDLE_LIST
directly.
asyncio.get_event_loop(), and, subsequently asyncio._get_running_loop()
are one of the most frequently executed functions in asyncio. They also
can't be sped up by third-party event loops like uvloop.
When implemented in C they become 4x faster.
* Add -X utf8 command line option, PYTHONUTF8 environment variable
and a new sys.flags.utf8_mode flag.
* If the LC_CTYPE locale is "C" at startup: enable automatically the
UTF-8 mode.
* Add _winapi.GetACP(). encodings._alias_mbcs() now calls
_winapi.GetACP() to get the ANSI code page
* locale.getpreferredencoding() now returns 'UTF-8' in the UTF-8
mode. As a side effect, open() now uses the UTF-8 encoding by
default in this mode.
* Py_DecodeLocale() and Py_EncodeLocale() now use the UTF-8 encoding
in the UTF-8 Mode.
* Update subprocess._args_from_interpreter_flags() to handle -X utf8
* Skip some tests relying on the current locale if the UTF-8 mode is
enabled.
* Add test_utf8mode.py.
* _Py_DecodeUTF8_surrogateescape() gets a new optional parameter to
return also the length (number of wide characters).
* pymain_get_global_config() and pymain_set_global_config() now
always copy flag values, rather than only copying if the new value
is greater than the old value.
Parse more env vars in Py_Main():
* Add more options to _PyCoreConfig:
* faulthandler
* tracemalloc
* importtime
* Move code to parse environment variables from _Py_InitializeCore()
to Py_Main(). This change fixes a regression from Python 3.6:
PYTHONUNBUFFERED is now read before calling pymain_init_stdio().
* _PyFaulthandler_Init() and _PyTraceMalloc_Init() now take an
argument to decide if the module has to be enabled at startup.
* tracemalloc_start() is now responsible to check the maximum number
of frames.
Other changes:
* Cleanup Py_Main():
* Rename some pymain_xxx() subfunctions
* Add pymain_run_python() subfunction
* Cleanup Py_NewInterpreter()
* _PyInterpreterState_Enable() now reports failure
* init_hash_secret() now considers pyurandom() failure as an "user
error": don't fail with abort().
* pymain_optlist_append() and pymain_strdup() now sets err on memory
allocation failure.
When a single .c file contains several functions and/or methods with
the same name, a safety _METHODDEF #define statement is generated
only for one of them.
This fixes the bug by using the full name of the function to avoid
duplicates rather than just the name.
Fix the pthread+semaphore implementation of
PyThread_acquire_lock_timed() when called with timeout > 0 and
intr_flag=0: recompute the timeout if sem_timedwait() is interrupted
by a signal (EINTR).
See also the PEP 475.
The pthread implementation of PyThread_acquire_lock() now fails with
a fatal error if the timeout is larger than PY_TIMEOUT_MAX, as done
in the Windows implementation.
The check prevents any risk of overflow in PyThread_acquire_lock().
Add also PY_DWORD_MAX constant.
Freeze all the objects tracked by gc - move them to a permanent generation
and ignore all the future collections. This can be used before a POSIX
fork() call to make the gc copy-on-write friendly or to speed up collection.
* Change NPN detection:
Version breakdown, support disabled (pre-patch/post-patch):
- pre-1.0.1: OPENSSL_NPN_NEGOTIATED will not be defined -> False/False
- 1.0.1 and 1.0.2: OPENSSL_NPN_NEGOTIATED will not be defined ->
False/False
- 1.1.0+: OPENSSL_NPN_NEGOTIATED will be defined and
OPENSSL_NO_NEXTPROTONEG will be defined -> True/False
Version breakdown support enabled (pre-patch/post-patch):
- pre-1.0.1: OPENSSL_NPN_NEGOTIATED will not be defined -> False/False
- 1.0.1 and 1.0.2: OPENSSL_NPN_NEGOTIATED will be defined and
OPENSSL_NO_NEXTPROTONEG will not be defined -> True/True
- 1.1.0+: OPENSSL_NPN_NEGOTIATED will be defined and
OPENSSL_NO_NEXTPROTONEG will not be defined -> True/True
* Refine NPN guard:
- If NPN is disabled, but ALPN is available we need our callback
- Make clinic's ssl behave the same way
This created a working ssl module for me, with NPN disabled and ALPN
enabled for OpenSSL 1.1.0f.
Concerns to address:
The initial commit for NPN support into OpenSSL [1], had the
OPENSSL_NPN_* variables defined inside the OPENSSL_NO_NEXTPROTONEG
guard. The question is if that ever made it into a release.
This would need an ugly hack, something like:
#if defined(OPENSSL_NO_NEXTPROTONEG) && \
!defined(OPENSSL_NPN_NEGOTIATED)
# define OPENSSL_NPN_UNSUPPORTED 0
# define OPENSSL_NPN_NEGOTIATED 1
# define OPENSSL_NPN_NO_OVERLAP 2
#endif
[1] https://github.com/openssl/openssl/commit/68b33cc5c7
The function '_PyArg_ParseStack()' and
'_PyArg_UnpackStack' were failing (with error
"XXX() takes Y argument (Z given)") before
the function '_PyArg_NoStackKeywords()' was called.
Thus, the latter did not raise its more meaningful
error : "XXX() takes no keyword arguments".
* bpo-16500: Allow registering at-fork handlers
* Address Serhiy's comments
* Add doc for new C API
* Add doc for new Python-facing function
* Add NEWS entry + doc nit
Element.getiterator() and the html parameter of XMLParser() were
deprecated only in the documentation (since Python 3.2 and 3.4 correspondintly).
Now using them emits a deprecation warning.
* Don’t need check_warnings any more.
* bpo-6532: Make the thread id an unsigned integer.
From C API side the type of results of PyThread_start_new_thread() and
PyThread_get_thread_ident(), the id parameter of
PyThreadState_SetAsyncExc(), and the thread_id field of PyThreadState
changed from "long" to "unsigned long".
* Restore a check in thread_get_ident().
Issue #29300: Rename struct.unpack() second parameter from "inputstr" to
"buffer", and use the Py_buffer type.
Fix also unit tests on struct.unpack() which passed a Unicode string instead of
a bytes string as struct.unpack() second parameter. The purpose of
test_trailing_counter() is to test invalid format strings, not to test the
buffer parameter.
* The struct module now requires contiguous buffers.
* Convert most functions and methods of the _struct module to Argument Clinic
* Use "Py_buffer" type for the "buffer" argument. Argument Clinic is
responsible to create and release the Py_buffer object.
* Use "PyStructObject *" type for self to avoid explicit conversions.
* Add an unit test on the _struct.Struct.unpack_from() method to test passing
arguments as keywords.
* Rephrase docstrings.
* Rename "fmt" argument to "format" in docstrings and the documentation.
As a side effect, functions and methods which used METH_VARARGS calling
convention like struct.pack() now use the METH_FASTCALL calling convention
which avoids the creation of temporary tuple to pass positional arguments and
so is faster. For example, struct.pack("i", 1) becomes 1.56x faster (-36%)::
$ ./python -m perf timeit \
-s 'import struct; pack=struct.pack' 'pack("i", 1)' \
--compare-to=../default-ref/python
Median +- std dev: 119 ns +- 1 ns -> 76.8 ns +- 0.4 ns: 1.56x faster (-36%)
Significant (t=295.91)
Patch co-written with Serhiy Storchaka.
Christian Heimes
Serhiy Storchaka
Rémi Lapeyre
Raymond Hettinger
Dong-hee Na
Steve Dower
Greg Price
Min ho Kim
Victor Stinner
Yash Aggarwal
Pablo Galindo
Alex Grönholm
Zackery Spytz
Gregory P. Smith
Dino Viehland
Antoine Pitrou
pxinwr
Mark Dickinson
Stefan Behnel
Steve Dower
Inada Naoki
animalize
Davin Potts
Anthony Sottile
Joannah Nanjekye
Ammar Askar
Vladimir Matveev
Tal Einat
Max Bélanger
BNMetrics
Srinivas Thatiparthy (శ్రీనివాస్ తాటిపర్తి)
William Orr
Tim Hoffmann
Lisa Roach
William Grzybowski
Alexander Buchkovsky
Giampaolo Rodola
Antoine Pitrou
Łukasz Langa
Antoine Pietri
Xiang Zhang
Ivan Levkivskyi
Barry Warsaw
Yury Selivanov
Victor Stinner
Benjamin Peterson
Segev Finer
Andrew Svetlov
Tal Einat
brainfvck
Oren Milman
Zachary Ware
Melvyn Sopacua
Yaron de Leeuw
Sylvain
Chandan Kumar
Yury Selivanov