Even though Python marks any handles it opens as non-inheritable there
is still a race when using `subprocess.Popen` since creating a process
with redirected stdio requires temporarily creating inheritable handles.
By implementing support for `subprocess.Popen(close_fds=True)` we fix
this race.
In order to implement this we use PROC_THREAD_ATTRIBUTE_HANDLE_LIST
which is available since Windows Vista. Which allows to pass an explicit
list of handles to inherit when creating a process.
This commit also adds `STARTUPINFO.lpAttributeList["handle_list"]`
which can be used to control PROC_THREAD_ATTRIBUTE_HANDLE_LIST
directly.
asyncio.get_event_loop(), and, subsequently asyncio._get_running_loop()
are one of the most frequently executed functions in asyncio. They also
can't be sped up by third-party event loops like uvloop.
When implemented in C they become 4x faster.
* Add -X utf8 command line option, PYTHONUTF8 environment variable
and a new sys.flags.utf8_mode flag.
* If the LC_CTYPE locale is "C" at startup: enable automatically the
UTF-8 mode.
* Add _winapi.GetACP(). encodings._alias_mbcs() now calls
_winapi.GetACP() to get the ANSI code page
* locale.getpreferredencoding() now returns 'UTF-8' in the UTF-8
mode. As a side effect, open() now uses the UTF-8 encoding by
default in this mode.
* Py_DecodeLocale() and Py_EncodeLocale() now use the UTF-8 encoding
in the UTF-8 Mode.
* Update subprocess._args_from_interpreter_flags() to handle -X utf8
* Skip some tests relying on the current locale if the UTF-8 mode is
enabled.
* Add test_utf8mode.py.
* _Py_DecodeUTF8_surrogateescape() gets a new optional parameter to
return also the length (number of wide characters).
* pymain_get_global_config() and pymain_set_global_config() now
always copy flag values, rather than only copying if the new value
is greater than the old value.
Parse more env vars in Py_Main():
* Add more options to _PyCoreConfig:
* faulthandler
* tracemalloc
* importtime
* Move code to parse environment variables from _Py_InitializeCore()
to Py_Main(). This change fixes a regression from Python 3.6:
PYTHONUNBUFFERED is now read before calling pymain_init_stdio().
* _PyFaulthandler_Init() and _PyTraceMalloc_Init() now take an
argument to decide if the module has to be enabled at startup.
* tracemalloc_start() is now responsible to check the maximum number
of frames.
Other changes:
* Cleanup Py_Main():
* Rename some pymain_xxx() subfunctions
* Add pymain_run_python() subfunction
* Cleanup Py_NewInterpreter()
* _PyInterpreterState_Enable() now reports failure
* init_hash_secret() now considers pyurandom() failure as an "user
error": don't fail with abort().
* pymain_optlist_append() and pymain_strdup() now sets err on memory
allocation failure.
When a single .c file contains several functions and/or methods with
the same name, a safety _METHODDEF #define statement is generated
only for one of them.
This fixes the bug by using the full name of the function to avoid
duplicates rather than just the name.
Fix the pthread+semaphore implementation of
PyThread_acquire_lock_timed() when called with timeout > 0 and
intr_flag=0: recompute the timeout if sem_timedwait() is interrupted
by a signal (EINTR).
See also the PEP 475.
The pthread implementation of PyThread_acquire_lock() now fails with
a fatal error if the timeout is larger than PY_TIMEOUT_MAX, as done
in the Windows implementation.
The check prevents any risk of overflow in PyThread_acquire_lock().
Add also PY_DWORD_MAX constant.
Freeze all the objects tracked by gc - move them to a permanent generation
and ignore all the future collections. This can be used before a POSIX
fork() call to make the gc copy-on-write friendly or to speed up collection.
* Change NPN detection:
Version breakdown, support disabled (pre-patch/post-patch):
- pre-1.0.1: OPENSSL_NPN_NEGOTIATED will not be defined -> False/False
- 1.0.1 and 1.0.2: OPENSSL_NPN_NEGOTIATED will not be defined ->
False/False
- 1.1.0+: OPENSSL_NPN_NEGOTIATED will be defined and
OPENSSL_NO_NEXTPROTONEG will be defined -> True/False
Version breakdown support enabled (pre-patch/post-patch):
- pre-1.0.1: OPENSSL_NPN_NEGOTIATED will not be defined -> False/False
- 1.0.1 and 1.0.2: OPENSSL_NPN_NEGOTIATED will be defined and
OPENSSL_NO_NEXTPROTONEG will not be defined -> True/True
- 1.1.0+: OPENSSL_NPN_NEGOTIATED will be defined and
OPENSSL_NO_NEXTPROTONEG will not be defined -> True/True
* Refine NPN guard:
- If NPN is disabled, but ALPN is available we need our callback
- Make clinic's ssl behave the same way
This created a working ssl module for me, with NPN disabled and ALPN
enabled for OpenSSL 1.1.0f.
Concerns to address:
The initial commit for NPN support into OpenSSL [1], had the
OPENSSL_NPN_* variables defined inside the OPENSSL_NO_NEXTPROTONEG
guard. The question is if that ever made it into a release.
This would need an ugly hack, something like:
#if defined(OPENSSL_NO_NEXTPROTONEG) && \
!defined(OPENSSL_NPN_NEGOTIATED)
# define OPENSSL_NPN_UNSUPPORTED 0
# define OPENSSL_NPN_NEGOTIATED 1
# define OPENSSL_NPN_NO_OVERLAP 2
#endif
[1] https://github.com/openssl/openssl/commit/68b33cc5c7
The function '_PyArg_ParseStack()' and
'_PyArg_UnpackStack' were failing (with error
"XXX() takes Y argument (Z given)") before
the function '_PyArg_NoStackKeywords()' was called.
Thus, the latter did not raise its more meaningful
error : "XXX() takes no keyword arguments".
* bpo-16500: Allow registering at-fork handlers
* Address Serhiy's comments
* Add doc for new C API
* Add doc for new Python-facing function
* Add NEWS entry + doc nit
Element.getiterator() and the html parameter of XMLParser() were
deprecated only in the documentation (since Python 3.2 and 3.4 correspondintly).
Now using them emits a deprecation warning.
* Don’t need check_warnings any more.
* bpo-6532: Make the thread id an unsigned integer.
From C API side the type of results of PyThread_start_new_thread() and
PyThread_get_thread_ident(), the id parameter of
PyThreadState_SetAsyncExc(), and the thread_id field of PyThreadState
changed from "long" to "unsigned long".
* Restore a check in thread_get_ident().
Issue #29300: Rename struct.unpack() second parameter from "inputstr" to
"buffer", and use the Py_buffer type.
Fix also unit tests on struct.unpack() which passed a Unicode string instead of
a bytes string as struct.unpack() second parameter. The purpose of
test_trailing_counter() is to test invalid format strings, not to test the
buffer parameter.
* The struct module now requires contiguous buffers.
* Convert most functions and methods of the _struct module to Argument Clinic
* Use "Py_buffer" type for the "buffer" argument. Argument Clinic is
responsible to create and release the Py_buffer object.
* Use "PyStructObject *" type for self to avoid explicit conversions.
* Add an unit test on the _struct.Struct.unpack_from() method to test passing
arguments as keywords.
* Rephrase docstrings.
* Rename "fmt" argument to "format" in docstrings and the documentation.
As a side effect, functions and methods which used METH_VARARGS calling
convention like struct.pack() now use the METH_FASTCALL calling convention
which avoids the creation of temporary tuple to pass positional arguments and
so is faster. For example, struct.pack("i", 1) becomes 1.56x faster (-36%)::
$ ./python -m perf timeit \
-s 'import struct; pack=struct.pack' 'pack("i", 1)' \
--compare-to=../default-ref/python
Median +- std dev: 119 ns +- 1 ns -> 76.8 ns +- 0.4 ns: 1.56x faster (-36%)
Significant (t=295.91)
Patch co-written with Serhiy Storchaka.
Issue #29286. Run Argument Clinic to get the new faster METH_FASTCALL calling
convention for functions using "boring" positional arguments.
Manually fix _elementtree: _elementtree_XMLParser_doctype() must remain
consistent with the clinic code.
This implementation provides additional 10-20% speed boost for
asyncio programs.
The patch also fixes _asynciomodule.c to use Arguments Clinic, and
makes '_schedule_callbacks' an overridable method (as it was in 3.5).
The underlying zlib library stores sizes in “unsigned int”. The corresponding
Python parameters are all sizes of buffers filled in by zlib, so it is okay
to reduce higher values to the UINT_MAX internal cap. OverflowError is still
raised for sizes that do not fit in Py_ssize_t.
Sizes are now limited to Py_ssize_t rather than unsigned long, because Python
byte strings cannot be larger than Py_ssize_t. Previously this could result
in a SystemError on 32-bit platforms.
This resolves a regression in the gzip module when reading more than UINT_MAX
or LONG_MAX bytes in one call, introduced by revision 62723172412c.
This changes the main documentation, doc strings, source code comments, and a
couple error messages in the test suite. In some cases the word was removed
or edited some other way to fix the grammar.
A deprecation warning no longer issued by XMLParser subclass with default
doctype() method. Direct call of doctype() now issues a warning. Parser's
doctype() now is not called if target's doctype() is called. Based on patch
by Martin Panter.
For details, see:
PEP 0485 -- A Function for testing approximate equality
Functions added: math.isclose() and cmath.isclose().
Original code by Chris Barker. Patch by Tal Einat.
(empty) definition of the methoddef macro: it's only generated once, even
if Argument Clinic processes the same symbol multiple times, and it's emitted
at the end of all processing rather than immediately after the first use.
Along the way, fix an argumrnt to fcntl.fcntl to be an int instead of
a long.
Thanks to Serhiy Storchaka for reviewing my Clinic patch and for
writing the patch to fix the long/int issue.
The new syntax is highly human readable while still preventing false
positives. The syntax also extends Python syntax to denote "self" and
positional-only parameters, allowing inspect.Signature objects to be
totally accurate for all supported builtins in Python 3.4.
annotate text signatures in docstrings, resulting in fewer false
positives. "self" parameters are also explicitly marked, allowing
inspect.Signature() to authoritatively detect (and skip) said parameters.
Issue #20326: Argument Clinic now generates separate checksums for the
input and output sections of the block, allowing external tools to verify
that the input has not changed (and thus the output is not out-of-date).
Yury Selivanov
Segev Finer
Andrew Svetlov
Serhiy Storchaka
Victor Stinner
Tal Einat
brainfvck
Oren Milman
Antoine Pitrou
Zachary Ware
Melvyn Sopacua
Yaron de Leeuw
Sylvain
Chandan Kumar
Gregory P. Smith
Xiang Zhang
Yury Selivanov
Mark Dickinson
Antoine Pitrou
Martin Panter
Berker Peksag
Benjamin Peterson
Steve Dower
Christian Heimes
Brett Cannon
Ethan Furman
Martin Panter
Charles-François Natali
Larry Hastings