Commit Graph

24196 Commits

Author SHA1 Message Date
Yurii Karabas 7301979b23
bpo-42202: Store func annotations as a tuple (GH-23316)
Reduce memory footprint and improve performance of loading modules having many func annotations.

  >>> sys.getsizeof({"a":"int","b":"int","return":"int"})
  232
  >>> sys.getsizeof(("a","int","b","int","return","int"))
  88

The tuple is converted into dict on the fly when `func.__annotations__` is accessed first.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Co-authored-by: Inada Naoki <songofacandy@gmail.com>
2020-11-25 19:43:18 +09:00
Julien Palard 4fedd7123e
bpo-12800: tarfile: Restore fix from 011525ee9 (GH-21409)
Restore fix from 011525ee92.
2020-11-25 10:23:17 +01:00
Julien Palard c9c6e9f89a
bpo-42238: Doc: Remove make suspicious from the CI and docs builds. (GH-23313)
It probably helped a lot a while back, but may not be as usefull
today.  We'll continue monitoring it before deletion, so true
positives can be migrated to rstlint.
2020-11-25 10:18:00 +01:00
Hai Shi ed1a5a5bac
bpo-40170: Hide impl detail of Py_TRASHCAN_BEGIN macro (GH-23235)
The Py_TRASHCAN_BEGIN macro no longer accesses PyTypeObject attributes,
but now can get the condition by calling the new private
_PyTrash_cond() function which hides implementation details.
2020-11-24 23:03:31 +01:00
Yurii Karabas 0ec34cab9d
bpo-42392: Remove loop parameter form asyncio locks and Queue (#23420)
Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
2020-11-24 20:08:54 +02:00
Victor Stinner ac7d0169d2
bpo-42212: smelly.py also checks the dynamic library (GH-23423)
The smelly.py script now also checks the Python dynamic library and
extension modules, not only the Python static library. Make also the
script more verbose: explain what it does.

The GitHub Action job now builds Python with the libpython dynamic
library.
2020-11-24 13:38:08 +01:00
Yaroslav Pankovych 79d2e62c00
Added support for negative indexes to PurePath.parents (GH-21799)
This commit also fixes up some of the overlapping documentation changed
in bpo-35498, which added support for indexing with slices.

Fixes bpo-21041.
https://bugs.python.org/issue21041

Co-authored-by: Paul Ganssle <p.ganssle@gmail.com>
Co-authored-by: Rémi Lapeyre <remi.lapeyre@henki.fr>
2020-11-23 15:06:22 -05:00
Nick Crews 2f2f9d0b5c
bpo-15450: Allow subclassing of dircmp (GH-23424) (#23424)
Co-authored-by: Chris Jerdonek <chris.jerdonek@gmail.com>
2020-11-23 18:29:37 +02:00
Irit Katriel ff420f0e08
bpo-28850: Fix PrettyPrinter.format overrides ignored for contents of small containers (GH-22120) 2020-11-23 15:31:31 +02:00
Serhiy Storchaka dd844a2916
bpo-42328: Fix tkinter.ttk.Style.map(). (GH-23300)
The function accepts now the representation of the default state as
empty sequence (as returned by Style.map()).
The structure of the result is now the same on all platform
and does not depend on the value of wantobjects.
2020-11-22 22:48:52 +02:00
Serhiy Storchaka 313467efdc
bpo-42435: Speed up comparison of bytes and bytearray object (GH--23461)
* Speed up comparison of bytes objects with non-bytes objects when
  option -b is specified.
* Speed up comparison of bytarray objects with non-buffer object.
2020-11-22 22:00:53 +02:00
Serhiy Storchaka 686c203cd4
bpo-42423: Accept single base class in PyType_FromModuleAndSpec() (GH-23441) 2020-11-22 13:25:02 +02:00
Ronald Oussoren 404a719b51
bpo-41116: Ensure system supplied libraries are found on macOS 11 (GH-23301)
On macOS system provided libraries are in a shared library cache
and not at their usual location. This PR teaches distutils to search
in the SDK, even if there was no "-sysroot" argument in
the compiler flags.
2020-11-22 00:14:25 -05:00
Serhiy Storchaka 453bc1da20
bpo-42426: IDLE: Fix reporting offset of the RE error in searchengine (GH-23447) 2020-11-22 00:06:51 -05:00
Ronald Oussoren 0f20bd9042
bpo-38443: Check that the specified universal architectures work (GH-22910)
As [bpo-38443]() says the error message from configure when specifying --enable-universalsdk with a set of architectures that is not supported by the compiler is not very helpful.   This PR explicitly checks if the compiler works and bails out if it doesn't.
2020-11-21 17:13:11 -08:00
Christian Heimes ea97ebab35
bpo-1635741: Port select module to multiphase init (GH-23409) 2020-11-21 20:29:26 +01:00
Christian Heimes 035deee265
bpo-1635741: Port _posixsubprocess module to multiphase init (GH-23406) 2020-11-21 20:28:14 +01:00
David CARLIER 0e62efc51e
bpo-42232: mmap module add Darwin specific madvise options. (GH-23076) 2020-11-21 03:39:56 -08:00
Filipe Laíns 01a202ab6b
bpo-40550: Fix time-of-check/time-of-action issue in subprocess.Popen.send_signal. (GH-20010)
send_signal() now swallows the exception if the process it thought was still alive winds up not to exist anymore (always a plausible race condition despite the checks).

Co-authored-by: Gregory P. Smith <greg@krypto.org>
2020-11-21 01:22:08 -08:00
Devin Jeanpierre 31729366e2
bpo-40791: Make compare_digest more constant-time. (GH-20444)
* bpo-40791: Make compare_digest more constant-time.

The existing volatile `left`/`right` pointers guarantee that the reads will all occur, but does not guarantee that they will be _used_. So a compiler can still short-circuit the loop, saving e.g. the overhead of doing the xors and especially the overhead of the data dependency between `result` and the reads. That would change performance depending on where the first unequal byte occurs. This change removes that optimization.

(This is change #1 from https://bugs.python.org/issue40791 .)
2020-11-21 00:55:23 -08:00
Joshua Cannon 4520584483
bpo-35498: Added slice support to PathLib parents attribute. (GH-11165)
Added slice support to the `pathlib.Path.parents` sequence. For a `Path` `p`, slices of `p.parents` should return the same thing as slices of `tuple(p.parents)`.
2020-11-20 10:40:39 -05:00
Erlend Egeberg Aasland a6109ef68d
bpo-1635741: Convert _sre types to heap types and establish module state (PEP 384) (GH-23393) 2020-11-20 21:36:23 +09:00
Christian Heimes 5c36da78d7
bpo-42333: Port _ssl extension module to heap types (GH-23392)
All types in _ssl module are now heap types.
2020-11-20 09:40:12 +01:00
Christian Heimes 03c8ddd9e9
bpo-42413: socket.timeout is now an alias of TimeoutError (GH-23413)
Signed-off-by: Christian Heimes <christian@python.org>
2020-11-20 00:26:07 -08:00
Terry Jan Reedy 7ddbaa7a1b
bpo-42416: Use inspect.getdoc for IDLE calltips (GH-23416)
Inspect.getdoc(ob) sometimes gets docstrings when ob.__doc__ is None.
2020-11-20 01:59:11 -05:00
Christian Heimes b437aa83f9
bpo-1635741: Port _posixshmem extension module to multiphase initialization (GH-23404)
Signed-off-by: Christian Heimes <christian@python.org>
2020-11-19 07:20:42 -08:00
Christian Heimes 646d7fdefb
bpo-1635741: Port gc module to multiphase initialization (GH-23377)
Signed-off-by: Christian Heimes <christian@python.org>

Automerge-Triggered-By: GH:tiran
2020-11-19 06:08:34 -08:00
Christian Heimes cfeb5437a8
bpo-1635741: Port _struct to multiphase initialization (GH-23398)
Signed-off-by: Christian Heimes <christian@python.org>
2020-11-19 22:36:23 +09:00
Victor Stinner 3390347aa0
bpo-42403: Simplify importlib external bootstrap (GH-23397)
Simplify the importlib external bootstrap code:
importlib._bootstrap_external now uses regular imports to import
builtin modules. When it is imported, the builtin __import__()
function is already fully working and so can be used to import
builtin modules like sys.
2020-11-19 13:43:43 +01:00
Christian Heimes bf9d70a1a5
bpo-1635741: Port spwd to multiphase initialization (GH-23390)
Signed-off-by: Christian Heimes <christian@python.org>
2020-11-19 10:54:03 +01:00
Christian Heimes 3094dd5fb5
bpo-1635741: Port _queue to multiphase initialization (GH-23376)
Signed-off-by: Christian Heimes <christian@python.org>

Automerge-Triggered-By: GH:tiran
2020-11-19 00:24:37 -08:00
Christian Heimes fa2eee975d
bpo-1635741: Port grp and pwd to multiphase initialization (GH-23360)
Signed-off-by: Christian Heimes <christian@python.org>
2020-11-18 23:47:32 -08:00
Christian Heimes cc0cd43c0f
bpo-1635741: Port _random to multiphase initialization (GH-23359)
Signed-off-by: Christian Heimes <christian@python.org>
2020-11-18 23:46:29 -08:00
David CARLIER 13b865f0e1
bpo-42375: subprocess DragonFlyBSD build update. (GH-23320)
Same as FreeBSD, file descriptors in /dev/fd id from 0 to 63.
2020-11-18 23:24:15 -08:00
Victor Stinner 6f4635fe20
bpo-1635741: Port _warnings to the multi-phase init (GH-23379)
Port the _warnings extension module to the multi-phase initialization
API (PEP 489).
2020-11-19 00:19:06 +01:00
Victor Stinner 0e2ac21dd4
bpo-39573: Convert Py_TYPE() and Py_SIZE() back to macros (GH-23366)
This change partically reverts
commit ad3252bad9
and the commit fe2978b3b9.

Many third party C extension modules rely on the ability of using
Py_TYPE() to set an object type: "Py_TYPE(obj) = type;" or to set an
object type using: "Py_SIZE(obj) = size;".
2020-11-18 18:48:06 +01:00
Christian Heimes 07f2adedf0
bpo-40998: Address compiler warnings found by ubsan (GH-20929)
Signed-off-by: Christian Heimes <christian@python.org>

Automerge-Triggered-By: GH:tiran
2020-11-18 07:38:53 -08:00
Christian Heimes 46f59ebd01
bpo-1635741: Port _hashlib to multiphase initialization (GH-23358)
Signed-off-by: Christian Heimes <christian@python.org>
2020-11-18 16:12:13 +01:00
Christian Heimes c7011012fa
bpo-1635741: Port symtable module to multiphase initialization (GH-23361)
Signed-off-by: Christian Heimes <christian@python.org>
2020-11-18 07:03:51 -08:00
Victor Stinner 8fba9523cf
bpo-42398: Fix "make regen-all" race condition (GH-23362)
Fix a race condition in "make regen-all" when make -jN option is used
to run jobs in parallel. The clinic.py script now only use atomic
write to write files. Moveover, generated files are now left
unchanged if the content does not change, to not change the file
modification time.

The "make regen-all" command runs "make clinic" and "make
regen-importlib" targets:

* "make regen-importlib" builds object files (ex: Modules/_weakref.o)
  from source files (ex: Modules/_weakref.c) and clinic files (ex:
  Modules/clinic/_weakref.c.h)
* "make clinic" always rewrites all clinic files
  (ex: Modules/clinic/_weakref.c.h)

Since there is no dependency between "clinic" and "regen-importlib"
Makefile targets, these two targets can be run in parallel. Moreover,
half of clinic.py file writes are not atomic and so there is a race
condition when "make regen-all" runs jobs in parallel using make -jN
option (which can be passed in MAKEFLAGS environment variable).

Fix clinic.py to make all file writes atomic:

* Add write_file() function to ensure that all file writes are
  atomic: write into a temporary file and then use os.replace().
* Moreover, write_file() doesn't recreate or modify the file if the
  content does not change to avoid modifying the file modification
  file.
* Update test_clinic to verify these assertions with a functional
  test.
* Remove Clinic.force attribute which was no longer used, whereas
  Clinic.verify remains useful.
2020-11-18 15:36:27 +01:00
Christian Heimes ce04e7105b
bpo-41561: skip test_min_max_version_mismatch (GH-22308)
skip test_min_max_version_mismatch when TLS 1.0 is not available

Signed-off-by: Christian Heimes <christian@python.org>
2020-11-18 13:10:53 +01:00
Victor Stinner 7f9b25a21a
bpo-41713: Port _signal module to multi-phase init (GH-23355)
Port the _signal extension module to the multi-phase initialization
API (PEP 489).

Co-Authored-By: Mohamed Koubaa <koubaa.m@gmail.com>
2020-11-17 23:28:25 +01:00
Mark Shannon 266b462238
bpo-42349: Compiler clean up. More yak-shaving for PEP 626. (GH-23267)
Make sure that CFG from compiler front-end is correct. Be a bit more aggressive in the compiler back-end.
2020-11-17 19:30:14 +00:00
Victor Stinner 0ae323b87f
bpo-41686: Always create the SIGINT event on Windows (GH-23344)
bpo-41686, bpo-41713: On Windows, the SIGINT event,
_PyOS_SigintEvent(), is now created even if Python is configured to
not install signal handlers (PyConfig.install_signal_handlers=0 or
Py_InitializeEx(0)).

Changes:

* Move global variables initialization from signal_exec() to
  _PySignal_Init() to clarify that they are global variables cleared
  by _PySignal_Fini().
* _PySignal_Fini() now closes sigint_event.
* IntHandler is no longer a global variable.
2020-11-17 18:15:20 +01:00
Erlend Egeberg Aasland a1f401a58b
bpo-42264: Deprecate sqlite3.OptimizedUnicode (GH-23163) 2020-11-17 17:55:12 +02:00
Victor Stinner 296a796951
bpo-41713: Remove PyOS_InitInterrupts() function (GH-23342)
Remove the undocumented PyOS_InitInterrupts() C function.

* Rename PyOS_InitInterrupts() to _PySignal_Init(). It now installs
  other signal handlers, not only SIGINT.
* Rename PyOS_FiniInterrupts() to _PySignal_Fini()
2020-11-17 16:22:23 +01:00
Erlend Egeberg Aasland 2ffba2a102
bpo-41861, _sqlite3 : Add NEWS entry and rename variables (GH-23337) 2020-11-17 13:52:54 +01:00
Yurii Karabas f03d318ca4
bpo-42345: Fix three issues with typing.Literal parameters (GH-23294)
Literal equality no longer depends on the order of arguments.

Fix issue related to `typing.Literal` caching by adding `typed` parameter to `typing._tp_cache` function.

Add deduplication of `typing.Literal` arguments.
2020-11-16 18:23:19 -08:00
Pablo Galindo b0aba1fcdc
bpo-42381: Allow walrus in set literals and set comprehensions (GH-23332)
Currently walruses are not allowerd in set literals and set comprehensions:

>>> {y := 4, 4**2, 3**3}
  File "<stdin>", line 1
    {y := 4, 4**2, 3**3}
       ^
SyntaxError: invalid syntax

but they should be allowed as well per PEP 572
2020-11-17 01:17:12 +00:00
Pablo Galindo a57b3d30f6
bpo-41625: Expose the splice() system call in the os module (GH-21947) 2020-11-17 00:00:38 +00:00
Steve Dower 9cc9e27725
bpo-42120: Remove macro defining copysign to _copysign on Windows (GH-23326) 2020-11-16 23:22:42 +00:00
Lysandros Nikolaou cae60187cf
bpo-42316: Allow unparenthesized walrus operator in indexes (GH-23317) 2020-11-17 01:09:35 +02:00
Lysandros Nikolaou cb3e5ed071
bpo-42374: Allow unparenthesized walrus in genexps (GH-23319)
This fixes a regression that was introduced by the new parser.

Automerge-Triggered-By: GH:lysnikolaou
2020-11-16 15:08:35 -08:00
Zackery Spytz f62dad16b8
bpo-38506: Fix the Windows py.exe launcher's misordering of 3.10 (GH-18307) 2020-11-16 21:32:35 +00:00
Kevin Adler c79667ff79
bpo-42087: Remove support for AIX 5.3 and below (GH-22830)
As AIX 5.3 and below do not support thread_cputime, it was decided in
https://bugs.python.org/issue40680 to require AIX 6.1 and above. This
commit removes workarounds for — and references to — older, unsupported
AIX versions.
2020-11-16 16:16:10 +01:00
Victor Stinner ae6cd7cfda
bpo-37205: time.time() cannot fail with fatal error (GH-23314)
time.time(), time.perf_counter() and time.monotonic() functions can
no longer fail with a Python fatal error, instead raise a regular
Python exception on failure.

Remove _PyTime_Init(): don't check system, monotonic and perf counter
clocks at startup anymore.

On error, _PyTime_GetSystemClock(), _PyTime_GetMonotonicClock() and
_PyTime_GetPerfCounter() now silently ignore the error and return 0.
They cannot fail with a Python fatal error anymore.

Add py_mach_timebase_info() and win_perf_counter_frequency()
sub-functions.
2020-11-16 16:08:05 +01:00
Victor Stinner 5909a494cd
bpo-42350: Fix Thread._reset_internal_locks() (GH-23268)
Fix the threading.Thread class at fork: do nothing if the thread is
already stopped (ex: fork called at Python exit). Previously, an
error was logged in the child process.
2020-11-16 15:20:34 +01:00
Victor Stinner 3df5c68487
bpo-37205: time.perf_counter() and time.monotonic() are system-wide (GH-23284)
time.perf_counter() on Windows and time.monotonic() on macOS are now
system-wide. Previously, they used an offset computed at startup to
reduce the precision loss caused by the float type. Use
time.perf_counter_ns() and time.monotonic_ns() added in Python 3.7 to
avoid this precision loss.
2020-11-16 13:21:45 +01:00
Yash Shete aa01011003
bpo-42153 Fix link to IMAP documents in imaplib.rst (GH-23297)
The University of Washington stopped hosting the IMAP documents. Link to a rescued copy on GitHub.
2020-11-15 21:02:35 -08:00
kj 384b7a4bd9
bpo-42332: Add weakref slot to types.GenericAlias (GH-23250)
Automerge-Triggered-By: GH:gvanrossum
2020-11-15 19:27:23 -08:00
Serhiy Storchaka a26215db11
bpo-42318: Fix support of non-BMP characters in Tkinter on macOS (GH-23281) 2020-11-15 18:16:59 +02:00
Brett Cannon d2e94bb084
bpo-42131: Add PEP 451-related methods to zipimport (GH-23187)
Specifically, find_spec(), create_module(), and exec_module().

Co-authored-by: Nick Coghlan <ncoghlan@gmail.com>
2020-11-13 15:14:58 -08:00
Steve Dower 9b6934230c
bpo-40754: Adds _testinternalcapi to Windows installer for test suite (GH-23271) 2020-11-13 22:24:36 +00:00
Christian Heimes cd9fed6afb
bpo-41001: Add os.eventfd() (#20930)
Co-authored-by: Kyle Stanley <aeros167@gmail.com>
2020-11-13 19:48:52 +01:00
Christian Heimes f97406be4c
bpo-40968: Send http/1.1 ALPN extension (#20959)
Signed-off-by: Christian Heimes <christian@python.org>
2020-11-13 16:37:52 +01:00
Victor Stinner b3b98082c5
bpo-41617: Add _Py__has_builtin() macro (GH-23260)
Fix building pycore_bitutils.h internal header on old clang version
without __builtin_bswap16() (ex: Xcode 4.6.3 on Mac OS X 10.7).

Add a new private _Py__has_builtin() macro to check for availability
of a preprocessor builtin function.

Co-Authored-By: Joshua Root <jmr@macports.org>

Co-authored-by: Joshua Root <jmr@macports.org>
2020-11-13 15:38:17 +01:00
Victor Stinner d96a7a8313
bpo-42296: On Windows, fix CTRL+C regression (GH-23257)
On Windows, fix a regression in signal handling which prevented to
interrupt a program using CTRL+C. The signal handler can be run in a
thread different than the Python thread, in which case the test
deciding if the thread can handle signals is wrong.

On Windows, _PyEval_SignalReceived() now always sets eval_breaker to
1 since it cannot test _Py_ThreadCanHandleSignals(), and
  eval_frame_handle_pending() always calls
  _Py_ThreadCanHandleSignals() to recompute eval_breaker.
2020-11-13 14:44:42 +01:00
Mario Corchero 750c5abf43
bpo-42308: Add threading.__excepthook__ (GH-23218)
Add threading.__excepthook__ to allow retrieving the original value
of threading.excepthook in case it is set to a broken or a different
value.
2020-11-12 18:27:44 +01:00
Victor Stinner b5cc05bbe6
bpo-38823: Always build _ctypes with wchar_t (GH-23248)
It is no longer possible to build the _ctypes extension module
without wchar_t type: remove CTYPES_UNICODE macro. Anyway, the
wchar_t type is required to build Python.
2020-11-12 15:38:13 +01:00
Jakub Stasiak fd4ed57674
bpo-42237: Fix os.sendfile() on illumos (GH-23154) 2020-11-12 11:49:30 +02:00
Mark Shannon 877df851c3
bpo-42246: Partial implementation of PEP 626. (GH-23113)
* Implement new line number table format, as defined in PEP 626.
2020-11-12 09:43:29 +00:00
Batuhan Taskaya fa476fe132
bpo-39411: pyclbr rewrite on AST (#18103)
- Rewrite pyclbr using an AST processor
- Add is_async to the pyclbr.Function
2020-11-11 10:14:12 +03:00
Victor Stinner 95ce7cd0a6
bpo-1635741: Fix typo in PyModule_AddObjectRef() doc (GH-23234)
It is similar to PyModule_AddObject(), not to itself.
2020-11-11 01:52:26 +01:00
Diogo Dutra 7e5ef0a571
bpo-42140: Improve asyncio.wait function (GH-22938)
# Improve asyncio.wait function

The original code creates the futures set two times.
We can create this set before, avoiding the second creation.

This new behaviour [breaks the aiokafka library](https://github.com/aio-libs/aiokafka/pull/672), because it gives an iterator to that function, so the second iteration become empty.

Automerge-Triggered-By: GH:1st1
2020-11-10 14:12:52 -08:00
Hai Shi a13b26cac1
bpo-41073: PyType_GetSlot() can now accept static types. (GH-21931)
PyType_GetSlot() can now accept static types.

Co-Authored-By: Petr Viktorin <encukou@gmail.com>

Automerge-Triggered-By: GH:encukou
2020-11-10 12:53:46 -08:00
Victor Stinner ace3f9a0ce
bpo-42260: Fix _PyConfig_Read() if compute_path_config=0 (GH-23220)
Fix _PyConfig_Read() if compute_path_config=0: use values set by
Py_SetPath(), Py_SetPythonHome() and Py_SetProgramName(). Add
compute_path_config parameter to _PyConfig_InitPathConfig().

The following functions now return NULL if called before
Py_Initialize():

* Py_GetExecPrefix()
* Py_GetPath()
* Py_GetPrefix()
* Py_GetProgramFullPath()
* Py_GetProgramName()
* Py_GetPythonHome()

These functions no longer automatically computes the Python Path
Configuration. Moreover, Py_SetPath() no longer computes
program_full_path.
2020-11-10 21:10:22 +01:00
Vladimir Matveev 1e996c3a3b
bpo-42085: Introduce dedicated entry in PyAsyncMethods for sending values (#22780) 2020-11-10 12:09:55 -08:00
Michal Čihař e59b2deffd
bpo-42014: shutil.rmtree: call onerror with correct function (GH-22585)
The onerror is supposed to be called with failed function, but in this case lstat is wrongly used instead of open.

Not sure if this needs bug or not...

Automerge-Triggered-By: GH:hynek
2020-11-10 08:06:02 -08:00
Andrew Svetlov 42d873c63a
bpo-42183: Fix a stack overflow error for asyncio Task or Future repr() (GH-23020)
The overflow occurs under some circumstances when a task or future
recursively returns itself.

Co-authored-by: Kyle Stanley <aeros167@gmail.com>
2020-11-10 15:58:31 +02:00
Petr Viktorin 0b9c4c6fcf
bpo-42171: Add PEP573-related items to the limited API (GH-23009) 2020-11-10 05:47:31 -08:00
jack1142 bfc6b63102
bpo-36310: Allow pygettext.py to detect calls to gettext in f-strings. (GH-19875)
Adds support to Tools/i18n/pygettext.py for gettext calls in f-strings. This process is done by parsing the f-strings, processing each value, and flagging the ones which contain a gettext call.

Co-authored-by: Batuhan Taskaya <batuhanosmantaskaya@gmail.com>
2020-11-10 01:50:45 +03:00
Tom Gringauz a117167d8d
bpo-41543: contextlib.nullcontext can fill in for an async context manager (GH-21870)
Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
2020-11-09 14:34:07 +02:00
Ammar Askar 97e8b1eaea
bpo-40624: Add support for the XPath != operator in xml.etree (GH-22147) 2020-11-09 08:02:39 +01:00
kj 4eb41d055e
bpo-42233: Add union type expression support for GenericAlias and fix de-duplicating of GenericAlias (GH-23077) 2020-11-08 20:00:13 -08:00
Ronald Oussoren 23831a7a90
bpo-41754: Ignore NotADirectoryError in invocation of xdg-settings (GH-23075)
It is not clear why this can happen, but several users have mentioned
getting this exception on macOS.
2020-11-08 10:46:55 +01:00
Ronald Oussoren 41761933c1
bpo-41100: Support macOS 11 and Apple Silicon (GH-22855)
Co-authored-by:  Lawrence D’Anna <lawrence_danna@apple.com>

* Add support for macOS 11 and Apple Silicon (aka arm64)
   
  As a side effect of this work use the system copy of libffi on macOS, and remove the vendored copy

* Support building on recent versions of macOS while deploying to older versions

  This allows building installers on macOS 11 while still supporting macOS 10.9.
2020-11-08 10:05:27 +01:00
Erlend Egeberg Aasland 01c6aa43dc
bpo-40077: Convert _queuemodule to use heap types (GH-23136)
@vstinner / @corona10, would you mind reviewing this?
2020-11-07 11:18:37 -08:00
Nick Coghlan 8805a4dad2
bpo-42282: Fold constants inside named expressions (GH-23190)
* The AST optimiser wasn't descending into named expressions, so
  any constant subexpressions weren't being folded at compile time
* Remove "default:" clauses inside the AST optimiser code to reduce the
  risk of similar bugs passing unnoticed in future compiler changes
2020-11-07 22:35:17 +10:00
Julien Palard ee2549c2ba
bpo-41028: Doc: Move switchers to docsbuild-scripts. (GH-20969) 2020-11-07 12:28:31 +01:00
Brett Cannon 825ac38332
bpo-42133: update parts of the stdlib to fall back to `__spec__.loader` when `__loader__` is missing (#22929) 2020-11-06 18:45:56 -08:00
Hai Shi 88c2cfd9ff
bpo-41832: PyType_FromModuleAndSpec() now accepts NULL tp_doc (GH-23123) 2020-11-06 17:04:47 +01:00
Zackery Spytz 91e93794d5
bpo-26389: Allow passing an exception object in the traceback module (GH-22610)
The format_exception(), format_exception_only(), and
print_exception() functions can now take an exception object as a positional-only argument.

Co-Authored-By: Matthias Bussonnier <bussonniermatthias@gmail.com>
2020-11-05 22:18:44 +00:00
Victor Stinner dc42af8fd1
bpo-42260: PyConfig_Read() only parses argv once (GH-23168)
The PyConfig_Read() function now only parses PyConfig.argv arguments
once: PyConfig.parse_argv is set to 2 after arguments are parsed.
Since Python arguments are strippped from PyConfig.argv, parsing
arguments twice would parse the application options as Python
options.

* Rework the PyConfig documentation.
* Fix _testinternalcapi.set_config() error handling.
* SetConfigTests no longer needs parse_argv=0 when restoring the old
  configuration.
2020-11-05 18:58:07 +01:00
vabr-g 4662fa9bfe
bpo-41877 Check for asert, aseert, assrt in mocks (GH-23165)
Currently, a Mock object which is not unsafe will raise an
AttributeError if an attribute with the prefix assert or assret is
accessed on it. This protects against misspellings of real assert
method calls, which lead to tests passing silently even if the tested
code does not satisfy the intended assertion.

Recently a check was done in a large code base (Google) and three
more frequent ways of misspelling assert were found causing harm:
asert, aseert, assrt. These are now added to the existing check.
2020-11-05 09:04:38 -08:00
Victor Stinner 53a03aafd5
bpo-42262: Add Py_NewRef() and Py_XNewRef() (GH-23152)
Added Py_NewRef() and Py_XNewRef() functions to increment the reference
count of an object and return the object.
2020-11-05 15:02:12 +01:00
Pablo Galindo 80449f243b
bpo-42266: Handle monkey-patching descriptors in LOAD_ATTR cache (GH-23157) 2020-11-05 09:23:15 +00:00
Kazantcev Andrey 178695b7ae
bpo-40816 Add AsyncContextDecorator class (GH-20516)
Co-authored-by: Yury Selivanov <yury@edgedb.com>
2020-11-05 10:52:24 +02:00
Victor Stinner af1d64d9f7
bpo-42260: Main init modify sys.flags in-place (GH-23150)
When Py_Initialize() is called twice, the second call now updates
more sys attributes for the configuration, rather than only sys.argv.

* Rename _PySys_InitMain() to _PySys_UpdateConfig().
* _PySys_UpdateConfig() now modifies sys.flags in-place, instead of
  creating a new flags object.
* Remove old commented sys.flags flags (unbuffered and skip_first).
* Add private _PySys_GetObject() function.
* When Py_Initialize(), Py_InitializeFromConfig() and
2020-11-04 17:34:34 +01:00
Victor Stinner 8021875bbc
bpo-1635741: Add PyModule_AddObjectRef() function (GH-23122)
Added PyModule_AddObjectRef() function: similar to
PyModule_AddObjectRef() but don't steal a reference to the value on
success.
2020-11-04 13:59:15 +01:00
Victor Stinner 3529718925
bpo-42236: os.device_encoding() respects UTF-8 Mode (GH-23119)
On Unix, the os.device_encoding() function now returns 'UTF-8' rather
than the device encoding if the Python UTF-8 Mode is enabled.
2020-11-04 11:20:10 +01:00
Mario Corchero 0001a1b69e
bpo-42251: Add gettrace and getprofile to threading (GH-23125)
This allows to retrieve the functions that were set in these two, which might differ from sys.gettrace and sys.getprofile within a thread.
2020-11-04 09:27:43 +00:00
Pablo Galindo a9abbda618 Python 3.10.0a2
-----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEz9yiRbEEPPKl+Xhl/+h0BBaL2EcFAl+gngEACgkQ/+h0BBaL
 2EfDABAAnQ/bHY40QjoQ+us2vufDj3uvm2S7w0LMM+kgrehhqEoG/FmVtfb8XIvD
 S8FFtSa/VkrVUXvxI/mMpWL78se+Qk9rVQJ8dUZ4SvFJVe1NJdCMNV2Ij263QgJU
 r6bNu42EODDrFAgG9vq5ImND1qnGkF+hKlMHYV3HZhTP4drBEY+TMaB55Vmsp9TH
 eBnfY50y+k8w0hJmsZA1LaVLAXU9WHaWVB90/D22Z9MJmmQF5ofsIFXYj+1IcEkE
 P5CLSCRSzVlpdeNiZWrcBIEK8ap4ZkHvwD8RI6Fu3E/h2TXeiDbp/nCHZDPZMVVk
 OVx3Z4BFTNXE4nmRDIiO5+GqIbgG10LFQKOrOvHQr6inookUaFGwJKwkaEheNr2T
 vcgZMy61OEOeiLqCe6AJ/PuNlQ267zslYucUpNqpQpCu8j691OGEHzv+NNcSRrEq
 T/S4spifDVpFPGLRDkFjJe1OQ33bVZFdJx7guY6WJXVkZILS8vghqQYng55CNzd/
 cdMqb5T6BBRb2SDqJ/qvUhyr7D8YML0pP5pywUUGKTdo84+mW5F/N2Q4g28ik9zv
 pgaehbAwS8jgoPbqH0Tz7qpXiWga9fTbFU47Az07xw5tQFD2+C5fdx+b1h2NBz2q
 exccvIw3Fn8zWikrKbeLgaSAWa2saw25dw0DMes8gG9uHD4+1JA=
 =XAGl
 -----END PGP SIGNATURE-----
gpgsig -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEz9yiRbEEPPKl+Xhl/+h0BBaL2EcFAl+hk9UACgkQ/+h0BBaL
 2EdkYxAAs5ujyihMfyoRpU9jV7pNIIbpCHMgV2s74ZBh5jnDPYRfIrGtaE/9HlTp
 PPAScsJvnPwBGNdg8Bs1cLAk/VgJC696ccJXmNugfpxs9CSzhEKQ0oF8PPm+oyqH
 LgU6Dv8dP3+ebOko/vnRPTXbL5gdbvVJvNiV5KUjE6FL+IRl2PHlvUdcEplkPXZ4
 179kgP+8hMCB1Ai1KbMDpO2XIgdkJO+iXgXg9QNDCv2zcyr6OegGYI4cPyB3m/YK
 ZiBTnYK48x/l28Gdi+LISOGa4NNQ7R0j+Qao7YDoqe1yy8Ubj3NrLub+nNnSqnqb
 8PYaFz0sWpJ4I/vjrMaUOJ+hKSwdRZqHF/LqfQBFyBCndmhi6tJNqAzcByqi03HL
 YnXlAYJI9T3rFzBOutfjvbFWkiM2DKAL6dWBIsMeuNEO6MQWTwQHKfidXJyx9zCo
 Vv5QLG5daLFDIgI4qBjJ4POxgl+7eR+5sJBNIVz43KhemiRKLdmFKWmv6moloVB+
 1xoNGmwqz4dzmlAMUj2vJiPuUYl6+P38q1dpAChOXx4hOQZ1ybsmUVUD4S+bLGVQ
 f/VYg6B2I/Fr7Yz9HVYe309abKUSzSeYnW9FkAuTHTmZPjgZHdIJl7euTE0FtNOd
 HQ0paNeC44AoqEnghH35FhFXs+wbedszyc++TltTWn32v0nSVk0=
 =qIu/
 -----END PGP SIGNATURE-----

Merge tag 'v3.10.0a2' into master

Python 3.10.0a2
2020-11-03 17:30:57 +00:00
Serhiy Storchaka 212d32f45c
bpo-42249: Fix writing binary Plist files larger than 4 GiB. (GH-23121) 2020-11-03 16:15:56 +02:00
Erlend Egeberg Aasland 74b4eda98b
bpo-40077: Convert mmap.mmap static type to a heap type (GH-23108) 2020-11-03 10:38:31 +01:00
Pablo Galindo 114ee5dec0
Python 3.10.0a2 2020-11-03 00:02:08 +00:00
Jakub Kulík 9568622c99
bpo-35455: Fix thread_time for Solaris OS (GH-11118) 2020-11-02 23:10:01 +00:00
Victor Stinner 45df61fd2d
bpo-26789: Fix logging.FileHandler._open() at exit (GH-23053)
The logging.FileHandler class now keeps a reference to the builtin
open() function to be able to open or reopen the file during Python
finalization.

Fix errors like:

    Exception ignored in: (...)
    Traceback (most recent call last):
      (...)
      File ".../logging/__init__.py", line 1463, in error
      File ".../logging/__init__.py", line 1577, in _log
      File ".../logging/__init__.py", line 1587, in handle
      File ".../logging/__init__.py", line 1649, in callHandlers
      File ".../logging/__init__.py", line 948, in handle
      File ".../logging/__init__.py", line 1182, in emit
      File ".../logging/__init__.py", line 1171, in _open
    NameError: name 'open' is not defined
2020-11-02 23:17:46 +01:00
Victor Stinner 5cf4782a26
bpo-41796: Make _ast module state per interpreter (GH-23024)
The ast module internal state is now per interpreter.

* Rename "astmodulestate" to "struct ast_state"
* Add pycore_ast.h internal header: the ast_state structure is now
  declared in pycore_ast.h.
* Add PyInterpreterState.ast (struct ast_state)
* Remove get_ast_state()
* Rename get_global_ast_state() to get_ast_state()
* PyAST_obj2mod() now handles get_ast_state() failures
2020-11-02 22:03:28 +01:00
Serhiy Storchaka 34637a0ce2
bpo-42103: Improve validation of Plist files. (GH-22882)
* Prevent some possible DoS attacks via providing invalid Plist files
  with extremely large number of objects or collection sizes.
* Raise InvalidFileException for too large bytes and string size instead of returning garbage.
* Raise InvalidFileException instead of ValueError for specific invalid datetime (NaN).
* Raise InvalidFileException instead of TypeError for non-hashable dict keys.
* Add more tests for invalid Plist files.
2020-11-02 23:01:40 +02:00
Irit Katriel 6fdfcec5b1
bpo-41943: Fix bug where assertLogs doesn't correctly filter messages… (GH-22565)
… by level

@vsajip , @pitrou

Automerge-Triggered-By: GH:vsajip
2020-11-02 11:25:29 -08:00
Jason R. Coombs aca67da4fe
Revert "bpo-37193: remove thread objects which finished process its request (GH-13893)" (GH-23107)
This reverts commit c415590212.
2020-11-02 16:48:56 +00:00
Julien Danjou 64366fa9b3
bpo-41435: Add sys._current_exceptions() function (GH-21689)
This adds a new function named sys._current_exceptions() which is equivalent ot
sys._current_frames() except that it returns the exceptions currently handled
by other threads. It is equivalent to calling sys.exc_info() for each running
thread.
2020-11-02 16:16:25 +02:00
Joongi Kim 6e8dcdaaa4
bpo-41229: Update docs for explicit aclose()-required cases and add contextlib.aclosing() method (GH-21545)
This is a PR to:

 * Add `contextlib.aclosing` which ia analogous to `contextlib.closing` but for async-generators with an explicit test case for [bpo-41229]()
 * Update the docs to describe when we need explicit `aclose()` invocation.

which are motivated by the following issues, articles, and examples:

 * [bpo-41229]()
 * https://github.com/njsmith/async_generator
 * https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#cleanup-in-generators-and-async-generators
 * https://www.python.org/dev/peps/pep-0533/
 * https://github.com/achimnol/aiotools/blob/ef7bf0cea7af/src/aiotools/context.py#L152

Particuarly regarding [PEP-533](https://www.python.org/dev/peps/pep-0533/), its acceptance (`__aiterclose__()`) would make this little addition of `contextlib.aclosing()` unnecessary for most use cases, but until then this could serve as a good counterpart and analogy to `contextlib.closing()`. The same applies for `contextlib.closing` with `__iterclose__()`.
Also, still there are other use cases, e.g., when working with non-generator objects with `aclose()` methods.
2020-11-02 00:02:48 -08:00
Tal Einat da7bb7b4d7
bpo-40511: Stop unwanted flashing of IDLE calltips (GH-20910)
They were occurring with both repeated 'force-calltip' invocations and by typing parentheses
 in expressions, strings, and comments in the argument code.

Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
2020-11-01 22:59:52 -05:00
MARUYAMA Norihiro c415590212
bpo-37193: remove thread objects which finished process its request (GH-13893)
* bpo-37193: remove the thread which finished process request from threads list

* rename variable t to thread.

* don't remove thread from list if it is daemon.

* use lock to protect self._threads.

* use finally block in case of exception from shutdown_request().

* check "not thread.daemon" before lock to avoid holding the lock if it's unnecessary.

* fix the place of _threads_lock.

* separate code to remove a current thread into a function.

* check ValueError when removing thread.

* fix wrong code which all instance shared same lock.

* Extract thread management into a _Threads class to encapsulate atomic operations and separate concerns.

* Replace multiple references of 'block_on_close' with one, avoiding the possibility that 'block_on_close' could change during the course of processing requests. Now, there's exactly one _threads object with behavior fixed for the duration.

* Add docstrings to private classes.

* Add test to ensure that a ThreadingTCPServer can be closed without serving any requests.

* Use _NoThreads as the default value. Fixes AttributeError when server is closed without serving any requests.

* Add blurb

* Add test capturing failure.

Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
2020-11-01 18:51:04 -05:00
Victor Stinner e662c398d8
bpo-42236: Use UTF-8 encoding if nl_langinfo(CODESET) fails (GH-23086)
If the nl_langinfo(CODESET) function returns an empty string, Python
now uses UTF-8 as the filesystem encoding.

In May 2010 (commit b744ba1d14), I
modified Python to log a warning and use UTF-8 as the filesystem
encoding (instead of None) if nl_langinfo(CODESET) returns an empty
string.

In August 2020 (commit 94908bbc15), I
modified Python startup to fail with a fatal error and a specific
error message if nl_langinfo(CODESET) returns an empty string. The
intent was to prevent guessing the encoding and also investigate user
configuration where this case happens.

In 10 years (2010 to 2020), I saw zero user report about the error
message related to nl_langinfo(CODESET) returning an empty string.

Today, UTF-8 became the defacto standard and it's safe to make the
assumption that the user expects UTF-8. For example,
nl_langinfo(CODESET) can return an empty string on macOS if the
LC_CTYPE locale is not supported, and UTF-8 is the default encoding
on macOS.

While this change is likely to not affect anyone in practice, it
should make UTF-8 lover happy ;-)

Rewrite also the documentation explaining how Python selects the
filesystem encoding and error handler.
2020-11-01 23:07:23 +01:00
kj 1f7dfb277e
bpo-42233: Correctly repr GenericAlias when used with typing module (GH-23081)
Noticed by @serhiy-storchaka in the bpo.  `typing`'s types were not showing the parameterized generic. 
Eg. previously:
```python
>>> typing.Union[dict[str, float], list[int]]
'typing.Union[dict, list]'
```
Now:
```python
>>> typing.Union[dict[str, float], list[int]]
'typing.Union[dict[str, float], list[int]]'
```

Automerge-Triggered-By: GH:gvanrossum
2020-11-01 10:13:38 -08:00
Ronald Oussoren 2165cea548
bpo-29566: binhex.binhex now consitently writes MacOS 9 line endings. (GH-23059)
[bpo-29566]() notes that binhex.binhex uses inconsistent line endings (both Unix and MacOS9 line endings are used). This PR changes this to use the MacOS9 line endings everywhere.
2020-11-01 01:08:48 -08:00
Lysandros Nikolaou 02cdfc93f8
bpo-42218: Correctly handle errors in left-recursive rules (GH-23065)
Left-recursive rules need to check for errors explicitly, since
even if the rule returns NULL, the parsing might continue and lead
to long-distance failures.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-10-31 20:31:41 +02:00
Inada Naoki 43ca084c88
Revert "bpo-42160: tempfile: Reduce overhead of pid check. (GH-22997)"
`_RandomNameSequence` is not true singleton so using `os.register_at_fork` doesn't make sense unlike `random._inst`.

This reverts commit 8e409cebad.
2020-10-31 11:15:38 +09:00
Pablo Galindo 06f8c3328d
bpo-42214: Fix check for NOTEQUAL token in the PEG parser for the barry_as_flufl rule (GH-23048) 2020-10-30 23:48:42 +00:00
Batuhan Taskaya 3af4b58552
bpo-42206: Propagate and raise errors from PyAST_Validate in the parser (GH-23035) 2020-10-30 11:48:41 +00:00
Eric W 8e409cebad
bpo-42160: tempfile: Reduce overhead of pid check. (GH-22997)
The _RandomSequence class in tempfile used to check the current pid every time its rng property was used.
This commit replaces this code with `os.register_at_fork` to reduce the overhead.
2020-10-30 13:56:28 +09:00
Teugea Ioan-Teodor 3317466061
bpo-42061: Document __format__ for IP addresses (GH-23018)
Automerge-Triggered-By: GH:ericvsmith
2020-10-29 15:17:59 -07:00
Yonatan Goldschmidt 350526105f
bpo-42143: Ensure PyFunction_NewWithQualName() can't fail after creating the func object (GH-22953)
func_dealloc() does not handle partially-created objects. Best not to give it any.
2020-10-29 11:58:52 +02:00
Zackery Spytz df59273c7a
bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve (GH-19639)
Use pickle.DEFAULT_PROTOCOL (currently 5) in shelve instead of a
hardcoded 3.
2020-10-29 02:44:35 -07:00
kj 4173320920
bpo-41805: Documentation for PEP 585 (GH-22615) 2020-10-27 14:37:18 -07:00
Lysandros Nikolaou 15acc4eaba
bpo-41659: Disallow curly brace directly after primary (GH-22996) 2020-10-27 20:54:20 +02:00
Victor Stinner 84f7382215
bpo-42157: Rename unicodedata.ucnhash_CAPI (GH-22994)
Removed the unicodedata.ucnhash_CAPI attribute which was an internal
PyCapsule object. The related private _PyUnicode_Name_CAPI structure
was moved to the internal C API.

Rename unicodedata.ucnhash_CAPI as unicodedata._ucnhash_CAPI.
2020-10-27 04:36:22 +01:00
Georges Toth 303aac8c56
bpo-30681: Support invalid date format or value in email Date header (GH-22090)
I am re-submitting an older PR which was abandoned but is still relevant, #10783 by @timb07.

The issue being solved () is still relevant. The original PR #10783 was closed as
the final request changes were not applied and since abandoned.

In this new PR I have re-used the original patch plus applied both comments from the review, by @maxking and @pganssle.


For reference, here is the original PR description:
In email.utils.parsedate_to_datetime(), a failure to parse the date, or invalid date components (such as hour outside 0..23) raises an exception. Document this behaviour, and add tests to test_email/test_utils.py to confirm this behaviour.

In email.headerregistry.DateHeader.parse(), check when parsedate_to_datetime() raises an exception and add a new defect InvalidDateDefect; preserve the invalid value as the string value of the header, but set the datetime attribute to None.

Add tests to test_email/test_headerregistry.py to confirm this behaviour; also added test to test_email/test_inversion.py to confirm emails with such defective date headers round trip successfully.

This pull request incorporates feedback gratefully received from @bitdancer, @brettcannon, @Mariatta and @warsaw, and replaces the earlier PR #2254.

Automerge-Triggered-By: GH:warsaw
2020-10-26 17:31:06 -07:00
Lysandros Nikolaou bca7014032
bpo-42123: Run the parser two times and only enable invalid rules on the second run (GH-22111)
* Implement running the parser a second time for the errors messages

The first parser run is only responsible for detecting whether
there is a `SyntaxError` or not. If there isn't the AST gets returned.
Otherwise, the parser is run a second time with all the `invalid_*`
rules enabled so that all the customized error messages get produced.
2020-10-27 00:42:04 +02:00
Victor Stinner c8c4200b65
bpo-42157: Convert unicodedata.UCD to heap type (GH-22991)
Convert the unicodedata extension module to the multiphase
initialization API (PEP 489) and convert the unicodedata.UCD static
type to a heap type.

Co-Authored-By: Mohamed Koubaa <koubaa.m@gmail.com>
2020-10-26 23:19:22 +01:00
Victor Stinner 920cb647ba
bpo-42157: unicodedata avoids references to UCD_Type (GH-22990)
* UCD_Check() uses PyModule_Check()
* Simplify the internal _PyUnicode_Name_CAPI structure:

  * Remove size and state members
  * Remove state and self parameters of getcode() and getname()
    functions

* Remove global_module_state
2020-10-26 19:19:36 +01:00
Lisa Roach 8374d2ee15
bpo-39101: Fixes BaseException hang in IsolatedAsyncioTestCase. (GH-22654) 2020-10-26 09:28:17 -07:00
Victor Stinner 47e1afd2a1
bpo-1635741: _PyUnicode_Name_CAPI moves to internal C API (GH-22713)
The private _PyUnicode_Name_CAPI structure of the PyCapsule API
unicodedata.ucnhash_CAPI moves to the internal C API. Moreover, the
structure gets a new state member which must be passed to the
getcode() and getname() functions.

* Move Include/ucnhash.h to Include/internal/pycore_ucnhash.h
* unicodedata module is now built with Py_BUILD_CORE_MODULE.
* unicodedata: move hashAPI variable into unicodedata_module_state.
2020-10-26 16:43:47 +01:00
Alexey Izbyshev c0590c0033
bpo-42146: Fix memory leak in subprocess.Popen() in case of uid/gid overflow (GH-22966)
Fix memory leak in subprocess.Popen() in case of uid/gid overflow

Also add a test that would catch this leak with `--huntrleaks`.

Alas, the test for `extra_groups` also exposes an inconsistency
in our error reporting: we use a custom ValueError for `extra_groups`,
but propagate OverflowError for `user` and `group`.
2020-10-25 17:09:32 -07:00
Pablo Galindo e68c67805e
bpo-42150: Avoid buffer overflow in the new parser (GH-22978) 2020-10-25 23:03:41 +00:00
Jason R. Coombs d1a0a960ee
bpo-42043: Add support for zipfile.Path subclasses (#22716)
* bpo-42043: Add support for zipfile.Path inheritance as introduced in zipp 3.2.0.

* Add blurb.
2020-10-25 14:45:05 -04:00
Jason R. Coombs df8d4c83a6
bpo-41490: ``path`` and ``contents`` to aggressively close handles (#22915)
* bpo-41490: ``path`` method to aggressively close handles

* Add blurb

* In ZipReader.contents, eagerly evaluate the contents to release references to the zipfile.

* Instead use _ensure_sequence to ensure any iterable from a reader is eagerly converted to a list if it's not already a sequence.
2020-10-25 14:21:46 -04:00
Mark Roseman 5df6c99cb4
bpo-33987: Add master ttk Frame to IDLE search dialogs (GH-22942) 2020-10-24 23:14:02 -04:00
Serhiy Storchaka 8cd1dbae32
bpo-41052: Fix pickling heap types implemented in C with protocols 0 and 1 (GH-22870) 2020-10-24 21:14:23 +03:00
Alexey Izbyshev 976da903a7
bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe (GH-11671)
* bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe

When used to run a new executable image, fork() is not a good choice
for process creation, especially if the parent has a large working set:
fork() needs to copy page tables, which is slow, and may fail on systems
where overcommit is disabled, despite that the child is not going to
touch most of its address space.

Currently, subprocess is capable of using posix_spawn() instead, which
normally provides much better performance. However, posix_spawn() does not
support many of child setup operations exposed by subprocess.Popen().
Most notably, it's not possible to express `close_fds=True`, which
happens to be the default, via posix_spawn(). As a result, most users
can't benefit from faster process creation, at least not without
changing their code.

However, Linux provides vfork() system call, which creates a new process
without copying the address space of the parent, and which is actually
used by C libraries to efficiently implement posix_spawn(). Due to sharing
of the address space and even the stack with the parent, extreme care
is required to use vfork(). At least the following restrictions must hold:

* No signal handlers must execute in the child process. Otherwise, they
  might clobber memory shared with the parent, potentially confusing it.

* Any library function called after vfork() in the child must be
  async-signal-safe (as for fork()), but it must also not interact with any
  library state in a way that might break due to address space sharing
  and/or lack of any preparations performed by libraries on normal fork().
  POSIX.1 permits to call only execve() and _exit(), and later revisions
  remove vfork() specification entirely. In practice, however, almost all
  operations needed by subprocess.Popen() can be safely implemented on
  Linux.

* Due to sharing of the stack with the parent, the child must be careful
  not to clobber local variables that are alive across vfork() call.
  Compilers are normally aware of this and take extra care with vfork()
  (and setjmp(), which has a similar problem).

* In case the parent is privileged, special attention must be paid to vfork()
  use, because sharing an address space across different privilege domains
  is insecure[1].

This patch adds support for using vfork() instead of fork() on Linux
when it's possible to do safely given the above. In particular:

* vfork() is not used if credential switch is requested. The reverse case
  (simple subprocess.Popen() but another application thread switches
  credentials concurrently) is not possible for pure-Python apps because
  subprocess.Popen() and functions like os.setuid() are mutually excluded
  via GIL. We might also consider to add a way to opt-out of vfork() (and
  posix_spawn() on platforms where it might be implemented via vfork()) in
  a future PR.

* vfork() is not used if `preexec_fn != None`.

With this change, subprocess will still use posix_spawn() if possible, but
will fallback to vfork() on Linux in most cases, and, failing that,
to fork().

[1] https://ewontfix.com/7

Co-authored-by: Gregory P. Smith [Google LLC] <gps@google.com>
2020-10-23 17:47:01 -07:00
Jacob Neil Taylor 16ee68da6e
bpo-38976: Add support for HTTP Only flag in MozillaCookieJar (#17471)
Add support for HTTP Only flag in MozillaCookieJar

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2020-10-23 15:48:55 -07:00
Christopher Marchfelder da6f098188
bpo-40592: shutil.which will not return None anymore if ; is the last char in PATHEXT (GH-20088)
shutil.which will not return None anymore for empty str in PATHEXT
Empty PATHEXT will now be defaulted to _WIN_DEFAULT_PATHEXT
2020-10-23 11:08:24 +01:00
Brett Cannon 3c69f0c933
bpo-41910: specify the default implementations of object.__eq__ and object.__ne__ (GH-22874)
See Objects/typeobject.c:object_richcompare() for the implementation of this in CPython.

Automerge-Triggered-By: GH:brettcannon
2020-10-21 16:24:38 -07:00
Pablo Galindo b451b0e9a7
bpo-38980: Add -fno-semantic-interposition when building with optimizations (GH-22862) 2020-10-21 22:46:52 +01:00
kpinc c60394c7fc
bpo-39416: Document some restrictions on the default string representations of numeric classes (GH-18111)
[bpo-39416](): Document string representations of the Numeric classes

This is a change to the specification of the Python language.

The idea here is to put sane minimal limits on the Python language's default
representations of its Numeric classes.  That way "Marty's Robotic Massage Parlor
and Python Interpreter" implementation of Python won't do anything too
crazy.

Some discussion in the email thread:
Subject: Documenting Python's float.__str__()
https://mail.python.org/archives/list/python-dev@python.org/thread/FV22TKT3S2Q3P7PNN6MCXI6IX3HRRNAL/
2020-10-21 10:13:50 -07:00
Batuhan Taskaya c7437e2c02
bpo-41747: Ensure all dataclass methods uses their parents' qualname (GH-22155)
* bpo-41747: Ensure all dataclass methods uses their parents' qualname

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2020-10-21 09:49:22 -04:00
Paul Ganssle 9a1ad2cf02
Update tzdata to 2020.3 (GH-22856)
TBH I had forgotten that we pin this 😅, and it's been updated quite a few times since we added this.
2020-10-21 06:40:43 -07:00
Dong-hee Na c0f22fb8b3
bpo-41902: Micro optimization for range.index if step is 1 (GH-22479) 2020-10-21 11:29:56 +09:00
Максим 5f22741340
bpo-23706: Add newline parameter to pathlib.Path.write_text (GH-22420) (GH-22420)
* Add _newline_ parameter to `pathlib.Path.write_text()`
* Update documentation of `pathlib.Path.write_text()`
* Add test case for `pathlib.Path.write_text()` calls with _newline_ parameter passed

Automerge-Triggered-By: GH:methane
2020-10-20 19:08:19 -07:00
Dong-hee Na 25492a5b59
bpo-41902: Micro optimization for compute_item of range (GH-22492) 2020-10-21 10:29:14 +09:00
kj 7cdf30fff3
bpo-42010: [docs] Clarify subscription of types (GH-22822) 2020-10-20 16:38:08 -07:00
Florian Dahlitz 2d55aa9e37
bpo-29981: Add examples and update index for set, dict, and generator comprehensions'(GH-20272)
Co-authored-by: Rémi Lapeyre <remi.lapeyre@henki.fr>
2020-10-20 17:27:07 -04:00
Steve Dower 6d883fbe14
bpo-38439: Update the Windows Store package's icons for IDLE. Artwork by Andrew Clover (GH-22817) 2020-10-20 15:54:13 +01:00
Andrey Doroschenko ec42789e6e
bpo-39693: mention KeyError in tarfile extractfile documentation (GH-18639)
Co-authored-by: Andrey Darascheka <andrei.daraschenka@leverx.com>
2020-10-20 10:05:01 -04:00
Miro Hrončok faddc7449d
bpo-38439: Add 256px IDLE icon to the .ico, drop gifs from it (GH-19648) 2020-10-20 13:21:08 +01:00
TIGirardi f2312037e3
bpo-38324: Fix test__locale.py Windows failures (GH-20529)
Use wide-char _W_* fields of lconv structure on Windows
Remove "ps_AF" from test__locale.known_numerics on Windows
2020-10-20 12:39:52 +01:00
Ronald Oussoren 3185267400
bpo-41491: plistlib: accept hexadecimal integer values in xml plist files (GH-22764) 2020-10-20 09:26:33 +02:00
Pablo Galindo 109826c850
bpo-42093: Add opcode cache for LOAD_ATTR (GH-22803) 2020-10-20 06:22:44 +01:00
Raymond Hettinger 871934d4cf
bpo-4356: Add key function support to the bisect module (GH-20556) 2020-10-19 22:04:01 -07:00
Justin Turner Arthur de73d432bb
bpo-38912: fix close before connect callback in test_asyncio SSL tests (GH-22691)
Reduces the rate at which the ENV CHANGED failure occurs in test_asyncio SSL tests (due to unclosed transport), but does not 100% resolve it.
2020-10-19 21:18:57 -04:00
Ruben Vorderman 23c0fb8edd
bpo-41586: Add pipesize parameter to subprocess & F_GETPIPE_SZ and F_SETPIPE_SZ to fcntl. (GH-21921)
* Add F_SETPIPE_SZ and F_GETPIPE_SZ to fcntl module
* Add pipesize parameter for subprocess.Popen class

This will allow the user to control the size of the pipes.
On linux the default is 64K. When a pipe is full it blocks for writing.
When a pipe is empty it blocks for reading. On processes that are
very fast this can lead to a lot of wasted CPU cycles. On a typical
Linux system the max pipe size is 1024K which is much better.
For high performance-oriented libraries such as xopen it is nice to
be able to set the pipe size.

The workaround without this feature is to use my_popen_process.stdout.fileno() in
conjuction with fcntl and 1031 (value of F_SETPIPE_SZ) to acquire this behavior.
2020-10-19 16:30:02 -07:00
Mark Sapiro bf838227c3
bpo-27321 Fix email.generator.py to not replace a non-existent header. (GH-18074)
This PR replaces #1977. The reason for the replacement is two-fold.

The fix itself is different is that if the CTE header doesn't exist in the original message, it is inserted. This is important because the new CTE could be quoted-printable whereas the original is implicit 8bit.

Also the tests are different. The test_nonascii_as_string_without_cte test in #1977 doesn't actually test the issue in that it passes without the fix. The test_nonascii_as_string_without_content_type_and_cte test is improved here, and even though it doesn't fail without the fix, it is included for completeness.

Automerge-Triggered-By: @warsaw
2020-10-19 15:49:19 -07:00
Zackery Spytz 1438c2ac77
bpo-41845: Move PyObject_GenericGetDict() back into the limited API (GH22646)
It was moved out of the limited API in 7d95e40721.
This change re-enables it from 3.10, to avoid generating invalid extension modules for earlier versions.
2020-10-19 23:47:37 +01:00
Alex Gaynor 3a8fdb2879
bpo-41784: make PyUnicode_AsUTF8AndSize part of the limited API (GH-22252) 2020-10-19 23:17:50 +01:00
Jason R. Coombs 5456e78f45
bpo-16396: Allow wintypes to be imported on non-Windows systems. (GH-21394)
Co-authored-by: Christian Heimes <christian@python.org>
2020-10-19 23:06:05 +01:00
Barry Warsaw 96ddc58281
bpo-42089: Sync with current cpython branch of importlib_metadata (GH-22775)
~~The only differences are in the test files.~~

Automerge-Triggered-By: @jaraco
2020-10-19 14:14:21 -07:00
Ronald Oussoren 93a1ccabde
bpo-41471: Ignore invalid prefix lengths in system proxy settings on macOS (GH-22762) 2020-10-19 20:16:21 +02:00
Ronald Oussoren 05ee790f4d
bpo-42051: Reject XML entity declarations in plist files (#22760) 2020-10-19 20:13:49 +02:00
Steve Dower 985f0ab3ad
bpo-39107: Updated Tcl and Tk to 8.6.10 in Windows installer (GH-22405) 2020-10-19 16:55:10 +01:00
Mark Shannon b580ed1d9d
Correct name of bytecode in change note. (GH-22723) 2020-10-19 13:20:33 +01:00
Bar Harel 5368c2b6e2
bpo-19270: Fixed sched.scheduler.cancel to cancel correct event (GH-22729) 2020-10-19 10:33:43 +03:00
Anthony Sottile 3c0ac18504
bpo-40492: Fix --outfile with relative path when the program changes it working dir (GH-19910) 2020-10-18 23:48:31 +03:00
Irit Katriel b81c833ab5
bpo-28660: Make TextWrapper break long words on hyphens (GH-22721) 2020-10-18 20:01:15 +03:00
scaramallion c304c9a7ef
bpo-41966: Fix pickling pure datetime.time subclasses (GH-22731) 2020-10-18 17:49:48 +03:00
Ma Lin a0c603cb9d
bpo-38252: Use 8-byte step to detect ASCII sequence in 64bit Windows build (GH-16334) 2020-10-18 17:48:38 +03:00
Max Bernstein 3635388f52
bpo-42065: Fix incorrectly formatted _codecs.charmap_decode error message (GH-19940) 2020-10-17 23:38:21 +03:00
Kevin Adler 1dd6d956a3
closes bpo-42030: Remove legacy AIX dynload support (GH-22717)
Since c19c5a6, AIX builds have defaulted to using dynload_shlib over
dynload_aix when dlopen is available. This function has been available
since AIX 4.3, which went out of support in 2003, the same year the
previously referenced commit was made. It has been nearly 20 years
since a version of AIX has been supported which has not used
dynload_shlib so there's no reason to keep this legacy code around.
2020-10-16 13:03:28 -05:00
Erlend Egeberg Aasland 644e94272a
bpo-42021: Fix possible ref leaks during _sqlite3 module init (GH-22673) 2020-10-15 21:20:15 +09:00
Kevin Adler 2d2af320d9
bpo-41894: Fix UnicodeDecodeError while loading native module (GH-22466)
When running in a non-UTF-8 locale, if an error occurs while importing a
native Python module (say because a dependent share library is missing),
the error message string returned may contain non-ASCII code points
causing a UnicodeDecodeError.

PyUnicode_DecodeFSDefault is used for buffers which may contain
filesystem  paths. For consistency with os.strerror(),
PyUnicode_DecodeLocale is used for buffers which contain system error
messages. While the shortname parameter is always encoded in ASCII
according to PEP 489, it is left decoded using PyUnicode_FromString to
minimize the changes and since it should not affect the decoding (albeit
_potentially_ slower).

In dynload_hpux, since the error buffer contains a message generated
from a static ASCII string and the module filesystem path,
PyUnicode_DecodeFSDefault is used instead of PyUnicode_DecodeLocale as
is used elsewhere.

* bpo-41894: Fix bugs in dynload error msg handling

For both dynload_aix and dynload_hpux, properly handle the possibility
that decoding strings may return NULL and when such an error happens,
properly decrement any previously decoded strings and return early.

In addition, in dynload_aix, ensure that we pass the decoded string
*object* pathname_ob to PyErr_SetImportError instead of the original
pathname buffer.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2020-10-15 10:53:27 +09:00
Brandt Bucher c13b847a6f
bpo-41984: GC track all user classes (GH-22701) 2020-10-14 18:44:07 -07:00
Hai Shi c5b049b91c
bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters (GH-22219) 2020-10-14 17:43:31 +02:00
Anatoliy Platonov b4d895336a
bpo-41876: Overload __repr__ for tkinter Font objects (GH-22450) 2020-10-14 13:02:51 +03:00
Yannick Jadoul 04b8631d84
bpo-42015: Reorder dereferencing calls in meth_dealloc, to make sure m_self is kept alive long enough (GH-22670) 2020-10-13 00:06:19 +03:00
Vladimir Matveev 24a54c0bd4
Delete PyGen_Send (#22663) 2020-10-12 12:10:42 -07:00
Victor Stinner 13ff396c01
bpo-41739: Fix test_logging.test_race_between_set_target_and_flush() (GH-22655)
The test now waits until all threads complete to avoid leaking
running threads.

Also, use regular threads rather than daemon threads.
2020-10-12 00:37:20 +02:00
Kyle Evans 1800c60080
bpo-40423: Optimization: use close_range(2) if available (GH-22651)
close_range(2) should be preferred at all times if it's available, otherwise we'll use closefrom(2) if available with a fallback to fdwalk(3) or plain old loop over fd range in order of most efficient to least.

[note that this version does check for ENOSYS, but currently ignores all other errors]

Automerge-Triggered-By: @pablogsal
2020-10-11 13:18:53 -07:00
Kyle Evans c230fde847
bpo-40422: create a common _Py_closerange API (GH-19754)
Such an API can be used both for os.closerange and subprocess. For the latter, this yields potential improvement for platforms that have fdwalk but wouldn't have used it there. This will prove even more beneficial later for platforms that have close_range(2), as the new API will prefer that over all else if it's available.

The new API is structured to look more like close_range(2), closing from [start, end] rather than the [low, high) of os.closerange().

Automerge-Triggered-By: @gpshead
2020-10-11 11:54:11 -07:00
Serhiy Storchaka 8287aadb75
bpo-41993: Fix possible issues in remove_module() (GH-22631)
* PyMapping_HasKey() is not safe because it silences all exceptions and can return incorrect result.
* Informative exceptions from PyMapping_DelItem() are overridden with RuntimeError and
  the original exception raised before calling remove_module() is lost.
* There is a race condition between PyMapping_HasKey() and PyMapping_DelItem().
2020-10-11 16:51:07 +03:00
Serhiy Storchaka 637a09b0d6
bpo-41986: Add Py_FileSystemDefaultEncodeErrors and Py_UTF8Mode back to limited API (GH-22621) 2020-10-10 17:09:45 +03:00
Vladimir Matveev 037245c5ac
bpo-41756: Add PyIter_Send function (#22443) 2020-10-09 17:15:15 -07:00
Serhiy Storchaka eb38c6b7aa
bpo-41831: Restore str implementation of __str__ in tkinter.EventType (GH-22355) 2020-10-09 22:57:34 +03:00
Serhiy Storchaka e2ec0b27c0
bpo-41974: Remove complex.__float__, complex.__floordiv__, etc (GH-22593)
Remove complex special methods __int__, __float__, __floordiv__,
__mod__, __divmod__, __rfloordiv__, __rmod__ and __rdivmod__
which always raised a TypeError.
2020-10-09 14:14:37 +03:00
Batuhan Taskaya 48f305fd12
bpo-41979: Accept star-unpacking on with-item targets (GH-22611)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-10-09 10:56:48 +01:00
Erlend Egeberg Aasland bfe6e03cd6
bpo-41557: Update Windows installer to use SQLite 3.33.0 (GH-21960) 2020-10-08 19:40:27 +01:00
Pablo Galindo 27ac19cca2
bpo-41976: Fix the fallback to gcc of ctypes.util.find_library when using gcc>9 (GH-22598) 2020-10-08 19:31:19 +01:00
E-Paine aecf036738
bpo-41306: Allow scale value to not be rounded (GH-21715)
This fixes the test failure with Tk 6.8.10 which is caused by changes to how Tk rounds the `from`, `to` and `tickinterval` arguments. This PR uses `noconv` if the patchlevel is greater than or equal to 8.6.10 (credit to Serhiy for this idea as it is much simpler than what I previously proposed).

Going into more detail for those who want it, the Tk change was made in [commit 591f68c](591f68cb38) and means that the arguments listed above are rounded relative to the value of `from`. However, when rounding the `from` argument ([line 623](591f68cb38/generic/tkScale.c (L623))), it is rounded relative to itself (i.e. rounding `0`) and therefore the assigned value for `from` is always what is given (no matter what values of `from` and `resolution`).

Automerge-Triggered-By: @pablogsal
2020-10-08 06:30:13 -07:00
Pablo Galindo 4a9f82f50d
bpo-41970: Avoid test failure in test_lib2to3 if the module is already imported (GH-22595)
…

Automerge-Triggered-By: @pablogsal
2020-10-08 06:24:28 -07:00
Raymond Hettinger 4e0ce82058
Revert "bpo-26680: Incorporate is_integer in all built-in and standard library numeric types (GH-6121)" (GH-22584)
This reverts commit 58a7da9e12.
2020-10-07 16:43:44 -07:00