Commit Graph

108446 Commits

Author SHA1 Message Date
Victor Stinner f3cb814315
bpo-42260: Add _PyConfig_FromDict() (GH-23167)
* Rename config_as_dict() to _PyConfig_AsDict().
* Add 'module_search_paths_set' to _PyConfig_AsDict().
* Add _PyConfig_FromDict().
* Add get_config() and set_config() to _testinternalcapi.
* Add config_check_consistency().
2020-11-05 18:12:33 +01:00
vabr-g 4662fa9bfe
bpo-41877 Check for asert, aseert, assrt in mocks (GH-23165)
Currently, a Mock object which is not unsafe will raise an
AttributeError if an attribute with the prefix assert or assret is
accessed on it. This protects against misspellings of real assert
method calls, which lead to tests passing silently even if the tested
code does not satisfy the intended assertion.

Recently a check was done in a large code base (Google) and three
more frequent ways of misspelling assert were found causing harm:
asert, aseert, assrt. These are now added to the existing check.
2020-11-05 09:04:38 -08:00
kj 133aa2d581
[docs] fix wrongly named AsyncContextDecorator (GH-23164)
Also added versionchanged.
2020-11-05 18:16:27 +02:00
Victor Stinner 53a03aafd5
bpo-42262: Add Py_NewRef() and Py_XNewRef() (GH-23152)
Added Py_NewRef() and Py_XNewRef() functions to increment the reference
count of an object and return the object.
2020-11-05 15:02:12 +01:00
Pablo Galindo 80449f243b
bpo-42266: Handle monkey-patching descriptors in LOAD_ATTR cache (GH-23157) 2020-11-05 09:23:15 +00:00
Kazantcev Andrey 178695b7ae
bpo-40816 Add AsyncContextDecorator class (GH-20516)
Co-authored-by: Yury Selivanov <yury@edgedb.com>
2020-11-05 10:52:24 +02:00
Victor Stinner 048a35659a
bpo-42260: Add _PyInterpreterState_SetConfig() (GH-23158)
* Inline _PyInterpreterState_SetConfig(): replace it with
  _PyConfig_Copy().
* Add _PyErr_SetFromPyStatus()
* Add _PyInterpreterState_GetConfigCopy()
* Add a new _PyInterpreterState_SetConfig() function.
* Add an unit which gets, modifies, and sets the config.
2020-11-05 00:45:56 +01:00
serge-sans-paille 100964e031
Disable peg generator tests when building with PGO (GH-23141)
Otherwise, when running the testsuite, test_peg_generator tries to compile C
code using the optimized flags and fails because it cannot find the profile
data.
2020-11-04 23:01:08 +00:00
Erlend Egeberg Aasland 789359f47c
bpo-1635741: _sqlite3 uses PyModule_AddObjectRef() (GH-23148) 2020-11-04 20:31:51 +01:00
Mohamed Koubaa 7184218e18
bpo-1635741: Fix PyInit_pyexpat() error handling (GH-22489)
Split PyInit_pyexpat() into sub-functions and fix reference leaks
on error paths.
2020-11-04 18:37:23 +01:00
Victor Stinner af1d64d9f7
bpo-42260: Main init modify sys.flags in-place (GH-23150)
When Py_Initialize() is called twice, the second call now updates
more sys attributes for the configuration, rather than only sys.argv.

* Rename _PySys_InitMain() to _PySys_UpdateConfig().
* _PySys_UpdateConfig() now modifies sys.flags in-place, instead of
  creating a new flags object.
* Remove old commented sys.flags flags (unbuffered and skip_first).
* Add private _PySys_GetObject() function.
* When Py_Initialize(), Py_InitializeFromConfig() and
2020-11-04 17:34:34 +01:00
Victor Stinner 58ca33b467
bpo-1635741: Fix ref leak in _PyWarnings_Init() error path (GH-23151)
Replace PyModule_AddObject() with PyModule_AddObjectRef() in the
_warnings module to fix a reference leak on error.

Use also PyModule_AddObjectRef() in importdl.c.
2020-11-04 17:33:06 +01:00
Victor Stinner 18ce7f1d0a
bpo-1635741: _ast uses PyModule_AddObjectRef() (GH-23146)
Replace PyModule_AddObject() with PyModule_AddObjectRef() in the _ast
module (Python-ast.c).
2020-11-04 16:37:07 +01:00
Victor Stinner 988f1ec8d2
bpo-1635741: _contextvars uses PyModule_AddType() (GH-23147)
Replace PyModule_AddObject() with PyModule_AddType() in the
_contextvars module (Python-ast.c).

Add also the module name to _contextvars types name.
2020-11-04 16:33:55 +01:00
Victor Stinner cfb41e80c1
bpo-42260: Reorganize PyConfig (GH-23149)
* Move orig_argv before argv
* Move program_name and platlibdir with other path configuration
  inputs

Give a name to the PyPreConfig and PyConfig structures and separate
the type definitions.
2020-11-04 16:15:54 +01:00
Victor Stinner 8021875bbc
bpo-1635741: Add PyModule_AddObjectRef() function (GH-23122)
Added PyModule_AddObjectRef() function: similar to
PyModule_AddObjectRef() but don't steal a reference to the value on
success.
2020-11-04 13:59:15 +01:00
Victor Stinner 3529718925
bpo-42236: os.device_encoding() respects UTF-8 Mode (GH-23119)
On Unix, the os.device_encoding() function now returns 'UTF-8' rather
than the device encoding if the Python UTF-8 Mode is enabled.
2020-11-04 11:20:10 +01:00
Mario Corchero 0001a1b69e
bpo-42251: Add gettrace and getprofile to threading (GH-23125)
This allows to retrieve the functions that were set in these two, which might differ from sys.gettrace and sys.getprofile within a thread.
2020-11-04 09:27:43 +00:00
Steve Dower db6434c474
Enable signing of nuget.org packages and update to supported timestamp server (GH-23132) 2020-11-03 22:31:49 +00:00
Matthew Suozzo bffb137cb5
Fix incorrect links in ast docs (GH-23017) 2020-11-03 23:28:42 +02:00
Petr Viktorin 57aaaa8d2a
Add _PyType_GetModuleByDef (GH-22835)
See https://mail.python.org/archives/list/capi-sig@python.org/thread/T3P2QNLNLBRFHWSKYSTPMVEIL2EEKFJU/ for discussion.

https://bugs.python.org/issue42100
2020-11-03 22:27:12 +01:00
Pablo Galindo a603c3d371
Post 3.10.0a2 2020-11-03 17:31:32 +00:00
Pablo Galindo a9abbda618 Python 3.10.0a2
-----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEz9yiRbEEPPKl+Xhl/+h0BBaL2EcFAl+gngEACgkQ/+h0BBaL
 2EfDABAAnQ/bHY40QjoQ+us2vufDj3uvm2S7w0LMM+kgrehhqEoG/FmVtfb8XIvD
 S8FFtSa/VkrVUXvxI/mMpWL78se+Qk9rVQJ8dUZ4SvFJVe1NJdCMNV2Ij263QgJU
 r6bNu42EODDrFAgG9vq5ImND1qnGkF+hKlMHYV3HZhTP4drBEY+TMaB55Vmsp9TH
 eBnfY50y+k8w0hJmsZA1LaVLAXU9WHaWVB90/D22Z9MJmmQF5ofsIFXYj+1IcEkE
 P5CLSCRSzVlpdeNiZWrcBIEK8ap4ZkHvwD8RI6Fu3E/h2TXeiDbp/nCHZDPZMVVk
 OVx3Z4BFTNXE4nmRDIiO5+GqIbgG10LFQKOrOvHQr6inookUaFGwJKwkaEheNr2T
 vcgZMy61OEOeiLqCe6AJ/PuNlQ267zslYucUpNqpQpCu8j691OGEHzv+NNcSRrEq
 T/S4spifDVpFPGLRDkFjJe1OQ33bVZFdJx7guY6WJXVkZILS8vghqQYng55CNzd/
 cdMqb5T6BBRb2SDqJ/qvUhyr7D8YML0pP5pywUUGKTdo84+mW5F/N2Q4g28ik9zv
 pgaehbAwS8jgoPbqH0Tz7qpXiWga9fTbFU47Az07xw5tQFD2+C5fdx+b1h2NBz2q
 exccvIw3Fn8zWikrKbeLgaSAWa2saw25dw0DMes8gG9uHD4+1JA=
 =XAGl
 -----END PGP SIGNATURE-----
gpgsig -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEz9yiRbEEPPKl+Xhl/+h0BBaL2EcFAl+hk9UACgkQ/+h0BBaL
 2EdkYxAAs5ujyihMfyoRpU9jV7pNIIbpCHMgV2s74ZBh5jnDPYRfIrGtaE/9HlTp
 PPAScsJvnPwBGNdg8Bs1cLAk/VgJC696ccJXmNugfpxs9CSzhEKQ0oF8PPm+oyqH
 LgU6Dv8dP3+ebOko/vnRPTXbL5gdbvVJvNiV5KUjE6FL+IRl2PHlvUdcEplkPXZ4
 179kgP+8hMCB1Ai1KbMDpO2XIgdkJO+iXgXg9QNDCv2zcyr6OegGYI4cPyB3m/YK
 ZiBTnYK48x/l28Gdi+LISOGa4NNQ7R0j+Qao7YDoqe1yy8Ubj3NrLub+nNnSqnqb
 8PYaFz0sWpJ4I/vjrMaUOJ+hKSwdRZqHF/LqfQBFyBCndmhi6tJNqAzcByqi03HL
 YnXlAYJI9T3rFzBOutfjvbFWkiM2DKAL6dWBIsMeuNEO6MQWTwQHKfidXJyx9zCo
 Vv5QLG5daLFDIgI4qBjJ4POxgl+7eR+5sJBNIVz43KhemiRKLdmFKWmv6moloVB+
 1xoNGmwqz4dzmlAMUj2vJiPuUYl6+P38q1dpAChOXx4hOQZ1ybsmUVUD4S+bLGVQ
 f/VYg6B2I/Fr7Yz9HVYe309abKUSzSeYnW9FkAuTHTmZPjgZHdIJl7euTE0FtNOd
 HQ0paNeC44AoqEnghH35FhFXs+wbedszyc++TltTWn32v0nSVk0=
 =qIu/
 -----END PGP SIGNATURE-----

Merge tag 'v3.10.0a2' into master

Python 3.10.0a2
2020-11-03 17:30:57 +00:00
Victor Stinner fd957c124c
bpo-41796: Call _PyAST_Fini() earlier to fix a leak (GH-23131)
Call _PyAST_Fini() on all interpreters, not only on the main
interpreter. Also, call it ealier to fix a reference leak.

Python types contain a reference to themselves in in their
PyTypeObject.tp_mro member. _PyAST_Fini() must called before the last
GC collection to destroy AST types.

_PyInterpreterState_Clear() now calls _PyAST_Fini(). It now also
calls _PyWarnings_Fini() on subinterpeters, not only on the main
interpreter.

Add an assertion in AST init_types() to ensure that the _ast module
is no longer used after _PyAST_Fini() has been called.
2020-11-03 18:07:15 +01:00
Serhiy Storchaka 212d32f45c
bpo-42249: Fix writing binary Plist files larger than 4 GiB. (GH-23121) 2020-11-03 16:15:56 +02:00
Erlend Egeberg Aasland 74b4eda98b
bpo-40077: Convert mmap.mmap static type to a heap type (GH-23108) 2020-11-03 10:38:31 +01:00
Pablo Galindo 114ee5dec0
Python 3.10.0a2 2020-11-03 00:02:08 +00:00
Jakub Kulík 9568622c99
bpo-35455: Fix thread_time for Solaris OS (GH-11118) 2020-11-02 23:10:01 +00:00
Victor Stinner 45df61fd2d
bpo-26789: Fix logging.FileHandler._open() at exit (GH-23053)
The logging.FileHandler class now keeps a reference to the builtin
open() function to be able to open or reopen the file during Python
finalization.

Fix errors like:

    Exception ignored in: (...)
    Traceback (most recent call last):
      (...)
      File ".../logging/__init__.py", line 1463, in error
      File ".../logging/__init__.py", line 1577, in _log
      File ".../logging/__init__.py", line 1587, in handle
      File ".../logging/__init__.py", line 1649, in callHandlers
      File ".../logging/__init__.py", line 948, in handle
      File ".../logging/__init__.py", line 1182, in emit
      File ".../logging/__init__.py", line 1171, in _open
    NameError: name 'open' is not defined
2020-11-02 23:17:46 +01:00
Victor Stinner 5cf4782a26
bpo-41796: Make _ast module state per interpreter (GH-23024)
The ast module internal state is now per interpreter.

* Rename "astmodulestate" to "struct ast_state"
* Add pycore_ast.h internal header: the ast_state structure is now
  declared in pycore_ast.h.
* Add PyInterpreterState.ast (struct ast_state)
* Remove get_ast_state()
* Rename get_global_ast_state() to get_ast_state()
* PyAST_obj2mod() now handles get_ast_state() failures
2020-11-02 22:03:28 +01:00
Serhiy Storchaka 34637a0ce2
bpo-42103: Improve validation of Plist files. (GH-22882)
* Prevent some possible DoS attacks via providing invalid Plist files
  with extremely large number of objects or collection sizes.
* Raise InvalidFileException for too large bytes and string size instead of returning garbage.
* Raise InvalidFileException instead of ValueError for specific invalid datetime (NaN).
* Raise InvalidFileException instead of TypeError for non-hashable dict keys.
* Add more tests for invalid Plist files.
2020-11-02 23:01:40 +02:00
Irit Katriel 6fdfcec5b1
bpo-41943: Fix bug where assertLogs doesn't correctly filter messages… (GH-22565)
… by level

@vsajip , @pitrou

Automerge-Triggered-By: GH:vsajip
2020-11-02 11:25:29 -08:00
Jason R. Coombs aca67da4fe
Revert "bpo-37193: remove thread objects which finished process its request (GH-13893)" (GH-23107)
This reverts commit c415590212.
2020-11-02 16:48:56 +00:00
Victor Stinner 4b9aad4999
bpo-42236: Enhance init and encoding documentation (GH-23109)
Enhance the documentation of the Python startup, filesystem encoding
and error handling, locale encoding. Add a new "Python UTF-8 Mode"
section.

* Add "locale encoding" and "filesystem encoding and error handler"
  to the glossary
* Remove documentation from Include/cpython/initconfig.h: move it to
  Doc/c-api/init_config.rst.
* Doc/c-api/init_config.rst:

  * Document command line options and environment variables
  * Document default values.

* Add a new "Python UTF-8 Mode" section in Doc/library/os.rst.
* Add warnings to Py_DecodeLocale() and Py_EncodeLocale() docs.
* Document how Python selects the filesystem encoding and error
  handler at a single place: PyConfig.filesystem_encoding and
  PyConfig.filesystem_errors.
* PyConfig: move orig_argv member at the right place.
2020-11-02 16:49:54 +01:00
Lysandros Nikolaou 301822859b
bpo-42224: Fix test_format when locale does not expect number grouping (GH-23067) 2020-11-02 17:27:30 +02:00
Julien Danjou 64366fa9b3
bpo-41435: Add sys._current_exceptions() function (GH-21689)
This adds a new function named sys._current_exceptions() which is equivalent ot
sys._current_frames() except that it returns the exceptions currently handled
by other threads. It is equivalent to calling sys.exc_info() for each running
thread.
2020-11-02 16:16:25 +02:00
Jakub Stasiak 3d86d090dc
bpo-42230: Improve asyncio documentation regarding accepting sets vs iterables (GH-23073)
People call wait() and as_completed() with various non-set iterables,
a list should be the most common but there are others as well[1].

Considering typeshed also documents wait()[2] and as_completed()[3]
as accepting arbitrary iterables I think it's a good idea to document
the status quo better.

[1] https://github.com/aio-libs/aiokafka/pull/672
[2] 620989bac5/stdlib/3/asyncio/tasks.pyi (L161)
[3] 620989bac5/stdlib/3/asyncio/tasks.pyi (L40)
2020-11-02 12:56:35 +02:00
Joongi Kim 6e8dcdaaa4
bpo-41229: Update docs for explicit aclose()-required cases and add contextlib.aclosing() method (GH-21545)
This is a PR to:

 * Add `contextlib.aclosing` which ia analogous to `contextlib.closing` but for async-generators with an explicit test case for [bpo-41229]()
 * Update the docs to describe when we need explicit `aclose()` invocation.

which are motivated by the following issues, articles, and examples:

 * [bpo-41229]()
 * https://github.com/njsmith/async_generator
 * https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/#cleanup-in-generators-and-async-generators
 * https://www.python.org/dev/peps/pep-0533/
 * https://github.com/achimnol/aiotools/blob/ef7bf0cea7af/src/aiotools/context.py#L152

Particuarly regarding [PEP-533](https://www.python.org/dev/peps/pep-0533/), its acceptance (`__aiterclose__()`) would make this little addition of `contextlib.aclosing()` unnecessary for most use cases, but until then this could serve as a good counterpart and analogy to `contextlib.closing()`. The same applies for `contextlib.closing` with `__iterclose__()`.
Also, still there are other use cases, e.g., when working with non-generator objects with `aclose()` methods.
2020-11-02 00:02:48 -08:00
Raymond Hettinger e9208f0e74
Minor formatting edits to the descriptor howto guide (GH-23092) 2020-11-01 20:15:50 -08:00
Tal Einat da7bb7b4d7
bpo-40511: Stop unwanted flashing of IDLE calltips (GH-20910)
They were occurring with both repeated 'force-calltip' invocations and by typing parentheses
 in expressions, strings, and comments in the argument code.

Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
2020-11-01 22:59:52 -05:00
Raymond Hettinger 74fa464b81
Add member objects to the descriptor howto guide (GH-23084) 2020-11-01 18:02:37 -08:00
MARUYAMA Norihiro c415590212
bpo-37193: remove thread objects which finished process its request (GH-13893)
* bpo-37193: remove the thread which finished process request from threads list

* rename variable t to thread.

* don't remove thread from list if it is daemon.

* use lock to protect self._threads.

* use finally block in case of exception from shutdown_request().

* check "not thread.daemon" before lock to avoid holding the lock if it's unnecessary.

* fix the place of _threads_lock.

* separate code to remove a current thread into a function.

* check ValueError when removing thread.

* fix wrong code which all instance shared same lock.

* Extract thread management into a _Threads class to encapsulate atomic operations and separate concerns.

* Replace multiple references of 'block_on_close' with one, avoiding the possibility that 'block_on_close' could change during the course of processing requests. Now, there's exactly one _threads object with behavior fixed for the duration.

* Add docstrings to private classes.

* Add test to ensure that a ThreadingTCPServer can be closed without serving any requests.

* Use _NoThreads as the default value. Fixes AttributeError when server is closed without serving any requests.

* Add blurb

* Add test capturing failure.

Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
2020-11-01 18:51:04 -05:00
Victor Stinner e662c398d8
bpo-42236: Use UTF-8 encoding if nl_langinfo(CODESET) fails (GH-23086)
If the nl_langinfo(CODESET) function returns an empty string, Python
now uses UTF-8 as the filesystem encoding.

In May 2010 (commit b744ba1d14), I
modified Python to log a warning and use UTF-8 as the filesystem
encoding (instead of None) if nl_langinfo(CODESET) returns an empty
string.

In August 2020 (commit 94908bbc15), I
modified Python startup to fail with a fatal error and a specific
error message if nl_langinfo(CODESET) returns an empty string. The
intent was to prevent guessing the encoding and also investigate user
configuration where this case happens.

In 10 years (2010 to 2020), I saw zero user report about the error
message related to nl_langinfo(CODESET) returning an empty string.

Today, UTF-8 became the defacto standard and it's safe to make the
assumption that the user expects UTF-8. For example,
nl_langinfo(CODESET) can return an empty string on macOS if the
LC_CTYPE locale is not supported, and UTF-8 is the default encoding
on macOS.

While this change is likely to not affect anyone in practice, it
should make UTF-8 lover happy ;-)

Rewrite also the documentation explaining how Python selects the
filesystem encoding and error handler.
2020-11-01 23:07:23 +01:00
Victor Stinner 82458b6cdb
bpo-42236: Enhance _locale._get_locale_encoding() (GH-23083)
* Rename _Py_GetLocaleEncoding() to _Py_GetLocaleEncodingObject()
* Add _Py_GetLocaleEncoding() which returns a wchar_t* string to
  share code between _Py_GetLocaleEncodingObject()
  and config_get_locale_encoding().
* _Py_GetLocaleEncodingObject() now decodes nl_langinfo(CODESET)
  from the current locale encoding with surrogateescape,
  rather than using UTF-8.
2020-11-01 20:59:35 +01:00
kj 1f7dfb277e
bpo-42233: Correctly repr GenericAlias when used with typing module (GH-23081)
Noticed by @serhiy-storchaka in the bpo.  `typing`'s types were not showing the parameterized generic. 
Eg. previously:
```python
>>> typing.Union[dict[str, float], list[int]]
'typing.Union[dict, list]'
```
Now:
```python
>>> typing.Union[dict[str, float], list[int]]
'typing.Union[dict[str, float], list[int]]'
```

Automerge-Triggered-By: GH:gvanrossum
2020-11-01 10:13:38 -08:00
Raymond Hettinger 148c76b27c
Expand and clarify the "Invoking Descriptors" section of the Descriptor HowTo (GH-23078) 2020-11-01 09:10:06 -08:00
Dong-hee Na 7feb54a634
bpo-37483: Add PyObject_CallOneArg() in the What's New in Python 3.9 (GH-23062) 2020-11-01 22:04:35 +09:00
Inada Naoki d2810054c7
tempfile: Use random.choises() instead of choise() (GH-23068) 2020-11-01 20:02:03 +09:00
Ronald Oussoren 2165cea548
bpo-29566: binhex.binhex now consitently writes MacOS 9 line endings. (GH-23059)
[bpo-29566]() notes that binhex.binhex uses inconsistent line endings (both Unix and MacOS9 line endings are used). This PR changes this to use the MacOS9 line endings everywhere.
2020-11-01 01:08:48 -08:00
Alexey Izbyshev d3b4e06807
bpo-42146: Unify cleanup in subprocess_fork_exec() (GH-22970)
* bpo-42146: Unify cleanup in subprocess_fork_exec()

Also ignore errors from _enable_gc():
* They are always suppressed by the current code due to a bug.
* _enable_gc() is only used if `preexec_fn != None`, which is unsafe.
* We don't have a good way to handle errors in case we successfully
  created a child process.

Co-authored-by: Gregory P. Smith <greg@krypto.org>
2020-10-31 22:33:08 -07:00