Commit Graph

12174 Commits

Author SHA1 Message Date
Gregory P. Smith da5c808fb5
bpo-36515: Disable unaligned memory access in _sha3 on ARM (GH-25927)
Contributed-By: Matthias Klose

Automerge-Triggered-By: GH:tiran
2021-05-05 13:55:33 -07:00
Brad Larsen 23362f8c30
Eliminate duplicated assignment in _randommodule.c (GH-25904) 2021-05-04 20:41:11 -07:00
Raymond Hettinger 6fdc4d37f3
bpo-40521: Convert deque freelist from global vars to instance vars (GH-25906) 2021-05-04 17:08:31 -07:00
Ken Jin 2f5baa1750
Add C-API tests (#25886) 2021-05-04 13:07:31 +02:00
Dennis Sweeney 80a2a4ed7d
bpo-38530: Refactor and improve AttributeError suggestions (GH-25776)
- Make case-swaps half the cost of any other edit
- Refactor Levenshtein code to not use memory allocator, and to bail early on no match.
- Add comments to Levenshtein distance code
- Add test cases for Levenshtein distance behind a debug macro
- Set threshold to `(name_size + item_size + 3) * MOVE_COST / 6`.
  - Reasoning: similar to `difflib.SequenceMatcher.ratio()` >= 2/3:
```
"Multiset Jaccard similarity" >= 2/3
matching letters / total letters >= 2/3
(name_size - distance + item_size - distance) / (name_size + item_size) >= 2/3
1 - (2*distance) / (name_size + item_size) >= 2/3
1/3 >= (2*distance) / (name_size + item_size)
(name_size + item_size) / 6 >= distance
With rounding:
(name_size + item_size + 3) // 6 >= distance
```

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2021-05-03 16:47:27 +01:00
Pablo Galindo c2931d31f8
bpo-43916: Move the _PyStructSequence_InitType function to the internal API (GH-25854) 2021-05-03 15:50:24 +01:00
Batuhan Taskaya ad106c68eb
bpo-42725: Render annotations effectless on symbol table with PEP 563 (GH-25583) 2021-05-03 10:43:00 +03:00
Mark Shannon 33ec88ac81
bpo-43977: Make sure that tp_flags for pattern matching are inherited correctly. (GH-25813) 2021-05-03 00:38:22 +01:00
Erlend Egeberg Aasland cbb7b9ed4a
bpo-43434: Clean up sqlite3.connect() after GH-25818 (GH-25823) 2021-05-02 23:10:51 +01:00
Erlend Egeberg Aasland c96cc089f6
bpo-43434: Move sqlite3.connect audit events to sqlite3.Connection.__init__ (GH-25818) 2021-05-02 22:25:17 +01:00
Zackery Spytz 73766b0341
bpo-32745: Fix a regression in the handling of ctypes' c_wchar_p type (#8721)
Embedded nulls would cause a ValueError to be raised. Thanks go to
Eryk Sun for their analysis.

Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2021-05-02 12:40:01 +02:00
Christian Heimes 91554e4c5c
bpo-43908: Mark ssl, hash, and hmac types as immutable (GH-25792)
Signed-off-by: Christian Heimes <christian@python.org>
2021-05-02 09:47:45 +02:00
Christian Heimes e983252b51
bpo-43998: Default to TLS 1.2 and increase cipher suite security (GH-25778)
The ssl module now has more secure default settings. Ciphers without forward
secrecy or SHA-1 MAC are disabled by default. Security level 2 prohibits
weak RSA, DH, and ECC keys with less than 112 bits of security.
:class:`~ssl.SSLContext` defaults to minimum protocol version TLS 1.2.
Settings are based on Hynek Schlawack's research.

```
$ openssl version
OpenSSL 1.1.1k  FIPS 25 Mar 2021
$ openssl ciphers -v '@SECLEVEL=2:ECDH+AESGCM:ECDH+CHACHA20:ECDH+AES:DHE+AES:!aNULL:!eNULL:!aDSS:!SHA1:!AESCCM'
TLS_AES_256_GCM_SHA384  TLSv1.3 Kx=any      Au=any  Enc=AESGCM(256) Mac=AEAD
TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 Kx=any      Au=any  Enc=CHACHA20/POLY1305(256) Mac=AEAD
TLS_AES_128_GCM_SHA256  TLSv1.3 Kx=any      Au=any  Enc=AESGCM(128) Mac=AEAD
TLS_AES_128_CCM_SHA256  TLSv1.3 Kx=any      Au=any  Enc=AESCCM(128) Mac=AEAD
ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(256) Mac=AEAD
ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD
ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(128) Mac=AEAD
ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(128) Mac=AEAD
ECDHE-ECDSA-CHACHA20-POLY1305 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=CHACHA20/POLY1305(256) Mac=AEAD
ECDHE-RSA-CHACHA20-POLY1305 TLSv1.2 Kx=ECDH     Au=RSA  Enc=CHACHA20/POLY1305(256) Mac=AEAD
ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA384
ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA384
ECDHE-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA256
ECDHE-RSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(128)  Mac=SHA256
DHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(256) Mac=AEAD
DHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(128) Mac=AEAD
DHE-RSA-AES256-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA256
DHE-RSA-AES128-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA256
```

Signed-off-by: Christian Heimes <christian@python.org>
2021-05-01 20:53:10 +02:00
Ma Lin 251ffa9d2b
bpo-41486: Fix initial buffer size can't > UINT32_MAX in zlib module (GH-25738)
* Fix initial buffer size can't > UINT32_MAX in zlib module

After commit f9bedb630e, in 64-bit build,
if the initial buffer size > UINT32_MAX, ValueError will be raised.

These two functions are affected:
1. zlib.decompress(data, /, wbits=MAX_WBITS, bufsize=DEF_BUF_SIZE)
2. zlib.Decompress.flush([length])

This commit re-allows the size > UINT32_MAX.

* adds curly braces per PEP 7.

* Renames `Buffer_*` to `OutputBuffer_*` for clarity
2021-04-30 16:32:49 -07:00
Victor Stinner 665c7746fc
bpo-43916: _md5.md5 uses Py_TPFLAGS_DISALLOW_INSTANTIATION (GH-25753)
The following types use Py_TPFLAGS_DISALLOW_INSTANTIATION flag:

* _md5.md5
* _sha1.sha1
* _sha256.sha224
* _sha256.sha256
* _sha512.sha384
* _sha512.sha512
2021-04-30 18:40:30 +02:00
Victor Stinner 7dcf0f6db3
bpo-43916: select.devpoll uses Py_TPFLAGS_DISALLOW_INSTANTIATION (GH-25751) 2021-04-30 18:19:57 +02:00
Erlend Egeberg Aasland 9746cda705
bpo-43916: Apply Py_TPFLAGS_DISALLOW_INSTANTIATION to selected types (GH-25748)
Apply Py_TPFLAGS_DISALLOW_INSTANTIATION to the following types:

* _dbm.dbm
* _gdbm.gdbm
* _multibytecodec.MultibyteCodec
* _sre..SRE_Scanner
* _thread._localdummy
* _thread.lock
* _winapi.Overlapped
* array.arrayiterator
* functools.KeyWrapper
* functools._lru_list_elem
* pyexpat.xmlparser
* re.Match
* re.Pattern
* unicodedata.UCD
* zlib.Compress
* zlib.Decompress
2021-04-30 16:04:57 +02:00
Erlend Egeberg Aasland 387397f8a4
bpo-43916: select.poll uses Py_TPFLAGS_DISALLOW_INSTANTIATION (GH-25750) 2021-04-30 15:49:17 +02:00
Victor Stinner 0cad068ec1
bpo-43916: Remove _disabled_new() function (GH-25745)
posix and _hashlib use the new Py_TPFLAGS_DISALLOW_INSTANTIATION
flag on their heap types, rather than using a custom tp_new function
(_disabled_new).
2021-04-30 14:06:49 +02:00
Victor Stinner 3bb09947ec
bpo-43916: Add Py_TPFLAGS_DISALLOW_INSTANTIATION type flag (GH-25721)
Add a new Py_TPFLAGS_DISALLOW_INSTANTIATION type flag to disallow
creating type instances: set tp_new to NULL and don't create the
"__new__" key in the type dictionary.

The flag is set automatically on static types if tp_base is NULL or
&PyBaseObject_Type and tp_new is NULL.

Use the flag on the following types:

* _curses.ncurses_version type
* _curses_panel.panel
* _tkinter.Tcl_Obj
* _tkinter.tkapp
* _tkinter.tktimertoken
* _xxsubinterpretersmodule.ChannelID
* sys.flags type
* sys.getwindowsversion() type
* sys.version_info type

Update MyStr example in the C API documentation to use
Py_TPFLAGS_DISALLOW_INSTANTIATION.

Add _PyStructSequence_InitType() function to create a structseq type
with the Py_TPFLAGS_DISALLOW_INSTANTIATION flag set.

type_new() calls _PyType_CheckConsistency() at exit.
2021-04-30 12:46:15 +02:00
Mark Shannon 069e81ab3d
bpo-43977: Use tp_flags for collection matching (GH-25723)
* Add Py_TPFLAGS_SEQUENCE and Py_TPFLAGS_MAPPING, add to all relevant standard builtin classes.

* Set relevant flags on collections.abc.Sequence and Mapping.

* Use flags in MATCH_SEQUENCE and MATCH_MAPPING opcodes.

* Inherit Py_TPFLAGS_SEQUENCE and Py_TPFLAGS_MAPPING.

* Add NEWS

* Remove interpreter-state map_abc and seq_abc fields.
2021-04-30 09:50:28 +01:00
Erlend Egeberg Aasland c6ad03fddf
bpo-43908: Make array.array type immutable (GH-25696)
Co-authored-by: Victor Stinner <vstinner@python.org>
2021-04-29 08:47:48 +02:00
Erlend Egeberg Aasland 5daf70b22e
bpo-43908: Make re types immutable (GH-25697)
Co-authored-by: Victor Stinner <vstinner@python.org>
2021-04-29 08:47:11 +02:00
Victor Stinner 103d5e420d
bpo-28254: _posixsubprocess uses PyGC_Enable/PyGC_Disable (GH-25693) 2021-04-28 19:09:29 +02:00
scoder 3cc481b9de
bpo-28254: Add a C-API for controlling the GC state (GH-25687)
Add new C-API functions to control the state of the garbage collector:
PyGC_Enable(), PyGC_Disable(), PyGC_IsEnabled(),
corresponding to the functions in the gc module.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
2021-04-28 18:12:16 +02:00
Ma Lin f9bedb630e
bpo-41486: Faster bz2/lzma/zlib via new output buffering (GH-21740)
Faster bz2/lzma/zlib via new output buffering.
Also adds .readall() function to _compression.DecompressReader class
to take best advantage of this in the consume-all-output at once scenario.

Often a 5-20% speedup in common scenarios due to less data copying.

Contributed by Ma Lin.
2021-04-27 23:58:54 -07:00
Victor Stinner a5e64444e6
bpo-43963: Add _signal module state (GH-25676)
* Add signal_state_t structure and signal_global_state variable.
* Add a module state to the _signal module.
* Move and rename variables:

  * DefaultHandler becomes state->default_handler
  * IgnoreHandler becomes state->ignore_handler
  * sigint_event becomes state->sigint_event
  * ItimerError becomes modstate->itimer_error

* Rename SetHandler() to set_handler() to be consistent with
  get_handler().
2021-04-28 03:02:55 +02:00
Victor Stinner a09766deab
bpo-43963: Fix import _signal in subinterpreters (GH-25674)
Importing the _signal module in a subinterpreter has no longer side
effects.

signal_module_exec() no longer modifies Handlers and no longer attempts
to set SIGINT signal handler in subinterpreters.
2021-04-28 01:50:04 +02:00
Ma Lin 93f411838a
Fix thread locks in zlib module may go wrong in rare case. (#22126)
Setting `next_in` before acquiring the thread lock may mix up compress/decompress state in other threads.
2021-04-27 10:37:11 +02:00
Erlend Egeberg Aasland 7244c0060d
bpo-43762: Add audit events for loading of sqlite3 extensions (GH-25246) 2021-04-27 00:16:46 +01:00
Christian Heimes 666991fc59
bpo-18233: Add internal methods to access peer chain (GH-25467)
The internal `_ssl._SSLSocket` object now provides methods to retrieve
the peer cert chain and verified cert chain as a list of Certificate
objects. Certificate objects have methods to convert the cert to a dict,
PEM, or DER (ASN.1).

These are private APIs for now. There is a slim chance to stabilize the
approach and provide a public API for 3.10. Otherwise I'll provide a
stable API in 3.11.

Signed-off-by: Christian Heimes <christian@python.org>
2021-04-26 15:01:40 +02:00
Serhiy Storchaka 172c0f2752
bpo-39529: Deprecate creating new event loop in asyncio.get_event_loop() (GH-23554)
asyncio.get_event_loop() emits now a deprecation warning when it creates a new event loop.
In future releases it will became an alias of asyncio.get_running_loop().
2021-04-25 13:40:44 +03:00
Segev Finer 5e437fb872
bpo-30555: Fix WindowsConsoleIO fails in the presence of fd redirection (GH-1927)
This works by not caching the handle and instead getting the handle from
the file descriptor each time, so that if the actual handle changes by
fd redirection closing/opening the console handle beneath our feet, we
will keep working correctly.
2021-04-23 23:00:27 +01:00
Steve Dower 019e9e8168
bpo-43538: Add extra arguments to os.startfile (GH-25538) 2021-04-23 18:03:17 +01:00
Zackery Spytz 7d37b86ad4
bpo-35114: Make ssl.RAND_status() return a bool (GH-20063) 2021-04-23 18:07:37 +02:00
Christian Heimes b9ad88be03
bpo-43920: Make load_verify_locations(cadata) error message consistent (GH-25554)
Signed-off-by: Christian Heimes <christian@python.org>
2021-04-23 13:51:40 +02:00
Erlend Egeberg Aasland e9194ea6ea
bpo-43852: Improve tuple creation in sqlite3 (GH-25421) 2021-04-23 14:21:08 +03:00
Steve Dower e07d809889
bpo-38222: Check specifically for a drive, not just a colon (GH-25540) 2021-04-23 00:30:37 +01:00
Steve Dower fe63a401a9
bpo-38822: Fixed os.stat failing on inaccessible directories. (GH-25527)
It would just fail if the path was inaccessible and had a trailing slash. It should fall back to the parent directory's metadata.
2021-04-22 20:45:02 +01:00
Raymond Hettinger a07da09ad5
bpo-43475: Fix worst case collision behavior for NaN instances (GH-25493) 2021-04-22 08:34:57 -07:00
Steve Dower dc516ef839
bpo-26227: Fixes decoding of host names on Windows from ANSI instead of UTF-8 (GH-25510) 2021-04-22 00:18:20 +01:00
Victor Stinner cdad2724e6
bpo-40137: Add pycore_moduleobject.h internal header (GH-25507)
Add pycore_moduleobject.h internal header file with static inline
functions to access module members:

* _PyModule_GetDict()
* _PyModule_GetDef()
* _PyModule_GetState()

These functions don't check at runtime if their argument has a valid
type and can be inlined even if Python is not built with LTO.

_PyType_GetModuleByDef() uses _PyModule_GetDef().

Replace PyModule_GetState() with _PyModule_GetState() in the
extension modules, considered as performance sensitive:

* _abc
* _functools
* _operator
* _pickle
* _queue
* _random
* _sre
* _struct
* _thread
* _winapi
* array
* posix

The following extensions are now built with the Py_BUILD_CORE_MODULE
macro defined, to be able to use the internal pycore_moduleobject.h
header: _abc, array, _operator, _queue, _sre, _struct.
2021-04-22 00:52:52 +02:00
Steve Dower 7b86e47617
bpo-43472: Ensure PyInterpreterState_New audit events are raised when called through _xxsubinterpreters module (GH-25506) 2021-04-21 23:34:29 +01:00
Raymond Hettinger 139c232f38
bpo-40137: Move state lookups out of the critical path (GH-25492) 2021-04-21 15:22:22 -07:00
Christian Heimes 3309113d61
bpo-43799: Also define SSLv3_method() (GH-25481)
Signed-off-by: Christian Heimes <christian@python.org>
2021-04-20 18:10:10 +02:00
l0x 64d975202f
bpo-40849: Expose X509_V_FLAG_PARTIAL_CHAIN ssl flag (GH-20463)
This short PR exposes an openssl flag that  wasn't exposed. I've also updated to doc to reflect the change. It's heavily inspired by 990fcaac3c.
2021-04-19 04:51:18 -07:00
Christian Heimes 2875c603b2
bpo-43880: Show DeprecationWarnings for deprecated ssl module features (GH-25455)
* ssl.OP_NO_SSLv2
* ssl.OP_NO_SSLv3
* ssl.OP_NO_TLSv1
* ssl.OP_NO_TLSv1_1
* ssl.OP_NO_TLSv1_2
* ssl.OP_NO_TLSv1_3
* ssl.PROTOCOL_SSLv2
* ssl.PROTOCOL_SSLv3
* ssl.PROTOCOL_SSLv23 (alias for PROTOCOL_TLS)
* ssl.PROTOCOL_TLS
* ssl.PROTOCOL_TLSv1
* ssl.PROTOCOL_TLSv1_1
* ssl.PROTOCOL_TLSv1_2
* ssl.TLSVersion.SSLv3
* ssl.TLSVersion.TLSv1
* ssl.TLSVersion.TLSv1_1
* ssl.wrap_socket()
* ssl.RAND_pseudo_bytes()
* ssl.RAND_egd() (already removed since it's not supported by OpenSSL 1.1.1)
* ssl.SSLContext() without a protocol argument
* ssl.match_hostname()
* hashlib.pbkdf2_hmac() (pure Python implementation, fast OpenSSL
  function will stay)

Signed-off-by: Christian Heimes <christian@python.org>
2021-04-19 07:27:10 +02:00
Christian Heimes 89d1550d14
bpo-42854: Use SSL_read/write_ex() (GH-25468)
The ssl module now uses ``SSL_read_ex`` and ``SSL_write_ex``
internally. The functions support reading and writing of data larger
than 2 GB. Writing zero-length data no longer fails with a protocol
violation error.

Signed-off-by: Christian Heimes <christian@python.org>
2021-04-19 06:55:30 +02:00
Christian Heimes aa6da32edc
bpo-43362: Fix invalid free and return check in _sha3 module (GH-25463)
Commit 93d50a6a8d / GH-21855 changed the
order of variable definitions, which introduced a potential invalid free
bug. Py_buffer object is now initialized earlier and the result of
Keccak initialize is verified.

Co-authored-by: Alex Henrie <alexhenrie24@gmail.com>
Signed-off-by: Christian Heimes <christian@python.org>

Co-authored-by: Alex Henrie <alexhenrie24@gmail.com>
2021-04-18 08:39:39 +02:00
Christian Heimes 7f1305ef9e
bpo-42333: Port _ssl extension to multiphase initialization (PEP 489) (GH-23253)
- Introduce sslmodule_slots
- Introduce sslmodulestate
- Use sslmodulestate
- Get rid of PyState_FindModule
- Move new structs and helpers to header file
- Use macros to access state
- Keep a strong ref to socket type
2021-04-17 20:06:38 +02:00