Commit Graph

11965 Commits

Author SHA1 Message Date
Serhiy Storchaka b4c98ed41e
bpo-41288: Refactor of unpickling NEWOBJ and NEWOBJ_EX opcodes. (GH-21472)
* Share code for NEWOBJ and NEWOBJ_EX.
* More detailed error messages.
2020-07-18 11:11:21 +03:00
Mark Shannon cb9879b948
bpo-40941: Unify implicit and explicit state in the frame and generator objects into a single value. (GH-20803)
* Merge gen and frame state variables into one.

* Replace stack pointer with depth in PyFrameObject. Makes code easier to read and saves a word of memory.
2020-07-17 11:44:23 +01:00
matthewhughes934 8e836bb21c
bpo-41195: Add getter for Openssl security level (GH-21282)
Add an accessor under SSLContext.security_level as a wrapper around
SSL_CTX_get_security_level, see:
https://www.openssl.org/docs/manmaster/man3/SSL_CTX_get_security_level.html


------
This is my first time contributing, so please pull me up on all the things I missed or did incorrectly.

Automerge-Triggered-By: @tiran
2020-07-17 01:59:15 -07:00
Zackery Spytz af4eda46d1
bpo-40150: Fix mismatched argument in RegisterWaitForSingleObject() call (GH-19686) 2020-07-15 21:43:00 +03:00
Benjamin Peterson 0108b2a240
Fix -Wstring-prototypes warnings in _zoneinfo.c. (GH-21478) 2020-07-15 12:02:14 -05:00
Felix Yan 015efdbef7
bpo-41302: Fix build with system libmpdec (GH-21481)
Move definition of UNUSED from modified headers of libmpdec to
_decimal.c itself. This makes the vendored source closer to the
standalone library and fixes build with --with-system-libmpdec.

Tested to build fine with either system libmpdec or the vendored one.
2020-07-15 14:14:11 +02:00
Zackery Spytz bbceef6851
bpo-20183: Convert _locale to the Argument Clinic (GH-14201) 2020-07-15 12:07:34 +03:00
Serhiy Storchaka 4f309abf55
bpo-41288: Fix a crash in unpickling invalid NEWOBJ_EX. (GH-21458)
Automerge-Triggered-By: @tiran
2020-07-13 05:49:26 -07:00
Serhiy Storchaka b0689ae7f9
bpo-41146: Convert signal.default_int_handler() to Argument Clinic (GH-21197) 2020-07-12 19:15:20 +03:00
Zackery Spytz 545b54d2ab
bpo-20175: Convert Modules/_multiprocessing to the Argument Clinic (GH-14245) 2020-07-12 19:11:11 +03:00
Zackery Spytz b7047e59a4
bpo-20181: Convert the readline module to the Argument Clinic (#14326) 2020-07-12 19:01:03 +03:00
Serhiy Storchaka 4c8f09d7ce
bpo-36346: Make using the legacy Unicode C API optional (GH-21437)
Add compile time option USE_UNICODE_WCHAR_CACHE. Setting it to 0
makes the interpreter not using the wchar_t cache and the legacy Unicode C API.
2020-07-10 23:26:06 +03:00
Zackery Spytz 9650fe0197
bpo-20179: Convert the _overlapped module to the Argument Clinic (GH-14275) 2020-07-10 20:43:37 +03:00
Victor Stinner 8182cc2e68
bpo-39573: Use the Py_TYPE() macro (GH-21433)
Replace obj->ob_type with Py_TYPE(obj).
2020-07-10 12:40:38 +02:00
Serhiy Storchaka d878349bac
bpo-36346: Do not use legacy Unicode C API in ctypes. (#21429) 2020-07-10 11:17:21 +03:00
Zackery Spytz ee96f32ca2
bpo-41252: Fix incorrect refcounting in _ssl.c's _servername_callback() (GH-21407) 2020-07-09 03:00:21 -07:00
Tony Solomonik 529f42645d
bpo-41247: asyncio.set_running_loop() cache running loop holder (GH-21401)
The running loop holder cache variable was always set to NULL when
calling set_running_loop.

Now set_running_loop saves the newly created running loop holder in the
cache variable for faster access in get_running_loop.

Automerge-Triggered-By: @1st1
2020-07-08 12:27:31 -07:00
Zackery Spytz aebc049557
closes bpo-41235: Fix the error handling in SSLContext.load_dh_params() (GH-21385) 2020-07-07 23:21:58 -05:00
Victor Stinner 8f42748ded
bpo-29778: test_embed tests the path configuration (GH-21306) 2020-07-08 00:20:37 +02:00
Steve Dower dcbaa1b49c
bpo-29778: Ensure python3.dll is loaded from correct locations when Python is embedded (GH-21297)
Also enables using debug build of `python3_d.dll`
Reference: CVE-2020-15523
2020-07-06 17:32:00 +01:00
Pablo Galindo e51dd9dad6
bpo-29727: Register array.array as a MutableSequence (GH-21338) 2020-07-05 22:43:14 +01:00
Serhiy Storchaka b3dd5cd4a3
bpo-36346: Undeprecate private function _PyUnicode_AsUnicode(). (GH-21336) 2020-07-05 18:53:45 +03:00
Inada Naoki 13c90e82b6
Uncomment Py_DEPRECATED for Py_UNICODE APIs (GH-21318)
PyUnicode_EncodeDecimal and PyUnicode_TransformDecimalToASCII
are deprecated since Python 3.3.
But Py_DEPRECATED(3.3) was commented out.
2020-07-05 11:01:54 +09:00
Dong-hee Na c0b214bc08
bpo-1635741: Port faulthandler module to multiphase initialization (GH-21294) 2020-07-04 01:36:47 +09:00
Mohamed Koubaa 9d006977d7
bpo-1635741: Port sha256 module to multiphase init (PEP 489) (GH-21189) 2020-07-03 17:59:47 +09:00
scoder 148f329135
bpo-39960: Allow heap types in the "Carlo Verre" hack check that override "tp_setattro()" (GH-21092)
Automerge-Triggered-By: @gvanrossum
2020-07-02 17:09:28 -07:00
Rémi Lapeyre 004e64e805
bpo-40967: Remove deprecated asyncio.Task.current_task() and asyncio.Task.all_tasks() (GH-20874) 2020-07-01 20:41:21 -07:00
Lawrence D'Anna 604d95e235
bpo-41100: fix _decimal for arm64 Mac OS (GH-21228)
Patch by Lawrence Danna.
2020-06-30 11:15:46 +02:00
Serhiy Storchaka 349f76c6aa
bpo-36346: Prepare for removing the legacy Unicode C API (AC only). (GH-21223) 2020-06-30 09:03:15 +03:00
Stefan Krah 8bea91b5e9
bpo-40874 Update the required libmpdec version for the decimal module (GH-21202) 2020-06-28 22:01:01 +02:00
Zackery Spytz cd3c2bdd5d
bpo-31082: Use "iterable" in the docstring for functools.reduce() (GH-20796) 2020-06-28 15:40:54 +09:00
Guido van Rossum 9d197c7d48
bpo-35975: Only use cf_feature_version if PyCF_ONLY_AST in cf_flags (#21021) 2020-06-27 17:33:49 -07:00
Christian Heimes 9cfcdb7d6e
Remove dead code from tracemalloc (GH-21029)
tracemalloc_get_frame() checked filename == NULL two times in a row.

Signed-off-by: Christian Heimes <christian@python.org>
2020-06-27 14:13:47 +02:00
Serhiy Storchaka 700cfa8c90
bpo-41069: Make TESTFN and the CWD for tests containing non-ascii characters. (GH-21035) 2020-06-25 17:56:31 +03:00
Serhiy Storchaka a7dc714701
bpo-41094: Additional fix for PYTHONSTARTUP. (GH-21119) 2020-06-24 19:46:30 +03:00
Serhiy Storchaka 6c6810d989
bpo-41094: Fix decoding errors with audit when open files. (GH-21095) 2020-06-24 08:46:05 +03:00
Anthony Sottile d051801052
bpo-31938: Fix default-value signatures of several functions in the select module (GH-21066) 2020-06-23 23:49:56 +02:00
Raymond Hettinger 930f4518ae
On path with known exact float, extract the double with the fast macro. (GH-21072) 2020-06-23 11:45:25 -07:00
Raymond Hettinger 32f2eda859
bpo-40521: Remove freelist from collections.deque() (GH-21073) 2020-06-23 06:50:15 -07:00
WildCard65 1d3dad5f96
bpo-41085: Fix array.array.index() on 64-bit Windows (GH-21071)
Fix integer overflow in the :meth:`array.array.index` method on 64-bit Windows
for index larger than ``2**31``.
2020-06-23 15:21:16 +02:00
Victor Stinner b4e85cadfb
bpo-40521: Make dict free lists per-interpreter (GH-20645)
Each interpreter now has its own dict free list:

* Move dict free lists into PyInterpreterState.
* Move PyDict_MAXFREELIST define to pycore_interp.h
* Add _Py_dict_state structure.
* Add tstate parameter to _PyDict_ClearFreeList() and _PyDict_Fini().
* In debug mode, ensure that the dict free lists are not used after
  _PyDict_Fini() is called.
* Remove "#ifdef EXPERIMENTAL_ISOLATED_SUBINTERPRETERS".
2020-06-23 11:33:18 +02:00
Dong-hee Na 1937edd376
bpo-1635741: Port _lzma module to multiphase initialization (GH-19382) 2020-06-23 00:53:07 +09:00
Victor Stinner c45dbe93b7
bpo-41078: Add pycore_list.h internal header file (GH-21057)
* Move _PyList_ITEMS() to pycore_list.h.
* The C extension "_heapq" is now built with Py_BUILD_CORE_MODULE
  macro defined to access the internal C API.
2020-06-22 17:39:32 +02:00
Victor Stinner 384621c42f
bpo-41078: Rename pycore_tupleobject.h to pycore_tuple.h (GH-21056) 2020-06-22 17:27:35 +02:00
Serhiy Storchaka c88239f864
bpo-26407: Do not mask errors in csv. (GH-20536)
Unexpected errors in calling the __iter__ method are no longer
masked by TypeError in csv.reader(), csv.writer.writerow() and
csv.writer.writerows().
2020-06-22 11:21:59 +03:00
Christian Heimes 4901ea9526
bpo-41061: Fix incorrect expressions in hashtable (GH-21028)
Signed-off-by: Christian Heimes <christian@python.org>
2020-06-22 00:41:48 -07:00
Gregory P. Smith d780fa7931
bpo-41056: Fix a possible MemoryError leak within zoneinfo. (GH-21007)
This was detected by our Coverity scan as a REVERSE_INULL issue.

Automerge-Triggered-By: @gpshead
2020-06-22 00:39:28 -07:00
Dong-hee Na 6989af0bc7
bpo-41052: Opt out serialization/deserialization for _random.Random (GH-21002) 2020-06-21 18:44:58 +09:00
Gregory P. Smith 3ccb96c978
bpo-41056: Use the fildes converter for fd to please Coverity. (GH-21011)
There are a bunch of other fd: int uses in this file, I expect many if not
all of them would be better off using the fildes converter.  This particular
one was flagged by Coverity as it presumably flags fpathconf as not accepting
negative fds.  I'd expect the other fd's to have been flagged as well
otherwise.

I'm marking this one as skip news as it really is a no-op.
2020-06-20 15:06:48 -07:00
Gregory P. Smith eb0d5c38de
bpo-41056: Fix a NULL pointer dereference on MemoryError within the ssl module. (GH-21009)
Detected by Coverity.
2020-06-20 12:15:03 -07:00
Dong-hee Na ec68918795
bpo-40077: Convert _bz2 module to use PyType_FromSpec (GH-20960) 2020-06-20 00:56:13 +09:00
Victor Stinner 37bb289556
bpo-40943: PY_SSIZE_T_CLEAN required for '#' formats (GH-20784)
The PY_SSIZE_T_CLEAN macro must now be defined to use
PyArg_ParseTuple() and Py_BuildValue() "#" formats: "es#", "et#",
"s#", "u#", "y#", "z#", "U#" and "Z#". See the PEP 353.

Update _testcapi.test_buildvalue_issue38913().
2020-06-19 11:45:31 +02:00
Inada Naoki 2c4928d37e
bpo-36346: Add Py_DEPRECATED to deprecated unicode APIs (GH-20878)
Co-authored-by: Kyle Stanley <aeros167@gmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
2020-06-17 20:09:44 +09:00
Dong-hee Na c4862e333a
bpo-1635741: Port _gdbm module to multiphase initialization (GH-20920) 2020-06-17 01:41:23 +09:00
David Szotten 8666356280
closes bpo-28557: error message for bad raw readinto (GH-7496)
Co-authored-by: Benjamin Peterson <benjamin@python.org>
2020-06-15 18:53:57 -05:00
Victor Stinner 04fc4f2a46
bpo-40989: PyObject_INIT() becomes an alias to PyObject_Init() (GH-20901)
The PyObject_INIT() and PyObject_INIT_VAR() macros become aliases to,
respectively, PyObject_Init() and PyObject_InitVar() functions.

Rename _PyObject_INIT() and _PyObject_INIT_VAR() static inline
functions to, respectively, _PyObject_Init() and _PyObject_InitVar(),
and move them to pycore_object.h. Remove their return value:
their return type becomes void.

The _datetime module is now built with the Py_BUILD_CORE_MODULE macro
defined.

Remove an outdated comment on _Py_tracemalloc_config.
2020-06-16 01:28:07 +02:00
Victor Stinner e822e37946
bpo-36020: Remove snprintf macro in pyerrors.h (GH-20889)
On Windows, #include "pyerrors.h" no longer defines "snprintf" and
"vsnprintf" macros.

PyOS_snprintf() and PyOS_vsnprintf() should be used to get portable
behavior.

Replace snprintf() calls with PyOS_snprintf() and replace vsnprintf()
calls with PyOS_vsnprintf().
2020-06-15 21:59:47 +02:00
Dong-hee Na bf69a8f99f
bpo-1635741: Port _dbm module to multiphase initialization (GH-20848) 2020-06-16 01:20:54 +09:00
Niklas Fiekas 794e7d1ab2
bpo-29782: Consolidate _Py_Bit_Length() (GH-20739)
In GH-2866, _Py_Bit_Length() was added to pymath.h for lack of a better
location. GH-20518 added a more appropriate header file for bit utilities. It
also shows how to properly use intrinsics. This allows reconsidering bpo-29782.

* Move the function to the new header.
* Changed return type to match __builtin_clzl() and reviewed usage.
* Use intrinsics where available.
* Pick a fallback implementation suitable for inlining.
2020-06-15 14:33:48 +02:00
Christian Heimes 0d3350daa8
bpo-40955: Fix memory leak in subprocess module (GH-20825)
```
Direct leak of 8 byte(s) in 1 object(s) allocated from:
    #0 0x7f008bf19667 in __interceptor_malloc (/lib64/libasan.so.6+0xb0667)
    #1 0x7f007a0bee4a in subprocess_fork_exec /home/heimes/dev/python/cpython/Modules/_posixsubprocess.c:774
    #2 0xe0305b in cfunction_call Objects/methodobject.c:546
```

Signed-off-by: Christian Heimes <christian@python.org>
2020-06-12 09:18:43 -07:00
Dong-hee Na ddef3bdc7b
bpo-40950: Port nis module to multiphase initialization (GH-20811) 2020-06-12 11:26:00 +09:00
Pablo Galindo 1ed83adb0e
bpo-40939: Remove the old parser (GH-20768)
This commit removes the old parser, the deprecated parser module, the old parser compatibility flags and environment variables and all associated support code and documentation.
2020-06-11 17:30:46 +01:00
Victor Stinner d72b9644a3
bpo-40947: getpath.c uses PyConfig.platlibdir (GH-20807)
Followup of bpo-40854, there is one remaining usage of PLATLIBDIR
which should be replaced by config->platlibdir.

test_sys checks that sys.platlibdir attribute exists and is a string.

Update Makefile: getpath.c and sysmodule.c no longer need PLATLIBDIR
macro, PyConfig.platlibdir member is used instead.

Co-authored-by: Sandro Mani <manisandro@gmail.com>
2020-06-11 17:28:52 +02:00
Victor Stinner 1bcc32f062
bpo-39465: Use _PyInterpreterState_GET() (GH-20788)
Replace _PyThreadState_GET() with _PyInterpreterState_GET() in:

* get_small_int()
* gcmodule.c: add also get_gc_state() function
* _PyTrash_deposit_object()
* _PyTrash_destroy_chain()
* warnings_get_state()
* Py_GetRecursionLimit()

Cleanup listnode.c: add 'parser' variable.
2020-06-10 20:08:26 +02:00
Victor Stinner d36cf5f1d2
bpo-40943: Replace PY_FORMAT_SIZE_T with "z" (GH-20781)
The PEP 353, written in 2005, introduced PY_FORMAT_SIZE_T. Python no
longer supports macOS 10.4 and Visual Studio 2010, but requires more
recent macOS and Visual Studio versions. In 2020 with Python 3.10, it
is now safe to use directly "%zu" to format size_t and "%zi" to
format Py_ssize_t.
2020-06-10 18:38:05 +02:00
Dong-hee Na 0383be4666
Remove usesless function from csv module (GH-20762) 2020-06-10 00:33:43 +09:00
Stefan Krah 951d680d56
Add multicore support to deccheck.py. (GH-20731) 2020-06-08 19:33:12 +02:00
Sandro Mani 8f023a2f66
bpo-40854: Allow overriding sys.platlibdir via PYTHONPLATLIBDIR env-var (GH-20605) 2020-06-08 17:28:11 +02:00
Victor Stinner c6b292cdee
bpo-29882: Add _Py_popcount32() function (GH-20518)
* Rename pycore_byteswap.h to pycore_bitutils.h.
* Move popcount_digit() to pycore_bitutils.h as _Py_popcount32().
* _Py_popcount32() uses GCC and clang builtin function if available.
* Add unit tests to _Py_popcount32().
2020-06-08 16:30:33 +02:00
Hai Shi 47a23fc63f
bpo-40898: Remove redundant if statements in tp_traverse (GH-20692) 2020-06-07 21:05:36 +09:00
Rémi Lapeyre b8867e5d5a
Fix return type of test helper function heapctypewithbuffer_releasebuffer() (GH-20685) 2020-06-07 09:05:33 +02:00
scoder f7c4e23642
bpo-40724: Support setting buffer slots from type specs (GH-20648)
This is not part of the limited API but makes the buffer slots available for type specs.
2020-06-06 21:35:10 +02:00
Lysandros Nikolaou ba6fd87e41
Refactor scripts in Tools/peg_generator/scripts (GH-20401) 2020-06-05 21:21:40 -07:00
Erlend Egeberg Aasland 45af786e11
bpo-40867: Remove unused include from Module/_randommodule.c (GH-20635) 2020-06-05 14:32:09 -07:00
Ram Rachum 235f918f44
bpo-40876: Clarify error message in the csv module (GH-20653) 2020-06-05 17:56:06 -03:00
Stefan Krah 087d612efe
bpo-40874: Update to libmpdec-2.5.0 (GH-20652) 2020-06-05 19:43:01 +02:00
Dong-hee Na 3ad52e366f
bpo-1635741: Port mmap module to multiphase initialization (GH-19459) 2020-06-06 00:01:02 +09:00
Victor Stinner e005ead49b
bpo-40521: Make context free list per-interpreter (GH-20644)
Each interpreter now has its own context free list:

* Move context free list into PyInterpreterState.
* Add _Py_context_state structure.
* Add tstate parameter to _PyContext_ClearFreeList()
  and _PyContext_Fini().
* Pass tstate to clear_freelists().
2020-06-05 02:56:37 +02:00
Victor Stinner 78a02c2568
bpo-40521: Make async gen free lists per-interpreter (GH-20643)
Each interpreter now has its own asynchronous generator free lists:

* Move async gen free lists into PyInterpreterState.
* Move _PyAsyncGen_MAXFREELIST define to pycore_interp.h
* Add _Py_async_gen_state structure.
* Add tstate parameter to _PyAsyncGen_ClearFreeLists
  and _PyAsyncGen_Fini().
2020-06-05 02:34:14 +02:00
Victor Stinner 88ec919010
bpo-40521: Make list free list per-interpreter (GH-20642)
Each interpreter now has its own list free list:

* Move list numfree and free_list into PyInterpreterState.
* Add _Py_list_state structure.
* Add tstate parameter to _PyList_ClearFreeList()
  and _PyList_Fini().
* Remove "#ifdef EXPERIMENTAL_ISOLATED_SUBINTERPRETERS".
* _PyGC_Fini() clears gcstate->garbage list which can be stored in
  the list free list. Call _PyGC_Fini() before _PyList_Fini() to
  prevent leaking this list.
2020-06-05 02:05:41 +02:00
Victor Stinner 3744ed2c9c
bpo-40521: Make frame free list per-interpreter (GH-20638)
Each interpreter now has its own frame free list:

* Move frame free list into PyInterpreterState.
* Add _Py_frame_state structure.
* Add tstate parameter to _PyFrame_ClearFreeList()
  and _PyFrame_Fini().
* Remove "#if PyFrame_MAXFREELIST > 0".
* Remove "#ifdef EXPERIMENTAL_ISOLATED_SUBINTERPRETERS".
2020-06-05 01:39:24 +02:00
Victor Stinner 2ba59370c3
bpo-40521: Make float free list per-interpreter (GH-20636)
Each interpreter now has its own float free list:

* Move tuple numfree and free_list into PyInterpreterState.
* Add _Py_float_state structure.
* Add tstate parameter to _PyFloat_ClearFreeList()
  and _PyFloat_Fini().
2020-06-05 00:50:05 +02:00
Victor Stinner 69ac6e58fd
bpo-40521: Make tuple free list per-interpreter (GH-20247)
Each interpreter now has its own tuple free lists:

* Move tuple numfree and free_list arrays into PyInterpreterState.
* Define PyTuple_MAXSAVESIZE and PyTuple_MAXFREELIST macros in
  pycore_interp.h.
* Add _Py_tuple_state structure. Pass it explicitly to tuple_alloc().
* Add tstate parameter to _PyTuple_ClearFreeList()
* Each interpreter now has its own empty tuple singleton.
2020-06-04 23:38:36 +02:00
Erlend Egeberg Aasland 6ed578f6db
bpo-40865: Remove unused insint() macro from hash modules (GH-20627)
Automerge-Triggered-By: @tiran
2020-06-04 13:08:42 -07:00
Dong-hee Na 586be6f3ff
Remove unused ReaderObject_Check macro (#20614) 2020-06-04 00:43:46 +09:00
Victor Stinner fa7ab6aa0f
bpo-40826: Add _PyOS_InterruptOccurred(tstate) function (GH-20599)
my_fgets() now calls _PyOS_InterruptOccurred(tstate) to check for
pending signals, rather calling PyOS_InterruptOccurred().

my_fgets() is called with the GIL released, whereas
PyOS_InterruptOccurred() must be called with the GIL held.

test_repl: use text=True and avoid SuppressCrashReport in
test_multiline_string_parsing().

Fix my_fgets() on Windows: fgets(fp) does crash if fileno(fp) is closed.
2020-06-03 14:39:59 +02:00
Victor Stinner 18a90248fd
bpo-32604: Fix reference leak in select module (GH-20600)
Fix reference leak in PyInit_select() of the select module:
remove Py_INCREF(poll_Type).
2020-06-03 14:36:46 +02:00
Victor Stinner 317bab0bf6
PyOS_AfterFork_Child() pass tstate to _PyEval_ReInitThreads() (GH-20598) 2020-06-02 18:44:54 +02:00
Victor Stinner 26881c8fae
PyOS_AfterFork_Child() uses PyStatus (GH-20596)
PyOS_AfterFork_Child() helper functions now return a PyStatus:
PyOS_AfterFork_Child() is now responsible to handle errors.

* Move _PySignal_AfterFork() to the internal C API
* Add #ifdef HAVE_FORK on _PyGILState_Reinit(), _PySignal_AfterFork()
  and _PyInterpreterState_DeleteExceptMain().
2020-06-02 15:51:37 +02:00
Victor Stinner cbe1296922
bpo-40826: PyOS_InterruptOccurred() requires GIL (GH-20578)
PyOS_InterruptOccurred() now fails with a fatal error if it is called
with the GIL released.
2020-06-01 20:34:15 +02:00
Ammar Askar 06e3a27a3c
bpo-39943: Fix MSVC warnings in sre extension (GH-20508) 2020-06-01 19:21:43 +02:00
Hai Shi a97011b9b8
bpo-39593: Add test on ctypes cfield.c s_set() (GH-18424) 2020-06-01 18:54:18 +02:00
Dong-hee Na e9684fac5a
bpo-1635741: Port fcntl module to multiphase initialization (GH-20540) 2020-06-02 01:12:24 +09:00
Christian Heimes a871f692b4
bpo-30008: Fix OpenSSL no-deprecated compilation (GH-20397)
Fix :mod:`ssl`` code to be compatible with OpenSSL 1.1.x builds that use
``no-deprecated`` and ``--api=1.1.0``.

Note: Tests assume full OpenSSL API and fail with limited API.

Signed-off-by: Christian Heimes <christian@python.org>
Co-authored-by: Mark Wright <gienah@gentoo.org>
2020-06-01 08:58:14 +02:00
Zackery Spytz 7b78e7f9fd
bpo-40061: Fix a possible refleak in _asynciomodule.c (GH-19748)
tup should be decrefed in the unlikely event of a PyList_New()
failure.
2020-05-30 01:22:02 -07:00
Petr Viktorin 459acc5516
bpo-40777: Initialize PyDateTime_IsoCalendarDateType.tp_base at run-time (GH-20493)
Recent changes to _datetimemodule broke compilation on mingw; see the comments in this change for details.

FWIW, @corona10: this issue is why `PyType_FromModuleAndSpec` & friends take the `bases` argument at run time.
2020-05-28 09:14:46 -07:00
Victor Stinner 0de437de62
bpo-25920: Remove socket.getaddrinfo() lock on macOS (GH-20177)
On macOS, socket.getaddrinfo() no longer uses an internal lock to
prevent race conditions when calling getaddrinfo(). getaddrinfo is
thread-safe is macOS 10.5, whereas Python 3.9 requires macOS 10.6 or
newer.

The lock was also used on FreeBSD older than 5.3, OpenBSD older than
201311 and NetBSD older than 4.
2020-05-28 17:23:39 +02:00
Serhiy Storchaka 5f4b229df7
bpo-40792: Make the result of PyNumber_Index() always having exact type int. (GH-20443)
Previously, the result could have been an instance of a subclass of int.

Also revert bpo-26202 and make attributes start, stop and step of the range
object having exact type int.

Add private function _PyNumber_Index() which preserves the old behavior
of PyNumber_Index() for performance to use it in the conversion functions
like PyLong_AsLong().
2020-05-28 10:33:45 +03:00
Victor Stinner 10228bad04
bpo-40795: ctypes calls unraisablehook with an exception (GH-20452)
If ctypes fails to convert the result of a callback or if a ctypes
callback function raises an exception, sys.unraisablehook is now
called with an exception set. Previously, the error was logged into
stderr by PyErr_Print().
2020-05-28 00:38:12 +02:00
YoSTEALTH 76ef255bde
bpo-37129: Add os.RWF_APPEND flag for os.pwritev() (GH-20336) 2020-05-27 23:32:22 +02:00
Pablo Galindo e4799b9594
Fix compiler warnings in _zoneinfo.c (GH-20342)
```
D:\a\cpython\cpython\Modules\_zoneinfo.c(903,52): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data [D:\a\cpython\cpython\PCbuild\_zoneinfo.vcxproj]
D:\a\cpython\cpython\Modules\_zoneinfo.c(904,44): warning C4267: '=': conversion from 'size_t' to 'unsigned int', possible loss of data [D:\a\cpython\cpython\PCbuild\_zoneinfo.vcxproj]
D:\a\cpython\cpython\Modules\_zoneinfo.c(1772,31): warning C4244: '=': conversion from 'ssize_t' to 'uint8_t', possible loss of data [D:\a\cpython\cpython\PCbuild\_zoneinfo.vcxproj]
```
2020-05-27 13:48:12 -07:00
Christian Heimes db5aed931f
bpo-40791: Use CRYPTO_memcmp() for compare_digest (#20456)
hashlib.compare_digest uses OpenSSL's CRYPTO_memcmp() function
when OpenSSL is available.

Note: The _operator module is a builtin module. I don't want to add
libcrypto dependency to libpython. Therefore I duplicated the wrapper
function and added a copy to _hashopenssl.c.
2020-05-27 21:50:06 +02:00
Sean Gillespie 29a1384c04
bpo-13097: ctypes: limit callback to 1024 arguments (GH-19914)
ctypes now raises an ArgumentError when a callback
is invoked with more than 1024 arguments.

The ctypes module allocates arguments on the stack in
ctypes_callproc() using alloca(), which is problematic
when large numbers of arguments are passed. Instead
of a stack overflow, this commit raises an ArgumentError
if more than 1024 parameters are passed.
2020-05-27 17:22:07 +02:00
Victor Stinner fe2978b3b9
bpo-39573: Convert Py_REFCNT and Py_SIZE to functions (GH-20429)
Convert Py_REFCNT() and Py_SIZE() macros to static inline functions.
They cannot be used as l-value anymore: use Py_SET_REFCNT() and
Py_SET_SIZE() to set an object reference count and size.

Replace &Py_SIZE(self) with &((PyVarObject*)self)->ob_size
in arraymodule.c.

This change is backward incompatible on purpose, to prepare the C API
for an opaque PyObject structure.
2020-05-27 14:55:10 +02:00
Pablo Galindo 1cf15af9a6 bpo-40217: Ensure Py_VISIT(Py_TYPE(self)) is always called for PyType_FromSpec types (reverts GH-19414) (GH-20264)
Heap types now always visit the type in tp_traverse. See added docs for details.

This reverts commit 0169d3003b.

Automerge-Triggered-By: @encukou
2020-05-27 02:03:38 -07:00
Ethan Steinberg 21fda91f8d
bpo-40611: Adds MAP_POPULATE to the mmap module (GH-20061)
MAP_POPULATE constant has now been added to the list of exported
mmap module flags.
2020-05-26 23:42:18 +02:00
Serhiy Storchaka 578c3955e0
bpo-37999: No longer use __int__ in implicit integer conversions. (GH-15636)
Only __index__ should be used to make integer conversions lossless.
2020-05-26 18:43:38 +03:00
Erlend Egeberg Aasland 5eb45d7d4e
bpo-40737: Fix possible reference leak for sqlite3 initialization (GH-20323) 2020-05-26 21:18:19 +09:00
Lysandros Nikolaou 9645930b5b
bpo-40688: Use the correct parser in the peg_generator scripts (GH-20235)
The scripts in `Tools/peg_generator/scripts` mostly assume that
`ast.parse` and `compile` use the old parser, since this was the
state of things, while we were developing them. They need to be
updated to always use the correct parser. `_peg_parser` is being
extended to support both parsing and compiling with both parsers.
2020-05-25 20:51:58 +01:00
Christian Heimes 20c22db602
bpo-40671: Prepare _hashlib for PEP 489 (GH-20180) 2020-05-25 10:44:51 +02:00
Zackery Spytz 3f59b55316
bpo-35714: Reject null characters in struct format strings (GH-16928)
struct.error is now raised if there is a null character in a struct
format string.
2020-05-25 10:55:09 +03:00
Jakub Stasiak 372ee27d49
bpo-38580: Document that select() accepts iterables, not just sequences (GH-16832) 2020-05-25 10:03:48 +03:00
Chris Jerdonek 7c30d12bd5
bpo-40696: Fix a hang that can arise after gen.throw() (GH-20287)
This updates _PyErr_ChainStackItem() to use _PyErr_SetObject()
instead of _PyErr_ChainExceptions(). This prevents a hang in
certain circumstances because _PyErr_SetObject() performs checks
to prevent cycles in the exception context chain while
_PyErr_ChainExceptions() doesn't.
2020-05-22 13:33:27 -07:00
Ammar Askar 06a1b8915d
bpo-40705: Fix use-after-free in _zoneinfo's module_free (GH-20280) 2020-05-22 12:10:55 -04:00
Raymond Hettinger 0226f3eba0
Let the argument clinic do the type checking for heapq (GH-20284) 2020-05-22 07:28:57 -07:00
Huon Wilson 8b62644831
bpo-40630: Add tracemalloc.reset_peak (GH-20102)
The reset_peak function sets the peak memory size to the current size,
representing a resetting of that metric. This allows for recording the
peak of specific sections of code, ignoring other code that may have
had a higher peak (since the most recent `tracemalloc.start()` or
tracemalloc.clear_traces()` call).
2020-05-22 16:18:51 +02:00
Dong-hee Na a487a39dca
bpo-40714: Remove compile warning from _zoneinfo.c (GH-20291) 2020-05-22 01:56:03 +09:00
Christian Heimes aca4670ad6
bpo-40645: restrict HMAC key len to INT_MAX (GH-20238)
Signed-off-by: Christian Heimes <christian@python.org>

Automerge-Triggered-By: @tiran
2020-05-19 15:35:51 -07:00
Pablo Galindo 2a561b5f68
bpo-37616: Handle version information more gracefully in getpath.c (GH-20214) 2020-05-19 17:28:05 +01:00
Minmin Gong 711f9e180a
bpo-40677: Define IO_REPARSE_TAG_APPEXECLINK explicitly (GH-20206)
This allows building with older versions of the Windows SDK where the value is not defined.
2020-05-19 13:22:16 +01:00
Pablo Galindo d4fe098d1e
Python 3.10.0a0 (GH-20198) 2020-05-19 03:33:01 +01:00
Minmin Gong 7f21c9ac87
bpo-40653: Move _dirnameW out of #ifdef HAVE_SYMLINK/#endif (GH-20144) 2020-05-18 17:17:19 +01:00
Kjell Braden 442634c42f
bpo-39148: enable ipv6 for datagrams in Proactor (GH-19121)
Ifdef is not necessary, as AF_INET6 is supported from Windows Vista, and other code in overlapped.c uses AF_INET6 and is not ifdef'd.
Change the raised exception so users are not fooled to think it comes from Windows API.

Automerge-Triggered-By: @njsmith
2020-05-17 23:21:30 -07:00
Chris Jerdonek da742ba826
bpo-31033: Improve the traceback for cancelled asyncio tasks (GH-19951)
When an asyncio.Task is cancelled, the exception traceback now
starts with where the task was first interrupted.  Previously,
the traceback only had "depth one."
2020-05-17 22:47:31 -07:00
Shantanu 3a855b26ae
bpo-40665: Use Argument Clinic for the bisect module (GH-20163) 2020-05-17 20:38:35 -07:00
Christian Heimes 62ecd8a8f9
bpo-37630: Fix spelling shake128 -> shake_128 (GH-20154) 2020-05-17 18:32:38 +02:00
Christian Heimes 54f2898fe7
bpo-40645: Implement HMAC in C (GH-20129)
The internal module ``_hashlib`` wraps and exposes OpenSSL's HMAC API. The
new code will be used in Python 3.10 after the internal implementation
details of the pure Python HMAC module are no longer part of the public API.

The code is based on a patch by Petr Viktorin for RHEL and Python 3.6.

Co-Authored-By: Petr Viktorin <encukou@gmail.com>
2020-05-17 13:49:10 +02:00
Batuhan Taskaya 31245d19f2
bpo-39740: Early declare devpoll_methods to support old compilers (GH-19281) 2020-05-16 23:36:14 +01:00
Christian Heimes d5b3f6b7f9
bpo-37630: Use SHA3 and SHAKE XOF from OpenSSL (GH-16049)
OpenSSL 1.1.1 comes with SHA3 and SHAKE builtin.

Signed-off-by: Christian Heimes <christian@python.org>

Automerge-Triggered-By: @tiran
2020-05-16 13:27:06 -07:00
Paul Ganssle 1b97b9b0ad
bpo-24416: Return named tuple from date.isocalendar() (GH-20113)
{date, datetime}.isocalendar() now return a private custom named tuple object
IsoCalendarDate rather than a simple tuple.

In order to leave IsocalendarDate as a private class and to improve what
backwards compatibility is offered for pickling the result of a
datetime.isocalendar() call, add a __reduce__ method to the named tuples that
reduces them to plain tuples. (This is the part of this PR most likely to cause
problems — if it causes major issues, switching to a strucseq or equivalent
would be prudent).

The pure python implementation of IsoCalendarDate uses positional-only
arguments, since it is private and only constructed by position anyway; the
equivalent change in the argument clinic on the C side would require us to move
the forward declaration of the type above the clinic import for whatever
reason, so it seems preferable to hold off on that for now.

bpo-24416: https://bugs.python.org/issue24416

Original PR by Dong-hee Na with only minor alterations by Paul Ganssle.

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
2020-05-16 10:02:59 -04:00
Batuhan Taskaya 4541086232
bpo-40192: Use thread_cputime for time.thread_time to improve resolution (GH-19381)
On AIX, time.thread_time() is now implemented with thread_cputime()
which has nanosecond resolution, rather than
clock_gettime(CLOCK_THREAD_CPUTIME_ID) which has a resolution of 10 ms.
2020-05-16 11:39:09 +02:00
Paul Ganssle 62972d9d73
bpo-40503: PEP 615: Tests and implementation for zoneinfo (GH-19909)
This is the initial implementation of PEP 615, the zoneinfo module,
ported from the standalone reference implementation (see
https://www.python.org/dev/peps/pep-0615/#reference-implementation for a
link, which has a more detailed commit history).

This includes (hopefully) all functional elements described in the PEP,
but documentation is found in a separate PR. This includes:

1. A pure python implementation of the ZoneInfo class
2. A C accelerated implementation of the ZoneInfo class
3. Tests with 100% branch coverage for the Python code (though C code
   coverage is less than 100%).
4. A compile-time configuration option on Linux (though not on Windows)

Differences from the reference implementation:

- The module is arranged slightly differently: the accelerated module is
  `_zoneinfo` rather than `zoneinfo._czoneinfo`, which also necessitates
  some changes in the test support function. (Suggested by Victor
  Stinner and Steve Dower.)
- The tests are arranged slightly differently and do not include the
  property tests. The tests live at test/test_zoneinfo/test_zoneinfo.py
  rather than test/test_zoneinfo.py or test/test_zoneinfo/__init__.py
  because we may do some refactoring in the future that would likely
  require this separation anyway; we may:
        - include the property tests
        - automatically run all the tests against both pure Python and C,
          rather than manually constructing C and Python test classes (similar
          to the way this works with test_datetime.py, which generates C
          and Python test cases from datetimetester.py).
- This includes a compile-time configuration option on Linux (though not
  on Windows); added with much help from Thomas Wouters.
- Integration into the CPython build system is obviously different from
  building a standalone zoneinfo module wheel.
- This includes configuration to install the tzdata package as part of
  CI, though only on the coverage jobs. Introducing a PyPI dependency as
  part of the CI build was controversial, and this is seen as less of a
  major change, since the coverage jobs already depend on pip and PyPI.

Additional changes that were introduced as part of this PR, most / all of
which were backported to the reference implementation:

- Fixed reference and memory leaks

    With much debugging help from Pablo Galindo

- Added smoke tests ensuring that the C and Python modules are built

    The import machinery can be somewhat fragile, and the "seamlessly falls
    back to pure Python" nature of this module makes it so that a problem
    building the C extension or a failure to import the pure Python version
    might easily go unnoticed.

- Adjustments to zoneinfo.__dir__

    Suggested by Petr Viktorin.

- Slight refactorings as suggested by Steve Dower.

- Removed unnecessary if check on std_abbr

    Discovered this because of a missing line in branch coverage.
2020-05-16 10:20:06 +02:00
Christian Heimes 6e8cda91d9
bpo-40457: Support OpenSSL without TLS 1.0/1.1 (GH-19862)
OpenSSL can be build without support for TLS 1.0 and 1.1. The ssl module
now correctly adheres to OPENSSL_NO_TLS1 and OPENSSL_NO_TLS1_1 flags.

Also update multissltest to test with latest OpenSSL and LibreSSL
releases.

Signed-off-by: Christian Heimes <christian@python.org>

Automerge-Triggered-By: @tiran
2020-05-15 18:33:05 -07:00
Chris Jerdonek 1ce5841eca
bpo-31033: Add a msg argument to Future.cancel() and Task.cancel() (GH-19979) 2020-05-15 16:55:50 -07:00
Christian Heimes c087a268a4
bpo-40515: Require OPENSSL_THREADS (GH-19953)
The ``ssl`` and ``hashlib`` modules now actively check that OpenSSL is
build with thread support. Python 3.7.0 made thread support mandatory and no
longer works safely with a no-thread builds.

Signed-off-by: Christian Heimes <christian@python.org>
2020-05-15 11:55:25 -07:00
Christian Heimes 16d4e6f6f5
bpo-40479: Fix hashlib issue with OpenSSL 3.0.0 (GH-20107)
OpenSSL 3.0.0-alpha2 was released today. The FIPS_mode() function has
been deprecated and removed. It no longer makes sense with the new
provider and context system in OpenSSL 3.0.0.

EVP_default_properties_is_fips_enabled() is good enough for our needs in
unit tests. It's an internal API, too.

Signed-off-by: Christian Heimes <christian@python.org>
2020-05-15 09:28:05 -07:00
Andrew York 003708bcf8
Trivial typo fix in _tkinter.c (GH-19622)
Change spelling of a #define in _tkinter.c from HAVE_LIBTOMMAMTH to HAVE_LIBTOMMATH, since this is used to keep track of tclTomMath.h, not tclTomMamth.h. No other file seems to refer to this variable.
2020-05-15 03:43:58 -07:00
Victor Stinner d2dc827d16
bpo-40602: _Py_hashtable_set() reports rehash failure (GH-20077)
If _Py_hashtable_set() fails to grow the hash table (rehash), it now
fails rather than ignoring the error.
2020-05-14 22:44:32 +02:00
Victor Stinner a482dc500b
bpo-40602: Write unit tests for _Py_hashtable_t (GH-20091)
Cleanup also hashtable.c.
Rename _Py_hashtable_t members:

* Rename entries to nentries
* Rename num_buckets to nbuckets
2020-05-14 21:55:47 +02:00
Victor Stinner f2c3b6823b
Revert "bpo-32604: [_xxsubinterpreters] Propagate exceptions. (GH-19768)" (GH-20089)
* Revert "bpo-40613: Remove compiler warning from _xxsubinterpretersmodule (GH-20069)"

This reverts commit fa0a66e62d.

* Revert "bpo-32604: [_xxsubinterpreters] Propagate exceptions. (GH-19768)"

This reverts commit a1d9e0accd.
2020-05-14 18:46:24 +02:00
Victor Stinner 97f33c3544
bpo-40549: posixmodule.c uses defining_class (GH-20075)
Pass PEP 573 defining_class to os.DirEntry methods. The module state
is now retrieve from defining_class rather than Py_TYPE(self), to
support subclasses (even if DirEntry doesn't support subclasses yet).

* Pass the module rather than defining_class to DirEntry_fetch_stat().
* Only get the module state once in _posix_clear(),
  _posix_traverse() and _posixmodule_exec().
2020-05-14 18:05:58 +02:00
Victor Stinner 3d17c045b4
bpo-40521: Add PyInterpreterState.unicode (GH-20081)
Move PyInterpreterState.fs_codec into a new
PyInterpreterState.unicode structure.

Give a name to the fs_codec structure and use this structure in
unicodeobject.c.
2020-05-14 01:48:38 +02:00
Dong-hee Na fa0a66e62d
bpo-40613: Remove compiler warning from _xxsubinterpretersmodule (GH-20069) 2020-05-13 22:38:27 +09:00
Victor Stinner 5b0a30354d
bpo-40609: _Py_hashtable_t values become void* (GH-20065)
_Py_hashtable_t values become regular "void *" pointers.

* Add _Py_hashtable_entry_t.data member
* Remove _Py_hashtable_t.data_size member
* Remove _Py_hashtable_t.get_func member. It is no longer needed
  to specialize _Py_hashtable_get() for a specific value size, since
  all entries now have the same size (void*).
* Remove the following macros:

  * _Py_HASHTABLE_GET()
  * _Py_HASHTABLE_SET()
  * _Py_HASHTABLE_SET_NODATA()
  * _Py_HASHTABLE_POP()

* Rename _Py_hashtable_pop() to _Py_hashtable_steal()
* _Py_hashtable_foreach() callback now gets key and value rather than
  entry.
* Remove _Py_hashtable_value_destroy_func type. value_destroy_func
  callback now only has a single parameter: data (void*).
2020-05-13 04:40:30 +02:00
Victor Stinner d95bd4214c
bpo-40609: _tracemalloc allocates traces (GH-20064)
Rewrite _tracemalloc to store "trace_t*" rather than directly
"trace_t" in traces hash tables. Traces are now allocated on the heap
memory, outside the hash table.

Add tracemalloc_copy_traces() and tracemalloc_copy_domains() helper
functions.

Remove _Py_hashtable_copy() function since there is no API to copy a
key or a value.

Remove also _Py_hashtable_delete() function which was commented.
2020-05-13 03:52:11 +02:00
Victor Stinner 2d0a3d682f
bpo-40609: Add destroy functions to _Py_hashtable (GH-20062)
Add key_destroy_func and value_destroy_func parameters to
_Py_hashtable_new_full().

marshal.c and _tracemalloc.c use these destroy functions.
2020-05-13 02:50:18 +02:00
Victor Stinner f9b3b582b8
bpo-40609: Remove _Py_hashtable_t.key_size (GH-20060)
Rewrite _Py_hashtable_t type to always store the key as
a "const void *" pointer. Add an explicit "key" member to
_Py_hashtable_entry_t.

Remove _Py_hashtable_t.key_size member.

hash and compare functions drop their hash table parameter, and their
'key' parameter type becomes "const void *".
2020-05-13 02:26:02 +02:00
Victor Stinner 9e2ca17420
bpo-40609: Rewrite how _tracemalloc handles domains (GH-20059)
Rewrite how the _tracemalloc module stores traces of other domains.
Rather than storing the domain inside the key, it now uses a new hash
table with the domain as the key, and the data is a per-domain traces
hash table.

* Add tracemalloc_domain hash table.
* Remove _Py_tracemalloc_config.use_domain.
* Remove pointer_t and related functions.
2020-05-13 01:36:47 +02:00
Steve Dower d6b727e2c9
bpo-40501: Replace ctypes code in uuid with native module (GH-19948) 2020-05-12 23:32:32 +01:00
Batuhan Taskaya f3a5b7ada0
bpo-39481: remove generic classes from ipaddress/mmap (GH-20045)
These were added by mistake (see https://bugs.python.org/issue39481#msg366288).
2020-05-11 19:32:40 -07:00
Victor Stinner b617993b7c
bpo-40602: Rename hashtable.h to pycore_hashtable.h (GH-20044)
* Move Modules/hashtable.h to Include/internal/pycore_hashtable.h
* Move Modules/hashtable.c to Python/hashtable.c
* Python is now linked to hashtable.c. _tracemalloc is no longer
  linked to hashtable.c. Previously, marshal.c got hashtable.c via
  _tracemalloc.c which is built as a builtin module.
2020-05-12 02:42:19 +02:00
Victor Stinner 4804b5b3df
bpo-39465: Don't access directly _Py_Identifier members (GH-20043)
* Replace id->object with _PyUnicode_FromId(&id)
* Use _Py_static_string_init(str) macro to initialize statically
  name_op in typeobject.c.
2020-05-12 01:43:38 +02:00
Inada Naoki d5d9a71866
bpo-36346: array: Don't use deprecated APIs (GH-19653)
* Py_UNICODE -> wchar_t
* Py_UNICODE -> unicode in Argument Clinic
* PyUnicode_AsUnicode -> PyUnicode_AsWideCharString
* Don't use "u#" format.

Co-authored-by: Victor Stinner <vstinner@python.org>
2020-05-11 15:37:25 +09:00
Victor Stinner 1c2fa78156
bpo-40549: Convert posixmodule.c to multiphase init (GH-19982)
Convert posixmodule.c ("posix" or "nt" module) to the multiphase
initialization (PEP 489).

* Create the module using PyModuleDef_Init().
* Create ScandirIteratorType and DirEntryType with the new
  PyType_FromModuleAndSpec() (PEP 573)
* Get the module state from ScandirIteratorType and DirEntryType with
  the new PyType_GetModule() (PEP 573)
* Pass module to functions which access the module state.
* convert_sched_param() gets a new module parameter. It is now called
  directly since Argument Clinic doesn't support passing the module
  to an argument converter callback.
* Remove _posixstate_global macro.
2020-05-10 11:05:29 +02:00
Dong-hee Na 77c614624b
bpo-40566: Apply PEP 573 to abc module (GH-20005) 2020-05-09 01:31:40 -07:00
Chris Jerdonek d2c349b190
bpo-40559: Add Py_DECREF to _asynciomodule.c:task_step_impl() (GH-19990)
This fixes a possible memory leak in the C implementation of
asyncio.Task.
2020-05-08 03:54:38 -07:00
Hai Shi c068b53a0c
bpo-38787: Update structures.rst docs (PEP 573) (GH-19980) 2020-05-07 19:16:01 +02:00
Eric Snow a1d9e0accd
bpo-32604: [_xxsubinterpreters] Propagate exceptions. (GH-19768)
(Note: PEP 554 is not accepted and the implementation in the code base is a private one for use in the test suite.)

If code running in a subinterpreter raises an uncaught exception then the "run" call in the calling interpreter fails. A RunFailedError is raised there that summarizes the original exception as a string. The actual exception type, __cause__, __context__, state, etc. are all discarded. This turned out to be functionally insufficient in practice. There is a more helpful solution (and PEP 554 has been updated appropriately).

This change adds the exception propagation behavior described in PEP 554 to the _xxsubinterpreters module. With this change a copy of the original exception is set to __cause__ on the RunFailedError. For now we are using "pickle", which preserves the exception's state. We also preserve the original __cause__, __context__, and __traceback__ (since "pickle" does not preserve those).

https://bugs.python.org/issue32604
2020-05-07 08:56:01 -06:00
Petr Viktorin e1becf46b4
bpo-38787: C API for module state access from extension methods (PEP 573) (GH-19936)
Module C state is now accessible from C-defined heap type methods (PEP 573).
Patch by Marcel Plch and Petr Viktorin.

Co-authored-by: Marcel Plch <mplch@redhat.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
2020-05-07 15:39:59 +02:00
Dong-hee Na 3466922320
bpo-1635741: Port errno module to multiphase initialization (GH-19923) 2020-05-07 10:17:16 +09:00
Victor Stinner d8135e913a
bpo-40533: Disable GC in subinterpreters (GH-19961)
When Python is built with experimental isolated interpreters, a
garbage collection now does nothing in an isolated interpreter.

Temporary workaround until subinterpreters stop sharing Python
objects.
2020-05-06 18:25:06 +02:00
Dennis Sweeney 1253c3ef70
bpo-40504: Allow weakrefs to lru_cache objects (GH-19938) 2020-05-05 14:14:32 -07:00
Victor Stinner fb2c7c4afb
bpo-40513: _xxsubinterpreters.run_string() releases the GIL (GH-19944)
In the experimental isolated subinterpreters build mode,
_xxsubinterpreters.run_string() now releases the GIL.
2020-05-05 20:33:06 +02:00
Dong-hee Na 92a98ed975
bpo-1635741: Port syslog module to multiphase initialization (GH-19907) 2020-05-05 10:49:46 +09:00
Serhiy Storchaka 531d1e5412
bpo-39435: Make the first argument of pickle.loads() positional-only. (GH-19846)
It was positional-only de facto: documentation and two implementations
used three different name.
2020-05-02 09:38:01 +03:00
Lysandros Nikolaou 03b7642265
bpo-40334: Make the PyPegen* and PyParser* APIs more consistent (GH-19839)
This commit makes both APIs more consistent by doing the following:
- Remove the `PyPegen_CodeObjectFrom*` functions, which weren't used 
  and will probably not be needed. Functions like `Py_CompileStringObject`
  can be used instead.
- Include a `const char *filename` parameter in `PyPegen_ASTFromString`.
- Rename `PyPegen_ASTFromFile` to `PyPegen_ASTFromFilename`, because
  its signature is not the same with `PyParser_ASTFromFile`.
2020-05-01 18:30:51 +01:00
Dong-hee Na 8727664557
bpo-32494: Use gdbm_count for dbm_length if possible (GH-19814) 2020-05-01 14:15:35 +02:00
Victor Stinner 252346acd9
bpo-40453: Add PyConfig._isolated_subinterpreter (GH-19820)
An isolated subinterpreter cannot spawn threads, spawn a child
process or call os.fork().

* Add private _Py_NewInterpreter(isolated_subinterpreter) function.
* Add isolated=True keyword-only parameter to
  _xxsubinterpreters.create().
* Allow again os.fork() in "non-isolated" subinterpreters.
2020-05-01 11:33:44 +02:00
Victor Stinner b66c0ff8af
bpo-1635741: Fix compiler warning in _stat.c (GH-19822)
Cast Py_ARRAY_LENGTH() size_t to int explicitly.
2020-05-01 00:44:03 +02:00
Lysandros Nikolaou 69e802ed81
bpo-40334: Fix test_peg_parser to actually use the old parser (GH-19778)
Now that the default parser is the new PEG parser, ast.parse uses it, which means that we don't actually test something in test_peg_parser. This commit introduces a new keyword argument (`oldparser`) for `_peg_parser.parse_string` for specifying that a string needs to be parsed with the old parser. This keyword argument is used in the tests to actually compare the ASTs the new parser generates with those generated by the old parser.
2020-04-29 23:53:30 +01:00
karl ding 360371f79c
bpo-40291: Add support for CAN_J1939 sockets (GH-19538)
Add support for CAN_J1939 sockets that wrap SAE J1939 protocol
functionality provided by Linux 5.4+ kernels.
2020-04-29 15:31:19 -07:00
Dong-hee Na 84724dd239
bpo-1635741: Port _stat module to multiphase initialization (GH-19798) 2020-04-30 03:20:27 +09:00
Dong-hee Na 113feb3ec2
bpo-40328: Add tool for generating cjk mapping headers (GH-19602) 2020-04-30 02:34:24 +09:00
Victor Stinner 2d8757758d
bpo-40286: Remove C implementation of Random.randbytes() (GH-19797)
Remove _random.Random.randbytes(): the C implementation of
randbytes(). Implement the method in Python to ease subclassing:
randbytes() now directly reuses getrandbits().
2020-04-29 18:49:00 +02:00
Victor Stinner e3dfb9b967
bpo-9216: Expose OpenSSL FIPS_mode() as _hashlib.get_fips_mode() (GH-19703)
test.pythoninfo logs OpenSSL FIPS_mode() and Linux
/proc/sys/crypto/fips_enabled in a new "fips" section.

Co-Authored-By: Petr Viktorin <encukou@gmail.com>
2020-04-29 09:04:22 -07:00
Victor Stinner 7036477323
bpo-40421: Add PyFrame_GetBack() function (GH-19765)
New PyFrame_GetBack() function: get the frame next outer frame.

Replace frame->f_back with PyFrame_GetBack(frame) in most code but
frameobject.c, ceval.c and genobject.c.
2020-04-29 03:28:46 +02:00
Victor Stinner 4386b9045e
bpo-40429: PyThreadState_GetFrame() returns a strong ref (GH-19781)
The PyThreadState_GetFrame() function now returns a strong reference
to the frame.
2020-04-29 03:01:43 +02:00
Victor Stinner ae00a5a885
bpo-40428: Remove PyTuple_ClearFreeList() function (GH-19769)
Remove the following function from the C API:

* PyAsyncGen_ClearFreeLists()
* PyContext_ClearFreeList()
* PyDict_ClearFreeList()
* PyFloat_ClearFreeList()
* PyFrame_ClearFreeList()
* PyList_ClearFreeList()
* PySet_ClearFreeList()
* PyTuple_ClearFreeList()

Make these functions private, move them to the internal C API and
change their return type to void.

Call explicitly PyGC_Collect() to free all free lists.

Note: PySet_ClearFreeList() did nothing.
2020-04-29 02:29:20 +02:00
Lewis Gaul f7bbf58aa9
bpo-38880: List interpreters associated with a channel end (GH-17323)
This PR adds the functionality requested by https://github.com/ericsnowcurrently/multi-core-python/issues/52.

Automerge-Triggered-By: @ericsnowcurrently
2020-04-28 17:18:42 -07:00
Victor Stinner 8852ad4208
bpo-40429: PyFrame_GetCode() now returns a strong reference (GH-19773) 2020-04-29 01:28:13 +02:00
Eric Snow 5e8c691594
bpo-32604: Add support for a "default" arg in channel_recv(). (GH-19770)
This allows the caller to avoid creation of an exception when the channel is empty (just like `dict.get()` works).  `ChannelEmptyError` is still raised if no default is provided.

Automerge-Triggered-By: @ericsnowcurrently
2020-04-28 16:11:32 -07:00
Victor Stinner 6d86a2331e
bpo-40429: PyFrame_GetCode() result cannot be NULL (GH-19772)
Add frame_nslots() to factorize duplicate code.
2020-04-29 00:56:58 +02:00
Victor Stinner a42ca74fa3
bpo-40421: Add PyFrame_GetCode() function (GH-19757)
PyFrame_GetCode(frame): return a borrowed reference to the frame
code.

Replace frame->f_code with PyFrame_GetCode(frame) in most code,
except in frameobject.c, genobject.c and ceval.c.

Also add PyFrame_GetLineNumber() to the limited C API.
2020-04-28 19:01:31 +02:00
Pablo Galindo 2b74c835a7
bpo-40334: Support CO_FUTURE_BARRY_AS_BDFL in the new parser (GH-19721)
This commit also allows to pass flags to the new parser in all interfaces and fixes a bug in the parser generator that was causing to inline rules with actions, making them disappear.
2020-04-27 18:02:07 +01:00
Victor Stinner e6f8abd500
bpo-38061: subprocess uses closefrom() on FreeBSD (GH-19697)
Optimize the subprocess module on FreeBSD using closefrom().
A single close(fd) syscall is cheap, but when sysconf(_SC_OPEN_MAX)
is high, the loop calling close(fd) on each file descriptor can take
several milliseconds.

The workaround on FreeBSD to improve performance was to load and
mount the fdescfs kernel module, but this is not enabled by default.

Initial patch by Ed Maste (emaste), Conrad Meyer (cem), Kyle Evans
(kevans) and Kubilay Kocak (koobs):
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274
2020-04-24 12:06:58 +02:00
Victor Stinner 162c567d16
bpo-38061: os.closerange() uses closefrom() on FreeBSD (GH-19696)
On FreeBSD, os.closerange(fd_low, fd_high) now calls
closefrom(fd_low) if fd_high is greater than or equal to
sysconf(_SC_OPEN_MAX).

Initial patch by Ed Maste (emaste), Conrad Meyer (cem), Kyle Evans
(kevans) and Kubilay Kocak (koobs):
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274
2020-04-24 12:00:51 +02:00
Lysandros Nikolaou ebebb6429c
bpo-40334: Improve various PEG-Parser related stuff (GH-19669)
The changes in this commit are all related to @vstinner's original review comments of the initial PEP 617 implementation PR.
2020-04-23 16:36:06 +01:00
Pablo Galindo c5fc156852
bpo-40334: PEP 617 implementation: New PEG parser for CPython (GH-19503)
Co-authored-by: Guido van Rossum <guido@python.org>
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
2020-04-22 23:29:27 +01:00
Victor Stinner 9bee32b34e
bpo-40138: Fix Windows os.waitpid() for large exit code (GH-19637)
Fix the Windows implementation of os.waitpid() for exit code
larger than "INT_MAX >> 8". The exit status is now interpreted as an
unsigned number.

os.waitstatus_to_exitcode() now accepts wait status larger than
INT_MAX.
2020-04-22 16:30:35 +02:00
Raymond Hettinger bba760e9b6
Fix uninitialized struct member (GH-19589) 2020-04-20 13:47:12 -07:00
Hai Shi 5dd21f5d1c
bpo-39849: Enable assertions in _testcapimodule.c and _testinternalcapi.c (GH-19623) 2020-04-21 01:49:13 +09:00
Serhiy Storchaka 2b5603140c
bpo-40178: Convert the remaining os functions to Argument Clinic. (GH-19360)
Convert os.getgrouplist(), os.initgroups(), os.sendfile() and
os.get_terminal_size().
2020-04-18 19:14:10 +03:00
Serhiy Storchaka 12446e6a60
bpo-40179: Fix translation of #elif in Argument Clinic (GH-19364)
Co-authored-by: Ammar Askar <ammar@ammaraskar.com>
2020-04-18 17:52:48 +03:00
Serhiy Storchaka 223221b290
bpo-40286: Makes simpler the relation between randbytes() and getrandbits() (GH-19574) 2020-04-17 23:51:28 +03:00
Antoine Pitrou 75a3378810
bpo-40282: Allow random.getrandbits(0) (GH-19539) 2020-04-17 19:32:14 +02:00
Victor Stinner 1a1bd2e238
bpo-40302: Replace PY_INT64_T with int64_t (GH-19573)
* Replace PY_INT64_T with int64_t
* Replace PY_UINT32_T with uint32_t
* Replace PY_UINT64_T with uint64_t

sha3module.c no longer checks if PY_UINT64_T is defined since it's
always defined and uint64_t is always available on platforms
supported by Python.
2020-04-17 19:13:06 +02:00
Victor Stinner 9f5fe7910f
bpo-40286: Add randbytes() method to random.Random (GH-19527)
Add random.randbytes() function and random.Random.randbytes()
method to generate random bytes.

Modify secrets.token_bytes() to use SystemRandom.randbytes()
rather than calling directly os.urandom().

Rename also genrand_int32() to genrand_uint32(), since it returns an
unsigned 32-bit integer, not a signed integer.

The _random module is now built with Py_BUILD_CORE_MODULE defined.
2020-04-17 19:05:35 +02:00
Victor Stinner 1ae035b7e8
bpo-40302: Add pycore_byteswap.h header file (GH-19552)
Add a new internal pycore_byteswap.h header file with the following
functions:

* _Py_bswap16()
* _Py_bswap32()
* _Py_bswap64()

Use these functions in _ctypes, sha256 and sha512 modules,
and also use in the UTF-32 encoder.

sha256, sha512 and _ctypes modules are now built with the internal
C API.
2020-04-17 17:47:20 +02:00
Jeffrey Quesnelle a75e730075
bpo-40294: Fix _asyncio when module is loaded/unloaded multiple times (GH-19542) 2020-04-17 04:09:45 +02:00
Victor Stinner 5eca75df03
bpo-40268: Reformat posixmodule.c includes (GH-19536) 2020-04-15 15:07:31 +02:00