Commit Graph

11484 Commits

Author SHA1 Message Date
Victor Stinner f0c85793d2
bpo-38353: Rework ismodule() in getpath.c (GH-16574)
* ismodule() no longer modify the input string.
* copy_absolute(), joinpath(): rename parameters and local variables.
2019-10-04 03:53:26 +02:00
Victor Stinner 03a8a56fac
bpo-38353: Add subfunctions to getpath.c (GH-16572)
Following symbolic links is now limited to 40 attempts, just to
prevent loops.

Add subfunctions:

* Add resolve_symlinks()
* Add calculate_argv0_path_framework()
* Add calculate_which()
* Add calculate_program_macos()

Fix also _Py_wreadlink(): readlink() result type is Py_ssize_t, not
int.
2019-10-04 02:22:39 +02:00
Victor Stinner 86ec5c65fe
bpo-38353: Fix calculate_argv0_path() for symlinks (GH-16549)
calculate_argv0_path() must read argv0_path link, not read
program_full_path link.
2019-10-03 23:17:00 +02:00
Victor Stinner 61691d8336
bpo-38353: Cleanup includes in the internal C API (GH-16548)
Use forward declaration of types to avoid includes in the internal C
API. Add also comment to justify other includes.
2019-10-02 23:51:20 +02:00
Victor Stinner 5e0ea7540f
bpo-37474: Don't call fedisableexcept() on FreeBSD (GH-16515)
On FreeBSD, Python no longer calls fedisableexcept() at startup to
control the floating point control mode. The call became useless
since FreeBSD 6: it became the default mode.
2019-10-01 13:12:29 +02:00
Victor Stinner 2f90261280
bpo-38321: Fix compiler warning in _randommodule.c (GH-16512)
Fix the GCC warning: "initialization discards ‘const’ qualifier from
pointer target type".
2019-10-01 12:45:52 +02:00
Victor Stinner 8462a4936b
bpo-38304: PyConfig_InitPythonConfig() cannot fail anymore (GH-16509)
PyConfig_InitPythonConfig() and PyConfig_InitIsolatedConfig() no
longer return PyStatus: they cannot fail anymore.
2019-10-01 12:06:16 +02:00
Victor Stinner 3c30a76f3d
bpo-38304: Remove PyConfig.struct_size (GH-16500) (GH-16508)
For now, we'll rely on the fact that the config structures aren't covered by the stable ABI.

We may revisit this in the future if we further explore the idea of offering a stable embedding API.

(cherry picked from commit bdace21b76)
2019-10-01 10:56:37 +02:00
Victor Stinner aca8c406ad
bpo-38321: Fix _testcapimodule.c warning (GH-16494)
Fix the following warning:

    modules\_testcapimodule.c(6409):
    warning C4146: unary minus operator applied to unsigned type,
    result still unsigned
2019-09-30 21:14:26 +02:00
Neil Schemenauer bcda460baf Clear weakrefs in garbage found by the GC (#16495)
Fix a bug due to the interaction of weakrefs and the cyclic garbage
collector. We must clear any weakrefs in garbage in order to prevent
their callbacks from executing and causing a crash.
2019-09-30 19:06:45 +02:00
Victor Stinner c9a413ede4 bpo-38321: Fix PyCStructUnionType_update_stgdict() warning (GH-16492)
bpo-22273, bpo-38321: Fix following warning:

    modules\_ctypes\stgdict.c(704):
    warning C4244: 'initializing': conversion from 'Py_ssize_t' to 'int', possible loss of data
2019-09-30 16:49:47 +01:00
Victor Stinner efe74b6369
bpo-38321: Fix _asynciomodule.c compiler warning (GH-16493)
bpo-38248, bpo-38321: Fix warning:

    modules\_asynciomodule.c(2667):
    warning C4102: 'set_exception': unreferenced label

The related goto has been removed by
commit edad4d89e3.
2019-09-30 16:41:34 +02:00
Victor Stinner 441b10cf28
bpo-38304: Add PyConfig.struct_size (GH-16451)
Add a new struct_size field to PyPreConfig and PyConfig structures to
allow to modify these structures in the future without breaking the
backward compatibility.

* Replace private _config_version field with public struct_size field
  in PyPreConfig and PyConfig.
* Public PyPreConfig_InitIsolatedConfig() and
  PyPreConfig_InitPythonConfig()
  return type becomes PyStatus, instead of void.
* Internal _PyConfig_InitCompatConfig(),
  _PyPreConfig_InitCompatConfig(), _PyPreConfig_InitFromConfig(),
  _PyPreConfig_InitFromPreConfig() return type becomes PyStatus,
  instead of void.
* Remove _Py_CONFIG_VERSION
* Update the Initialization Configuration documentation.
2019-09-28 04:28:35 +02:00
Christian Heimes 9055815809 bpo-38270: More fixes for strict crypto policy (GH-16418)
test_hmac and test_hashlib test built-in hashing implementations and
OpenSSL-based hashing implementations. Add more checks to skip OpenSSL
implementations when a strict crypto policy is active.

Use EVP_DigestInit_ex() instead of EVP_DigestInit() to initialize the
EVP context. The EVP_DigestInit() function clears alls flags and breaks
usedforsecurity flag again.

Signed-off-by: Christian Heimes <christian@python.org>



https://bugs.python.org/issue38270
2019-09-27 06:03:53 -07:00
Benjamin Peterson 52b9408038
closes bpo-38174: Update vendored expat library to 2.2.8. (GH-16346)
Fixes CVE-2019-15903. See full changelog at https://github.com/libexpat/libexpat/blob/R_2_2_8/expat/Changes.
2019-09-25 21:33:58 -07:00
Christian Heimes df69e75edc
bpo-38142: Updated _hashopenssl.c to be PEP 384 compliant (#16071)
* Updated _hashopenssl.c to be PEP 384 compliant
* Remove refleak test from test_hashlib. The updated type no longer accepts random arguments to __init__.
2019-09-25 23:03:30 +02:00
Yury Selivanov edad4d89e3 bpo-38248: Fix inconsistent immediate asyncio.Task cancellation (GH-16330) 2019-09-25 03:32:08 -07:00
Vinay Sajip c64af8fad3
Changed conditions for ctypes array-in-struct handling. (GH-16381) 2019-09-25 11:11:57 +01:00
Vinay Sajip 57dc7d5ae8
bpo-22273: Disabled tests while investigating buildbot failures on ARM7L/PPC64. (GH-16377) 2019-09-25 07:58:32 +01:00
Dong-hee Na ad7736faf5 bpo-38265: Update os.pread to accept the length type as Py_ssize_t. (GH-16359) 2019-09-25 08:47:04 +03:00
Vinay Sajip 12f209eccb
bpo-22273: Update ctypes to correctly handle arrays in small structur… (GH-15839) 2019-09-25 04:38:44 +01:00
Victor Stinner 221fd84703
bpo-38234: Cleanup getpath.c (GH-16367)
* search_for_prefix() directly calls reduce() if found is greater
  than 0.
* Add calculate_pybuilddir() subfunction.
* search_for_prefix(): add path string buffer for readability.
* Fix some error handling code paths: release resources on error.
* calculate_read_pyenv(): rename tmpbuffer to filename.
* test.pythoninfo now also logs windows.dll_path
2019-09-25 02:54:25 +02:00
Victor Stinner bb6bf7d342
bpo-38234: Add tests for Python init path config (GH-16358) 2019-09-24 18:21:02 +02:00
Victor Stinner 85ce0a7178
bpo-38234: read_pth_file() now returns PyStatus (GH-16338)
Refactor path configuration code:

* read_pth_file() now returns PyStatus to report errors, rather than
  calling Py_FatalError().
* Move argv0_path and zip_path buffers out of PyCalculatePath
  structures.
* On Windows, _PyPathConfig.home is now preferred over PyConfig.home.
2019-09-24 00:55:48 +02:00
Victor Stinner 9c42f8cda5
bpo-38234: Fix _PyConfig_InitPathConfig() (GH-16335)
* _PyConfig_InitPathConfig() now starts by copying the global path
  configuration, and then override values set in PyConfig.
* _PyPathConfig_Calculate() implementations no longer override
  _PyPathConfig fields which are already computed. For example,
  if _PyPathConfig.prefix is not NULL, leave it unchanged.
* If Py_SetPath() has been called, _PyConfig_InitPathConfig() doesn't
  call _PyPathConfig_Calculate() anymore.
* _PyPathConfig_Calculate() no longer uses PyConfig,
  except to initialize PyCalculatePath structure.
* pathconfig_calculate(): remove useless temporary
  "_PyPathConfig new_config" variable.
* calculate_module_search_path(): remove hack to workaround memory
  allocation failure, call Py_FatalError() instead.
* Fix get_program_full_path(): handle memory allocation failure.
2019-09-23 18:47:29 +02:00
Victor Stinner e267793aa4
bpo-38234: Fix PyConfig_Read() when Py_SetPath() was called (GH-16298)
* If Py_SetPath() has been called, _PyConfig_InitPathConfig() now
  uses its value.
* Py_Initialize() now longer copies path configuration from PyConfig
  to the global path configuration (_Py_path_config).
2019-09-21 01:50:16 +02:00
Orivej Desh 77abf23c67 bpo-6559: Update _posixsubprocess.fork_exec doc (GH-16283)
It did not list the argument added in d4cc7bf993.


https://bugs.python.org/issue6559



Automerge-Triggered-By: @gpshead
2019-09-20 10:01:09 -07:00
Eddie Elizondo 3368f3c6ae bpo-38140: Make dict and weakref offsets opaque for C heap types (#16076)
* Make dict and weakref offsets opaque for C heap types

* Add news
2019-09-19 17:29:05 +01:00
Victor Stinner 0a963fbc9c
bpo-38203: faulthandler.dump_traceback_later() is always available (GH-16249)
dump_traceback_later() and cancel_dump_traceback_later() functions of
the faulthandler module are always available since Python 3.7.
2019-09-18 14:15:10 +02:00
Serhiy Storchaka f669581a95
bpo-38185: Fixed case-insensitive string comparison in sqlite3.Row indexing. (GH-16190) 2019-09-17 09:20:56 +03:00
Christian Heimes 724f1a5723 bpo-33936: Don't call obsolete init methods with OpenSSL 1.1.0+ (GH-16140)
``OPENSSL_VERSION_1_1`` was never defined in ``_hashopenssl.c``. 


https://bugs.python.org/issue33936
2019-09-16 12:10:05 -07:00
Serhiy Storchaka 8debfa5040
bpo-38175: Fix a memory leak in comparison of sqlite3.Row objects. (GH-16155) 2019-09-16 20:15:18 +03:00
Hai Shi 56a45142e7 Fix a possbile refleak in setint() of mmapmodule.c (GH-16136) 2019-09-16 13:56:57 +08:00
Raymond Hettinger 6e27a0d775
bpo-37798: Prevent undefined behavior in direct calls to the C helper function. (#16149) 2019-09-15 09:36:38 -07:00
Dino Viehland 09dc2c672f
Fix missing dec ref (#16158) 2019-09-15 15:51:44 +01:00
Christian Heimes eb2b0c694a
bpo-38153: detect shake independently from sha3 (GH-16143)
XOF digests (SHAKE) are not available in OpenSSL 1.1.0 but SHA3 fixed-length digests are.

Signed-off-by: Christian Heimes <christian@python.org>
2019-09-14 17:29:54 +02:00
Dino Viehland f919054e53
bpo-38116: Convert select module to PEP-384 (#15971)
* Convert select module to PEP-384

Summary: Do the necessary versions to be Pyro-compatible, including migrating `PyType_Ready` to `PyType_FromSpec` and moving static data into a new `_selectstate` struct.

* 📜🤖 Added by blurb_it.

* Fixup Mac OS/X build
2019-09-14 15:20:27 +01:00
Serhiy Storchaka 279f44678c
bpo-37206: Unrepresentable default values no longer represented as None. (GH-13933)
In ArgumentClinic, value "NULL" should now be used only for unrepresentable default values
(like in the optional third parameter of getattr). "None" should be used if None is accepted
as argument and passing None has the same effect as not passing the argument at all.
2019-09-14 12:24:05 +03:00
Serhiy Storchaka bf169915ec
bpo-38005: Fixed comparing and creating of InterpreterID and ChannelID. (GH-15652)
* Fix a crash in comparing with float (and maybe other crashes).
* They are now never equal to strings and non-integer numbers.
* Comparison with a large number no longer raises OverflowError.
* Arbitrary exceptions no longer silenced in constructors and comparisons.
* TypeError raised in the constructor contains now the name of the type.
* Accept only ChannelID and int-like objects in channel functions.
* Accept only InterpreterId, int-like objects and str in the InterpreterId constructor.
* Accept int-like objects, not just int in interpreter related functions.
2019-09-13 22:50:27 +03:00
Eddie Elizondo a67ac2f2d9 bpo-38150 Fix refleak in the finalizer of a _testcapimodule type (GH-16115)
The PyLong created in the finalizer was not being cleaned up


https://bugs.python.org/issue38150



Automerge-Triggered-By: @matrixise
2019-09-13 09:48:03 -07:00
Christian Heimes 995b5d38e7 bpo-38153: Normalize hashlib algorithm names (GH-16083)
Signed-off-by: Christian Heimes <christian@python.org>
2019-09-13 14:31:19 +01:00
Dino Viehland 04f0bbfbed bpo-38075: Port _randommodule.c to PEP-384 (GH-15798)
- Migrate `Random_Type` to `PyType_FromSpec`
- To simulate an old use of `PyLong_Type.tp_as_number->nb_absolute`, I added
  code to the module init function to stash `int.__abs__` for later
  use. Ideally we'd use `PyType_GetSlot()` instead, but it doesn't currently
  work for static types in CPython, and implementing it just for this case
  doesn't seem worth it.
- Do exact check for long and dispatch to PyNumber_Absolute, use vector call when not exact.
2019-09-13 03:12:27 -07:00
Christian Heimes 7cad53e6b0 bpo-9216: Add usedforsecurity to hashlib constructors (GH-16044)
The usedforsecurity keyword only argument added to the hash constructors is useful for FIPS builds and similar restrictive environment with non-technical requirements that legacy algorithms be forbidden by their implementations without being explicitly annotated as not being used for any security related purposes.  Linux distros with FIPS support benefit from this being standard rather than making up their own way(s) to do it.

Contributed and Signed-off-by: Christian Heimes christian@python.org
2019-09-12 19:30:00 -05:00
Patrick McLean 2b2ead7438 bpo-36046: Add user and group parameters to subprocess (GH-11950)
* subprocess: Add user, group and extra_groups paremeters to subprocess.Popen

This adds a `user` parameter to the Popen constructor that will call
setreuid() in the child before calling exec(). This allows processes
running as root to safely drop privileges before running the subprocess
without having to use a preexec_fn.

This also adds a `group` parameter that will call setregid() in
the child process before calling exec().

Finally an `extra_groups` parameter was added that will call
setgroups() to set the supplimental groups.
2019-09-12 18:15:44 +01:00
Christian Heimes 9a4963b932 bpo-38137: Re-add OpenSSL 1.0.2 compat (GH-16051)
The defines are required for OpenSSL 1.0.2 and LibreSSL.


https://bugs.python.org/issue38134



Automerge-Triggered-By: @tiran
2019-09-12 07:33:26 -07:00
animalize 4ffd05d7ec bpo-21872: fix lzma library decompresses data incompletely (GH-14048)
* 1. add test case with wrong behavior
* 2. fix bug when max_length == -1
* 3. allow b"" as valid input data for decompress_buf()
* 4. when max_length >= 0, let needs_input mechanism works
* add more asserts to test case
2019-09-12 15:20:37 +01:00
Christian Heimes 8c74574e0a bpo-38132: Check EVP_DigestUpdate for error (GH-16041) 2019-09-12 14:30:47 +01:00
Christian Heimes 5a4f82f457 bpo-38132: Simplify _hashopenssl code (GH-16023)
Signed-off-by: Christian Heimes <christian@python.org>
2019-09-12 13:42:07 +01:00
Christian Heimes 64117e059b bpo-38134: Remove PKBDF2_HMAC_fast from _hashopenssl (GH-16028)
Signed-off-by: Christian Heimes <christian@python.org>
2019-09-12 12:46:37 +01:00
Zackery Spytz ea683deccc closes bpo-38127: _ctypes: PyObject_IsSubclass() should be checked for failure. (GH-16011)
An exception may occur during a PyObject_IsSubclass() call.
2019-09-12 11:09:32 +01:00
bggardner 954900a3f9 closes bpo-37405: Make socket.getsockname() always return a tuple for AF_CAN. (GH-14392)
This fixes a regression from 3.5. In recent releases, `getsockname()` in the AF_CAN case has returned a string.
2019-09-12 11:02:48 +01:00
Petr Viktorin 5e9caeec76
bpo-37879: Fix warnings in _testcapimodule (GH-16004) 2019-09-12 10:12:53 +01:00
Steve Dower 0b72ccff56 bpo-32592: Set Windows 8 as the minimum required version for API support (GH-15951) 2019-09-11 17:03:37 +01:00
Steve Dower aa929273ca bpo-33166: Change os.cpu_count to return active (real) processors (GH-15949) 2019-09-11 16:15:39 +01:00
Jakub Kulík e20134f889 bpo-38110: Use fdwalk for os.closerange() when available. (GH-15224)
Use fdwalk() on platforms that support it to implement os.closerange().
2019-09-11 10:11:57 -05:00
Eddie Elizondo ff023ed36e bpo-37879: Suppress subtype_dealloc decref when base type is a C heap type (GH-15323)
The instance destructor for a type is responsible for preparing
an instance for deallocation by decrementing the reference counts
of its referents.

If an instance belongs to a heap type, the type object of an instance
has its reference count decremented while for static types, which
are permanently allocated, the type object is unaffected by the
instance destructor.

Previously, the default instance destructor searched the class
hierarchy for an inherited instance destructor and, if present,
would invoke it.

Then, if the instance type is a heap type, it would decrement the
reference count of that heap type.  However, this could result in the
premature destruction of a type because the inherited instance
destructor should have already decremented the reference count
of the type object.

This change avoids the premature destruction of the type object
by suppressing the decrement of its reference count when an
inherited, non-default instance destructor has been invoked.

Finally, an assertion on the Py_SIZE of a type was deleted.  Heap
types have a non zero size, making this into an incorrect assertion.

https://github.com/python/cpython/pull/15323
2019-09-11 10:17:13 +01:00
Jordon Xu 2ec7010206 bpo-37752: Delete redundant Py_CHARMASK in normalizestring() (GH-15095) 2019-09-10 17:04:08 +01:00
Dino Viehland b7f8e52433 bpo-38073: Make pwd module PEP-384 compatible (GH-15790)
Makes the pwd module PEP-384 compatible


https://bugs.python.org/issue38073



Automerge-Triggered-By: @tiran
2019-09-10 05:59:43 -07:00
Petr Viktorin f958377b67 bpo-37499: Test various C calling conventions (GH-15776)
Add functions with various calling conventions to `_testcapi`, expose them as module-level functions, bound methods, class methods, and static methods, and test calling them and introspecting them through GDB.


https://bugs.python.org/issue37499


Co-authored-by: Jeroen Demeyer <J.Demeyer@UGent.be>
Automerge-Triggered-By: @pganssle
2019-09-10 04:21:09 -07:00
Dino Viehland 5a7d2e11aa bpo-38069: Convert _posixsubprocess to PEP-384 (GH-15780)
Summary:
Eliminate uses of `_Py_IDENTIFIER` from `_posixsubprocess`, replacing them with interned strings.

Also tries to find an existing version of the module, which will allow subinterpreters.



https://bugs.python.org/issue38069
2019-09-10 04:01:20 -07:00
Dino Viehland bd0c7a12d9 bpo-38071: Make termios PEP-384 compatible (GH-15785)
Make the termios module PEP-384 compatible.
2019-09-10 03:33:30 -07:00
Dino Viehland 40a5313edf bpo-38072: PEP-384 grpmodule (GH-15788)
Make the grp module PEP-384 compliant.
2019-09-10 03:30:36 -07:00
Kenta Murata 9e61066355 Fix calling order of PyEval_InitThreads. (GH-4602)
As described in Doc/c-api/init.rst, PyEval_InitThreads() cannot be called
before Py_Initialize() function.
2019-09-10 11:29:56 +01:00
Dino Viehland a1ffad0719 bpo-38074: Make zlib extension module PEP-384 compatible (GH-15792)
Updated zlibmodule.c to be PEP 384 compliant.
2019-09-10 03:27:03 -07:00
Dino Viehland 4f384af067 bpo-38076: Make struct module PEP-384 compatible (#15805)
* PEP-384 _struct

* More PEP-384 fixes for _struct

Summary: Add a couple of more fixes for `_struct` that were previously missed such as removing `tp_*` accessors and using `PyBytesWriter` instead of calling `PyBytes_FromStringAndSize` with `NULL`. Also added a test to confirm that `iter_unpack` type is still uninstantiable.

* 📜🤖 Added by blurb_it.
2019-09-10 03:18:37 -07:00
Sergey Fedoseev 92709a263e bpo-37840: Fix handling of negative indices in bytearray_getitem() (GH-15250) 2019-09-09 09:28:34 -07:00
Christian Heimes 915cd3f069 bpo-35941: Fix performance regression in new code (GH-12610)
Accumulate certificates in a set instead of doing a costly list contain
operation. A Windows cert store can easily contain over hundred
certificates. The old code would result in way over 5,000 comparison
operations

Signed-off-by: Christian Heimes <christian@python.org>
2019-09-09 09:06:55 -07:00
Orivej Desh 09090d04ef bpo-37649: Fix exec_prefix check (GH-14897) 2019-09-09 09:05:21 -07:00
Zackery Spytz 682107cf45 bpo-36279: Ensure os.wait3() rusage is initialized (GH-15111)
Co-Authored-By: David Wilson <dw@botanicus.net>
2019-09-09 08:48:32 -07:00
Victor Stinner d91d4de317
bpo-38070: visit_decref() calls _PyObject_IsFreed() (GH-15782)
In debug mode, visit_decref() now calls _PyObject_IsFreed() to ensure
that the object is not freed. If it's freed, the program fails with
an assertion error and Python dumps informations about the freed
object.
2019-09-09 17:44:59 +02:00
animalize 77643c486f bpo-38037: Fix reference counters in signal module (GH-15753) 2019-09-09 15:46:26 +02:00
neonene ed70129e15 bpo-37702: Fix SSL's certificate-store leak on Windows (GH-15632)
ssl_collect_certificates function in _ssl.c has a memory leak.
Calling CertOpenStore() and CertAddStoreToCollection(), a store's refcnt gets incremented by 2.
But CertCloseStore() is called only once and the refcnt leaves 1.
2019-09-09 05:33:43 -07:00
Zackery Spytz a6563650c8 bpo-37445: Include FORMAT_MESSAGE_IGNORE_INSERTS in FormatMessageW() calls (GH-14462)
If FormatMessageW() is passed the FORMAT_MESSAGE_FROM_SYSTEM flag without FORMAT_MESSAGE_IGNORE_INSERTS, it will fail if there are insert sequences in the message definition.
2019-09-09 02:20:38 -07:00
Greg Price 7669cb8b21 bpo-38043: Use `bool` for boolean flags on is_normalized_quickcheck. (GH-15711) 2019-09-09 02:16:31 -07:00
Serhiy Storchaka 526a01467b
bpo-34410: Fix a crash in the tee iterator when re-enter it. (GH-15625)
RuntimeError is now raised in this case.
2019-09-09 11:47:14 +03:00
Serhiy Storchaka 918b468b7d
Revert "Raise a RuntimeError when tee iterator is consumed from different threads (GH-15567)" (GH-15736)
This reverts commit fa220ec763.
2019-09-09 11:18:16 +03:00
HongWeipeng 3c87a667bb bpo-36946:Fix possible signed integer overflow when handling slices. (GH-15639)
This is a complement to PR 13375.
2019-09-08 13:15:56 +03:00
Sergey Fedoseev 32f825393e Remove some unused defines in multiprocessing.h. (GH-15661)
It looks like they are unused since
87cf220972.
2019-09-07 10:44:36 +03:00
Joannah Nanjekye 2bc43cdc01 bpo-37878: Remove PyThreadState_DeleteCurrent() function (GH-15315)
* Rename PyThreadState_DeleteCurrent()
  to _PyThreadState_DeleteCurrent()
* Move it to the internal C API

Co-Authored-By: Carol Willing <carolcode@willingconsulting.com>
2019-09-05 18:06:49 +02:00
Steve Dower 772ec0fad5
bpo-38030: Fix os.stat failures on block devices on Windows (GH-15681) 2019-09-04 14:42:54 -07:00
Greg Price 2f09413947 closes bpo-37966: Fully implement the UAX #15 quick-check algorithm. (GH-15558)
The purpose of the `unicodedata.is_normalized` function is to answer
the question `str == unicodedata.normalized(form, str)` more
efficiently than writing just that, by using the "quick check"
optimization described in the Unicode standard in UAX #15.

However, it turns out the code doesn't implement the full algorithm
from the standard, and as a result we often miss the optimization and
end up having to compute the whole normalized string after all.

Implement the standard's algorithm.  This greatly speeds up
`unicodedata.is_normalized` in many cases where our partial variant
of quick-check had been returning MAYBE and the standard algorithm
returns NO.

At a quick test on my desktop, the existing code takes about 4.4 ms/MB
(so 4.4 ns per byte) when the partial quick-check returns MAYBE and it
has to do the slow normalize-and-compare:

  $ build.base/python -m timeit -s 'import unicodedata; s = "\uf900"*500000' \
      -- 'unicodedata.is_normalized("NFD", s)'
  50 loops, best of 5: 4.39 msec per loop

With this patch, it gets the answer instantly (58 ns) on the same 1 MB
string:

  $ build.dev/python -m timeit -s 'import unicodedata; s = "\uf900"*500000' \
      -- 'unicodedata.is_normalized("NFD", s)'
  5000000 loops, best of 5: 58.2 nsec per loop

This restores a small optimization that the original version of this
code had for the `unicodedata.normalize` use case.

With this, that case is actually faster than in master!

$ build.base/python -m timeit -s 'import unicodedata; s = "\u0338"*500000' \
    -- 'unicodedata.normalize("NFD", s)'
500 loops, best of 5: 561 usec per loop

$ build.dev/python -m timeit -s 'import unicodedata; s = "\u0338"*500000' \
    -- 'unicodedata.normalize("NFD", s)'
500 loops, best of 5: 512 usec per loop
2019-09-03 19:45:44 -07:00
Steve Dower 993ac92418
bpo-38020: Fixes crash in os.readlink() on Windows (GH-15663) 2019-09-03 12:50:51 -07:00
Dong-hee Na 0cf832a9ef bpo-37798: Fix _statistics module doc (GH-15546) 2019-09-03 12:21:45 +03:00
Serhiy Storchaka 1f21eaa15e
bpo-15999: Clean up of handling boolean arguments. (GH-15610)
* Use the 'p' format unit instead of manually called PyObject_IsTrue().
* Pass boolean value instead 0/1 integers to functions that needs boolean.
* Convert some arguments to boolean only once.
2019-09-01 12:16:51 +03:00
Serhiy Storchaka 5eca7f3f38
bpo-15999: Always pass bool instead of int to socket.setblocking(). (GH-15621) 2019-09-01 12:12:52 +03:00
Serhiy Storchaka 41c57b3353
bpo-37994: Fix silencing all errors if an attribute lookup fails. (GH-15630)
Only AttributeError should be silenced.
2019-09-01 12:03:39 +03:00
Serhiy Storchaka f02ea6225b
bpo-36543: Remove old-deprecated ElementTree features. (GH-12707)
Remove methods Element.getchildren(), Element.getiterator() and
ElementTree.getiterator() and the xml.etree.cElementTree module.
2019-09-01 11:18:35 +03:00
Inada Naoki 013e52fd34
bpo-37990: fix gc stats (GH-15626) 2019-08-31 09:13:42 +09:00
Min ho Kim 39d87b5471 Fix typos mostly in comments, docs and test names (GH-15209) 2019-08-30 16:21:19 -04:00
Victor Stinner 96b4087ce7
bpo-37140: Fix StructUnionType_paramfunc() (GH-15612)
Fix a ctypes regression of Python 3.8. When a ctypes.Structure is
passed by copy to a function, ctypes internals created a temporary
object which had the side effect of calling the structure finalizer
(__del__) twice. The Python semantics requires a finalizer to be
called exactly once. Fix ctypes internals to no longer call the
finalizer twice.

Create a new internal StructParam_Type which is only used by
_ctypes_callproc() to call PyMem_Free(ptr) on Py_DECREF(argument).
StructUnionType_paramfunc() creates such object.
2019-08-30 14:30:33 +02:00
Sergey Fedoseev 6a650aaf77 bpo-37976: Prevent shadowing of TypeError in zip() (GH-15592) 2019-08-29 21:25:48 -07:00
Thomas A Caswell e278335a6e bpo-37933: Fix faulthandler.cancel_dump_traceback_later() (GH-15440)
Fix faulthandler.cancel_dump_traceback_later() call
if cancel_dump_traceback_later() was not called previously.
2019-08-29 18:30:04 +02:00
Rémi Lapeyre 4901fe274b bpo-37034: Display argument name on errors with keyword arguments with Argument Clinic. (GH-13593) 2019-08-29 17:49:08 +03:00
Joannah Nanjekye 2c5fb17118 bpo-36833: Add tests for Datetime C API Macros (GH-14842)
Added tests for PyDateTime_xxx_GET_xxx() macros of the C API of
the datetime module.
2019-08-29 14:54:46 +02:00
Justin Blanchard 122376df55 bpo-37372: Fix error unpickling datetime.time objects from Python 2 with seconds>=24. (GH-14307) 2019-08-29 10:36:15 +03:00
Serhiy Storchaka b235a1b473
bpo-37960: Silence only necessary errors in repr() of buffered and text streams. (GH-15543) 2019-08-29 09:25:22 +03:00
HongWeipeng fa220ec763 Raise a RuntimeError when tee iterator is consumed from different threads (GH-15567) 2019-08-28 20:39:25 -07:00
Vinay Sharma 13f37f2ba8 closes bpo-37964: add F_GETPATH command to fcntl (GH-15550)
https://bugs.python.org/issue37964



Automerge-Triggered-By: @benjaminp
2019-08-28 18:56:17 -07:00
Christian Heimes 98d90f745d
bpo-37951: Lift subprocess's fork() restriction (GH-15544) 2019-08-27 23:36:56 +02:00
vrajivk 8bf5fef873 bpo-36205: Fix the rusage implementation of time.process_time() (GH-15538) 2019-08-27 00:13:12 -04:00
Raymond Hettinger 6fee0f8ea7
bpo-37798: Minor code formatting and comment clean-ups. (GH-15526) 2019-08-26 11:25:58 -07:00
Inada Naoki b27cbec801 bpo-37055: fix warnings in _blake2 module (GH-14646)
https://bugs.python.org/issue37055



Automerge-Triggered-By: @tiran
2019-08-26 10:52:36 -07:00
Raymond Hettinger aef9ad82f7
bpo-37942: Improve argument clinic float converter (GH-15470) 2019-08-24 19:10:39 -07:00
Dong-hee Na 0a18ee4be7 bpo-37798: Add C fastpath for statistics.NormalDist.inv_cdf() (GH-15266) 2019-08-23 15:20:30 -07:00
Pablo Galindo 4be11c009a bpo-37915: Fix comparison between tzinfo objects and timezone objects (GH-15390)
https://bugs.python.org/issue37915



Automerge-Triggered-By: @pablogsal
2019-08-22 12:24:25 -07:00
Steve Dower df2d4a6f3d
bpo-37834: Normalise handling of reparse points on Windows (GH-15231)
bpo-37834: Normalise handling of reparse points on Windows
* ntpath.realpath() and nt.stat() will traverse all supported reparse points (previously was mixed)
* nt.lstat() will let the OS traverse reparse points that are not name surrogates (previously would not traverse any reparse point)
* nt.[l]stat() will only set S_IFLNK for symlinks (previous behaviour)
* nt.readlink() will read destinations for symlinks and junction points only

bpo-1311: os.path.exists('nul') now returns True on Windows
* nt.stat('nul').st_mode is now S_IFCHR (previously was an error)
2019-08-21 15:27:33 -07:00
Stefan Krah bcc446f525
Revert mode change that loses information in directory listings on Linux. (#15366) 2019-08-21 23:00:04 +02:00
Victor Stinner d8c5adf6f8
bpo-37851: faulthandler allocates its stack on demand (GH-15358)
The faulthandler module no longer allocates its alternative stack at
Python startup. Now the stack is only allocated at the first
faulthandler usage.

faulthandler no longer ignores memory allocation failure when
allocating the stack. sigaltstack() failure now raises an OSError
exception, rather than being ignored.

The alternative stack is no longer used if sigaction() is
not available. In practice, sigaltstack() should only be available
when sigaction() is avaialble, so this change should have no effect
in practice.

faulthandler.dump_traceback_later() internal locks are now only
allocated at the first dump_traceback_later() call, rather than
always being allocated at Python startup.
2019-08-21 13:40:42 +01:00
Greg Price 9ece4a5057 Unmark files as executable that can't actually be executed. (GH-15353)
There are plenty of legitimate scripts in the tree that begin with a
`#!`, but also a few that seem to be marked executable by mistake.

Found them with this command -- it gets executable files known to Git,
filters to the ones that don't start with a `#!`, and then unmarks
them as executable:

    $ git ls-files --stage \
      | perl -lane 'print $F[3] if (!/^100644/)' \
      | while read f; do
          head -c2 "$f" | grep -qxF '#!' \
          || chmod a-x "$f"; \
        done

Looking at the list by hand confirms that we didn't sweep up any
files that should have the executable bit after all.  In particular

 * The `.psd` files are images from Photoshop.

 * The `.bat` files sure look like things that can be run.
   But we have lots of other `.bat` files, and they don't have
   this bit set, so it must not be needed for them.



Automerge-Triggered-By: @benjaminp
2019-08-20 21:53:59 -07:00
Brett Cannon 1407038e0b
Remove a dead comment from ossaudiodev.c (#15346) 2019-08-20 12:20:47 -07:00
Joannah Nanjekye 9e66aba999 bpo-15913: Implement PyBuffer_SizeFromFormat() (GH-13873)
Implement PyBuffer_SizeFromFormat() function (previously
documented but not implemented): call struct.calcsize().
2019-08-20 15:46:36 +01:00
Alex Gaynor 40dad9545a Replace usage of the obscure PEM_read_bio_X509_AUX with the more standard PEM_read_bio_X509 (GH-15303)
X509_AUX is an odd, note widely used, OpenSSL extension to the X509 file format. This function doesn't actually use any of the extra metadata that it parses, so just use the standard API.

Automerge-Triggered-By: @tiran
2019-08-15 05:31:28 -07:00
Victor Stinner ac827edc49
bpo-21131: Fix faulthandler.register(chain=True) stack (GH-15276)
faulthandler now allocates a dedicated stack of SIGSTKSZ*2 bytes,
instead of just SIGSTKSZ bytes. Calling the previous signal handler
in faulthandler signal handler uses more than SIGSTKSZ bytes of stack
memory on some platforms.
2019-08-14 23:35:27 +02:00
Artem Khramov 2814620657 bpo-37811: FreeBSD, OSX: fix poll(2) usage in sockets module (GH-15202)
FreeBSD implementation of poll(2) restricts the timeout argument to be
either zero, or positive, or equal to INFTIM (-1).

Unless otherwise overridden, socket timeout defaults to -1. This value
is then converted to milliseconds (-1000) and used as argument to the
poll syscall. poll returns EINVAL (22), and the connection fails.

This bug was discovered during the EINTR handling testing, and the
reproduction code can be found in
https://bugs.python.org/issue23618 (see connect_eintr.py,
attached). On GNU/Linux, the example runs as expected.

This change is trivial:
If the supplied timeout value is negative, truncate it to -1.
2019-08-14 23:21:48 +02:00
Victor Stinner 077af8c2c9
bpo-37738: Fix curses addch(str, color_pair) (GH-15071)
Fix the implementation of curses addch(str, color_pair): pass the
color pair to setcchar(), instead of always passing 0 as the color
pair.
2019-08-14 12:31:43 +02:00
Greg Price 51aac15f6d Delete leftover clinic-generated file for C zipimport. (GH-15174) 2019-08-10 10:20:27 +03:00
Ngalim Siregar 92c7e30adf bpo-37642: Update acceptable offsets in timezone (GH-14878)
This fixes an inconsistency between the Python and C implementations of
the datetime module. The pure python version of the code was not
accepting offsets greater than 23:59 but less than 24:00. This is an
accidental legacy of the original implementation, which was put in place
before tzinfo allowed sub-minute time zone offsets.

GH-14878
2019-08-09 10:22:16 -04:00
Inada Naoki 2a570af12a
bpo-37587: optimize json.loads (GH-15134)
Use a tighter scope temporary variable to help register allocation.
1% speedup for large string.

Use PyDict_SetItemDefault() for memoizing keys.
At most 4% speedup when the cache hit ratio is low.
2019-08-08 17:57:10 +09:00
Sergey Fedoseev 3e41f3cabb bpo-34488: optimize BytesIO.writelines() (GH-8904)
Avoid the creation of unused int object for each line.
2019-08-07 09:38:31 +09:00
Serhiy Storchaka 18b711c5a7
bpo-37648: Fixed minor inconsistency in some __contains__. (GH-14904)
The collection's item is now always at the left and
the needle is on the right of ==.
2019-08-04 14:12:48 +03:00
Serhiy Storchaka 17e52649c0
bpo-37685: Fixed comparisons of datetime.timedelta and datetime.timezone. (GH-14996)
There was a discrepancy between the Python and C implementations.

Add singletons ALWAYS_EQ, LARGEST and SMALLEST in test.support
to test mixed type comparison.
2019-08-04 12:38:46 +03:00
Greg Bowser 8fbece135d bpo-36590: Add Bluetooth RFCOMM and support for Windows. (GH-12767)
Support for RFCOMM, L2CAP, HCI, SCO is based on the BTPROTO_* macros
being defined. Winsock only supports RFCOMM, even though it has a
BTHPROTO_L2CAP macro. L2CAP support would build on windows, but not
necessarily work.

This also adds some basic unittests for constants (all of which existed
prior to this commit, just not on windows) and creating sockets.

pair: Nate Duarte <slacknate@gmail.com>
2019-08-02 13:29:52 -07:00
Inada Naoki bf8162c8c4
bpo-37729: gc: write stats at once (GH-15050)
gc used several PySys_WriteStderr() calls to write stats.
It caused stats mixed up when stderr is shared by multiple
processes like this:

  gc: collecting generation 2...
  gc: objects in each generation: 0 0gc: collecting generation 2...
  gc: objects in each generation: 0 0 126077 126077
  gc: objects in permanent generation: 0

  gc: objects in permanent generation: 0
  gc: done, 112575 unreachable, 0 uncollectablegc: done, 112575 unreachable, 0 uncollectable, 0.2223s elapsed
  , 0.2344s elapsed
2019-08-02 16:25:29 +09:00
Anthony Sottile c9345e382c bpo-37695: Correct unget_wch error message. (GH-14986) 2019-07-31 15:11:24 +03:00
karl ding 31c4fd2a10 bpo-37085: Expose SocketCAN bcm_msg_head flags (#13646)
Expose the CAN_BCM SocketCAN constants used in the bcm_msg_head struct
flags (provided by <linux/can/bcm.h>) under the socket library.

This adds the following constants with a CAN_BCM prefix:

  * SETTIMER
  * STARTTIMER
  * TX_COUNTEVT
  * TX_ANNOUNCE
  * TX_CP_CAN_ID
  * RX_FILTER_ID
  * RX_CHECK_DLC
  * RX_NO_AUTOTIMER
  * RX_ANNOUNCE_RESUME
  * TX_RESET_MULTI_IDX
  * RX_RTR_FRAME
  * CAN_FD_FRAME

The CAN_FD_FRAME flag was introduced in the 4.8 kernel, while the other
ones were present since SocketCAN drivers were mainlined in 2.6.25. As
such, it is probably unnecessary to guard against these constants being
missing.
2019-07-31 10:47:16 +02:00
Min ho Kim c4cacc8c5e Fix typos in comments, docs and test names (#15018)
* Fix typos in comments, docs and test names

* Update test_pyparse.py

account for change in string length

* Apply suggestion: splitable -> splittable

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Apply suggestion: splitable -> splittable

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Apply suggestion: Dealloccte -> Deallocate

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Update posixmodule checksum.

* Reverse idlelib changes.
2019-07-30 18:16:13 -04:00
Marco Paolini 8a758f5b99 bpo-37587: Make json.loads faster for long strings (GH-14752)
When scanning the string, most characters are valid, so
checking for invalid characters first means never needing
to check the value of strict on valid strings, and only
needing to check it on invalid characters when doing
non-strict parsing of invalid strings.

This provides a measurable reduction in per-character
processing time (~11% in the pre-merge patch testing).
2019-07-31 00:16:34 +10:00
Pablo Galindo 9211e2fd81 bpo-37268: Add deprecation notice and a DeprecationWarning for the parser module (GH-15017)
Deprecate the parser module and add a deprecation warning triggered on import and a warning block in the documentation.





https://bugs.python.org/issue37268



Automerge-Triggered-By: @pablogsal
2019-07-30 04:04:01 -07:00
Raymond Hettinger 6b5f1b496f
bpo-37691: Let math.dist() accept sequences and iterables for coordinates (GH-14975) 2019-07-27 14:04:29 -07:00
Markus Mohrhard 898318b53d bpo-37502: handle default parameter for buffers argument of pickle.loads correctly (GH-14593) 2019-07-25 18:00:34 +02:00
Stefan Behnel c6cb4cdd21
bpo-37399: Correctly attach tail text to the last element/comment/pi (GH-14856)
* bpo-37399: Correctly attach tail text to the last element/comment/pi, even when comments or pis are discarded.
Also fixes the insertion of PIs when "insert_pis=True" is configured for a TreeBuilder.
2019-07-24 20:08:02 +02:00
Min ho Kim 96e12d5f4f Fix typos in docs, comments and test assert messages (#14872) 2019-07-21 16:12:33 -04:00
Hai Shi 5623ac87bb bpo-37476: Adding tests for asutf8 and asutf8andsize (GH-14531) 2019-07-20 15:56:23 +08:00
Zackery Spytz d395209653 bpo-37547: Fix a compiler warning in winconsoleio.c (GH-14785)
The compiler warning was introduced in
59ad110d7a.
2019-07-19 15:07:06 +09:00
Sergey Fedoseev 1c5e68e714 bpo-34749: Improved performance of binascii.a2b_base64(). (GH-9444)
https://bugs.python.org/issue34749
2019-07-14 05:15:32 -07:00
Mark Dickinson 014847034b
Fix inconsequential typo in math.remainder algorithm comments. (#14746) 2019-07-13 16:50:03 +01:00
Jeroen Demeyer ed184c06e2 bpo-37358: Use vectorcall for functools.partial (GH-14284)
https://bugs.python.org/issue37358
2019-07-13 07:39:18 -07:00
Giovanni Cappellotto dc3f99fa77 bpo-37548: Document range of atan, acos and asin (GH-14717) 2019-07-13 14:59:55 +01:00
gescheit b9a0376b0d closes bpo-37347: Fix refcount problem in sqlite3. (GH-14268) 2019-07-12 20:15:48 -07:00
Disconnect3d 13ab570feb Remove redundant check from arraymodule b_getitem (GH-14676)
The `arraymodule`'s `b_getitem` function returns a `PyLong` converted 
from `arrayobject`'s array, by dereferencing a pointer to `char`.

When the `char` type is `signed char`, the `if (x >= 128) x -= 256;` comparison/code is redundant because a `signed char` will have a value of `[-128, 127]` and so `x` will never be greater or equal than 128.

This check was indeed needed for situations where a given compiler would assume `char` being `unsigned char` which would make `x` in `[0, 256]` range.

However, the check can be removed if we cast the `ob_item` into a signed char pointer (`signed char*`) instead of `char*`.

This PR/commit introduces this change.
2019-07-11 14:57:42 -07:00
Jeroen Demeyer 1dbd084f1f bpo-29548: no longer use PyEval_Call* functions (GH-14683) 2019-07-12 00:57:32 +09:00
Jeroen Demeyer 59ad110d7a bpo-37547: add _PyObject_CallMethodOneArg (GH-14685) 2019-07-11 17:59:05 +09:00
Stefan Krah f117d871c4
Rename memory.c to mpalloc.c for consistency with the header file. (#14687) 2019-07-10 18:27:38 +02:00
Stefan Krah 4749dbe54c
Really remove vcstdint.h. (#14686) 2019-07-10 17:55:48 +02:00
Hai Shi 3a3db970de bpo-27679: Remove set_bitfields() from _ctypes_test (GH-14648) 2019-07-09 20:00:27 +02:00
Victor Stinner 76611c7c0a bpo-37120: Fix _ssl get_num_tickets() (GH-14668)
Replace PyLong_FromLong() with PyLong_FromSize_t():
SSL_CTX_get_num_tickets() return type is size_t.


https://bugs.python.org/issue37120
2019-07-09 04:30:52 -07:00
Jeroen Demeyer 762f93ff2e bpo-37337: Add _PyObject_CallMethodNoArgs() (GH-14267) 2019-07-08 17:19:25 +09:00
Min ho Kim f7d72e48fb Fix some typos (GH-14435) 2019-07-05 14:39:32 -07:00
Jeroen Demeyer 7f41c8e0dd bpo-37493: use _PyObject_CallNoArg in more places (GH-14575) 2019-07-04 19:35:31 +09:00
Jeroen Demeyer 196a530e00 bpo-37483: add _PyObject_CallOneArg() function (#14558) 2019-07-04 19:31:34 +09:00
Benjamin Peterson 2cd07920bb
Put pyexpatns.h include back. bpo-37437 (GH-14539) 2019-07-01 22:07:44 -07:00
Steve Dower e226e83d36
bpo-37363: Add audit events on startup for the run commands (GH-14524) 2019-07-01 16:03:53 -07:00
Victor Stinner 67310023f2
bpo-36763: Use PyConfig_Clear() (GH-14445)
Stop using "static PyConfig", PyConfig must now always use
dynamically allocated strings: use PyConfig_SetString(),
PyConfig_SetArgv() and PyConfig_Clear().
2019-07-01 19:52:45 +02:00
Christian Heimes f0f5930ac8 bpo-37428: Don't set PHA verify flag on client side (GH-14421)
SSLContext.post_handshake_auth = True no longer sets
SSL_VERIFY_POST_HANDSHAKE verify flag for client connections. Although the
option is documented as ignored for clients, OpenSSL implicitly enables cert
chain validation when the flag is set.

Signed-off-by: Christian Heimes <christian@python.org>



https://bugs.python.org/issue37428
2019-06-30 23:29:17 -07:00
Ammar Askar 5cbbbd73a6 bpo-29505: Add more fuzzing for re.compile, re.load and csv.reader (GH-14255)
Add more fuzz testing for re.compile, re.load and csv.reader
2019-06-29 22:54:42 -07:00
Victor Stinner ec3e20a2d1
bpo-37412: Fix os.getcwd() for long path on Windows (GH-14424)
* Fix test for integer overflow.
* Add an unit test.
2019-06-28 18:01:59 +02:00
Victor Stinner 3029035ef3
bpo-20443: Fix calculate_program_full_path() warning (GH-14446)
Don't call _Py_isabs() with a bytes string (char*), the function
expects as wide string.
2019-06-28 16:49:38 +02:00
Jeroen Demeyer b1263d5a60 bpo-37337: Add _PyObject_VectorcallMethod() (GH-14228) 2019-06-28 18:49:00 +09:00
Benjamin Peterson 3b03b09fc9
closes bpo-37437: Update vendorized expat to 2.2.7. (GH-14436) 2019-06-27 20:54:44 -07:00
Brandt Bucher 45a30af109 closes bpo-37420: Handle errors during iteration in os.sched_setaffinity. (GH-14414) 2019-06-27 09:10:57 -07:00
Zackery Spytz d52a83a3d4 bpo-37419: Fix possible segfaults when passing large sequences to os.posix_spawn() (GH-14409)
Use Py_ssize_t instead of int for i.
2019-06-26 21:54:19 +01:00
Victor Stinner 689830ee62
bpo-37412: os.getcwdb() now uses UTF-8 on Windows (GH-14396)
The os.getcwdb() function now uses the UTF-8 encoding on Windows,
rather than the ANSI code page: see PEP 529 for the rationale. The
function is no longer deprecated on Windows.

os.getcwd() and os.getcwdb() now detect integer overflow on memory
allocations. On Unix, these functions properly report MemoryError on
memory allocation failure.
2019-06-26 17:31:12 +02:00
Victor Stinner c6a2320e87
bpo-37406: sqlite3 raises TypeError for wrong operation type (GH-14386)
The sqlite3 module now raises TypeError, rather than ValueError, if
operation argument type is not str: execute(), executemany() and
calling a connection.
2019-06-26 03:16:24 +02:00
Victor Stinner 22eb689cf3
bpo-37388: Development mode check encoding and errors (GH-14341)
In development mode and in debug build, encoding and errors arguments
are now checked on string encoding and decoding operations. Examples:
open(), str.encode() and bytes.decode().

By default, for best performances, the errors argument is only
checked at the first encoding/decoding error, and the encoding
argument is sometimes ignored for empty strings.
2019-06-26 00:51:05 +02:00
Victor Stinner 3939c321c9
bpo-20443: _PyConfig_Read() gets the absolute path of run_filename (GH-14053)
Python now gets the absolute path of the script filename specified on
the command line (ex: "python3 script.py"): the __file__ attribute of
the __main__ module, sys.argv[0] and sys.path[0] become an absolute
path, rather than a relative path.

* Add _Py_isabs() and _Py_abspath() functions.
* _PyConfig_Read() now tries to get the absolute path of
  run_filename, but keeps the relative path if _Py_abspath() fails.
* Reimplement os._getfullpathname() using _Py_abspath().
* Use _Py_isabs() in getpath.c.
2019-06-25 15:02:43 +02:00
ziheng 6ffd9b05df bpo-32627: Fix compile error when conflicting `_uuid` headers included (GH-11751) 2019-06-24 13:59:50 -04:00
Steve Dower 60419a7e96
bpo-37363: Add audit events for a range of modules (GH-14301) 2019-06-24 08:42:54 -07:00
Gabe Appleton 2ac3bab2a6 bpo-37345: Add formal UDPLITE support (GH-14258)
At the moment you can definitely use UDPLITE sockets on Linux systems, but it would be good if this support were formalized such that you can detect support at runtime easily.

At the moment, to make and use a UDPLITE socket requires something like the following code:

```
>>> import socket
>>> a = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, 136)
>>> b = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, 136)
>>> a.bind(('localhost', 44444))
>>> b.sendto(b'test'*256, ('localhost', 44444))
>>> b.setsockopt(136, 10, 16)
>>> b.sendto(b'test'*256, ('localhost', 44444))
>>> b.setsockopt(136, 10, 32)
>>> b.sendto(b'test'*256, ('localhost', 44444))
>>> b.setsockopt(136, 10, 64)
>>> b.sendto(b'test'*256, ('localhost', 44444))
```

If you look at this through Wireshark, you can see that the packets are different in that the checksums and checksum coverages change.

With the pull request that I am submitting momentarily, you could do the following code instead:

```
>>> import socket
>>> a = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDPLITE)
>>> b = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDPLITE)
>>> a.bind(('localhost', 44444))
>>> b.sendto(b'test'*256, ('localhost', 44444))
>>> b.set_send_checksum_coverage(16)
>>> b.sendto(b'test'*256, ('localhost', 44444))
>>> b.set_send_checksum_coverage(32)
>>> b.sendto(b'test'*256, ('localhost', 44444))
>>> b.set_send_checksum_coverage(64)
>>> b.sendto(b'test'*256, ('localhost', 44444))
```

One can also detect support for UDPLITE just by checking

```
>>> hasattr(socket, 'IPPROTO_UDPLITE')
```


https://bugs.python.org/issue37345
2019-06-24 02:58:56 -07:00
Zackery Spytz 08286d52b2 bpo-37316: mmap.mmap() passes the wrong variable to PySys_Audit() (GH-14152)
Also, add a missing call to va_end() in PySys_Audit().
2019-06-21 08:31:59 -07:00
Benjamin Peterson 7821b4c6d2
Fix name of '\0'. (GH-14222)
'\0' is the NUL byte not NULL.
2019-06-18 21:37:58 -07:00
Victor Stinner 01b63ecac6
bpo-35134: Add Include/cpython/import.h header file (GH-14213)
* Add Include/cpython/import.h and Include/internal/pycore_import.h
  header files.
* Move _PyImport_ReInitLock() to the internal C API. Don't export the
  symbol anymore.
2019-06-19 00:48:09 +02:00
Serhiy Storchaka 231aad3849
bpo-37315: Deprecate accepting floats in math.factorial(). (GH-14147) 2019-06-17 16:57:27 +03:00
Victor Stinner 2ff58a24e8
bpo-37194: Add a new public PyObject_CallNoArgs() function (GH-13890)
Add a new public PyObject_CallNoArgs() function to the C API: call a
callable Python object without any arguments.

It is the most efficient way to call a callback without any argument.
On x86-64, for example, PyObject_CallFunctionObjArgs(func, NULL)
allocates 960 bytes on the stack per call, whereas
PyObject_CallNoArgs(func) only allocates 624 bytes per call.

It is excluded from stable ABI 3.8.

Replace private _PyObject_CallNoArg() with public
PyObject_CallNoArgs() in C extensions: _asyncio, _datetime,
_elementtree, _pickle, _tkinter and readline.
2019-06-17 14:27:23 +02:00
Mark Dickinson 2dfeaa9222
Turn math.isqrt assertion into a comment to clarify its purpose. (GH-14131) 2019-06-16 17:53:21 +01:00
Mark Dickinson 45e0411eee
Simplify negativity checks in math.comb and math.perm. (GH-13870) 2019-06-16 11:06:06 +01:00
Victor Stinner 066e5b1a91
bpo-37266: Daemon threads are now denied in subinterpreters (GH-14049)
In a subinterpreter, spawning a daemon thread now raises an
exception. Daemon threads were never supported in subinterpreters.
Previously, the subinterpreter finalization crashed with a Pyton
fatal error if a daemon thread was still running.

* Add _thread._is_main_interpreter()
* threading.Thread.start() now raises RuntimeError if the thread is a
  daemon thread and the method is called from a subinterpreter.
* The _thread module now uses Argument Clinic for the new function.
* Use textwrap.dedent() in test_threading.SubinterpThreadingTests
2019-06-14 18:55:22 +02:00
Jeffrey Kintscher 8725c83ed5 bpo-35070: test_getgrouplist may fail on macOS if too many groups (GH-13071) 2019-06-13 03:01:29 -04:00
Victor Stinner 37d66d7d4b
bpo-37253: Add _PyCompilerFlags_INIT macro (GH-14018)
Add a new _PyCompilerFlags_INIT macro to initialize PyCompilerFlags
variables, rather than initializing cf_flags and cf_feature_version
explicitly in each variable.
2019-06-13 02:16:41 +02:00
Paul Monson b4c7defe58 bpo-36779: time.tzname returns empty string on Windows if default cod… (GH-13073)
Calling setlocale(LC_CTYPE, "") on a system where GetACP() returns CP_UTF8 results in empty strings in _tzname[].

This causes time.tzname to be an empty string.
I have reported the bug to the UCRT team and will follow up, but it will take some time get a fix into production.

In the meantime one possible workaround is to temporarily change the locale by calling setlocale(LC_CTYPE, "C") before calling _tzset and restore the current locale after if the GetACP() == CP_UTF8 or CP_UTF7

@zooba 


https://bugs.python.org/issue36779
2019-06-12 16:13:27 -07:00
Ammar Askar a6e190e94b bpo-29505: Fuzz json module, enforce size limit on int(x) fuzz (GH-13991)
* bpo-29505: Enable fuzz testing of the json module, enforce size limit on int(x) fuzz and json input size to avoid timeouts.

Contributed by by Ammar Askar for Google.
2019-06-11 21:30:34 -07:00
aaronpaulhurst 408a2ef1ac closes bpo-35184: Fix XML_POOR_ENTROPY option that breaks makesetup parsing of pyexpat line in Setup. (GH-13064)
When the line is uncommented, the equals character causes it to be incorrectly interpreted
as a macro definition by makesetup.  This results in invalid Makefile output.

The expat code only requires XML_POOR_ENTROPY to be defined; the value is unnecessary.
2019-06-10 18:54:24 -07:00
Raymond Hettinger e119b3d136
bpo-37178: Allow a one argument form of math.perm() (GH-13905) 2019-06-08 08:58:11 -07:00
Ammar Askar a15a7bcaea bpo-29505: Fix interpreter in fuzzing targets to be relocatable (GH-13907) 2019-06-08 07:43:16 -07:00
Victor Stinner 3bf0f3ad20
bpo-37169: Rewrite _PyObject_IsFreed() unit tests (GH-13888)
Replace two Python function calls with a single one to ensure that no
memory allocation is done between the invalid object is created and
when _PyObject_IsFreed() is called.
2019-06-07 16:22:21 +02:00
Eric Wieser 0690c79c41 bpo-37188: Fix a divide-by-zero in arrays of size-0 objects (#13881) 2019-06-07 10:13:26 +02:00
Zackery Spytz dc2476500d bpo-37170: Fix the cast on error in PyLong_AsUnsignedLongLongMask() (GH-13860) 2019-06-06 22:39:23 +02:00
Raymond Hettinger e985804207
bpo-37165: Convert _count_elements to the argument clinic (GH-13848) 2019-06-05 16:05:25 -07:00
Raymond Hettinger 963eb0f473
bpo-35431: Drop the k <= n requirement (GH-13798) 2019-06-04 01:23:06 -07:00
Victor Stinner 0fd2c300c2
Revert "bpo-36818: Add PyInterpreterState.runtime field. (gh-13129)" (GH-13795)
This reverts commit 396e0a8d9d.
2019-06-04 03:15:09 +02:00
Christian Heimes 78c7d52779
bpo-37120: Add SSLContext.num_tickets (GH-13719)
Signed-off-by: Christian Heimes <christian@python.org>
2019-06-03 21:00:10 +02:00
Christian Heimes e35d1ba9ea
bpo-34271: Fix compatibility with 1.0.2 (GH-13728)
Fix various compatibility issues with LibreSSL and OpenSSL 1.0.2
introduced by bpo-34271.

Signed-off-by: Christian Heimes <christian@python.org>
2019-06-03 20:40:15 +02:00
Victor Stinner e225bebc14
Revert "bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (gh-13714)" (GH-13780)
This reverts commit 6a150bcaeb.
2019-06-03 18:14:24 +02:00
Petr Viktorin e584cbff1e
bpo-36027 bpo-36974: Fix "incompatible pointer type" compiler warnings (GH-13758) 2019-06-03 01:08:14 +02:00
Petr Viktorin fb9423fd0a
bpo-36974: Make tp_call=PyVectorcall_Call work for inherited types (GH-13699)
When inheriting a heap subclass from a vectorcall class that sets
`.tp_call=PyVectorcall_Call` (as recommended in PEP 590), the subclass does
not inherit `_Py_TPFLAGS_HAVE_VECTORCALL`, and thus `PyVectorcall_Call` does
not work for it.

This attempts to solve the issue by:
* always inheriting `tp_vectorcall_offset` unless `tp_call` is overridden
  in the subclass
* inheriting _Py_TPFLAGS_HAVE_VECTORCALL for static types, unless `tp_call`
  is overridden
* making `PyVectorcall_Call` ignore `_Py_TPFLAGS_HAVE_VECTORCALL`

This means it'll be ever more important to only call `PyVectorcall_Call`
on classes that support vectorcall. In `PyVectorcall_Call`'s intended role
as `tp_call` filler, that's not a problem.
2019-06-02 23:52:20 +02:00
Serhiy Storchaka 5ae299ac78
bpo-37128: Add math.perm(). (GH-13731) 2019-06-02 11:16:49 +03:00
Eric Snow 6a150bcaeb
bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (gh-13714) 2019-06-01 15:39:46 -06:00
Zackery Spytz 36dcaab7fd Fix the error handling in bytesio_sizeof(). (GH-10459)
bytesio_sizeof() must check if an error has occurred in _PySys_GetSizeOf().
2019-06-02 00:07:45 +03:00
Sergey Fedoseev e5f6207ba6 bpo-34303: Micro-optimizations in functools.reduce() (GH-8598) 2019-06-01 13:32:17 -07:00
Serhiy Storchaka 2b843ac0ae
bpo-35431: Refactor math.comb() implementation. (GH-13725)
* Fixed some bugs.
* Added support for index-likes objects.
* Improved error messages.
* Cleaned up and optimized the code.
* Added more tests.
2019-06-01 22:09:02 +03:00
Yash Aggarwal 4a686504eb bpo-35431: Implemented math.comb (GH-11414) 2019-06-01 00:21:27 -07:00