Commit Graph

102945 Commits

Author SHA1 Message Date
Victor Stinner 7ddd56f4d8
bpo-35233: Rewrite test_embed.InitConfigTests (GH-10524)
* Fix _PyCoreConfig_SetGlobalConfig(): set also Py_FrozenFlag
* Fix _PyCoreConfig_AsDict(): export also xoptions
* Add _Py_GetGlobalVariablesAsDict() and _testcapi.get_global_config()
* test.pythoninfo: dump also global configuration variables
* _testembed now serializes global, core and main configurations
  using JSON to reuse _Py_GetGlobalVariablesAsDict(),
  _PyCoreConfig_AsDict() and _PyMainInterpreterConfig_AsDict(),
  rather than duplicating code.
* test_embed.InitConfigTests now test much more configuration
  variables
2018-11-14 00:24:28 +01:00
Gregory P. Smith 746b2d35ea
bpo-35214: Fix OOB memory access in unicode escape parser (GH-10506)
Discovered using clang's MemorySanitizer when it ran python3's
test_fstring test_misformed_unicode_character_name.

An msan build will fail by simply executing: ./python -c 'u"\N"'
2018-11-13 13:16:54 -08:00
Victor Stinner 00b137c72f
bpo-35233: Fix _PyMainInterpreterConfig_Copy() (GH-10519)
* Fix _PyMainInterpreterConfig_Copy():
   copy 'install_signal_handlers' attribute
* Add _PyMainInterpreterConfig_AsDict()
* Add unit tests on the main interpreter configuration
  to test_embed.InitConfigTests
* test.pythoninfo: log also main_config
2018-11-13 19:59:26 +01:00
Victor Stinner f966e5397e
bpo-29564:_PyMem_DumpTraceback() suggests enabling tracemalloc (GH-10510)
If tracemalloc is not tracing Python memory allocations,
_PyMem_DumpTraceback() now suggests to enable tracemalloc
to get the traceback where the memory block has been allocated.
2018-11-13 15:14:58 +01:00
Eddie Elizondo 474eedfb3d bpo-34784: Fix PyStructSequence_NewType with heap-allocated StructSequence (GH-9665) 2018-11-13 13:09:31 +01:00
Victor Stinner 1a6be91e6f
bpo-35081: Make some _PyGC macros internal (GH-10507)
* Move "GC" macros together:

  * PyObject_IS_GC()
  * _Py_AS_GC()
  * _PyObject_GC_IS_TRACKED()
  * _PyObject_GC_MAY_BE_TRACKED()

* Mark other GC macros as internal (#ifdef Py_BUILD_CORE):

  * _PyGCHead_NEXT(g), _PyGCHead_SET_NEXT(g, p)
  * _PyGCHead_PREV(g), _PyGCHead_SET_PREV(g, p)
  * _PyGCHead_FINALIZED(g), _PyGCHead_SET_FINALIZED(g)
  * _PyGC_FINALIZED(o), _PyGC_SET_FINALIZED(o)
  * _PyObject_GC_TRACK(o), _PyObject_GC_UNTRACK(o)
  * _PyGC_PREV_MASK_FINALIZED
  * _PyGC_PREV_MASK_COLLECTING
  * _PyGC_PREV_SHIFT
  * _PyGC_PREV_MASK

* Replace _PyGC_generation0 with _PyRuntime.gc.generation0
* _queuemodule.c: replace _PyObject_GC_UNTRACK()
  with with PyObject_GC_UnTrack()
* Document that  _PyObject_GC_TRACK() _PyObject_GC_UNTRACK() macros
  have been removed from the public C API.
2018-11-13 12:52:18 +01:00
Paul Ganssle 0d12672b30 bpo-35081: Remove Py_BUILD_CORE from datetime.h (GH-10416)
Datetime macros like PyDate_Check() have two implementations, one using
the C API capsule and one using direct access to the datetime type
symbols defined in _datetimemodule.c. Since the direct access versions
of the macros are only used in _datetimemodule.c, they have been moved
out of "datetime.h" and into _datetimemodule.c.

The _PY_DATETIME_IMPL macro is currently necessary in order to avoid
both duplicate definitions of these macros in _datetimemodule.c and
unnecessary declarations of C API capsule-related macros and varibles in
datetime.h.

Co-Authored-By: Victor Stinner <vstinner@redhat.com>
2018-11-13 09:02:25 +01:00
Gregory P. Smith 3015fb8ce4
bpo-35214: Add _Py_ prefix to MEMORY_SANITIZER def. (GH-10503)
Rename our new MEMORY_SANITIZER define to _Py_MEMORY_SANITIZER.
Project based C Preprocessor namespacing at its finest. :P
2018-11-12 22:01:22 -08:00
Windson yang 0dc1e45dfd Fix a couple documentation typos. (GH-10498)
reproduciblity -> reproducibility
PyPA are the group -> PyPA is the group
2018-11-12 19:56:25 -08:00
Gregory P. Smith e6c77d8301
Disable getc_unlocked() with MemorySanitizer. (GH-10499)
clang's MemorySanitizer understand getc() but does not understand
getc_unlocked().  Workaround: Don't use it on msan builds.
2018-11-12 19:47:13 -08:00
Victor Stinner 2c07c493d2
bpo-29564: warnings suggests to enable tracemalloc (GH-10486)
The warnings module now suggests to enable tracemalloc if the source
is specified, tracemalloc module is available, but tracemalloc is not
tracing memory allocations.
2018-11-13 02:41:00 +01:00
Gregory P. Smith 1584a00815
bpo-35214: Initial clang MemorySanitizer support (GH-10479)
Adds configure flags for msan and ubsan builds to make it easier to enable.
These also encode the detail that address sanitizer and memory sanitizer
should disable pymalloc.

Define MEMORY_SANITIZER when appropriate at build time and adds workarounds
to existing code to mark things as initialized where the sanitizer is otherwise unable to
determine that.  This lets our build succeed under the memory sanitizer.  not all tests
pass without sanitizer failures yet but we're in pretty good shape after this.
2018-11-12 12:07:14 -08:00
Windson yang a9655b7f71 Improve grammar in Glossary. (GH-10474)
a asynchronous generator -> an asynchronous generator
2018-11-12 09:42:38 -08:00
Andrés Delfino 7610f4fe93 Correct grammar mistake in stdtypes.rst (GH-10481) 2018-11-12 09:24:00 -08:00
Andrew Svetlov 7438792607
bpo-30064: Refactor sock_* asyncio API (#10419) 2018-11-12 19:00:22 +02:00
Andrés Delfino 9404e7737b Linkify PEP 8 in unix.rst (GH-10482) 2018-11-12 08:31:57 -08:00
Victor Stinner 621cebe81b
bpo-35081: Rename internal headers (GH-10275)
Rename Include/internal/ headers:

* pycore_hash.h -> pycore_pyhash.h
* pycore_lifecycle.h -> pycore_pylifecycle.h
* pycore_mem.h -> pycore_pymem.h
* pycore_state.h -> pycore_pystate.h

Add missing headers to Makefile.pre.in and PCbuild:

* pycore_condvar.h.
* pycore_hamt.h
* pycore_pyhash.h
2018-11-12 16:53:38 +01:00
Giampaolo Rodola 19c46a4c96
bpo-33695 shutil.copytree() + os.scandir() cache (#7874) 2018-11-12 06:18:15 -08:00
Srinivas Thatiparthy (శ్రీనివాస్ తాటిపర్తి) cd449806fa Minor grammar improvement to io documentation. (GH-10329)
Independently of -> Independent of
2018-11-11 20:06:18 -08:00
Jakub Stasiak 7432f0929f Update subprocess.Popen documentation wrt universal_newlines arg (GH-10337)
* universal_newlines defaulting to False would suggest, that not
  specifying universal_newlines explicitly and setting text to True
  should cause an error, which is not the case.
* The run function didn't have the universal_newlines parameter
  documented
* The check_output function didn't have its text parameter documented
2018-11-11 19:40:42 -08:00
Julien Palard 082875dcd6
bpo-33878: Doc: Fix missing case by simplifying. (GH-7762)
The documentation was not covering multiple targets enclosed by
parenthesis nor multiple targets enclosed by brackets, adding them all
would be heavy, an else cover them all and is lighter to read.
2018-11-12 00:59:39 +01:00
Victor Stinner 5f2df88b63
bpo-35177: Add dependencies between header files (GH-10361)
* ast.h now includes Python-ast.h and node.h
* parsetok.h now includes node.h and grammar.h
* symtable.h now includes Python-ast.h
* Modify asdl_c.py to enhance Python-ast.h:

  * Add #ifndef/#define Py_PYTHON_AST_H to be able to include the header
    twice
  * Add "extern { ... }" for C++
  * Undefine "Yield" macro conflicting with winbase.h

* Remove "#undef Yield" from C files, it's now done in Python-ast.h
* Remove now useless includes in C files
2018-11-12 00:56:19 +01:00
Alexey Izbyshev fd3a91cbf9 closes bpo-35204: Disable thread and memory sanitizers for address_in_range(). (GH-10442)
This function may access memory which is mapped but is considered
free by libc allocator. It behaves so by design, therefore we
need to suppress sanitizer reports.

GCC doesn't support MSan, so disable only TSan for it.
2018-11-11 15:14:51 -08:00
Raymond Hettinger f9ec1b9f52
Neaten the code without any algorithmic change. (GH-10466)
Remove unneeded assertion (we already know so is a PySetObject *).
2018-11-11 14:35:47 -08:00
Andrés Delfino b086c8afdb bpo-33699: Describe try's else clause with the rest of the try clause (GH-7252)
https://bugs.python.org/issue33699
2018-11-11 11:33:51 -08:00
Andrés Delfino 76e8fd7b7e dict insertion order is guaranteed since 3.7 (GH-10431) 2018-11-11 08:56:47 -08:00
Terry Jan Reedy 50ff02b431
bpo-34864: Document two IDLE on MacOS issues. (GH-10456)
The System Preferences Dock "prefer tabs always" setting disables some
IDLE features.  Menus are a bit different than as described for Windows
and Linux.
2018-11-10 23:26:31 -05:00
Srinivas Thatiparthy (శ్రీనివాస్ తాటిపర్తి) b9498e2367 bpo-35202: Remove unused imports in Lib directory. (GH-10446) 2018-11-10 09:22:02 +02:00
Srinivas Thatiparthy (శ్రీనివాస్ తాటిపర్తి) 43a74abb3a bpo-35202: Remove unused imports in idlelib (GH-10438) 2018-11-10 01:45:31 -05:00
Srinivas Thatiparthy (శ్రీనివాస్ తాటిపర్తి) 2d1bc537fe bpo-35202: Remove unused imports in Lib directory. (GH-10445) 2018-11-10 07:45:28 +02:00
David Kleuker 9d43fa1bb2 Doc: Simplify Copyright line in README (GH-10287)
This also makes it consistent with other places like:

- https://docs.python.org/dev/copyright.html
- https://www.python.org/ footer
- e42b705188/Python/getcopyright.c (L7)
2018-11-09 19:45:40 -08:00
Géry Ogam 009b2f0204 Correct a typo in the Unittest documentation (GH-10397)
Co-Authored-By: maggyero <gery.ogam@gmail.com>
2018-11-09 14:34:54 -05:00
Victor Stinner d17a693fa0
bpo-35199: Add an internal _PyTuple_ITEMS() macro (GH-10434)
* _PyTuple_ITEMS() gives access to the tuple->ob_item field and cast the
  first argument to PyTupleObject*. This internal macro is only usable if
  Py_BUILD_CORE is defined.
* Replace &PyTuple_GET_ITEM(ob, 0) with _PyTuple_ITEMS(ob).
* Replace PyTuple_GET_ITEM(op, 1) with &_PyTuple_ITEMS(ob)[1].
2018-11-09 16:56:48 +01:00
Victor Stinner 130893debf
bpo-35081: Internal headers require Py_BUILD_CORE (GH-10363)
* All internal header files now require Py_BUILD_CORE or
  Py_BUILD_CORE_BUILTIN to be defined.
* _json.c is now compiled with Py_BUILD_CORE_BUILTIN to access
  pycore_accu.h header.
* Add an example to Modules/Setup to show how to build _json
  as a built-in module; it requires non trivial compiler options.
2018-11-09 13:03:37 +01:00
Raymond Hettinger 0a18e0510a Hoist the float conversion out of the inner loop. (GH-10430)
Currently, the *n* and *total* variables get converted to floats each time they are multiplied by random().  This minor tweak does the conversion just once and gets a small speedup (approx 3%).
2018-11-09 02:39:50 -08:00
Raymond Hettinger cf5863faab Optimize set.pop() to advance a pointer instead of indexing. (GH-10429)
Gives approx 20% speed-up using clang depending on the number of elements in the set (the less dense the set, the more the speed-up).

Uses the same entry++ logic used elsewhere in the setobject.c code.
2018-11-09 02:31:56 -08:00
Raymond Hettinger b83942c755 Cleanup and improve the regex tokenizer example. (GH-10426)
1) Convert weird field name "typ" to the more standard "type".
2) For the NUMBER type, convert the value to an int() or float().
3) Simplify ``group(kind)`` to the shorter and faster ``group()`` call.
4) Simplify logic go a single if-elif chain to make this easier to extend.
5) Reorder the tests to match the order the tokens are specified.
   This isn't necessary for correctness but does make the example
   easier to follow.
6) Move the "column" calculation before the if-elif chain so that
   users have the option of using this value in error messages.
2018-11-09 01:19:33 -08:00
Raymond Hettinger 216aaaa056 Fixing wording in comment. (GH-10425)
Since the n==1 case just returns *max*, it cannot be larger
than the magnitude of the vector entry.
2018-11-09 01:06:02 -08:00
Alexey Izbyshev 7a69cf47a9 bpo-35194: Fix a wrong constant in cp932 codec (GH-10420)
This typo doesn't affect the result because wrong bits are discarded
on implicit conversion to unsigned char, but it trips UBSan
with -fsanitize=implicit-integer-truncation.





https://bugs.python.org/issue35194
2018-11-08 23:12:06 -08:00
Lisa Roach 0f221d09ca
bpo-24412: Adds cleanUps for setUpClass and setUpModule. (GH-9190) 2018-11-08 18:34:33 -08:00
Gregory P. Smith 49fa4a9f1e
bpo-35193: Fix an off by one error in the RETURN_VALUE case. (GH-10418)
Fix an off by one error in the peephole optimizer when checking for unreachable code beyond a return.

Do a bounds check within find_op so it can return before going past the end as a safety measure.

7db3c48833 (diff-a33329ae6ae0bb295d742f0caf93c137)
introduced this off by one error while fixing another one nearby.

This bug was shipped in all Python 3.6 and 3.7 releases.

The included unittest won't fail unless you do a clang msan build.
2018-11-08 17:55:07 -08:00
Vincent Michel fd512d7645 bpo-35065: Remove `StreamReaderProtocol._untrack_reader` (#10212)
The call to `_untrack_reader` is performed too soon, causing the protocol
to forget about the reader before `connection_lost` can run and feed the
EOF to the reader. See bpo-35065.
2018-11-08 14:21:47 +02:00
Zackery Spytz 5d95312fb8 Replace dead code with an assertion in winreg.c. (GH-10028) 2018-11-08 11:45:00 +02:00
Serhiy Storchaka a44d34e179
bpo-34966: Improve support of method aliases in pydoc. (GH-9823)
Pydoc now does not duplicate docstrings for aliases of inherited methods.
2018-11-08 08:48:11 +02:00
Pablo Galindo 6843ffe453
Revert "bpo-32409: Fix regression in activate.bat on international Windows (GH-10295)" (GH-10403)
This reverts commit c64583b6d3 due to multiple buildbot failures when building it.
2018-11-07 23:55:40 +00:00
Julien Palard 556d50d03d
Doc: Make all versions sidebars the same for consistency. (GH-10288) 2018-11-08 00:11:49 +01:00
Pablo Galindo b93925047a
Fix the construction of subprocess.CalledProcessError in test_venv (GH-10400)
The constructor of subprocess.CalledProcessError in the check_output function had an extra None in it.
2018-11-07 22:21:17 +00:00
Julien Palard beed84ca5e
bpo-35015: Doc: Fix internationalisation of the availability directive. (GH-10360) 2018-11-07 22:42:40 +01:00
Elvis Pranskevichus a6e956bcb0 bpo-34726: Fix handling of hash-based pycs in zipimport. (GH-10327)
Current support for hash-based bytecode files in `zipimport` is rather
sparse, which leads to test failures when the test suite is ran with
the ``SOURCE_DATE_EPOCH`` environment variable set.

This teaches zipimport to handle hash-based pycs properly.
2018-11-07 20:34:59 +02:00
Andrés Delfino bfe1839aa9 Add link to PEP 525 in Expressions. (GH-10333) 2018-11-07 20:12:12 +02:00