Commit Graph

743 Commits

Author SHA1 Message Date
Victor Stinner 4d61e6e3b8
Revert: bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (GH-11617) (GH-12159)
* Revert "bpo-36097: Use only public C-API in the_xxsubinterpreters module (adding as necessary). (#12003)"

This reverts commit bcfa450f21.

* Revert "bpo-33608: Simplify ceval's DISPATCH by hoisting eval_breaker ahead of time. (gh-12062)"

This reverts commit bda918bf65.

* Revert "bpo-33608: Use _Py_AddPendingCall() in _PyCrossInterpreterData_Release(). (gh-12024)"

This reverts commit b05b711a2c.

* Revert "bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (GH-11617)"

This reverts commit ef4ac967e2.
2019-03-04 14:21:28 +01:00
Eric Snow bcfa450f21
bpo-36097: Use only public C-API in the_xxsubinterpreters module (adding as necessary). (#12003) 2019-03-01 16:50:31 -07:00
Pablo Galindo 1f24a719e7
bpo-35808: Retire pgen and use pgen2 to generate the parser (GH-11814)
Pgen is the oldest piece of technology in the CPython repository, building it requires various #if[n]def PGEN hacks in other parts of the code and it also depends more and more on CPython internals. This commit removes the old pgen C code and replaces it for a new version implemented in pure Python. This is a modified and adapted version of lib2to3/pgen2 that can generate grammar files compatibles with the current parser.

This commit also eliminates all the #ifdef and code branches related to pgen, simplifying the code and making it more maintainable. The regen-grammar step now uses $(PYTHON_FOR_REGEN) that can be any version of the interpreter, so the new pgen code maintains compatibility with older versions of the interpreter (this also allows regenerating the grammar with the current CI solution that uses Python3.5). The new pgen Python module also makes use of the Grammar/Tokens file that holds the token specification, so is always kept in sync and avoids having to maintain duplicate token definitions.
2019-03-01 15:34:44 -08:00
Victor Stinner 91b9ecf82c
bpo-36142: Add preconfig.c (GH-12128)
* Add _PyArgv_Decode() function
* Move _Py_ClearFileSystemEncoding() and _Py_SetFileSystemEncoding()
  to preconfig.c.
2019-03-01 17:52:56 +01:00
Victor Stinner f684d83d86
bpo-36142: Exclude coreconfig.h from Py_LIMITED_API (GH-12111)
The whole coreconfig.h header is now excluded from Py_LIMITED_API.
Move functions definitions into a new internal pycore_coreconfig.h
header.

* Move Include/coreconfig.h to Include/cpython/coreconfig.h
* coreconfig.h header is now excluded from Py_LIMITED_API
* Move functions to pycore_coreconfig.h
2019-03-01 03:44:13 +01:00
Inada Naoki 001fee14e0
bpo-12822: use monotonic clock for condvar if possible (GH-11723) 2019-02-20 10:00:09 +09:00
Jakub Kulík 5c8f537669 bpo-35520: Fix build with dtrace support on certain systems. (#11194) 2019-01-24 18:29:48 +01:00
Ned Deily 44a3ee07e3 bpo-35257: fix broken BLDSHARED - needs LDFLAGS too (GH-11297)
`BLDSHARED` needs to have both `LDFLAGS` and `LDFLAGS_NODIST`, not just `LDFLAGS_NODIST`; `PY_CORE_LDFLAGS` provides both.  For example, as it stands now with just `LDFLAGS_NODIST`, macOS universal builds are broken as the necessary `-arch` flags are no longer passed to the standard library extension module link step from `setup.py` resulting in extension modules being single architecture only.


https://bugs.python.org/issue35257
2018-12-23 12:32:48 -08:00
Serhiy Storchaka 8ac658114d
bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370)
"Include/token.h", "Lib/token.py" (containing now some data moved from
"Lib/tokenize.py") and new files "Parser/token.c" (containing the code
moved from "Parser/tokenizer.c") and "Doc/library/token-list.inc" (included
in "Doc/library/token.rst") are now generated from "Grammar/Tokens" by
"Tools/scripts/generate_token.py". The script overwrites files only if
needed and can be used on the read-only sources tree.

"Lib/symbol.py" is now generated by "Tools/scripts/generate_symbol_py.py"
instead of been executable itself.

Added new make targets "regen-token" and "regen-symbol" which are now
dependencies of "regen-all".

The documentation contains now strings for operators and punctuation tokens.
2018-12-22 11:18:40 +02:00
stratakis cf10a750f4 bpo-35257: Avoid leaking LTO linker flags into distutils (GH-10900)
When compiling 3rd party C extensions, the linker flags used by the
compiler for the interpreter and the stdlib modules, will get
leaked into distutils. In order to avoid that, the PY_CORE_LDFLAGS
and PY_LDFLAGS_NODIST are introduced to keep those flags separated.
2018-12-19 18:19:01 +01:00
Victor Stinner 640ed520dd
bpo-35499: make profile-opt don't override CFLAGS_NODIST (GH-11164)
"make profile-opt" no longer replaces CFLAGS_NODIST with CFLAGS. It
now adds profile-guided optimization (PGO) flags to CFLAGS_NODIST,
existing CFLAGS_NODIST flags are kept.
2018-12-16 18:00:42 +01:00
jdemeyer 25648d05ac bpo-34245: install Python shared library with more standard 0755 mode (GH-8492) 2018-12-09 02:46:50 -05:00
Victor Stinner 54ba556c6c
bpo-35134: Create Include/cpython/tupleobject.h (GH-10764)
Move tupleobject.h code surrounded by "#ifndef Py_LIMITED_API"
to a new Include/cpython/tupleobject.h header file.

Add cpython/ header files to Makefile.pre.in and pythoncore project
of PCbuild.
2018-11-28 13:01:32 +01:00
Victor Stinner 480833808e
bpo-35134: Update "make tags": add Include/cpython/ (GH-10739)
"make tags" and "make TAGS" now also parse Include/cpython/ header
files.
2018-11-27 12:42:25 +01:00
E. M. Bray da324d53d4 closes bpo-34212: Build core extension modules with Py_BUILD_CORE_BUILTIN. (GH-8712) 2018-11-26 20:21:31 -06:00
Victor Stinner ec13b9322d
bpo-35081: Add Include/internal/pycore_tupleobject.h (GH-10705)
Move _PyTuple_ITEMS() to a new header file:
Include/internal/pycore_tupleobject.h
2018-11-25 23:56:17 +01:00
Victor Stinner e421106b9e
bpo-35134: Create Include/cpython/ subdirectory (GH-10624)
Include/*.h should be the "portable Python API", whereas
Include/cpython/*.h should be the "CPython API": CPython
implementation details.

Changes:

* Create Include/cpython/ subdirectory
* "make install" now creates $prefix/include/cpython and copy
  Include/cpython/* to $prefix/include/cpython
* Create Include/cpython/objimpl.h: move objimpl.h code
  surrounded by "#ifndef Py_LIMITED_API" to cpython/objimpl.h.
* objimpl.h now includes cpython/objimpl.h
* Windows installer (MSI) now also install Include/ subdirectories:
  Include/cpython/ and Include/internal/.
2018-11-23 17:00:00 +01:00
Victor Stinner f653fd4d95
bpo-35296: make install now installs the internal API (GH-10665)
make install now also installs the internal API: Include/internal/*.h
header files.
2018-11-23 16:30:12 +01:00
Victor Stinner 984061eeb4
bpo-35081: Add new internal headers to Makefile (GH-10670)
Add pycore_fileutils.h and pycore_object.h to Makefile.pre.in and to
the pythoncore project of PCbuild/.
2018-11-23 12:30:40 +01:00
Victor Stinner 621cebe81b
bpo-35081: Rename internal headers (GH-10275)
Rename Include/internal/ headers:

* pycore_hash.h -> pycore_pyhash.h
* pycore_lifecycle.h -> pycore_pylifecycle.h
* pycore_mem.h -> pycore_pymem.h
* pycore_state.h -> pycore_pystate.h

Add missing headers to Makefile.pre.in and PCbuild:

* pycore_condvar.h.
* pycore_hamt.h
* pycore_pyhash.h
2018-11-12 16:53:38 +01:00
Victor Stinner a1c249c405
bpo-35081: And pycore_lifecycle.h and pycore_pathconfig.h (GH-10273)
* And pycore_lifecycle.h and pycore_pathconfig.h headers to
  Include/internal/
* Move Py_BUILD_CORE specific code from coreconfig.h and
  pylifecycle.h to pycore_pathconfig.h and pycore_lifecycle.h
* Move _Py_wstrlist_XXX() definitions and _PyPathConfig code
  from pycore_state.h to pycore_pathconfig.h
* Move "Init" and "Fini" function definitions from pylifecycle.c to
  pycore_lifecycle.h.
2018-11-01 03:15:58 +01:00
Victor Stinner e281f7d80c
bpo-35081: Move accu.h to Include/internal/pycore_accu.h (GH-10271)
The accu.h header is no longer part of the Python C API: it has been
moved to the "internal" headers which are restricted to Python
itself.

Replace #include "accu.h" with #include "pycore_accu.h".
2018-11-01 02:30:36 +01:00
Victor Stinner 27e2d1f219
bpo-35081: Add pycore_ prefix to internal header files (GH-10263)
* Rename Include/internal/ header files:

  * pyatomic.h -> pycore_atomic.h
  * ceval.h -> pycore_ceval.h
  * condvar.h -> pycore_condvar.h
  * context.h -> pycore_context.h
  * pygetopt.h -> pycore_getopt.h
  * gil.h -> pycore_gil.h
  * hamt.h -> pycore_hamt.h
  * hash.h -> pycore_hash.h
  * mem.h -> pycore_mem.h
  * pystate.h -> pycore_state.h
  * warnings.h -> pycore_warnings.h

* PCbuild project, Makefile.pre.in, Modules/Setup: add the
  Include/internal/ directory to the search paths of header files.
* Update includes. For example, replace #include "internal/mem.h"
  with #include "pycore_mem.h".
2018-11-01 00:52:28 +01:00
Victor Stinner 31368a4f0e
bpo-35081: Move Include/pyatomic.c to Include/internal/ (GH-10239)
Add pyatomic.h to the VS project (it wasn't referenced).
2018-10-30 15:14:25 +01:00
Victor Stinner 6279c1c500
bpo-35053: Add Include/tracemalloc.h (GH-10091)
* Modify object.h to ensure that pymem.h is included,
  to get _Py_tracemalloc_config variable.
* Move _PyTraceMalloc_XXX() functions to tracemalloc.h,
  they need PyObject type. Break circular dependency between pymem.h
  and object.h.
2018-10-25 15:54:13 +02:00
Benjamin Peterson 65ed12cb7c
closes bpo-34869: Remove LDLAST. (GH-9667) 2018-10-03 19:23:24 -07:00
Serhiy Storchaka 79d1c2e6c9
bpo-25711: Rewrite zipimport in pure Python. (GH-6809) 2018-09-18 22:22:29 +03:00
Victor Stinner 6c785c0ebd
bpo-34170: Add Python/coreconfig.c for _PyCoreConfig (GH-8607)
* Add Include/coreconfig.h
* Move config_*() and _PyCoreConfig_*() functions from Modules/main.c
  to a new Python/coreconfig.c file.
* Inline _Py_ReadHashSeed() into config_init_hash_seed()
* Move global configuration variables to coreconfig.c
2018-08-01 17:56:14 +02:00
Antoine Pitrou 961d54c5c1
bpo-32430: Rename Modules/Setup.dist to Modules/Setup (GH-8229)
bpo-32430: Rename Modules/Setup.dist to Modules/Setup

Remove the necessity to copy the former manually to the latter when updating the local source tree.
2018-07-16 19:03:03 +02:00
Matthias Klose 10f715d712
Enable the verbose build for extension modules with GNU make (#6659) 2018-04-30 20:07:55 +02:00
Brett Cannon a8c342465b
bpo-25427: Remove pyvenv (GH-5962) 2018-04-20 14:15:40 -07:00
xdegaye 063db62aab bpo-32232: by default, Setup modules are no longer built with -DPy_BUILD_CORE (GH-6489)
Setup modules are no longer built with -DPy_BUILD_CORE by default,
as using that flag may now require including additional internal-only header files.

Instead, only the modules that specifically need it use that setting.
2018-04-21 01:03:49 +10:00
Zachary Ware d6ff8a7037
bpo-30638: Add clinic to `make regen-all` (GH-5671)
Also, use PYTHON_FOR_REGEN for clinic and blake2s_impl.c rather than PYTHON_FOR_BUILD, and update .travis.yml to make use of the change.
2018-02-17 17:58:57 -06:00
Guido van Rossum 95e4d58913 String annotations [PEP 563] (#4390)
* Document `from __future__ import annotations`
* Provide plumbing and tests for `from __future__ import annotations`
* Implement unparsing the AST back to string form

This is required for PEP 563 and as such only implements a part of the
unparsing process that covers expressions.
2018-01-26 08:20:18 -08:00
Yury Selivanov f23746a934
bpo-32436: Implement PEP 567 (#5027) 2018-01-22 19:11:18 -05:00
Christian Heimes ff5be6e810
bpo-32598: Use autoconf to detect usable OpenSSL (#5242)
Add https://www.gnu.org/software/autoconf-archive/ax_check_openssl.html
to auto-detect compiler flags, linker flags and libraries to compile
OpenSSL extensions. The M4 macro uses pkg-config and falls back to
manual detection.

Add autoconf magic to detect usable X509_VERIFY_PARAM_set1_host()
and related functions.

Refactor setup.py to use new config vars to compile _ssl and _hashlib
modules.

Signed-off-by: Christian Heimes <christian@python.org>
2018-01-20 13:19:21 +01:00
Christian Heimes ced9cb5303
bpo-32549: Compile OpenSSL 1.1.0 on Travis CI (#5180)
Use an improved version of multissl test helper to compile a local copy
of OpenSSL 1.1.0g.

Signed-off-by: Christian Heimes <christian@python.org>
2018-01-16 21:02:26 +01:00
Ned Deily ca0c5f2656
bpo-32248: install new test_importlib directories (#5138) 2018-01-08 21:07:59 -05:00
Benjamin Peterson e425bd7517
move pygetopt.h to internal (closes bpo-32264) (#4830) 2017-12-14 23:48:12 -08:00
INADA Naoki 7ea143ae79
bpo-29469: Move constant folding to AST optimizer (GH-2858) 2017-12-14 16:47:20 +09:00
Victor Stinner 31a8393cf6
Revert "bpo-32197: Try to fix a compiler error on OS X introduced in bpo-32030. (#4681)" (#4694)
* Revert "bpo-32197: Try to fix a compiler error on OS X introduced in bpo-32030. (#4681)"

This reverts commit 13badcbc60.

Re-apply commits:

* "bpo-32030: _PyPathConfig_Init() sets home and program_name (#4673)"
  commit af5a895073.
* "bpo-32030: Fix config_get_program_name() on macOS (#4669)"
  commit e23c06e2b0.
* "bpo-32030: Add Python/pathconfig.c (#4668)"
  commit 0ea395ae96.
* "bpo-32030: Don't call _PyPathConfig_Fini() in Py_FinalizeEx() (#4667)"
  commit ebac19dad6.
* "bpo-32030: Fix Py_GetPath(): init program_name (#4665)"
  commit 9ac3d88827.

* Fix compilation error on macOS
2017-12-04 13:39:15 +01:00
Serhiy Storchaka 13badcbc60
bpo-32197: Try to fix a compiler error on OS X introduced in bpo-32030. (#4681)
* Revert "bpo-32030: _PyPathConfig_Init() sets home and program_name (#4673)"

This reverts commit af5a895073.

* Revert "bpo-32030: Fix config_get_program_name() on macOS (#4669)"

This reverts commit e23c06e2b0.

* Revert "bpo-32030: Add Python/pathconfig.c (#4668)"

This reverts commit 0ea395ae96.

* Revert "bpo-32030: Don't call _PyPathConfig_Fini() in Py_FinalizeEx() (#4667)"

This reverts commit ebac19dad6.

* Revert "bpo-32030: Fix Py_GetPath(): init program_name (#4665)"

This reverts commit 9ac3d88827.
2017-12-02 21:36:00 +02:00
Victor Stinner 0ea395ae96
bpo-32030: Add Python/pathconfig.c (#4668)
* Factorize code from PC/getpathp.c and Modules/getpath.c to remove
  duplicated code
* rename pathconfig_clear() to _PyPathConfig_Clear()
* Inline _PyPathConfig_Fini() in pymain_impl() and then remove it,
  since it's a oneliner
2017-12-01 20:50:58 +01:00
Victor Stinner 3be3b97a97
make tags: index also Modules/_ctypes/ (#4648)
Avoid also "cd $(srcdir)" to not change the current directory.
2017-11-30 22:49:10 +01:00
Victor Stinner 9e87e7776f
bpo-32096: Remove obj and mem from _PyRuntime (#4532)
bpo-32096, bpo-30860:  Partially revert the commit
2ebc5ce42a8a9e047e790aefbf9a94811569b2b6:

* Move structures back from Include/internal/mem.h to
  Objects/obmalloc.c
* Remove _PyObject_Initialize() and _PyMem_Initialize()
* Remove Include/internal/pymalloc.h
* Add test_capi.test_pre_initialization_api():
   Make sure that it's possible to call Py_DecodeLocale(), and then call
   Py_SetProgramName() with the decoded string, before Py_Initialize().

PyMem_RawMalloc() and Py_DecodeLocale() can be called again before
_PyRuntimeState_Init().

Co-Authored-By: Eric Snow <ericsnowcurrently@gmail.com>
2017-11-24 12:09:24 +01:00
xdegaye 0de92859ca
bpo-31934: Abort when building out of a not clean source tree (GH-4255) 2017-11-08 16:03:58 +01:00
luzpaz a5293b4ff2 Fix miscellaneous typos (#4275) 2017-11-05 15:37:50 +02:00
Neil Schemenauer 4e38d71a2b
bpo-28643: Record profile-opt build progress with stamp files (#4223)
* bpo-28643: Record profile-opt build progress with stamp files

The profile-opt makefile target is expensive to build. Since the
makefile does not contain complete dependency information for this
target, much extra work can get done if the build is interrupted and
re-started.  Even running "make" a second time will result in a huge
amount of redundant work.

As a minimal fix (rather than removing recursive "make" and adding a
proper dependency graph), split the profile-opt target into parts:

- ensure tree is clean (profile-clean-stamp)
- build with profile generation enabled (profile-gen-stamp)
- run task to generate profile information (profile-run-stamp)
- build optimized Python using above information (profile-opt)

We use "stamp" files to record completion of the steps.  Running
"make clean" will not remove the profile-run-stamp file.

Other minor changes:

- remove the "build_all_use_profile" target.  I don't expect callers
  of the makefile to use this target so that should be safe.

- remove execution of "profile-removal" at end of "profile-opt".  I
  don't see any reason to not to keep the profile information, given
  the cost to generate it.  Removing the "profile-run-stamp" file
  will force re-generation of it.
2017-11-02 10:46:02 -07:00
Victor Stinner 87d332dcdb bpo-31810: Add smelly.py to check exported symbols (#4057)
* Add Tools/scripts/smelly.py: script checking if all symbols
  exported by libpython start with "Py" or "_Py".
* Modify "make smelly" to run smelly.py: the command now fails with a
  non-zero exit code if libpython leaks a "smelly" symbol.
* Travis CI now runs "make smelly"
2017-10-24 01:29:53 -07:00
Benjamin Peterson 6fb0e4a6d0 explicitly list objects for the ar command (#3824)
$^ is not portable.

closes bpo-31625
2017-09-29 09:26:48 -07:00