Pgen is the oldest piece of technology in the CPython repository, building it requires various #if[n]def PGEN hacks in other parts of the code and it also depends more and more on CPython internals. This commit removes the old pgen C code and replaces it for a new version implemented in pure Python. This is a modified and adapted version of lib2to3/pgen2 that can generate grammar files compatibles with the current parser.
This commit also eliminates all the #ifdef and code branches related to pgen, simplifying the code and making it more maintainable. The regen-grammar step now uses $(PYTHON_FOR_REGEN) that can be any version of the interpreter, so the new pgen code maintains compatibility with older versions of the interpreter (this also allows regenerating the grammar with the current CI solution that uses Python3.5). The new pgen Python module also makes use of the Grammar/Tokens file that holds the token specification, so is always kept in sync and avoids having to maintain duplicate token definitions.
The whole coreconfig.h header is now excluded from Py_LIMITED_API.
Move functions definitions into a new internal pycore_coreconfig.h
header.
* Move Include/coreconfig.h to Include/cpython/coreconfig.h
* coreconfig.h header is now excluded from Py_LIMITED_API
* Move functions to pycore_coreconfig.h
This change adds the necessary items to the build projects to avoid erroring out right at the start. It does not add _support_ for targeting Windows on ARM32, but is a necessary prerequisite for adding it.
"Include/token.h", "Lib/token.py" (containing now some data moved from
"Lib/tokenize.py") and new files "Parser/token.c" (containing the code
moved from "Parser/tokenizer.c") and "Doc/library/token-list.inc" (included
in "Doc/library/token.rst") are now generated from "Grammar/Tokens" by
"Tools/scripts/generate_token.py". The script overwrites files only if
needed and can be used on the read-only sources tree.
"Lib/symbol.py" is now generated by "Tools/scripts/generate_symbol_py.py"
instead of been executable itself.
Added new make targets "regen-token" and "regen-symbol" which are now
dependencies of "regen-all".
The documentation contains now strings for operators and punctuation tokens.
Move tupleobject.h code surrounded by "#ifndef Py_LIMITED_API"
to a new Include/cpython/tupleobject.h header file.
Add cpython/ header files to Makefile.pre.in and pythoncore project
of PCbuild.
Include/*.h should be the "portable Python API", whereas
Include/cpython/*.h should be the "CPython API": CPython
implementation details.
Changes:
* Create Include/cpython/ subdirectory
* "make install" now creates $prefix/include/cpython and copy
Include/cpython/* to $prefix/include/cpython
* Create Include/cpython/objimpl.h: move objimpl.h code
surrounded by "#ifndef Py_LIMITED_API" to cpython/objimpl.h.
* objimpl.h now includes cpython/objimpl.h
* Windows installer (MSI) now also install Include/ subdirectories:
Include/cpython/ and Include/internal/.
* And pycore_lifecycle.h and pycore_pathconfig.h headers to
Include/internal/
* Move Py_BUILD_CORE specific code from coreconfig.h and
pylifecycle.h to pycore_pathconfig.h and pycore_lifecycle.h
* Move _Py_wstrlist_XXX() definitions and _PyPathConfig code
from pycore_state.h to pycore_pathconfig.h
* Move "Init" and "Fini" function definitions from pylifecycle.c to
pycore_lifecycle.h.
The accu.h header is no longer part of the Python C API: it has been
moved to the "internal" headers which are restricted to Python
itself.
Replace #include "accu.h" with #include "pycore_accu.h".
Visual Studio solution: Set InlineFunctionExpansion to
OnlyExplicitInline ("/Ob1" option) on all projects (in
pyproject.props) in Debug mode on Win32 and x64 platforms to expand
functions marked as inline.
This change should make Python compiled in Debug mode a little bit
faster on Windows. On Unix, GCC uses -Og optimization level for
./configure --with-pydebug.
* Modify object.h to ensure that pymem.h is included,
to get _Py_tracemalloc_config variable.
* Move _PyTraceMalloc_XXX() functions to tracemalloc.h,
they need PyObject type. Break circular dependency between pymem.h
and object.h.
* Add Include/coreconfig.h
* Move config_*() and _PyCoreConfig_*() functions from Modules/main.c
to a new Python/coreconfig.c file.
* Inline _Py_ReadHashSeed() into config_init_hash_seed()
* Move global configuration variables to coreconfig.c
This is not the ideal solution; this means that a test module is now
always included in the main python3x.dll. However, we're already
including xxsubtype, so why not?
* Document `from __future__ import annotations`
* Provide plumbing and tests for `from __future__ import annotations`
* Implement unparsing the AST back to string form
This is required for PEP 563 and as such only implements a part of the
unparsing process that covers expressions.
* Factorize code from PC/getpathp.c and Modules/getpath.c to remove
duplicated code
* rename pathconfig_clear() to _PyPathConfig_Clear()
* Inline _PyPathConfig_Fini() in pymain_impl() and then remove it,
since it's a oneliner
bpo-32096, bpo-30860: Partially revert the commit
2ebc5ce42a8a9e047e790aefbf9a94811569b2b6:
* Move structures back from Include/internal/mem.h to
Objects/obmalloc.c
* Remove _PyObject_Initialize() and _PyMem_Initialize()
* Remove Include/internal/pymalloc.h
* Add test_capi.test_pre_initialization_api():
Make sure that it's possible to call Py_DecodeLocale(), and then call
Py_SetProgramName() with the decoded string, before Py_Initialize().
PyMem_RawMalloc() and Py_DecodeLocale() can be called again before
_PyRuntimeState_Init().
Co-Authored-By: Eric Snow <ericsnowcurrently@gmail.com>
* Don't use "Python runtime" anymore to parse command line options or
to get environment variables: pymain_init() is now a strict
separation.
* Use an error message rather than "crashing" directly with
Py_FatalError(). Limit the number of calls to Py_FatalError(). It
prepares the code to handle errors more nicely later.
* Warnings options (-W, PYTHONWARNINGS) and "XOptions" (-X) are now
only added to the sys module once Python core is properly
initialized.
* _PyMain is now the well identified owner of some important strings
like: warnings options, XOptions, and the "program name". The
program name string is now properly freed at exit.
pymain_free() is now responsible to free the "command" string.
* Rename most methods in Modules/main.c to use a "pymain_" prefix to
avoid conflits and ease debug.
* Replace _Py_CommandLineDetails_INIT with memset(0)
* Reorder a lot of code to fix the initialization ordering. For
example, initializing standard streams now comes before parsing
PYTHONWARNINGS.
* Py_Main() now handles errors when adding warnings options and
XOptions.
* Add _PyMem_GetDefaultRawAllocator() private function.
* Cleanup _PyMem_Initialize(): remove useless global constants: move
them into _PyMem_Initialize().
* Call _PyRuntime_Initialize() as soon as possible:
_PyRuntime_Initialize() now returns an error message on failure.
* Add _PyInitError structure and following macros:
* _Py_INIT_OK()
* _Py_INIT_ERR(msg)
* _Py_INIT_USER_ERR(msg): "user" error, don't abort() in that case
* _Py_INIT_FAILED(err)
* group the (stateful) runtime globals into various topical structs
* consolidate the topical structs under a single top-level _PyRuntimeState struct
* add a check-c-globals.py script that helps identify runtime globals
Other globals are excluded (see globals.txt and check-c-globals.py).
* group the (stateful) runtime globals into various topical structs
* consolidate the topical structs under a single top-level _PyRuntimeState struct
* add a check-c-globals.py script that helps identify runtime globals
Other globals are excluded (see globals.txt and check-c-globals.py).
Fix logic for retrying nuget.exe download with Python.
Add support for HOST_PYTHON variable.
Clear internal environment variables used in find_python.bat
Use HOST_PYTHON as the actual Python if it is recent enough.
Adds HOST_PYTHON variable to AppVeyor configuration
* bpo-30726: Fix elementtree warnings on Windows
Caused by usage of `getenv` which should be safe. And a few integer
truncations which should also be ok.
* bpo-30726: Don't ignore libexpat warnings which haypo intends to fix upstream
bpo-30726, bpo-29591: libexpat 2.2.1 of Modules/expat/ now uses
a winconfig.h configuration file which already defines:
* XML_NS
* XML_DTD
* BYTEORDER=1234
* XML_CONTEXT_BYTES=1024
* HAVE_MEMMOVE
Remove these defines from PCbuild/_elementtree.vcxproj to prevent
compiler warnings.
Co-Authored-By: Jeremy Kloth <jeremy.kloth@gmail.com>
The Windows build now depends on Python 3.6 to fetch externals, but it will be downloaded via NuGet (which is downloaded via PowerShell) if it is not available via `py -3.6`. This means the only thing that must be installed on a modern Windows box to do a full build of CPython with all extensions is Visual Studio.
Also fixes an outdated note about _lzma in PCbuild/readme.txt
* Improves test_underpth_nosite_file to reveal why it fails.
* Enable building with Windows 10 SDK.
* Fix WinSDK detection
* Fix initialization on Windows when a ._pth file exists.
* Fix tabs
* Adds comment about Py_GetPath call.
PEP 432 specifies a number of large changes to interpreter startup code, including exposing a cleaner C-API. The major changes depend on a number of smaller changes. This patch includes all those smaller changes.
* Adds lib.pyproj file to solution so that VS with Python support can open all the files in the standard library.
* Remove unexpected solution configuration.
* Remove lib.pyproj from solution to avoid memory issues on VS 2015.
Using a response file will eliminate the headache associated with batch argument/quote processing.
For example I unsucessfully compiled python with visualcpptools when specifying VSInstallDir in the batch file (cannot find vcruntime.h)
```batch
build.bat -p x64 -e -M --no-tkinter "/p:VCInstallDir=%VCInstallDir%"
```
but it build successfully when specifying it in a response file
msbuild.rsp:
```
/p:VCInstallDir=%VCInstallDir%
```
Use --short form of git hash. Use output from "git describe" for tag.
Expected outputs:
1. previous hg
2. previous git
3. updated git
Release (tagged) build:
1. Python 3.7.0a0 (v3.7.0a0:4def2a2901a5, ...
2. Python 3.7.0a0 (v3.7.0a0^0:05f53735c8912f8df1077e897f052571e13c3496, ...
3. Python 3.7.0a0 (v3.7.0a0:05f53735c8, ...
Development build:
1. Python 3.7.0a0 (default:41df79263a11, ...
2. Python 3.7.0a0 (master:05f53735c8912f8df1077e897f052571e13c3496, ...
3. Python 3.7.0a0 (heads/master-dirty:05f53735c8, ...
"dirty" means the working tree has uncommitted changes.
See "git help describe" for more info.
Eric Snow
Pablo Galindo
Victor Stinner
Paul Monson
Steve Dower
antektek
Gregory Szorc
Serhiy Storchaka
Jeremy Kloth
Carl Meyer
Ivan Levkivskyi
Zachary Ware
Ned Deily
Guido van Rossum
Yury Selivanov
Antoine Pitrou
Benjamin Peterson
INADA Naoki
Victor Stinner
native-api
Mariatta
Stefan Grönke
Segev Finer
Charles
Xiang Zhang