Add Include/cpython/bytearrayobject.h and
Include/cpython/bytesobject.h header files.
Move CPython C API from Include/bytesobject.h into a new
Include/cpython/bytesobject.h header file which is included by
Include/bytesobject.h. Do a similar change for
Include/bytearrayobject.h.
Move the dtoa.h header file to the internal C API as pycore_dtoa.h:
it only contains private functions (prefixed by "_Py").
The math and cmath modules must now be compiled with the
Py_BUILD_CORE macro defined.
Move the bytes_methods.h header file to the internal C API as
pycore_bytes_methods.h: it only contains private symbols (prefixed by
"_Py"), except of the PyDoc_STRVAR_shared() macro.
Move listobject.h code surrounded by "#ifndef Py_LIMITED_API"
to a new Include/cpython/listobject.h header file.
Add cpython/ header files to Makefile.pre.in and pythoncore project
of PCbuild.
Add _PyList_CAST() macro.
As Windows 7 is not supported by Python 3.9, we just replace the dynamic load with a static import. Backports will have a different fix to ensure they continue to behave the same.
Note that the support is not actually enabled yet, and so we won't be publishing these packages. However, for those who want to build it themselves (even by reusing the Azure Pipelines definition), it's now relatively easy to enable.
Provide Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() as
regular functions for the limited API. Previously, there were defined
as macros, but these macros didn't work with the limited API which
cannot access PyThreadState.recursion_depth field.
Remove _Py_CheckRecursionLimit from the stable ABI.
Add Include/cpython/ceval.h header file.
Having these in a separate file from the one that's named after the
module in the usual way makes it very easy to miss them when looking
for tests for these two functions.
(In fact when working recently on is_normalized, I'd been surprised to
see no tests for it here and concluded the function had evaded being
tested at all. I'd gone as far as to write up some tests myself
before I spotted this other file.)
Mostly this just means moving all the one file's code into the other,
and moving code from the module toplevel to inside the test class to
keep it tidily separate from the rest of the file's code.
There's one substantive change, which reduces by a bit the amount of
code to be moved: we drop the `x > sys.maxunicode` conditional and all
the `RangeError` logic behind it. Now if that condition ever occurs
it will cause an error at `chr(x)`, and a test failure. That's the
right result because, since PEP 393 in Python 3.3, there is no longer
such a thing as an "unsupported character".
This is the converse of GH-15353 -- in addition to plenty of
scripts in the tree that are marked with the executable bit
(and so can be directly executed), there are a few that have
a leading `#!` which could let them be executed, but it doesn't
do anything because they don't have the executable bit set.
Here's a command which finds such files and marks them. The
first line finds files in the tree with a `#!` line *anywhere*;
the next-to-last step checks that the *first* line is actually of
that form. In between we filter out files that already have the
bit set, and some files that are meant as fragments to be
consumed by one or another kind of preprocessor.
$ git grep -l '^#!' \
| grep -vxFf <( \
git ls-files --stage \
| perl -lane 'print $F[3] if (!/^100644/)' \
) \
| grep -ve '\.in$' -e '^Doc/includes/' \
| while read f; do
head -c2 "$f" | grep -qxF '#!' \
&& chmod a+x "$f"; \
done
* Add Include/cpython/import.h and Include/internal/pycore_import.h
header files.
* Move _PyImport_ReInitLock() to the internal C API. Don't export the
symbol anymore.
The multiprocessing.resource_tracker replaces the multiprocessing.semaphore_tracker module. Other than semaphores, resource_tracker also tracks shared_memory segments. Patch by Pierre Glaser.
Add a new _testinternalcapi module to test the internal C API.
Move _Py_GetConfigsAsDict() function to the internal C API:
_testembed now uses _testinternalcapi to access the function.
Change PyAPI_FUNC(type), PyAPI_DATA(type) and PyMODINIT_FUNC macros
of pyport.h when Py_BUILD_CORE_MODULE is defined.
The Py_BUILD_CORE_MODULE define must be now be used to build a C
extension as a dynamic library accessing Python internals: export the
PyInit_xxx() function in DLL exports on Windows.
Changes:
* Py_BUILD_CORE_BUILTIN and Py_BUILD_CORE_MODULE now imply
Py_BUILD_CORE directy in pyport.h.
* ceval.c compilation now fails with an error if Py_BUILD_CORE is not
defined, just to ensure that Python is build with the correct
defines.
* setup.py now compiles _pickle.c with Py_BUILD_CORE_MODULE define.
* setup.py compiles _json.c with Py_BUILD_CORE_MODULE define, rather
than Py_BUILD_CORE_BUILTIN define
* PCbuild/pythoncore.vcxproj: Add Py_BUILD_CORE_BUILTIN define.
* Revert "bpo-36097: Use only public C-API in the_xxsubinterpreters module (adding as necessary). (#12003)"
This reverts commit bcfa450f21.
* Revert "bpo-33608: Simplify ceval's DISPATCH by hoisting eval_breaker ahead of time. (gh-12062)"
This reverts commit bda918bf65.
* Revert "bpo-33608: Use _Py_AddPendingCall() in _PyCrossInterpreterData_Release(). (gh-12024)"
This reverts commit b05b711a2c.
* Revert "bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (GH-11617)"
This reverts commit ef4ac967e2.
Pgen is the oldest piece of technology in the CPython repository, building it requires various #if[n]def PGEN hacks in other parts of the code and it also depends more and more on CPython internals. This commit removes the old pgen C code and replaces it for a new version implemented in pure Python. This is a modified and adapted version of lib2to3/pgen2 that can generate grammar files compatibles with the current parser.
This commit also eliminates all the #ifdef and code branches related to pgen, simplifying the code and making it more maintainable. The regen-grammar step now uses $(PYTHON_FOR_REGEN) that can be any version of the interpreter, so the new pgen code maintains compatibility with older versions of the interpreter (this also allows regenerating the grammar with the current CI solution that uses Python3.5). The new pgen Python module also makes use of the Grammar/Tokens file that holds the token specification, so is always kept in sync and avoids having to maintain duplicate token definitions.
The whole coreconfig.h header is now excluded from Py_LIMITED_API.
Move functions definitions into a new internal pycore_coreconfig.h
header.
* Move Include/coreconfig.h to Include/cpython/coreconfig.h
* coreconfig.h header is now excluded from Py_LIMITED_API
* Move functions to pycore_coreconfig.h
This change adds the necessary items to the build projects to avoid erroring out right at the start. It does not add _support_ for targeting Windows on ARM32, but is a necessary prerequisite for adding it.
"Include/token.h", "Lib/token.py" (containing now some data moved from
"Lib/tokenize.py") and new files "Parser/token.c" (containing the code
moved from "Parser/tokenizer.c") and "Doc/library/token-list.inc" (included
in "Doc/library/token.rst") are now generated from "Grammar/Tokens" by
"Tools/scripts/generate_token.py". The script overwrites files only if
needed and can be used on the read-only sources tree.
"Lib/symbol.py" is now generated by "Tools/scripts/generate_symbol_py.py"
instead of been executable itself.
Added new make targets "regen-token" and "regen-symbol" which are now
dependencies of "regen-all".
The documentation contains now strings for operators and punctuation tokens.
Move tupleobject.h code surrounded by "#ifndef Py_LIMITED_API"
to a new Include/cpython/tupleobject.h header file.
Add cpython/ header files to Makefile.pre.in and pythoncore project
of PCbuild.
Include/*.h should be the "portable Python API", whereas
Include/cpython/*.h should be the "CPython API": CPython
implementation details.
Changes:
* Create Include/cpython/ subdirectory
* "make install" now creates $prefix/include/cpython and copy
Include/cpython/* to $prefix/include/cpython
* Create Include/cpython/objimpl.h: move objimpl.h code
surrounded by "#ifndef Py_LIMITED_API" to cpython/objimpl.h.
* objimpl.h now includes cpython/objimpl.h
* Windows installer (MSI) now also install Include/ subdirectories:
Include/cpython/ and Include/internal/.
* And pycore_lifecycle.h and pycore_pathconfig.h headers to
Include/internal/
* Move Py_BUILD_CORE specific code from coreconfig.h and
pylifecycle.h to pycore_pathconfig.h and pycore_lifecycle.h
* Move _Py_wstrlist_XXX() definitions and _PyPathConfig code
from pycore_state.h to pycore_pathconfig.h
* Move "Init" and "Fini" function definitions from pylifecycle.c to
pycore_lifecycle.h.
The accu.h header is no longer part of the Python C API: it has been
moved to the "internal" headers which are restricted to Python
itself.
Replace #include "accu.h" with #include "pycore_accu.h".
Visual Studio solution: Set InlineFunctionExpansion to
OnlyExplicitInline ("/Ob1" option) on all projects (in
pyproject.props) in Debug mode on Win32 and x64 platforms to expand
functions marked as inline.
This change should make Python compiled in Debug mode a little bit
faster on Windows. On Unix, GCC uses -Og optimization level for
./configure --with-pydebug.
* Modify object.h to ensure that pymem.h is included,
to get _Py_tracemalloc_config variable.
* Move _PyTraceMalloc_XXX() functions to tracemalloc.h,
they need PyObject type. Break circular dependency between pymem.h
and object.h.
* Add Include/coreconfig.h
* Move config_*() and _PyCoreConfig_*() functions from Modules/main.c
to a new Python/coreconfig.c file.
* Inline _Py_ReadHashSeed() into config_init_hash_seed()
* Move global configuration variables to coreconfig.c
This is not the ideal solution; this means that a test module is now
always included in the main python3x.dll. However, we're already
including xxsubtype, so why not?
* Document `from __future__ import annotations`
* Provide plumbing and tests for `from __future__ import annotations`
* Implement unparsing the AST back to string form
This is required for PEP 563 and as such only implements a part of the
unparsing process that covers expressions.
* Factorize code from PC/getpathp.c and Modules/getpath.c to remove
duplicated code
* rename pathconfig_clear() to _PyPathConfig_Clear()
* Inline _PyPathConfig_Fini() in pymain_impl() and then remove it,
since it's a oneliner
bpo-32096, bpo-30860: Partially revert the commit
2ebc5ce42a8a9e047e790aefbf9a94811569b2b6:
* Move structures back from Include/internal/mem.h to
Objects/obmalloc.c
* Remove _PyObject_Initialize() and _PyMem_Initialize()
* Remove Include/internal/pymalloc.h
* Add test_capi.test_pre_initialization_api():
Make sure that it's possible to call Py_DecodeLocale(), and then call
Py_SetProgramName() with the decoded string, before Py_Initialize().
PyMem_RawMalloc() and Py_DecodeLocale() can be called again before
_PyRuntimeState_Init().
Co-Authored-By: Eric Snow <ericsnowcurrently@gmail.com>
* Don't use "Python runtime" anymore to parse command line options or
to get environment variables: pymain_init() is now a strict
separation.
* Use an error message rather than "crashing" directly with
Py_FatalError(). Limit the number of calls to Py_FatalError(). It
prepares the code to handle errors more nicely later.
* Warnings options (-W, PYTHONWARNINGS) and "XOptions" (-X) are now
only added to the sys module once Python core is properly
initialized.
* _PyMain is now the well identified owner of some important strings
like: warnings options, XOptions, and the "program name". The
program name string is now properly freed at exit.
pymain_free() is now responsible to free the "command" string.
* Rename most methods in Modules/main.c to use a "pymain_" prefix to
avoid conflits and ease debug.
* Replace _Py_CommandLineDetails_INIT with memset(0)
* Reorder a lot of code to fix the initialization ordering. For
example, initializing standard streams now comes before parsing
PYTHONWARNINGS.
* Py_Main() now handles errors when adding warnings options and
XOptions.
* Add _PyMem_GetDefaultRawAllocator() private function.
* Cleanup _PyMem_Initialize(): remove useless global constants: move
them into _PyMem_Initialize().
* Call _PyRuntime_Initialize() as soon as possible:
_PyRuntime_Initialize() now returns an error message on failure.
* Add _PyInitError structure and following macros:
* _Py_INIT_OK()
* _Py_INIT_ERR(msg)
* _Py_INIT_USER_ERR(msg): "user" error, don't abort() in that case
* _Py_INIT_FAILED(err)
* group the (stateful) runtime globals into various topical structs
* consolidate the topical structs under a single top-level _PyRuntimeState struct
* add a check-c-globals.py script that helps identify runtime globals
Other globals are excluded (see globals.txt and check-c-globals.py).
* group the (stateful) runtime globals into various topical structs
* consolidate the topical structs under a single top-level _PyRuntimeState struct
* add a check-c-globals.py script that helps identify runtime globals
Other globals are excluded (see globals.txt and check-c-globals.py).
Fix logic for retrying nuget.exe download with Python.
Add support for HOST_PYTHON variable.
Clear internal environment variables used in find_python.bat
Use HOST_PYTHON as the actual Python if it is recent enough.
Adds HOST_PYTHON variable to AppVeyor configuration
* bpo-30726: Fix elementtree warnings on Windows
Caused by usage of `getenv` which should be safe. And a few integer
truncations which should also be ok.
* bpo-30726: Don't ignore libexpat warnings which haypo intends to fix upstream
bpo-30726, bpo-29591: libexpat 2.2.1 of Modules/expat/ now uses
a winconfig.h configuration file which already defines:
* XML_NS
* XML_DTD
* BYTEORDER=1234
* XML_CONTEXT_BYTES=1024
* HAVE_MEMMOVE
Remove these defines from PCbuild/_elementtree.vcxproj to prevent
compiler warnings.
Co-Authored-By: Jeremy Kloth <jeremy.kloth@gmail.com>
The Windows build now depends on Python 3.6 to fetch externals, but it will be downloaded via NuGet (which is downloaded via PowerShell) if it is not available via `py -3.6`. This means the only thing that must be installed on a modern Windows box to do a full build of CPython with all extensions is Visual Studio.
Also fixes an outdated note about _lzma in PCbuild/readme.txt
* Improves test_underpth_nosite_file to reveal why it fails.
* Enable building with Windows 10 SDK.
* Fix WinSDK detection
* Fix initialization on Windows when a ._pth file exists.
* Fix tabs
* Adds comment about Py_GetPath call.
PEP 432 specifies a number of large changes to interpreter startup code, including exposing a cleaner C-API. The major changes depend on a number of smaller changes. This patch includes all those smaller changes.
* Adds lib.pyproj file to solution so that VS with Python support can open all the files in the standard library.
* Remove unexpected solution configuration.
* Remove lib.pyproj from solution to avoid memory issues on VS 2015.
Using a response file will eliminate the headache associated with batch argument/quote processing.
For example I unsucessfully compiled python with visualcpptools when specifying VSInstallDir in the batch file (cannot find vcruntime.h)
```batch
build.bat -p x64 -e -M --no-tkinter "/p:VCInstallDir=%VCInstallDir%"
```
but it build successfully when specifying it in a response file
msbuild.rsp:
```
/p:VCInstallDir=%VCInstallDir%
```
Use --short form of git hash. Use output from "git describe" for tag.
Expected outputs:
1. previous hg
2. previous git
3. updated git
Release (tagged) build:
1. Python 3.7.0a0 (v3.7.0a0:4def2a2901a5, ...
2. Python 3.7.0a0 (v3.7.0a0^0:05f53735c8912f8df1077e897f052571e13c3496, ...
3. Python 3.7.0a0 (v3.7.0a0:05f53735c8, ...
Development build:
1. Python 3.7.0a0 (default:41df79263a11, ...
2. Python 3.7.0a0 (master:05f53735c8912f8df1077e897f052571e13c3496, ...
3. Python 3.7.0a0 (heads/master-dirty:05f53735c8, ...
"dirty" means the working tree has uncommitted changes.
See "git help describe" for more info.
* Move all functions to call objects in a new Objects/call.c file.
* Rename fast_function() to _PyFunction_FastCallKeywords().
* Copy null_error() from Objects/abstract.c
* Inline type_error() in call.c to not have to copy it, it was only
called once.
* Export _PyEval_EvalCodeWithName() since it is now called
from call.c.
* Move all functions to call objects in a new Objects/call.c file.
* Rename fast_function() to _PyFunction_FastCallKeywords().
* Copy null_error() from Objects/abstract.c
* Inline type_error() in call.c to not have to copy it, it was only
called once.
* Export _PyEval_EvalCodeWithName() since it is now called
from call.c.
It can now handle OpenSSL versions 1.0.2e and greater, which don't
include include files in include/.
Note that sources prepared by this script no longer support the old
project files for 2.7; you now have to have Perl available to use
the old build_ssl.py script with sources from svn.python.org.
This modifies the behavior of the '-e' flag to PCbuild\build.bat: when '-e'
is not supplied, no attempt will be made to build extension modules that
require external libraries, even if the external libraries are present.
Also adds '--no-<module>' flags to PCbuild\build.bat, where '<module>' is
one of 'ssl', 'tkinter', or 'bsddb', to allow skipping just those modules
(if '-e' is given).
This removes the need to add externals/tcltk[64]/bin to PATH to be able
to import tkinter. Also documents the necessity for the DLLs to be
on PATH or in python.exe's directory.
Rather than using Burn "Persisted" variables we now add registry keys for each added feature. These can be detected by the installer regardless of which version installed them, and we use this for Modify and Upgrade. In particular, Upgrades can't access the Persisted variables, but can find well-known registry keys.
There are also some changes to the bootstrap app to properly handle upgrades.
Finally, a few minor improvements to the Windows build to keep things tidier.
Rather than using Burn "Persisted" variables we now add registry keys for each added feature. These can be detected by the installer regardless of which version installed them, and we use this for Modify and Upgrade. In particular, Upgrades can't access the Persisted variables, but can find well-known registry keys.
There are also some changes to the bootstrap app to properly handle upgrades.
Finally, a few minor improvements to the Windows build to keep things tidier.
This will prevent failures from the version of NASM installed on buildbots,
if the ability to override the version used is desired again later, we can
figure out a way to opt in.
This change affects the makefiles checked into svn.python.org, which the 3.5
build no longer uses. 3.4 and 2.7 both still use those makefiles, but their
build_ssl.py scripts don't require an update; if the script is running the
'fix_makefiles' method it already has Perl available anyway.
Victor Stinner
Anthony Shaw
Steve Dower
Dino Viehland
Segev Finer
Tal Einat
Benjamin Peterson
Greg Price
Dong-hee Na
Min ho Kim
Stefan Krah
Victor Stinner
animalize
Paul Monson
Łukasz Langa
Inada Naoki
Zackery Spytz
Antoine Pitrou
Steve Dower
Pierre Glaser
Pablo Galindo
Jeroen Demeyer
Isuru Fernando
Jess
Eric Snow
antektek
Gregory Szorc
Serhiy Storchaka
Jeremy Kloth
Carl Meyer
Ivan Levkivskyi
Zachary Ware
Ned Deily
Guido van Rossum
Yury Selivanov
Antoine Pitrou
INADA Naoki
Victor Stinner
native-api
Mariatta
Stefan Grönke
Charles
Xiang Zhang
Christian Heimes
Brett Cannon
Larry Hastings
Martin Panter
Yury Selivanov
Gregory P. Smith