Pgen is the oldest piece of technology in the CPython repository, building it requires various #if[n]def PGEN hacks in other parts of the code and it also depends more and more on CPython internals. This commit removes the old pgen C code and replaces it for a new version implemented in pure Python. This is a modified and adapted version of lib2to3/pgen2 that can generate grammar files compatibles with the current parser.
This commit also eliminates all the #ifdef and code branches related to pgen, simplifying the code and making it more maintainable. The regen-grammar step now uses $(PYTHON_FOR_REGEN) that can be any version of the interpreter, so the new pgen code maintains compatibility with older versions of the interpreter (this also allows regenerating the grammar with the current CI solution that uses Python3.5). The new pgen Python module also makes use of the Grammar/Tokens file that holds the token specification, so is always kept in sync and avoids having to maintain duplicate token definitions.
Add TEST_EXTENSIONS constant to setup.py to allow to not build test
extensions like _testcapi.
Changes:
* Add add_ldflags_cppflags() subfunction
* Rename add_compiler_directories() to configure_compiler().
* Remove unused COMPILED_WITH_PYDEBUG constant.
* Use self.add() rather than accessing directly self.extensions.
* Remove module_enabled() function: check differently if curses
extension is built or not.
* Move set_compiler_flags() calls and concurrent.future hack from
module top-level to main()
* Remove unused variables 'macros' and 'libraries' from
detect_multiprocessing().
* Move SUMMARY and CLASSIFIERS constants at the top, move
set_compiler_flags() function below these constants.
* Add some empty new lines to respect PEP 8.
* Split PyBuildExt.detect_modules() huge function into subfunctions.
* Move curses, hashlib and some other code to reorganize the code.
* detect_tkinter() now returns False if the extension is missing.
* Add PyBuildExt.config_h_vars attribute
* Add _PyCoreConfig_ReadFromArgv() function which parses command line
options: move code from main.c to coreconfig.c.
* Add _PyCoreConfig_Write() to write the new configuration: coerce
the LC_CTYPE locale, set Py_xxx global configuration variables,
etc.
* _PyCoreConfig_ReadFromArgv() now only changes the LC_CTYPE locale
temporarily. _PyCoreConfig_Write() becomes responsible to set the
LC_CTYPE locale.
* Add _Py_SetArgcArgv() and _Py_ClearArgcArgv() functions
* Rename many "pymain_xxx()" functions
* Add "const" to some function parameters
* Reorganize main.c to declare functions in the order in which they
are called.
* Add PyBuildExt.srcdir atribute in setup.py: the source directory is
now always absolute.
* Add PyBuildExt.inc_dirs and PyBuildExt.lib_dirs attributes:
replace 'inc_dirs' and 'lib_dirs' local variables of
detect_modules().
* Replace "from distutils.errors import *"
with "from distutils.errors import CCompilerError, DistutilsError"
to be able to use static analyzers like pyflakes
* Reorder imports.
* Add PyBuildExt.add() which adds the extension directly to
self.extensions, rather than using a temporary 'exts' local
variable in detect_modules() and then add 'exts' to self.extensions
* Convert 'missing' local variable from detect_modules()
into PyBuildExt.missing attribute
* _detect_openssl(), _decimal_ext() and _detect_nis() now call
directly self.add(), rather than returning an extension
(or None if not found).
* Rename _decimal_ext() to _detect_decimal() for consistency with
other methods.
* Move fields from _PyMain to _PyCoreConfig:
* skip_first_line
* run_command
* run_module
* run_filename
* Replace _PyMain.stdin_is_interactive with a new
stdin_is_interactive(config) function
* Rename _PyMain to _PyArgv. Add "const _PyArgv *args" field
to _PyCmdline.
Add a new _Py_INIT_EXIT() macro to be able to exit Python with an
exitcode using _PyInitError API. Rewrite function calls by
pymain_main() to use _PyInitError.
Changes:
* Remove _PyMain.err and _PyMain.status field
* Add _Py_INIT_EXIT() macro and _PyInitError.exitcode field.
* Rename _Py_FatalInitError() to _Py_ExitInitError().
The whole coreconfig.h header is now excluded from Py_LIMITED_API.
Move functions definitions into a new internal pycore_coreconfig.h
header.
* Move Include/coreconfig.h to Include/cpython/coreconfig.h
* coreconfig.h header is now excluded from Py_LIMITED_API
* Move functions to pycore_coreconfig.h
Use locale.getpreferredencoding() rather than locale.getlocale() to
get the locale encoding. With some locales, locale.getlocale()
returns the wrong encoding.
For example, on Fedora 29, locale.getlocale() returns ISO-8859-1
encoding for the "en_IN" locale, whereas
locale.getpreferredencoding() reports the correct encoding: UTF-8.
* Rename globals to upper case to better distinguish if a variable is
global or local:
* Rename cross_compiling to CROSS_COMPILING
* Rename host_platform to HOST_PLATFORM
* Rename disabled_module_list to DISABLED_MODULE_LIST
* Add MS_WINDOWS, CYGWIN and MACOS constants.
* Use booleans: replace "return 0" with "return False"
and replace "return 1" with "return True"
The standard math library (libm) may follow IEEE-754 recommendation to
include an implementation of sinPi(), i.e. sinPi(x):=sin(pi*x).
And this triggers a name clash, found by FreeBSD developer
Steve Kargl, who worken on putting sinpi into libm used on FreeBSD
(it has to be named "sinpi", not "sinPi", cf. e.g.
https://en.cppreference.com/w/c/experimental/fpext4).
Per POSIX, `nice(3)` requires `unistd.h` and `exit(3)` requires `stdlib.h`.
Fixing the test will prevent false positives with pedantic compilers like clang.
Pablo Galindo
Ned Deily
Stéphane Wirtel
Eric Snow
Victor Stinner
Xtreak
Cheryl Sabella
Raymond Hettinger
Zhiming Wang
Gregory P. Smith
Shiv Dhar
pxinwr
Joannah Nanjekye
Giampaolo Rodola
Benjamin Peterson
Dima Pasechnik
Serhiy Storchaka
ngie-eign
Zackery Spytz
Davin Potts
Anthony Sottile
Sergey Fedoseev
Łukasz Langa
Andre Delfino