Commit Graph

809 Commits

Author SHA1 Message Date
Paul Ganssle 2abededbc4
bpo-40683: Add zoneinfo to LIBSUBDIRS (#20229)
Without this, only the _zoneinfo module is getting installed, not the
zoneinfo module. I believe this was not noticed earlier because
test.test_zoneinfo was also not being installed.
2020-05-19 11:55:18 -04:00
Batuhan Taskaya 63b8e0cba3
bpo-40528: Improve AST generation script to do builds simultaneously (GH-19968)
- Switch from getopt to argparse.
- Removed the limitation of not being able to produce both C and H simultaneously.

This will make it run faster since it parses the asdl definition once and uses the generated tree to generate both the header and the C source.
2020-05-18 18:42:10 +01:00
Ned Deily 951ab58024
Revert "bpo-26317: Support OBJC and OBJCXX configure command line variables (GH-20176)" (GH-20182)
This reverts commit 0da5466650.

The commit is causing make failures on a FreeBSD buildbot.
Due to the imminent 3.9.0b1 cutoff, revert this commit for
now pending further investigation.
2020-05-18 11:31:21 -04:00
Ned Deily 0da5466650
bpo-26317: Support OBJC and OBJCXX configure command line variables (GH-20176)
Add support to the configure script for OBJC and OBJCXX command line options so that the macOS builds can use the clang compiler for the macOS-specific Objective C source files. This allows third-party compilers, like GNU gcc, to be used to build the rest of the project since some of the Objective C system header files are not compilable by GNU gcc.

Co-authored-by: Jeffrey Kintscher <websurfer@surf2c.net>
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
2020-05-18 09:17:22 -04:00
Paul Ganssle 62972d9d73
bpo-40503: PEP 615: Tests and implementation for zoneinfo (GH-19909)
This is the initial implementation of PEP 615, the zoneinfo module,
ported from the standalone reference implementation (see
https://www.python.org/dev/peps/pep-0615/#reference-implementation for a
link, which has a more detailed commit history).

This includes (hopefully) all functional elements described in the PEP,
but documentation is found in a separate PR. This includes:

1. A pure python implementation of the ZoneInfo class
2. A C accelerated implementation of the ZoneInfo class
3. Tests with 100% branch coverage for the Python code (though C code
   coverage is less than 100%).
4. A compile-time configuration option on Linux (though not on Windows)

Differences from the reference implementation:

- The module is arranged slightly differently: the accelerated module is
  `_zoneinfo` rather than `zoneinfo._czoneinfo`, which also necessitates
  some changes in the test support function. (Suggested by Victor
  Stinner and Steve Dower.)
- The tests are arranged slightly differently and do not include the
  property tests. The tests live at test/test_zoneinfo/test_zoneinfo.py
  rather than test/test_zoneinfo.py or test/test_zoneinfo/__init__.py
  because we may do some refactoring in the future that would likely
  require this separation anyway; we may:
        - include the property tests
        - automatically run all the tests against both pure Python and C,
          rather than manually constructing C and Python test classes (similar
          to the way this works with test_datetime.py, which generates C
          and Python test cases from datetimetester.py).
- This includes a compile-time configuration option on Linux (though not
  on Windows); added with much help from Thomas Wouters.
- Integration into the CPython build system is obviously different from
  building a standalone zoneinfo module wheel.
- This includes configuration to install the tzdata package as part of
  CI, though only on the coverage jobs. Introducing a PyPI dependency as
  part of the CI build was controversial, and this is seen as less of a
  major change, since the coverage jobs already depend on pip and PyPI.

Additional changes that were introduced as part of this PR, most / all of
which were backported to the reference implementation:

- Fixed reference and memory leaks

    With much debugging help from Pablo Galindo

- Added smoke tests ensuring that the C and Python modules are built

    The import machinery can be somewhat fragile, and the "seamlessly falls
    back to pure Python" nature of this module makes it so that a problem
    building the C extension or a failure to import the pure Python version
    might easily go unnoticed.

- Adjustments to zoneinfo.__dir__

    Suggested by Petr Viktorin.

- Slight refactorings as suggested by Steve Dower.

- Removed unnecessary if check on std_abbr

    Discovered this because of a missing line in branch coverage.
2020-05-16 10:20:06 +02:00
Victor Stinner b617993b7c
bpo-40602: Rename hashtable.h to pycore_hashtable.h (GH-20044)
* Move Modules/hashtable.h to Include/internal/pycore_hashtable.h
* Move Modules/hashtable.c to Python/hashtable.c
* Python is now linked to hashtable.c. _tracemalloc is no longer
  linked to hashtable.c. Previously, marshal.c got hashtable.c via
  _tracemalloc.c which is built as a builtin module.
2020-05-12 02:42:19 +02:00
Petr Viktorin e1becf46b4
bpo-38787: C API for module state access from extension methods (PEP 573) (GH-19936)
Module C state is now accessible from C-defined heap type methods (PEP 573).
Patch by Marcel Plch and Petr Viktorin.

Co-authored-by: Marcel Plch <mplch@redhat.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
2020-05-07 15:39:59 +02:00
Pablo Galindo b7226eb0b7
Call $(MKDIR_P) before regenerating the PEG meta-parser (GH-19861) 2020-05-02 16:15:27 +01:00
Pablo Galindo d2baff4301
bpo-40334: regenerate metaparser as part of regen-all (GH-19854) 2020-05-02 05:23:39 +01:00
Victor Stinner b8f704d219
bpo-40421: Add Include/cpython/code.h header file (GH-19756)
bpo-35134, bpo-40421: Add Include/cpython/code.h header file.

code.h now defines PyCodeObject type in the limited C API. It is now
included by Python.h.

Give a name to the PyCodeObject structure: it is now called
"struct PyCodeObject". So it becomes possible to define PyCodeObject
as "struct PyCodeObject" in the limited C API without defining the
structure.
2020-04-28 17:07:12 +02:00
Victor Stinner 7c59d7c986
bpo-40421: Add pyframe.h header file (GH-19755)
Add a new separated pyframe.h header file of the PyFrame public C
API: it is included by Python.h.

Add PyFrame_GetLineNumber() to the limited C API.

Replace "struct _frame" with "PyFrameObject" in header files.
PyFrameObject is now defined as struct _frame by pyframe.h which is
included early enough in Python.h.
2020-04-28 16:32:48 +02:00
Pablo Galindo 5b9f4988c9
bpo-40334: Refactor peg_generator to receive a Tokens file when building c code (GH-19745) 2020-04-28 13:11:55 +01:00
Pablo Galindo 4044c843a5
Add files in tests/test_peg_generator to the install target lists (GH-19723)
Update the "Makefile.pre.in" template and the "PCbuild/lib.pyproj" with the files in "Lib/test/test/test_peg_generator" so they get correctly installed along the rest of the standard library.
2020-04-27 11:18:04 +01:00
Guido van Rossum 3e89251ba8
Fix broken mkdir -p call in regen-pegen (#19695)
We should use `$(MKDIR_P) <directory>`, not `$(MKDIR_P) -p <directory>`.
2020-04-23 16:30:42 -07:00
Lysandros Nikolaou ebebb6429c
bpo-40334: Improve various PEG-Parser related stuff (GH-19669)
The changes in this commit are all related to @vstinner's original review comments of the initial PEP 617 implementation PR.
2020-04-23 16:36:06 +01:00
Pablo Galindo a25f3c4c8f
bpo-40334: Fix builds outside the source directory and regenerate autoconf files (GH-19667) 2020-04-23 01:38:11 +01:00
Pablo Galindo c5fc156852
bpo-40334: PEP 617 implementation: New PEG parser for CPython (GH-19503)
Co-authored-by: Guido van Rossum <guido@python.org>
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
2020-04-22 23:29:27 +01:00
Victor Stinner 1ae035b7e8
bpo-40302: Add pycore_byteswap.h header file (GH-19552)
Add a new internal pycore_byteswap.h header file with the following
functions:

* _Py_bswap16()
* _Py_bswap32()
* _Py_bswap64()

Use these functions in _ctypes, sha256 and sha512 modules,
and also use in the UTF-32 encoder.

sha256, sha512 and _ctypes modules are now built with the internal
C API.
2020-04-17 17:47:20 +02:00
Victor Stinner 0c13e1f96a
bpo-40241: Add pycore_interp.h header (GH-19499)
Move PyInterpreterState and related functions to a new internal
pycore_interp.h header file.
2020-04-13 12:47:17 +02:00
Victor Stinner 1c4cbdf94d
bpo-40268: Add pycore_runtime.h header file (GH-19493)
Move PyRuntimeState from pycore_pystate.h to pycore_runtime.h.

Remove _PyGILState_check_enabled macro: access directly
_PyRuntime.gilstate.check_enabled.
2020-04-13 11:45:21 +02:00
Victor Stinner a15e260b70
bpo-40170: Add _PyIndex_Check() internal function (GH-19426)
Add _PyIndex_Check() function to the internal C API: fast inlined
verson of PyIndex_Check().

Add Include/internal/pycore_abstract.h header file.

Replace PyIndex_Check() with _PyIndex_Check() in C files of Objects
and Python subdirectories.
2020-04-08 02:01:56 +02:00
Guido van Rossum 48b069a003
bpo-39481: Implementation for PEP 585 (#18239)
This implements things like `list[int]`,
which returns an object of type `types.GenericAlias`.
This object mostly acts as a proxy for `list`,
but has attributes `__origin__` and `__args__`
that allow recovering the parts (with values `list` and `(int,)`.

There is also an approximate notion of type variables;
e.g. `list[T]` has a `__parameters__` attribute equal to `(T,)`.
Type variables are objects of type `typing.TypeVar`.
2020-04-07 09:50:06 -07:00
Victor Stinner 08faf0016e
bpo-38644: Add _PySys_Audit() which takes tstate (GH-19180)
Add _PySys_Audit() function to the internal C API: similar to
PySys_Audit(), but requires a mandatory tstate parameter.

Cleanup sys_audit_tstate() code: remove code path for NULL tstate,
since the function exits at entry if tstate is NULL. Remove also code
path for NULL tstate->interp: should_audit() now ensures that it is
not NULL (even if tstate->interp cannot be NULL in practice).

PySys_AddAuditHook() now checks if tstate is not NULL to decide if
tstate can be used or not, and tstate is set to NULL if the runtime
is not initialized yet.

Use _PySys_Audit() in sysmodule.c.
2020-03-26 18:57:32 +01:00
Batuhan Taşkaya 1b6b0e78fe
bpo-40069: Clear out .lst files on make clean (GH-19169)
Files created on AIX by xlc (C compiler).
2020-03-26 14:10:04 +01:00
Victor Stinner 8510f43078
bpo-1294959: Add sys.platlibdir attribute (GH-18381)
Add --with-platlibdir option to the configure script: name of the
platform-specific library directory, stored in the new sys.platlitdir
attribute. It is used to build the path of platform-specific dynamic
libraries and the path of the standard library.

It is equal to "lib" on most platforms. On Fedora and SuSE, it is
equal to "lib64" on 64-bit systems.

Co-Authored-By: Jan Matějek <jmatejek@suse.com>
Co-Authored-By: Matěj Cepl <mcepl@cepl.eu>
Co-Authored-By: Charalampos Stratakis <cstratak@redhat.com>
2020-03-10 09:53:09 +01:00
Victor Stinner 8c3aee65ed
bpo-35134: Add Include/cpython/fileutils.h header file (GH-18493)
Move CPython C API from Include/fileutils.h into a new
Include/cpython/fileutils.h header file which is included by
Include/fileutils.h.

Exclude the following private symbols from the limited C API:

* _Py_error_handler
* _Py_GetErrorHandler()
* _Py_DecodeLocaleEx()
* _Py_EncodeLocaleEx()
2020-02-12 23:55:09 +01:00
Victor Stinner 98921aeaf5
bpo-35134: Add Include/cpython/bytesobject.h file (GH-18494)
Add Include/cpython/bytearrayobject.h and
Include/cpython/bytesobject.h header files.

Move CPython C API from Include/bytesobject.h into a new
Include/cpython/bytesobject.h header file which is included by
Include/bytesobject.h. Do a similar change for
Include/bytearrayobject.h.
2020-02-12 23:54:31 +01:00
Victor Stinner e9e7d284c4
bpo-35081: Move dtoa.h header to the internal C API (GH-18489)
Move the dtoa.h header file to the internal C API as pycore_dtoa.h:
it only contains private functions (prefixed by "_Py").

The math and cmath modules must now be compiled with the
Py_BUILD_CORE macro defined.
2020-02-12 22:54:42 +01:00
Victor Stinner 45876a90e2
bpo-35081: Move bytes_methods.h to the internal C API (GH-18492)
Move the bytes_methods.h header file to the internal C API as
pycore_bytes_methods.h: it only contains private symbols (prefixed by
"_Py"), except of the PyDoc_STRVAR_shared() macro.
2020-02-12 22:32:34 +01:00
Fangrui Song 9a978ddb93
closes bpo-39575: Change -lgcov to --coverage. (GH-18382)
This allows clang to get rid of the dependency on libgcov.
When linking, GCC passes -lgcov while clang passes the path to libclang_rt.profile-$arch.a
2020-02-07 15:46:29 -08:00
Victor Stinner bec4186c67
bpo-35134: Create Include/cpython/listobject.h (GH-18395)
Move listobject.h code surrounded by "#ifndef Py_LIMITED_API"
to a new Include/cpython/listobject.h header file.

Add cpython/ header files to Makefile.pre.in and pythoncore project
of PCbuild.

Add _PyList_CAST() macro.
2020-02-07 09:20:21 +01:00
Dino Viehland 2528a6c3d0
Add test.test_import.data.unwritable package to makefile (#18211) 2020-01-27 14:04:56 -08:00
Anthony Shaw ef7eaafc9d bpo-39144 Align ctags and etags behaviours in the makefile and include Python stdlib files (GH-17721) 2019-12-28 02:32:48 +00:00
Victor Stinner 4d231bcc77
bpo-38644: Add _PyObject_Call() (GH-17089)
* Add pycore_call.h internal header file.
* Add _PyObject_Call(): PyObject_Call() with tstate
* Add _PyObject_CallNoArgTstate(): _PyObject_CallNoArg() with tstate
* Add _PyObject_FastCallDictTstate(): _PyObject_FastCallDict()
  with tstate
* _PyObject_Call_Prepend() now takes tstate
* Replace _PyObject_FastCall() calls
  with _PyObject_VectorcallTstate() calls
2019-11-14 13:36:21 +01:00
Victor Stinner f4b1e3d7c6
bpo-38644: Add Py_EnterRecursiveCall() to the limited API (GH-17046)
Provide Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() as
regular functions for the limited API. Previously, there were defined
as macros, but these macros didn't work with the limited API which
cannot access PyThreadState.recursion_depth field.

Remove _Py_CheckRecursionLimit from the stable ABI.

Add Include/cpython/ceval.h header file.
2019-11-04 19:48:34 +01:00
Gregory P. Smith c37447481e
bpo-28494: install ziptestdata to fix install bot (GH-15902) 2019-09-11 05:31:37 -05:00
Neil Schemenauer c6bbcd2583 bpo-37725: have "make clean" remove PGO task data (#15033)
Change "clean" makefile target to also clean the program guided
optimization (PGO) data.  Previously you would have to use "make
clean" and "make profile-removal", or "make clobber".
2019-09-10 10:44:20 +01:00
T. Wouters b4612f5d54
bpo-37589: Add a few missing dependencies on .h files in the Makefile. (GH-15757)
The missing dependencies prevented incremental builds from working when you touched any
of these files. Based on GH-14758 by @vemakereporter.
2019-09-09 05:17:18 -07:00
Erlend Egeberg Aasland 387c3c5a04 bpo-37653: Fix libinstall target in Makefile (GH-14911) 2019-07-23 18:11:50 +02:00
Neil Schemenauer 4e16a4a311
bpo-36044: Reduce number of unit tests run for PGO build (GH-14702)
Reduce the number of unit tests run for the PGO generation task.  This
speeds up the task by a factor of about 15x.  Running the full unit test
suite is slow.  This change may result in a slightly less optimized build
since not as many code branches will be executed.  If you are willing to
wait for the much slower build, the old behavior can be restored using
'./configure [..] PROFILE_TASK="-m test --pgo-extended"'.  We make no
guarantees as to which PGO task set produces a faster build.  Users who
care should run their own relevant benchmarks as results can depend on
the environment, workload, and compiler tool chain.
2019-07-22 12:54:25 -07:00
Victor Stinner e8692818af
bpo-37468: make install no longer install wininst-*.exe files (GH-14511)
make install no longer installs "wininst-*.exe" files used by
distutils bdist_wininst: bdist_wininst only works on Windows.
2019-07-15 17:18:42 +02:00
Victor Stinner 47fbc4e45b
bpo-37359: Add --cleanup option to python3 -m test (GH-14332)
* regrtest: Add --cleanup option to remove "test_python_*" directories
  of previous failed test jobs.
* Add "make cleantest" to run "python3 -m test --cleanup".
2019-06-24 12:03:00 +02:00
Victor Stinner 01b63ecac6
bpo-35134: Add Include/cpython/import.h header file (GH-14213)
* Add Include/cpython/import.h and Include/internal/pycore_import.h
  header files.
* Move _PyImport_ReInitLock() to the internal C API. Don't export the
  symbol anymore.
2019-06-19 00:48:09 +02:00
Christian Heimes 8a8b59c979 bpo-37215: Fix dtrace issue introduce by bpo-36842 (GH-13940)
Signed-off-by: Christian Heimes <christian@python.org>


https://bugs.python.org/issue37215
2019-06-10 08:19:48 -07:00
Inada Naoki 91234a1636
bpo-26219: per opcode cache for LOAD_GLOBAL (GH-12884)
This patch implements per opcode cache mechanism, and use it in
only LOAD_GLOBAL opcode.

Based on Yury's opcache3.patch in bpo-26219.
2019-06-03 21:30:58 +09:00
Victor Stinner 331a6a56e9
bpo-36763: Implement the PEP 587 (GH-13592)
* Add a whole new documentation page:
  "Python Initialization Configuration"
* PyWideStringList_Append() return type is now PyStatus,
  instead of int
* PyInterpreterState_New() now calls PyConfig_Clear() if
  PyConfig_InitPythonConfig() fails.
* Rename files:

  * Python/coreconfig.c => Python/initconfig.c
  * Include/cpython/coreconfig.h => Include/cpython/initconfig.h
  * Include/internal/: pycore_coreconfig.h => pycore_initconfig.h

* Rename structures

  * _PyCoreConfig => PyConfig
  * _PyPreConfig => PyPreConfig
  * _PyInitError => PyStatus
  * _PyWstrList => PyWideStringList

* Rename PyConfig fields:

  * use_module_search_paths => module_search_paths_set
  * module_search_path_env => pythonpath_env

* Rename PyStatus field: _func => func
* PyInterpreterState: rename core_config field to config
* Rename macros and functions:

  * _PyCoreConfig_SetArgv() => PyConfig_SetBytesArgv()
  * _PyCoreConfig_SetWideArgv() => PyConfig_SetArgv()
  * _PyCoreConfig_DecodeLocale() => PyConfig_SetBytesString()
  * _PyInitError_Failed() => PyStatus_Exception()
  * _Py_INIT_ERROR_TYPE_xxx enums => _PyStatus_TYPE_xxx
  * _Py_UnixMain() => Py_BytesMain()
  * _Py_ExitInitError() => Py_ExitStatusException()
  * _Py_PreInitializeFromArgs() => Py_PreInitializeFromBytesArgs()
  * _Py_PreInitializeFromWideArgs() => Py_PreInitializeFromArgs()
  * _Py_PreInitialize() => Py_PreInitialize()
  * _Py_RunMain() => Py_RunMain()
  * _Py_InitializeFromConfig() => Py_InitializeFromConfig()
  * _Py_INIT_XXX() => _PyStatus_XXX()
  * _Py_INIT_FAILED() => _PyStatus_EXCEPTION()

* Rename 'err' PyStatus variables to 'status'
* Convert RUN_CODE() macro to config_run_code() static inline function
* Remove functions:

  * _Py_InitializeFromArgs()
  * _Py_InitializeFromWideArgs()
  * _PyInterpreterState_GetCoreConfig()
2019-05-27 16:39:22 +02:00
Antoine Pitrou 91f4380ced
bpo-36785: PEP 574 implementation (GH-7076) 2019-05-26 17:10:09 +02:00
Jason R. Coombs f7fba6cfb6
bpo-34632 fix buildbots and remove artifact (GH-13566)
* bpo-34632: Also include the test data directory.

* bpo-34632: remove the framework resources artifacts, accidentally added in 1bbf7b661f (ccbccce)
2019-05-25 10:00:21 -04:00
Chih-Hsuan Yen c3738cfe63 bpo-34632: fix installation of importlib.metadata (#13563) 2019-05-25 04:09:34 -04:00
Victor Stinner bc66faccb8
bpo-36721: Fix pkg-config symbolic links on "make install" (GH-13551) 2019-05-24 18:27:13 +02:00