Commit Graph

49 Commits

Author SHA1 Message Date
Anthony Sottile 360ef843d8
gh-99108: Add missing md5/sha1 defines to Modules/Setup (#102308) 2023-02-28 22:34:06 +01:00
Jonathan Protzenko d43c2652d4
gh-99108: Followup fix for Modules/Setup (GH-102183)
Automerge-Triggered-By: GH:erlend-aasland
2023-02-23 09:23:57 -08:00
Jonathan Protzenko fcadc7e405
gh-99108: Import MD5 and SHA1 from HACL* (#102089)
Replaces our fallback non-OpenSSL MD5 and SHA1 implementations with those from HACL* as we've already done with SHA2.
2023-02-22 13:18:43 -08:00
Gregory P. Smith 0b13575e74
gh-99108: Refactor _sha256 & _sha512 into _sha2. (#101924)
This merges their code. They're backed by the same single HACL* static library, having them be a single module simplifies maintenance.

This should unbreak the wasm enscripten builds that currently fail due to linking in --whole-archive mode and the HACL* library appearing twice.

Long unnoticed error fixed: _sha512.SHA384Type was doubly assigned and was actually SHA512Type. Nobody depends on those internal names.

Also rename LIBHACL_ make vars to LIBHACL_SHA2_ in preperation for other future HACL things.
2023-02-15 22:08:20 -08:00
Eric Snow c67b00534a
gh-101524: Split Up the _xxsubinterpreters Module (gh-101526)
This is step 1 in potentially dropping all the "channel"-related code. Channels have already been removed from PEP 554.

https://github.com/python/cpython/issues/101524
2023-02-03 18:14:43 -07:00
Eric Snow d45cc80452
gh-98627: Add the _testsinglephase Module (gh-99039)
This makes it more clear that a given test is definitely testing against a single-phase init (legacy) extension module.  The new module is a companion to _testmultiphase.

https://github.com/python/cpython/issues/98627
2022-11-08 09:58:11 -07:00
Christian Heimes 81dca70d70
gh-93939: Build C extensions without setup.py (GH-94474)
Combines GH-93940, GH-94452, and GH-94433
2022-07-14 00:51:49 -07:00
Christian Heimes 5596909eac
bpo-45847: Port _scproxy to PY_STDLIB_MOD (GH-29644) 2021-11-20 10:18:48 +01:00
Christian Heimes 7e44dc0ba7
bpo-45573: Move mandatory core modules to Modules/Setup.bootstrap (GH-29616) 2021-11-19 16:40:57 +01:00
Erlend Egeberg Aasland 29e5874d5a
bpo-45774: Autoconfiscate SQLite detection (GH-29507)
Co-authored-by: Christian Heimes <christian@python.org>
2021-11-19 15:10:41 +01:00
Christian Heimes 25ecc040d0
bpo-45573: Introduce extension module flags in Makefile (GH-29594)
``configure`` now uses a standardized format to forward state, compiler
flags, and linker flags to ``Makefile``, ``setup.py``, and
``Modules/Setup``. ``makesetup`` use the new variables by default if a
module line does not contain any compiler or linker flags. ``setup.py``
has a new function ``addext()``.

For a module ``egg``, configure adds:

* ``MODULE_EGG`` with value yes, missing, disabled, or n/a
* ``MODULE_EGG_CFLAGS``
* ``MODULE_EGG_LDFLAGS``

``Makefile.pre.in`` may also provide ``MODULE_EGG_DEPS`` that lists
dependencies such as header files and static libs.

Signed-off-by: Christian Heimes <christian@python.org>
2021-11-18 09:18:44 +01:00
Christian Heimes 464e6616be
bpo-45800: Move pyexpat build setup into configure (GH-29547)
Settings for :mod:`pyexpat` C extension are now detected by ``configure``.
The bundled ``expat`` library is built in ``Makefile``.

Signed-off-by: Christian Heimes <christian@python.org>
2021-11-14 10:02:24 +01:00
Christian Heimes 0486570f7b
bpo-45798: Move _decimal build setup into configure (GH-29541)
Settings for :mod:`decimal` internal C extension are now detected by
:program:`configure`. The bundled `libmpdec` library is built in
``Makefile``.

Signed-off-by: Christian Heimes <christian@python.org>
2021-11-13 13:56:16 +01:00
Christian Heimes 0a9f69539b
bpo-45747: Detect gdbm/dbm dependencies in configure (GH-29467)
Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@innova.no>
2021-11-10 20:26:55 +01:00
Christian Heimes f0150ac94a
bpo-45548: Some test modules must be built as shared libs (GH-29268)
Some test cases don't work when test modules are static extensions.

Add dependency on Modules/config.c to trigger a rebuild whenever a
module build type is changed.

``makesetup`` puts shared extensions into ``Modules/`` directory. Create
symlinks from pybuilddir so the extensions can be imported.

Note: It is not possible to use the content of pybuilddir.txt as a build
target. Makefile evaluates target variables in the first pass. The
pybuilddir.txt file does not exist at that point.
2021-10-29 17:49:57 +02:00
Christian Heimes 4c95fb4640
bpo-45548: Fix out-of-tree and Debian builds (GH-29263)
Add Modules subdirs to SRCDIRS to generate directories for out-of-tree
object files.

Debian wants ncurses lib. Works on Fedora, too.

Debian also needs pkg-config to detect correct flags.

Remove more outdated comments. Makefile now tracks header dependencies
-lintl is injected by configure when needed. Build _dbm with
gdbm-compat.

Group some modules by purpose. socket, select, and mmap work on Windows,
too.
2021-10-28 11:57:38 +02:00
Christian Heimes ece916e92c
bpo-45548: Add missing extensions to Modules/Setup (GH-29199)
Adds _ctypes, _decimal, _multiprocessing, _posixshmem, _scproxy, _sqlite3,
and _uuid.

Use Makefile variables to build OpenSSL and TCL/TK modules.

Drop ``-lm`` from math lines. Extensions are always linked against libm.

Signed-off-by: Christian Heimes <christian@python.org>
2021-10-25 20:01:52 +02:00
Christian Heimes fa26245a1c
bpo-45548: Remove _math.c workarounds for pre-C99 libm (GH-29179)
The :mod:`math` and :mod:`cmath` implementation now require a C99 compatible
``libm`` and no longer ship with workarounds for missing acosh, asinh,
expm1, and log1p functions.

The changeset also removes ``_math.c`` and moves the last remaining
workaround into ``_math.h``. This simplifies static builds with
``Modules/Setup`` and resolves symbol conflicts.

Co-authored-by: Mark Dickinson <mdickinson@enthought.com>
Co-authored-by: Brett Cannon <brett@python.org>
Signed-off-by: Christian Heimes <christian@python.org>
2021-10-25 01:25:27 -07:00
Christian Heimes ec93721e00
bpo-45570: Simplify setup macros for pyexpat (GH-29159)
* ``HAVE_EXPAT_CONFIG_H`` is not used by our code and not used by
  system-wide expat header files
* ``USE_PYEXPAT_CAPI`` is no longer used by our code
* ``XML_POOR_ENTROPY`` should be defined in expat_config.h

Signed-off-by: Christian Heimes <christian@python.org>
2021-10-22 09:28:23 -07:00
Christian Heimes 03e9f5dc75
bpo-43974: Move Py_BUILD_CORE_MODULE into module code (GH-29157)
setup.py no longer defines Py_BUILD_CORE_MODULE. Instead every
module defines the macro before #include "Python.h" unless
Py_BUILD_CORE_BUILTIN is already defined.

Py_BUILD_CORE_BUILTIN is defined for every module that is built by
Modules/Setup.

The PR also simplifies Modules/Setup. Makefile and makesetup
already define Py_BUILD_CORE_BUILTIN and include Modules/internal
for us.

Signed-off-by: Christian Heimes <christian@python.org>
2021-10-22 15:36:28 +02:00
Brett Cannon 01cf4fb9c1
bpo-45548: Make `Modules/Setup` easier to read (GH-29143) 2021-10-21 17:07:26 -07:00
Brett Cannon dd86f63b55
bpo-45548: add some missing entries to `Modules/Setup` (GH-29115)
Also remove a duplicate entry for `_weakref`.
2021-10-20 17:34:49 -07:00
Victor Stinner bbe7497c5a
bpo-45434: Remove pystrhex.h header file (GH-28923)
Move Include/pystrhex.h to Include/internal/pycore_strhex.h.
The header file only contains private functions.

The following C extensions are now built with Py_BUILD_CORE_MODULE
macro defined to get access to the internal C API:

* _blake2
* _hashopenssl
* _md5
* _sha1
* _sha3
* _ssl
* binascii
2021-10-13 15:22:35 +02:00
Yurii Karabas 96c4cbd96c
bpo-44353: Implement typing.NewType __call__ method in C (#27262)
Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com>
Co-authored-by: Denis Laxalde <denis@laxalde.org>
2021-07-22 23:06:54 +02:00
Victor Stinner cdad2724e6
bpo-40137: Add pycore_moduleobject.h internal header (GH-25507)
Add pycore_moduleobject.h internal header file with static inline
functions to access module members:

* _PyModule_GetDict()
* _PyModule_GetDef()
* _PyModule_GetState()

These functions don't check at runtime if their argument has a valid
type and can be inlined even if Python is not built with LTO.

_PyType_GetModuleByDef() uses _PyModule_GetDef().

Replace PyModule_GetState() with _PyModule_GetState() in the
extension modules, considered as performance sensitive:

* _abc
* _functools
* _operator
* _pickle
* _queue
* _random
* _sre
* _struct
* _thread
* _winapi
* array
* posix

The following extensions are now built with the Py_BUILD_CORE_MODULE
macro defined, to be able to use the internal pycore_moduleobject.h
header: _abc, array, _operator, _queue, _sre, _struct.
2021-04-22 00:52:52 +02:00
Christian Heimes 39258d3595
bpo-43669: PEP 644: Require OpenSSL 1.1.1 or newer (GH-23014)
- Remove HAVE_X509_VERIFY_PARAM_SET1_HOST check
- Update hashopenssl to require OpenSSL 1.1.1
- multissltests only OpenSSL > 1.1.0
- ALPN is always supported
- SNI is always supported
- Remove deprecated NPN code. Python wrappers are no-op.
- ECDH is always supported
- Remove OPENSSL_VERSION_1_1 macro
- Remove locking callbacks
- Drop PY_OPENSSL_1_1_API macro
- Drop HAVE_SSL_CTX_CLEAR_OPTIONS macro
- SSL_CTRL_GET_MAX_PROTO_VERSION is always defined now
- security level is always available now
- get_num_tickets is available with TLS 1.3
- X509_V_ERR MISMATCH is always available now
- Always set SSL_MODE_RELEASE_BUFFERS
- X509_V_FLAG_TRUSTED_FIRST is always available
- get_ciphers is always supported
- SSL_CTX_set_keylog_callback is always available
- Update Modules/Setup with static link example
- Mention PEP in whatsnew
- Drop 1.0.2 and 1.1.0 from GHA tests
2021-04-17 11:36:35 +02:00
Victor Stinner 37834136d0
bpo-42161: Modules/ uses _PyLong_GetZero() and _PyLong_GetOne() (GH-22998)
Use _PyLong_GetZero() and _PyLong_GetOne() in Modules/ directory.

_cursesmodule.c and zoneinfo.c are now built with
Py_BUILD_CORE_MODULE macro defined.
2020-10-27 17:12:53 +01:00
Victor Stinner 47e1afd2a1
bpo-1635741: _PyUnicode_Name_CAPI moves to internal C API (GH-22713)
The private _PyUnicode_Name_CAPI structure of the PyCapsule API
unicodedata.ucnhash_CAPI moves to the internal C API. Moreover, the
structure gets a new state member which must be passed to the
getcode() and getname() functions.

* Move Include/ucnhash.h to Include/internal/pycore_ucnhash.h
* unicodedata module is now built with Py_BUILD_CORE_MODULE.
* unicodedata: move hashAPI variable into unicodedata_module_state.
2020-10-26 16:43:47 +01:00
Kyle Evans 7992579cd2
bpo-40422: Move _Py_closerange to fileutils.c (GH-22680)
This API is relatively lightweight and organizationally, given that it's
used by multiple modules, it makes sense to move it to fileutils.

Requires making sure that _posixsubprocess is compiled with the appropriate
Py_BUIILD_CORE_BUILTIN macro.
2020-10-13 22:04:44 +02:00
Victor Stinner c45dbe93b7
bpo-41078: Add pycore_list.h internal header file (GH-21057)
* Move _PyList_ITEMS() to pycore_list.h.
* The C extension "_heapq" is now built with Py_BUILD_CORE_MODULE
  macro defined to access the internal C API.
2020-06-22 17:39:32 +02:00
Pablo Galindo 1ed83adb0e
bpo-40939: Remove the old parser (GH-20768)
This commit removes the old parser, the deprecated parser module, the old parser compatibility flags and environment variables and all associated support code and documentation.
2020-06-11 17:30:46 +01:00
Paul Ganssle 62972d9d73
bpo-40503: PEP 615: Tests and implementation for zoneinfo (GH-19909)
This is the initial implementation of PEP 615, the zoneinfo module,
ported from the standalone reference implementation (see
https://www.python.org/dev/peps/pep-0615/#reference-implementation for a
link, which has a more detailed commit history).

This includes (hopefully) all functional elements described in the PEP,
but documentation is found in a separate PR. This includes:

1. A pure python implementation of the ZoneInfo class
2. A C accelerated implementation of the ZoneInfo class
3. Tests with 100% branch coverage for the Python code (though C code
   coverage is less than 100%).
4. A compile-time configuration option on Linux (though not on Windows)

Differences from the reference implementation:

- The module is arranged slightly differently: the accelerated module is
  `_zoneinfo` rather than `zoneinfo._czoneinfo`, which also necessitates
  some changes in the test support function. (Suggested by Victor
  Stinner and Steve Dower.)
- The tests are arranged slightly differently and do not include the
  property tests. The tests live at test/test_zoneinfo/test_zoneinfo.py
  rather than test/test_zoneinfo.py or test/test_zoneinfo/__init__.py
  because we may do some refactoring in the future that would likely
  require this separation anyway; we may:
        - include the property tests
        - automatically run all the tests against both pure Python and C,
          rather than manually constructing C and Python test classes (similar
          to the way this works with test_datetime.py, which generates C
          and Python test cases from datetimetester.py).
- This includes a compile-time configuration option on Linux (though not
  on Windows); added with much help from Thomas Wouters.
- Integration into the CPython build system is obviously different from
  building a standalone zoneinfo module wheel.
- This includes configuration to install the tzdata package as part of
  CI, though only on the coverage jobs. Introducing a PyPI dependency as
  part of the CI build was controversial, and this is seen as less of a
  major change, since the coverage jobs already depend on pip and PyPI.

Additional changes that were introduced as part of this PR, most / all of
which were backported to the reference implementation:

- Fixed reference and memory leaks

    With much debugging help from Pablo Galindo

- Added smoke tests ensuring that the C and Python modules are built

    The import machinery can be somewhat fragile, and the "seamlessly falls
    back to pure Python" nature of this module makes it so that a problem
    building the C extension or a failure to import the pure Python version
    might easily go unnoticed.

- Adjustments to zoneinfo.__dir__

    Suggested by Petr Viktorin.

- Slight refactorings as suggested by Steve Dower.

- Removed unnecessary if check on std_abbr

    Discovered this because of a missing line in branch coverage.
2020-05-16 10:20:06 +02:00
Victor Stinner b617993b7c
bpo-40602: Rename hashtable.h to pycore_hashtable.h (GH-20044)
* Move Modules/hashtable.h to Include/internal/pycore_hashtable.h
* Move Modules/hashtable.c to Python/hashtable.c
* Python is now linked to hashtable.c. _tracemalloc is no longer
  linked to hashtable.c. Previously, marshal.c got hashtable.c via
  _tracemalloc.c which is built as a builtin module.
2020-05-12 02:42:19 +02:00
Pablo Galindo c5fc156852
bpo-40334: PEP 617 implementation: New PEG parser for CPython (GH-19503)
Co-authored-by: Guido van Rossum <guido@python.org>
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
2020-04-22 23:29:27 +01:00
Victor Stinner 9f5fe7910f
bpo-40286: Add randbytes() method to random.Random (GH-19527)
Add random.randbytes() function and random.Random.randbytes()
method to generate random bytes.

Modify secrets.token_bytes() to use SystemRandom.randbytes()
rather than calling directly os.urandom().

Rename also genrand_int32() to genrand_uint32(), since it returns an
unsigned 32-bit integer, not a signed integer.

The _random module is now built with Py_BUILD_CORE_MODULE defined.
2020-04-17 19:05:35 +02:00
Victor Stinner 1ae035b7e8
bpo-40302: Add pycore_byteswap.h header file (GH-19552)
Add a new internal pycore_byteswap.h header file with the following
functions:

* _Py_bswap16()
* _Py_bswap32()
* _Py_bswap64()

Use these functions in _ctypes, sha256 and sha512 modules,
and also use in the UTF-32 encoder.

sha256, sha512 and _ctypes modules are now built with the internal
C API.
2020-04-17 17:47:20 +02:00
Victor Stinner e9e7d284c4
bpo-35081: Move dtoa.h header to the internal C API (GH-18489)
Move the dtoa.h header file to the internal C API as pycore_dtoa.h:
it only contains private functions (prefixed by "_Py").

The math and cmath modules must now be compiled with the
Py_BUILD_CORE macro defined.
2020-02-12 22:54:42 +01:00
Dong-hee Na 0a18ee4be7 bpo-37798: Add C fastpath for statistics.NormalDist.inv_cdf() (GH-15266) 2019-08-23 15:20:30 -07:00
aaronpaulhurst 408a2ef1ac closes bpo-35184: Fix XML_POOR_ENTROPY option that breaks makesetup parsing of pyexpat line in Setup. (GH-13064)
When the line is uncommented, the equals character causes it to be incorrectly interpreted
as a macro definition by makesetup.  This results in invalid Makefile output.

The expat code only requires XML_POOR_ENTROPY to be defined; the value is unnecessary.
2019-06-10 18:54:24 -07:00
Victor Stinner 23bace26ec
bpo-36635: Add _testinternalcapi module (GH-12841)
Add a new _testinternalcapi module to test the internal C API.

Move _Py_GetConfigsAsDict() function to the internal C API:
_testembed now uses _testinternalcapi to access the function.
2019-04-18 11:37:26 +02:00
Victor Stinner 5c75f37d47
bpo-36635: Change pyport.h for Py_BUILD_CORE_MODULE define (GH-12853)
Change PyAPI_FUNC(type), PyAPI_DATA(type) and PyMODINIT_FUNC macros
of pyport.h when Py_BUILD_CORE_MODULE is defined.

The Py_BUILD_CORE_MODULE define must be now be used to build a C
extension as a dynamic library accessing Python internals: export the
PyInit_xxx() function in DLL exports on Windows.

Changes:

* Py_BUILD_CORE_BUILTIN and Py_BUILD_CORE_MODULE now imply
  Py_BUILD_CORE directy in pyport.h.
* ceval.c compilation now fails with an error if Py_BUILD_CORE is not
  defined, just to ensure that Python is build with the correct
  defines.
* setup.py now compiles _pickle.c with Py_BUILD_CORE_MODULE define.
* setup.py compiles _json.c with Py_BUILD_CORE_MODULE define, rather
  than Py_BUILD_CORE_BUILTIN define
* PCbuild/pythoncore.vcxproj: Add Py_BUILD_CORE_BUILTIN define.
2019-04-17 23:02:26 +02:00
Victor Stinner 02e6bf7f20
bpo-28604: Fix localeconv() for different LC_MONETARY (GH-10606)
locale.localeconv() now sets temporarily the LC_CTYPE locale to the
LC_MONETARY locale if the two locales are different and monetary
strings are non-ASCII. This temporary change affects other threads.

Changes:

* locale.localeconv() can now set LC_CTYPE to LC_MONETARY to decode
  monetary fields.
* Add LocaleInfo.grouping_buffer: copy localeconv() grouping string
  since it can be replaced anytime if a different thread calls
  localeconv().
* _Py_GetLocaleconvNumeric() now requires a "struct lconv *"
  structure, so locale.localeconv() now longer calls localeconv()
  twice. Moreover, the function now requires all arguments to be
  non-NULL.
* Rename STATIC_LOCALE_INFO_INIT to LocaleInfo_STATIC_INIT.
* Move _Py_GetLocaleconvNumeric() definition from fileutils.h
  to pycore_fileutils.h. pycore_fileutils.h now includes locale.h.
* The _locale module is now built with Py_BUILD_CORE defined.
2018-11-20 16:20:16 +01:00
Victor Stinner 130893debf
bpo-35081: Internal headers require Py_BUILD_CORE (GH-10363)
* All internal header files now require Py_BUILD_CORE or
  Py_BUILD_CORE_BUILTIN to be defined.
* _json.c is now compiled with Py_BUILD_CORE_BUILTIN to access
  pycore_accu.h header.
* Add an example to Modules/Setup to show how to build _json
  as a built-in module; it requires non trivial compiler options.
2018-11-09 13:03:37 +01:00
Max Bélanger 318ab63c01 closes bpo-35139: The `pyexpat` module's macros in `Modules/Setup` now match `setup.py` (GH-10289)
This could cause compile errors on macOS or other platforms.
2018-11-01 19:49:46 -07:00
Victor Stinner 27e2d1f219
bpo-35081: Add pycore_ prefix to internal header files (GH-10263)
* Rename Include/internal/ header files:

  * pyatomic.h -> pycore_atomic.h
  * ceval.h -> pycore_ceval.h
  * condvar.h -> pycore_condvar.h
  * context.h -> pycore_context.h
  * pygetopt.h -> pycore_getopt.h
  * gil.h -> pycore_gil.h
  * hamt.h -> pycore_hamt.h
  * hash.h -> pycore_hash.h
  * mem.h -> pycore_mem.h
  * pystate.h -> pycore_state.h
  * warnings.h -> pycore_warnings.h

* PCbuild project, Makefile.pre.in, Modules/Setup: add the
  Include/internal/ directory to the search paths of header files.
* Update includes. For example, replace #include "internal/mem.h"
  with #include "pycore_mem.h".
2018-11-01 00:52:28 +01:00
Victor Stinner c89a932714
bpo-35053: Define _PyTraceMalloc_NewReference in object.h (GH-10107)
_PyTraceMalloc_NewReference() is now called by _Py_NewReference(), so
move its definition to object.h. Moreover, define it even if
Py_LIMITED_API is defined, since _Py_NewReference() is also exposed
even if Py_LIMITED_API is defined.
2018-10-26 00:01:56 +02:00
Serhiy Storchaka 79d1c2e6c9
bpo-25711: Rewrite zipimport in pure Python. (GH-6809) 2018-09-18 22:22:29 +03:00
Benjamin Peterson b9bf9d025e
Delete old expat comment. (GH-9197) 2018-09-11 17:17:39 -07:00
Antoine Pitrou 961d54c5c1
bpo-32430: Rename Modules/Setup.dist to Modules/Setup (GH-8229)
bpo-32430: Rename Modules/Setup.dist to Modules/Setup

Remove the necessity to copy the former manually to the latter when updating the local source tree.
2018-07-16 19:03:03 +02:00