Commit Graph

4022 Commits

Author SHA1 Message Date
Victor Stinner 6387b5313c
gh-108494: Document how to add a project in PCbuild/readme.txt (#110077)
Add _testclinic_limited to Tools/msi/test/test_files.wxs.
2023-10-02 18:53:38 +02:00
dependabot[bot] a431a0f988
build(deps): bump hypothesis from 6.84.0 to 6.87.1 in /Tools (#110174)
Bumps [hypothesis](https://github.com/HypothesisWorks/hypothesis) from 6.84.0 to 6.87.1.
- [Release notes](https://github.com/HypothesisWorks/hypothesis/releases)
- [Commits](https://github.com/HypothesisWorks/hypothesis/compare/hypothesis-python-6.84.0...hypothesis-python-6.87.1)

---
updated-dependencies:
- dependency-name: hypothesis
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-01 17:25:29 +01:00
dependabot[bot] 038c3564fb
PEG generator: bump types-setuptools from 68.1.0.1 to 68.2.0.0 (#110175)
build(deps-dev): bump types-setuptools in /Tools

Bumps [types-setuptools](https://github.com/python/typeshed) from 68.1.0.1 to 68.2.0.0.
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-setuptools
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-01 12:38:46 +01:00
Victor Stinner a46e960768
gh-109649: Use os.process_cpu_count() (#110165)
Replace os.cpu_count() with os.process_cpu_count() in modules:

* compileall
* concurrent.futures
* multiprocessing

Replace os.cpu_count() with os.process_cpu_count() in programs:

* _decimal deccheck.py test
* freeze.py
* multissltests.py
* python -m test (regrtest)
* wasm_build.py

Other changes:

* test.pythoninfo logs os.process_cpu_count().
* regrtest gets os.process_cpu_count() / os.cpu_count() in headers.
2023-10-01 03:14:57 +02:00
Victor Stinner 53eb9a676f
gh-110152: regrtest handles cross compilation and HOSTRUNNER (#110156)
* _add_python_opts() now handles cross compilation and HOSTRUNNER.
* display_header() now tells if Python is cross-compiled, display
  HOSTRUNNER, and get the host platform.
* Remove Tools/scripts/run_tests.py script.
* Remove "make hostrunnertest": use "make buildbottest"
  or "make test" instead.
2023-09-30 22:37:23 +00:00
Victor Stinner 74e425ec18
gh-110014: Fix _POSIX_THREADS and _POSIX_SEMAPHORES usage (#110139)
* pycore_pythread.h is now the central place to make sure that
  _POSIX_THREADS and _POSIX_SEMAPHORES macros are defined if
  available.
* Make sure that pycore_pythread.h is included when _POSIX_THREADS
  and _POSIX_SEMAPHORES macros are tested.
* PY_TIMEOUT_MAX is now defined as a constant, since its value
  depends on _POSIX_THREADS, instead of being defined as a macro.
* Prevent integer overflow in the preprocessor when computing
  PY_TIMEOUT_MAX_VALUE on Windows:
  replace "0xFFFFFFFELL * 1000 < LLONG_MAX"
  with "0xFFFFFFFELL < LLONG_MAX / 1000".
* Document the change and give hints how to fix affected code.
* Add an exception for PY_TIMEOUT_MAX  name to smelly.py
* Add PY_TIMEOUT_MAX to the stable ABI
2023-09-30 19:25:54 +02:00
Victor Stinner f3bb00ea12
gh-107954: Refactor initconfig.c: add CONFIG_SPEC (#110146)
Add a specification of the PyConfig structure to factorize the code.
2023-09-30 17:24:06 +00:00
Ned Deily c88037d137
gh-109991: Update GitHub CI workflows to use OpenSSL 3.0.11 and multissltests to use 1.1.1w, 3.0.11, and 3.1.3. (gh-110002) 2023-09-28 00:45:13 -04:00
Guido van Rossum 5bb6f0fcba
gh-104909: Split some more insts into ops (#109943)
These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR`
that weren't already viable uops:

* Split LOAD_ATTR_METHOD_WITH_VALUES
* Split LOAD_ATTR_METHOD_NO_DICT
* Split LOAD_ATTR_SLOT
* Split STORE_ATTR_SLOT
* Split STORE_ATTR_INSTANCE_VALUE

Also:

* Add `-v` flag to code generator which prints a list of non-viable uops
  (easter-egg: it can print execution counts -- see source)
* Double _Py_UOP_MAX_TRACE_LENGTH to 128



I had dropped one of the DEOPT_IF() calls! :-(
2023-09-27 15:27:44 -07:00
Victor Stinner b89ed9df39
gh-109615: Fix support test_copy_python_src_ignore() (#109958)
Fix the test when run on an installed Python: use "abs_srcdir" of
sysconfig, and skip the test if the Python source code cannot be
found.

* Tools/patchcheck/patchcheck.py, Tools/freeze/test/freeze.py and
  Lib/test/libregrtest/utils.py now first try to get "abs_srcdir"
  from sysconfig, before getting "srcdir" from sysconfig.
* test.pythoninfo logs sysconfig "abs_srcdir".
2023-09-27 12:32:12 +02:00
Victor Stinner 1512d6c6ee
gh-109615: Fix test_tools.test_freeze SRCDIR (#109935)
Fix copy_source_tree() function of test_tools.test_freeze:

* Don't copy SRC_DIR/build/ anymore. This directory is modified by
  other tests running in parallel.
* Add test.support.copy_python_src_ignore().
* Use sysconfig to get the source directory.
* Use sysconfig.get_config_var() to get CONFIG_ARGS variable.
2023-09-27 08:18:39 +00:00
Ammar Askar a829356f86
gh-109098: Fuzz re module instead of internal sre (#109911)
* gh-109098: Fuzz re module instead of internal sre
* Fix c-analyzer globals test failure
* Put globals exception in ignored.tsv
2023-09-26 15:35:49 -07:00
OmniTroid 4390c13148
Fix argument ordering of embuilder command documented in `Tools/wasm/README.md` (GH-109863) 2023-09-26 13:22:00 -07:00
Victor Stinner fbfec5642e
gh-109566: regrtest reexecutes the process (#109909)
When --fast-ci or --slow-ci option is used, regrtest now replaces the
current process with a new process to add "-u -W default -bb -E"
options to Python.

Changes:

* PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add
  "-u -W default -bb -E" options to Python: it's now done by
  regrtest.
* Fix Tools/scripts/run_tests.py: flush stdout before replacing the
  process. Previously, buffered messages were lost.
2023-09-26 20:46:52 +02:00
Victor Stinner 859618c8cd
gh-109566, regrtest: Add --fast-ci and --slow-ci options (#109570)
* Add --fast-ci and --slow-ci options to libregrtest:

  * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
    (skip slowest tests).
  * --slow-ci uses a default timeout of 20 minues and "-u all" (run
    all tests).

* regrtest header now lists test resources.
* Makefile changes:

  * "make test", "make hostrunnertest" and "make coverage-report" now
    use --fast-ci option and TESTTIMEOUT variable.
  * "make buildbottest" now uses "--slow-ci". Remove options which
    became redundant with "--slow-ci".
  * "make testall" and "make testuniversal" now use --slow-ci option
    and TESTTIMEOUT variable.
  * "make testall" now uses "find -exec rm ..." instead of
    "find ... -print|xargs rm ...", same as "make clean".

* GitHub Actions workflow:

  * Ubuntu and Address Sanitizer jobs now use "make test". Remove
    options which became redundant with "--fast-ci".
  * Windows jobs now use --fast-ci option.
  * Use -j0 to detect the number of CPUs.

* Set Makefile TESTTIMEOUT default to an empty string, since
  --slow-ci and --fast-ci use different default timeout. It's now
  accepted to pass "--timeout=" to regrtest: treated as not timeout.
* Tools/scripts/run_tests.py now uses --fast-ci option.
* Tools/buildbot/test.bat now uses --slow-ci option. Remove
  --timeout=1200 option, redundant with --slow-ci.
2023-09-26 17:22:50 +02:00
Hugo van Kemenade f19416534a
Code: Update Donghee Na's name (#109744) 2023-09-25 18:17:34 +03:00
Pablo Galindo Salgado b28ffaa193
gh-109596: Ensure repeated rules in the grammar are not allowed and fix incorrect soft keywords (#109606) 2023-09-22 19:03:23 +01:00
Victor Stinner 09a25616a9
gh-109723: Disable Py_BUILD_CORE in _testcapi (#109727)
Make sure that the internal C API is not tested by mistake by
_testcapi.

Undefine Py_BUILD_CORE_BUILTIN and Py_BUILD_CORE_MODULE macros in
Modules/_testcapi/parts.h: move code from _testcapimodule.c.

heaptype_relative.c and vectorcall_limited.c are using the limited C
API which is incompatible with the internal C API.

Move test_long_numbits() from _testcapi to _testinternalcapi since it
uses the internal C API "pycore_long.h".

Fix Modules/_testcapi/pyatomic.c: don't include Python.h directly,
just include _testcapi/parts.h.

Ajust "make check-c-globals" for these changes.
2023-09-22 14:54:37 +00:00
Victor Stinner 3f5c5649cf
gh-104469: Disallow using Py_LIMITED_API with Py_BUILD_CORE (#109690)
Fix make check-c-globals: complete USE_LIMITED_C_API list of the
c-analyzer.
2023-09-22 01:21:07 +02:00
Heinz-Alexander Fuetterer ef6d475db3
Fix typos in docs and comments (#109619) 2023-09-20 16:58:23 +00:00
James Gerity def828995a
fixes gh-109559: Update `unicodedata` for Unicode 15.1.0 (GH-109560)
---------

Co-authored-by: Benjamin Peterson <benjamin@python.org>
2023-09-19 22:07:47 -07:00
Sam Gross 0c89056fe5
gh-108724: Add PyMutex and _PyParkingLot APIs (gh-109344)
PyMutex is a one byte lock with fast, inlineable lock and unlock functions for the common uncontended case.  The design is based on WebKit's WTF::Lock.

PyMutex is built using the _PyParkingLot APIs, which provides a cross-platform futex-like API (based on WebKit's WTF::ParkingLot).  This internal API will be used for building other synchronization primitives used to implement PEP 703, such as one-time initialization and events.

This also includes tests and a mini benchmark in Tools/lockbench/lockbench.py to compare with the existing PyThread_type_lock.

Uncontended acquisition + release:
* Linux (x86-64): PyMutex: 11 ns, PyThread_type_lock: 44 ns
* macOS (arm64): PyMutex: 13 ns, PyThread_type_lock: 18 ns
* Windows (x86-64): PyMutex: 13 ns, PyThread_type_lock: 38 ns

PR Overview:

The primary purpose of this PR is to implement PyMutex, but there are a number of support pieces (described below).

* PyMutex:  A 1-byte lock that doesn't require memory allocation to initialize and is generally faster than the existing PyThread_type_lock.  The API is internal only for now.
* _PyParking_Lot:  A futex-like API based on the API of the same name in WebKit.  Used to implement PyMutex.
* _PyRawMutex:  A word sized lock used to implement _PyParking_Lot.
* PyEvent:  A one time event.  This was used a bunch in the "nogil" fork and is useful for testing the PyMutex implementation, so I've included it as part of the PR.
* pycore_llist.h:  Defines common operations on doubly-linked list.  Not strictly necessary (could do the list operations manually), but they come up frequently in the "nogil" fork. ( Similar to https://man.freebsd.org/cgi/man.cgi?queue)

---------

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
2023-09-19 09:54:29 -06:00
Victor Stinner 67d9363372
gh-109566: Run GHA and buildbot tests with --fail-rerun (#109567) 2023-09-19 15:50:27 +02:00
Nikita Sobolev f65497fd25
gh-109125: Run mypy on `Tools/wasm` (#109126) 2023-09-18 23:49:26 -06:00
DongWoo Son df8b3a46a7
Fix a typo in c-analyzer (#109213)
Co-authored-by: sunmy2019 <59365878+sunmy2019@users.noreply.github.com>
Co-authored-by: Dale Collison <92315623+dcollison@users.noreply.github.com>
2023-09-18 00:27:54 -07:00
Michael Droettboom 19f5effc27
GH-109373: Store metadata required for pystats comparison in the JSON (GH-109374) 2023-09-15 13:10:46 -07:00
Guido van Rossum a7a079798d
gh-109287: Desugar inst(X) to op(X); macro(X) = X (#109294)
This makes the internal representation in the code generator simpler: there's a list of ops, and a list of macros, and there's no special-casing needed for ops that aren't macros. (There's now special-casing for ops that are also macros, but that's simpler.)
2023-09-15 08:39:05 -07:00
Serhiy Storchaka 1f885df2a5
gh-107782: Use _testcapi to test non-representable signatures (GH-109325)
Builtin functions and methods that have non-representable signatures today
will have representable signatures yesterday, and they will become unusable
for testing this feature.

So we need to add special functions and methods to the _testcapi module
that always have non-representable signatures.
2023-09-14 09:12:17 +03:00
Sam Gross baaac99487
Fix invocation of wasm_build.py for node (GH-109383) 2023-09-13 15:28:08 -07:00
Michael Droettboom 5dcbbd8861
GH-109330: Dump and compare stats using opcode names, not numbers (GH-109335) 2023-09-12 14:12:57 -07:00
Guido van Rossum b86ce91bfe
gh-106581: Honor 'always_exits' in write_components() (#109338)
I must have overlooked this when refactoring the code generator.
The Tier 1 interpreter contained a few silly things like
```
            goto resume_frame;
            STACK_SHRINK(1);
```
(and other variations, some where the unconditional `goto` was hidden in a macro).
2023-09-12 17:58:40 +00:00
Irit Katriel 8b55adfa8f
gh-109256: allocate opcode IDs for internal opcodes in their own range (#109269) 2023-09-12 10:36:17 +00:00
Guido van Rossum fbaf77eb9b
gh-109214: Rename SAVE_IP to _SET_IP, and similar (#109285)
* Rename SAVE_IP to _SET_IP
* Rename EXIT_TRACE to _EXIT_TRACE
* Rename SAVE_CURRENT_IP to _SAVE_CURRENT_IP
* Rename INSERT to _INSERT (This is for Ken Jin's abstract interpreter)
* Rename IS_NONE to _IS_NONE
* Rename JUMP_TO_TOP to _JUMP_TO_TOP
2023-09-11 15:39:19 -07:00
Guido van Rossum bcce5e2718
gh-109039: Branch prediction for Tier 2 interpreter (#109038)
This adds a 16-bit inline cache entry to the conditional branch instructions POP_JUMP_IF_{FALSE,TRUE,NONE,NOT_NONE} and their instrumented variants, which is used to keep track of the branch direction.

Each time we encounter these instructions we shift the cache entry left by one and set the bottom bit to whether we jumped.

Then when it's time to translate such a branch to Tier 2 uops, we use the bit count from the cache entry to decided whether to continue translating the "didn't jump" branch or the "jumped" branch.

The counter is initialized to a pattern of alternating ones and zeros to avoid bias.

The .pyc file magic number is updated. There's a new test, some fixes for existing tests, and a few miscellaneous cleanups.
2023-09-11 18:20:24 +00:00
Alex Waygood 697c9dcf8f
gh-108455: peg_generator: enable mypy's `--warn-unreachable` setting and `redundant-expr` error code (#109160) 2023-09-08 22:05:40 +01:00
Michael Droettboom 52beebc856
gh-109136: Fix summarize_stats.py tool (#109137) 2023-09-08 18:23:58 +02:00
Mark Shannon 15d4c9fabc
GH-108716: Turn off deep-freezing of code objects. (GH-108722) 2023-09-08 10:34:40 +01:00
Mark Shannon 0858328ca2
GH-108614: Add `RESUME_CHECK` instruction (GH-108630) 2023-09-07 14:39:03 +01:00
Victor Stinner db1ee6a19a
gh-108740: Fix "make regen-all" race condition (#108741)
Fix a race condition in "make regen-all". The deepfreeze.c source and
files generated by Argument Clinic are now generated or updated
before generating "global objects". Previously, some identifiers may
miss depending on the order in which these files were generated.

* "make regen-global-objects": Make sure that deepfreeze.c is
  generated and up to date, and always run "make clinic".
* "make clinic" no longer runs generate_global_objects.py script.
* "make regen-deepfreeze" now only updates deepfreeze.c (C file).
  It doesn't build deepfreeze.o (object) anymore.
* Remove misleading messages in "make regen-global-objects" and
  "make clinic". They are now outdated, these commands are now
  safe to use.
* Document generates files in Doc/using/configure.rst.

Co-authored-by: Erlend E. Aasland <erlend@python.org>
2023-09-06 20:09:21 +02:00
Łukasz Langa f8a047941f
gh-109002: Ensure only one wheel for each vendored package (#109003)
Output with one wheel:
```
❯ GITHUB_ACTIONS=true ./Tools/build/verify_ensurepip_wheels.py
Verifying checksum for /Volumes/RAMDisk/cpython/Lib/ensurepip/_bundled/pip-23.2.1-py3-none-any.whl.
Expected digest: 7ccf472345f20d35bdc9d1841ff5f313260c2c33fe417f48c30ac46cccabf5be
Actual digest:   7ccf472345f20d35bdc9d1841ff5f313260c2c33fe417f48c30ac46cccabf5be
::notice file=/Volumes/RAMDisk/cpython/Lib/ensurepip/_bundled/pip-23.2.1-py3-none-any.whl::Successfully verified the checksum of the pip wheel.
```

Output with two wheels:
```
❯ GITHUB_ACTIONS=true ./Tools/build/verify_ensurepip_wheels.py
::error file=/Volumes/RAMDisk/cpython/Lib/ensurepip/_bundled/pip-22.0.4-py3-none-any.whl::Found more than one wheel for package pip.

::error file=/Volumes/RAMDisk/cpython/Lib/ensurepip/_bundled/pip-23.2.1-py3-none-any.whl::Found more than one wheel for package pip.
```

Output without wheels:
```
❯ GITHUB_ACTIONS=true ./Tools/build/verify_ensurepip_wheels.py
::error file=::Could not find a pip wheel on disk.
```
2023-09-06 16:49:44 +02:00
KH 44892b1c52
Fix a typo in umarshal.py (#108803)
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
2023-09-06 09:06:41 +00:00
Guido van Rossum b87263be9b
gh-106581: Support multiple uops pushing new values (#108895)
Also avoid the need for the awkward .clone() call in the argument
to mgr.adjust_inverse() and mgr.adjust().
2023-09-05 13:58:39 -07:00
Nikita Sobolev 6ead5bd6ae
Disable `differing_test_runners` health check (#108886) 2023-09-04 18:31:58 +00:00
Serhiy Storchaka 1796c191b4
gh-108494: Argument Clinic: inline parsing code for positional-only parameters in the limited C API (GH-108622) 2023-09-03 17:28:14 +03:00
Victor Stinner ad73674283
gh-107603: Argument Clinic: Only include pycore_gc.h if needed (#108726)
Argument Clinic now only includes pycore_gc.h if PyGC_Head is needed,
and only includes pycore_runtime.h if _Py_ID() is needed.

* Add 'condition' optional argument to Clinic.add_include().
* deprecate_keyword_use() includes pycore_runtime.h when using
  the _PyID() function.
* Fix rendering of includes: comments start at the column 35.
* Mark PC/clinic/_wmimodule.cpp.h and
  "Objects/stringlib/clinic/*.h.h" header files as generated in
  .gitattributes.

Effects:

* 42 header files generated by AC no longer include the internal C
  API, instead of 4 header files before. For example,
  Modules/clinic/_abc.c.h no longer includes the internal C API.
* Fix _testclinic_depr.c.h: it now always includes pycore_runtime.h
  to get _Py_ID().
2023-08-31 23:42:34 +02:00
Mark Shannon 059bd4d299
GH-108614: Remove non-debug uses of `#if TIER_ONE` and `#if TIER_TWO` from `_POP_FRAME` op. (GH-108685) 2023-08-31 11:34:52 +01:00
Alex Waygood b89b838ebc
gh-108455: peg_generator: use `types-setuptools==68.1.0.1` in CI (#108697) 2023-08-31 08:41:25 +01:00
Alex Waygood 77e8f233ac
gh-108455: peg_generator: install two stubs packages before running mypy (#108637) 2023-08-29 20:14:08 +01:00
Guido van Rossum 4f22152713
gh-107557: Remove unnecessary SAVE_IP instructions (#108583)
Also remove NOP instructions.

The "stubs" are not optimized in this fashion (their SAVE_IP should always be preserved since it's where to jump next, and they don't contain NOPs by their nature).
2023-08-29 16:51:51 +00:00
Nikita Sobolev f8be2e262c
gh-108455: peg generator: Use `strict_optional=True` for `grammar_parser` (#108629) 2023-08-29 13:02:52 +01:00