Commit Graph

3900 Commits

Author SHA1 Message Date
Guido van Rossum 14b7f00fdf
GH-98831: Update generate_cases.py: register inst, opcode_metadata.h (#100735)
(These aren't used yet, but may be coming soon,
and it's easier to keep this tool the same between branches.)

Added a sanity check for all this to compile.c.

Co-authored-by: Irit Katriel <iritkatriel@yahoo.com>
2023-01-05 13:01:07 -08:00
Mark Shannon 15aecf8dd7
GH-100719: Remove the `co_nplaincellvars` field from code objects. (GH-100721) 2023-01-04 15:41:39 +00:00
Irit Katriel c31e356a10
gh-100720: refactor calculation of number of frame slots for a code object into the new function _PyFrame_NumSlotsForCodeObject (#100722) 2023-01-04 13:37:06 +00:00
Eli Schwartz 8586949833
gh-89419: gdb: fix bug causing `AttributeError` in py-locals when no frame is available (#100611) 2023-01-03 20:26:57 +05:30
Zachary Ware 2df82db485
gh-100540: Remove obsolete Modules/_ctypes/darwin/ dlfcn shim (GH-100541)
As far as I can tell, this hasn't been actually used since Mac OS X 10.2.
2022-12-29 16:13:28 -06:00
Zachary Ware 025b5c3780
gh-100540: Remove unused Modules/_ctypes/libffi_osx/ (GH-100543)
It was an ancient, modified copy of libffi that has not been in use
since GH-22855.
2022-12-28 13:38:52 -06:00
Nikita Sobolev 3da71ff0f2
Fix copy-paste error in `Tools/clinic.py` (#100560) 2022-12-28 09:57:57 +05:30
Kumar Aditya 7cf164ad5e
GH-100342: check for allocation failure in AC `*args` parsing (#100343) 2022-12-28 09:46:28 +05:30
Guido van Rossum 08e5594cf3
GH-98831: Modernize a ton of simpler instructions (#100545)
* load_const and load_fast aren't families for now
* Don't decref unmoved names
* Modernize GET_ANEXT
* Modernize GET_AWAITABLE
* Modernize ASYNC_GEN_WRAP
* Modernize YIELD_VALUE
* Modernize POP_EXCEPT (in more than one way)
* Modernize PREP_RERAISE_STAR
* Modernize LOAD_ASSERTION_ERROR
* Modernize LOAD_BUILD_CLASS
* Modernize STORE_NAME
* Modernize LOAD_NAME
* Modernize LOAD_CLASSDEREF
* Modernize LOAD_DEREF
* Modernize STORE_DEREF
* Modernize COPY_FREE_VARS (mark it as done)
* Modernize LIST_TO_TUPLE
* Modernize LIST_EXTEND
* Modernize SET_UPDATE
* Modernize SETUP_ANNOTATIONS
* Modernize DICT_UPDATE
* Modernize DICT_MERGE
* Modernize MAP_ADD
* Modernize IS_OP
* Modernize CONTAINS_OP
* Modernize CHECK_EXC_MATCH
* Modernize IMPORT_NAME
* Modernize IMPORT_STAR
* Modernize IMPORT_FROM
* Modernize JUMP_FORWARD (mark it as done)
* Modernize JUMP_BACKWARD (mark it as done)
2022-12-27 17:11:03 -08:00
Nikita Sobolev b0ea28913e
Remove `NoneType` redefinition from `clinic.py` (#100551) 2022-12-27 21:41:39 +05:30
Illia Volochii a23cb72ac8
gh-100454: Start running SSL tests with OpenSSL 3.1.0-beta1 (#100456) 2022-12-23 18:04:20 -08:00
penguin_wwy a02161286a
GH-99770: Make the correct call specialization fail kind show up in the stats (GH-99771) 2022-12-22 12:44:59 +00:00
Guido van Rossum 9cdd2fa63b
GH-98831: Add DECREF_INPUTS(), expanding to DECREF() each stack input (#100205)
The presence of this macro indicates that a particular instruction
may be considered for conversion to a register-based format
(see https://github.com/faster-cpython/ideas/issues/485).

An invariant (currently unchecked) is that `DEOPT_IF()` may only
occur *before* `DECREF_INPUTS()`, and `ERROR_IF()` may only occur
*after* it. One reason not to check this is that there are a few
places where we insert *two* `DECREF_INPUTS()` calls, in different
branches of the code. The invariant checking would have to be able
to do some flow control analysis to understand this.

Note that many instructions, especially specialized ones,
can't be converted to use this macro straightforwardly.
This is because the generator currently only generates plain
`Py_DECREF(variable)` statements, and cannot generate
things like `_Py_DECREF_SPECIALIZED()` let alone deal with
`_PyList_AppendTakeRef()`.
2022-12-16 20:45:55 -08:00
Eric Snow 0415cf895f
gh-81057: Move the Cached Parser Dummy Name to _PyRuntimeState (#100277) 2022-12-16 13:48:03 +00:00
Mark Shannon 5693f45b19
Assorted minor fixes for specialization stats. (GH-100219) 2022-12-14 15:50:02 +00:00
Mark Shannon 6997e77bdf
GH-100222: Redefine _Py_CODEUNIT as a union to clarify structure of code unit. (GH-100223) 2022-12-14 11:12:53 +00:00
Eric Snow 0e081a089e
gh-81057: Move _Py_RefTotal to the "Ignored Globals" List (gh-100203)
We can't move it to _PyRuntimeState because the symbol is exposed in the stable ABI. We'll have to sort that out before a per-interpreter GIL, but it shouldn't be too hard.

https://github.com/python/cpython/issues/81057
2022-12-12 16:51:27 -07:00
Eric Snow 5eb28bca9f
gh-81057: Move Signal-Related Globals to _PyRuntimeState (gh-100085)
https://github.com/python/cpython/issues/81057
2022-12-12 16:50:19 -07:00
Eric Snow 53d9cd95cd
gh-81057: Move faulthandler Globals to _PyRuntimeState (gh-100152)
https://github.com/python/cpython/issues/81057
2022-12-12 09:58:46 -07:00
Eric Snow 8790d4d31f
gh-81057: Move tracemalloc Globals to _PyRuntimeState (gh-100151)
https://github.com/python/cpython/issues/81057
2022-12-12 08:44:23 -07:00
Michael Droettboom 1583c6e326
GH-100143: Improve collecting pystats for parts of runs (GH-100144)
* pystats off by default

* Add -Xpystats flag

* Always dump pystats, even if turned off
2022-12-12 14:50:43 +00:00
Shantanu 621a1790c4
gh-100176: remove incorrect version compatibility check from argument clinic (#100190) 2022-12-12 17:22:12 +05:30
Mark Shannon fb713b2183
GH-98522: Add version number to code objects. (GH-98525)
* Add version number to code object for better versioning of functions.

* Improves specialization for closures and list comprehensions.
2022-12-09 12:18:45 +00:00
Eric Snow 1160001b34
gh-81057: Move Threading-Related Globals to _PyRuntimeState (#100084)
https://github.com/python/cpython/issues/81057
2022-12-08 17:50:58 -07:00
Eric Snow bc8cdf8c3d
gh-81057: Move Ceval Trampoline Globals to _PyRuntimeState (gh-100083)
https://github.com/python/cpython/issues/81057
2022-12-08 17:17:20 -07:00
Guido van Rossum 1cfa704f64
GH-98831: Generate things in the input order (#100123)
This makes it easier to see what changed in the generated code
when converting an instruction to super or macro.
2022-12-08 15:54:07 -08:00
Eric Snow 8a3f06c54b
gh-81057: Move time Globals to _PyRuntimeState (gh-100122)
https://github.com/python/cpython/issues/81057
2022-12-08 16:46:09 -07:00
Eric Snow cda9f0236f
gh-81057: Move OS-Related Globals to _PyRuntimeState (gh-100082)
https://github.com/python/cpython/issues/81057
2022-12-08 15:38:06 -07:00
Guido van Rossum c85be734d1
GH-98831: Typed stack effects, and more instructions converted (#99764)
Stack effects can now have a type, e.g. `inst(X, (left, right -- jump/uint64_t)) { ... }`.

Instructions converted to the non-legacy format:

* COMPARE_OP
* COMPARE_OP_FLOAT_JUMP
* COMPARE_OP_INT_JUMP
* COMPARE_OP_STR_JUMP
* STORE_ATTR
* DELETE_ATTR
* STORE_GLOBAL
* STORE_ATTR_INSTANCE_VALUE
* STORE_ATTR_WITH_HINT
* STORE_ATTR_SLOT, and complete the store_attr family
* Complete the store_subscr family: STORE_SUBSCR{,DICT,LIST_INT}
  (STORE_SUBSCR was alread half converted,
  but wasn't using cache effects yet.)
* DELETE_SUBSCR
* PRINT_EXPR
* INTERPRETER_EXIT (a bit weird, ends in return)
* RETURN_VALUE
* GET_AITER (had to restructure it some)
  The original had mysterious `SET_TOP(NULL)` before `goto error`.
  I assume those just account for `obj` having been decref'ed,
  so I got rid of them in favor of the cleanup implied by `ERROR_IF()`.
* LIST_APPEND (a bit unhappy with it)
* SET_ADD (also a bit unhappy with it)

Various other improvements/refactorings as well.
2022-12-08 13:31:27 -08:00
Eric Snow 91a8e002c2
gh-81057: Move More Globals to _PyRuntimeState (gh-100092)
https://github.com/python/cpython/issues/81057
2022-12-07 15:56:31 -07:00
Eric Snow d47ffeb9e3
gh-90110: Clean Up the C-analyzer Globals Lists (gh-100091)
https://github.com/python/cpython/issues/90110
2022-12-07 15:02:47 -07:00
Michael Droettboom 9dc787ea96
GH-100026: Include the number of raw input files in summarize_stats.py (GH-100027) 2022-12-06 11:14:47 +00:00
Eric Snow 530cc9dbb6
gh-99741: Implement Multi-Phase Init for the _xxsubinterpreters Module (gh-99742)
_xxsubinterpreters is an internal module used for testing.

https://github.com/python/cpython/issues/99741
2022-12-05 13:40:20 -07:00
Guido van Rossum acf9184e6b
GH-98831: Support cache effects in super- and macro instructions (#99601) 2022-12-02 19:57:30 -08:00
Michael Droettboom bf94c653f4
GH-99905: Fix output of misses in summarize_stats.py execution counts (GH-99906)
This was an indentation error introduced in 2844aa6a
2022-11-30 19:17:08 +00:00
colorfulappl 0da728387c
gh-64490: Fix bugs in argument clinic varargs processing (#32092) 2022-11-24 20:56:50 +01:00
colorfulappl 8dbe08eb7c
gh-99240: Fix double-free bug in Argument Clinic str_converter generated code (GH-99241)
Fix double-free bug mentioned at https://github.com/python/cpython/issues/99240,
by moving memory clean up out of "exit" label.

Automerge-Triggered-By: GH:erlend-aasland
2022-11-24 06:01:26 -08:00
colorfulappl 69f6cc77d0
gh-64490: Fix refcount error when arguments are packed to tuple in argument clinic (#99233) 2022-11-24 17:54:18 +05:30
Guido van Rossum 8f18ac04d3
GH-98831: Add `macro` and `op` and their implementation to DSL (#99495)
Newly supported interpreter definition syntax:
- `op(NAME, (input_stack_effects -- output_stack_effects)) { ... }`
- `macro(NAME) = OP1 + OP2;`

Also some other random improvements:
- Convert `WITH_EXCEPT_START` to use stack effects
- Fix lexer to balk at unrecognized characters, e.g. `@`
- Fix moved output names; support object pointers in cache
- Introduce `error()` method to print errors
- Introduce read_uint16(p) as equivalent to `*p`

Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
2022-11-22 16:04:57 -08:00
colorfulappl c450c8c9ed
gh-96002: Add functional test for Argument Clinic (#96178)
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2022-11-21 15:08:45 +01:00
Guido van Rossum 4f5e1cb00a
GH-98831: Refactor and fix cases generator (#99526)
Also complete cache effects for BINARY_SUBSCR family.
2022-11-17 17:06:07 -08:00
Eric Snow 9db1e17c80
gh-81057: Move the global Dict-Related Versions to _PyRuntimeState (gh-99497)
We also move the global func version.

https://github.com/python/cpython/issues/81057
2022-11-16 10:37:29 -07:00
Eric Snow 01fa907aa8
gh-81057: Move contextvars-related Globals to _PyRuntimeState (gh-99400)
This is part of the effort to consolidate global variables, to make them easier to manage (and make it easier to later move some of them to PyInterpreterState).

https://github.com/python/cpython/issues/81057
2022-11-16 09:54:28 -07:00
Eric Snow 5f55067e23
gh-81057: Move More Globals in Core Code to _PyRuntimeState (gh-99516)
https://github.com/python/cpython/issues/81057
2022-11-16 09:37:14 -07:00
Guido van Rossum e37744f289
GH-98831: Implement basic cache effects (#99313) 2022-11-15 19:59:19 -08:00
Eric Snow 3c57971a2d
gh-81057: Move Globals in Core Code to _PyRuntimeState (gh-99496)
This is the first of several changes to consolidate non-object globals in core code.

https://github.com/python/cpython/issues/81057
2022-11-15 09:45:11 -07:00
Eric Snow e874c2f198
gh-81057: Move the Remaining Import State Globals to _PyRuntimeState (gh-99488)
https://github.com/python/cpython/issues/81057
2022-11-14 15:56:16 -07:00
Eric Snow a088290f9d
gh-81057: Move Global Variables Holding Objects to _PyRuntimeState. (gh-99487)
This moves nearly all remaining object-holding globals in core code (other than static types).

https://github.com/python/cpython/issues/81057
2022-11-14 13:50:56 -07:00
Eric Snow 7f3a4b967c
gh-81057: Move PyImport_Inittab to _PyRuntimeState (gh-99402)
We actually don't move PyImport_Inittab.  Instead, we make a copy that we keep on _PyRuntimeState and use only that after Py_Initialize().  We also prevent folks from modifying PyImport_Inittab (the best we can) after that point.

https://github.com/python/cpython/issues/81057
2022-11-11 17:06:05 -07:00
Eric Snow 67807cfc87
gh-81057: Move the Allocators to _PyRuntimeState (gh-99217)
The global allocators were stored in 3 static global variables: _PyMem_Raw, _PyMem, and _PyObject.  State for the "small block" allocator was stored in another 13.  That makes a total of 16 global variables. We are moving all 16 to the _PyRuntimeState struct as part of the work for gh-81057.  (If PEP 684 is accepted then we will follow up by moving them all to PyInterpreterState.)

https://github.com/python/cpython/issues/81057
2022-11-11 16:30:46 -07:00
Eric Snow f531b6879b
gh-81057: Add PyInterpreterState.static_objects (gh-99397)
As we consolidate global variables, we find some objects that are almost suitable to add to _PyRuntimeState.global_objects, but have some small/sneaky bit of per-interpreter state (e.g. a weakref list). We're adding PyInterpreterState.static_objects so we can move such objects there. (We'll removed the _not_used field once we've added others.)

https://github.com/python/cpython/issues/81057
2022-11-11 14:24:18 -07:00
Eric Snow dd36b71fa6
gh-81057: Move the Extension Modules Cache to _PyRuntimeState (gh-99355)
We also move the closely related max_module_number and add comments documenting the group of struct members.

https://github.com/python/cpython/issues/81057
2022-11-11 14:16:28 -07:00
Eric Snow fe55ff3f68
gh-81057: Generate a Separate Initializer For Each Part of the Global Objects Initializer (gh-99389)
Up until now we had a single generated initializer macro for all the statically declared global objects in _PyRuntimeState, including several one-offs (e.g. the empty tuple). The one-offs don't need to be generated, but were because we had one big initializer. Having separate initializers for set of generated global objects allows us to generate only the ones we need to.  This allows us to add initializers for one-off global objects without having to generate them.

https://github.com/python/cpython/issues/81057
2022-11-11 13:23:41 -07:00
Ikko Ashimine 8fe1a88e6a
Fix typo in Tools/wasm/README.md (GH-99384)
Noticable -> Noticeable

Automerge-Triggered-By: GH:AlexWaygood
2022-11-11 08:38:09 -08:00
Gregory P. Smith e00d730efe
Update github CI to use OpenSSL 1.1.1s and 3.0.7. (#99364) 2022-11-11 01:33:55 -08:00
Eric Snow 73679b13ca
gh-90110: Update the C-analyzer Tool (gh-99307) 2022-11-10 09:03:57 -07:00
Victor Stinner 4ce2a202c7
gh-99300: Use Py_NewRef() in Parser/ directory (#99330)
Replace Py_INCREF() with Py_NewRef() in C files of the Parser/
directory and in the PEG generator.
2022-11-10 15:30:05 +01:00
Mark Shannon 1e197e63e2
GH-96421: Insert shim frame on entry to interpreter (GH-96319)
* Adds EXIT_INTERPRETER instruction to exit PyEval_EvalDefault()

* Simplifies RETURN_VALUE, YIELD_VALUE and RETURN_GENERATOR instructions as they no longer need to check for entry frames.
2022-11-10 12:34:57 +00:00
Brandt Bucher c7f5708714
GH-98686: Get rid of "adaptive" and "quick" instructions (GH-99182) 2022-11-09 10:50:09 -08:00
Victor Stinner 0124b5dd28
gh-90868: Add _PyStaticObject_CheckRefcnt() function (#99261)
Add _PyStaticObject_CheckRefcnt() function to make
_PyStaticObjects_CheckRefcnt() shorter. Use
_PyObject_ASSERT_FAILED_MSG() to log the object causing the fatal
error.
2022-11-09 08:40:40 +01:00
Eric Snow 52f91c642b
gh-90868: Adjust the Generated Objects (gh-99223)
We do the following:

* move the generated _PyUnicode_InitStaticStrings() to its own file
* move the generated _PyStaticObjects_CheckRefcnt() to its own file
* include pycore_global_objects.h in extension modules instead of pycore_runtime_init.h

These changes help us avoid including things that aren't needed.

https://github.com/python/cpython/issues/90868
2022-11-08 10:03:03 -07:00
Eric Snow d45cc80452
gh-98627: Add the _testsinglephase Module (gh-99039)
This makes it more clear that a given test is definitely testing against a single-phase init (legacy) extension module.  The new module is a companion to _testmultiphase.

https://github.com/python/cpython/issues/98627
2022-11-08 09:58:11 -07:00
Guido van Rossum f1a654648b
GH-98831: Simple input-output stack effects for bytecodes.c (#99120) 2022-11-08 08:22:56 -08:00
Nikita Sobolev e56e33d271
gh-72719: Remove asyncore and asynchat modules (#96580)
Remove modules asyncore and asynchat, which were deprecated by PEP 594.

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2022-11-08 16:48:58 +01:00
Guido van Rossum 7dcd28eb41
GH-98831: Implement super-instruction generation (#99084)
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
2022-11-06 09:40:47 -08:00
Guido van Rossum d04899abb0
GH-99104: Update headers for bytecodes.c and generate_cases.py (#99112)
Also tweak the labels near the end of bytecodes.c.
2022-11-04 17:40:43 -07:00
Guido van Rossum c885623e9f
GH-98831: Auto-generate PREDICTED() macro calls (#99102)
Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
2022-11-04 15:30:17 -07:00
Michael Droettboom 2844aa6a8e
Support comparing two sets of pystats (GH-98816)
This adds support for comparing pystats collected from two different builds.

- The `--json-output` can be used to load in a set of raw stats and output a
  JSON file.
- Two of these JSON files can be provided on the next run, and then comparative
  results between the two are output.
2022-11-04 10:15:54 +00:00
Victor Stinner 0faa0ba240
gh-92584: Remove the distutils package (#99061)
Remove the distutils package. It was deprecated in Python 3.10 by PEP
632 "Deprecate distutils module". For projects still using distutils
and cannot be updated to something else, the setuptools project can
be installed: it still provides distutils.

* Remove Lib/distutils/ directory
* Remove test_distutils
* Remove references to distutils
* Skip test_check_c_globals and test_peg_generator since they use
  distutils
2022-11-03 19:27:27 +01:00
Victor Stinner e99c507013
gh-92584: Remove references to removed _bootsubprocess (#99062)
The _bootsubprocess module was removed in gh-93939
by commit 81dca70d70.
2022-11-03 18:12:45 +01:00
Victor Stinner a60ddd31be
gh-98401: Invalid escape sequences emits SyntaxWarning (#99011)
A backslash-character pair that is not a valid escape sequence now
generates a SyntaxWarning, instead of DeprecationWarning.  For
example, re.compile("\d+\.\d+") now emits a SyntaxWarning ("\d" is an
invalid escape sequence), use raw strings for regular expression:
re.compile(r"\d+\.\d+"). In a future Python version, SyntaxError will
eventually be raised, instead of SyntaxWarning.

Octal escapes with value larger than 0o377 (ex: "\477"), deprecated
in Python 3.11, now produce a SyntaxWarning, instead of
DeprecationWarning. In a future Python version they will be
eventually a SyntaxError.

codecs.escape_decode() and codecs.unicode_escape_decode() are left
unchanged: they still emit DeprecationWarning.

* The parser only emits SyntaxWarning for Python 3.12 (feature
  version), and still emits DeprecationWarning on older Python
  versions.
* Fix SyntaxWarning by using raw strings in Tools/c-analyzer/ and
  wasm_build.py.
2022-11-03 17:53:25 +01:00
Guido van Rossum 916af11a97
GH-98831: Remove redundant extract_cases.py script (GH-99065)
Debt I owe from PR GH-98830.
2022-11-03 09:50:35 -07:00
serge-sans-paille 8c4de57de9
gh-98948: Remove obsolete readelf dependency (#98949)
This got introduced in commit 5884449539
to determine if readline is already linked against curses or tinfo in
the setup.py, which is no longer present.
2022-11-03 14:31:37 +01:00
Guido van Rossum 41bc101dd6
GH-98831: "Generate" the interpreter (#98830)
The switch cases (really TARGET(opcode) macros) have been moved from ceval.c to generated_cases.c.h. That file is generated from instruction definitions in bytecodes.c (which impersonates a C file so the C code it contains can be edited without custom support in e.g. VS Code).

The code generator lives in Tools/cases_generator (it has a README.md explaining how it works). The DSL used to describe the instructions is a work in progress, described in https://github.com/faster-cpython/ideas/blob/main/3.12/interpreter_definition.md.

This is surely a work-in-progress. An easy next step could be auto-generating super-instructions.

**IMPORTANT: Merge Conflicts**

If you get a merge conflict for instruction implementations in ceval.c, your best bet is to port your changes to bytecodes.c. That file looks almost the same as the original cases, except instead of `TARGET(NAME)` it uses `inst(NAME)`, and the trailing `DISPATCH()` call is omitted (the code generator adds it automatically).
2022-11-02 21:31:26 -07:00
Serhiy Storchaka f520d720f6
gh-99016: Make build scripts compatible with Python 3.8 (GH-99017) 2022-11-02 20:30:09 +02:00
Brandt Bucher 276d77724f
GH-98686: Quicken everything (GH-98687) 2022-11-02 10:42:57 -07:00
Neil Schemenauer de6981680b
gh-90716: add _pylong.py module (#96673)
Add Python implementations of certain longobject.c functions. These use
asymptotically faster algorithms that can be used for operations on
integers with many digits. In those cases, the performance overhead of
the Python implementation is not significant since the asymptotic
behavior is what dominates runtime. Functions provided by this module
should be considered private and not part of any public API.

Co-author: Tim Peters <tim.peters@gmail.com>
Co-author: Mark Dickinson <dickinsm@gmail.com>
Co-author: Bjorn Martinsson
2022-10-25 22:00:50 -07:00
Pablo Galindo Salgado 1f737edb67
gh-96143: Move the perf trampoline files to the Python directory (#98675) 2022-10-25 23:34:22 +01:00
Victor Stinner 1863302d61
gh-97669: Create Tools/build/ directory (#97963)
Create Tools/build/ directory. Move the following scripts from
Tools/scripts/ to Tools/build/:

* check_extension_modules.py
* deepfreeze.py
* freeze_modules.py
* generate_global_objects.py
* generate_levenshtein_examples.py
* generate_opcode_h.py
* generate_re_casefix.py
* generate_sre_constants.py
* generate_stdlib_module_names.py
* generate_token.py
* parse_html5_entities.py
* smelly.py
* stable_abi.py
* umarshal.py
* update_file.py
* verify_ensurepip_wheels.py

Update references to these scripts.
2022-10-17 12:01:00 +02:00
Jakub Kuczys 120b4ab2b6
gh-95731: Fix module docstring extraction in pygettext (#95732) 2022-10-15 07:57:53 -07:00
Victor Stinner d09d2c7c91
gh-97669: Remove Tools/scripts/startuptime.py (#98214)
The "pyperf command" tool be used instead. Example:

    $ python3 -m pyperf command -- python3 -c pass
    .....................
    command: Mean +- std dev: 17.8 ms +- 0.4 ms

pyperf also computes the standard deviation which gives an idea of
the benchmark looks reliable or not.
2022-10-12 17:53:46 +02:00
Victor Stinner 0895c2a066
gh-97669: Create Tools/patchcheck/ directory (#98186)
Move patchcheck.py, reindent.py and untabify.py scripts to a new
Tools/patchcheck/ directory.
2022-10-12 10:09:21 +02:00
Victor Stinner 454a6d61bc
gh-97669: Remove abitype.py and pep384_macrocheck.py (#98165)
Remove abitype.py and pep384_macrocheck.py scripts of Tools/scripts/.
2022-10-11 13:18:53 +02:00
Victor Stinner e0ae9ddffe
gh-97669: Remove outdated example scripts (#97675) (#98167)
Remove outdated example scripts of the Tools/scripts/ directory:

* gprof2html.py
* md5sum.py
* nm2def.py
* pathfix.py
* win_add2path.py

Remove test_gprof2html, test_md5sum and test_pathfix of test_tools.
2022-10-11 10:07:57 +02:00
Ken Jin b399115ef1
gh-95756: Lazily created cached co_* attrs (GH-97791) 2022-10-11 11:26:08 +08:00
Joannah Nanjekye 4ed00be98f
gh-68686: Retire eptag ptag scripts (#98064)
* Retire eptag ptag scripts

* 📜🤖 Added by blurb_it.

* fix news entry error

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-10-08 07:57:47 -07:00
Irit Katriel 34e6f3567e
gh-92886: [clinic.py] raise exception on invalid input instead of assertion (GH-98051)
Tests should pass with -O (assertions off).

Automerge-Triggered-By: GH:iritkatriel
2022-10-07 14:41:35 -07:00
Victor Stinner 002252c4ad
gh-97669: Move difflib examples to Doc/includes/ (#97964)
Remove diff.py and ndiff.py scripts of Tools/scripts/: move them to
Doc/includes/.

* diff.py and ndiff.py files are no longer executable. Remove also
  their shebang ("#!/usr/bin/env python3").
* Remove the -profile command from ndiff.py to simply the code.
* Remove ndiff.py copyright and history command. The Python
  documentation examples are distributed under the "Zero Clause BSD
  License".
2022-10-07 10:57:48 -07:00
Łukasz Langa bbc7cd649a
gh-97008: Add a Python implementation of AttributeError and NameError suggestions (#97022)
Relevant tests moved from test_exceptions to test_traceback to be able to
compare both implementations.

Co-authored-by: Carl Friedrich Bolz-Tereick <cfbolz@gmx.de>
2022-10-04 15:31:16 -07:00
Victor Stinner 116fa62c6e
gh-97670: Remove sys.getdxp() and analyze_dxp.py script (#97671)
Remove the sys.getdxp() function and the Tools/scripts/analyze_dxp.py
script. DXP stands for "dynamic execution pairs". They were related
to DYNAMIC_EXECUTION_PROFILE and DXPAIRS macros which have been
removed in Python 3.11. Python can now be built with "./configure
--enable-pystats" to gather statistics on Python opcodes.
2022-10-04 15:28:57 +02:00
Victor Stinner 6cbbc26a73
gh-97669: Remove outdated example scripts (#97675)
Remove outdated example scripts of the Tools/scripts/ directory. A
copy can be found in the old-demos project:
https://github.com/gvanrossum/old-demos

Removed scripts (39):

* byext.py
* byteyears.py
* cleanfuture.py
* copytime.py
* crlf.py
* db2pickle.py
* dutree.doc
* dutree.py
* find-uname.py
* find_recursionlimit.py
* finddiv.py
* findlinksto.py
* findnocoding.py
* fixcid.py
* fixdiv.py
* fixheader.py
* fixnotice.py
* fixps.py
* get-remote-certificate.py
* google.py
* highlight.py
* ifdef.py
* import_diagnostics.py
* lfcr.py
* linktree.py
* lll.py
* mailerdaemon.py
* make_ctype.py
* mkreal.py
* objgraph.py
* pdeps.py
* pickle2db.py
* pindent.py
* pysource.py
* reindent-rst.py
* rgrep.py
* suff.py
* texi2html.py
* which.py

Changes:

* Remove test_fixcid, test_lll, test_pdeps and test_pindent
  of test.test_tools.
* Remove get-remote-certificate.py changelog entry, since the script
  was removed.

Note: there is a copy of crlf.py in Lib/test/test_lib2to3/data/.
2022-10-04 10:49:00 +02:00
Victor Stinner 64fe343717
gh-97681: Remove Tools/demo/ directory (#97682)
Remove the Tools/demo/ directory which contained old demo scripts. A
copy can be found in the old-demos project:
https://github.com/gvanrossum/old-demos

Remove the following old demo scripts:

* beer.py
* eiffel.py
* hanoi.py
* life.py
* markov.py
* mcast.py
* queens.py
* redemo.py
* rpython.py
* rpythond.py
* sortvisu.py
* spreadsheet.py
* vector.py

Changes:

* Remove a reference to the redemo.py script in the regex howto
  documentation.
* Remove a reference to the removed Tools/demo/ directory in the
  curses documentation.
* Update PC/layout/ to remove the reference to Tools/demo/ directory.
2022-10-03 17:09:02 +02:00
Serhiy Storchaka 0ee9619a4c
gh-97728: Argument Clinic: Fix uninitialized variable in the Py_UNICODE converter (GH-97729)
It affects function os.system() on Windows and Windows-specific modules
winreg, _winapi, _overlapped, and _msi.
2022-10-03 10:42:54 +03:00
Eric Snow 273a819ed2
gh-90110: Update the c-analyzer Tool (gh-97695)
https://github.com/python/cpython/issues/90110
2022-09-30 19:32:46 -06:00
Steve Dower ff54dd96cb
gh-90989: Install Windows launcher per-user, and clarify some installer text (GH-97655) 2022-09-30 10:29:31 +01:00
Steve Dower 73942e475c
gh-97649: The Tools directory is no longer installed on Windows (GH-97653) 2022-09-30 10:25:00 +01:00
Victor Stinner 83a0f44ffd
gh-97612: Fix shell injection in get-remote-certificate.py (#97613)
Fix a shell code injection vulnerability in the
get-remote-certificate.py example script. The script no longer uses a
shell to run "openssl" commands. Issue reported and initial fix by
Caleb Shortt.

Remove the Windows code path to send "quit" on stdin to the "openssl
s_client" command: use DEVNULL on all platforms instead.

Co-authored-by: Caleb Shortt <caleb@rgauge.com>
2022-09-29 01:17:27 +02:00
LiarPrincess 0c1d7a06ed
bpo-47243: Duplicate entry in 'Objects/unicodetype_db.h' (GH-32376)
Fix for duplicate 1st entry in 'Objects/unicodetype_db.h':

```c
/* a list of unique character type descriptors */
const _PyUnicode_TypeRecord _PyUnicode_TypeRecords[] = {
    {0, 0, 0, 0, 0, 0},
    {0, 0, 0, 0, 0, 0}, <--- HERE
    {0, 0, 0, 0, 0, 32},
    {0, 0, 0, 0, 0, 48},
    …
```

https://bugs.python.org/issue47243

Automerge-Triggered-By: GH:isidentical
2022-09-28 06:57:14 -07:00
Christian Heimes 0b62964b04
gh-96883: browser: include concurrent.futures (GH-96886) 2022-09-17 13:23:39 +02:00
Benjamin Peterson fd1e477f53
closes gh-96734: Update to Unicode 15.0.0. (GH-96809) 2022-09-13 15:45:12 -07:00
Eric Snow 1756ffd66a
gh-90110: Fix the c-analyzer Tool (gh-96731)
This includes:

* update the whitelists
* fixes so we can stop ignoring some of the files
* ensure Include/cpython/*.h get analyzed
2022-09-12 11:09:31 -06:00
adang1345 662782e95f
gh-96729: Ensure installers built with Tools/msi/buildrelease.bat have matching UUIDs with official releases (GH-96755) 2022-09-12 15:23:13 +01:00
Christian Heimes a36235d5c7
gh-84461: Omit resource mod and getresuid funcs on Emscripten (GH-96303) 2022-09-11 10:10:00 +02:00
Christian Heimes 1fc8bd3710
gh-95853: Multiple ops and debug for wasm_build.py (#96744) 2022-09-11 09:51:23 +02:00
Kumar Aditya 4e4bfffe2d
GH-90699: use statically allocated interned strings in typeobject's slotdefs (GH-94706) 2022-09-07 15:02:08 -07:00
Steve Dower de33df27aa
gh-89545: Updates platform module to use new internal _wmi module on Windows to directly query OS properties (GH-96289) 2022-09-07 21:09:20 +01:00
Kumar Aditya 6dab8c95bd
GH-96458: Statically initialize utf8 representation of static strings (#96481) 2022-09-02 23:43:08 -07:00
Christian Heimes 2eb9008d72
gh-95853: Improve WASM build script (GH-96389)
- pre-build Emscripten ports and system libraries
- check for broken EMSDK versions
- use EMSDK's node for wasm32-emscripten
- warn when PKG_CONFIG_PATH is set
- add support level information
2022-08-30 07:56:26 +02:00
Christian Heimes d0b3d235db
gh-96320: WASI socket fixes (#96388)
* gh-96320: WASI socket fixes

- ignore missing functions in ``socket.__repr__``
- bundle network files with assets

* blurb
2022-08-30 06:36:11 +02:00
Christian Heimes 873554ef84
gh-94682: Build and test with OpenSSL 1.1.1q (gh-94683) 2022-08-29 18:19:15 +02:00
Carl Friedrich Bolz-Tereick 9c197bc8bf
GH-96172 fix unicodedata.east_asian_width being wrong on unassigned code points (#96207) 2022-08-26 19:29:39 +03:00
Eric Snow d45d5c277a
gh-90110: Update the c-analyzer Tool (gh-96255)
Here we automatically ignore uses of _PyArg_Parser, "kwlist" arrays, and module/type defs. That way new uses don't trigger false positives in the c-analyzer check script.
2022-08-25 15:46:08 -06:00
conioh 1288097088
bpo-46744: Support "-Win32" and make platform flags case insensitive in Windows build scripts. (GH-31803) 2022-08-25 20:23:42 +01:00
Mark Shannon a4a9f2e879
GH-96177: Move GIL and eval breaker code out of ceval.c into ceval_gil.c. (GH-96204) 2022-08-24 14:21:01 +01:00
Carl Friedrich Bolz-Tereick 2d9f252c0c
gh-96019: Fix caching of decompositions in makeunicodedata (GH-96020) 2022-08-19 12:20:44 +03:00
Christian Heimes 6087f491ea
gh-95853: WASM: better version and asset handling in scripts (GH-96045)
- support EMSDK tot-upstream and git releases
- allow WASM assents for wasm64-emscripten and WASI. This makes single file distributions on WASI easier.
- decouple WASM assets from browser builds
2022-08-19 08:36:12 +02:00
Eric Snow 586fc02be5
gh-90110: Update the c-analyzer Tool (gh-96058) 2022-08-17 16:54:59 -06:00
Eric Snow 5aac85101b
gh-90110: Get the C Analyzer Tool Working Again (gh-96057)
We broke it with a recent `_PyArg_Parser` change.

Also:
* moved the `_PyArg_Parser` whitelist entries over to ignored.tsv now that they are thread-safe
* added some known globals from a currently-excluded file
* dropped some outdated globals from the whitelist
2022-08-17 13:55:01 -06:00
Mark Shannon 3ef3c6306d
GH-95707: Fix uses of `Py_TPFLAGS_MANAGED_DICT` (GH-95854)
* Make sure that tp_dictoffset is correct with Py_TPFLAGS_MANAGED_DICT is set.

* Avoid traversing managed dict twice when subclassing class with Py_TPFLAGS_MANAGED_DICT set.
2022-08-15 12:29:27 +01:00
Christian Heimes 4a7f5a55dc
gh-95853: Address wasm build and test issues (GH-95985) 2022-08-15 07:41:10 +02:00
Christian Heimes 32ac98e899
gh-95853: Add script to automate WASM build (GH-95828)
Automate WASM build with a new Python script. The script provides
several build profiles with configure flags for Emscripten flavors
and WASI. The script can detect and use Emscripten SDK and WASI SDK from
default locations or env vars.

``configure`` now detects Node arguments and creates HOSTRUNNER
arguments for Node 16. It also sets correct arguments for
``wasm64-emscripten``.

Co-authored-by: Brett Cannon <brett@python.org>
2022-08-13 21:56:08 +02:00
Erlend E. Aasland f07adf82f3
gh-90928: Improve static initialization of keywords tuple in AC (#95907) 2022-08-13 12:09:40 +02:00
Eric Snow 6f6a4e6cc5
gh-90928: Statically Initialize the Keywords Tuple in Clinic-Generated Code (gh-95860)
We only statically initialize for core code and builtin modules.  Extension modules still create
the tuple at runtime.  We'll solve that part of interpreter isolation separately.

This change includes generated code. The non-generated changes are in:

* Tools/clinic/clinic.py
* Python/getargs.c
* Include/cpython/modsupport.h
* Makefile.pre.in (re-generate global strings after running clinic)
* very minor tweaks to Modules/_codecsmodule.c and Python/Python-tokenize.c

All other changes are generated code (clinic, global strings).
2022-08-11 15:25:49 -06:00
Serhiy Storchaka cc9160a29b
gh-91838: Resolve more HTTP links which redirect to HTTPS (GH-95650) 2022-08-08 14:00:17 +03:00
Oleg Iarygin 56d16e8cb4
gh-93243: Make smtpd private before porting its users (GH-93246)
gh-93243

This PR is required to reduce diffs of the following porting (no need to either maintain documentation and tests consistent with each porting step, or try to port everything and remove smtpd in a single PR).

Automerge-Triggered-By: GH:warsaw
2022-08-05 17:41:29 -07:00
Eric Snow 60f54d9485
gh-90110: Update the C Analyzer Tool Whitelists (gh-95628) 2022-08-04 10:47:02 -06:00
Steve Dower 5b6acbaa20
gh-95587: Fixes some upgrade detection issues in the Windows installer (GH-95631) 2022-08-04 17:12:08 +01:00
Eric Snow a7161f95aa
gh-90110: Get the C Analyzer Tool Working Again (gh-95545) 2022-08-01 17:13:23 -06:00
neonene 7d8973870b
gh-95445: Ensure Windows msi uninstalls document folder successfully (GH-95465) 2022-08-01 16:37:32 +01:00
Mark Shannon de388c0a7b
GH-95245: Store object values and dict pointers in single tagged pointer. (GH-95278) 2022-08-01 14:34:54 +01:00
Christian Heimes 0fe645d6fd
gh-95174: Add pthread stubs for WASI (GH-95234)
Co-authored-by: Brett Cannon <brett@python.org>
2022-07-27 20:28:06 +02:00
Christian Heimes 2833f3798d
gh-95174: Move WASIX logic into wasi-env (GH-95320)
wasi-env now sets WASIX flags. This allows us to control all build
parameter for wasm32-wasi buildbot from CPython repository.

Also export and improve SYSROOT parameter.
2022-07-27 15:30:36 +02:00
Christian Heimes 8b24d60f1b
gh-95174: WASI: skip missing sockets functions (GH-95179) 2022-07-27 08:19:23 +02:00
Christian Heimes daa64d6a59
gh-95085: Promote Emscripten and WASI to PEP 11 tier 3 (GH-95086) 2022-07-27 08:18:34 +02:00
Christian Heimes e8f3e8f0ab
gh-95205: Improve WASM README.md (GH-95267)
Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@protonmail.com>
2022-07-26 11:12:42 +02:00
Dong-hee Na a15ae19ffb
gh-85454: Remove distutils.ccompiler from Tools/c-analyzer (GH-95171) 2022-07-25 21:10:30 +02:00
Kumar Aditya 73ee5a6b86
GH-94851: check refcnt of immortal objects after finalization (GH-95001) 2022-07-25 10:43:59 -07:00
Erlend Egeberg Aasland 310f94871a
gh-95205: Improve wasm README (#95206)
Co-authored-by: Christian Heimes <christian@python.org>
2022-07-25 09:42:50 +02:00
Brandt Bucher e402b26b7f
GH-95113: Don't use EXTENDED_ARG_QUICK in unquickened code (GH-95121) 2022-07-22 11:04:20 -07:00
Christian Heimes 0f34c7e2d3
gh-93939: Use new MODULE_name_STATE in wasm_assets script (GH-95035) 2022-07-20 08:53:26 +02:00
Serhiy Storchaka 74b5e4ce80
gh-95007: Remove the NoneType return converter (GH-95019)
It has confusing semantic which does not provide any benefit (the
only difference is that you should write "return Py_None" instead
of "Py_RETURN_NONE"), it is not currently used, and it is broken.
2022-07-20 08:42:02 +03:00
Eric Snow 7a1a85d640
gh-94673: [c-analyzer] Add a Script to Identify Static Types (#94989)
issue: https://github.com/python/cpython/issues/94673
2022-07-18 19:03:57 -06:00
Christian Heimes 81dca70d70
gh-93939: Build C extensions without setup.py (GH-94474)
Combines GH-93940, GH-94452, and GH-94433
2022-07-14 00:51:49 -07:00
Christian Heimes 0c66074e9f
gh-94773: deepfreeze: support frozensets with unsortable types (GH-94775) 2022-07-12 18:09:47 +02:00
Kumar Aditya 9dff9f4814
GH-90699: Intern statically allocated strings (GH-93597)
This is similar to how strings are interned for deepfreeze.
2022-07-08 10:47:37 -07:00
Kumar Aditya 71697664d7
GH-90699: Move generated static initializer to pycore_runtime_generated.h (GH-94051) 2022-07-07 13:04:05 -07:00
Erlend Egeberg Aasland 8bbd70b4d1
gh-94430: Allow params named `module` or `self` with custom C names in Argument Clinic (#94431) 2022-07-07 11:29:34 +02:00
Erlend Egeberg Aasland 2b8ed4d3d4
gh-94538: Fix Argument Clinic output to custom file (#94539) 2022-07-05 11:06:04 +02:00
Christian Heimes 7bd67d1d88
gh-93939: Add script to check extension modules (#94545)
Add script ``Tools/scripts/check_modules.py`` to check and validate builtin
and shared extension modules. The script also handles ``Modules/Setup`` and
will eventually replace ``setup.py``.

Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@protonmail.com>
2022-07-05 09:25:45 +02:00
Irit Katriel c57aad777a
gh-94216: add pseudo instructions to the dis/opcodes modules (GH-94241) 2022-07-01 15:33:35 +01:00
Petr Viktorin 1df9449db2
gh-84461: Tool/wasm/python.html: Add subresource integrity hashes (#93953) 2022-07-01 11:54:30 +02:00
Trey Hunner a8e333d79a
gh-84461: Improve WebAssembly in-browser demo (#91879)
* Buffer standard input line-by-line

* Add non-root .editorconfig for JS & HTML indent

* Add support for clearing REPL with CTRL+L

* Support unicode in stdout and stderr

* Remove \r\n normalization

* Note that local .editorconfig file extends root

* Only normalize lone \r characters (convert to \n)

* Skip non-printable characters in buffered input

* Fix Safari bug (regex lookbehind not supported)

Co-authored-by: Christian Heimes <christian@python.org>
2022-07-01 11:52:58 +02:00
yutotnh 5631013617
Fix typos in comments (#94305) 2022-06-28 16:28:21 +02:00
Christian Heimes 44fa03d748
gh-93939: Create and install scripts in Makefile (GH-94324)
Scripts for 2to3, idle, and pydoc are now created and installed by make.
2022-06-28 14:56:06 +02:00
Mark Shannon c0453a40fa
GH-94163: Add BINARY_SLICE and STORE_SLICE instructions. (GH-94168) 2022-06-27 12:24:23 +01:00
Ken Jin 50a5ab2c0b
gh-93382: Sync up `co_code` changes with 3.11 (GH-94227)
Sync up co_code changes with 3.11 commit 852b4d4bcd.
2022-06-25 01:55:18 +08:00
Christian Heimes e69306f08b
gh-84461: Build Emscripten with WASM BigInt support (#94219) 2022-06-24 17:03:42 +02:00
Christian Heimes 8625802d85
gh-84461: Fix ctypes and test_ctypes on Emscripten (#94142)
- c_longlong and c_longdouble need experimental WASM bigint.
- Skip tests that need threading
- Define ``CTYPES_MAX_ARGCOUNT`` for Emscripten. libffi-emscripten 2022-06-23 supports up to 1000 args.
2022-06-24 12:40:43 +02:00
Guido van Rossum 4e796f5646
gh-93771: Clarify how deepfreeze.py is run (#94150) 2022-06-23 09:31:39 -07:00
Victor Stinner c1fb12e5af
gh-54781: Move Lib/tkinter/test/test_ttk/ to Lib/test/test_ttk/ (#94070)
* Move Lib/tkinter/test/test_tkinter/ to Lib/test/test_tkinter/.
* Move Lib/tkinter/test/test_ttk/ to Lib/test/test_ttk/.
* Add Lib/test/test_ttk/__init__.py based on test_ttk_guionly.py.
* Add Lib/test/test_tkinter/__init__.py
* Remove old Lib/test/test_tk.py.
* Remove old Lib/test/test_ttk_guionly.py.
* Add __main__ sub-modules.
* Update imports and update references to rename files.
2022-06-22 22:23:37 +02:00
Victor Stinner 47e35625ff
gh-84623: Remove unused imports (#94132) 2022-06-22 19:14:27 +02:00
Illia Volochii d36954b7ea
gh-91172: Create a workflow for verifying bundled pip and setuptools (GH-31885)
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
2022-06-22 15:32:22 +02:00
Mark Shannon f805d37641
Stats: Add summary of top instructions for misses and deferred specialization. (GH-94072) 2022-06-22 08:50:02 +01:00
Ezio Melotti f28ec34c5c
gh-82927: Update files related to HTML entities. (GH-92504) 2022-06-21 22:03:12 +02:00
Victor Stinner 616fa3465d
gh-54781: Move Lib/lib2to3/tests/ to Lib/test/test_lib2to3/ (#94049)
* Move Lib/lib2to3/tests/ to Lib/test/test_lib2to3/.
* Remove Lib/test/test_lib2to3.py.
* Update imports.
* all_project_files(): use different paths and sort files
  to make the tests more reproducible.
* Update references to tests.
2022-06-21 15:21:22 +02:00
Victor Stinner c735d54534
gh-93839: Move Lib/unttest/test/ to Lib/test/test_unittest/ (#94043)
* Move Lib/unittest/test/ to Lib/test/test_unittest/
* Remove Lib/test/test_unittest.py
* Replace unittest.test with test.test_unittest
* Remove unittest.load_tests()
* Rewrite unittest __init__.py and __main__.py
* Update build system, CODEOWNERS, and wasm_assets.py
2022-06-21 10:27:59 +02:00
Victor Stinner d82e0bfe8b
gh-93839: Move Lib/ctypes/test/ to Lib/test/test_ctypes/ (#94041)
* Move Lib/ctypes/test/ to Lib/test/test_ctypes/
* Remove Lib/test/test_ctypes.py
* Update imports and build system.
2022-06-21 10:24:33 +02:00
Mark Shannon ab0e601016
GH-93516: Speedup line number checks when tracing. (GH-93763)
* Use a lookup table to reduce overhead of getting line numbers during tracing.
2022-06-20 13:00:42 +01:00
Mark Shannon 45e62a2bc1
GH-93897: Store frame size in code object and de-opt if insufficient space on thread frame stack. (GH-93908) 2022-06-20 12:59:25 +01:00
Christian Heimes 2702e408fd
gh-84461: Fix Emscripten umask and permission issues (GH-94002)
- Emscripten's default umask is too strict, see
  https://github.com/emscripten-core/emscripten/issues/17269
- getuid/getgid and geteuid/getegid are stubs that always return 0
  (root). Disable effective uid/gid syscalls and fix tests that use
  chmod() current user.
- Cannot drop X bit from directory.
2022-06-19 18:28:55 +02:00
Christian Heimes 7a2cc35e1c
gh-84461: Fix pydebug Emscripten browser builds (GH-93982)
wasm_assets script did not take the ABIFLAG flag of sysconfigdata into
account.
2022-06-18 14:51:50 +02:00
Christian Heimes 8ba1c7f720
gh-84461: Document how to install SDKs manually (GH-93844)
Co-authored-by: Brett Cannon <brett@python.org>
2022-06-15 21:28:34 +02:00
Steve Dower 99be1cbeb3
gh-93824: Reenable installation of shell extension on Windows ARM64 (GH-93825) 2022-06-15 17:07:55 +01:00
Mark Shannon cdf7097612
Include freelists in allocation total. (GH-93799) 2022-06-14 14:44:26 +01:00
Mark Shannon 3cd1a5d3ec
GH-93516: Store offset of first traceable instruction in code object (GH-93769) 2022-06-14 11:09:30 +01:00
Christian Heimes c2007573dd
gh-84461: Fix parallel testing on WebAssembly (GH-93768) 2022-06-13 19:51:04 +02:00
Davide Rizzo 733e15f170
gh-84508: tool to generate cjk traditional chinese mappings (gh-93272) 2022-06-11 23:19:41 +09:00
Christian Heimes dc5e02b2f4
gh-84461: Use HOSTRUNNER to run regression tests (GH-93694)
Co-authored-by: Brett Cannon <brett@python.org>
2022-06-11 08:42:23 +02:00
Christian Heimes 70690c7233
gh-84461: Emscripten's faccessat() does not accept flags (GHß92353) 2022-06-07 14:04:27 +02:00
Christian Heimes 22fed605e0
gh-90473: Make chmod a dummy on WASI, skip chmod tests (GH-93534)
WASI does not have the ``chmod(2)`` syscall yet.
2022-06-06 19:24:11 +02:00
Christian Heimes 4c71d22c4f
gh-90473: WASI requires proper open(2) flags (GH-93529) 2022-06-06 12:08:07 +02:00
Christian Heimes 069c96f84c
gh-90473: Skip and document more failing tests on WASI (GH-93436)
- Mark more ``umask()`` cases
- ``dup()`` is not supported
- ``/dev/null`` is not available
- document missing features
- mark more modules as not available
2022-06-03 00:44:48 +02:00
Dong-hee Na 749dc4b9c2
Revert "gh-84508: Add mapping files for Korean and Japanese. (gh-93309)" (#93320)
This reverts commit dec1e9346d.
2022-05-29 09:49:19 +09:00
Dong-hee Na dec1e9346d
gh-84508: Add mapping files for Korean and Japanese. (gh-93309) 2022-05-28 12:32:00 +09:00
Mark Shannon bbcf42449e
GH-90230: Add stats to breakdown the origin of calls to `PyEval_EvalFrame` (GH-93284) 2022-05-27 16:31:41 +01:00
Christian Heimes 1f134e96ba
gh-90473: Misc test fixes for WASI (GH-93218)
* ``sys.executable`` is not set
* WASI does not support subprocess
* ``pwd`` module is not available
* WASI checks ``open`` syscall flags more strict, needs r, w, rw flag.
* ``umask`` is not available
* ``/dev/null`` may not be accessible
2022-05-25 15:57:26 +02:00
Victor Stinner 5115a16831
gh-93103: Parser uses PyConfig.parser_debug instead of Py_DebugFlag (#93106)
* Replace deprecated Py_DebugFlag with PyConfig.parser_debug in the
  parser.
* Add Parser.debug member.
* Add tok_state.debug member.
* Py_FrozenMain(): Replace Py_VerboseFlag with PyConfig.verbose.
2022-05-24 22:35:08 +02:00
Christian Heimes 760ec8940a
gh-90473: WASI: skip gethostname tests (GH-93092)
- WASI's ``gethostname()`` is a stub that always fails with OSError
  ``ENOTSUP``
- skip mailcap ``test`` if subprocess is not available
- WASI process_time clock does not work.
2022-05-23 10:39:57 +02:00
Dong-hee Na 909868db2a
Update globals-to-fix.tsv to follow recent changes (gh-92959) 2022-05-20 14:06:54 +09:00
Christian Heimes 137fd3d88a
gh-90473: Decrease recursion limit and skip tests on WASI (GH-92803) 2022-05-19 12:43:16 +02:00
Mark Shannon a4460f2eb8
Split refcount stats into 'interpreter' and 'non-interpreter' (GH-92919) 2022-05-18 14:38:43 +01:00
Zackery Spytz 6b932cb133
bpo-38704: Prevent installation on unsupported Windows versions (GH-17950) 2022-05-17 23:24:17 +01:00
Mark Shannon 93fc14933b
Summarize stats: Increase number of predecessor/successor pairs shown from 3 to 5. (GH-92853) 2022-05-17 10:59:24 +01:00
Mark Shannon fa2b8b75eb
Improve object stats (#92845)
* Add incref/decref stats

* Show ratios for allocation in summary
2022-05-16 14:35:11 +01:00
Inada Naoki f9c9354a7a
gh-92536: PEP 623: Remove wstr and legacy APIs from Unicode (GH-92537) 2022-05-12 14:48:38 +09:00
Victor Stinner da5727a120
gh-92651: Remove the Include/token.h header file (#92652)
Remove the token.h header file. There was never any public tokenizer
C API. The token.h header file was only designed to be used by Python
internals.

Move Include/token.h to Include/internal/pycore_token.h. Including
this header file now requires that the Py_BUILD_CORE macro is
defined. It no longer checks for the Py_LIMITED_API macro.

Rename functions:

* PyToken_OneChar() => _PyToken_OneChar()
* PyToken_TwoChars() => _PyToken_TwoChars()
* PyToken_ThreeChars() => _PyToken_ThreeChars()
2022-05-11 23:22:50 +02:00
Erlend Egeberg Aasland 4bd07d1dbd
gh-92256: Improve Argument Clinic parser error messages (#92268)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
2022-05-10 09:23:42 +02:00
Serhiy Storchaka 3680ebed7f
bpo-44712: Replace "type(literal)" with corresponding builtin types (GH-27294)
I suppose it is a remnants of very old code written when str, int, list, dict, etc
were functions and not classes.
2022-05-08 17:10:11 +03:00
Victor Stinner 329afe78c3
gh-57684: Update tests for PYTHONSAFEPATH=1 (#92358)
Fix tests failing with the PYTHONSAFEPATH=1 env var.

Enhance also -P help in Python usage (python --help).
2022-05-06 03:41:24 +02:00
Eric Snow 1f631ae3a1
gh-81057: Fix indent. (gh-92267) 2022-05-03 15:44:52 -06:00
Victor Stinner 804f2529d8
gh-91320: Use _PyCFunction_CAST() (#92251)
Replace "(PyCFunction)(void(*)(void))func" cast with
_PyCFunction_CAST(func).

Change generated by the command:

sed -i -e \
  's!(PyCFunction)(void(\*)(void)) *\([A-Za-z0-9_]\+\)!_PyCFunction_CAST(\1)!g' \
  $(find -name "*.c")
2022-05-03 21:42:14 +02:00
Eric Snow 456cd513e3
gh-81057: Get the c-analyzer tool working again. (gh-92246) 2022-05-03 13:18:27 -06:00
Victor Stinner b270b82f11
gh-91320: Argument Clinic uses _PyCFunction_CAST() (#32210)
Replace "(PyCFunction)(void(*)(void))func" cast with
_PyCFunction_CAST(func).
2022-05-03 20:25:41 +02:00
Dennis Sweeney b156578bd6
gh-92031: Deoptimize Static Code at Finalization (GH-92039) 2022-05-03 08:59:12 -06:00
Serhiy Storchaka a055dac0b4
gh-91583: AC: Fix regression for functions with defining_class (GH-91739)
Argument Clinic now generates the same efficient code as before
adding the defining_class parameter.
2022-04-30 13:15:02 +03:00
Petr Viktorin 83bce8ef14
gh-91324: Convert the stable ABI manifest to TOML (GH-92026) 2022-04-29 16:18:08 +02:00
Hugo van Kemenade 89c6b2b8f6
bpo-36329: Remove 'make -C Doc serve' in favour of 'make -C Doc htmlview' (GH-32354)
Also updated `make -C htmlview` so it used a full path with `file://`, because the original didn't open the page (macOS).

For example:
```sh
cd Doc

# Doesn't open anything:
python3 -c "import webbrowser; webbrowser.open('build/html/index.html')"

# Opens the docs page e.g. file:///Users/hugo/github/cpython/Doc/build/html/index.html :
python3 -c "import os, webbrowser; webbrowser.open('file://' + os.path.realpath('build/html/index.html'))"
```

https://bugs.python.org/issue36329
2022-04-29 15:47:26 +02:00
Petr Viktorin 6dcbc08c95
gh-91324: List feature macros in the stable ABI manifest, improve tests (GH-32415) 2022-04-28 16:30:28 +02:00
Victor Stinner 64a54e511d
gh-91719: Add pycore_opcode.h internal header file (#91906)
Move the following API from Include/opcode.h (public C API) to a new
Include/internal/pycore_opcode.h header file (internal C API):

* EXTRA_CASES
* _PyOpcode_Caches
* _PyOpcode_Deopt
* _PyOpcode_Jump
* _PyOpcode_OpName
* _PyOpcode_RelativeJump
2022-04-26 00:14:30 +02:00
Christian Heimes 92c1037afc
gh-84461: Add --enable-wasm-pthreads and more file systems (GH-91820) 2022-04-23 14:59:33 +02:00
Christian Heimes 9b5ca5405e
gh-84461: Add sys._emscripten_info, improve docs and build (gh-91781) 2022-04-23 09:52:16 +02:00
Serhiy Storchaka f912cc0e41
gh-91575: Add a script for generating data for case-insensitive matching in re (GH-91660)
Also test that all extra cases are in BMP.
2022-04-22 21:37:46 +03:00
Guido van Rossum f8dc6186d1
GH-91719: Make MSVC generate somewhat faster switch code (#91718)
Apparently a switch on an 8-bit quantity where all cases are
present generates a more efficient jump (doing only one indexed
memory load instead of two).

So we make opcode and use_tracing uint8_t, and generate a macro
full of extra `case NNN:` lines for all unused opcodes.

See https://github.com/faster-cpython/ideas/issues/321#issuecomment-1103263673
2022-04-21 11:53:57 -07:00
Petr Viktorin ac4ffd3be2
bpo-47169: Export PyOS_CheckStack on Windows (GH-32414) 2022-04-21 18:03:25 +02:00
Mark Shannon 944fffee89
GH-88116: Use a compact format to represent end line and column offsets. (GH-91666)
* Stores all location info in linetable to conform to PEP 626.

* Remove column table from code objects.

* Remove end-line table from code objects.

* Document new location table format
2022-04-21 16:10:37 +01:00
Kumar Aditya ab0d35d70d
bpo-46712: share more global strings in deepfreeze (gh-32152)
(for gh-90868)
2022-04-19 11:41:36 -06:00
Barry Warsaw dbd9d75fed
Add some information about where pynche has moved to (#91650) 2022-04-18 07:34:53 -07:00
Barry Warsaw 7173fd5de0
Remove the ancient Pynche color editor (#91554)
Closes #91551
2022-04-17 15:38:44 -07:00
Dennis Sweeney 37965d2fb4
gh-78607: Replace __ltrace__ with __lltrace__ (GH-91619) 2022-04-16 18:57:00 -04:00
Yu Liu 9300b6d729
gh-91595: fix the comparison of character and integer by using ord() (#91596)
* fix the comparison of character and integer by using ord()

* 📜🤖 Added by blurb_it.

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2022-04-16 16:34:48 +02:00
Brandt Bucher 1b34b5687b
gh-91404: Use computed gotos and reduce indirection in re (#91495) 2022-04-15 09:26:44 -07:00
Dennis Sweeney c9d41bcd68
gh-91428: include specialized opcodes in _PyOpcode_OpName (GH-91467) 2022-04-14 16:00:58 -04:00
Inada Naoki 13b17e2a0a
gh-91156: Fix `encoding="locale"` in UTF-8 mode (GH-70056) 2022-04-14 16:00:35 +09:00
Serhiy Storchaka 474fdbe9e4
bpo-47152: Automatically regenerate sre_constants.h (GH-91439)
* Move the code for generating Modules/_sre/sre_constants.h from
  Lib/re/_constants.py into a separate script
  Tools/scripts/generate_sre_constants.py.
* Add target `regen-sre` in the makefile.
* Make target `regen-all` depending on `regen-sre`.
2022-04-12 18:34:06 +03:00
Dennis Sweeney 8a35ce3796
gh-91428: Add _PyOpcode_OpName to opcode.h of debug builds (GH-91430) 2022-04-11 18:33:00 -04:00
Christian Heimes defbbd68f7
bpo-40280: WASM docs and smaller browser builds (GH-32412)
Co-authored-by: Brett Cannon <brett@python.org>
2022-04-10 09:29:51 +02:00
Petr Viktorin 1c2fddddae
Add feature macro PY_HAVE_THREAD_NATIVE_ID to the stable ABI definition (GH-32365) 2022-04-08 14:35:11 +02:00
Brandt Bucher ef6a482b02
bpo-47177: Replace `f_lasti` with `prev_instr` (GH-32208) 2022-04-07 12:31:01 -07:00
Jeremy Kloth 612e422c6e
bpo-46576: Speed up test_peg_generator by using a static library for shared sources (GH-32338)
Speed up test_peg_generator by using a static library for shared sources to avoid recompiling as much code.
2022-04-06 14:55:58 -07:00
Petr Viktorin d79f118d04
bpo-47115: Document which parts of structs are in limited API/stable ABI (GH-32196)
Co-authored-by: Erlend Egeberg Aasland <erlend.aasland@innova.no>
2022-04-06 16:50:45 +02:00
Christian Heimes 96e09837fb
bpo-40280: Add limited Emscripten REPL (GH-32284)
Co-authored-by: Katie Bell <katie@katharos.id.au>
2022-04-05 11:21:11 +02:00
Christian Heimes c9844cb8aa
bpo-40280: Add --enable-wasm-dynamic-linking (GH-32253) 2022-04-04 19:31:31 +02:00
Serhiy Storchaka 1578f06c1c
bpo-47152: Move sources of the _sre module into a subdirectory (GH-32290) 2022-04-04 10:53:26 +03:00
Christian Heimes 4ed8a9a589
bpo-40280: Enable most file-at() and nanosleep APIs again (GH-32238) 2022-04-02 10:13:44 +02:00
Eric Snow e7bb7c2f04
bpo-47146: Stop Depending On regen-deepfreeze For regen-global-objects (gh-32218)
This effectively reverts the Makefile change in gh-31637. I've added some notes so it is more clear what is going on.

We also update the "Check if generated files are up to date" job to run "make regen-deepfreeze" to ensure "make regen-global-objects" catches deepfreeze.c.

https://bugs.python.org/issue47146
2022-03-31 14:29:52 -06:00
Eric Snow db4dada510
bpo-47146: Avoid Using make Recursively (gh-32206)
https://bugs.python.org/issue47146
2022-03-30 19:24:02 -06:00
Steve Dower 2ab609dd61
bpo-47171: Enable installing the py.exe launcher on Windows ARM64 (GH-32203) 2022-03-30 21:20:38 +01:00
Hugo van Kemenade 6881ea936e
bpo-47126: Update to canonical PEP URLs specified by PEP 676 (GH-32124) 2022-03-30 12:00:27 +01:00
Steve Dower bad86a621a
bpo-46566: Add new py.exe launcher implementation (GH-32062) 2022-03-29 00:21:08 +01:00
Kumar Aditya 785cc67705
bpo-46429: tweak deepfreeze output (#32107) 2022-03-27 11:46:22 -07:00
Eric Snow febf54bcf3
bpo-46712: Do not Regen Deep-Frozen Modules before Generating Global Objects (gh-32061)
We have to run "make regen-deepfreeze" before running Tools/scripts/generate-global-objects.py; otherwise we will miss any changes to global objects in deep-frozen modules (which aren't committed in the repo).  However, building $(PYTHON_FOR_FREEZE) fails if one of its source files had a global object (e.g. via _Py_ID(...)) added or removed, without generate-global-objects.py running first.  So "make regen-global-objects" would sometimes fail.

We solve this by running generate-global-objects.py before *and* after "make regen-deepfreeze". To speed things up and cut down on noise, we also avoid updating the global objects files if there are no changes to them.

https://bugs.python.org/issue46712
2022-03-23 09:55:52 -06:00
Christian Heimes 4aea656d62
bpo-32033: Finalize WASI configure options (GH-32053) 2022-03-22 18:42:09 +01:00
Jeremy Kloth 88872a29f1
bpo-47084: Clear Unicode cached representations on finalization (GH-32032) 2022-03-22 13:53:51 +01:00
Steve Dower 3751b6b030
bpo-47086: Remove .chm from Windows installer and add HTML docs (GH-32038) 2022-03-22 01:08:37 +00:00
Brandt Bucher 2bde6827ea
bpo-46841: Quicken code in-place (GH-31888)
* Moves the bytecode to the end of the corresponding PyCodeObject, and quickens it in-place.

* Removes the almost-always-unused co_varnames, co_freevars, and co_cellvars member caches

* _PyOpcode_Deopt is a new mapping from all opcodes to their un-quickened forms.

* _PyOpcode_InlineCacheEntries is renamed to _PyOpcode_Caches

* _Py_IncrementCountAndMaybeQuicken is renamed to _PyCode_Warmup

* _Py_Quicken is renamed to _PyCode_Quicken

* _co_quickened is renamed to _co_code_adaptive (and is now a read-only memoryview).

* Do not emit unused nonzero opargs anymore in the compiler.
2022-03-21 11:11:17 +00:00
Dennis Sweeney d7a93cbf4b
summarize_stats.py: add pairs by opcode (GH-31957) 2022-03-18 10:10:31 +00:00
Christian Heimes ef1327e3b6
bpo-40280: Skip more tests on Emscripten (GH-31947)
- lchmod, lchown are not fully implemented
- skip umask tests
- cannot fstat unlinked or renamed files yet
- ignore musl libc issues that affect Emscripten
2022-03-17 12:09:57 +01:00
Steve Dower 7c77652141
bpo-47032: Ensure Windows install builds fail correctly with a non-zero exit code when part of the build fails (GH-31921) 2022-03-16 01:36:20 +00:00
Steve Dower 7088120853
bpo-46948: Fix launcher installer build failure due to first part of fix (GH-31920) 2022-03-16 00:46:33 +00:00
Christian Heimes af0a50de4b
bpo-47024: Update OpenSSL to 1.1.1n (GH-31895)
Co-authored-by: Zachary Ware <zachary.ware@gmail.com>
2022-03-15 21:03:04 +01:00
Inada Naoki 52f6ce396d
Tools/gdb: Drop code to support Python 2. (GH-31717) 2022-03-15 17:04:11 +09:00
Hugo van Kemenade 23abae621f
CI: Fix patchcheck (GH-31708) 2022-03-14 15:28:57 +02:00
Christian Heimes ecfff63e06
bpo-40280: Disable AF_UNIX, AF_PACKET, SO_REUSE* on Emscripten (#31829)
Emscripten's socket emulation is limited. AF_UNIX, AF_PACKET, setsockopt(), and most SO_* constants are not supported.
2022-03-11 23:25:14 +01:00
Christian Heimes de554d6e02
bpo-40280: Skip more tests/features that don't apply to Emscripten (GH-31791)
- fd inheritance can't be modified because Emscripten doesn't support subprocesses anyway.
- setpriority always fails
- geteuid no longer causes problems with latest emsdk
- umask is a stub
- geteuid / getuid always return 0, but process cannot chown to random uid.
2022-03-10 13:43:40 +01:00
Kumar Aditya 8714b6fa27
bpo-46881: Statically allocate and initialize the latin1 characters. (GH-31616) 2022-03-09 15:02:00 -08:00
Christian Heimes 5081e78efd
bpo-40280: Block more non-working syscalls in Emscripten (GH-31757)
- getgroups always fails.
- geteuid and getegid always return 0 (root), which confuse tarfile and
  tests.
- hardlinks (link, linkat) always fails.
- non-encodable file names are not supported by NODERAWFS layer.
- mark more tests with dependency on subprocess and multiprocessing.
  Mocking does not work if the module fails to import.
2022-03-08 12:17:30 +01:00
Steve Dower 77446d2aa5
bpo-46948: Fix CVE-2022-26488 by ensuring the Windows Installer correctly uses the install path during repair (GH-31726) 2022-03-07 17:23:11 +00:00
Luca Chiodini 7f07b5ee9c
Update grammar_grapher with the new forced (&&) directive (#31704) 2022-03-06 15:26:28 +01:00
Serhiy Storchaka 6927632492
Remove trailing spaces (GH-31695) 2022-03-05 17:47:00 +02:00
Steve Dower 8f31bf4698
bpo-46744: Move Windows ARM64 installation directory to correct ProgramFiles (GH-31677) 2022-03-04 00:50:42 +00:00
Inada Naoki 9833bb91e4
bpo-46845: Reduce dict size when all keys are Unicode (GH-31564) 2022-03-02 08:09:28 +09:00
Eric Snow 21099fc064
bpo-46712: Let generate_global_objects.py Run on Earlier Python Versions (gh-31637)
https://bugs.python.org/issue46712
2022-03-01 14:29:54 -07:00
Eric Snow 08deed1af5
bpo-46753: Add the empty tuple to the _PyRuntimeState.global_objects. (gh-31345)
https://bugs.python.org/issue46753
2022-02-28 15:15:48 -07:00
Mark Shannon 4558af5a8f
bpo-46841: Move the cache for `LOAD_GLOBAL` inline. (GH-31575) 2022-02-28 12:56:29 +00:00
Steve Dower da7d99a4de
bpo-46567: Add Tcl/Tk build for Windows ARM64 (GH-31574) 2022-02-28 12:06:43 +00:00
Kumar Aditya 0d9b565e62
Propagate errors (however unlikely) from _Py_Deepfreeze_Init() (GH-31596) 2022-02-26 08:35:03 -08:00
Kumar Aditya eb002dbe0d
bpo-46712: Share global string identifiers in deepfreeze (GH-31261)
Where appropriate, deepfreeze.c now uses `&_Py_ID(blah)` references instead of locally defining constants. This saves some space.
2022-02-25 10:05:24 -08:00
Brandt Bucher 0f41aac109
bpo-46841: Use *inline* caching for `BINARY_OP` (GH-31543) 2022-02-25 12:11:34 +00:00
Petr Viktorin 2c228a7b8f
bpo-46748: Don't import <stdbool.h> in public headers (GH-31553)
<stdbool.h> is the standard/modern way to define embedd/extends Python free to define bool, true and false, but there are existing applications that use slightly different redefinitions, which fail if the header is included.

It's OK to use stdbool outside the public headers, though.

https://bugs.python.org/issue46748
2022-02-25 09:25:54 +01:00
Kumar Aditya 4dc746310b
bpo-46430: Fix memory leak in interned strings of deep-frozen modules (GH-31549) 2022-02-24 17:54:06 +01:00
Victor Stinner 042f31da55
bpo-45459: C API uses type names rather than structure names (GH-31528)
Thanks to the new pytypedefs.h, it becomes to use type names like
PyObject rather like structure names like "struct _object".
2022-02-24 17:51:59 +01:00
Eric Snow 1f455361ec
bpo-46765: Replace Locally Cached Strings with Statically Initialized Objects (gh-31366)
https://bugs.python.org/issue46765
2022-02-22 17:23:51 -07:00
Mark Shannon 09487c11a5
Fix reporting of specialization stats. (GH-31503) 2022-02-22 17:18:10 +00:00
Mark Shannon 1320d56ff7
Remove pair-counts from specialization stats. (GH-31500) 2022-02-22 15:44:31 +00:00
Eric Snow 4d8a515d19
bpo-46541: Scan Fewer Files in generate_global_objects.py (gh-31364)
https://bugs.python.org/issue46541
2022-02-15 20:07:11 -07:00
Eric Snow 6c89589486
bpo-46541: Drop the check for orphaned global strings. (gh-31363)
https://bugs.python.org/issue46541
2022-02-15 20:06:38 -07:00
Eric Snow 12360aa159
bpo-46541: Discover the global strings. (gh-31346)
Instead of manually enumerating the global strings in generate_global_objects.py, we extrapolate the list from usage of _Py_ID() and _Py_STR() in the source files.

This is partly inspired by gh-31261.

https://bugs.python.org/issue46541
2022-02-14 17:36:51 -07:00
Eric Snow 1d9036425e
Regen the global objects using PYTHON_FOR_REGEN. (gh-31344)
https://bugs.python.org/issue46541
2022-02-14 16:07:09 -07:00
Mark Shannon 0ade875ebe
Add pair counts to stats output and summary. (GH-31324) 2022-02-14 15:53:38 +00:00
Eric Snow 80e4f262aa
bpo-36876: Make sure the c-analyzer is checking all the source files.' (gh-31264)
https://bugs.python.org/issue36876
2022-02-10 16:14:19 -07:00
Pablo Galindo Salgado 390459de6d
Allow the parser to avoid nested processing of invalid rules (GH-31252) 2022-02-10 13:12:14 +00:00
Mark Shannon b0662ae5c8
Add stats for PRECALL_FUNCTION. (GH-31250) 2022-02-10 11:47:52 +00:00
Mark Shannon cfc1cecd7b
Right justify numeric columns in stats summary script. (GH-31234) 2022-02-10 11:01:18 +00:00
Eric Snow cb68788dca
bpo-36876: Minor cleanup to c-analyzer "ignored" data.' (gh-31239)
https://bugs.python.org/issue36876
2022-02-09 17:10:53 -07:00
Kumar Aditya c0a5ebeb12
bpo-46430: Intern strings in deep-frozen modules (GH-30683) 2022-02-09 08:52:42 -08:00
Mark Shannon f71a69aa92
bpo-46072: Output stats as markdown with collapsible sections. (GH-31228) 2022-02-09 12:30:26 +00:00
Eric Snow 77bab59c8a
bpo-36876: Update the c-analyzer whitelist. (gh-31225)
This change adds variables that had been added since the last time the whitelist was updated. It also cleans up the list a little.

https://bugs.python.org/issue36876
2022-02-08 18:37:57 -07:00
Eric Snow 81c72044a1
bpo-46541: Replace core use of _Py_IDENTIFIER() with statically initialized global objects. (gh-30928)
We're no longer using _Py_IDENTIFIER() (or _Py_static_string()) in any core CPython code.  It is still used in a number of non-builtin stdlib modules.

The replacement is: PyUnicodeObject (not pointer) fields under _PyRuntimeState, statically initialized as part of _PyRuntime.  A new _Py_GET_GLOBAL_IDENTIFIER() macro facilitates lookup of the fields (along with _Py_GET_GLOBAL_STRING() for non-identifier strings).

https://bugs.python.org/issue46541#msg411799 explains the rationale for this change.

The core of the change is in:

* (new) Include/internal/pycore_global_strings.h - the declarations for the global strings, along with the macros
* Include/internal/pycore_runtime_init.h - added the static initializers for the global strings
* Include/internal/pycore_global_objects.h - where the struct in pycore_global_strings.h is hooked into _PyRuntimeState
* Tools/scripts/generate_global_objects.py - added generation of the global string declarations and static initializers

I've also added a --check flag to generate_global_objects.py (along with make check-global-objects) to check for unused global strings.  That check is added to the PR CI config.

The remainder of this change updates the core code to use _Py_GET_GLOBAL_IDENTIFIER() instead of _Py_IDENTIFIER() and the related _Py*Id functions (likewise for _Py_GET_GLOBAL_STRING() instead of _Py_static_string()).  This includes adding a few functions where there wasn't already an alternative to _Py*Id(), replacing the _Py_Identifier * parameter with PyObject *.

The following are not changed (yet):

* stop using _Py_IDENTIFIER() in the stdlib modules
* (maybe) get rid of _Py_IDENTIFIER(), etc. entirely -- this may not be doable as at least one package on PyPI using this (private) API
* (maybe) intern the strings during runtime init

https://bugs.python.org/issue46541
2022-02-08 13:39:07 -07:00
Eric Snow c018d3037b
bpo-45952: Get the C analyzer tool working again. (gh-31220)
https://bugs.python.org/issue45952
2022-02-08 12:37:53 -07:00
Eric Snow 1e6214dbd6
bpo-45952: Get the C analyzer tool working again. (gh-31219)
https://bugs.python.org/issue45952
2022-02-08 11:42:17 -07:00
Mark Shannon 328fe3fd20
Print summary stats for overall success of specialization. (GH-31211) 2022-02-08 11:50:02 +00:00
Christian Heimes 96b344c2f1
bpo-40280: Address more test failures on Emscripten (GH-31050)
Co-authored-by: Brett Cannon <brett@python.org>
2022-02-05 20:52:01 +01:00
Kumar Aditya 9d4161a60c
bpo-46608: Fix argument parsing in freeze_modules.py (GH-31131) 2022-02-05 07:59:48 -08:00
Kumar Aditya bf95ff91f2
bpo-46608: exclude marshalled-frozen data if deep-freezing to save 300 KB space (GH-31074)
This reduces the size of the data segment by **300 KB** of the executable because if the modules are deep-frozen then the marshalled frozen data just wastes space. This was inspired by comment by @gvanrossum in https://github.com/python/cpython/pull/29118#issuecomment-958521863. Note: There is a new option `--deepfreeze-only` in `freeze_modules.py` to change this behavior, it is on be default to save disk space.
```console 
# du -s ./python before
27892   ./python
# du -s ./python after
27524   ./python
```

Automerge-Triggered-By: GH:ericsnowcurrently
2022-02-04 09:57:03 -08:00
Mark Shannon 832876b992
Add miss stats for specialized instructions. (GH-31108) 2022-02-04 09:56:46 +00:00
Manish Kumar ⛄ ba650af7d6
Optimize images by IMGbot (GH-21348)
Co-authored-by: ImgBotApp <ImgBotHelp@gmail.com>
2022-02-04 15:49:43 +09:00
Gregory P. Smith 164a017e13
bpo-46576: bpo-46524: Disable compiler optimization within test_peg_generator. (#31015)
Disable compiler optimization within test_peg_generator.

This speed up test_peg_generator by always disabling compiler
optimizations by using -O0 or equivalent when the test is building its
own C extensions.

A build not using --with-pydebug in order to speed up test execution
winds up with this test taking a very long time as it would do
repeated compilation of parser C code using the same optimization
flags as CPython was built with.

This speeds the test up 6-8x on gps-raspbian.

Also incorporate's #31017's win32 conditional and flags.

Co-authored-by: Kumar Aditya kumaraditya303
2022-02-02 12:15:16 -08:00
Mark Shannon 0d05da1fbf
Add specialization stats for FOR_ITER. (GH-31079) 2022-02-02 15:56:47 +00:00
Mark Shannon 187930f74c
bpo-46072: Add some frame stats. (GH-31060) 2022-02-02 11:01:33 +00:00
Mark Shannon 48be46ec1f
bpo-46072: Add some object layout and allocation stats (GH-31051) 2022-02-01 15:05:18 +00:00
Steve Dower 45faf151c6
bpo-33125: Enables building traditional installer for Windows ARM64 (GH-30885)
Also makes a few general improvements to the build process and removes some dead code.
2022-01-28 16:48:06 +00:00
Mark Shannon 90ab138bbd
bpo-46072: Add simple stats for Python calls. (GH-30989) 2022-01-28 15:20:33 +00:00
Kumar Aditya 5a9e423473
bpo-46449: deepfreeze get_code() now returns strong ref (GH-30987) 2022-01-28 13:41:13 +01:00
Christian Heimes 26b0482393
bpo-46476: Simplify and fix _PyStaticCode_Dealloc (GH-30965) 2022-01-27 19:32:12 +00:00
Eric Snow 247480a21c
bpo-46541: Generate the global objects initializer. (gh-30941)
This change is a prerequisite for generating code for other global objects (like strings in gh-30928).

(We borrowed some code from Tools/scripts/deepfreeze.py.)

https://bugs.python.org/issue46541
2022-01-27 11:06:09 -07:00
Kumar Aditya c7f810b34d
bpo-46476: Fix memory leak in code objects generated by deepfreeze (GH-30853)
Add _Py_Deepfreeze_Fini() and _PyStaticCode_Dealloc() functions.
2022-01-27 14:03:47 +01:00
Christian Heimes d5fd438b38
bpo-40280: Get help() working and more (GH-30858) 2022-01-24 22:02:01 +01:00
Kumar Aditya ca78130d7e
bpo-46471: Use single byte singletons (GH-30781) 2022-01-23 09:45:39 -08:00
Christian Heimes c02e860ee7
bpo-40280: Misc fixes for wasm32-emscripten (GH-30722) 2022-01-20 18:56:33 +01:00
Kumar Aditya ef3ef6fa43
bpo-46429: Merge all deepfrozen files into one (GH-30572) 2022-01-20 08:38:39 -08:00
Kumar Aditya 194ecc6d44
bpo-46443: deepfreeze: use small ints and singleton zero bytes (GH-30715) 2022-01-19 22:13:21 -08:00
bneuburg c47c9e6589
bpo-44934: Add optional feature AppendPath to Windows MSI installer (GH-27889)
The option must be enabled from the command line
2022-01-18 00:18:44 +00:00
Christian Heimes ee1a8b336d
bpo-40280: Block more syscalls that are causing crashes in tests (GH-30601) 2022-01-14 18:48:44 +01:00
Christian Heimes a6ca8eee22
bpo-46315: Add ifdef HAVE_ feature checks for WASI compatibility (GH-30507) 2022-01-13 09:46:04 +01:00
Christian Heimes 43839ba438
bpo-40280: Add --with-emscripten-target to build for browser or node (GH-30552)
Co-authored-by: Ethan Smith <ethan@ethanhs.me>
2022-01-12 16:08:19 +01:00
Ethan Smith 5c66414b55
bpo-40280: Disable epoll_create in Emscripten config.site (GH-30494)
Co-authored-by: nick.pope@infogrid.io
2022-01-09 11:58:59 +01:00
Brandt Bucher 332e6b9725
bpo-45256: Don't track the exact depth of each `InterpreterFrame` (GH-30372) 2022-01-05 11:30:26 +00:00
Dennis Sweeney 7537f60087
bpo-45609: More specialization stats for STORE_SUBSCR (GH-30193) 2022-01-04 18:05:09 +00:00
Pablo Galindo Salgado dd6c35761a
bpo-46110: Restore commit e9898bf153
This restores commit e9898bf153 .
2022-01-03 19:54:06 +00:00
Pablo Galindo Salgado 9d35dedc5e
Revert "bpo-46110: Add a recursion check to avoid stack overflow in the PEG parser (GH-30177)" (GH-30363)
This reverts commit e9898bf153 temporarily as we want to confirm if this commit is the cause of a slowdown at startup time.
2022-01-03 18:29:18 +00:00
Hugo van Kemenade 2cf7d02b99
bpo-46178: Remove/rename redundant Travis CI code (#30309) 2021-12-30 16:16:27 -08:00
Dong-hee Na 196b53eb1e
bpo-45189: Drop the "list_frozen" command from _test_embed. (GH-30273) 2021-12-28 11:05:50 +09:00
Kumar Aditya fc54e722a2
bpo-46106: Update OpenSSL to 1.1.1m (GH-30211)
Co-authored-by: Ned Deily <nad@python.org>
2021-12-21 21:20:16 -05:00
Pablo Galindo Salgado e9898bf153
bpo-46110: Add a recursion check to avoid stack overflow in the PEG parser (GH-30177)
Co-authored-by: Batuhan Taskaya <isidentical@gmail.com>
2021-12-20 15:43:26 +00:00
Raymond Hettinger 29ea68bd1d
Revert "bpo-46131: add fastpath for PyFloat_Check() (GH-30200)" (GH-30208)
This reverts commit 2ef06d4125.
2021-12-19 15:05:46 -06:00
Matti Picus 2ef06d4125
bpo-46131: add fastpath for PyFloat_Check() (#30200) 2021-12-19 14:24:30 -06:00
Christian Heimes 0339434835
bpo-40280: Add Tools/wasm with helpers for cross building (GH-29984)
Co-authored-by: Ethan Smith <ethan@ethanhs.me>
Co-authored-by: Brett Cannon <brett@python.org>
2021-12-18 15:54:02 +01:00
Christian Heimes 2985feac4e
bpo-46114: Fix OpenSSL version check for 3.0.1 (GH-30170) 2021-12-17 16:17:32 +01:00
Mark Shannon efd6236d36
bpo-46072: Add top level stats struct (GH-30169) 2021-12-17 14:48:01 +00:00
Mark Shannon 4506bbede1
bpo-46072: Document --enable-stats option. (GH-30139) 2021-12-16 13:40:54 +00:00
Mark Shannon 342b93f9f2
bpo-46072: Add --with-pystats configure option to simplify gathering of VM stats (GH-30116)
* Simplify specialization stats collection macros.

* Add --enable-pystats option to configure.

* Update specialization summary script to handle larger number of kinds
2021-12-15 15:32:32 +00:00
Christian Heimes eb483c46d6
bpo-45949: Pure Python freeze module for cross builds (GH-29899) 2021-12-13 20:48:46 +01:00
Petr Viktorin 98e506ae8a
bpo-43795: Document stable_abi.txt format and contents (GH-29956)
Also mention that removals generally aren't allowed.
2021-12-11 00:34:31 +01:00
Christian Heimes 16638a4bdb
bpo-45654: No need to freeze types (GH-30028) 2021-12-10 19:09:09 +01:00
Serhiy Storchaka d8a464ef03
Remove an outdated comment. (GH-30010) 2021-12-09 20:44:36 +02:00
Kumar Aditya 44b0e76f2a
bpo-45654: Freeze the runpy module and stuff it imports (GH-29903) 2021-12-09 08:51:09 -08:00
Mark Shannon 8319114fee
bpo-45947: Place dict and values pointer at fixed (negative) offset just before GC header. (GH-29879)
* Place __dict__ immediately before GC header for plain Python objects.

* Fix up lazy dict creation logic to use managed dict pointers.

* Manage values pointer, placing them directly before managed dict pointers.

* Convert hint-based load/store attr specialization target managed dict classes.

* Specialize LOAD_METHOD for managed dict objects.

* Remove unsafe _PyObject_GC_Calloc function.

* Remove unsafe _PyObject_GC_Malloc() function.

* Add comment explaning use of Py_TPFLAGS_MANAGED_DICT.
2021-12-07 16:02:53 +00:00
Crowthebird d9301703fb
bpo-45816: Support building with VS 2022 (v143 toolset) on Windows (GH-29577) 2021-12-03 19:47:38 +00:00
Christian Heimes 84ca1232b0
bpo-45950: Introduce Bootstrap Python again (#29859)
The build system now uses a :program:`_bootstrap_python` interpreter for
freezing and deepfreezing again. To speed up build process the build tools
:program:`_bootstrap_python` and :program:`_freeze_module` are no longer
build with LTO.

Cross building depends on a build Python interpreter, which must have same
version and bytecode as target host Python.
2021-12-03 16:01:11 +01:00
Eric Snow ee94aa0850
bpo-45952: Get the C analyzer tool working again. (gh-29882)
There wasn't much that needed to be done. Mostly it was just a few new files that got added.

https://bugs.python.org/issue45952
2021-12-01 11:20:20 -07:00
Inada Naoki 9a5dec4e97
Fix EncodingWarning in Tools/freeze/test/freeze.py (GH-29742) 2021-11-30 13:16:40 +09:00
Kumar Aditya b0b10e146b
bpo-45019: Cleanup module freezing and deepfreeze (#29772) 2021-11-26 08:50:54 -08:00
Victor Stinner 253b7a0a9f
bpo-45866: pegen strips directory of "generated from" header (GH-29777)
"make regen-all" now produces the same output when run from a
directory other than the source tree: when building Python out of the
source tree.
2021-11-26 11:50:34 +01:00
Kumar Aditya 256f9c4588
Fix typo in freeze_modules (#29744) 2021-11-24 11:12:03 -08:00
Christian Heimes 5c4b19ec49
bpo-45020: Fix strict-prototypes warning (GH-29755) 2021-11-24 20:01:39 +01:00
Christian Heimes dd8ce9ea8d
bpo-45886: Allow overriding freeze command for cross compiling (GH-29735) 2021-11-24 08:07:15 +01:00
Christian Heimes f840398a5f
bpo-45873: Restore Python 3.6 compatibility (GH-29730)
Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>
2021-11-23 21:36:40 +01:00
Guido van Rossum 5be98e57b3
bpo-45873: Get rid of bootstrap_python (#29717)
Instead we use $(PYTHON_FOR_REGEN) .../deepfreeze.py with the
frozen .h file as input, as we did for Windows in bpo-45850.

We also get rid of the code that generates the .h files
when make regen-frozen is run (i.e., .../make_frozen.py),
and the MANIFEST file.

Restore Python 3.8 and 3.9 as Windows host Python again

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
2021-11-23 08:56:06 -08:00
Eric Snow 8ed1495ad9
bpo-45783: Preserve file moves and deletions in the tests for the freeze tool. (GH-29527)
Use shutil.copytree rather than Git, which might be missing (or configured
differently) when testing Python built from a source release.
2021-11-23 14:43:40 +01:00
Neil Schemenauer 457e6a6e96
bpo-45561: Run smelly.py tool from $(srcdir) (GH-29138) 2021-11-23 08:51:02 +01:00