* pycore_intrinsics.h does nothing if included twice
(add #ifndef and #define).
* Update Tools/cases_generator/generate_cases.py to generate the
Py_BUILD_CORE test.
* _bz2, _lzma, _opcode and zlib extensions now define the
Py_BUILD_CORE_MODULE macro to use internal headers
(pycore_code.h, pycore_intrinsics.h and pycore_blocks_output_buffer.h).
It is now possible to deprecate passing keyword arguments for
keyword-or-positional parameters with Argument Clinic, using the new
'/ [from X.Y]' syntax.
(To be read as "positional-only from Python version X.Y")
Co-authored-by: Erlend E. Aasland <erlend@python.org>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
This finishes the work begun in gh-107760. When, while projecting a superblock, we encounter a call to a short, simple function, the superblock will now enter the function using `_PUSH_FRAME`, continue through it, and leave it using `_POP_FRAME`, and then continue through the original code. Multiple frame pushes and pops are even possible. It is also possible to stop appending to the superblock in the middle of a called function, when running out of space or encountering an unsupported bytecode.
'not hasattr(parent, "classes")' is always false, since 'parent' is an
instance of either the Module, Class, or Clinic classes, and all of
them has a "classes" attribute.
* Split `CALL_PY_EXACT_ARGS` into uops
This is only the first step for doing `CALL` in Tier 2.
The next step involves tracing into the called code object and back.
After that we'll have to do the remaining `CALL` specialization.
Finally we'll have to deal with `KW_NAMES`.
Note: this moves setting `frame->return_offset` directly in front of
`DISPATCH_INLINED()`, to make it easier to move it into `_PUSH_FRAME`.
* Instead of calling get_identifiers_and_strings(), extract identifiers and strings from pycore_global_strings.h.
* Avoid ast.literal_eval(), it's very slow.
Move the "deprecated positinal" tests from clinic.test.c to
_testclinic.c. Mock PY_VERSION_HEX in order to prevent generated
compiler warnings/errors to trigger. Put clinic code for deprecated
positionals in Modules/clinic/_testclinic_depr_star.c.h for easy
inspection of the generated code.
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Extract helper methods for formatting the signature and parameter
sections, and clean up the remaining function body.
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
- Generalize the syntax for the type of a stack effect to allow a trailing `*`,
so we can declare something as e.g. `PyCodeObject *`.
- When generating assignments for stack effects,
the type of the value on the stack should be the default (i.e., `PyObject *`)
even when the variable copied to/from it has a different type,
so that an appropriate cast is generated
However, not when the variable is an array --
then the type is taken from the variable (as it is always `PyObject **`).
It is now possible to deprecate passing parameters positionally with
Argument Clinic, using the new '* [from X.Y]' syntax.
(To be read as "keyword-only from Python version X.Y")
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
This fixes two tiny defects in analysis.py that I didn't catch on time in #107564:
- `get_var_names` in `check_macro_consistency` should skip `UNUSED` names.
- Fix an occurrence of `is UNUSED` (should be `==`).
Introducing a new file, stacking.py, that takes over several responsibilities related to symbolic evaluation of push/pop operations, with more generality.
- always wrap the offending line, token, or name in quotes
- in most cases, put the entire error message on one line
Added tests for uncovered branches that were touched by this PR.
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Introduce ClinicError, and use it in fail(). The CLI runs main(),
catches ClinicError, formats the error message, prints to stderr
and exits with an error.
As a side effect, this refactor greatly improves the accuracy of
reported line numbers in case of error.
Also, adapt the test suite to work with ClinicError.
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
The motivation for this change is to clean up the output_templates()
method a little bit, as it accounts for ~10% of the lines of code in
clinic.py; removing some clutter helps readability.
Previously, only function docstrings were checked for non-ASCII characters.
Also, improve the warn() message.
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Introduce docstring_append() helper, and use it for both parameter and
function docstrings. Remove docstring fixup from
do_post_block_processing_cleanup(); instead, make sure no fixup is needed.
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
- Use ArgumentParser.error() to handle CLI errors
- Put the entire CLI in main()
- Rework ClinicExternalTest to call main() instead of using subprocesses
Co-authored-by: AlexWaygood <alex.waygood@gmail.com>
The _xxsubinterpreters module should not rely on internal API. Some of the functions it uses were recently moved there however. Here we move them back (and expose them properly).
We tried this before with a dict and for all interned strings. That ran into problems due to interpreter isolation. However, exclusively using a per-interpreter cache caused some inconsistency that can eliminate the benefit of interning. Here we circle back to using a global cache, but only for statically allocated strings. We also use a more-basic _Py_hashtable_t for that global cache instead of a dict.
Ideally we would only have the global cache, but the optional isolation of each interpreter's allocator means that a non-static string object must not outlive its interpreter. Thus we would have to store a copy of each such interned string in the global cache, tied to the main interpreter.
This mostly extracts a whole bunch of stuff out of generate_cases.py into separate files, but there are a few other things going on here.
- analysis.py: `Analyzer` etc.
- instructions.py: `Instruction` etc.
- flags.py: `InstructionFlags`, `variable_used`, `variable_used_unspecialized`
- formatting.py: `Formatter` etc.
- Rename parser.py to parsing.py, to avoid conflict with stdlib parser.py
- Blackify most things
- Fix most mypy errors
- Remove output filenames from Generator state, add them to `write_instructions()` etc.
- Fix unit tests
Move the private _PyInterpreterID C API to the internal C API: add a
new pycore_interp_id.h header file.
Remove Include/interpreteridobject.h and
Include/cpython/interpreteridobject.h header files.
Add test for the 'destination <name> clear' command,
and the 'destination' directive in general.
Fix two bugs in 'destination <name> clear' command:
1. The text attribute of the allocator is called 'text', not '_text'
2. Return after processing the 'clear' command,
instead of proceeding directly to the fail().
This restores a corner case: when the generator is run with working directory set to Tools/cases_generator, the source filenames listed in the generated provenance header should be relative to the repo root directory.
By turning `assert(kwnames == NULL)` into a macro that is not in the "forbidden" list, many instructions that formerly were skipped because they contained such an assert (but no other mention of `kwnames`) are now supported in Tier 2. This covers 10 instructions in total (all specializations of `CALL` that invoke some C code):
- `CALL_NO_KW_TYPE_1`
- `CALL_NO_KW_STR_1`
- `CALL_NO_KW_TUPLE_1`
- `CALL_NO_KW_BUILTIN_O`
- `CALL_NO_KW_BUILTIN_FAST`
- `CALL_NO_KW_LEN`
- `CALL_NO_KW_ISINSTANCE`
- `CALL_NO_KW_METHOD_DESCRIPTOR_O`
- `CALL_NO_KW_METHOD_DESCRIPTOR_NOARGS`
- `CALL_NO_KW_METHOD_DESCRIPTOR_FAST`
Add various missing annotations in the following classes:
- BlockPrinter
- CConverter
- CLanguage
- FormatCounterFormatter
- Language
- _TextAccumulator
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
This moves EXIT_TRACE, SAVE_IP, JUMP_TO_TOP, and
_POP_JUMP_IF_{FALSE,TRUE} from ceval.c to bytecodes.c.
They are no less special than before, but this way
they are discoverable o the copy-and-patch tooling.
Fetch CONFIG_ARGS from the original source directory, instead of from
the copied source tree. When "make clean" is executed in the copied
source tree, the build directory is cleared and the configure argument
lookup fails. However, the original source directory still contains this
information.
During superblock generation, a JUMP_BACKWARD instruction is translated to either a JUMP_TO_TOP micro-op (when the target of the jump is exactly the beginning of the superblock, closing the loop), or a SAVE_IP + EXIT_TRACE pair, when the jump goes elsewhere.
The new JUMP_TO_TOP instruction includes a CHECK_EVAL_BREAKER() call, so a closed loop can still be interrupted.
- Hand-written uops JUMP_IF_{TRUE,FALSE}.
These peek at the top of the stack.
The jump target (in superblock space) is absolute.
- Hand-written translation for POP_JUMP_IF_{TRUE,FALSE},
assuming the jump is unlikely.
Once we implement jump-likelihood profiling,
we can implement the jump-unlikely case (in another PR).
- Tests (including some test cleanup).
- Improvements to len(ex) and ex[i] to expose the whole trace.
This adds several of unspecialized opcodes to superblocks:
TO_BOOL, BINARY_SUBSCR, STORE_SUBSCR,
UNPACK_SEQUENCE, LOAD_GLOBAL, LOAD_ATTR,
COMPARE_OP, BINARY_OP.
While we may not want that eventually, for now this helps finding bugs.
There is a rudimentary test checking for UNPACK_SEQUENCE.
Once we're ready to undo this, that would be simple:
just replace the call to variable_used_unspecialized
with a call to variable_used (as shown in a comment).
Or add individual opcdes to FORBIDDEN_NAMES_IN_UOPS.
Instead of special-casing specific instructions,
we add a few more special values to the 'size' field of expansions,
so in the future we can automatically handle
additional super-instructions in the generator.
- Tweak uops debugging output
- Fix the bug from gh-106290
- Rename `SET_IP` to `SAVE_IP` (per https://github.com/faster-cpython/ideas/issues/558)
- Add a `SAVE_IP` uop at the start of the trace (ditto)
- Allow `unbound_local_error`; this gives us uops for `LOAD_FAST_CHECK`, `LOAD_CLOSURE`, and `DELETE_FAST`
- Longer traces
- Support `STORE_FAST_LOAD_FAST`, `STORE_FAST_STORE_FAST`
- Add deps on pycore_uops.h to Makefile(.pre.in)
Annotate the following method signatures:
- state_dsl_start()
- state_parameter_docstring_start()
- state_parameters_start()
Inverting ignore_line() logic, add type hints (including type guard) to
it, and rename to valid_line().
Remove the "cpython/pytime.h" header file: it only contained private
functions. Move functions to the internal pycore_time.h header file.
Move tests from _testcapi to _testinternalcapi. Rename also test
methods to have the same name than tested C functions.
No longer export these functions:
* _PyTime_Add()
* _PyTime_As100Nanoseconds()
* _PyTime_FromMicrosecondsClamp()
* _PyTime_FromTimespec()
* _PyTime_FromTimeval()
* _PyTime_GetPerfCounterWithInfo()
* _PyTime_MulDiv()
- Establish global state struct
- Convert static types to heap types and add them to global state:
* PyDecContextManager_Type
* PyDecContext_Type
* PyDecSignalDictMixin_Type
* PyDec_Type
- Add to global state:
* PyDecSignalDict_Type
* DecimalTuple
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Co-authored-by: Erlend E. Aasland <erlend@python.org>
This produces longer traces (superblocks?).
Also improved debug output (uop names are now printed instead of numeric opcodes). This would be simpler if the numeric opcode values were generated by generate_cases.py, but that's another project.
Refactored some code in generate_cases.py so the essential algorithm for cache effects is only run once. (Deciding which effects are used and what the total cache size is, regardless of what's used.)
Added a new, experimental, tracing optimizer and interpreter (a.k.a. "tier 2"). This currently pessimizes, so don't use yet -- this is infrastructure so we can experiment with optimizing passes. To enable it, pass ``-Xuops`` or set ``PYTHONUOPS=1``. To get debug output, set ``PYTHONUOPSDEBUG=N`` where ``N`` is a debug level (0-4, where 0 is no debug output and 4 is excessively verbose).
All of this code is likely to change dramatically before the 3.13 feature freeze. But this is a first step.
* Add table describing possible executable classes for out-of-process debuggers.
* Remove shim code object creation code as it is no longer needed.
* Make lltrace a bit more robust w.r.t. non-standard frames.
Remove compatibility code for Python 2 and early Python 3 versions.
* Remove os_fsencode() reimplementation: use os.fsencode() directly.
os.fsencode() was added to Python 3.2.
* Remove references to Python 2 and "Python 3": just say "Python".
* Remove outdated u'' string format: use '' instead.
This fixes a race during import. The existing _PyRuntimeState.imports.pkgcontext is shared between interpreters, and occasionally this would cause a crash when multiple interpreters were importing extensions modules at the same time. To solve this we add a thread-local variable for the value. We also leave the existing state (and infrequent race) in place for platforms that do not support thread-local variables.
For a while now, pending calls only run in the main thread (in the main interpreter). This PR changes things to allow any thread run a pending call, unless the pending call was explicitly added for the main thread to run.
The _xxsubinterpreters module was meant to only use public API. Some internal C-API usage snuck in over the last few years (e.g. gh-28969). This fixes that.
When I added the relevant condition to type_ready_set_bases() in gh-103912, I had missed that the function also sets tp_base and ob_type (if necessary). That led to problems for third-party static types.
We fix that here, by making those extra operations distinct and by adjusting the condition to be more specific.
Upgrade builds to OpenSSL 1.1.1u.
This OpenSSL version addresses a pile if less-urgent CVEs since 1.1.1t.
The Mac/BuildScript/build-installer.py was already updated.
Also updates _ssl_data_111.h from OpenSSL 1.1.1u, _ssl_data_300.h from 3.0.9, and adds a new _ssl_data_31.h file from 3.1.1 along with the ssl.c code to use it.
Manual edits to the _ssl_data_300.h file prevent it from removing any existing definitions in case those exist in some peoples builds and were important (avoiding regressions during backporting).
backports of this prior to 3.12 will not include the openssl 3.1 header.
* refcounts.dat:
* Remove Py_UNICODE functions.
* Replace Py_UNICODE argument type with wchar_t.
* _PyUnicode_ToLowercase(), _PyUnicode_ToUppercase(),
_PyUnicode_ToTitlecase() are no longer deprecated in comments.
It's no longer needed since they now use Py_UCS4 type, rather than
the deprecated Py_UNICODE type.
* gdb: Remove unused char_width() method.
* socket_helper.transient_internet() no longer imports nntplib to
catch nntplib.NNTPTemporaryError.
* ssltests.py no longer runs test_nntplib.
* "make quicktest" no longer runs test_nntplib.
* WASM: remove nntplib from OMIT_NETWORKING_FILES.
* Remove mentions to nntplib in the email documentation.
- Make some string interpolations more readable using f-strings or
explicit parametrisation
- Remove unneeded open() mode specifiers
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
The following local variables were assigned but never used:
- line 551: result
- line 1341: groups
- line 1431: default_return_converter
- line 1529: ignore_self
- line 1809: input_checksum
- line 4224: new'
---
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
For code readability. Instances of `builtins.dict` have been ordered since 3.6, and have been guaranteed by the language to be ordered since Python 3.7. Argument Clinic now requires Python 3.10+.
Annotate the following:
- methods of class Class
- methods of class Module
- methods of class PythonParser
- function compute_checksum()
- function parse_file()
- global variable unsupported_special_methods
- Convert `unspecified` and `unknown` to be members of a `Sentinels` enum, rather than instances of bespoke classes.
- An enum feels more idiomatic here, and works better with type checkers.
- Convert some `==` and `!=` checks for these values to identity checks, which are more idiomatic with sentinels.
- _Don't_ do the same for `Null`, as this needs to be a distinct type due to its usage in `clinic.py`.
- Use `object` as the annotation for `default` across `clinic.py`. `default` can be literally any object, so `object` is the correct annotation here.
---
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Introduce TypeSet, and use it to annotate the 'accept' keyword of
various C converters. Also add some missing return annotations for
converter init functions.
* Add basic mypy workflow to CI
* Make the type check pass
---------
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
When monitoring LINE events, instrument all instructions that can have a predecessor on a different line.
Then check that the a new line has been hit in the instrumentation code.
This brings the behavior closer to that of 3.11, simplifying implementation and porting of tools.
Use the unused keyword param in the converter to explicitly
mark an argument as unused:
/*[clinic input]
SomeBaseClass.stubmethod
flag: bool(unused=True)
[clinic start generated code]*/
Replaces our built-in SHA3 implementation with a verified one from the HACL* project.
This implementation is used when OpenSSL does not provide SHA3 or is not present.
3.11 shiped with a very slow tiny sha3 implementation to get off of the <=3.10 reference implementation that wound up having serious bugs. This brings us back to a reasonably performing built-in implementation consistent with what we've just replaced our other guaranteed available standard hash algorithms with: code from the HACL* project.
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Victor Stinner
Serhiy Storchaka
Alex Waygood
Ned Deily
Dong-hee Na
Guido van Rossum
Erlend E. Aasland
Irit Katriel
Ken Jin
Martin DeMello
Carl Meyer
Mark Shannon
Charlie Zhao
Eric Snow
Kumar Aditya
Pablo Galindo Salgado
Brandt Bucher
Nikita Sobolev
Kevin Diem
Ijtaba Hussain
Eisuke Kawashima
dependabot[bot]
Kirill Podoprigora
YAMAMOTO Takashi
Brett Cannon
Gregory P. Smith
Lysandros Nikolaou
Steve Dower
Petr Viktorin
Zachary Ware
Erlend E. Aasland
gsallam
Jelle Zijlstra
Kumar Aditya
penguin_wwy
Hugo van Kemenade
Jonathan Protzenko