In translate(), generate unique group names across calls.
The restores the undocumented ability to get a valid regexp
by joining multiple translate() results via `|`.
With the new parser, the error message contains always the trailing
newlines, causing the comparison of the repr of the error messages
in codeop to fail. This commit makes the new parser mirror the old parser's
behaviour regarding trailing newlines.
This is for the C generator:
- Disallow rule and variable names starting with `_`
- Rename most local variable names generated by the parser to start with `_`
Exceptions:
- Renaming `p` to `_p` will be a separate PR
- There are still some names that might clash, e.g.
- anything starting with `Py`
- C reserved words (`if` etc.)
- Macros like `EXTRA` and `CHECK`
* bpo-39791: Update importlib.resources to support files() API (importlib_resources 1.5).
* 📜🤖 Added by blurb_it.
* Add some documentation about the new objects added.
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
(Note: PEP 554 is not accepted and the implementation in the code base is a private one for use in the test suite.)
If code running in a subinterpreter raises an uncaught exception then the "run" call in the calling interpreter fails. A RunFailedError is raised there that summarizes the original exception as a string. The actual exception type, __cause__, __context__, state, etc. are all discarded. This turned out to be functionally insufficient in practice. There is a more helpful solution (and PEP 554 has been updated appropriately).
This change adds the exception propagation behavior described in PEP 554 to the _xxsubinterpreters module. With this change a copy of the original exception is set to __cause__ on the RunFailedError. For now we are using "pickle", which preserves the exception's state. We also preserve the original __cause__, __context__, and __traceback__ (since "pickle" does not preserve those).
https://bugs.python.org/issue32604
Module C state is now accessible from C-defined heap type methods (PEP 573).
Patch by Marcel Plch and Petr Viktorin.
Co-authored-by: Marcel Plch <mplch@redhat.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
When parsing something like `f(g()=2)`, where the name of a default arg
is not a NAME, but an arbitrary expression, a specialised error message
is emitted.
When parsing a string with an invalid escape, the old parser used to
point to the beginning of the invalid string. This commit changes the new
parser to match that behaviour, since it's currently pointing to the
end of the string (or to be more precise, to the beginning of the next
token).
Make the design more object-oriented.
Split _GenericAlias on two almost independent classes: for special
generic aliases like List and for parametrized generic aliases like List[int].
Add specialized subclasses for Callable, Callable[...], Tuple and Union[...].
_PyErr_ChainExceptions() now ensures that the first parameter is an
exception type, as done by _PyErr_SetObject().
* The following function now check PyExceptionInstance_Check() in an
assertion using a new _PyBaseExceptionObject_cast() helper
function:
* PyException_GetTraceback(), PyException_SetTraceback()
* PyException_GetCause(), PyException_SetCause()
* PyException_GetContext(), PyException_SetContext()
* PyExceptionClass_Name() now checks PyExceptionClass_Check() with an
assertion.
* Remove XXX comment and add gi_exc_state variable to _gen_throw().
* Remove comment from test_generators
Due to backwards compatibility concerns regarding keywords immediately followed by a string without whitespace between them (like in `bg="#d00" if clear else"#fca"`) will fail to parse,
commit 41d5b94af4 has to be reverted.
When parsing things like `def f(*): pass` the old parser used to output `SyntaxError: named arguments must follow bare *`, which the new parser wasn't able to do.
This is a follow-up to GH-19823 that removes the check that the
exception value isn't NULL, prior to calling _PyErr_ChainExceptions().
This enables implicit exception chaining for gen.throw() in more
circumstances.
The commit also adds a test that a particular code snippet involving
gen.throw() doesn't crash. The test shows why the new
`gi_exc_state.exc_type != Py_None` check that was added is necessary.
Without the new check, the code snippet (as well as a number of other
tests) crashes on certain platforms (e.g. Fedora but not Mac).
Before this commit, if an exception was active inside a generator
when calling gen.throw(), that exception was lost (i.e. there was
no implicit exception chaining). This commit fixes that by
setting exc.__context__ when calling gen.throw(exc).
An isolated subinterpreter cannot spawn threads, spawn a child
process or call os.fork().
* Add private _Py_NewInterpreter(isolated_subinterpreter) function.
* Add isolated=True keyword-only parameter to
_xxsubinterpreters.create().
* Allow again os.fork() in "non-isolated" subinterpreters.
`ast.parse` and `compile` support a `feature_version` parameter that
tells the parser to parse the input string, as if it were written in
an older Python version.
The `feature_version` is propagated to the tokenizer, which uses it
to handle the three different stages of support for `async` and
`await`. Additionally, it disallows the following at parser level:
- The '@' operator in < 3.5
- Async functions in < 3.5
- Async comprehensions in < 3.6
- Underscores in numeric literals in < 3.6
- Await expression in < 3.5
- Variable annotations in < 3.6
- Async for-loops in < 3.5
- Async with-statements in < 3.5
- F-strings in < 3.6
Closeswe-like-parsers/cpython#124.
Before this commit, if an exception was active inside a generator
when calling gen.throw(), then that exception was lost (i.e. there
was no implicit exception chaining). This commit fixes that.
This implements full support for # type: <type> comments, # type: ignore <stuff> comments, and the func_type parsing mode for ast.parse() and compile().
Closes https://github.com/we-like-parsers/cpython/issues/95.
(For now, you need to use the master branch of mypy, since another issue unique to 3.9 had to be fixed there, and there's no mypy release yet.)
The only thing missing is `feature_version=N`, which is being tracked in https://github.com/we-like-parsers/cpython/issues/124.
Now that the default parser is the new PEG parser, ast.parse uses it, which means that we don't actually test something in test_peg_parser. This commit introduces a new keyword argument (`oldparser`) for `_peg_parser.parse_string` for specifying that a string needs to be parsed with the old parser. This keyword argument is used in the tests to actually compare the ASTs the new parser generates with those generated by the old parser.
Remove _random.Random.randbytes(): the C implementation of
randbytes(). Implement the method in Python to ease subclassing:
randbytes() now directly reuses getrandbits().
test.pythoninfo logs OpenSSL FIPS_mode() and Linux
/proc/sys/crypto/fips_enabled in a new "fips" section.
Co-Authored-By: Petr Viktorin <encukou@gmail.com>
After parsing is done in single statement mode, the tokenizer buffer has to be checked for additional lines and a `SyntaxError` must be raised, in case there are any.
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
This allows the caller to avoid creation of an exception when the channel is empty (just like `dict.get()` works). `ChannelEmptyError` is still raised if no default is provided.
Automerge-Triggered-By: @ericsnowcurrently
An E_EOF error was only being caught after the parser exited before this commit. There are some cases though, where the tokenizer returns ERRORTOKEN *and* has set an E_EOF error (like when EOF directly follows a line continuation character) which weren't correctly handled before.
Now only test_error_during_result_unpickle_in_result_handler()
captures and ignores sys.stderr in the test process.
Tools like test.bisect_cmd don't support subTest() but only
work with the granularity of one method.
Remove unused ExecutorDeadlockTest._sleep_id() method.
This commit also allows to pass flags to the new parser in all interfaces and fixes a bug in the parser generator that was causing to inline rules with actions, making them disappear.
* Move socket related functions from test.support to socket_helper.
* Import socket, nntplib and urllib.error lazily in transient_internet().
* Remove importing multiprocess.
Previously every test was building an extension module and
loading it into sys.modules. The tearDown function was thus
not able to clean up correctly, resulting in memory leaks.
With this commit, every test function now builds the extension
module and runs the actual test code in a new process
(using assert_python_ok), so that sys.modules stays intact
and no memory gets leaked.
Log "Warning -- ..." test warnings into sys.__stderr__ rather than
sys.stderr, to ensure to display them even if sys.stderr is captured.
test.libregrtest.utils.print_warning() now calls
test.support.print_warning().
* Rename PyConfig.use_peg to _use_peg_parser
* Document PyConfig._use_peg_parser and mark it a deprecated
* Mark -X oldparser option and PYTHONOLDPARSER env var as deprecated
in the documentation.
* Add use_old_parser() and skip_if_new_parser() to test.support
* Remove sys.flags.use_peg: use_old_parser() uses
_testinternalcapi.get_configs() instead.
* Enhance test_embed tests
* subprocess._args_from_interpreter_flags() copies -X oldparser
Added str.removeprefix and str.removesuffix methods and corresponding
bytes, bytearray, and collections.UserString methods to remove affixes
from a string if present. See PEP 616 for a full description.
The constant values of future flags in the __future__ module
is updated in order to prevent collision with compiler flags.
Previously PyCF_ALLOW_TOP_LEVEL_AWAIT was clashing
with CO_FUTURE_DIVISION.
Fix the Windows implementation of os.waitpid() for exit code
larger than "INT_MAX >> 8". The exit status is now interpreted as an
unsigned number.
os.waitstatus_to_exitcode() now accepts wait status larger than
INT_MAX.
It is possible to use either '-isysroot /some/path' (with a space) or
'-isysroot/some/path' (no space in between). Support both forms in
places where special handling of -isysroot is done, rather than just
the first form.
Co-authored-by: Ned Deily <nad@python.org>
Avoid linear runtime of ShareableList.__getitem__ and
ShareableList.__setitem__ by storing running allocated bytes in
ShareableList._allocated_bytes instead of the number of bytes for
a particular stored item.
Co-authored-by: Antoine Pitrou <antoine@python.org>
* Use ast module to find class definition
* Add NEWS entry
* Fix class with multiple children and move decorator code to the method
* Fix PR comments
1. Use node.decorator_list to select decorators
2. Remove unwanted variables in ClassVisitor
3. Simplify stack management as per review
* Add test for nested functions and async calls
* Fix pydoc test since comments are returned now correctly
* Set event loop policy as None to fix environment related change
* Refactor visit_AsyncFunctionDef and tests
* Refactor to use local variables and fix tests
* Add patch attribution
* Use self.addCleanup for asyncio
* Rename ClassVisitor to ClassFinder and fix asyncio cleanup
* Return first class inside conditional in case of multiple definitions. Remove decorator for class source.
* Add docstring to make the test correct
* Modify NEWS entry regarding decorators
* Return decorators too for bpo-15856
* Move ast and the class source code to top. Use proper Exception.
The uname binary on Android does not support -p [1]. Here is a sample
log:
```
0:06:03 load avg: 0.56 [254/421/8] test_platform failed -- running: test_asyncio (5 min 53 sec)
uname: Unknown option p (see "uname --help")
test test_platform failed -- Traceback (most recent call last):
File "/data/local/tmp/lib/python3.9/test/test_platform.py", line 170, in test_uname_processor
proc_res = subprocess.check_output(['uname', '-p'], text=True).strip()
File "/data/local/tmp/lib/python3.9/subprocess.py", line 420, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/data/local/tmp/lib/python3.9/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['uname', '-p']' returned non-zero exit status 1.
```
[1] https://android.googlesource.com/platform/external/toybox/+/refs/heads/master/toys/posix/uname.c
Automerge-Triggered-By: @jaraco
Add random.randbytes() function and random.Random.randbytes()
method to generate random bytes.
Modify secrets.token_bytes() to use SystemRandom.randbytes()
rather than calling directly os.urandom().
Rename also genrand_int32() to genrand_uint32(), since it returns an
unsigned 32-bit integer, not a signed integer.
The _random module is now built with Py_BUILD_CORE_MODULE defined.
Add a new internal pycore_byteswap.h header file with the following
functions:
* _Py_bswap16()
* _Py_bswap32()
* _Py_bswap64()
Use these functions in _ctypes, sha256 and sha512 modules,
and also use in the UTF-32 encoder.
sha256, sha512 and _ctypes modules are now built with the internal
C API.
* Replace flag-flip indirection with direct inspection
* Use any for simpler code
* Avoid flag flip and set results directly.
* Resolve processor in a single function.
* Extract processor handling into a namespace (class)
* Remove _syscmd_uname, unused
* Restore platform.processor behavior to match prior expectation (reliant on uname -p in a subprocess).
* Extract '_unknown_as_blank' function.
* Override uname_result to resolve the processor late.
* Add a test intended to capture the expected values from 'uname -p'
* Instead of trying to keep track of all of the possible outputs on different systems (probably a fool's errand), simply assert that except for the known platform variance, uname().processor matches the output of 'uname -p'
* Use a skipIf directive
* Use contextlib.suppress to suppress the error. Inline strip call.
* 📜🤖 Added by blurb_it.
* Remove use of contextlib.suppress (it would fail with NameError if it had any effect). Rely on _unknown_as_blank to replace unknown with blank.
Co-authored-by: blurb-it[bot] <blurb-it[bot]@users.noreply.github.com>
* bpo-35967: Make test more lenient to satisfy build bots.
* Update Lib/test/test_platform.py
Co-Authored-By: Kyle Stanley <aeros167@gmail.com>
* Expect '' for 'unknown'
Co-authored-by: Kyle Stanley <aeros167@gmail.com>
* Add a test intended to capture the expected values from 'uname -p'
* Instead of trying to keep track of all of the possible outputs on different systems (probably a fool's errand), simply assert that except for the known platform variance, uname().processor matches the output of 'uname -p'
* Use a skipIf directive
* Use contextlib.suppress to suppress the error. Inline strip call.
When there is a SyntaxError after reading the last input character from
the tokenizer and if no newline follows it, the error message used to be
`unexpected EOF while parsing`, which is wrong.
Add the encoding in ftplib.FTP and ftplib.FTP_TLS to the
constructor as keyword-only and change the default from "latin-1" to "utf-8"
to follow RFC 2640.
Fix a hang at fork in the logging module: the new private
_at_fork_reinit() method is now used to reinitialize locks at fork in
the child process.
The createLock() method is no longer used at fork.
Move the PyGC_Head structure and the following private macros to the
internal C API:
* _PyGCHead_FINALIZED()
* _PyGCHead_NEXT()
* _PyGCHead_PREV()
* _PyGCHead_SET_FINALIZED()
* _PyGCHead_SET_NEXT()
* _PyGCHead_SET_PREV()
* _PyGC_FINALIZED()
* _PyGC_PREV_MASK
* _PyGC_PREV_MASK_COLLECTING
* _PyGC_PREV_MASK_FINALIZED
* _PyGC_PREV_SHIFT
* _PyGC_SET_FINALIZED()
* _PyObject_GC_IS_TRACKED()
* _PyObject_GC_MAY_BE_TRACKED()
* _Py_AS_GC(o)
Keep the private _PyGC_FINALIZED() macro in the public C API for
backward compatibility with Python 3.8: make it an alias to the new
PyObject_GC_IsFinalized() function.
Move the SIZEOF_PYGC_HEAD constant from _testcapi module to
_testinternalcapi module.
* bpo-39011: Preserve line endings within attributes
Line endings within attributes were previously normalized to "\n" in Py3.7/3.8.
This patch removes that normalization, as line endings which were
replaced by entity numbers should be preserved in original form.
Add a private _at_fork_reinit() method to _thread.Lock,
_thread.RLock, threading.RLock and threading.Condition classes:
reinitialize the lock after fork in the child process; reset the lock
to the unlocked state.
Rename also the private _reset_internal_locks() method of
threading.Event to _at_fork_reinit().
* Add _PyThread_at_fork_reinit() private function. It is excluded
from the limited C API.
* threading.Thread._reset_internal_locks() now calls
_at_fork_reinit() on self._tstate_lock rather than creating a new
Python lock object.
This implements things like `list[int]`,
which returns an object of type `types.GenericAlias`.
This object mostly acts as a proxy for `list`,
but has attributes `__origin__` and `__args__`
that allow recovering the parts (with values `list` and `(int,)`.
There is also an approximate notion of type variables;
e.g. `list[T]` has a `__parameters__` attribute equal to `(T,)`.
Type variables are objects of type `typing.TypeVar`.
test_builtin.PtyTests now registers an handler for SIGHUP signal.
Closing the PTY file descriptor can emit a SIGHUP signal: just ignore
it.
run_child() now also closes the PTY file descriptor before waiting
for the process completition, otherwise the test hangs on AIX.
There are two copies of the grammar -- the one used by Python itself as
Grammar/Grammar, and the one used by lib2to3 which has necessarily diverged at
Lib/lib2to3/Grammar.txt because it needs to support older syntax an we want it
to be reasonable stable to avoid requiring fixer rewrites.
This brings suport for syntax like `if x:= foo():` to match what the live
Python grammar does.
This should've been added at the time of the walrus operator itself, but lib2to3 being
independent is often overlooked. So we do consider this a bugfix rather than enhancement.
CGIHTTPRequestHandler of http.server now logs the CGI script exit
code, rather than the CGI script exit status of os.waitpid().
For example, if the script is killed by signal 11, it now logs:
"CGI script exit code -11."
The AbstractBasicAuthHandler class of the urllib.request module uses
an inefficient regular expression which can be exploited by an
attacker to cause a denial of service. Fix the regex to prevent the
catastrophic backtracking. Vulnerability reported by Ben Caller
and Matt Schwager.
AbstractBasicAuthHandler of urllib.request now parses all
WWW-Authenticate HTTP headers and accepts multiple challenges per
header: use the realm of the first Basic challenge.
Co-Authored-By: Serhiy Storchaka <storchaka@gmail.com>
Add os.waitstatus_to_exitcode() function to convert a wait status to an
exitcode.
Suggest waitstatus_to_exitcode() usage in the documentation when
appropriate.
Use waitstatus_to_exitcode() in:
* multiprocessing, os, subprocess and _bootsubprocess modules;
* test.support.wait_process();
* setup.py: run_command();
* and many tests.
Running `test_socket` or anything that depends on it (like python -m
test.pythoninfo) crashes if IOCTL_VM_SOCKETS_GET_LOCAL_CID does not
exist in the socket module.
Automerge-Triggered-By: @pablogsal
* Add _waitstatus_to_exitcode() helper function to _bootsubprocess.
* Enhance check_output() error message if the command fails.
_bootsubprocess no longer handles WIFSTOPPED() case: it now raises a
ValueError.
* Rewrite test_thread.test_forkinthread() to use
support.wait_process() and wait for the child process in the main
thread, not in the spawned thread.
* test_threading now uses support.wait_process() and checks the child
process exit code to detect crashes.
* test_fork1: remove duplicated wait_impl() method: reuse
fork_wait.py implementation instead.
* Use exit code different than 0 to ensure that we executed the
expected code path.
Moreover, the following tests now check the child process exit code:
* test_os.PtyTests
* test_mailbox.test_lock_conflict()
* test_tempfile.test_process_awareness()
* test_uuid.testIssue8621()
* multiprocessing resource tracker tests
Replace statically allocated types with heap allocated types:
use PyType_FromSpec().
Add a module state to store the _abc_data_type.
Add traverse, clear and free functions to the module.
Change 0/1 assignments to 'executing', 'canceled', 'reading', 'endoffile'.
These are not used outside of pyshell. Other bools already use False/True.
Add comment about int needed for Windows call.
Remove self.more, unused in idlelib and code.InteractiveInterpreter.
The latter uses 'more' as a local.
Remove daemon threads from :mod:`concurrent.futures` by adding
an internal `threading._register_atexit()`, which calls registered functions
prior to joining all non-daemon threads. This allows for compatibility
with subinterpreters, which don't support daemon threads.
If fork was not called by a thread spawned by threading.Thread,
threading._after_fork() now creates a _MainThread instance for
_main_thread, instead of a _DummyThread instance.
This pull request fixes the newline conversion bug originally reported in bpo-1812. When that issue was originally submitted, the open builtin did not default to universal newline mode; now it does, which makes the issue fix simpler, since the only code path that needs to be changed is the one in doctest._load_testfile where the file is loaded from a package whose loader has a get_data method.
Remove two unused imports: _thread and _weakref. Avoid creating a new
winreg builtin module if it's already available in sys.modules.
The winreg module is now stored as "winreg" rather than "_winreg".
PyThreadState.frame is a borrowed reference, not a strong reference:
PyThreadState_Clear() must not call Py_CLEAR(tstate->frame).
Remove test_threading.test_warnings_at_exit(): we cannot warranty
that the Python thread state of daemon threads is cleared in a
reliable way during Python shutdown.
* Update ChainMap to include | and |=
Created __ior__, __or__ and __ror__ methods in ChainMap class.
* Update ACKS
* Update docs
* Update test_collections.py to include test_issue584().
Added testing for | and |= operators for ChainMap objects.
* Update test_union_operators
Renamed test_union operators, fixed errors and style problems raised by brandtbucher.
* Update test_union_operators in TestChainMap
Added testing for union operator between ChainMap and iterable of key-value pairs.
* Update test_union operators in test_collections.py
Gave more descriptive variable names and eliminated unnecessary tmp variable.
* Update test_union_operators in test_collections.py
Added cm3
* Check .maps rather than Chainmap equality.
* Add news entry
* Update Lib/test/test_collections.py
Co-Authored-By: Brandt Bucher <brandtbucher@gmail.com>
* Removed whitespace
* Added Guido's changes
* Fixed Docs
* Removed whitespace
Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
* Re-add removed classes Suite, slice, Param, AugLoad and AugStore.
* Add docstrings for dummy classes.
* Add docstrings for attribute aliases.
* Set __module__ to "ast" instead of "_ast".
* bpo-22490: Remove "__PYVENV_LAUNCHER__" from the shell environment on macOS
This changeset removes the environment varialbe "__PYVENV_LAUNCHER__"
during interpreter launch as it is only needed to communicate between
the stub executable in framework installs and the actual interpreter.
Leaving the environment variable present may lead to misbehaviour when
launching other scripts.
* Actually commit the changes for issue 22490...
* Correct typo
Co-Authored-By: Nicola Soranzo <nicola.soranzo@gmail.com>
* Run make patchcheck
Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>
Do not apply AST-based optimizations if 'from __future__ import annotations' is used in order to
prevent information lost in the final version of the annotations.
Serhiy Storchaka
Batuhan Taskaya
Tim Peters
Raymond Hettinger
Shantanu
Brad Solomon
Pablo Galindo
Jason R. Coombs
Zackery Spytz
Eric Snow
Petr Viktorin
Lysandros Nikolaou
Victor Stinner
Dennis Sweeney
Curtis Bucher
Hai Shi
Dong-hee Na
Chris Jerdonek
Sander
Furkan Önder
Noah Doersing
Guido van Rossum
Jelle Zijlstra
lrjball
Vlad Serebrennikov
karl ding
Mark Shannon
Lewis Gaul
Miro Hrončok
Karthikeyan Singaravelan
Ethan Onstott
Łukasz Langa
Nickolena Fisher
Carl Meyer
Anthony Sottile
Steve Dower
Joshua Root
HongWeipeng
Antoine Pitrou
Barry
Galden
Tim Hoffmann
Thomas Krennwallner
Kyle Stanley
Hakan Çelik
Chih-Hsuan Yen
Barney Gale
Mariusz Felisiak
Inada Naoki
Russell Davis
Ammar Askar
Ethan Smith
Sebastian Pedersen
Oren Milman
Alex Itkes
mefistotelis
pxinwr
Zachary Ware
Tal Einat
Michael Felt
Tim Hatch
Derek Keeler
Ashwin Ramaswami
Terry Jan Reedy
Peter Donis
Furkan Önder
Ram Rachum
Roman Yurchak
Ronald Oussoren
Batuhan Taşkaya
Jens Reidel