We tried this before with a dict and for all interned strings. That ran into problems due to interpreter isolation. However, exclusively using a per-interpreter cache caused some inconsistency that can eliminate the benefit of interning. Here we circle back to using a global cache, but only for statically allocated strings. We also use a more-basic _Py_hashtable_t for that global cache instead of a dict.
Ideally we would only have the global cache, but the optional isolation of each interpreter's allocator means that a non-static string object must not outlive its interpreter. Thus we would have to store a copy of each such interned string in the global cache, tied to the main interpreter.
The two tests are crashing periodically in CI and on buildbots. I suspect the problem is in the _xxsubinterpreters module.
Regardless, I'm disabling the tests temporarily, to reduce the noise as we approach 3.12rc1. I'll be investigating the crashes separately.
Adding the `rtype_cache` to the `warnings.warn` message improves the
previous, somewhat vague message from
```
/Users/username/cpython/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown
```
to
```
/Users/username/cpython/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown: {'/mp-yor5cvj8', '/mp-10jx8eqr', '/mp-eobsx9tt', '/mp-0lml23vl', '/mp-9dgtsa_m', '/mp-frntyv4s'}
```
---------
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
test_logging: Fix test_udp_reconnection() by increasing the timeout
from 100 ms to 5 minutes (LONG_TIMEOUT).
Replace also blocking wait() with wait(LONG_TIMEOUT) in
test_output() to prevent the test to hang.
Instead of hacking into the Clinic class, use the Argument Clinic tool
to run the ClinicExternalTest test suite.
Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
Move private _PyMem functions to the internal C API (pycore_pymem.h):
* _PyMem_GetCurrentAllocatorName()
* _PyMem_RawStrdup()
* _PyMem_RawWcsdup()
* _PyMem_Strdup()
No longer export these functions.
Move pymem_getallocatorsname() function from _testcapi to _testinternalcapi,
since the API moved to the internal C API.
This mostly extracts a whole bunch of stuff out of generate_cases.py into separate files, but there are a few other things going on here.
- analysis.py: `Analyzer` etc.
- instructions.py: `Instruction` etc.
- flags.py: `InstructionFlags`, `variable_used`, `variable_used_unspecialized`
- formatting.py: `Formatter` etc.
- Rename parser.py to parsing.py, to avoid conflict with stdlib parser.py
- Blackify most things
- Fix most mypy errors
- Remove output filenames from Generator state, add them to `write_instructions()` etc.
- Fix unit tests
Move private debug _PyObject functions to the internal C API
(pycore_object.h):
* _PyDebugAllocatorStats()
* _PyObject_CheckConsistency()
* _PyObject_DebugTypeStats()
* _PyObject_IsFreed()
No longer export most of these functions, except of
_PyObject_IsFreed().
Move test functions using _PyObject_IsFreed() from _testcapi to
_testinternalcapi. check_pyobject_is_freed() test no longer catch
_testcapi.error: the tested function cannot raise _testcapi.error.
Use lowercase `mail from` and `rcpt to` in `smtplib.SMTP`
SMTP commands are case-insensitive. `smtplib` uses lowercase commands,
however it writes `mail FROM` and `rcpt TO`, lacking consistency.
There are 3 paths to use `locale` argument in
`calendar.Locale{Text|HTML}Calendar.__init__(..., locale=None)`:
(1) `locale=None` -- denotes the "default locale"[1]
(2) `locale=""` -- denotes the native environment
(3) `locale=other_valid_locale` -- denotes a custom locale
So far case (2) is covered and case (1) is in 78935daf5a (same branch).
This commit adds a remaining case (3).
[1] In the current implementation, this translates into the following
approach:
GET current locale
IF current locale == "C" THEN
SET current locale TO ""
GET current locale
ENDIF
* Remove unreachable code (and increase test coverage)
This condition cannot be true. `_locale.setlocale()` from the C module
raises `locale.Error` instead of returning `None` for
`different_locale.__enter__` (where `self.oldlocale` is set).
* Expand the try clause to calls to `LocaleTextCalendar.formatmonthname()`.
This method temporarily changes the current locale to the given locale,
so `_locale.setlocale()` may raise `local.Error`.
Co-authored-by: Rohit Mediratta <rohitm@gmail.com>
Co-authored-by: Jessica McKellar <jesstess@mit.edu>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>
Add test for the 'destination <name> clear' command,
and the 'destination' directive in general.
Fix two bugs in 'destination <name> clear' command:
1. The text attribute of the allocator is called 'text', not '_text'
2. Return after processing the 'clear' command,
instead of proceeding directly to the fail().
* Add PyDict_GetItemRef() and PyDict_GetItemStringRef() functions.
Add these functions to the stable ABI version 3.13.
* Add unit tests on the PyDict C API in test_capi.
threadingmock: Improve test suite to avoid race conditions
Simplify tests and split them into multiple tests to prevent assertions
from triggering race conditions.
Additionally, we rely on calling the mocks without delay to validate the
functionality of matching calls.
These aren't automatically translated because (ironically)
they are macros deferring to POP_JUMP_IF_{TRUE,FALSE},
which are not viable uops (being manually translated).
The hack is that we emit IS_NONE and then set opcode and
jump to the POP_JUMP_IF_{TRUE,FALSE} translation code.
The Tier 2 opcode _IS_ITER_EXHAUSTED_LIST (and _TUPLE)
didn't set it->it_seq to NULL, causing a subtle bug
that resulted in test_exhausted_iterator in list_tests.py
to fail when running all tests with -Xuops.
The bug was introduced in gh-106696.
Added this as an explicit test.
Also fixed the dependencies for ceval.o -- it depends on executor_cases.c.h.
During superblock generation, a JUMP_BACKWARD instruction is translated to either a JUMP_TO_TOP micro-op (when the target of the jump is exactly the beginning of the superblock, closing the loop), or a SAVE_IP + EXIT_TRACE pair, when the jump goes elsewhere.
The new JUMP_TO_TOP instruction includes a CHECK_EVAL_BREAKER() call, so a closed loop can still be interrupted.
gh-86618 assumed a-b-c = a-(b+c) = a-d where d = b+d.
For floats 2.0, 1.0, and 0.9999999999999999, this assumption
is false. The net change of 1.1102230246251565e-16 to 0.0
results in division by 0. Revert the replacement. Add test.
* Convert PyObject_DelAttr() and PyObject_DelAttrString() macros to
functions.
* Add PyObject_DelAttr() and PyObject_DelAttrString() functions to
the stable ABI.
* Replace PyObject_SetAttr(obj, name, NULL) with
PyObject_DelAttr(obj, name).
* lambda has a name of __none__, but no async lambda so this branch is not needed
* _get_signature_object only returns None for bound builtins. There are no async builtins so this branch isn't needed
* Exclude a couple of methods from coverage checking in the downstream rolling backport of mock
- Hand-written uops JUMP_IF_{TRUE,FALSE}.
These peek at the top of the stack.
The jump target (in superblock space) is absolute.
- Hand-written translation for POP_JUMP_IF_{TRUE,FALSE},
assuming the jump is unlikely.
Once we implement jump-likelihood profiling,
we can implement the jump-unlikely case (in another PR).
- Tests (including some test cleanup).
- Improvements to len(ex) and ex[i] to expose the whole trace.
Detect email address parsing errors and return empty tuple to indicate the parsing error (old API). This fixes or at least ameliorates CVE-2023-27043.
---------
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Fixes a hang in multiprocessing process pool executor when a child process crashes and code could otherwise block on writing to the pipe. See GH-94777 for more details.
Reduce test noise by fixing or catching and testing stderr messages from individual tests.
test_cmd_line_script.test_script_as_dev_fd calls spawn_python and hence subprocess.Popen with incompatible arguments. On POSIX, pass_fds forces close_fds to be True (subprocess.py line 848). Correct the call.
test_uuid.test_cli_namespace_required_for_uuid3: when the namespace is omitted, uuid.main calls argparse.Argument_Parser.error, which prints to stderr before calling sys.exit, which raises SystemExit. Unittest assertRaises catches the exception but not the previous output. Catch the output and test it.
test_warnings.test_catchwarnings_with_simplefilter_error similarly prints before raising. Catch the output and test it.
---------
Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
threadingmock: Remove unused branch for `timeout`
This is no longer needed as the mock does not hold a "timeout"
parameter, the timeout is stored in `_mock_wait_timeout`.
Some items remained uninitialized if _sre.template() was called with invalid
indices. Then attempt to clear them in the destructor led to dereferencing
of uninitialized pointer.
This adds several of unspecialized opcodes to superblocks:
TO_BOOL, BINARY_SUBSCR, STORE_SUBSCR,
UNPACK_SEQUENCE, LOAD_GLOBAL, LOAD_ATTR,
COMPARE_OP, BINARY_OP.
While we may not want that eventually, for now this helps finding bugs.
There is a rudimentary test checking for UNPACK_SEQUENCE.
Once we're ready to undo this, that would be simple:
just replace the call to variable_used_unspecialized
with a call to variable_used (as shown in a comment).
Or add individual opcdes to FORBIDDEN_NAMES_IN_UOPS.
Prevent `multiprocessing.spawn` from failing to *import* in environments
where `sys.executable` is `None`. This regressed in 3.11 with the addition
of support for path-like objects in multiprocessing.
Adds a test decorator to have tests only run when part of test_multiprocessing_spawn to `_test_multiprocessing.py` so we can start to avoid re-running the same not-global-state specific test in all 3 modes when there is no need.
The uops test wasn't testing anything by default,
and was failing when run with -Xuops.
Made the two executor-related context managers global,
so TestUops can use them (notably `with temporary_optimizer(opt)`).
Made clear_executor() a little more thorough.
Fixed a crash upon finalizing a uop optimizer,
by adding a `tp_dealloc` handler.
Mark `testthreadingmock.py` with `threading_helper.requires_working_threading`.
Also add longer delays to reduce the change of a race conditions on the tests that validate short timeouts.
mock: Rename `wait_until_any_call` to `wait_until_any_call_with`
Rename the method to be more explicit that it expects the args and
kwargs to wait for.
Remove private _PyUnicode_TransformDecimalAndSpaceToASCII() and other
private _PyUnicode C API functions: move them to the internal C API
(pycore_unicodeobject.h). No longer most of these functions.
Replace _testcapi.unicode_transformdecimalandspacetoascii() with
_testinternal._PyUnicode_TransformDecimalAndSpaceToASCII().
We match paths using the `_lines` attribute, which is derived from the
path's string representation. The bug arises because an empty path's string
representation is `'.'` (not `''`), which is matched by the `'*'` wildcard.
mock: Add `ThreadingMock` class
Add a new class that allows to wait for a call to happen by using
`Event` objects. This mock class can be used to test and validate
expectations of multithreading code.
It uses two attributes for events to distinguish calls with any argument
and calls with specific arguments.
The calls with specific arguments need a lock to prevent two calls in
parallel from creating the same event twice.
The timeout is configured at class and constructor level to allow users
to set a timeout, we considered passing it as an argument to the
function but it could collide with a function parameter. Alternatively
we also considered passing it as positional only but from an API
caller perspective it was unclear what the first number meant on the
function call, think `mock.wait_until_called(1, "arg1", "arg2")`, where
1 is the timeout.
Lastly we also considered adding the new attributes to magic mock
directly rather than having a custom mock class for multi threading
scenarios, but we preferred to have specialised class that can be
composed if necessary. Additionally, having added it to `MagicMock`
directly would have resulted in `AsyncMock` having this logic, which
would not work as expected, since when if user "waits" on a
coroutine does not have the same meaning as waiting on a standard
call.
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* Remove private _PyTraceMalloc C API functions: move them to the
internal C API.
* Don't export most of these functions anymore, but still export
_PyTraceMalloc_GetTraceback() used by tests.
* Rename Include/tracemalloc.h to Include/cpython/tracemalloc.h
If the output arg to Helper() is a stream rather than the default None, which means 'page to stdout', the ImportError from pydoc.resolve is currently not caught in pydoc.doc. The same error is caught when output is None.
---------
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Remove the "cpython/pytime.h" header file: it only contained private
functions. Move functions to the internal pycore_time.h header file.
Move tests from _testcapi to _testinternalcapi. Rename also test
methods to have the same name than tested C functions.
No longer export these functions:
* _PyTime_Add()
* _PyTime_As100Nanoseconds()
* _PyTime_FromMicrosecondsClamp()
* _PyTime_FromTimespec()
* _PyTime_FromTimeval()
* _PyTime_GetPerfCounterWithInfo()
* _PyTime_MulDiv()
Make assertions about the `st_mode`, `st_ino` and `st_dev` attributes of
the stat results from two files and a directory, rather than checking if
`chmod()` affects `st_mode` (which is already tested elsewhere).
- Split out dedicated test for unbuffered `open()`
- Split out dedicated test for `is_mount()` at the filesystem root
- Avoid `os.stat()` when checking that empty paths point to '.'
- Remove unnecessary `rmtree()` call
- Remove unused `assertSame()` method
Charlie Zhao
Inada Naoki
James Cave
Alex Waygood
Tom Niget
Felipe A. Hernandez
Eric Snow
Mark Shannon
Irit Katriel
János Kukovecz
Pablo Galindo Salgado
Erlend E. Aasland
Nikita Sobolev
shailshouryya
Victor Stinner
Paul Moore
Carl Meyer
Tian Gao
Guido van Rossum
Dong-hee Na
Tomas R
htsedebenham
Gilles Bassière
J. Nick Koston
Jochem Boersma
Matthieu Caneill
Bart Skowron
James Hilton-Balfe
Gregory P. Smith
Brandt Bucher
Pieter Eendebak
Barney Gale
Yonatan Bitton
Serhiy Storchaka
Mario Corchero
Grigoriev Semyon
Jason R. Coombs
Chris Brett
CF Bolz-Tereick
Jim Porter
Dennis Sweeney
Prince Roshan
Kirill Podoprigora
Jelle Zijlstra
Terry Jan Reedy
Ethan Furman
Chris Withers
Thomas Dwyer
Louis Paulot
Ijtaba Hussain
Hugo van Kemenade
Radislav Chugunov
Desmond Cheong
Andrew Geng
Dustin Rodrigues
Ariel Eizenberg
JosephSBoyle
Lysandros Nikolaou