Skip test_freeze_simple_script() of test_tools.test_freeze if Python
is built with "./configure --enable-optimizations", which means with
Profile Guided Optimization (PGO): it just makes the test too slow.
The freeze tool is tested by many other CIs with other (faster)
compiler flags.
test.pythoninfo now gets also get_build_info() of
test.libregrtests.utils.
The filename was previously intentionally omitted from exception because
"it might confuse the user". Uncaught exceptions are not generally a
replacement for user-facing error messages, so obscuring this
information only has the effect of making the programmer's life more
difficult.
PyMutex is a one byte lock with fast, inlineable lock and unlock functions for the common uncontended case. The design is based on WebKit's WTF::Lock.
PyMutex is built using the _PyParkingLot APIs, which provides a cross-platform futex-like API (based on WebKit's WTF::ParkingLot). This internal API will be used for building other synchronization primitives used to implement PEP 703, such as one-time initialization and events.
This also includes tests and a mini benchmark in Tools/lockbench/lockbench.py to compare with the existing PyThread_type_lock.
Uncontended acquisition + release:
* Linux (x86-64): PyMutex: 11 ns, PyThread_type_lock: 44 ns
* macOS (arm64): PyMutex: 13 ns, PyThread_type_lock: 18 ns
* Windows (x86-64): PyMutex: 13 ns, PyThread_type_lock: 38 ns
PR Overview:
The primary purpose of this PR is to implement PyMutex, but there are a number of support pieces (described below).
* PyMutex: A 1-byte lock that doesn't require memory allocation to initialize and is generally faster than the existing PyThread_type_lock. The API is internal only for now.
* _PyParking_Lot: A futex-like API based on the API of the same name in WebKit. Used to implement PyMutex.
* _PyRawMutex: A word sized lock used to implement _PyParking_Lot.
* PyEvent: A one time event. This was used a bunch in the "nogil" fork and is useful for testing the PyMutex implementation, so I've included it as part of the PR.
* pycore_llist.h: Defines common operations on doubly-linked list. Not strictly necessary (could do the list operations manually), but they come up frequently in the "nogil" fork. ( Similar to https://man.freebsd.org/cgi/man.cgi?queue)
---------
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
On a Python built in debug mode, Py_DECREF() now calls
_Py_NegativeRefcount() if the object is a dangling pointer to
deallocated memory: memory filled with 0xDD "dead byte" by the debug
hook on memory allocators. The fix is to check the reference count
*before* checking for _Py_IsImmortal().
Add test_decref_freed_object() to test_capi.test_misc.
This feature is off by default via code but on by default via the CLI. The `.gitignore` file contains `*` which causes the entire directory to be ignored.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
libregrtest now decodes stdout of test worker processes with the
"backslashreplace" error handler to log corrupted stdout, instead of
failing with an error and not logging the stdout.
Builtin functions and methods that have non-representable signatures today
will have representable signatures yesterday, and they will become unusable
for testing this feature.
So we need to add special functions and methods to the _testcapi module
that always have non-representable signatures.
Fix test_gdb on Python built with LLVM clang 16 on Linux ppc64le (ex:
Fedora 38). Search patterns in gdb "bt" command output to detect
when gdb fails to retrieve the traceback. For example, skip a test if
"Backtrace stopped: frame did not save the PC" is found.
The test now calls free_tool_id() so it can be run multiple times in
the same process. For example, the following command no longer fails:
python -m test test_monitoring -R 3:3
On WASI platform, get_temp_dir() should behave differently since the
parent process is a WASI process and uses a different get_temp_dir()
path.
Fix also WorkerThread._runtest(): don't read JSON file if the worker
process exit code is non-zero.
On Emscripten and WASI, or if --python command line is used,
libregrtest now writes JSON into stdout, instead of using a name
file.
* Add JsonFileType.STDOUT.
* Remove JsonFileType.FILENAME.
* test.pythoninfo logs environment variables related to
cross-compilation and running Python on Emscripten/WASI.
On Emscripten and WASI platforms, or if --python command line option
is used, libregrtest now uses a filename for the JSON file.
Emscripten and WASI buildbot workers run the main test process with a
different Python (Linux) which spawns Emscripten/WASI processes using
the command specified in --python command line option. Passing a file
descriptor from the parent process to the child process doesn't work
in this case.
* Add JsonFile and JsonFileType classes
* Add RunTests.json_file_use_filename() method.
* Add a test in test_regrtest on the --python command line option.
* test_regrtest: add parallel=False parameter.
* Split long RunWorkers._runtest() function into sub-functions.
Fix Emscripten and WASI: start the test worker process in the Python
source code directory, where 'python.js' and 'python.wasm' can be
found. Then worker_process() changes to a temporary directory created
to run tests.
* create_worker_process() uses os_helper.SAVEDCWD as cwd.
* worker_process() uses get_temp_dir() as the parent directory for
get_work_dir().
* Don't use plural but singual for "test" in "Run 1 test ..."
message.
* Remove unused imports.
* Add WORK_DIR_PREFIX and WORKER_WORK_DIR_PREFIX constants.
Don't spawn more threads than the number of jobs: these worker
threads would never get anything to do.
* Add the number of tests in "Run ... tests in ..." message.
* Add RunTests.get_jobs() method.
* Add plural() function.
* count() uses f-string.
libregrtest now calls random.seed() before running each test file
when -r/--randomize command line option is used. Moreover, it's also
called in worker processes. It should help to make tests more
deterministic. Previously, it was only called once in the main
process before running all test files and it was not called in worker
processes.
* Convert some f-strings to regular strings in test_regrtest when
f-string is not needed.
* Remove unused all_methods variable from test_regrtest.
* Add RunTests members are now mandatory.
On Windows, skip the test if the current working directory and the
Python source code directory have different mount drives. It happens
if the temporary directory is on a different mount drive than the
Python source code.
Fix test_os.test_access_denied() when the TEMP environment variable
is equal to the current working directory. Run the test using a
different filename, since self.fname already exists in this case.
test_asyncio and test_compileall now clean up multiprocessing by
calling multiprocessing _cleanup_tests(): explicitly clean up
resources and stop background processes like the resource tracker.
* Rename SAVE_IP to _SET_IP
* Rename EXIT_TRACE to _EXIT_TRACE
* Rename SAVE_CURRENT_IP to _SAVE_CURRENT_IP
* Rename INSERT to _INSERT (This is for Ken Jin's abstract interpreter)
* Rename IS_NONE to _IS_NONE
* Rename JUMP_TO_TOP to _JUMP_TO_TOP
This adds a 16-bit inline cache entry to the conditional branch instructions POP_JUMP_IF_{FALSE,TRUE,NONE,NOT_NONE} and their instrumented variants, which is used to keep track of the branch direction.
Each time we encounter these instructions we shift the cache entry left by one and set the bottom bit to whether we jumped.
Then when it's time to translate such a branch to Tier 2 uops, we use the bit count from the cache entry to decided whether to continue translating the "didn't jump" branch or the "jumped" branch.
The counter is initialized to a pattern of alternating ones and zeros to avoid bias.
The .pyc file magic number is updated. There's a new test, some fixes for existing tests, and a few miscellaneous cleanups.
libregrtest now uses a separated file descriptor to write test result
as JSON. Previously, if a test wrote debug messages late around the
JSON, the main test process failed to parse JSON.
Rename TestResult.write_json() to TestResult.write_json_into().
worker_process() no longer writes an empty line at the end. There is
no need to separate test process output from the JSON output anymore,
since JSON is now written into a separated file descriptor.
create_worker_process() now always spawn the process with
close_fds=True.
* Rename runtest_mp.py to run_workers.py
* Move exit_timeout() and temp_cwd() context managers from
Regrtest.main() to Regrtest.run_tests(). Actions like --list-tests
or --list-cases don't need these protections.
* Regrtest: remove selected and tests attributes. Pass 'selected' to
list_tests(), list_cases() and run_tests(). display_result() now
expects a TestTuple, instead of TestList.
* Rename setup_tests() to setup_process() and rename setup_support()
to setup_tests().
* Move _adjust_resource_limits() to utils and rename it to
adjust_rlimit_nofile().
* Move replace_stdout() to utils.
* Fix RunTests.verbose type: it's an int.
* Pass results, quiet and pgo to Logger constructor.
* Move display_progress() method from Regrtest to Logger.
* No longer pass Regrtest to RunWorkers, but logger and results.
* Add single.py and result.py files.
* Rename runtest.py to runtests.py.
* Move run_single_test() function and its helper functions to
single.py.
* Move remove_testfn(), abs_module_name() and normalize_test_name()
to utils.py.
* Move setup_support() to setup.py.
* Move type hints like TestName to utils.py.
* Rename runtest.py to runtests.py.
Add new worker.py file:
* Move create_worker_process() and worker_process() to this file.
* Add main() function to worker.py. create_worker_process() now
runs the command: "python -m test.libregrtest.worker JSON".
* create_worker_process() now starts the worker process in the
current working directory. Regrtest now gets the absolute path of
the reflog.txt filename: -R command line option filename.
* Remove --worker-json command line option.
Remove test_regrtest.test_worker_json().
Related changes:
* Add write_json() and from_json() methods to TestResult.
* Rename select_temp_dir() to get_temp_dir() and move it to utils.
* Rename make_temp_dir() to get_work_dir() and move it to utils.
It no longer calls os.makedirs(): Regrtest.main() now calls it.
* Move fix_umask() to utils. The function is now called by
setup_tests().
* Move StrPath to utils.
* Add exit_timeout() context manager to utils.
* RunTests: Replace junit_filename (StrPath) with use_junit (bool).
Fix test_pyexpat.test_exception(): it can now be run from a directory
different than Python source code directory. Before, the test failed
in this case.
Skip the test if Modules/pyexpat.c source is not available. Skip also
the test on Python implementations other than CPython.
Fix test_site.test_underpth_basic() when the working directory
contains at least one non-ASCII character: encode the "._pth" file to
UTF-8 and enable the UTF-8 Mode to use UTF-8 for the child process
stdout.
Attempts to pickle or create a shallow or deep copy of codecs streams
now raise a TypeError.
Previously, copying failed with a RecursionError, while pickling
produced wrong results that eventually caused unpickling to fail with
a RecursionError.
* Add attributes to Regrtest and RunTests:
* fail_env_changed
* num_workers
* Rename MultiprocessTestRunner to RunWorkers. Add num_workers
parameters to RunWorkers constructor. Remove RunWorkers.ns
attribute.
* Rename TestWorkerProcess to WorkerThread.
* get_running() now returns a string like: "running (...): ...".
* Regrtest.action_run_tests() now selects the number of worker
processes, instead of the command line parser.
* Add attributes to Regrtest and RunTests:
* gc_threshold
* memory_limit
* python_cmd
* use_resources
* Remove WorkerJob class. Add as_json() and from_json() methods to
RunTests. A worker process now only uses RunTests for all
parameters.
* Add tests on support.set_memlimit() in test_support. Create
_parse_memlimit() and also adds tests on it.
* Remove 'ns' parameter from runtest.py.
* Rename dash_R() runtest_refleak(). The function now gets
huntrleaks and quiet arguments, instead of 'ns' argument.
* Add attributes to Regrtest and RunTests:
* verbose
* quiet
* huntrleaks
* test_dir
* Add HuntRefleak class.
Refator Regrtest class:
* Rename finalize() finalize_tests().
* Pass tracer to run_test() and finalize_tests(). Remove Regrtest.tracer.
* run_test() does less things: move code to its caller.
* Regrtest.__init__() now copies 'ns' namespace attributes to
Regrtest attributes. Regrtest match_tests and ignore_tests
attributes have type FilterTuple (tuple), instead of a list.
* Add RunTests.copy(). Regrtest._rerun_failed_tests() now uses
RunTests.copy().
* Replace Regrtest.all_tests (list) with Regrtest.first_runtests
(RunTests).
* Make random_seed maximum 10x larger (9 digits, instead of 8).
* main() now calls _parse_args() and pass 'ns' to Regrtest
constructor. Remove kwargs argument from Regrtest.main().
* _parse_args() checks ns.huntrleaks.
* set_temp_dir() is now responsible to call expanduser().
* Regrtest.main() sets self.tests earlier.
* Add TestTuple and TestList types.
* Rename MatchTests to FilterTuple and rename MatchTestsDict
to FilterTestDict.
* TestResult.get_rerun_match_tests() return type
is now FilterTuple: return a tuple instead of a list.
RunTests.tests type becomes TestTuple.
Symbols of the C API should be prefixed by "Py_" to avoid conflict
with existing names in 3rd party C extensions on "#include <Python.h>".
test.pythoninfo now logs Py_C_RECURSION_LIMIT constant and other
_testcapi and _testinternalcapi constants.
add guard so that ``Enum('bar')`` raises a TypeError instead of
creating a new enum class called `bar`. To create the new but
empty class, use:
huh = Enum('bar', names=())
Benjamin Peterson
Jelle Zijlstra
Victor Stinner
Ronan Pigott
Sam Gross
Nikita Sobolev
Jenner
Serhiy Storchaka
Tian Gao
Shantanu
Alex Waygood
Brett Cannon
Gregory P. Smith
buermarc
Carl Meyer
Irit Katriel
Brandt Bucher
Hugo van Kemenade
Guido van Rossum
Mark Shannon
云line
Pieter Eendebak
AN Long
Pablo Galindo Salgado
Kumar Aditya
Ethan Furman