Add write_all() helper function to test_pty to raise an exception on
short write: if os.writes() does not write all bytes. It should not
happen for a PTY.
The docs state that the space, tab, colon, and comma characters are
ignored in Py_BuildValue() format strings.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
It does not already work (because it locks only addCleanup(), not doCleanups()),
and it is no longer needed since the clean up procedure waits for all test threads to join.
Ensure that `PurePath('foo/a').with_name('.')` raises `ValueError`
Ensure that `PureWindowsPath('foo/a').with_name('a:b')` does not raise
`ValueError`.
multiprocessing test_terminate() and test_wait_socket_slow() no
longer test the CI performance: no longer check maximum elapsed time.
Add CLOCK_RES constant: tolerate a difference of 100 ms.
Deprecation warning about non-integer numbers in gettext now always refers
to the line in the user code where gettext function or method is used.
Previously, it could refer to a line in gettext code.
Also, increase test coverage for NullTranslations and domain-aware functions
like dngettext().
* test_timerfd_TFD_TIMER_ABSTIME() and
test_timerfd_ns_TFD_TIMER_ABSTIME() tolerate a difference of 50 us.
* test_timerfd_negative() checks if os.TFD_TIMER_CANCEL_ON_SET is
defined.
Add wrapper for timerfd_create, timerfd_settime, and timerfd_gettime to os module.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
sys.audit() now has assertions to check that the event argument is
not NULL and that the format argument does not use the "N" format.
Add tests on PySys_AuditTuple().
Replace os.kill() with proc.kill() which catchs PermissionError.
Rewrite _kill_with_event():
* Use subprocess context manager ("with proc:").
* Use sleeping_retry() to wait until the child process is ready.
* Replace SIGINT with proc.kill() on error.
* Replace 10 seconds with SHORT_TIMEOUT to wait until the process is
ready.
* Replace 0.5 seconds with SHORT_TIMEOUT to wait for the process
exit.
Increase support.LOOPBACK_TIMEOUT from 5 to 10 seconds. Also increase
the timeout depending on the --timeout option. For example, for a
test timeout of 40 minutes (ARM Raspbian 3.x), use LOOPBACK_TIMEOUT
of 20 seconds instead of 5 seconds before.
Fix a deadlock in test_socket when server fails with a timeout but
the client is still running in its thread. Don't hold a lock to call
cleanup functions in doCleanups(). One of the cleanup function waits
until the client completes, whereas the client could deadlock if it
called addCleanup() in such situation.
doCleanups() is called when the server completed, but the client can
still be running in its thread especially if the server failed with a
timeout. Don't put a lock on doCleanups() to prevent deadlock between
addCleanup() called in the client and doCleanups() waiting for
self.done.wait of ThreadableTest._setUp().
test_builtin and test_socketserver no longer use signal.alarm() to
implement a watchdog with a hardcoded timeout (2 and 60 seconds).
Python test runner regrtest has two watchdogs: faulthandler and
timeout on running worker processes. Tests using short hardcoded
timeout can fail on slowest buildbots just because the timeout is too
short.
* Use `FindFirstFile` Win32 API to fix a bug where `ntpath.realpath()`
breaks out of traversing a series of paths where a (handled)
`ERROR_ACCESS_DENIED` or `ERROR_SHARING_VIOLATION` occurs.
* Update docs to reflect that `ntpath.realpath()` eliminates MS-DOS
style names.
When using worker processes (-jN) with --verbose3 option, regrtest
can now display the worker output even if a worker process does
crash. Previously, sys.stdout and sys.stderr were replaced and so
the worker output was lost on a crash.
CFunctionFullTests now also runs "bt" command before "py-bt-full",
similar to CFunctionTests which also runs "bt" command before
"py-bt". So test_gdb can skip the test if patterns like "?? ()" are
found in the gdb output.
dataclasses.replace() now raises TypeError instead of ValueError if
specify keyword argument for a field declared with init=False or miss keyword
argument for required InitVar field.
The test had an instability issue due to the ordering of the dummy
queue operation and the real wakeup pipe operations. Both primitives
are thread safe but not done atomically as a single update and may
interleave arbitrarily. With the old order of operations this can lead
to an incorrect state where the dummy queue is full but the wakeup
pipe is empty. By swapping the order in clear() I think this can no
longer happen in any possible operation interleaving (famous last
words).
In Python/bytecodes.c, you now write
```
DEOPT_IF(condition);
```
The code generator expands this to
```
DEOPT_IF(condition, opcode);
```
where `opcode` is the name of the unspecialized instruction.
This works inside macro expansions too.
**CAVEAT:** The entire `DEOPT_IF(condition)` statement must be on a single line.
If it isn't, the substitution will fail; an error will be printed by the code generator
and the C compiler will report some errors.
In a few places we switch to another interpreter without knowing if it has a thread state associated with the current thread. For the main interpreter there wasn't much of a problem, but for subinterpreters we were *mostly* okay re-using the tstate created with the interpreter (located via PyInterpreterState_ThreadHead()). There was a good chance that tstate wasn't actually in use by another thread.
However, there are no guarantees of that. Furthermore, re-using an already used tstate is currently fragile. To address this, now we create a new thread state in each of those places and use it.
One consequence of this change is that PyInterpreterState_ThreadHead() may not return NULL (though that won't happen for the main interpreter).
The existence of background threads running on a subinterpreter was preventing interpreters from getting properly destroyed, as well as impacting the ability to run the interpreter again. It also affected how we wait for non-daemon threads to finish.
We add PyInterpreterState.threads.main, with some internal C-API functions.
This change makes sure sys.path[0] is set properly for subinterpreters. Before, it wasn't getting set at all. This PR does not address the broader concerns from gh-109853.
If the timeout is greater than PY_TIMEOUT_MAX,
PyThread_acquire_lock_timed() uses a timeout of PY_TIMEOUT_MAX
microseconds, which is around 280.6 years. This case is unlikely and
limiting a timeout to 280.6 years sounds like a reasonable trade-off.
The constant PY_TIMEOUT_MAX is not used in PyPI top 5,000 projects.
This method supports file URIs (including variants) as described in RFC 8089, such as URIs generated by `pathlib.Path.as_uri()` and `urllib.request.pathname2url()`.
The method is added to `Path` rather than `PurePath` because it uses `os.fsdecode()`, and so its results vary from system to system. I intend to deprecate `PurePath.as_uri()` and move it to `Path` for the same reason.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
* _add_python_opts() now handles cross compilation and HOSTRUNNER.
* display_header() now tells if Python is cross-compiled, display
HOSTRUNNER, and get the host platform.
* Remove Tools/scripts/run_tests.py script.
* Remove "make hostrunnertest": use "make buildbottest"
or "make test" instead.
* Refactor os_sched_getaffinity_impl(): move variable definitions to
their first assignment.
* Fix test_posix.test_sched_getaffinity(): restore the old CPU mask
when the test completes!
* Doc: Specify that os.cpu_count() counts *logicial* CPUs.
* Doc: Specify that os.sched_getaffinity(0) is related to the calling
thread.
Rename WORKER_ERROR to WORKER_BUG. Add WORKER_FAILED state: it does
not stop the manager, whereas WORKER_BUG does.
Change also TestResults.display_result() order: display failed tests
at the end, the important important information.
WorkerThread now tries to get the signal name for negative exit code.
Call also copy_python_src_ignore() on listdir() names.
shutil.copytree(): replace set() with an empty tuple. An empty tuple
becomes a constant in the compiler and checking if an item is in an
empty tuple is cheap.
* pycore_pythread.h is now the central place to make sure that
_POSIX_THREADS and _POSIX_SEMAPHORES macros are defined if
available.
* Make sure that pycore_pythread.h is included when _POSIX_THREADS
and _POSIX_SEMAPHORES macros are tested.
* PY_TIMEOUT_MAX is now defined as a constant, since its value
depends on _POSIX_THREADS, instead of being defined as a macro.
* Prevent integer overflow in the preprocessor when computing
PY_TIMEOUT_MAX_VALUE on Windows:
replace "0xFFFFFFFELL * 1000 < LLONG_MAX"
with "0xFFFFFFFELL < LLONG_MAX / 1000".
* Document the change and give hints how to fix affected code.
* Add an exception for PY_TIMEOUT_MAX name to smelly.py
* Add PY_TIMEOUT_MAX to the stable ABI
Add private `pathlib._PathBase` class. This will be used by an experimental PyPI package to incubate a `tarfile.TarPath` class.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
concurrent.futures: The *executor manager thread* now catches
exceptions when adding an item to the *call queue*. During Python
finalization, creating a new thread can now raise RuntimeError. Catch
the exception and call terminate_broken() in this case.
Add test_python_finalization_error() to test_concurrent_futures.
concurrent.futures._ExecutorManagerThread changes:
* terminate_broken() no longer calls shutdown_workers() since the
call queue is no longer working anymore (read and write ends of
the queue pipe are closed).
* terminate_broken() now terminates child processes, not only
wait until they complete.
* _ExecutorManagerThread.terminate_broken() now holds shutdown_lock
to prevent race conditons with ProcessPoolExecutor.submit().
multiprocessing.Queue changes:
* Add _terminate_broken() method.
* _start_thread() sets _thread to None on exception to prevent
leaking "dangling threads" even if the thread was not started
yet.
* Add context manager on Bunch class.
* Bunch now catchs exceptions on executed functions and re-raise them
at __exit__() as an ExceptionGroup.
* Rewrite BarrierProxy.test_default_timeout(). Use a single thread.
Only check that barrier.wait() blocks for at least default timeout
seconds.
* test_with(): inline _with() function.
Fix test_asyncio timeouts: don't measure the maximum duration, a test
should not measure a CI performance. Only measure the minimum
duration when a task has a timeout or delay. Add CLOCK_RES to
test_asyncio.utils.
In the subprocess code path, wait until the child process completes
with a timeout of EXIT_TIMEOUT seconds.
Fix create_worker_process() regression: use start_new_session=True if
USE_PROCESS_GROUP is true.
WorkerThread.wait_stopped() uses a timeout of 60 seconds, instead of
30 seconds.
On Windows, multiprocessing Popen.terminate() now catchs
PermissionError and get the process exit code. If the process is
still running, raise again the PermissionError. Otherwise, the
process terminated as expected: store its exit code.
Fix race conditions in test_threading lock tests. Wait until a
condition is met rather than using time.sleep() with a hardcoded
number of seconds.
* Replace sleeping loops with support.sleeping_retry() which raises
an exception on timeout.
* Add wait_threads_blocked(nthread) which computes a sleep depending
on the number of threads. Remove _wait() function.
* test_set_and_clear(): use a way longer Event.wait() timeout.
* BarrierTests.test_repr(): wait until the 2 threads are waiting for
the barrier. Use a way longer timeout for Barrier.wait() timeout.
* test_thread_leak() no longer needs to count
len(threading.enumerate()): Bunch uses
threading_helper.wait_threads_exit() internally which does it in
wait_for_finished().
* Add BaseLockTests.wait_phase() which implements a timeout.
test_reacquire() and test_recursion_count() use wait_phase().
In 2003, test_pty got a hardcoded timeout of 10 seconds to prevent
hanging on AIX & HPUX "if run after test_openpty":
commit 7d8145268e. Since 2003, test_pty
was no longer reported to hang on AIX. But today, the test can fail
simply because a CI is busy running other tests in parallel.
The timeout of 10 seconds is no longer needed, just remove it.
Moreover, regrtest now has multiple built-in generic timeout
mecanisms.
Fix test_interprocess_signal() of test_signal. Make sure that the
subprocess.Popen object is deleted before the test raising an
exception in a signal handler. Otherwise, Popen.__del__() can get the
exception which is logged as "Exception ignored in: ...." and the
test fails.
* Split test_pycfunction.py: add test_cfunction_full.py.
Split the function into the following 6 functions. In verbose
mode, these "pycfunction" tests now log each tested call.
* test_pycfunction_noargs()
* test_pycfunction_o()
* test_pycfunction_varargs()
* test_pycfunction_varargs_keywords()
* test_pycfunction_fastcall()
* test_pycfunction_fastcall_keywords()
* Move get_gdb_repr() to PrettyPrintTests.
* Replace DebuggerTests.get_sample_script() with SAMPLE_SCRIPT.
* Rename checkout_hook_path to CHECKOUT_HOOK_PATH.
* Rename gdb_version to GDB_VERSION_TEXT.
* Replace (gdb_major_version, gdb_minor_version) with GDB_VERSION.
* run_gdb() uses "backslashreplace" error handler instead of "replace".
* Add check_gdb() function to util.py.
* Enhance support.check_cflags_pgo(): check also for sysconfig
PGO_PROF_USE_FLAG (if available) in compiler flags.
* Move some SkipTest checks to test_gdb/__init__.py.
* Elaborate why gdb cannot be tested on Windows: gdb doesn't support
PDB debug symbol files.
Fix test_timeout() of test_concurrent_futures.test_wait. Remove the
future which may or may not complete depending if it takes longer
than the timeout ot not. Keep the second future which does not
complete before wait(). Make also the test faster: 0.5 second instead
of 6 seconds, so remove @support.requires_resource('walltime')
decorator.
Split test_gdb.py file into a test_gdb package made of multiple
tests, so tests can now be run in parallel.
* Create Lib/test/test_gdb/ directory.
* Split test_gdb.py into multiple files in Lib/test/test_gdb/
directory.
* Move Lib/test/gdb_sample.py to Lib/test/test_gdb/ directory.
Update get_sample_script(): use __file__ to locate gdb_sample.py.
* Move gdb_has_frame_select() and HAS_PYUP_PYDOWN to test_misc.py.
* Explicitly skip test_gdb on Windows. Previously, test_gdb was
skipped even if gdb was available because of
gdb_has_frame_select().
Fix the test when run on an installed Python: use "abs_srcdir" of
sysconfig, and skip the test if the Python source code cannot be
found.
* Tools/patchcheck/patchcheck.py, Tools/freeze/test/freeze.py and
Lib/test/libregrtest/utils.py now first try to get "abs_srcdir"
from sysconfig, before getting "srcdir" from sysconfig.
* test.pythoninfo logs sysconfig "abs_srcdir".
WASM and WASI buildbots use multiple PYTHON environment variables
such as PYTHONPATH and _PYTHON_HOSTRUNNER. Don't use -E if the
--python=COMMAND option is used.
Fix copy_source_tree() function of test_tools.test_freeze:
* Don't copy SRC_DIR/build/ anymore. This directory is modified by
other tests running in parallel.
* Add test.support.copy_python_src_ignore().
* Use sysconfig to get the source directory.
* Use sysconfig.get_config_var() to get CONFIG_ARGS variable.
* On Windows, use subprocess.run() instead of os.execv().
* Only add needed options
* Rename reexec parameter to _add_python_opts.
* Rename --no-reexec option to --dont-add-python-opts.
Remove replace_stdout(): call sys.stdout.reconfigure() instead of set
the error handler to backslashreplace.
display_header() logs an empty line and flush stdout.
Remove encoding workaround in display_header() since stdout error
handler is now set to backslashreplace earlier.
When --fast-ci or --slow-ci option is used, regrtest now replaces the
current process with a new process to add "-u -W default -bb -E"
options to Python.
Changes:
* PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add
"-u -W default -bb -E" options to Python: it's now done by
regrtest.
* Fix Tools/scripts/run_tests.py: flush stdout before replacing the
process. Previously, buffered messages were lost.
* Add --fast-ci and --slow-ci options to libregrtest:
* --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
(skip slowest tests).
* --slow-ci uses a default timeout of 20 minues and "-u all" (run
all tests).
* regrtest header now lists test resources.
* Makefile changes:
* "make test", "make hostrunnertest" and "make coverage-report" now
use --fast-ci option and TESTTIMEOUT variable.
* "make buildbottest" now uses "--slow-ci". Remove options which
became redundant with "--slow-ci".
* "make testall" and "make testuniversal" now use --slow-ci option
and TESTTIMEOUT variable.
* "make testall" now uses "find -exec rm ..." instead of
"find ... -print|xargs rm ...", same as "make clean".
* GitHub Actions workflow:
* Ubuntu and Address Sanitizer jobs now use "make test". Remove
options which became redundant with "--fast-ci".
* Windows jobs now use --fast-ci option.
* Use -j0 to detect the number of CPUs.
* Set Makefile TESTTIMEOUT default to an empty string, since
--slow-ci and --fast-ci use different default timeout. It's now
accepted to pass "--timeout=" to regrtest: treated as not timeout.
* Tools/scripts/run_tests.py now uses --fast-ci option.
* Tools/buildbot/test.bat now uses --slow-ci option. Remove
--timeout=1200 option, redundant with --slow-ci.
test_error_at_task_unpickle() and
test_error_during_result_unpickle_in_result_handler() now restore
sys.stderr which is overriden by _raise_error_ignore_stderr().
Follow-up of gh-107219.
* Only close the connection writer on Windows.
* Also use existing constant _winapi.ERROR_OPERATION_ABORTED instead of
WSA_OPERATION_ABORTED.
* Refactor zipfile._strip_extra to use higher level abstractions for extras instead of a heavy-state loop.
* Add blurb
* Remove _strip_extra and use _Extra.strip directly.
* Use memoryview to avoid unnecessary copies while splitting Extras.
Fix test_zippath_from_non_installed_posix() of test_venv: don't copy
__pycache__/ sub-directories, because they can be modified by other
Python tests running in parallel.
* Factorize code listing "bad / env changed / ..." tests.
* Add TestResults.is_all_good() method.
* Move "All 400 tests OK." to the end
* Move "Test suite interrupted by signal SIGINT." to the end.
Expect the test to be "short" but don't measure the exact performance
of the CI. SHORT_TIMEOUT is about 30 seconds whereas the cancelled
coroutine takes around 1 hour.
Replace harcoded sleep of 500 ms with synchronization using a pipe.
Fix also Process._feed_stdin(): catch also BrokenPipeError on
stdin.write(input), not only on stdin.drain().
This fixes issue #105829, https://github.com/python/cpython/issues/105829
Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Co-authored-by: Chris Withers <chris@withers.org>
Co-authored-by: Thomas Moreau <thomas.moreau.2010@gmail.com>
Sometimes the child_handled event was missing because either
the child quits before it gets a chance to handle the signal,
or the parent asserts before the event notification is
delivered via IPC. Synchronize explicitly to avoid this.
SubprocessProtocol process_exited() method can be called before
pipe_data_received() and pipe_connection_lost() methods. Document it
and adapt the test for that.
Revert commit 282edd7b2a.
_child_watcher_callback() calls immediately _process_exited(): don't
add an additional delay with call_soon(). The reverted change didn't
make _process_exited() more determistic: it can still be called
before pipe_connection_lost() for example.
Co-authored-by: Davide Rizzo <sorcio@gmail.com>
Skip test_freeze_simple_script() of test_tools.test_freeze if Python
is built with "./configure --enable-optimizations", which means with
Profile Guided Optimization (PGO): it just makes the test too slow.
The freeze tool is tested by many other CIs with other (faster)
compiler flags.
test.pythoninfo now gets also get_build_info() of
test.libregrtests.utils.
Lysandros Nikolaou
Filipe Laíns
Jelle Zijlstra
Alex Waygood
Pablo Galindo Salgado
Thomas Grainger
Nikita Sobolev
Hugo van Kemenade
Victor Stinner
J. Nick Koston
Tian Gao
Serhiy Storchaka
Zackery Spytz
Barney Gale
Radislav Chugunov
Yilei Yang
Donghee Na
Eric Snow
Masaru Tsuchiyama
Carl Meyer
Sam Gross
AN Long
박문식
sunmy2019
Xuehai Pan
elfstrom
Guido van Rossum
Steve Dower
Charles Machalow
Raymond Hettinger
Furkan Onder
James Hilton-Balfe
Pieter Eendebak
Irit Katriel
Davide Rizzo
Kristján Valur Jónsson
Dale Collison
Tom Gillespie
Antoine Pitrou
Jason R. Coombs
DongWoo Son
EliseevEgor
Heinz-Alexander Fuetterer
Benjamin Peterson