Commit Graph

30471 Commits

Author SHA1 Message Date
Irit Katriel 8b55adfa8f
gh-109256: allocate opcode IDs for internal opcodes in their own range (#109269) 2023-09-12 10:36:17 +00:00
Nikita Sobolev 1110c5bc82
gh-108303: Move tokenize-related data to Lib/test/tokenizedata (GH-109265) 2023-09-12 09:37:42 +03:00
Victor Stinner 8c813faf86
gh-109276: libregrtest: limit number workers (#109288)
Don't spawn more threads than the number of jobs: these worker
threads would never get anything to do.

* Add the number of tests in "Run ... tests in ..." message.
* Add RunTests.get_jobs() method.
* Add plural() function.
* count() uses f-string.
2023-09-12 03:47:04 +00:00
Victor Stinner a84cb74d42
gh-109276: libregrtest calls random.seed() before each test (#109279)
libregrtest now calls random.seed() before running each test file
when -r/--randomize command line option is used. Moreover, it's also
called in worker processes. It should help to make tests more
deterministic. Previously, it was only called once in the main
process before running all test files and it was not called in worker
processes.

* Convert some f-strings to regular strings in test_regrtest when
  f-string is not needed.
* Remove unused all_methods variable from test_regrtest.
* Add RunTests members are now mandatory.
2023-09-12 05:35:08 +02:00
Victor Stinner 4e77645986
gh-109276: libregrtest only checks saved_test_environment() once (#109278)
There is no need to check for environment changes twice.
2023-09-12 05:01:33 +02:00
Victor Stinner df4f0fe203
gh-109276: Complete test.pythoninfo (#109312)
* Enable collect_sysconfig() on Windows.
* Add sysconfig 'abs_builddir' and 'srcdir'
* Add sysconfig.is_python_build()
* Add tempfile.gettempdir()
* Remove compatiblity with Python 2.7 (print_function).
2023-09-12 02:50:05 +00:00
Victor Stinner 9363769161
gh-109295: Skip test_generated_cases if different mount drives (#109308)
On Windows, skip the test if the current working directory and the
Python source code directory have different mount drives. It happens
if the temporary directory is on a different mount drive than the
Python source code.
2023-09-12 04:30:07 +02:00
Victor Stinner 7dedfd36dc
gh-109295: Fix test_os.test_access_denied() for TEMP=cwd (#109299)
Fix test_os.test_access_denied() when the TEMP environment variable
is equal to the current working directory. Run the test using a
different filename, since self.fname already exists in this case.
2023-09-12 03:35:26 +02:00
Victor Stinner 09ea4b8706
gh-109295: Clean up multiprocessing in test_asyncio and test_compileall (#109298)
test_asyncio and test_compileall now clean up multiprocessing by
calling multiprocessing _cleanup_tests(): explicitly clean up
resources and stop background processes like the resource tracker.
2023-09-12 03:31:15 +02:00
Jelle Zijlstra b88d9e75f6
gh-109118: Disallow nested scopes within PEP 695 scopes within classes (#109196)
Fixes #109118. Fixes #109194.

Co-authored-by: Carl Meyer <carl@oddbird.net>
2023-09-11 17:11:06 -07:00
Carl Meyer ceeb4173ae
gh-109195: fix source location for super load before LOAD_SUPER_ATTR (#109289) 2023-09-11 17:35:49 -06:00
Guido van Rossum fbaf77eb9b
gh-109214: Rename SAVE_IP to _SET_IP, and similar (#109285)
* Rename SAVE_IP to _SET_IP
* Rename EXIT_TRACE to _EXIT_TRACE
* Rename SAVE_CURRENT_IP to _SAVE_CURRENT_IP
* Rename INSERT to _INSERT (This is for Ken Jin's abstract interpreter)
* Rename IS_NONE to _IS_NONE
* Rename JUMP_TO_TOP to _JUMP_TO_TOP
2023-09-11 15:39:19 -07:00
Guido van Rossum bcce5e2718
gh-109039: Branch prediction for Tier 2 interpreter (#109038)
This adds a 16-bit inline cache entry to the conditional branch instructions POP_JUMP_IF_{FALSE,TRUE,NONE,NOT_NONE} and their instrumented variants, which is used to keep track of the branch direction.

Each time we encounter these instructions we shift the cache entry left by one and set the bottom bit to whether we jumped.

Then when it's time to translate such a branch to Tier 2 uops, we use the bit count from the cache entry to decided whether to continue translating the "didn't jump" branch or the "jumped" branch.

The counter is initialized to a pattern of alternating ones and zeros to avoid bias.

The .pyc file magic number is updated. There's a new test, some fixes for existing tests, and a few miscellaneous cleanups.
2023-09-11 18:20:24 +00:00
Irit Katriel ecd21a629a
gh-109179: Fix traceback display for SyntaxErrors with notes (#109197) 2023-09-11 19:18:34 +01:00
Victor Stinner de5f8f7d13
gh-109276: libregrtest: use separated file for JSON (#109277)
libregrtest now uses a separated file descriptor to write test result
as JSON.  Previously, if a test wrote debug messages late around the
JSON, the main test process failed to parse JSON.

Rename TestResult.write_json() to TestResult.write_json_into().

worker_process() no longer writes an empty line at the end. There is
no need to separate test process output from the JSON output anymore,
since JSON is now written into a separated file descriptor.

create_worker_process() now always spawn the process with
close_fds=True.
2023-09-11 19:33:42 +02:00
Nikita Sobolev baa6dc8e38
gh-90805: Make sure test_functools works with and without _functoolsmodule (GH-108644) 2023-09-11 19:13:37 +03:00
Serhiy Storchaka c0f488b88f
gh-109182: Fix and improve tests for gh-108654 (GH-109189) 2023-09-11 17:50:33 +03:00
Mark Shannon 4a69301ea4
GH-108976. Keep monitoring data structures valid during de-optimization during callback. (GH-109131) 2023-09-11 14:37:09 +01:00
Serhiy Storchaka 60b8341d07
Better integration of doctest and unittest in test_ctypes.test_objects (GH-108922)
Better integration of docrtest and unittest in test_ctypes.test_objects
2023-09-11 14:05:30 +03:00
Serhiy Storchaka 0abc935086
Test DocTestFinder directly instead of calling support.run_doctest() (GH-108917) 2023-09-11 14:03:30 +03:00
Victor Stinner c439f6a72d
gh-109162: libregrtest: move code around (#109253)
* Move Regrtest.display_header() to utils.py.
* Move cleanup_temp_dir() to utils.py.
* Move list_cases() to findtests.py.
2023-09-11 08:52:03 +00:00
Victor Stinner 7aa8fcc8eb
gh-109162: libregrtest: use relative imports (#109250)
libregrtest.__init__ no longer exposes any symbol, so
"python -m test.libregrtest.worker" imports less modules.
2023-09-11 07:02:35 +00:00
Victor Stinner 0c139b5f2f
gh-109162: libregrtest: rename runtest_mp.py to run_workers.py (#109248)
* Rename runtest_mp.py to run_workers.py
* Move exit_timeout() and temp_cwd() context managers from
  Regrtest.main() to Regrtest.run_tests(). Actions like --list-tests
  or --list-cases don't need these protections.
* Regrtest: remove selected and tests attributes. Pass 'selected' to
  list_tests(), list_cases() and run_tests(). display_result() now
  expects a TestTuple, instead of TestList.
* Rename setup_tests() to setup_process() and rename setup_support()
  to setup_tests().
* Move _adjust_resource_limits() to utils and rename it to
  adjust_rlimit_nofile().
* Move replace_stdout() to utils.
* Fix RunTests.verbose type: it's an int.
2023-09-11 03:27:37 +00:00
Victor Stinner 0b6b05391b
gh-109162: libregrtest: fix Logger (#109246)
* Pass results, quiet and pgo to Logger constructor.
* Move display_progress() method from Regrtest to Logger.
* No longer pass Regrtest to RunWorkers, but logger and results.
2023-09-11 01:46:26 +00:00
Victor Stinner 1ec45378e9
gh-109162: libregrtest: add single.py and result.py (#109243)
* Add single.py and result.py files.
* Rename runtest.py to runtests.py.
* Move run_single_test() function and its helper functions to
  single.py.
* Move remove_testfn(), abs_module_name() and normalize_test_name()
  to utils.py.
* Move setup_support() to setup.py.
* Move type hints like TestName to utils.py.
* Rename runtest.py to runtests.py.
2023-09-11 00:07:18 +00:00
Victor Stinner a939b65aa6
gh-109162: libregrtest: add worker.py (#109229)
Add new worker.py file:

* Move create_worker_process() and worker_process() to this file.
* Add main() function to worker.py. create_worker_process() now
  runs the command: "python -m test.libregrtest.worker JSON".
* create_worker_process() now starts the worker process in the
  current working directory. Regrtest now gets the absolute path of
  the reflog.txt filename: -R command line option filename.
* Remove --worker-json command line option.
  Remove test_regrtest.test_worker_json().

Related changes:

* Add write_json() and from_json() methods to TestResult.
* Rename select_temp_dir() to get_temp_dir() and move it to utils.
* Rename make_temp_dir() to get_work_dir() and move it to utils.
  It no longer calls os.makedirs(): Regrtest.main() now calls it.
* Move fix_umask() to utils. The function is now called by
  setup_tests().
* Move StrPath to utils.
* Add exit_timeout() context manager to utils.
* RunTests: Replace junit_filename (StrPath) with use_junit (bool).
2023-09-11 01:11:22 +02:00
Victor Stinner e55aab9578
gh-109230: test_pyexpat no longer depends on the current directory (#109233)
Fix test_pyexpat.test_exception(): it can now be run from a directory
different than Python source code directory. Before, the test failed
in this case.

Skip the test if Modules/pyexpat.c source is not available. Skip also
the test on Python implementations other than CPython.
2023-09-10 22:16:59 +00:00
Victor Stinner cbb3a6f8ad
gh-109237: Fix test_site for non-ASCII working directory (#109238)
Fix test_site.test_underpth_basic() when the working directory
contains at least one non-ASCII character: encode the "._pth" file to
UTF-8 and enable the UTF-8 Mode to use UTF-8 for the child process
stdout.
2023-09-10 22:04:35 +00:00
Serhiy Storchaka d6892c2b92
gh-50644: Forbid pickling of codecs streams (GH-109180)
Attempts to pickle or create a shallow or deep copy of codecs streams
now raise a TypeError.

Previously, copying failed with a RecursionError, while pickling
produced wrong results that eventually caused unpickling to fail with
a RecursionError.
2023-09-10 20:06:09 +03:00
云line 4297499696
gh-109207: Fix SystemError when printing symtable entry object. (GH-109225) 2023-09-10 15:04:24 +03:00
Pieter Eendebak 85a5d3dbe1
gh-93627: Align Python implementation of pickle with C implementation of pickle (GH-103035)
If a method like __reduce_ex_ or __reduce__ is set to None, a TypeError is raised.
2023-09-10 10:06:08 +03:00
Serhiy Storchaka 92578919a6
gh-109174: Add support of SimpleNamespace in copy.replace() (GH-109175) 2023-09-10 08:09:25 +03:00
Victor Stinner 0eab2427b1
gh-109162: libregrtest: add Logger class (#109212)
* Add Logger class in a new logger.py file.
* Move Regrtest attributes to Logger:

  * start_time
  * test_count_text
  * test_count_width
  * win_load_tracker

* Move Regrtest method to Logger:

  * log()
  * getloadavg(): rename to get_load_avg()
  * set_tests()

* Add methods to the Logger class:

  * start_load_tracker()
  * stop_load_tracker()
2023-09-10 03:04:26 +00:00
Victor Stinner db5bfe91f8
gh-109162: libregrtest: add TestResults class (#109208)
* Add TestResults class.
* Move Regrtest methods to TestResults:

  * accumulate_result(): now takes a RunTests parameter
  * get_executed()
  * no_tests_run()

* Add methods to TestResults:

  * add_junit()
  * display_result()
  * display_summary()
  * need_rerun()
  * prepare_rerun()
  * write_junit()

* Rename 'need_rerun' attribute to 'bad_results'.
* Rename 'total_stats' attribute to 'stats'.
2023-09-10 02:30:43 +00:00
Victor Stinner a341750078
gh-109162: Refactor libregrtest.Regrtest (#109206)
* Add type hint types: TestName, StrPath, StrJSON.
* Add attributes to Regrtest:

  * cmdline_args
  * coverage
  * coverage_dir
  * fail_rerun
  * next_single_filename
  * print_slowest
  * tmp_dir
  * want_rerun
  * want_run_leaks

* Remove Regrtest.ns attribute.
* Rename Regrtest methods:

  * cleanup() => cleanup_temp_dir()
  * create_temp_dir() => make_temp_dir()
  * set_temp_dir() => select_temp_dir()

* Convert Regrtest methods to static methods:

  * cleanup_temp_dir()
  * display_header()
  * fix_umask()
  * get_rerun_match_tests()
  * list_tests()
  * make_temp_dir()
  * select_temp_dir()

* Remove display_sanitizers() method: move code into
  display_header().
* Rename 'test_cwd' variable to 'work_dir'.
2023-09-10 03:07:05 +02:00
Victor Stinner 0553fdfe30
gh-109162: Refactor libregrtest.runtest_mp (#109205)
* Add attributes to Regrtest and RunTests:

  * fail_env_changed
  * num_workers

* Rename MultiprocessTestRunner to RunWorkers. Add num_workers
  parameters to RunWorkers constructor. Remove RunWorkers.ns
  attribute.
* Rename TestWorkerProcess to WorkerThread.
* get_running() now returns a string like: "running (...): ...".
* Regrtest.action_run_tests() now selects the number of worker
  processes, instead of the command line parser.
2023-09-10 00:24:38 +00:00
Victor Stinner 0c0f254230
gh-109162: libregrtest: remove WorkerJob class (#109204)
* Add attributes to Regrtest and RunTests:

  * gc_threshold
  * memory_limit
  * python_cmd
  * use_resources

* Remove WorkerJob class. Add as_json() and from_json() methods to
  RunTests. A worker process now only uses RunTests for all
  parameters.
* Add tests on support.set_memlimit() in test_support. Create
  _parse_memlimit() and also adds tests on it.
* Remove 'ns' parameter from runtest.py.
2023-09-10 01:41:21 +02:00
Victor Stinner 24fa8f2046
gh-109162: libregrtest: fix _decode_worker_job() (#109202)
Decode also HuntRefleak() object inside the RunTests object.

Add an unit test on huntrleaks with multiprocessing (-R -jN).
2023-09-10 00:51:24 +02:00
Victor Stinner e21c89f984
gh-109162: Refactor libregrtest.RunTests (#109177)
* Rename dash_R() runtest_refleak(). The function now gets
  huntrleaks and quiet arguments, instead of 'ns' argument.
* Add attributes to Regrtest and RunTests:

  * verbose
  * quiet
  * huntrleaks
  * test_dir

* Add HuntRefleak class.
2023-09-09 09:18:14 +00:00
Serhiy Storchaka b4131a13cb
gh-109050: Remove remaining tests for legacy Unicode C API (GH-109068) 2023-09-09 05:44:46 +00:00
Jelle Zijlstra 17f994174d
gh-109118: Fix runtime crash when NameError happens in PEP 695 function (#109123) 2023-09-09 02:49:20 +00:00
Victor Stinner e9e2ca7a7b
gh-109162: Refactor libregrtest.runtest (#109172)
* Rename runtest() to run_single_test().
* Pass runtests to run_single_test().
* Add type annotation to Regrtest attributes. Add missing attributes
  to Namespace.
* Add attributes to Regrtest and RunTests:

  * fail_fast
  * ignore_tests
  * match_tests
  * output_on_failure
  * pgo
  * pgo_extended
  * timeout

* Get pgo from 'runtests', rather than from 'ns'.
* Remove WorkerJob.match_tests.
* setup_support() now gets pgo_extended from runtests.
* save_env(): change parameter order, pass test_name first.
* Add setup_test_dir() function.
* Pass runtests to setup_tests().
2023-09-09 01:37:48 +00:00
Tian Gao 057bc72490
gh-109052: Use the base opcode when comparing code objects (gh-109107) 2023-09-09 10:24:49 +09:00
Victor Stinner a56c928756
gh-109162: Refactor libregrtest WorkerJob (#109171)
* Rename --worker-args command line option to --worker-json.
* Rename _parse_worker_args() to _parse_worker_json().
* WorkerJob:

  * Add runtests attribute
  * Remove test_name and rerun attribute

* Rename run_test_in_subprocess() to create_worker_process().
* Rename run_tests_worker() to worker_process().
* create_worker_process() uses json.dump(): write directly JSON to
  stdout.
* Convert MultiprocessResult to a frozen dataclass.
* Rename RunTests.match_tests to RunTests.match_tests_dict.
2023-09-09 01:03:39 +00:00
Victor Stinner 489ca0acf0
gh-109162: Refactor Regrtest.action_run_tests() (#109170)
Refator Regrtest class:

* Rename finalize() finalize_tests().
* Pass tracer to run_test() and finalize_tests(). Remove Regrtest.tracer.
* run_test() does less things: move code to its caller.
2023-09-09 00:30:28 +00:00
Victor Stinner 2fafc3d5c6
gh-108996: Skip broken test_msvcrt for now (#109169) 2023-09-08 23:56:53 +00:00
Victor Stinner ac8409b38b
gh-109162: Regrtest copies 'ns' attributes (#109168)
* Regrtest.__init__() now copies 'ns' namespace attributes to
  Regrtest attributes. Regrtest match_tests and ignore_tests
  attributes have type FilterTuple (tuple), instead of a list.
* Add RunTests.copy(). Regrtest._rerun_failed_tests() now uses
  RunTests.copy().
* Replace Regrtest.all_tests (list) with Regrtest.first_runtests
  (RunTests).
* Make random_seed maximum 10x larger (9 digits, instead of 8).
2023-09-08 23:48:54 +00:00
Victor Stinner 5b7303e265
gh-109162: Refactor Regrtest.main() (#109163)
* main() now calls _parse_args() and pass 'ns' to Regrtest
  constructor.  Remove kwargs argument from Regrtest.main().
* _parse_args() checks ns.huntrleaks.
* set_temp_dir() is now responsible to call expanduser().
* Regrtest.main() sets self.tests earlier.
* Add TestTuple and TestList types.
* Rename MatchTests to FilterTuple and rename MatchTestsDict
  to FilterTestDict.
* TestResult.get_rerun_match_tests() return type
  is now FilterTuple: return a tuple instead of a list.
  RunTests.tests type becomes TestTuple.
2023-09-08 22:41:26 +00:00
AN Long bcb2ab5ef8
gh-108996: add tests for msvcrt (#109004)
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Co-authored-by: Steve Dower <steve.dower@microsoft.com>
2023-09-08 18:38:38 -04:00
Pablo Galindo Salgado 5bda2f637e
gh-109114: Relax the check for invalid lambdas inside f-strings to avoid false positives (#109121) 2023-09-08 17:00:23 +00:00
Kumar Aditya ccd48623d4
GH-109067: fix randomly failing `test_async_gen_asyncio_gc_aclose_09` test (#109142)
Use `asyncio.sleep(0)` instead of short sleeps.
2023-09-08 16:27:58 +00:00
Pablo Galindo Salgado 6275c67ea6
gh-106922: Fix error location for constructs with spaces and parentheses (#108959) 2023-09-08 17:18:35 +01:00
Hugo van Kemenade aa51182320
gh-109140: Rename duplicated tests in `test_binascii` (#109141) 2023-09-08 15:30:28 +00:00
Victor Stinner b0edf3b98e
GH-91079: Rename C_RECURSION_LIMIT to Py_C_RECURSION_LIMIT (#108507)
Symbols of the C API should be prefixed by "Py_" to avoid conflict
with existing names in 3rd party C extensions on "#include <Python.h>".

test.pythoninfo now logs Py_C_RECURSION_LIMIT constant and other
_testcapi and _testinternalcapi constants.
2023-09-08 09:48:28 +00:00
Mark Shannon 15d4c9fabc
GH-108716: Turn off deep-freezing of code objects. (GH-108722) 2023-09-08 10:34:40 +01:00
Ethan Furman c74e440168
gh-109022: [Enum] require `names=()` to create empty enum type (GH-109048)
add guard so that ``Enum('bar')`` raises a TypeError instead of
creating a new enum class called `bar`.  To create the new but
empty class, use:

    huh = Enum('bar', names=())
2023-09-07 18:19:03 -07:00
Serhiy Storchaka b9831e5c98
Use unittest test runner for doctests in test_statistics (GH-108921) 2023-09-07 23:08:55 +03:00
Serhiy Storchaka 7e1a7abb98
gh-68403: Fix test_coverage in test_trace (GH-108910)
Its behavior no longer affected by test running options such as -m.
2023-09-07 21:28:18 +03:00
Ee Durbin 1829a3c9a3
gh-75743: Restore test_timeout.testConnectTimeout() (#109087)
This un-skips this test now that pythontest.net implements appropriate firewall
rules for it.
2023-09-07 20:13:32 +02:00
Serhiy Storchaka f9f085c326
gh-103186: Make test_generated_cases less noisy by default (GH-109100)
Print additional details only when tests are run with -vv.
2023-09-07 20:53:38 +03:00
Irit Katriel 96396962ce
gh-109094: remove unnecessary updates of frame->prev_instr in instrumentation functions (#109076) 2023-09-07 18:23:11 +01:00
Carl Meyer f2584eade3
gh-108732: include comprehension locals in frame.f_locals (#109026)
Co-authored-by: Radislav Chugunov <52372310+chgnrdv@users.noreply.github.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
2023-09-07 08:56:43 -06:00
Serhiy Storchaka b2729e93e9
gh-88943: Improve syntax error for non-ASCII character that follows a numerical literal (GH-109081)
It now points on the invalid non-ASCII character, not on the valid numerical literal.
2023-09-07 17:00:13 +03:00
Mark Shannon 0858328ca2
GH-108614: Add `RESUME_CHECK` instruction (GH-108630) 2023-09-07 14:39:03 +01:00
Serhiy Storchaka d485551c9d
gh-103186: Suppress RuntimeWarning about unclosed async iterator in test_sys_settrace (GH-109075) 2023-09-07 16:34:44 +03:00
Serhiy Storchaka e4bb0026b9
gh-103186: Remove debug print in test_sys_settrace (GH-109077) 2023-09-07 16:30:41 +03:00
Ijtaba Hussain 3e53ac9903
gh-103186: Suppress and assert expected RuntimeWarnings in test_sys_settrace (GH-103244)
Caused as a result of frame manipulation where locals are never assigned / initialised.
2023-09-07 10:41:38 +03:00
Victor Stinner babdced23f
test.pythoninfo logs freedesktop_os_release() (#109057) 2023-09-07 05:43:32 +00:00
Anders Kaseorg f42edf1e7b
gh-109045: Remove remaining LIMITED_API_AVAILABLE checks in tests (#109046)
Commit 13a00078b8 (#108663) made all
Python builds compatible with the Limited API, and removed the
LIMITED_API_AVAILABLE flag.  However, some tests were still checking
for that flag, so they were now being incorrectly skipped.  Remove
these checks to let these tests run again.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2023-09-07 04:42:58 +02:00
Dong-hee Na 3bfa24e29f
gh-107265: Remove all ENTER_EXECUTOR when execute _Py_Instrument (gh-108539) 2023-09-07 09:53:54 +09:00
Victor Stinner a52a350977
gh-109015: Add test.support.socket_helper.tcp_blackhole() (#109016)
Skip test_asyncio, test_imaplib and test_socket tests if FreeBSD TCP
blackhole is enabled (net.inet.tcp.blackhole=2).
2023-09-07 01:58:03 +02:00
Serhiy Storchaka 6f3c138dfa
gh-108751: Add copy.replace() function (GH-108752)
It creates a modified copy of an object by calling the object's
__replace__() method.

It is a generalization of dataclasses.replace(), named tuple's _replace()
method and replace() methods in various classes, and supports all these
stdlib classes.
2023-09-06 23:55:42 +03:00
Serhiy Storchaka 3a08db8d13
gh-106307: Fix PyMapping_GetOptionalItemString() (GH-108797)
The resulting pointer was not set to NULL if the creation of a temporary
string object was failed.

The tests were also missed due to oversight.
2023-09-06 19:47:38 +00:00
Victor Stinner a0773b89df
gh-108753: Enhance pystats (#108754)
Statistics gathering is now off by default. Use the "-X pystats"
command line option or set the new PYTHONSTATS environment variable
to 1 to turn statistics gathering on at Python startup.

Statistics are no longer dumped at exit if statistics gathering was
off or statistics have been cleared.

Changes:

* Add PYTHONSTATS environment variable.
* sys._stats_dump() now returns False if statistics are not dumped
  because they are all equal to zero.
* Add PyConfig._pystats member.
* Add tests on sys functions and on setting PyConfig._pystats to 1.
* Add Include/cpython/pystats.h and Include/internal/pycore_pystats.h
  header files.
* Rename '_py_stats' variable to '_Py_stats'.
* Exclude Include/cpython/pystats.h from the Py_LIMITED_API.
* Move pystats.h include from object.h to Python.h.
* Add _Py_StatsOn() and _Py_StatsOff() functions. Remove
  '_py_stats_struct' variable from the API: make it static in
  specialize.c.
* Document API in Include/pystats.h and Include/cpython/pystats.h.
* Complete pystats documentation in Doc/using/configure.rst.
* Don't write "all zeros" stats: if _stats_off() and _stats_clear()
  or _stats_dump() were called.
* _PyEval_Fini() now always call _Py_PrintSpecializationStats() which
  does nothing if stats are all zeros.

Co-authored-by: Michael Droettboom <mdboom@gmail.com>
2023-09-06 15:54:59 +00:00
Victor Stinner 8ff1142578
gh-108851: Fix tomllib recursion tests (#108853)
* Add get_recursion_available() and get_recursion_depth() functions
  to the test.support module.
* Change infinite_recursion() default max_depth from 75 to 100.
* Fix test_tomllib recursion tests for WASI buildbots: reduce the
  recursion limit and compute the maximum nested array/dict depending
  on the current available recursion limit.
* test.pythoninfo logs sys.getrecursionlimit().
* Enhance test_sys tests on sys.getrecursionlimit()
  and sys.setrecursionlimit().
2023-09-06 17:34:31 +02:00
Victor Stinner fbce43a251
gh-91960: Skip test_gdb if gdb cannot retrive Python frames (#108999)
Skip test_gdb if gdb is unable to retrieve Python frame objects: if a
frame is "<optimized out>". When Python is built with "clang -Og",
gdb can fail to retrive the 'frame' parameter of
_PyEval_EvalFrameDefault(). In this case, tests like py_bt() are
likely to fail. Without getting access to Python frames,
python-gdb.py is mostly clueless on retrieving the Python traceback.
Moreover, test_gdb is no longer skipped on macOS if Python is built
with Clang.
2023-09-06 14:34:35 +00:00
Victor Stinner a8cae4071c
gh-107219: Fix concurrent.futures terminate_broken() (#108974)
Fix a race condition in _ExecutorManagerThread.terminate_broken():
ignore the InvalidStateError on future.set_exception(). It can happen
if the future is cancelled before the caller.

Moreover, test_crash_big_data() now waits explicitly until the
executor completes.
2023-09-06 15:57:01 +02:00
Victor Stinner 14d6e197cc
gh-108303: Create Lib/test/test_dataclasses/ directory (#108978)
Move test_dataclasses.py and its "dataclass_*.py" modules into the
new Lib/test/test_dataclasses/ subdirectory.
2023-09-06 15:54:16 +02:00
Nikita Sobolev 1fb20d42c5
gh-108983: Add more PEP 526 tests to `test_grammar` (#108984) 2023-09-06 06:41:38 -07:00
Matthias Bussonnier 5f3433f210
gh-106670: Fix Pdb handling of chained Exceptions with no stacks. (#108865) 2023-09-06 09:41:56 +00:00
Victor Stinner cd2ef21b07
gh-108962: Skip test_tempfile.test_flags() if not supported (#108964)
Skip test_tempfile.test_flags() if chflags() fails with "OSError:
[Errno 45] Operation not supported" (ex: on FreeBSD 13).
2023-09-05 21:59:40 +00:00
Guido van Rossum b87263be9b
gh-106581: Support multiple uops pushing new values (#108895)
Also avoid the need for the awkward .clone() call in the argument
to mgr.adjust_inverse() and mgr.adjust().
2023-09-05 13:58:39 -07:00
Anthony Shaw 2c4c26c4ce
gh-108469: Update ast.unparse for unescaped quote support from PEP701 [3.12] (#108553)
Co-authored-by: sunmy2019 <59365878+sunmy2019@users.noreply.github.com>
2023-09-05 21:01:23 +01:00
Nikita Sobolev 3f89b25763
gh-108927: Fix test_import + test_importlib + test_unittest problem (#108929) 2023-09-05 18:57:48 +00:00
Serhiy Storchaka 1e0d62793a
gh-108416: Mark slow but not CPU bound test methods with requires_resource('walltime') (GH-108480) 2023-09-05 17:56:30 +03:00
Serhiy Storchaka f980cc19b9
gh-89392: Use unittest test runner for doctests in test_getopt (GH-108916) 2023-09-05 17:35:28 +03:00
Serhiy Storchaka eaabaac7c0
gh-89392: Use normal unittest runner in test_type_cache (GH-108911) 2023-09-05 13:46:17 +00:00
Serhiy Storchaka 420c636219
Add missed "f" in an f-string (GH-108906) 2023-09-05 12:00:28 +00:00
Nikita Sobolev b4c8cce9a7
gh-108840: Remove unused `TestEnumTypeSubclassing` from `test_enum` (#108841) 2023-09-05 09:45:54 +01:00
Petr Viktorin 230649f538
gh-108294: Add error handling for time.sleep audit event (GH-108363)
I've also cleaned the tests up a bit to make this easier to test.
2023-09-05 10:25:08 +02:00
Mark Shannon 5a2a046151
GH-108390: Prevent non-local events being set with `sys.monitoring.set_local_events()` (GH-108420) 2023-09-05 08:03:53 +01:00
Serhiy Storchaka 04a0830b00
gh-89392: Remove support of test_main() in libregrtest (GH-108876) 2023-09-05 08:36:43 +03:00
Victor Stinner 1170d5a292
gh-108834: regrtest --fail-rerun exits with code 5 (#108896)
When the --fail-rerun option is used and a test fails and then pass,
regrtest now uses exit code 5 ("rerun) instead of 2 ("bad test").
2023-09-05 01:09:42 +00:00
Victor Stinner 676593859e
gh-106320: Remove private _PyErr_WriteUnraisableMsg() (#108863)
Move the private _PyErr_WriteUnraisableMsg() functions to the
internal C API (pycore_pyerrors.h).

Move write_unraisable_exc() from _testcapi to _testinternalcapi.
2023-09-05 01:54:55 +02:00
Tian Gao 6304d983a0
gh-108463: Make expressions/statements work as expected in pdb (#108464) 2023-09-04 21:44:40 +00:00
Nikita Sobolev 6ead5bd6ae
Disable `differing_test_runners` health check (#108886) 2023-09-04 18:31:58 +00:00
Petr Viktorin 40e52c94a2
gh-107902: Don't test setting suid/sgid on systems that don't support them (GH-108368) 2023-09-04 15:46:00 +02:00
Serhiy Storchaka f3b6608ba2
gh-89392: Fix running test_pep646_syntax as script (GH-108875) 2023-09-04 15:24:03 +03:00
Serhiy Storchaka 074ac1f72e
bpo-45229: Make ElementTree tests discoverable (GH-108859) 2023-09-04 13:04:32 +03:00
Serhiy Storchaka d0b22f6bd8
gh-89392: Make test_pep646_syntax discoverable (GH-108861) 2023-09-04 12:41:58 +03:00
Serhiy Storchaka 76f3c043b6
gh-89392: Remove test_main() in test_netrc (GH-108860) 2023-09-04 12:41:13 +03:00
Victor Stinner 5a79d2ae57
Revert "gh-46376: Return existing pointer when possible in ctypes (#1… (#108688)
This reverts commit 08447b5deb.

Revert also _ctypes.c changes of the PyDict_ContainsString() change,
commit 6726626646.
2023-09-04 11:21:47 +02:00
Victor Stinner 31c2945f14
gh-108834: regrtest reruns failed tests in subprocesses (#108839)
When using --rerun option, regrtest now re-runs failed tests
in verbose mode in fresh worker processes to have more
deterministic behavior. So it can write its final report even
if a test killed a worker progress.

Add --fail-rerun option to regrtest: exit with non-zero exit code
if a test failed pass passed when re-run in verbose mode (in a
fresh process). That's now more useful since tests can pass
when re-run in a fresh worker progress, whereas they failed
when run after other tests when tests are run sequentially.

Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a
deprecated alias.

Changes:

* Fix and enhance statistics in regrtest summary. Add "(filtered)"
  when --match and/or --ignore options are used.
* Add RunTests class.
* Add TestResult.get_rerun_match_tests() method
* Rewrite code to serialize/deserialize worker arguments as JSON
  using a new WorkerJob class.
* Fix stats when a test is run with --forever --rerun.
* If failed test names cannot be parsed, log a warning and don't
  filter tests.
* test_regrtest.test_rerun_success() now uses a marker file, since
  the test is re-run in a separated process.
* Add tests on normalize_test_name() function.
* Add test_success() and test_skip() tests to test_regrtest.
2023-09-03 21:37:15 +00:00
Sangyun_LEE 0c369d6cb8
Update Lib/test/test_unittest/testmock/testmock.py: fix typo RuntimError to RuntimeError (#108847) 2023-09-03 22:19:49 +01:00
Serhiy Storchaka 509bb61977
Reorder some test's decorators (GH-108804)
For example, do not demand the 'cpu' resource if the test cannot be run
due to non-working threads.
2023-09-03 15:21:43 +03:00
Sergey B Kirpichev 9c995abd78
gh-102837: improve test coverage for math module (#102523)
- input checks for math_1(L989), math_1a(L1023), math_2(L1064,L1071), hypot(L2682), log(L2307), ldexp(L2168), ceil(L1165), floor(L1236,L1239) and dist(L2587,L2588,L2628).
- drop inaccessible "if" branch (L3518) in perm_comb_small()
- improve fsum coverage for exceptional cases (L1433,L1438,L1451,L1497), ditto fmod(L2378)
- rewrite modf to fix inaccessible case(L2229), ditto for pow(L2988)
    
(all line numbers are wrt the main branch at 5e6661bce9)
2023-09-03 01:48:47 -07:00
Victor Stinner d4e534cbb3
regrtest computes statistics (#108793)
test_netrc, test_pep646_syntax and test_xml_etree now return results
in the test_main() function.

Changes:

* Rewrite TestResult as a dataclass with a new State class.
* Add test.support.TestStats class and Regrtest.stats_dict attribute.
* libregrtest.runtest functions now modify a TestResult instance
  in-place.
* libregrtest summary lists the number of run tests and skipped
  tests, and denied resources.
* Add TestResult.has_meaningful_duration() method.
* Compute TestResult duration in the upper function.
* Use time.perf_counter() instead of time.monotonic().
* Regrtest: rename 'resource_denieds' attribute to 'resource_denied'.
* Rename CHILD_ERROR to MULTIPROCESSING_ERROR.
* Use match/case syntadx to have different code depending on the
  test state.

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-09-02 18:09:36 +02:00
Barney Gale bdc3c884cd
GH-78722: Raise exceptions from `pathlib.Path.iterdir()` without delay. (#107320)
`pathlib.Path.iterdir()` now immediately raises any `OSError`
exception from `os.listdir()`, rather than waiting until its
result is iterated over.
2023-09-02 16:08:03 +01:00
Victor Stinner 4f9b706c6f
gh-108794: doctest counts skipped tests (#108795)
* Add 'skipped' attribute to TestResults.
* Add 'skips' attribute to DocTestRunner.
* Rename private DocTestRunner._name2ft attribute
  to DocTestRunner._stats.
* Use f-string for string formatting.
* Add some tests.
* Document DocTestRunner attributes and its API for statistics.
* Document TestResults class.

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-09-02 16:42:07 +02:00
Serhiy Storchaka f3ba0a74cd
gh-108416: Mark slow test methods with @requires_resource('cpu') (GH-108421)
Only mark tests which spend significant system or user time,
by itself or in subprocesses.
2023-09-02 07:45:34 +03:00
Guido van Rossum 76ce537fb1
Fix test_regrtest when run with uops always on (#108778)
The fix has two parts:
- When `-X uops` is detected, pass it on to the subprocess created to run the manufactured test.
  I need this so I can run `./python -Xuops -m test test_regrtest` and see it fail without the next fix.
- Use `-R 6:3:` in `ArgsTestCase.test_huntrleaks` instead of `-R 3:3:` -- it takes longer to settle with `-X uops`.
2023-09-01 16:27:09 -07:00
Nikita Sobolev 3b73f9f00e
gh-107862: Add roundtrip `hypothesis` tests to `test_binascii` (#107863) 2023-09-01 21:04:07 +00:00
Alex Waygood a1cbace91b
gh-105509: Simplify implementation of `typing.Annotated` (#105510) 2023-09-01 13:57:25 -07:00
Victor Stinner 23f54c1200
Make test_fcntl quiet (#108758)
Running test_fcntl logs two "struct.pack: ..." lines because
multiprocessing imports test_fcntl twice with
test.support.verbose=1. Move get_lockdata() inside TestFcntl test
case and only call it where it's needed, to stop logging these lines.
2023-09-01 14:59:09 +02:00
Irit Katriel 844f4c2e12
gh-108727: Fix segfault due to missing tp_dealloc definition for CounterOptimizer_Type (GH-108734) 2023-09-01 10:16:09 +01:00
Nikita Sobolev 044b8b3b6a
gh-107805: Fix signatures of module-level generated functions in `turtle` (#107807)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-09-01 01:18:15 -07:00
Victor Stinner ad73674283
gh-107603: Argument Clinic: Only include pycore_gc.h if needed (#108726)
Argument Clinic now only includes pycore_gc.h if PyGC_Head is needed,
and only includes pycore_runtime.h if _Py_ID() is needed.

* Add 'condition' optional argument to Clinic.add_include().
* deprecate_keyword_use() includes pycore_runtime.h when using
  the _PyID() function.
* Fix rendering of includes: comments start at the column 35.
* Mark PC/clinic/_wmimodule.cpp.h and
  "Objects/stringlib/clinic/*.h.h" header files as generated in
  .gitattributes.

Effects:

* 42 header files generated by AC no longer include the internal C
  API, instead of 4 header files before. For example,
  Modules/clinic/_abc.c.h no longer includes the internal C API.
* Fix _testclinic_depr.c.h: it now always includes pycore_runtime.h
  to get _Py_ID().
2023-08-31 23:42:34 +02:00
Victor Stinner 2bd960b579
gh-108337: Add pyatomic.h header (#108701)
This adds a new header that provides atomic operations on common data
types. The intention is that this will be exposed through Python.h,
although that is not the case yet. The only immediate use is in
the test file.

Co-authored-by: Sam Gross <colesbury@gmail.com>
2023-08-31 21:41:18 +00:00
Victor Stinner c6d56135e1
gh-108638: Fix tests when _stat extension is missing (#108689)
Fix test_inspect and test_pydoc when the _stat extension is missing.
Skip tests relying on _stat when _stat is missing.
2023-08-31 23:14:23 +02:00
Ethan Furman d48760b2f1
gh-108682: [Enum] raise TypeError if super().__new__ called in custom __new__ (GH-108704)
When overriding the `__new__` method of an enum, the underlying data type should be created directly; i.e. .

    member = object.__new__(cls)
    member = int.__new__(cls, value)
    member = str.__new__(cls, value)

Calling `super().__new__()` finds the lookup version of `Enum.__new__`, and will now raise an exception when detected.
2023-08-31 12:45:12 -07:00
Victor Stinner 13a00078b8
gh-108634: Py_TRACE_REFS uses a hash table (#108663)
Python built with "configure --with-trace-refs" (tracing references)
is now ABI compatible with Python release build and debug build.
Moreover, it now also supports the Limited API.

Change Py_TRACE_REFS build:

* Remove _PyObject_EXTRA_INIT macro.
* The PyObject structure no longer has two extra members (_ob_prev
  and _ob_next).
* Use a hash table (_Py_hashtable_t) to trace references (all
  objects): PyInterpreterState.object_state.refchain.
* Py_TRACE_REFS build is now ABI compatible with release build and
  debug build.
* Limited C API extensions can now be built with Py_TRACE_REFS:
  xxlimited, xxlimited_35, _testclinic_limited.
* No longer rename PyModule_Create2() and PyModule_FromDefAndSpec2()
  functions to PyModule_Create2TraceRefs() and
  PyModule_FromDefAndSpec2TraceRefs().
* _Py_PrintReferenceAddresses() is now called before
  finalize_interp_delete() which deletes the refchain hash table.
* test_tracemalloc find_trace() now also filters by size to ignore
  the memory allocated by _PyRefchain_Trace().

Test changes for Py_TRACE_REFS:

* Add test.support.Py_TRACE_REFS constant.
* Add test_sys.test_getobjects() to test sys.getobjects() function.
* test_exceptions skips test_recursion_normalizing_with_no_memory()
  and test_memory_error_in_PyErr_PrintEx() if Python is built with
  Py_TRACE_REFS.
* test_repl skips test_no_memory().
* test_capi skisp test_set_nomemory().
2023-08-31 18:33:34 +02:00
Victor Stinner 79823c103b
gh-106320: Remove private _PyErr_ChainExceptions() (#108713)
Remove _PyErr_ChainExceptions(), _PyErr_ChainExceptions1() and
_PyErr_SetFromPyStatus() functions from the public C API.

* Move the private _PyErr_ChainExceptions() and
  _PyErr_ChainExceptions1() function to the internal C API
  (pycore_pyerrors.h).
* Move the private _PyErr_SetFromPyStatus() to the internal C API
  (pycore_initconfig.h).
* No longer export the _PyErr_ChainExceptions() function.
* Move run_in_subinterp_with_config() from _testcapi to
  _testinternalcapi.
2023-08-31 13:53:19 +02:00
Carl Meyer 157b89e55e
gh-108696: revert bypassing import cache in test_import helper (#108698) 2023-08-30 18:15:31 -06:00
Carl Meyer d52c4482a8
gh-108654: restore comprehension locals before handling exception (#108659)
Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
2023-08-30 17:50:50 -06:00
Victor Stinner f59c66e8c8
gh-108297: Remove test_crashers (#108690)
The test was skipped in 2011 by
commit 89ba56d5fb.

Scripts in Lib/test/crashers/ do not crash on a reliable way. They
rely on undefined behaviors, like state of the stack memory, and so
may or may not crash. It is not worth it to make sure that they crash
in a continious integration, they should be run manually time to time
instead.
2023-08-30 21:33:04 +02:00
albanD add8d45cbe
gh-108520: Fix bad fork detection in nested multiprocessing use case (#108568)
gh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the `is_fork_ctx` attribute would be left missing after the first deserialization.

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
2023-08-30 17:07:41 +00:00
Erlend E. Aasland 2a3926fa51
gh-108590: Revert gh-108657 (commit 400a1cebc) (#108686)
Reverted per Serhiy's request.
2023-08-30 14:53:10 +00:00
Corvin 400a1cebc7
gh-108590: Fix sqlite3.iterdump for invalid Unicode in TEXT columns (#108657)
Co-authored-by: Erlend E. Aasland <erlend@python.org>
2023-08-30 09:06:21 +00:00
Serhiy Storchaka 210a5d7b8b
Revert "gh-103224: Use the realpath of the Python executable in `test_venv` (GH-103243)" (GH-108667)
This reverts commit 85b0b0cd94.

It broke builtbots.
2023-08-30 11:41:04 +03:00
Serhiy Storchaka d08d49dd09
Revert "Use non alternate name for Kyiv (GH-108533)" (GH-108649)
This reverts commit 7659128b9d.

It broke tests on the Debian and macOS buildbots.
2023-08-29 23:08:45 +03:00
Erlend E. Aasland 0b0c1d046c
gh-108278: Deprecate passing the first param of sqlite3.Connection callback APIs by keyword (#108632)
Deprecate passing the callback callable by keyword for the following
sqlite3.Connection APIs:

- set_authorizer(authorizer_callback)
- set_progress_handler(progress_handler, ...)
- set_trace_callback(trace_callback)

The affected parameters will become positional-only in Python 3.15.
2023-08-29 22:02:12 +02:00
Guido van Rossum 59e46932c8
gh-108488: Initialize JUMP_BACKWARD cache to 0, not 17 (#108591)
This mis-initialization caused the executor optimization to kick in sooner than intended. It also set the lower 4 bits of the counter to `1` -- those bits are supposed to be reserved (the actual counter is in the upper 12 bits).
2023-08-29 18:14:56 +00:00
Victor Stinner 83e191ba76
test_sys: remove debug print() (#108642) 2023-08-29 16:04:07 +00:00
Victor Stinner b62a76043e
gh-108638: Fix stat.filemode() when _stat is missing (#108639)
Change the pure Python implementation of stat.filemode() for unknown
file type: use "?", as done by the _stat.filemode().

test_stat skips TestFilemodeCStat if the _stat extension is missing.
2023-08-29 15:46:46 +00:00
Jochen Sprickerhof 7659128b9d
Use non alternate name for Kyiv (GH-108533)
tzdata provides Kiev as an alternative to Kyiv:

https://sources.debian.org/src/tzdata/2023c-10/backward/?hl=314#L314

But Debian moved it to the tzdata-legacy package breaking the test:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1050530

This patch switches to the name provided by tzdata.
2023-08-29 18:00:43 +03:00
Serhiy Storchaka ecb2bf02a4
gh-108617: Extend interactive session tests for sqlite3 (GH-108556) 2023-08-29 13:20:32 +03:00
Edward Schauman-Haigh 6eaddc10e9
gh-108558: Improve sqlite3 row factory tests (#108578)
Add test_sqlite_row_keys() to explicitly test sqlite3.Row.keys().

Cleanups:
- Reduce test noise by converting docstrings to regular comments
- Reduce boilerplate code by adding a setUp() method to RowFactoryTests

Co-authored-by: Erlend E. Aasland <erlend@python.org>
2023-08-29 08:51:36 +00:00
Adam Turner 0bd2ba553d
GH-107603: Clinic: Pass specific attributes to ``print_block()`` (#108581) 2023-08-28 21:31:45 +00:00
Matthias Bussonnier f75cefd402
gh-106670: Allow Pdb to move between chained exceptions (#106676) 2023-08-28 18:31:03 +00:00
Alex Waygood 242bef459b
gh-108494: Argument clinic: Improve the `parse_file()` API (#108575)
Co-authored-by: Victor Stinner <vstinner@python.org>
2023-08-28 18:25:16 +00:00
Erlend E. Aasland 4116592b6f
gh-108278: Deprecate passing the three first params as keyword args for sqlite3 UDF creation APIs (#108281)
Deprecate passing name, number of arguments, and the callable as keyword
arguments, for the following sqlite3.Connection APIs:

- create_function(name, nargs, callable, ...)
- create_aggregate(name, nargs, callable)

The affected parameters will become positional-only in Python 3.15.
2023-08-28 13:32:07 +00:00
Serhiy Storchaka bc5356bb5d
gh-108494: Argument Clinic: fix support of Limited C API (GH-108536) 2023-08-28 16:04:27 +03:00
Erlend E. Aasland d90973340b
gh-104683: Argument Clinic: Refactor the module and class resolver (#108552) 2023-08-28 14:41:05 +02:00
Erlend E. Aasland 0e8b3fc718
gh-108550: Speed up sqlite3 tests (#108551)
Refactor the CLI so we can easily invoke it and mock command-line
arguments. Adapt the CLI tests so we no longer have to launch a
separate process.

Disable the busy handler for all concurrency tests; we have full
control over the order of the SQLite C API calls, so we can safely
do this.

The sqlite3 test suite now completes ~8 times faster than before.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2023-08-28 14:17:34 +02:00
Erlend E. Aasland d0160c7c22
gh-64662: Add virtual table support to sqlite3.Connection.iterdump (#108340)
Co-authored-by: Aviv Palivoda <palaviv@gmail.com>
2023-08-28 00:18:32 +02:00
Jelle Zijlstra 482fad7f01
gh-108295: Fix crashes with TypeVar weakrefs (#108517) 2023-08-26 17:13:33 -07:00
denballakh e407cea193
gh-107406: Add better `struct.Struct` repr (#107407) 2023-08-26 15:24:16 +05:30
Victor Stinner 86bc9e35c4
gh-108494: AC supports pos-only args in limited C API (#108498)
AC now checks for "#define Py_LIMITED_API" pattern to use the limited
C API.
2023-08-26 00:39:24 +02:00
Victor Stinner 1dd9510977
gh-108494: Argument Clinic partial supports of Limited C API (#108495)
Argument Clinic now has a partial support of the
Limited API:

* Add --limited option to clinic.c.
* Add '_testclinic_limited' extension which is built with
  the limited C API version 3.13.
* For now, hardcode in clinic.py that "_testclinic_limited.c" targets
  the limited C API.
2023-08-25 23:22:08 +02:00
Guido van Rossum 5347018409
gh-108311: Fix test_store_attr_with_hint by disabling optimizer in decorator (#108312)
See https://github.com/python/cpython/issues/108311#issuecomment-1693569380

---------

Co-authored-by: AlexWaygood <alex.waygood@gmail.com>
2023-08-25 16:40:27 +00:00
Mark Shannon bc2f9e6258
gh-107932: Fix merge conflict in test_dis (GH-108478)
Fix merge conflict in test_dis
2023-08-25 13:02:10 +01:00
Serhiy Storchaka 4ae3edf300
gh-108418: Speed up bigmem compression tests in dry mode (GH-108419)
Only generate and compress small amount of random data in dry run.
2023-08-25 12:37:27 +03:00
Corvin 24aa249a66
gh-107932: Fix dis module for bytecode that does not have an associated source line (GH-107988) 2023-08-25 09:31:26 +01:00
Peeyush Aggarwal 8d4052075e
gh-103384: Generalize the regex pattern `BaseConfigurator.INDEX_PATTERN` to allow spaces and non-alphanumeric characters in keys. (GH-103391)
Co-authored-by: Vinay Sajip <vinay_sajip@yahoo.co.uk>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Erlend E. Aasland <erlend@python.org>
2023-08-25 09:15:26 +01:00
Victor Stinner 4e5a7284ee
gh-108444: Argument Clinic uses PyLong_AsInt() (#108458)
Argument Clinic now uses the new public PyLong_AsInt(), rather than
the old name _PyLong_AsInt().
2023-08-25 00:51:22 +02:00
Victor Stinner be436e08b8
gh-108444: Add PyLong_AsInt() public function (#108445)
* Rename _PyLong_AsInt() to PyLong_AsInt().
* Add documentation.
* Add test.
* For now, keep _PyLong_AsInt() as an alias to PyLong_AsInt().
2023-08-24 23:55:30 +02:00
Filipe Laíns feb9a49c9c
GH-103247: bypass the import cache on the _require_loader helper 2023-08-24 21:05:03 +00:00
Victor Stinner 26893016a7
gh-106320: Remove private _PyDict functions (#108449)
Move private functions to the internal C API (pycore_dict.h):

* _PyDictView_Intersect()
* _PyDictView_New()
* _PyDict_ContainsId()
* _PyDict_DelItemId()
* _PyDict_DelItem_KnownHash()
* _PyDict_GetItemIdWithError()
* _PyDict_GetItem_KnownHash()
* _PyDict_HasSplitTable()
* _PyDict_NewPresized()
* _PyDict_Next()
* _PyDict_Pop()
* _PyDict_SetItemId()
* _PyDict_SetItem_KnownHash()
* _PyDict_SizeOf()

No longer export most of these functions.

Move also the _PyDictViewObject structure to the internal C API.

Move dict_getitem_knownhash() function from _testcapi to the
_testinternalcapi extension. Update test_capi.test_dict for this
change.
2023-08-24 20:01:50 +00:00
Victor Stinner 995f4c48e1
gh-80527: Change support.requires_legacy_unicode_capi() (#108438)
The decorator now requires to be called with parenthesis:

    @support.requires_legacy_unicode_capi()

instead of:

    @support.requires_legacy_unicode_capi

The implementation now only imports _testcapi when the decorator is
called, so "import test.support" no longer imports the _testcapi
extension.
2023-08-24 20:09:23 +02:00
Victor Stinner 480a337366
gh-106320: Remove private _PyContext_NewHamtForTests() (#108434)
Move the function to the internal C API.
2023-08-24 19:37:41 +02:00
Victor Stinner aa6f787faa
gh-108388: Convert test_concurrent_futures to package (#108401)
Convert test_concurrent_futures to a package of sub-tests.
2023-08-24 19:21:44 +02:00
Victor Stinner 6726626646
gh-108314: Add PyDict_ContainsString() function (#108323) 2023-08-24 15:59:12 +02:00
Chris Markiewicz 2eb60c1934
gh-108111: Flush gzip write buffer before seeking, fixing bad writes (#108341)
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
2023-08-24 10:23:01 +02:00
Victor Stinner aa9a359ca2
gh-108388: Split test_multiprocessing_spawn (#108396)
Split test_multiprocessing_fork, test_multiprocessing_forkserver and
test_multiprocessing_spawn into test packages. Each package is made
of 4 sub-tests: processes, threads, manager and misc. It allows
running more tests in parallel and so reduce the total test duration.
2023-08-24 05:35:39 +02:00
Victor Stinner 174e9da083
gh-108388: regrtest splits test_asyncio package (#108393)
Currently, test_asyncio package is only splitted into sub-tests when
using command "./python -m test". With this change, it's also
splitted when passing it on the command line:
"./python -m test test_asyncio".

Remove the concept of "STDTESTS". Python is now mature enough to not
have to bother with that anymore. Removing STDTESTS simplify the
code.
2023-08-24 02:44:58 +00:00
Victor Stinner 7a6cc3eb66
test_peg_generator and test_freeze require cpu (#108386)
The test_peg_generator and test_tools.test_freeze tests now require
the 'cpu' resource. Skip these tests unless the 'cpu' resource is
enabled (it is disabled by default).

These tests are no longer skipped if Python is built with ASAN or
MSAN sanitizer.
2023-08-24 00:52:48 +02:00
Victor Stinner 592bacb6fc
gh-108342: Make ssl TestPreHandshakeClose more reliable (#108370)
* In preauth tests of test_ssl, explicitly break reference cycles
  invoving SingleConnectionTestServerThread to make sure that the
  thread is deleted. Otherwise, the test marks the environment as
  altered because the threading module sees a "dangling thread"
  (SingleConnectionTestServerThread). This test leak was introduced
  by the test added for the fix of issue gh-108310.
* Use support.SHORT_TIMEOUT instead of hardcoded 1.0 or 2.0 seconds
  timeout.
* SingleConnectionTestServerThread.run() catchs TimeoutError
* Fix a race condition (missing synchronization) in
  test_preauth_data_to_tls_client(): the server now waits until the
  client connect() completed in call_after_accept().
* test_https_client_non_tls_response_ignored() calls server.join()
  explicitly.
* Replace "localhost" with server.listener.getsockname()[0].
2023-08-23 21:57:11 +00:00
albanD 1700d34d31
gh-77377: Ensure multiprocessing SemLock is valid for spawn-based Process before serializing it (#107275)
Ensure multiprocessing SemLock is valid for spawn Process before serializing it.

Creating a multiprocessing SemLock with a fork context, and then trying to pass it to a spawn-created Process, would segfault if not detected early.

---------

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
2023-08-23 20:27:35 +00:00
Nikita Sobolev 3f61cf646d
gh-108303: Move `ann_module*.py` files to `typinganndata/` folder (#108354) 2023-08-23 17:42:08 +02:00
Serhiy Storchaka 154477be72
gh-50002: xml.dom.minidom now preserves whitespaces in attributes (GH-107947)
Also double quotes (") are now only quoted in attributes.
2023-08-23 15:23:41 +03:00
Mariusz Felisiak 29bc6165ab
gh-105539: Fix ResourceWarning from unclosed SQLite connections in test_sqlite3 (#108360)
Follow up to 1a1bfc2891.

Explicitly manage connections in:

- test_audit.test_sqlite3
- test_sqlite3.test_audit
- test_sqlite3.test_backup

Co-authored-by: Erlend E. Aasland <erlend@python.org>
2023-08-23 10:10:08 +00:00
Petr Viktorin 31b61d19ab
gh-108294: Add time.sleep audit event (GH-108298) 2023-08-23 11:00:22 +02:00
Irit Katriel 2dfbd4f36d
gh-108113: Make it possible to optimize an AST (#108282) 2023-08-23 09:01:17 +01:00
Victor Stinner 9173b2bbe1
gh-105776: Fix test_cppext when CC contains -std=c11 option (#108343)
Fix test_cppext when the C compiler command has the "-std=c11" option.
Remove "-std=" options from the compiler command.
2023-08-23 02:52:56 +00:00
Victor Stinner 3a1ac87f8f
gh-90791: test.pythoninfo logs ASAN_OPTIONS env var (#108289)
* Cleanup libregrtest code logging ASAN_OPTIONS.
* Fix a typo on "ASAN_OPTIONS" vs "MSAN_OPTIONS".
2023-08-23 01:39:50 +02:00
Victor Stinner 21dda09600
gh-108303: Add Lib/test/test_cppext/ sub-directory (#108325)
* Move test_cppext to its own directory
* Rename setup_testcppext.py to setup.py
* Rename _testcppext.cpp to extension.cpp
* The source (extension.cpp) is now also copied by the test.
2023-08-22 18:30:18 +00:00
Łukasz Langa 0cb0c238d5
gh-108310: Fix CVE-2023-40217: Check for & avoid the ssl pre-close flaw (#108315)
Instances of `ssl.SSLSocket` were vulnerable to a bypass of the TLS handshake
and included protections (like certificate verification) and treating sent
unencrypted data as if it were post-handshake TLS encrypted data.

The vulnerability is caused when a socket is connected, data is sent by the
malicious peer and stored in a buffer, and then the malicious peer closes the
socket within a small timing window before the other peers’ TLS handshake can
begin. After this sequence of events the closed socket will not immediately
attempt a TLS handshake due to not being connected but will also allow the
buffered data to be read as if a successful TLS handshake had occurred.

Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
2023-08-22 19:53:15 +02:00
Victor Stinner d2879f2095
gh-108303: Remove unused Lib/test/sgml_input.html (#108305)
In Python 2.7, the file was used by Lib/test/test_sgmllib.py to test
Lib/sgmllib.py. The sgmllib module and its tests have been removed in
Python 3.0.
2023-08-22 19:12:51 +02:00
Victor Stinner adfc118fda
gh-106016: Add Lib/test/test_module/ directory (#108293)
* Move Python scripts related to test_module to this new directory:
  good_getattr.py and bad_getattrX.py scripts.
* Move Lib/test/test_module.py to Lib/test/test_module/__init__.py.
2023-08-22 16:53:49 +02:00
Victor Stinner a541e01537
gh-90791: Enable test___all__ on ASAN build (#108286)
* Only skip modules and tests related to X11 on ASAN builds: run
  other tests with ASAN.
* Use print(flush=True) to see output earlier when it's redirected to
  a pipe.
* Update issue reference: replace bpo-46633 with gh-90791.
2023-08-22 15:52:32 +02:00
Erlend E. Aasland 1a1bfc2891
gh-105539: Emit ResourceWarning if sqlite3 database is not closed explicitly (#108015) 2023-08-22 13:10:29 +02:00
Dennis Sweeney 86617518c4
gh-108179: Add error message for parser stack overflows (#108256) 2023-08-22 08:41:50 +01:00
Victor Stinner 58f9c63500
Fix test_faulthandler for sanitizers (#108245)
Set environment options to ask sanitizers to not handle SIGSEGV.

This change allows running test_enable_fd() and test_enable_file()
with sanitizers. Previously, they were skipped.
2023-08-22 01:16:12 +00:00
Steve Dower de33b5c662
gh-106242: Make ntpath.realpath errors consistent with abspath when there are embedded nulls (GH-108248)
* gh-106242: Make ntpath.realpath errors consistent with abspath when there are embedded nulls

* Update 2023-08-22-00-36-57.gh-issue-106242.q24ITw.rst

mention Windows and the former incorrect ValueError.

---------

Co-authored-by: Gregory P. Smith <greg@krypto.org>
2023-08-22 00:16:02 +00:00
Victor Stinner 531930f47f
Fix test_generators: save/restore warnings filters (#108246)
Previously, depending on existing filters, the test
could modify the warnings and so fail as "env changed".
2023-08-21 23:41:13 +00:00
Irit Katriel a1cc74c4ee
gh-107901: Fix missing line number on BACKWARD_JUMP at the end of a for loop (#108242) 2023-08-21 23:44:31 +01:00
Dong-hee Na e6db23f66d
gh-107265: Fix code_hash for ENTER_EXECUTOR case (#108188) 2023-08-21 14:22:18 -07:00
Victor Stinner 5afe0c17ca
gh-108223: test.pythoninfo and libregrtest log Py_NOGIL (#108238)
Enable with --disable-gil --without-pydebug:

    $ make pythoninfo|grep NOGIL
    sysconfig[Py_NOGIL]: 1

    $ ./python -m test
    ...
    == Python build: nogil debug
    ...
2023-08-21 22:16:23 +02:00
Irit Katriel 10a91d7e98
gh-108113: Make it possible to create an optimized AST (#108154) 2023-08-21 16:31:30 +00:00
Serhiy Storchaka 60942cccb1
gh-95065, gh-107704: Argument Clinic: support multiple '/ [from ...]' and '* [from ...]' markers (GH-108132) 2023-08-21 13:59:58 +00:00
Nikita Sobolev 13104f3b74
gh-107905: Test raising `__value__` for `TypeAliasType` (#107997) 2023-08-21 13:52:37 +00:00
balmeida-nokia 37135d25e2
gh-107396: tarfiles: set self.exception before _init_read_gz() (GH-107485)
In the stack call of: _init_read_gz()
```
_read, tarfile.py:548
read, tarfile.py:526
_init_read_gz, tarfile.py:491
```
a try;except exists that uses `self.exception`, so it needs to be set before
calling _init_read_gz().
2023-08-21 11:39:06 +00:00
Petr Viktorin acbd3f9c5c
gh-107845: Fix symlink handling for tarfile.data_filter (GH-107846)
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Lumír 'Frenzy' Balhar <frenzy.madness@gmail.com>
2023-08-21 12:56:46 +02:00
Raymond Hettinger 20cc90c0df
gh-105736: Sync pure python version of OrderedDict with the C version (#108098) 2023-08-21 11:49:08 +02:00
Joon Hwan 김준환 014a5b71e7
gh-107895: Fix test_asyncio.test_runners when run it in CPython's "development mode" (GH-108168) 2023-08-21 07:55:09 +00:00
Dong-hee Na 4fdf3fda0f
gh-107265: Fix code_richcompare for ENTER_EXECUTOR case (gh-108165) 2023-08-21 05:50:09 +00:00
Dong-hee Na 04f7875c44
gh-107526: Fix test_module_level_callable_unrepresentable_default (gh-108187) 2023-08-21 04:51:31 +00:00
Serhiy Storchaka 633ea217a8
gh-107915: Handle errors in C API functions PyErr_Set*() and PyErr_Format() (GH-107918)
Such C API functions as PyErr_SetString(), PyErr_Format(),
PyErr_SetFromErrnoWithFilename() and many others no longer crash or
ignore errors if it failed to format the error message or decode the
filename. Instead, they keep a corresponding error.
2023-08-19 14:51:03 +03:00
Serhiy Storchaka 79db9d9a0e
gh-72684: Tkinter: provide interface for "tk busy" subcommands (GH-107684)
Add tkinter.Misc methods: tk_busy_hold(), tk_busy_configure(), tk_busy_cget(),
tk_busy_forget(), tk_busy_current(), and tk_busy_status().
2023-08-19 14:48:02 +03:00
Serhiy Storchaka 2f311437cd
gh-107704: Argument Clinic: add support for deprecating keyword use of parameters (GH-107984)
It is now possible to deprecate passing keyword arguments for
keyword-or-positional parameters with Argument Clinic, using the new
'/ [from X.Y]' syntax.
(To be read as "positional-only from Python version X.Y")

Co-authored-by: Erlend E. Aasland <erlend@python.org>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2023-08-19 10:13:35 +03:00