cpython/Tools
Victor Stinner c53c3f4000
bpo-42398: Fix "make regen-all" race condition (GH-23362) (GH-23367)
Fix a race condition in "make regen-all" when make -jN option is used
to run jobs in parallel. The clinic.py script now only use atomic
write to write files. Moveover, generated files are now left
unchanged if the content does not change, to not change the file
modification time.

The "make regen-all" command runs "make clinic" and "make
regen-importlib" targets:

* "make regen-importlib" builds object files (ex: Modules/_weakref.o)
  from source files (ex: Modules/_weakref.c) and clinic files (ex:
  Modules/clinic/_weakref.c.h)
* "make clinic" always rewrites all clinic files
  (ex: Modules/clinic/_weakref.c.h)

Since there is no dependency between "clinic" and "regen-importlib"
Makefile targets, these two targets can be run in parallel. Moreover,
half of clinic.py file writes are not atomic and so there is a race
condition when "make regen-all" runs jobs in parallel using make -jN
option (which can be passed in MAKEFLAGS environment variable).

Fix clinic.py to make all file writes atomic:

* Add write_file() function to ensure that all file writes are
  atomic: write into a temporary file and then use os.replace().
* Moreover, write_file() doesn't recreate or modify the file if the
  content does not change to avoid modifying the file modification
  file.
* Update test_clinic to verify these assertions with a functional
  test.
* Remove Clinic.force attribute which was no longer used, whereas
  Clinic.verify remains useful.

(cherry picked from commit 8fba9523cf)
2020-11-18 17:11:09 +01:00
..
buildbot bpo-41173: Copy test results file from ARM worker before uploading (GH-21305) 2020-07-07 16:45:17 -07:00
c-analyzer bpo-41971: Fix test failure in test.test_tools.test_c_analyzer when mutating global state (GH-22652) (GH-22653) 2020-10-11 21:54:35 +01:00
ccbench bpo-38839: Fix some unused functions in tests (GH-17189) 2019-11-19 11:45:20 -08:00
clinic bpo-42398: Fix "make regen-all" race condition (GH-23362) (GH-23367) 2020-11-18 17:11:09 +01:00
demo bpo-38539: Finish rename of ss1.py to spreadsheet.py (GH-16896) 2019-10-22 21:05:02 -07:00
freeze bpo-22831: Use "with" to avoid possible fd leaks in tools (part 2). (GH-10927) 2019-03-30 08:33:02 +02:00
gdb bpo-40019: Skip test_gdb if Python was optimized (GH-19081) 2020-03-20 08:23:26 +01:00
i18n bpo-22831: Use "with" to avoid possible fd leaks in tools (part 2). (GH-10927) 2019-03-30 08:33:02 +02:00
importbench Tools/importbench: Fix a misplaced stderr= (GH-12690) 2019-04-05 09:18:19 +02:00
iobench Replace KB unit with KiB (#4293) 2017-11-08 14:44:44 -08:00
msi bpo-40754: Adds _testinternalcapi to Windows installer for test suite (GH-23271) 2020-11-13 14:47:27 -08:00
nuget bpo-41744: Package python.props with correct name in NuGet package (GH-22154) 2020-10-21 14:15:18 +09:00
peg_generator [3.9] bpo-42218: Correctly handle errors in left-recursive rules (GH-23065) (GH-23066) 2020-10-31 12:06:03 -07:00
pynche Remove code commented for more than 10 years (GH-16965) 2019-10-28 22:35:31 +01:00
scripts closes bpo-40385: Remove Tools/scripts/checkpyc.py (GH-19709) 2020-04-24 23:33:59 -05:00
ssl [3.9] bpo-41043: Escape literal part of the path for glob(). (GH-20994). (GH-21275) 2020-07-02 10:05:16 +03:00
stringbench Issue #19936: Added executable bits or shebang lines to Python scripts which 2014-01-16 17:33:23 +02:00
test2to3 Issue #19936: Remove executable bits from C source files and several forgotten 2014-01-16 18:48:45 +02:00
tz bpo-29919: Remove unused imports found by pyflakes (#137) 2017-03-27 16:05:26 +02:00
unicode bpo-40328: Add tool for generating cjk mapping headers (GH-19602) 2020-04-30 02:34:24 +09:00
unittestgui Removed unintentional trailing spaces in text files. 2015-03-29 19:12:58 +03:00
README bpo-40334: PEP 617 implementation: New PEG parser for CPython (GH-19503) 2020-04-22 23:29:27 +01:00

README

This directory contains a number of Python programs that are useful
while building or extending Python.

buildbot        Batchfiles for running on Windows buildbot workers.

ccbench         A Python threads-based concurrency benchmark. (*)

demo            Several Python programming demos.

freeze          Create a stand-alone executable from a Python program.

gdb             Python code to be run inside gdb, to make it easier to
                debug Python itself (by David Malcolm).

i18n            Tools for internationalization. pygettext.py
                parses Python source code and generates .pot files,
                and msgfmt.py generates a binary message catalog
                from a catalog in text format.

iobench         Benchmark for the new Python I/O system. (*)

msi             Support for packaging Python as an MSI package on Windows.

parser          Un-parsing tool to generate code from an AST.

peg_generator   PEG-based parser generator (pegen) used for new parser.

pynche          A Tkinter-based color editor.

scripts         A number of useful single-file programs, e.g. tabnanny.py
                by Tim Peters, which checks for inconsistent mixing of
                tabs and spaces, and 2to3, which converts Python 2 code
                to Python 3 code.

stringbench     A suite of micro-benchmarks for various operations on
                strings (both 8-bit and unicode). (*)

test2to3        A demonstration of how to use 2to3 transparently in setup.py.

unicode         Tools for generating unicodedata and codecs from unicode.org
                and other mapping files (by Fredrik Lundh, Marc-Andre Lemburg
                and Martin von Loewis).

unittestgui     A Tkinter based GUI test runner for unittest, with test
                discovery.


(*) A generic benchmark suite is maintained separately at https://github.com/python/performance