Commit Graph

3080 Commits

Author SHA1 Message Date
Victor Stinner c53c3f4000
bpo-42398: Fix "make regen-all" race condition (GH-23362) (GH-23367)
Fix a race condition in "make regen-all" when make -jN option is used
to run jobs in parallel. The clinic.py script now only use atomic
write to write files. Moveover, generated files are now left
unchanged if the content does not change, to not change the file
modification time.

The "make regen-all" command runs "make clinic" and "make
regen-importlib" targets:

* "make regen-importlib" builds object files (ex: Modules/_weakref.o)
  from source files (ex: Modules/_weakref.c) and clinic files (ex:
  Modules/clinic/_weakref.c.h)
* "make clinic" always rewrites all clinic files
  (ex: Modules/clinic/_weakref.c.h)

Since there is no dependency between "clinic" and "regen-importlib"
Makefile targets, these two targets can be run in parallel. Moreover,
half of clinic.py file writes are not atomic and so there is a race
condition when "make regen-all" runs jobs in parallel using make -jN
option (which can be passed in MAKEFLAGS environment variable).

Fix clinic.py to make all file writes atomic:

* Add write_file() function to ensure that all file writes are
  atomic: write into a temporary file and then use os.replace().
* Moreover, write_file() doesn't recreate or modify the file if the
  content does not change to avoid modifying the file modification
  file.
* Update test_clinic to verify these assertions with a functional
  test.
* Remove Clinic.force attribute which was no longer used, whereas
  Clinic.verify remains useful.

(cherry picked from commit 8fba9523cf)
2020-11-18 17:11:09 +01:00
Miss Islington (bot) faadc52e75
bpo-40754: Adds _testinternalcapi to Windows installer for test suite (GH-23271)
(cherry picked from commit 9b6934230c)

Co-authored-by: Steve Dower <steve.dower@python.org>
2020-11-13 14:47:27 -08:00
Miss Skeleton (bot) e899c016ec
Enable signing of nuget.org packages and update to supported timestamp server (GH-23132)
(cherry picked from commit db6434c474)

Co-authored-by: Steve Dower <steve.dower@python.org>
2020-11-03 14:52:41 -08:00
Lysandros Nikolaou cfcb952e30
[3.9] bpo-42218: Correctly handle errors in left-recursive rules (GH-23065) (GH-23066)
Left-recursive rules need to check for errors explicitly, since
even if the rule returns NULL, the parsing might continue and lead
to long-distance failures.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
(cherry picked from commit 02cdfc93f8)

Automerge-Triggered-By: GH:lysnikolaou
2020-10-31 12:06:03 -07:00
Lysandros Nikolaou 24a7c298d4
[3.9] bpo-42123: Run the parser two times and only enable invalid rules on the second run (GH-22111) (GH-23011)
* Implement running the parser a second time for the errors messages

The first parser run is only responsible for detecting whether
there is a `SyntaxError` or not. If there isn't the AST gets returned.
Otherwise, the parser is run a second time with all the `invalid_*`
rules enabled so that all the customized error messages get produced.

(cherry picked from commit bca7014032)
2020-10-28 02:14:15 +02:00
Lysandros Nikolaou 8aee019d71
[3.9] Remove git conflict lines from test_parse_directory in peg_generator (GH-23007) 2020-10-28 00:37:36 +02:00
Miss Skeleton (bot) d0bfce992c
bpo-41744: Package python.props with correct name in NuGet package (GH-22154)
NuGet automatically includes .props file from the build directory in the
target using the package, but only if the .props file has the correct
name: it must be $(id).props

Rename python.props correspondingly in all the nuspec variants. Also
keep python.props as it were for backward compatibility.
(cherry picked from commit 7c11a9acca)

Co-authored-by: Václav Slavík <vaclav@slavik.io>
2020-10-21 14:15:18 +09:00
Miss Skeleton (bot) 4af6729219
bpo-41971: Fix test failure in test.test_tools.test_c_analyzer when mutating global state (GH-22652) (GH-22653)
(cherry picked from commit 47ecfd8030)

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-10-11 21:54:35 +01:00
Pablo Galindo 270e249290
[3.9] Fix 'gather' rules in the python parser generator (GH-22021) (GH-22080)
Currently, empty sequences in gather rules make the conditional for
gather rules fail as empty sequences evaluate as "False". We need to
explicitly check for "None" (the failure condition) to avoid false
negatives.
(cherry picked from commit e55a0e9)

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-09-03 16:35:36 +01:00
Miss Islington (bot) 75c2281762
bpo-41526: Fixed layout of final page of the installer (GH-21871)
(cherry picked from commit 6444ca9469)

Co-authored-by: Steve Dower <steve.dower@python.org>
2020-08-13 16:55:37 -07:00
Miss Islington (bot) 95cc37f6b8
bpo-41412 and bpo-40948: Windows installer updates (GH-21656)
Prevent installation on Windows 8 and earlier.
Download UCRT on demand when required (non-updated Windows 8.1 only)
Add reference to py launcher to post-install message
(cherry picked from commit 37a06cbe5c)

Co-authored-by: Steve Dower <steve.dower@python.org>
2020-07-28 08:55:38 -07:00
Miss Islington (bot) 366cfc65f2
bpo-41173: Copy test results file from ARM worker before uploading (GH-21305)
(cherry picked from commit 10772ec150)

Co-authored-by: Steve Dower <steve.dower@python.org>
2020-07-07 16:45:17 -07:00
Pablo Galindo 54f115dd53
[3.9] bpo-41215: Don't use NULL by default in the PEG parser keyword list (GH-21355) (GH-21356)
(cherry picked from commit 39e76c0fb0)

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>

Automerge-Triggered-By: @lysnikolaou
2020-07-06 12:29:59 -07:00
Serhiy Storchaka ecfecc2d6c
[3.9] bpo-41043: Escape literal part of the path for glob(). (GH-20994). (GH-21275)
(cherry picked from commit 9355868458)
2020-07-02 10:05:16 +03:00
Miss Islington (bot) 9862181750
bpo-41044: Generate valid PEG python parsers for opt+seq rules (GH-20995)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
(cherry picked from commit 55460ee6dc)

Co-authored-by: Batuhan Taskaya <isidentical@gmail.com>
2020-06-20 10:59:32 -07:00
Pablo Galindo 145fa86598
[3.9] Include soft keywords in keyword.py (GH-20877). (GH-20880)
(cherry picked from commit 78319e373d)

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-06-15 04:31:05 +01:00
Miss Islington (bot) 961edf7979
bpo-40939: Generate keyword.py using the new parser (GH-20800)
(cherry picked from commit 9727694f08)

Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
2020-06-11 06:02:37 -07:00
Miss Islington (bot) 18f1226884
Refactor scripts in Tools/peg_generator/scripts (GH-20401)
(cherry picked from commit ba6fd87e41)

Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
2020-06-05 21:41:12 -07:00
Miss Islington (bot) 24a88b008b
bpo-30008: Fix OpenSSL no-deprecated compilation (GH-20397)
Fix :mod:`ssl`` code to be compatible with OpenSSL 1.1.x builds that use
``no-deprecated`` and ``--api=1.1.0``.

Note: Tests assume full OpenSSL API and fail with limited API.

Signed-off-by: Christian Heimes <christian@python.org>
Co-authored-by: Mark Wright <gienah@gentoo.org>
(cherry picked from commit a871f692b4)

Co-authored-by: Christian Heimes <christian@python.org>
2020-06-01 00:18:39 -07:00
Lysandros Nikolaou 1bfe659ee5
[3.9] Backport GH-20370 and GH-20436: Soft keywords (GH-20458) 2020-05-27 21:20:07 +01:00
Miss Islington (bot) fda47659bb
Fix peg_generator compiler warnings under MSVC (GH-20405)
(cherry picked from commit a2bbedc8b1)

Co-authored-by: Ammar Askar <ammar@ammaraskar.com>
2020-05-25 21:52:21 -07:00
Miss Islington (bot) 3c6c86ab77
bpo-40688: Use the correct parser in the peg_generator scripts (GH-20235)
The scripts in `Tools/peg_generator/scripts` mostly assume that
`ast.parse` and `compile` use the old parser, since this was the
state of things, while we were developing them. They need to be
updated to always use the correct parser. `_peg_parser` is being
extended to support both parsing and compiling with both parsers.
(cherry picked from commit 9645930b5b)

Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
2020-05-25 13:11:36 -07:00
Miss Islington (bot) 31084be618
bpo-40750: Do not expand the new parser debug flags if Py_BUILD_CORE is not defined (GH-20393)
(cherry picked from commit deb4355a37)

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-05-25 12:37:56 -07:00
Miss Islington (bot) 82da2c3eb4
bpo-40750: Support -d flag in the new parser (GH-20340)
(cherry picked from commit 800a35c623)

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-05-25 10:58:03 -07:00
Miss Islington (bot) 82c274e3ba
bpo-40334: Support suppressing of multiple optional variables in Pegen (GH-20367)
(cherry picked from commit cba5031510)

Co-authored-by: Batuhan Taskaya <batuhanosmantaskaya@gmail.com>
2020-05-24 15:38:47 -07:00
Miss Islington (bot) 427c696400
Fix debug output in PEG parser generator (GH-20308)
(cherry picked from commit b831129123)

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-05-21 19:05:20 -07:00
Pablo Galindo 7178b1bb68
[3.9] Fix typing problems reported by mypy in pegen (GH-20297) (GH-20300)
(cherry picked from commit d10fef35c6)

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-05-21 22:09:43 +01:00
Miss Islington (bot) c06983bc30
bpo-39631: Fix file association MIME type in the Windows installer (GH-20205)
Use text/x-python instead of text/plain to avoid issues with tools assuming that "ShellExecute(script)" is a non-executable operation.
(cherry picked from commit 8c862e5124)

Co-authored-by: Zackery Spytz <zspytz@gmail.com>
2020-05-19 05:39:16 -07:00
Miss Islington (bot) 3d062829de
bpo-40669: Use requirements.pip when installing PEG dependencies (GH-20194)
(cherry picked from commit 3764069f3b)

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-05-18 19:31:28 -07:00
Lysandros Nikolaou dc31800f86
bpo-40669: Install PEG benchmarking dependencies in a venv (GH-20183)
Create a `make venv` target, that creates a virtual environment
and installs the dependency in that venv. `make time` and all
the related targets are changed to use the virtual environment
python.

Automerge-Triggered-By: @pablogsal
2020-05-18 11:27:40 -07:00
Lysandros Nikolaou 7b7a21bc4f
bpo-40661: Fix segfault when parsing invalid input (GH-20165)
Fix segfaults when parsing very complex invalid input, like `import äˆ ð£„¯ð¢·žð±‹á”€ð””ð‘©±å®ä±¬ð©¾\n𗶽`.

Co-authored-by: Guido van Rossum <guido@python.org>
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2020-05-18 18:32:03 +01:00
Lysandros Nikolaou 2c8cd06afe
bpo-40334: Improvements to error-handling code in the PEG parser (GH-20003)
The following improvements are implemented in this commit:
- `p->error_indicator` is set, in case malloc or realloc fail.
- Avoid memory leaks in the case that realloc fails.
- Call `PyErr_NoMemory()` instead of `PyErr_Format()`, because it requires no memory.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-05-17 04:19:23 +01:00
Paul Ganssle 62972d9d73
bpo-40503: PEP 615: Tests and implementation for zoneinfo (GH-19909)
This is the initial implementation of PEP 615, the zoneinfo module,
ported from the standalone reference implementation (see
https://www.python.org/dev/peps/pep-0615/#reference-implementation for a
link, which has a more detailed commit history).

This includes (hopefully) all functional elements described in the PEP,
but documentation is found in a separate PR. This includes:

1. A pure python implementation of the ZoneInfo class
2. A C accelerated implementation of the ZoneInfo class
3. Tests with 100% branch coverage for the Python code (though C code
   coverage is less than 100%).
4. A compile-time configuration option on Linux (though not on Windows)

Differences from the reference implementation:

- The module is arranged slightly differently: the accelerated module is
  `_zoneinfo` rather than `zoneinfo._czoneinfo`, which also necessitates
  some changes in the test support function. (Suggested by Victor
  Stinner and Steve Dower.)
- The tests are arranged slightly differently and do not include the
  property tests. The tests live at test/test_zoneinfo/test_zoneinfo.py
  rather than test/test_zoneinfo.py or test/test_zoneinfo/__init__.py
  because we may do some refactoring in the future that would likely
  require this separation anyway; we may:
        - include the property tests
        - automatically run all the tests against both pure Python and C,
          rather than manually constructing C and Python test classes (similar
          to the way this works with test_datetime.py, which generates C
          and Python test cases from datetimetester.py).
- This includes a compile-time configuration option on Linux (though not
  on Windows); added with much help from Thomas Wouters.
- Integration into the CPython build system is obviously different from
  building a standalone zoneinfo module wheel.
- This includes configuration to install the tzdata package as part of
  CI, though only on the coverage jobs. Introducing a PyPI dependency as
  part of the CI build was controversial, and this is seen as less of a
  major change, since the coverage jobs already depend on pip and PyPI.

Additional changes that were introduced as part of this PR, most / all of
which were backported to the reference implementation:

- Fixed reference and memory leaks

    With much debugging help from Pablo Galindo

- Added smoke tests ensuring that the C and Python modules are built

    The import machinery can be somewhat fragile, and the "seamlessly falls
    back to pure Python" nature of this module makes it so that a problem
    building the C extension or a failure to import the pure Python version
    might easily go unnoticed.

- Adjustments to zoneinfo.__dir__

    Suggested by Petr Viktorin.

- Slight refactorings as suggested by Steve Dower.

- Removed unnecessary if check on std_abbr

    Discovered this because of a missing line in branch coverage.
2020-05-16 10:20:06 +02:00
Christian Heimes 6e8cda91d9
bpo-40457: Support OpenSSL without TLS 1.0/1.1 (GH-19862)
OpenSSL can be build without support for TLS 1.0 and 1.1. The ssl module
now correctly adheres to OPENSSL_NO_TLS1 and OPENSSL_NO_TLS1_1 flags.

Also update multissltest to test with latest OpenSSL and LibreSSL
releases.

Signed-off-by: Christian Heimes <christian@python.org>

Automerge-Triggered-By: @tiran
2020-05-15 18:33:05 -07:00
Christian Heimes 938717fd04
bpo-40163: Fix multissltest download of old OpenSSL (GH-19329) 2020-05-15 22:32:25 +02:00
Christian Heimes 62d618c06b
bpo-40479: Test with latest OpenSSL versions (GH-20108)
* 1.0.2u (EOL)
* 1.1.0l (EOL)
* 1.1.1g
* 3.0.0-alpha2 (disabled for now)

Build the FIPS provider and create a FIPS configuration file for OpenSSL
3.0.0.

Signed-off-by: Christian Heimes <christian@python.org>

Automerge-Triggered-By: @tiran
2020-05-15 09:48:25 -07:00
Victor Stinner 97e1568325
bpo-38787: Fix Argument Clinic defining_class_converter (GH-20074)
Don't hardcode defining_class parameter name to "cls":

* Define CConverter.set_template_dict(): do nothing by default
* CLanguage.render_function() now calls set_template_dict() on all
  converters.
2020-05-14 00:31:31 +02:00
Steve Dower d6b727e2c9
bpo-40501: Replace ctypes code in uuid with native module (GH-19948) 2020-05-12 23:32:32 +01:00
Pablo Galindo ac7a92cc0a
bpo-40334: Avoid collisions between parser variables and grammar variables (GH-19987)
This is for the C generator:
- Disallow rule and variable names starting with `_`
- Rename most local variable names generated by the parser to start with `_`

Exceptions:
- Renaming `p` to `_p` will be a separate PR
- There are still some names that might clash, e.g.
  - anything starting with `Py`
  - C reserved words (`if` etc.)
  - Macros like `EXTRA` and `CHECK`
2020-05-09 21:34:50 -07:00
Pablo Galindo db9163ceef
bpo-40555: Check for p->error_indicator in loop rules after the main loop is done (GH-19986) 2020-05-08 03:38:44 +01:00
Petr Viktorin e1becf46b4
bpo-38787: C API for module state access from extension methods (PEP 573) (GH-19936)
Module C state is now accessible from C-defined heap type methods (PEP 573).
Patch by Marcel Plch and Petr Viktorin.

Co-authored-by: Marcel Plch <mplch@redhat.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
2020-05-07 15:39:59 +02:00
Pablo Galindo 470aac4d8e
bpo-40334: Generate comments in the parser code to improve debugging (GH-19966) 2020-05-06 23:14:43 +01:00
Anthony Shaw c95e691c90
Clean up unused imports for the peg generator module (GH-19891) 2020-05-04 03:03:05 +01:00
Pablo Galindo d2baff4301
bpo-40334: regenerate metaparser as part of regen-all (GH-19854) 2020-05-02 05:23:39 +01:00
Pablo Galindo 9dbaa8d9f0
Fix some scripts in the peg generator folder (GH-19853) 2020-05-02 05:23:06 +01:00
Pablo Galindo 7ba08ff7b4
bpo-40334: use the TOKENS file when checking dangling rules (GH-19849) 2020-05-01 23:14:12 +01:00
Pablo Galindo 6bd99d5f00
Fix the Tools/peg_generator/scripts/benchmark.py script (GH-19848) 2020-05-01 22:33:54 +01:00
Pablo Galindo b796b3fb48
bpo-40334: Simplify type handling in the PEG c_generator (GH-19818) 2020-05-01 12:32:26 +01:00
Dong-hee Na 113feb3ec2
bpo-40328: Add tool for generating cjk mapping headers (GH-19602) 2020-04-30 02:34:24 +09:00
Pablo Galindo 4db245ee9d
bpo-40334: refactor and cleanup for the PEG generators (GH-19775) 2020-04-29 10:42:21 +01:00