Commit Graph

3111 Commits

Author SHA1 Message Date
Victor Stinner ac7d0169d2
bpo-42212: smelly.py also checks the dynamic library (GH-23423)
The smelly.py script now also checks the Python dynamic library and
extension modules, not only the Python static library. Make also the
script more verbose: explain what it does.

The GitHub Action job now builds Python with the libpython dynamic
library.
2020-11-24 13:38:08 +01:00
Eric Snow 9f02b479e6
bpo-36876: [c-analyzer tool] Tighten up the results and output. (GH-23431)
We also update the "ignored" file with a temporary list of all known globals.
2020-11-20 15:39:28 -07:00
Victor Stinner 8fba9523cf
bpo-42398: Fix "make regen-all" race condition (GH-23362)
Fix a race condition in "make regen-all" when make -jN option is used
to run jobs in parallel. The clinic.py script now only use atomic
write to write files. Moveover, generated files are now left
unchanged if the content does not change, to not change the file
modification time.

The "make regen-all" command runs "make clinic" and "make
regen-importlib" targets:

* "make regen-importlib" builds object files (ex: Modules/_weakref.o)
  from source files (ex: Modules/_weakref.c) and clinic files (ex:
  Modules/clinic/_weakref.c.h)
* "make clinic" always rewrites all clinic files
  (ex: Modules/clinic/_weakref.c.h)

Since there is no dependency between "clinic" and "regen-importlib"
Makefile targets, these two targets can be run in parallel. Moreover,
half of clinic.py file writes are not atomic and so there is a race
condition when "make regen-all" runs jobs in parallel using make -jN
option (which can be passed in MAKEFLAGS environment variable).

Fix clinic.py to make all file writes atomic:

* Add write_file() function to ensure that all file writes are
  atomic: write into a temporary file and then use os.replace().
* Moreover, write_file() doesn't recreate or modify the file if the
  content does not change to avoid modifying the file modification
  file.
* Update test_clinic to verify these assertions with a functional
  test.
* Remove Clinic.force attribute which was no longer used, whereas
  Clinic.verify remains useful.
2020-11-18 15:36:27 +01:00
Steve Dower 9b6934230c
bpo-40754: Adds _testinternalcapi to Windows installer for test suite (GH-23271) 2020-11-13 22:24:36 +00:00
Mark Shannon 877df851c3
bpo-42246: Partial implementation of PEP 626. (GH-23113)
* Implement new line number table format, as defined in PEP 626.
2020-11-12 09:43:29 +00:00
jack1142 bfc6b63102
bpo-36310: Allow pygettext.py to detect calls to gettext in f-strings. (GH-19875)
Adds support to Tools/i18n/pygettext.py for gettext calls in f-strings. This process is done by parsing the f-strings, processing each value, and flagging the ones which contain a gettext call.

Co-authored-by: Batuhan Taskaya <batuhanosmantaskaya@gmail.com>
2020-11-10 01:50:45 +03:00
Yash Shete 1f73c320e2
bpo-41712: Avoid runaway regex match in upload scripts (GH-23166) 2020-11-09 17:38:09 +00:00
Steve Dower db6434c474
Enable signing of nuget.org packages and update to supported timestamp server (GH-23132) 2020-11-03 22:31:49 +00:00
Lysandros Nikolaou 02cdfc93f8
bpo-42218: Correctly handle errors in left-recursive rules (GH-23065)
Left-recursive rules need to check for errors explicitly, since
even if the rule returns NULL, the parsing might continue and lead
to long-distance failures.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-10-31 20:31:41 +02:00
Eric Snow 4fe72090de
bpo-36876: Small adjustments to the C-analyzer tool. (GH-23045)
This is a little bit of clean-up, small fixes, and additional helpers prior to building an updated & accurate list of globals to eliminate.
2020-10-30 15:46:52 -06:00
Victor Stinner b9ee4af4c6
bpo-42208: Fix test_gdb for gc_collect_main() name (GH-23041)
The gcmodule.c collect() function was renamed to gc_collect_main():
update gdb/libpython.py (python-gdb.py).
2020-10-30 21:09:48 +01:00
Victor Stinner c310185c08
bpo-42161: Remove private _PyLong_Zero and _PyLong_One (GH-23003)
Use PyLong_FromLong(0) and PyLong_FromLong(1) of the public C API
instead. For Python internals, _PyLong_GetZero() and _PyLong_GetOne()
of pycore_long.h can be used.
2020-10-27 21:34:33 +01:00
Lysandros Nikolaou bca7014032
bpo-42123: Run the parser two times and only enable invalid rules on the second run (GH-22111)
* Implement running the parser a second time for the errors messages

The first parser run is only responsible for detecting whether
there is a `SyntaxError` or not. If there isn't the AST gets returned.
Otherwise, the parser is run a second time with all the `invalid_*`
rules enabled so that all the customized error messages get produced.
2020-10-27 00:42:04 +02:00
Eric Snow 345cd37abe
bpo-36876: Fix the C analyzer tool. (GH-22841)
The original tool wasn't working right and it was simpler to create a new one, partially re-using some of the old code. At this point the tool runs properly on the master. (Try: ./python Tools/c-analyzer/c-analyzer.py analyze.)  It take ~40 seconds on my machine to analyze the full CPython code base.

Note that we'll need to iron out some OS-specific stuff (e.g. preprocessor). We're okay though since this tool isn't used yet in our workflow. We will also need to verify the analysis results in detail before activating the check in CI, though I'm pretty sure it's close.

https://bugs.python.org/issue36876
2020-10-22 18:42:51 -06:00
Pablo Galindo 47ecfd8030
bpo-41971: Fix test failure in test.test_tools.test_c_analyzer when mutating global state (GH-22652) 2020-10-11 21:34:51 +01:00
Serhiy Storchaka 9975cc5008
bpo-41985: Add _PyLong_FileDescriptor_Converter and AC converter for "fildes". (GH-22620) 2020-10-09 23:00:45 +03:00
Andre Delfino e8a2076e14
Revert "Fix all Python Cookbook links (#22205)" (GH-22424)
This commit reverts commit ac0333e1e1 as the original links are working again and they provide extended features such as comments and alternative versions.
2020-09-27 01:47:25 +01:00
Victor Stinner 19c3ac92bf
bpo-41834: Remove _Py_CheckRecursionLimit variable (GH-22359)
Remove the global _Py_CheckRecursionLimit variable: it has been
replaced by ceval.recursion_limit of the PyInterpreterState
structure.

There is no need to keep the variable for the stable ABI, since
Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() were not usable
in Python 3.8 and older: these macros accessed PyThreadState members,
whereas the PyThreadState structure is opaque in the limited C API.
2020-09-23 14:04:57 +02:00
Serhiy Storchaka dcfaa520c4
bpo-41715: Fix potential catastrofic backtracking in c_analyzer. (GH-22091) 2020-09-17 10:34:20 +03:00
Pablo Galindo a5634c4067
bpo-41746: Add type information to asdl_seq objects (GH-22223)
* Add new capability to the PEG parser to type variable assignments. For instance:
```
       | a[asdl_stmt_seq*]=';'.small_stmt+ [';'] NEWLINE { a }
```

* Add new sequence types from the asdl definition (automatically generated)
* Make `asdl_seq` type a generic aliasing pointer type.
* Create a new `asdl_generic_seq` for the generic case using `void*`.
* The old `asdl_seq_GET`/`ast_seq_SET` macros now are typed.
* New `asdl_seq_GET_UNTYPED`/`ast_seq_SET_UNTYPED` macros for dealing with generic sequences.
* Changes all possible `asdl_seq` types to use specific versions everywhere.
2020-09-16 19:42:00 +01:00
Andre Delfino ac0333e1e1
Fix all Python Cookbook links (#22205) 2020-09-15 21:13:26 +01:00
Václav Slavík 7c11a9acca
bpo-41744: Package python.props with correct name in NuGet package (GH-22154)
NuGet automatically includes .props file from the build directory in the
target using the package, but only if the .props file has the correct
name: it must be $(id).props

Rename python.props correspondingly in all the nuspec variants. Also
keep python.props as it were for backward compatibility.
2020-09-14 20:30:15 +01:00
Pablo Galindo e55a0e971b
Fix 'gather' rules in the python parser generator (GH-22021)
Currently, empty sequences in gather rules make the conditional for
gather rules fail as empty sequences evaluate as "False". We need to
explicitly check for "None" (the failure condition) to avoid false
negatives.
2020-09-03 15:29:55 +01:00
Steve Dower 6444ca9469
bpo-41526: Fixed layout of final page of the installer (GH-21871) 2020-08-14 00:35:52 +01:00
Victor Stinner fabd7bb8e0
bpo-41521: Replace whitelist/blacklist with allowlist/denylist (GH-21822)
Automerge-Triggered-By: @tiran
2020-08-11 06:26:59 -07:00
Mark Shannon 582aaf19e8
bpo-41463: Generate information about jumps from 'opcode.py' rather than duplicating it in 'compile.c' (GH-21714)
Generate information about jumps from 'opcode.py' rather than duplicate it in 'compile.c'
2020-08-04 17:30:11 +01:00
Steve Dower 37a06cbe5c
bpo-41412 and bpo-40948: Windows installer updates (GH-21656)
Prevent installation on Windows 8 and earlier.
Download UCRT on demand when required (non-updated Windows 8.1 only)
Add reference to py launcher to post-install message
2020-07-28 16:35:46 +01:00
Lysandros Nikolaou b3fbff7289
bpo-40939: Remove even more references to the old parser (GH-21642)
Automerge-Triggered-By: @lysnikolaou
2020-07-27 12:52:59 -07:00
Guido van Rossum 508ed2d912
Delete remaining references to Grammar/Grammar from docs (#21624)
(Ironically, the file itself remains, see https://github.com/we-like-parsers/cpython/issues/135.)
2020-07-26 08:27:52 -07:00
Victor Stinner 8182cc2e68
bpo-39573: Use the Py_TYPE() macro (GH-21433)
Replace obj->ob_type with Py_TYPE(obj).
2020-07-10 12:40:38 +02:00
Steve Dower 10772ec150
bpo-41173: Copy test results file from ARM worker before uploading (GH-21305) 2020-07-08 00:24:39 +01:00
Pablo Galindo 1ac0cbca36
bpo-41215: Don't use NULL by default in the PEG parser keyword list (GH-21355)
Automerge-Triggered-By: @lysnikolaou
2020-07-06 12:31:16 -07:00
Serhiy Storchaka b3dd5cd4a3
bpo-36346: Undeprecate private function _PyUnicode_AsUnicode(). (GH-21336) 2020-07-05 18:53:45 +03:00
Serhiy Storchaka 349f76c6aa
bpo-36346: Prepare for removing the legacy Unicode C API (AC only). (GH-21223) 2020-06-30 09:03:15 +03:00
Nikita Nemkin bbf36e8903
bpo-41070: Simplify pyshellext.dll build (GH-21037)
Replace MIDL-generated file with manual GUID definition.
Use the same .def file for release and debug builds.
Update setup build to support latest toolset
2020-06-24 00:32:23 +01:00
Lysandros Nikolaou 314858e276
bpo-40939: Remove the old parser (Part 2) (GH-21005)
Remove some remaining files and Makefile targets for the old parser
2020-06-20 19:07:25 +01:00
Batuhan Taskaya 55460ee6dc
bpo-41044: Generate valid PEG python parsers for opt+seq rules (GH-20995)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-06-20 18:40:06 +01:00
Serhiy Storchaka 9355868458
bpo-41043: Escape literal part of the path for glob(). (GH-20994) 2020-06-20 11:10:31 +03:00
Pablo Galindo 5fc4f8ae68
Fix trailing whitespace in keyword.py (GH-20881) 2020-06-15 04:33:33 +01:00
Pablo Galindo 78319e373d
Include soft keywords in keyword.py (GH-20877) 2020-06-15 03:55:15 +01:00
Pablo Galindo 756180b4bf
bpo-40939: Clean and adapt the peg_generator directory after deleting the old parser (GH-20822) 2020-06-12 01:55:35 +01:00
Pablo Galindo 436b648910
bpo-40939: Remove some extra references to PYTHONOLDPARSER (GH-20815)
Automerge-Triggered-By: @pablogsal
2020-06-11 10:08:05 -07:00
Pablo Galindo 1ed83adb0e
bpo-40939: Remove the old parser (GH-20768)
This commit removes the old parser, the deprecated parser module, the old parser compatibility flags and environment variables and all associated support code and documentation.
2020-06-11 17:30:46 +01:00
Lysandros Nikolaou 9727694f08
bpo-40939: Generate keyword.py using the new parser (GH-20800) 2020-06-11 13:45:15 +01:00
Lysandros Nikolaou ba6fd87e41
Refactor scripts in Tools/peg_generator/scripts (GH-20401) 2020-06-05 21:21:40 -07:00
Christian Heimes a871f692b4
bpo-30008: Fix OpenSSL no-deprecated compilation (GH-20397)
Fix :mod:`ssl`` code to be compatible with OpenSSL 1.1.x builds that use
``no-deprecated`` and ``--api=1.1.0``.

Note: Tests assume full OpenSSL API and fail with limited API.

Signed-off-by: Christian Heimes <christian@python.org>
Co-authored-by: Mark Wright <gienah@gentoo.org>
2020-06-01 08:58:14 +02:00
Serhiy Storchaka 5f4b229df7
bpo-40792: Make the result of PyNumber_Index() always having exact type int. (GH-20443)
Previously, the result could have been an instance of a subclass of int.

Also revert bpo-26202 and make attributes start, stop and step of the range
object having exact type int.

Add private function _PyNumber_Index() which preserves the old behavior
of PyNumber_Index() for performance to use it in the conversion functions
like PyLong_AsLong().
2020-05-28 10:33:45 +03:00
Pablo Galindo 404b23b85b
Fix lookahead of soft keywords in the PEG parser (GH-20436)
Automerge-Triggered-By: @gvanrossum
2020-05-26 16:15:52 -07:00
Guido van Rossum b45af1a569
Add soft keywords (GH-20370)
These are like keywords but they only work in context; they are not reserved except when there is an exact match.

This would enable things like match statements without reserving `match` (which would be bad for the `re.match()` function and probably lots of other places).

Automerge-Triggered-By: @gvanrossum
2020-05-26 10:58:44 -07:00
Serhiy Storchaka 578c3955e0
bpo-37999: No longer use __int__ in implicit integer conversions. (GH-15636)
Only __index__ should be used to make integer conversions lossless.
2020-05-26 18:43:38 +03:00