Commit Graph

3099 Commits

Author SHA1 Message Date
Lysandros Nikolaou bca7014032
bpo-42123: Run the parser two times and only enable invalid rules on the second run (GH-22111)
* Implement running the parser a second time for the errors messages

The first parser run is only responsible for detecting whether
there is a `SyntaxError` or not. If there isn't the AST gets returned.
Otherwise, the parser is run a second time with all the `invalid_*`
rules enabled so that all the customized error messages get produced.
2020-10-27 00:42:04 +02:00
Eric Snow 345cd37abe
bpo-36876: Fix the C analyzer tool. (GH-22841)
The original tool wasn't working right and it was simpler to create a new one, partially re-using some of the old code. At this point the tool runs properly on the master. (Try: ./python Tools/c-analyzer/c-analyzer.py analyze.)  It take ~40 seconds on my machine to analyze the full CPython code base.

Note that we'll need to iron out some OS-specific stuff (e.g. preprocessor). We're okay though since this tool isn't used yet in our workflow. We will also need to verify the analysis results in detail before activating the check in CI, though I'm pretty sure it's close.

https://bugs.python.org/issue36876
2020-10-22 18:42:51 -06:00
Pablo Galindo 47ecfd8030
bpo-41971: Fix test failure in test.test_tools.test_c_analyzer when mutating global state (GH-22652) 2020-10-11 21:34:51 +01:00
Serhiy Storchaka 9975cc5008
bpo-41985: Add _PyLong_FileDescriptor_Converter and AC converter for "fildes". (GH-22620) 2020-10-09 23:00:45 +03:00
Andre Delfino e8a2076e14
Revert "Fix all Python Cookbook links (#22205)" (GH-22424)
This commit reverts commit ac0333e1e1 as the original links are working again and they provide extended features such as comments and alternative versions.
2020-09-27 01:47:25 +01:00
Victor Stinner 19c3ac92bf
bpo-41834: Remove _Py_CheckRecursionLimit variable (GH-22359)
Remove the global _Py_CheckRecursionLimit variable: it has been
replaced by ceval.recursion_limit of the PyInterpreterState
structure.

There is no need to keep the variable for the stable ABI, since
Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() were not usable
in Python 3.8 and older: these macros accessed PyThreadState members,
whereas the PyThreadState structure is opaque in the limited C API.
2020-09-23 14:04:57 +02:00
Serhiy Storchaka dcfaa520c4
bpo-41715: Fix potential catastrofic backtracking in c_analyzer. (GH-22091) 2020-09-17 10:34:20 +03:00
Pablo Galindo a5634c4067
bpo-41746: Add type information to asdl_seq objects (GH-22223)
* Add new capability to the PEG parser to type variable assignments. For instance:
```
       | a[asdl_stmt_seq*]=';'.small_stmt+ [';'] NEWLINE { a }
```

* Add new sequence types from the asdl definition (automatically generated)
* Make `asdl_seq` type a generic aliasing pointer type.
* Create a new `asdl_generic_seq` for the generic case using `void*`.
* The old `asdl_seq_GET`/`ast_seq_SET` macros now are typed.
* New `asdl_seq_GET_UNTYPED`/`ast_seq_SET_UNTYPED` macros for dealing with generic sequences.
* Changes all possible `asdl_seq` types to use specific versions everywhere.
2020-09-16 19:42:00 +01:00
Andre Delfino ac0333e1e1
Fix all Python Cookbook links (#22205) 2020-09-15 21:13:26 +01:00
Václav Slavík 7c11a9acca
bpo-41744: Package python.props with correct name in NuGet package (GH-22154)
NuGet automatically includes .props file from the build directory in the
target using the package, but only if the .props file has the correct
name: it must be $(id).props

Rename python.props correspondingly in all the nuspec variants. Also
keep python.props as it were for backward compatibility.
2020-09-14 20:30:15 +01:00
Pablo Galindo e55a0e971b
Fix 'gather' rules in the python parser generator (GH-22021)
Currently, empty sequences in gather rules make the conditional for
gather rules fail as empty sequences evaluate as "False". We need to
explicitly check for "None" (the failure condition) to avoid false
negatives.
2020-09-03 15:29:55 +01:00
Steve Dower 6444ca9469
bpo-41526: Fixed layout of final page of the installer (GH-21871) 2020-08-14 00:35:52 +01:00
Victor Stinner fabd7bb8e0
bpo-41521: Replace whitelist/blacklist with allowlist/denylist (GH-21822)
Automerge-Triggered-By: @tiran
2020-08-11 06:26:59 -07:00
Mark Shannon 582aaf19e8
bpo-41463: Generate information about jumps from 'opcode.py' rather than duplicating it in 'compile.c' (GH-21714)
Generate information about jumps from 'opcode.py' rather than duplicate it in 'compile.c'
2020-08-04 17:30:11 +01:00
Steve Dower 37a06cbe5c
bpo-41412 and bpo-40948: Windows installer updates (GH-21656)
Prevent installation on Windows 8 and earlier.
Download UCRT on demand when required (non-updated Windows 8.1 only)
Add reference to py launcher to post-install message
2020-07-28 16:35:46 +01:00
Lysandros Nikolaou b3fbff7289
bpo-40939: Remove even more references to the old parser (GH-21642)
Automerge-Triggered-By: @lysnikolaou
2020-07-27 12:52:59 -07:00
Guido van Rossum 508ed2d912
Delete remaining references to Grammar/Grammar from docs (#21624)
(Ironically, the file itself remains, see https://github.com/we-like-parsers/cpython/issues/135.)
2020-07-26 08:27:52 -07:00
Victor Stinner 8182cc2e68
bpo-39573: Use the Py_TYPE() macro (GH-21433)
Replace obj->ob_type with Py_TYPE(obj).
2020-07-10 12:40:38 +02:00
Steve Dower 10772ec150
bpo-41173: Copy test results file from ARM worker before uploading (GH-21305) 2020-07-08 00:24:39 +01:00
Pablo Galindo 1ac0cbca36
bpo-41215: Don't use NULL by default in the PEG parser keyword list (GH-21355)
Automerge-Triggered-By: @lysnikolaou
2020-07-06 12:31:16 -07:00
Serhiy Storchaka b3dd5cd4a3
bpo-36346: Undeprecate private function _PyUnicode_AsUnicode(). (GH-21336) 2020-07-05 18:53:45 +03:00
Serhiy Storchaka 349f76c6aa
bpo-36346: Prepare for removing the legacy Unicode C API (AC only). (GH-21223) 2020-06-30 09:03:15 +03:00
Nikita Nemkin bbf36e8903
bpo-41070: Simplify pyshellext.dll build (GH-21037)
Replace MIDL-generated file with manual GUID definition.
Use the same .def file for release and debug builds.
Update setup build to support latest toolset
2020-06-24 00:32:23 +01:00
Lysandros Nikolaou 314858e276
bpo-40939: Remove the old parser (Part 2) (GH-21005)
Remove some remaining files and Makefile targets for the old parser
2020-06-20 19:07:25 +01:00
Batuhan Taskaya 55460ee6dc
bpo-41044: Generate valid PEG python parsers for opt+seq rules (GH-20995)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-06-20 18:40:06 +01:00
Serhiy Storchaka 9355868458
bpo-41043: Escape literal part of the path for glob(). (GH-20994) 2020-06-20 11:10:31 +03:00
Pablo Galindo 5fc4f8ae68
Fix trailing whitespace in keyword.py (GH-20881) 2020-06-15 04:33:33 +01:00
Pablo Galindo 78319e373d
Include soft keywords in keyword.py (GH-20877) 2020-06-15 03:55:15 +01:00
Pablo Galindo 756180b4bf
bpo-40939: Clean and adapt the peg_generator directory after deleting the old parser (GH-20822) 2020-06-12 01:55:35 +01:00
Pablo Galindo 436b648910
bpo-40939: Remove some extra references to PYTHONOLDPARSER (GH-20815)
Automerge-Triggered-By: @pablogsal
2020-06-11 10:08:05 -07:00
Pablo Galindo 1ed83adb0e
bpo-40939: Remove the old parser (GH-20768)
This commit removes the old parser, the deprecated parser module, the old parser compatibility flags and environment variables and all associated support code and documentation.
2020-06-11 17:30:46 +01:00
Lysandros Nikolaou 9727694f08
bpo-40939: Generate keyword.py using the new parser (GH-20800) 2020-06-11 13:45:15 +01:00
Lysandros Nikolaou ba6fd87e41
Refactor scripts in Tools/peg_generator/scripts (GH-20401) 2020-06-05 21:21:40 -07:00
Christian Heimes a871f692b4
bpo-30008: Fix OpenSSL no-deprecated compilation (GH-20397)
Fix :mod:`ssl`` code to be compatible with OpenSSL 1.1.x builds that use
``no-deprecated`` and ``--api=1.1.0``.

Note: Tests assume full OpenSSL API and fail with limited API.

Signed-off-by: Christian Heimes <christian@python.org>
Co-authored-by: Mark Wright <gienah@gentoo.org>
2020-06-01 08:58:14 +02:00
Serhiy Storchaka 5f4b229df7
bpo-40792: Make the result of PyNumber_Index() always having exact type int. (GH-20443)
Previously, the result could have been an instance of a subclass of int.

Also revert bpo-26202 and make attributes start, stop and step of the range
object having exact type int.

Add private function _PyNumber_Index() which preserves the old behavior
of PyNumber_Index() for performance to use it in the conversion functions
like PyLong_AsLong().
2020-05-28 10:33:45 +03:00
Pablo Galindo 404b23b85b
Fix lookahead of soft keywords in the PEG parser (GH-20436)
Automerge-Triggered-By: @gvanrossum
2020-05-26 16:15:52 -07:00
Guido van Rossum b45af1a569
Add soft keywords (GH-20370)
These are like keywords but they only work in context; they are not reserved except when there is an exact match.

This would enable things like match statements without reserving `match` (which would be bad for the `re.match()` function and probably lots of other places).

Automerge-Triggered-By: @gvanrossum
2020-05-26 10:58:44 -07:00
Serhiy Storchaka 578c3955e0
bpo-37999: No longer use __int__ in implicit integer conversions. (GH-15636)
Only __index__ should be used to make integer conversions lossless.
2020-05-26 18:43:38 +03:00
Ammar Askar a2bbedc8b1
Fix peg_generator compiler warnings under MSVC (GH-20405) 2020-05-26 05:33:35 +01:00
Lysandros Nikolaou 9645930b5b
bpo-40688: Use the correct parser in the peg_generator scripts (GH-20235)
The scripts in `Tools/peg_generator/scripts` mostly assume that
`ast.parse` and `compile` use the old parser, since this was the
state of things, while we were developing them. They need to be
updated to always use the correct parser. `_peg_parser` is being
extended to support both parsing and compiling with both parsers.
2020-05-25 20:51:58 +01:00
Pablo Galindo deb4355a37
bpo-40750: Do not expand the new parser debug flags if Py_BUILD_CORE is not defined (GH-20393) 2020-05-25 20:17:12 +01:00
Pablo Galindo 800a35c623
bpo-40750: Support -d flag in the new parser (GH-20340) 2020-05-25 18:38:45 +01:00
Batuhan Taskaya cba5031510
bpo-40334: Support suppressing of multiple optional variables in Pegen (GH-20367) 2020-05-24 23:20:18 +01:00
Pablo Galindo b831129123
Fix debug output in PEG parser generator (GH-20308) 2020-05-22 02:48:09 +01:00
Pablo Galindo d10fef35c6
Fix typing problems reported by mypy in pegen (GH-20297) 2020-05-21 21:39:44 +01:00
Batuhan Taskaya f50516e6a9
bpo-40334: Correctly generate C parser when assigned var is None (GH-20296)
When there are 2 negative lookaheads in the same rule, let's say `!"(" blabla "," !")"`, there will the 2 `FunctionCall`'s where assigned value is None. Currently when the `add_var` is called
the first one will be ignored but when the second lookahead's var is sent to dedupe it
will be returned as `None_1` and this won't be ignored by the declaration generator in the `visit_Alt`. This patch adds an explicit check to `add_var` to distinguish whether if there is a variable or not.
2020-05-21 20:57:52 +01:00
Zackery Spytz 8c862e5124
bpo-39631: Fix file association MIME type in the Windows installer (GH-20205)
Use text/x-python instead of text/plain to avoid issues with tools assuming that "ShellExecute(script)" is a non-executable operation.
2020-05-19 13:20:39 +01:00
Pablo Galindo 3764069f3b
bpo-40669: Use requirements.pip when installing PEG dependencies (GH-20194) 2020-05-18 23:37:06 +01:00
Lysandros Nikolaou dc31800f86
bpo-40669: Install PEG benchmarking dependencies in a venv (GH-20183)
Create a `make venv` target, that creates a virtual environment
and installs the dependency in that venv. `make time` and all
the related targets are changed to use the virtual environment
python.

Automerge-Triggered-By: @pablogsal
2020-05-18 11:27:40 -07:00
Lysandros Nikolaou 7b7a21bc4f
bpo-40661: Fix segfault when parsing invalid input (GH-20165)
Fix segfaults when parsing very complex invalid input, like `import äˆ ð£„¯ð¢·žð±‹á”€ð””ð‘©±å®ä±¬ð©¾\n𗶽`.

Co-authored-by: Guido van Rossum <guido@python.org>
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2020-05-18 18:32:03 +01:00