Commit Graph

607 Commits

Author SHA1 Message Date
Yury Selivanov 96ec934e75 Issue #24619: Simplify async/await tokenization.
This commit simplifies async/await tokenization in tokenizer.c,
tokenize.py & lib2to3/tokenize.py.  Previous solution was to keep
a stack of async-def & def blocks, whereas the new approach is just
to remember position of the outermost async-def block.

This change won't bring any parsing performance improvements, but
it makes the code much easier to read and validate.
2015-07-23 15:01:58 +03:00
Yury Selivanov 8fb307cd65 Issue #24619: New approach for tokenizing async/await.
This commit fixes how one-line async-defs and defs are tracked
by tokenizer.  It allows to correctly parse invalid code such
as:

>>> async def f():
...     def g(): pass
...     async = 10

and valid code such as:

>>> async def f():
...     async def g(): pass
...     await z

As a consequence, is is now possible to have one-line
'async def foo(): await ..' functions:

>>> async def foo(): return await bar()
2015-07-22 13:33:45 +03:00
Yury Selivanov 8085b80c18 Issue 24226: Fix parsing of many sequential one-line 'def' statements. 2015-05-18 12:50:52 -04:00
Yury Selivanov 7544508f02 PEP 0492 -- Coroutines with async and await syntax. Issue #24017. 2015-05-11 22:57:16 -04:00
Benjamin Peterson 025e9ebd0a PEP 448: additional unpacking generalizations (closes #2292)
Patch by Neil Girdhar.
2015-05-05 20:16:41 -04:00
Benjamin Peterson 273a720f87 merge 3.4 (#24022) 2015-04-21 12:07:06 -04:00
Benjamin Peterson d73aca769f do not call into python api if an exception is set (#24022) 2015-04-21 12:05:19 -04:00
Serhiy Storchaka 45ec3288d0 Removed trailing whitespaces in miscalenous files. 2015-04-03 19:42:32 +03:00
Serhiy Storchaka a8cd4d482f Got rid of warnings "suggest braces around empty body in an ‘else’ statement"
in Parser/pgen.c.
2015-04-03 15:24:33 +03:00
Raymond Hettinger df1b699447 Issue #22823: Use set literals instead of creating a set from a list 2014-11-09 15:56:33 -08:00
Serhiy Storchaka 67c719b34b Silenced some warnings about comparison between signed and unsigned integer
expressions.
2014-09-05 10:10:23 +03:00
Guido van Rossum 416b516d46 Fix bootstrapping asdl -- it didn't work with Python 2.7. 2014-07-08 16:22:48 -07:00
Benjamin Peterson 3e439797ba merge 3.4 (#21642) 2014-06-07 12:39:51 -07:00
Benjamin Peterson c416162302 allow the keyword else immediately after (no space) an integer (closes #21642) 2014-06-07 12:36:39 -07:00
Eli Bendersky 5e3d338a74 Issue #19655: Replace the ASDL parser carried with CPython
The new parser does not rely on Spark (which is now removed from our repo),
uses modern 3.x idioms and is significantly smaller and simpler.

It generates exactly the same AST files (.h and .c), so in practice no builds
should be affected.
2014-05-09 17:58:22 -07:00
Benjamin Peterson d51374ed78 PEP 465: a dedicated infix operator for matrix multiplication (closes #21176) 2014-04-09 23:55:56 -04:00
Martin v. Löwis 78f1e4c865 Merge with 3.3 2014-02-28 15:43:36 +01:00
Martin v. Löwis 815b41b1cd Issue #20731: Properly position in source code files even if they
are opened in text mode. Patch by Serhiy Storchaka.
2014-02-28 15:27:29 +01:00
Benjamin Peterson 42ec031fe7 merge 3.3 (#20588) 2014-02-10 22:41:40 -05:00
Benjamin Peterson c2f665e721 don't put runtime values in array initializer for C89 compliance (closes #20588) 2014-02-10 22:19:02 -05:00
Serhiy Storchaka 5940b92909 Do not reset the line number because we already set file position to correct
value.

(fixes error in patch for issue #18960)
2014-01-09 20:13:52 +02:00
Serhiy Storchaka 1064a13bb0 Do not reset the line number because we already set file position to correct
value.

(fixes error in patch for issue #18960)
2014-01-09 20:12:49 +02:00
Serhiy Storchaka 7282ff6d5b Issue #18960: Fix bugs with Python source code encoding in the second line.
* The first line of Python script could be executed twice when the source
encoding (not equal to 'utf-8') was specified on the second line.

* Now the source encoding declaration on the second line isn't effective if
the first line contains anything except a comment.

* As a consequence, 'python -x' works now again with files with the source
encoding declarations specified on the second file, and can be used again
to make Python batch files on Windows.

* The tokenize module now ignore the source encoding declaration on the second
line if the first line contains anything except a comment.

* IDLE now ignores the source encoding declaration on the second line if the
first line contains anything except a comment.

* 2to3 and the findnocoding.py script now ignore the source encoding
declaration on the second line if the first line contains anything except
a comment.
2014-01-09 18:41:59 +02:00
Serhiy Storchaka 768c16ce02 Issue #18960: Fix bugs with Python source code encoding in the second line.
* The first line of Python script could be executed twice when the source
encoding (not equal to 'utf-8') was specified on the second line.

* Now the source encoding declaration on the second line isn't effective if
the first line contains anything except a comment.

* As a consequence, 'python -x' works now again with files with the source
encoding declarations specified on the second file, and can be used again
to make Python batch files on Windows.

* The tokenize module now ignore the source encoding declaration on the second
line if the first line contains anything except a comment.

* IDLE now ignores the source encoding declaration on the second line if the
first line contains anything except a comment.

* 2to3 and the findnocoding.py script now ignore the source encoding
declaration on the second line if the first line contains anything except
a comment.
2014-01-09 18:36:09 +02:00
Christian Heimes af01f66817 Issue #16136: Remove VMS support and VMS-related code 2013-12-21 16:19:10 +01:00
Christian Heimes 724b828e79 upcast int to size_t to silence two autological-constant-out-of-range-compare warnings with clang. 2013-12-04 08:42:46 +01:00
Victor Stinner cad876d542 Fix a compiler warning on Windows 64-bit in parsetok.c
Python parser doesn't support lines longer than INT_MAX bytes yet
2013-11-18 01:09:51 +01:00
Victor Stinner 3a8a333942 Fix compiler warnings on Windows 64-bit in grammar.c
INT_MAX states and labels should be enough for everyone
2013-11-18 01:07:38 +01:00
Serhiy Storchaka c679227e31 Issue #1772673: The type of `char*` arguments now changed to `const char*`. 2013-10-19 21:03:34 +03:00
Victor Stinner c548660af5 Issue #16742: My fix on PyOS_StdioReadline() was incomplete, PyMem_FREE() was
not patched
2013-10-19 02:40:16 +02:00
Antoine Pitrou d01d396e7f Issue #4555: All exported C symbols are now prefixed with either "Py" or "_Py".
("make smelly" now clean)
2013-10-12 22:52:43 +02:00
Victor Stinner 2fe9bac4dc Close #16742: Fix misuse of memory allocations in PyOS_Readline()
The GIL must be held to call PyMem_Malloc(), whereas PyOS_Readline() releases
the GIL to read input.

The result of the C callback PyOS_ReadlineFunctionPointer must now be a string
allocated by PyMem_RawMalloc() or PyMem_RawRealloc() (or NULL if an error
occurred), instead of a string allocated by PyMem_Malloc() or PyMem_Realloc().

Fixing this issue was required to setup a hook on PyMem_Malloc(), for example
using the tracemalloc module.

PyOS_Readline() copies the result of PyOS_ReadlineFunctionPointer() into a new
buffer allocated by PyMem_Malloc(). So the public API of PyOS_Readline() does
not change.
2013-10-10 16:18:20 +02:00
Eli Bendersky 1891cff587 Move open outside try/finally 2013-09-26 09:35:39 -07:00
Eli Bendersky 99081238e9 Don't use fancy new Python features like 'with' - some bots don't have them
and can't bootstrap the parser.
2013-09-26 06:41:36 -07:00
Eli Bendersky 58fe1b1307 Normalize whitespace 2013-09-26 06:32:22 -07:00
Eli Bendersky b788a385cd Small fixes in Parser/asdl.py - no change in functionality.
1. Make it work when invoked directly from the command-line. It was failing
   due to a couple of stale function/class usages in the __main__ section.
2. Close the parsed file in the parse() function after opening it.
2013-09-26 06:31:32 -07:00
Victor Stinner daf455554b Issue #18571: Implementation of the PEP 446: file descriptors and file handles
are now created non-inheritable; add functions os.get/set_inheritable(),
os.get/set_handle_inheritable() and socket.socket.get/set_inheritable().
2013-08-28 00:53:59 +02:00
Victor Stinner 14e461d5b9 Close #11619: The parser and the import machinery do not encode Unicode
filenames anymore on Windows.
2013-08-26 22:28:21 +02:00
Ezio Melotti d640fe2af5 #18803: merge with 3.3. 2013-08-26 01:33:30 +03:00
Ezio Melotti 7c4a7e6f3c #18803: fix more typos. Patch by Févry Thibault. 2013-08-26 01:32:56 +03:00
Antoine Pitrou 9ed5f27266 Issue #18722: Remove uses of the "register" keyword in C code. 2013-08-13 20:18:52 +02:00
Christian Heimes 73207e03ad Issue #18368: PyOS_StdioReadline() no longer leaks memory when realloc() fails. 2013-08-06 16:03:33 +02:00
Christian Heimes 9ae513caa7 Issue #18368: PyOS_StdioReadline() no longer leaks memory when realloc() fails. 2013-08-06 15:59:16 +02:00
Christian Heimes 1289565f4b Silence warning about set but unused variable inside compile_atom() in non-debug builds 2013-07-31 23:48:04 +02:00
Christian Heimes 5e4d372524 Silence warning about set but unused variable inside compile_atom() in non-debug builds 2013-07-31 23:47:56 +02:00
Christian Heimes b7f1b38dea Issue #18552: Check return value of PyArena_AddPyObject() in obj2ast_object(). 2013-07-27 00:33:35 +02:00
Christian Heimes 70c94e7896 Issue #18552: Check return value of PyArena_AddPyObject() in obj2ast_object(). 2013-07-27 00:33:13 +02:00
Victor Stinner b318990cac (Merge 3.3) Parser/asdl_c.py: use Py_CLEAR() 2013-07-27 00:04:42 +02:00
Victor Stinner 1acc129d48 Parser/asdl_c.py: use Py_CLEAR() 2013-07-27 00:03:47 +02:00
Victor Stinner ee4b59c0f8 (Merge 3.3) According to the PEP 7, C code must "use 4-space indents"
Replace 8 spaces with 4.
2013-07-27 00:01:35 +02:00