Commit Graph

665 Commits

Author SHA1 Message Date
Albert-Jan Nijburg c9ccacea3f bpo-25324: add missing comma in Parser/tokenizer.c (GH-1910) 2017-06-01 13:51:27 -07:00
Albert-Jan Nijburg fc354f0785 bpo-25324: copy tok_name before changing it (#1608)
* add test to check if were modifying token

* copy list so import tokenize doesnt have side effects on token

* shorten line

* add tokenize tokens to token.h to get them to show up in token

* move ERRORTOKEN back to its previous location, and fix nitpick

* copy comments from token.h automatically

* fix whitespace and make more pythonic

* change to fix comments from @haypo

* update token.rst and Misc/NEWS

* change wording

* some more wording changes
2017-05-31 16:00:21 +02:00
Jim Fasarakis-Hilliard cf1958af4c Remove obsolete declaration in tokenizer.h (#962) 2017-04-03 19:18:32 +03:00
Serhiy Storchaka 0b3ec19225 Use NULL rather than 0. (#778)
There was few cases of using literal 0 instead of NULL in the context of
pointers.  While this was a legitimate C code, using NULL rather than 0 makes
the code clearer.
2017-03-23 17:53:47 +02:00
INADA Naoki 4c78c527d2 bpo-29622: Make AST constructor to accept less than enough number of positional arguments (GH-249)
bpo-29463 added optional "docstring" field to 4 AST types.
While it is optional, it breaks backward compatibility because AST constructor
requires number of positional argument is same to number of fields.

AST types accepts empty arguments, and incomplete keyword arguments.
But it's not big problem because field can be filled after creation, and checked when compiling.
So stop requiring complete set of fields for positional arguments too.
2017-02-24 02:48:17 +09:00
INADA Naoki cb41b2766d bpo-29463: Add docstring field to some AST nodes. (#46)
* bpo-29463: Add docstring field to some AST nodes.

ClassDef, ModuleDef, FunctionDef, and AsyncFunctionDef has docstring
field for now.  It was first statement of there body.

* fix document.  thanks travis!

* doc fixes
2017-02-22 16:31:59 +01:00
Berker Peksag d2f4404bbb Issue #28489: Merge from 3.6 2017-02-05 04:33:11 +03:00
Berker Peksag 6f80562862 Issue #28489: Fix comment in tokenizer.c
Patch by Ryan Gonzalez.
2017-02-05 04:32:39 +03:00
INADA Naoki fc489082c8 Issue #29369: Use Py_IDENTIFIER in Python-ast.c 2017-01-25 22:33:43 +09:00
Victor Stinner a5ed5f000a Use _PyObject_CallNoArg()
Replace:
    PyObject_CallObject(callable, NULL)
with:
    _PyObject_CallNoArg(callable)
2016-12-06 18:45:50 +01:00
Serhiy Storchaka 06515833fe Replaced outdated macros _PyUnicode_AsString and _PyUnicode_AsStringAndSize
with PyUnicode_AsUTF8 and PyUnicode_AsUTF8AndSize.
2016-11-20 09:13:07 +02:00
Steve Dower 6c2b9d3479 Issue #28333: Fixes off-by-one error that was adding an extra space. 2016-10-25 11:51:54 -07:00
Steve Dower 59bd34fa8a Issue #28333: Remove unnecessary increment. 2016-10-08 12:20:45 -07:00
Steve Dower 3cd187b9f5 Issue #28333: Enables Unicode for ps1/ps2 and input() prompts. (Patch by Eryk Sun) 2016-10-08 12:18:16 -07:00
Serhiy Storchaka 5e80855af3 Issue #24098: Fixed possible crash when AST is changed in process of
compiling it.
2016-10-07 21:55:49 +03:00
Serhiy Storchaka cf3806026b Issue #24098: Fixed possible crash when AST is changed in process of
compiling it.
2016-10-07 21:51:28 +03:00
Benjamin Peterson e2e792d98f merge 3.5 (#28184) 2016-09-19 22:17:16 -07:00
Benjamin Peterson f5e8e8fc2b merge 3.5 (#24022) 2016-09-18 23:44:02 -07:00
Benjamin Peterson 57bda335e1 merge 3.4 2016-09-18 23:43:18 -07:00
Benjamin Peterson 26d998cfdd properly handle the single null-byte file (closes #24022) 2016-09-18 23:41:11 -07:00
Benjamin Peterson 9ac11a752a properly free memory in pgen 2016-09-18 18:00:25 -07:00
Benjamin Peterson 5a715cfc57 merge 3.5 (#27981) 2016-09-12 22:07:14 -07:00
Benjamin Peterson 35ee948fa5 restructure fp_setreadl so as to avoid refleaks (closes #27981) 2016-09-12 22:06:58 -07:00
Brett Cannon a721abac29 Issue #26331: Implement the parsing part of PEP 515.
Thanks to Georg Brandl for the patch.
2016-09-09 14:57:09 -07:00
Yury Selivanov 52c4e7cc84 Issue #28008: Implement PEP 530 -- asynchronous comprehensions. 2016-09-09 10:36:01 -07:00
Yury Selivanov f8cb8a16a3 Issue #27985: Implement PEP 526 -- Syntax for Variable Annotations.
Patch by Ivan Levkivskyi.
2016-09-08 20:50:03 -07:00
Christian Heimes c6cc23d0b9 Skip unused value in tokenizer code
In the case of an escape character, c is never read. tok_next() is
used to advance the pointer.

CID 1225097
2016-09-09 00:09:45 +02:00
Steve Dower 3929499914 Issue #1602: Windows console doesn't input or print Unicode (PEP 528)
Closes #17602: Adds a readline implementation for the Windows console
2016-08-30 21:22:36 -07:00
Steve Dower 940f33a50f Issue #23524: Finish removing _PyVerify_fd from sources 2016-09-08 11:21:54 -07:00
Benjamin Peterson 2f8bfef158 replace PY_SIZE_MAX with SIZE_MAX 2016-09-07 09:26:18 -07:00
Benjamin Peterson ca47063998 replace Py_(u)intptr_t with the c99 standard types 2016-09-06 13:47:26 -07:00
Victor Stinner 4bb31e90f0 Fix a clang warning in grammar.c
Clang is smarter than GCC and emits a warning for dead code after a function
declared with __attribute__((__noreturn__)) (Py_FatalError).
2016-08-19 15:11:56 +02:00
Berker Peksag 531396c764 Issue #27336: Fix compilation failures --without-threads 2016-06-17 13:25:01 +03:00
Serhiy Storchaka ec39756960 Issue #22570: Renamed Py_SETREF to Py_XSETREF. 2016-04-06 09:50:03 +03:00
Serhiy Storchaka 48842714b9 Issue #22570: Renamed Py_SETREF to Py_XSETREF. 2016-04-06 09:45:48 +03:00
Berker Peksag 2a65ecb780 Issue #26130: Remove redundant variable 's' from Parser/parser.c
Patch by Oren Milman.
2016-03-28 00:45:28 +03:00
Benjamin Peterson 7285d520e0 remove duplicated check for fractions and complex numbers (closes #26076)
Patch by Oren Milman.
2016-03-24 22:43:23 -07:00
Serhiy Storchaka a051bf3afb Issue #26581: Use the first coding cookie on a line, not the last one. 2016-03-20 23:47:48 +02:00
Serhiy Storchaka e431d3c9aa Issue #26581: Use the first coding cookie on a line, not the last one. 2016-03-20 23:36:29 +02:00
Victor Stinner 0611c26a58 On memory error, dump the memory block traceback
Issue #26564: _PyObject_DebugDumpAddress() now dumps the traceback where a
memory block was allocated on memory block. Use the tracemalloc module to get
the traceback.
2016-03-15 22:22:13 +01:00
Victor Stinner 8a1be61849 Add more checks on the GIL
Issue #10915, #15751, #26558:

* PyGILState_Check() now returns 1 (success) before the creation of the GIL and
  after the destruction of the GIL. It allows to use the function early in
  Python initialization and late in Python finalization.
* Add a flag to disable PyGILState_Check(). Disable PyGILState_Check() when
  Py_NewInterpreter() is called
* Add assert(PyGILState_Check()) to: _Py_dup(), _Py_fstat(), _Py_read()
  and _Py_write()
2016-03-14 22:07:55 +01:00
Victor Stinner 25219f596a Issue #26146: remove useless code
obj2ast_constant() code is baesd on obj2ast_object() which has a special case
for Py_None. But in practice, we don't need to have a special case for
constants.

Issue noticed by Joseph Jevnik on a review.
2016-01-27 00:37:59 +01:00
Victor Stinner f2c1aa1661 Add ast.Constant
Issue #26146: Add a new kind of AST node: ast.Constant. It can be used by
external AST optimizers, but the compiler does not emit directly such node.

An optimizer can replace the following AST nodes with ast.Constant:

* ast.NameConstant: None, False, True
* ast.Num: int, float, complex
* ast.Str: str
* ast.Bytes: bytes
* ast.Tuple if items are constants too: tuple
* frozenset

Update code to accept ast.Constant instead of ast.Num and/or ast.Str:

* compiler
* docstrings
* ast.literal_eval()
* Tools/parser/unparse.py
2016-01-26 00:40:57 +01:00
Serhiy Storchaka ef1585eb9a Issue #25923: Added more const qualifiers to signatures of static and private functions. 2015-12-25 20:01:53 +02:00
Serhiy Storchaka 2d06e84455 Issue #25923: Added the const qualifier to static constant arrays. 2015-12-25 19:53:18 +02:00
Serhiy Storchaka f006940351 Issue #20440: Massive replacing unsafe attribute setting code with special
macro Py_SETREF.
2015-12-24 10:39:57 +02:00
Serhiy Storchaka 5a57ade58e Issue #20440: Massive replacing unsafe attribute setting code with special
macro Py_SETREF.
2015-12-24 10:35:59 +02:00
Serhiy Storchaka 0304729ec4 Issue #25388: Fixed tokenizer crash when processing undecodable source code
with a null byte.
2015-11-14 15:12:04 +02:00
Serhiy Storchaka 7e2b870b85 Issue #25388: Fixed tokenizer crash when processing undecodable source code
with a null byte.
2015-11-14 15:11:17 +02:00
Serhiy Storchaka 0d441119f5 Issue #25388: Fixed tokenizer crash when processing undecodable source code
with a null byte.
2015-11-14 15:10:35 +02:00