Commit Graph

41 Commits

Author SHA1 Message Date
Yury Selivanov 96ec934e75 Issue #24619: Simplify async/await tokenization.
This commit simplifies async/await tokenization in tokenizer.c,
tokenize.py & lib2to3/tokenize.py.  Previous solution was to keep
a stack of async-def & def blocks, whereas the new approach is just
to remember position of the outermost async-def block.

This change won't bring any parsing performance improvements, but
it makes the code much easier to read and validate.
2015-07-23 15:01:58 +03:00
Yury Selivanov 8fb307cd65 Issue #24619: New approach for tokenizing async/await.
This commit fixes how one-line async-defs and defs are tracked
by tokenizer.  It allows to correctly parse invalid code such
as:

>>> async def f():
...     def g(): pass
...     async = 10

and valid code such as:

>>> async def f():
...     async def g(): pass
...     await z

As a consequence, is is now possible to have one-line
'async def foo(): await ..' functions:

>>> async def foo(): return await bar()
2015-07-22 13:33:45 +03:00
Yury Selivanov 7544508f02 PEP 0492 -- Coroutines with async and await syntax. Issue #24017. 2015-05-11 22:57:16 -04:00
Serhiy Storchaka c679227e31 Issue #1772673: The type of `char*` arguments now changed to `const char*`. 2013-10-19 21:03:34 +03:00
Victor Stinner fe7c5b5bdf Issue #9319: Include the filename in "Non-UTF8 code ..." syntax error. 2011-04-05 01:48:03 +02:00
Victor Stinner 7f2fee3640 Issue #10785: Store the filename as Unicode in the Python parser. 2011-04-05 00:39:01 +02:00
Georg Brandl 2b15bd810d #10222: fix for overzealous AIX compiler. 2010-10-29 04:54:13 +00:00
Victor Stinner 4c7c8c3023 Issue #9713, #10114: Parser functions (eg. PyParser_ASTFromFile) expects
filenames encoded to the filesystem encoding with surrogateescape error handler
(to support undecodable bytes), instead of UTF-8 in strict mode.
2010-10-16 13:14:10 +00:00
Victor Stinner 22a351aabf Issue #10095: fp_setreadl() doesn't reopen the file, reuse instead the file
descriptor.
2010-10-14 12:04:34 +00:00
Antoine Pitrou f95a1b3c53 Recorded merge of revisions 81029 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r81029 | antoine.pitrou | 2010-05-09 16:46:46 +0200 (dim., 09 mai 2010) | 3 lines

  Untabify C files. Will watch buildbots.
........
2010-05-09 15:52:27 +00:00
Benjamin Peterson aeaa592516 Merged revisions 76230 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r76230 | benjamin.peterson | 2009-11-12 17:39:44 -0600 (Thu, 12 Nov 2009) | 2 lines

  fix several compile() issues by translating newlines in the tokenizer
........
2009-11-13 00:17:59 +00:00
Benjamin Peterson f5b52246ed ignore the coding cookie in compile(), exec(), and eval() if the source is a string #4626 2009-03-02 23:31:26 +00:00
Brett Cannon da78043237 Latin-1 source code was not being properly decoded when passed through
compile(). This was due to left-over special-casing before UTF-8 became the
default source encoding.

Closes issue #3574. Thanks to Victor Stinner for help with the patch.
2008-10-17 03:38:50 +00:00
Guido van Rossum 40d20bcf1f Issue 1267, continued.
Additional patch by Christian Heimes to deal more cleanly with the
FILE* vs file-descriptor issues.
I cleaned up his code a bit, and moved the lseek() call into import.c.
2007-10-22 00:09:51 +00:00
Guido van Rossum ce3a72aec6 Patch 1267 by Christian Heimes.
Move the initialization of sys.std{in,out,err} and __builtin__.open
to C code.
This solves the problem that "python -S" wouldn't work.
2007-10-19 23:16:50 +00:00
Neil Schemenauer 3f993c3b52 Use an enum for decoding_state. It makes the code a little more
understandable.
2007-09-21 20:50:26 +00:00
Thomas Wouters 89d996e5c2 Merged revisions 57778-58052 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r57820 | georg.brandl | 2007-08-31 08:59:27 +0200 (Fri, 31 Aug 2007) | 2 lines

  Document new shorthand notation for index entries.
........
  r57827 | georg.brandl | 2007-08-31 10:47:51 +0200 (Fri, 31 Aug 2007) | 2 lines

  Fix subitem markup.
........
  r57833 | martin.v.loewis | 2007-08-31 12:01:07 +0200 (Fri, 31 Aug 2007) | 1 line

  Mark registry components as 64-bit on Win64.
........
  r57854 | bill.janssen | 2007-08-31 21:02:23 +0200 (Fri, 31 Aug 2007) | 1 line

  deprecate use of FakeSocket
........
  r57855 | bill.janssen | 2007-08-31 21:02:46 +0200 (Fri, 31 Aug 2007) | 1 line

  remove mentions of socket.ssl in comments
........
  r57856 | bill.janssen | 2007-08-31 21:03:31 +0200 (Fri, 31 Aug 2007) | 1 line

  remove use of non-existent SSLFakeSocket in apparently untested code
........
  r57859 | martin.v.loewis | 2007-09-01 08:36:03 +0200 (Sat, 01 Sep 2007) | 3 lines

  Bug #1737210: Change Manufacturer of Windows installer to PSF.
  Will backport to 2.5.
........
  r57865 | georg.brandl | 2007-09-01 09:51:24 +0200 (Sat, 01 Sep 2007) | 2 lines

  Fix RST link (backport from Py3k).
........
  r57876 | georg.brandl | 2007-09-01 17:49:49 +0200 (Sat, 01 Sep 2007) | 2 lines

  Document sets' ">" and "<" operations (backport from py3k).
........
  r57878 | skip.montanaro | 2007-09-01 19:40:03 +0200 (Sat, 01 Sep 2007) | 4 lines

  Added a note and examples to explain that re.split does not split on an
  empty pattern match. (issue 852532).
........
  r57879 | walter.doerwald | 2007-09-01 20:18:09 +0200 (Sat, 01 Sep 2007) | 2 lines

  Fix wrong function names.
........
  r57880 | walter.doerwald | 2007-09-01 20:34:05 +0200 (Sat, 01 Sep 2007) | 2 lines

  Fix typo.
........
  r57889 | andrew.kuchling | 2007-09-01 22:31:59 +0200 (Sat, 01 Sep 2007) | 1 line

  Markup fix
........
  r57892 | andrew.kuchling | 2007-09-01 22:43:36 +0200 (Sat, 01 Sep 2007) | 1 line

  Add various items
........
  r57895 | andrew.kuchling | 2007-09-01 23:17:58 +0200 (Sat, 01 Sep 2007) | 1 line

  Wording change
........
  r57896 | andrew.kuchling | 2007-09-01 23:18:31 +0200 (Sat, 01 Sep 2007) | 1 line

  Add more items
........
  r57904 | ronald.oussoren | 2007-09-02 11:46:07 +0200 (Sun, 02 Sep 2007) | 3 lines

  Macosx: this patch ensures that the value of MACOSX_DEPLOYMENT_TARGET used
  by the Makefile is also used at configure-time.
........
  r57925 | georg.brandl | 2007-09-03 09:16:46 +0200 (Mon, 03 Sep 2007) | 2 lines

  Fix #883466: don't allow Unicode as arguments to quopri and uu codecs.
........
  r57936 | matthias.klose | 2007-09-04 01:33:04 +0200 (Tue, 04 Sep 2007) | 2 lines

  - Added support for linking the bsddb module against BerkeleyDB 4.6.x.
........
  r57954 | mark.summerfield | 2007-09-04 10:16:15 +0200 (Tue, 04 Sep 2007) | 3 lines

  Added cross-references plus a note about dict & list shallow copying.
........
  r57958 | martin.v.loewis | 2007-09-04 11:51:57 +0200 (Tue, 04 Sep 2007) | 3 lines

  Document that we rely on the OS to release the crypto
  context. Fixes #1626801.
........
  r57960 | martin.v.loewis | 2007-09-04 15:13:14 +0200 (Tue, 04 Sep 2007) | 3 lines

  Patch #1388440: Add set_completion_display_matches_hook and
  get_completion_type to readline.
........
  r57961 | martin.v.loewis | 2007-09-04 16:19:28 +0200 (Tue, 04 Sep 2007) | 3 lines

  Patch #1031213: Decode source line in SyntaxErrors back to its original
  source encoding. Will backport to 2.5.
........
  r57972 | matthias.klose | 2007-09-04 20:17:36 +0200 (Tue, 04 Sep 2007) | 3 lines

  - Makefile.pre.in(buildbottest): Run an optional script pybuildbot.identify
    to include some information about the build environment.
........
  r57973 | matthias.klose | 2007-09-04 21:05:38 +0200 (Tue, 04 Sep 2007) | 2 lines

  - Makefile.pre.in(buildbottest): Remove whitespace at eol.
........
  r57975 | matthias.klose | 2007-09-04 22:46:02 +0200 (Tue, 04 Sep 2007) | 2 lines

  - Fix libffi configure for hppa*-*-linux* | parisc*-*-linux*.
........
  r57980 | bill.janssen | 2007-09-05 02:46:27 +0200 (Wed, 05 Sep 2007) | 1 line

  SSL certificate distinguished names should be represented by tuples
........
  r57985 | martin.v.loewis | 2007-09-05 08:39:17 +0200 (Wed, 05 Sep 2007) | 3 lines

  Patch #1105: Explain that one needs to build the solution
  to get dependencies right.
........
  r57987 | armin.rigo | 2007-09-05 09:51:21 +0200 (Wed, 05 Sep 2007) | 4 lines

  PyDict_GetItem() returns a borrowed reference.
  There are probably a number of places that are open to attacks
  such as the following one, in bltinmodule.c:min_max().
........
  r57991 | martin.v.loewis | 2007-09-05 13:47:34 +0200 (Wed, 05 Sep 2007) | 3 lines

  Patch #786737: Allow building in a tree of symlinks pointing to
  a readonly source.
........
  r57993 | georg.brandl | 2007-09-05 15:36:44 +0200 (Wed, 05 Sep 2007) | 2 lines

  Backport from Py3k: Bug #1684991: explain lookup semantics for __special__ methods (new-style classes only).
........
  r58004 | armin.rigo | 2007-09-06 10:30:51 +0200 (Thu, 06 Sep 2007) | 4 lines

  Patch #1733973 by peaker:
  ptrace_enter_call() assumes no exception is currently set.
  This assumption is broken when throwing into a generator.
........
  r58006 | armin.rigo | 2007-09-06 11:30:38 +0200 (Thu, 06 Sep 2007) | 4 lines

  PyDict_GetItem() returns a borrowed reference.
  This attack is against ceval.c:IMPORT_NAME, which calls an
  object (__builtin__.__import__) without holding a reference to it.
........
  r58013 | georg.brandl | 2007-09-06 16:49:56 +0200 (Thu, 06 Sep 2007) | 2 lines

  Backport from 3k: #1116: fix reference to old filename.
........
  r58021 | thomas.heller | 2007-09-06 22:26:20 +0200 (Thu, 06 Sep 2007) | 1 line

  Fix typo:  c_float represents to C float type.
........
  r58022 | skip.montanaro | 2007-09-07 00:29:06 +0200 (Fri, 07 Sep 2007) | 3 lines

  If this is correct for py3k branch and it's already in the release25-maint
  branch, seems like it ought to be on the trunk as well.
........
  r58023 | gregory.p.smith | 2007-09-07 00:59:59 +0200 (Fri, 07 Sep 2007) | 4 lines

  Apply the fix from Issue1112 to make this test more robust and keep
  windows happy.
........
  r58031 | brett.cannon | 2007-09-07 05:17:50 +0200 (Fri, 07 Sep 2007) | 4 lines

  Make uuid1 and uuid4 tests conditional on whether ctypes can be imported;
  implementation of either function depends on ctypes but uuid as a whole does
  not.
........
  r58032 | brett.cannon | 2007-09-07 06:18:30 +0200 (Fri, 07 Sep 2007) | 6 lines

  Fix a crasher where Python code managed to infinitely recurse in C code without
  ever going back out to Python code in PyObject_Call().  Required introducing a
  static RuntimeError instance so that normalizing an exception there is no
  reliance on a recursive call that would put the exception system over the
  recursion check itself.
........
  r58034 | thomas.heller | 2007-09-07 08:32:17 +0200 (Fri, 07 Sep 2007) | 1 line

  Add a 'c_longdouble' type to the ctypes module.
........
  r58035 | thomas.heller | 2007-09-07 11:30:40 +0200 (Fri, 07 Sep 2007) | 1 line

  Remove unneeded #include.
........
  r58036 | thomas.heller | 2007-09-07 11:33:24 +0200 (Fri, 07 Sep 2007) | 6 lines

  Backport from py3k branch:

  Add a workaround for a strange bug on win64, when _ctypes is compiled
  with the SDK compiler.  This should fix the failing
  Lib\ctypes\test\test_as_parameter.py test.
........
  r58037 | georg.brandl | 2007-09-07 16:14:40 +0200 (Fri, 07 Sep 2007) | 2 lines

  Fix a wrong indentation for sublists.
........
  r58043 | georg.brandl | 2007-09-07 22:10:49 +0200 (Fri, 07 Sep 2007) | 2 lines

  #1095: ln -f doesn't work portably, fix in Makefile.
........
  r58049 | skip.montanaro | 2007-09-08 02:34:17 +0200 (Sat, 08 Sep 2007) | 1 line

  be explicit about the actual location of the missing file
........
2007-09-08 17:39:28 +00:00
Martin v. Löwis 85bcc66bb4 Convert code from sys.stdin.encoding to UTF-8 in
interactive mode. Fixes #1100.
2007-09-04 09:18:06 +00:00
Martin v. Löwis 49c5da1d88 Patch #1440601: Add col_offset attribute to AST nodes. 2006-03-01 22:49:05 +00:00
Martin v. Löwis 66485ae571 Remove unused field. 2006-03-01 04:04:20 +00:00
Martin v. Löwis 95292d6caa Constify filenames and scripts. Fixes #651362. 2002-12-11 14:04:59 +00:00
Martin v. Löwis f62a89b1e0 Ignore encoding declarations inside strings. Fixes #603509. 2002-09-03 11:52:44 +00:00
Martin v. Löwis 1ee99d31d9 Make pgen compile with pydebug. Duplicate normalized names, as it may
be longer than the old string.
2002-08-04 20:10:29 +00:00
Martin v. Löwis 00f1e3f5a5 Patch #534304: Implement phase 1 of PEP 263. 2002-08-04 17:29:52 +00:00
Guido van Rossum 8586991099 REMOVED all CWI, CNRI and BeOpen copyright markings.
This should match the situation in the 1.6b1 tree.
2000-09-01 23:29:29 +00:00
Tim Peters dbd9ba6a6c Nuke all remaining occurrences of Py_PROTO and Py_FPROTO. 2000-07-09 03:09:57 +00:00
Guido van Rossum ffcc3813d8 Change copyright notice - 2nd try. 2000-06-30 23:58:06 +00:00
Guido van Rossum fd71b9e9d4 Change copyright notice. 2000-06-30 23:50:40 +00:00
Guido van Rossum 926f13a081 Add checking for inconsistent tab usage 1998-04-09 21:38:06 +00:00
Guido van Rossum 86bea46b3d Another directory quickly renamed. 1997-04-29 21:03:06 +00:00
Guido van Rossum d266eb460e New permission notice, includes CNRI. 1996-10-25 14:44:06 +00:00
Guido van Rossum b9f8d6e54d Added 1995 to copyright message. 1995-01-04 19:08:09 +00:00
Guido van Rossum 1d5735e846 Merge back to main trunk 1994-08-30 08:27:36 +00:00
Guido van Rossum 248a50c168 * Grammar: corrected old typo (class instead of 'class')
* dosmodule.c: MSDOS specific stuff from posixmodule.c.
* posixmodule.c: removed all MSDOS specific stuff.
* tokenizer.h, parsetok.h: in prototypes, don't mix named and unnamed
  parameters (MSC doesn't like this).
1993-12-20 12:53:10 +00:00
Guido van Rossum a3309960a5 * Added support for X11 modules.
* Makefile: change location of FORMS library.
* posixmodule.c: turn #if 0 into #ifdef MSDOS (stuff in unistd.h or not)
* Almost all .h files: added CPP magic to avoid duplicate inclusions and
  to support inclusion from C++.
1993-07-28 09:05:47 +00:00
Guido van Rossum a849b834f1 * selectmodule.c: fix (another!) two memory leaks -- this time in list2set
* tokenizer.[ch]: allow continuation without \ inside () [] {}.
1993-05-12 11:35:44 +00:00
Guido van Rossum 6ac258d381 * pythonrun.c: Print exception type+arg *after* stack trace instead of
before it.
* ceval.c, object.c: moved testbool() to object.c (now extern visible)
* stringobject.c: fix bugs in and rationalize string resize in formatstring()
* tokenizer.[ch]: fix non-working code for lines longer than BUFSIZ
1993-05-12 08:24:20 +00:00
Guido van Rossum 9bfef44d97 * Changed all copyright messages to include 1993.
* Stubs for faster implementation of local variables (not yet finished)
* Added function name to code object.  Print it for code and function
  objects.  THIS MAKES THE .PYC FILE FORMAT INCOMPATIBLE (the version
  number has changed accordingly)
* Print address of self for built-in methods
* New internal functions getattro and setattro (getattr/setattr with
  string object arg)
* Replaced "dictobject" with more powerful "mappingobject"
* New per-type functio tp_hash to implement arbitrary object hashing,
  and hashobject() to interface to it
* Added built-in functions hash(v) and hasattr(v, 'name')
* classobject: made some functions static that accidentally weren't;
  added __hash__ special instance method to implement hash()
* Added proper comparison for built-in methods and functions
1993-03-29 10:43:31 +00:00
Guido van Rossum 5113f5fd34 Copyright for 1992 added 1992-04-05 14:20:22 +00:00
Guido van Rossum f70e43a073 Added copyright notice. 1991-02-19 12:39:46 +00:00
Guido van Rossum 85a5fbbdfe Initial revision 1990-10-14 12:07:46 +00:00