This commit simplifies async/await tokenization in tokenizer.c,
tokenize.py & lib2to3/tokenize.py. Previous solution was to keep
a stack of async-def & def blocks, whereas the new approach is just
to remember position of the outermost async-def block.
This change won't bring any parsing performance improvements, but
it makes the code much easier to read and validate.
This commit fixes how one-line async-defs and defs are tracked
by tokenizer. It allows to correctly parse invalid code such
as:
>>> async def f():
... def g(): pass
... async = 10
and valid code such as:
>>> async def f():
... async def g(): pass
... await z
As a consequence, is is now possible to have one-line
'async def foo(): await ..' functions:
>>> async def foo(): return await bar()
compile(). This was due to left-over special-casing before UTF-8 became the
default source encoding.
Closes issue #3574. Thanks to Victor Stinner for help with the patch.
Additional patch by Christian Heimes to deal more cleanly with the
FILE* vs file-descriptor issues.
I cleaned up his code a bit, and moved the lseek() call into import.c.
svn+ssh://pythondev@svn.python.org/python/trunk
........
r57820 | georg.brandl | 2007-08-31 08:59:27 +0200 (Fri, 31 Aug 2007) | 2 lines
Document new shorthand notation for index entries.
........
r57827 | georg.brandl | 2007-08-31 10:47:51 +0200 (Fri, 31 Aug 2007) | 2 lines
Fix subitem markup.
........
r57833 | martin.v.loewis | 2007-08-31 12:01:07 +0200 (Fri, 31 Aug 2007) | 1 line
Mark registry components as 64-bit on Win64.
........
r57854 | bill.janssen | 2007-08-31 21:02:23 +0200 (Fri, 31 Aug 2007) | 1 line
deprecate use of FakeSocket
........
r57855 | bill.janssen | 2007-08-31 21:02:46 +0200 (Fri, 31 Aug 2007) | 1 line
remove mentions of socket.ssl in comments
........
r57856 | bill.janssen | 2007-08-31 21:03:31 +0200 (Fri, 31 Aug 2007) | 1 line
remove use of non-existent SSLFakeSocket in apparently untested code
........
r57859 | martin.v.loewis | 2007-09-01 08:36:03 +0200 (Sat, 01 Sep 2007) | 3 lines
Bug #1737210: Change Manufacturer of Windows installer to PSF.
Will backport to 2.5.
........
r57865 | georg.brandl | 2007-09-01 09:51:24 +0200 (Sat, 01 Sep 2007) | 2 lines
Fix RST link (backport from Py3k).
........
r57876 | georg.brandl | 2007-09-01 17:49:49 +0200 (Sat, 01 Sep 2007) | 2 lines
Document sets' ">" and "<" operations (backport from py3k).
........
r57878 | skip.montanaro | 2007-09-01 19:40:03 +0200 (Sat, 01 Sep 2007) | 4 lines
Added a note and examples to explain that re.split does not split on an
empty pattern match. (issue 852532).
........
r57879 | walter.doerwald | 2007-09-01 20:18:09 +0200 (Sat, 01 Sep 2007) | 2 lines
Fix wrong function names.
........
r57880 | walter.doerwald | 2007-09-01 20:34:05 +0200 (Sat, 01 Sep 2007) | 2 lines
Fix typo.
........
r57889 | andrew.kuchling | 2007-09-01 22:31:59 +0200 (Sat, 01 Sep 2007) | 1 line
Markup fix
........
r57892 | andrew.kuchling | 2007-09-01 22:43:36 +0200 (Sat, 01 Sep 2007) | 1 line
Add various items
........
r57895 | andrew.kuchling | 2007-09-01 23:17:58 +0200 (Sat, 01 Sep 2007) | 1 line
Wording change
........
r57896 | andrew.kuchling | 2007-09-01 23:18:31 +0200 (Sat, 01 Sep 2007) | 1 line
Add more items
........
r57904 | ronald.oussoren | 2007-09-02 11:46:07 +0200 (Sun, 02 Sep 2007) | 3 lines
Macosx: this patch ensures that the value of MACOSX_DEPLOYMENT_TARGET used
by the Makefile is also used at configure-time.
........
r57925 | georg.brandl | 2007-09-03 09:16:46 +0200 (Mon, 03 Sep 2007) | 2 lines
Fix#883466: don't allow Unicode as arguments to quopri and uu codecs.
........
r57936 | matthias.klose | 2007-09-04 01:33:04 +0200 (Tue, 04 Sep 2007) | 2 lines
- Added support for linking the bsddb module against BerkeleyDB 4.6.x.
........
r57954 | mark.summerfield | 2007-09-04 10:16:15 +0200 (Tue, 04 Sep 2007) | 3 lines
Added cross-references plus a note about dict & list shallow copying.
........
r57958 | martin.v.loewis | 2007-09-04 11:51:57 +0200 (Tue, 04 Sep 2007) | 3 lines
Document that we rely on the OS to release the crypto
context. Fixes#1626801.
........
r57960 | martin.v.loewis | 2007-09-04 15:13:14 +0200 (Tue, 04 Sep 2007) | 3 lines
Patch #1388440: Add set_completion_display_matches_hook and
get_completion_type to readline.
........
r57961 | martin.v.loewis | 2007-09-04 16:19:28 +0200 (Tue, 04 Sep 2007) | 3 lines
Patch #1031213: Decode source line in SyntaxErrors back to its original
source encoding. Will backport to 2.5.
........
r57972 | matthias.klose | 2007-09-04 20:17:36 +0200 (Tue, 04 Sep 2007) | 3 lines
- Makefile.pre.in(buildbottest): Run an optional script pybuildbot.identify
to include some information about the build environment.
........
r57973 | matthias.klose | 2007-09-04 21:05:38 +0200 (Tue, 04 Sep 2007) | 2 lines
- Makefile.pre.in(buildbottest): Remove whitespace at eol.
........
r57975 | matthias.klose | 2007-09-04 22:46:02 +0200 (Tue, 04 Sep 2007) | 2 lines
- Fix libffi configure for hppa*-*-linux* | parisc*-*-linux*.
........
r57980 | bill.janssen | 2007-09-05 02:46:27 +0200 (Wed, 05 Sep 2007) | 1 line
SSL certificate distinguished names should be represented by tuples
........
r57985 | martin.v.loewis | 2007-09-05 08:39:17 +0200 (Wed, 05 Sep 2007) | 3 lines
Patch #1105: Explain that one needs to build the solution
to get dependencies right.
........
r57987 | armin.rigo | 2007-09-05 09:51:21 +0200 (Wed, 05 Sep 2007) | 4 lines
PyDict_GetItem() returns a borrowed reference.
There are probably a number of places that are open to attacks
such as the following one, in bltinmodule.c:min_max().
........
r57991 | martin.v.loewis | 2007-09-05 13:47:34 +0200 (Wed, 05 Sep 2007) | 3 lines
Patch #786737: Allow building in a tree of symlinks pointing to
a readonly source.
........
r57993 | georg.brandl | 2007-09-05 15:36:44 +0200 (Wed, 05 Sep 2007) | 2 lines
Backport from Py3k: Bug #1684991: explain lookup semantics for __special__ methods (new-style classes only).
........
r58004 | armin.rigo | 2007-09-06 10:30:51 +0200 (Thu, 06 Sep 2007) | 4 lines
Patch #1733973 by peaker:
ptrace_enter_call() assumes no exception is currently set.
This assumption is broken when throwing into a generator.
........
r58006 | armin.rigo | 2007-09-06 11:30:38 +0200 (Thu, 06 Sep 2007) | 4 lines
PyDict_GetItem() returns a borrowed reference.
This attack is against ceval.c:IMPORT_NAME, which calls an
object (__builtin__.__import__) without holding a reference to it.
........
r58013 | georg.brandl | 2007-09-06 16:49:56 +0200 (Thu, 06 Sep 2007) | 2 lines
Backport from 3k: #1116: fix reference to old filename.
........
r58021 | thomas.heller | 2007-09-06 22:26:20 +0200 (Thu, 06 Sep 2007) | 1 line
Fix typo: c_float represents to C float type.
........
r58022 | skip.montanaro | 2007-09-07 00:29:06 +0200 (Fri, 07 Sep 2007) | 3 lines
If this is correct for py3k branch and it's already in the release25-maint
branch, seems like it ought to be on the trunk as well.
........
r58023 | gregory.p.smith | 2007-09-07 00:59:59 +0200 (Fri, 07 Sep 2007) | 4 lines
Apply the fix from Issue1112 to make this test more robust and keep
windows happy.
........
r58031 | brett.cannon | 2007-09-07 05:17:50 +0200 (Fri, 07 Sep 2007) | 4 lines
Make uuid1 and uuid4 tests conditional on whether ctypes can be imported;
implementation of either function depends on ctypes but uuid as a whole does
not.
........
r58032 | brett.cannon | 2007-09-07 06:18:30 +0200 (Fri, 07 Sep 2007) | 6 lines
Fix a crasher where Python code managed to infinitely recurse in C code without
ever going back out to Python code in PyObject_Call(). Required introducing a
static RuntimeError instance so that normalizing an exception there is no
reliance on a recursive call that would put the exception system over the
recursion check itself.
........
r58034 | thomas.heller | 2007-09-07 08:32:17 +0200 (Fri, 07 Sep 2007) | 1 line
Add a 'c_longdouble' type to the ctypes module.
........
r58035 | thomas.heller | 2007-09-07 11:30:40 +0200 (Fri, 07 Sep 2007) | 1 line
Remove unneeded #include.
........
r58036 | thomas.heller | 2007-09-07 11:33:24 +0200 (Fri, 07 Sep 2007) | 6 lines
Backport from py3k branch:
Add a workaround for a strange bug on win64, when _ctypes is compiled
with the SDK compiler. This should fix the failing
Lib\ctypes\test\test_as_parameter.py test.
........
r58037 | georg.brandl | 2007-09-07 16:14:40 +0200 (Fri, 07 Sep 2007) | 2 lines
Fix a wrong indentation for sublists.
........
r58043 | georg.brandl | 2007-09-07 22:10:49 +0200 (Fri, 07 Sep 2007) | 2 lines
#1095: ln -f doesn't work portably, fix in Makefile.
........
r58049 | skip.montanaro | 2007-09-08 02:34:17 +0200 (Sat, 08 Sep 2007) | 1 line
be explicit about the actual location of the missing file
........
* dosmodule.c: MSDOS specific stuff from posixmodule.c.
* posixmodule.c: removed all MSDOS specific stuff.
* tokenizer.h, parsetok.h: in prototypes, don't mix named and unnamed
parameters (MSC doesn't like this).
* Makefile: change location of FORMS library.
* posixmodule.c: turn #if 0 into #ifdef MSDOS (stuff in unistd.h or not)
* Almost all .h files: added CPP magic to avoid duplicate inclusions and
to support inclusion from C++.
before it.
* ceval.c, object.c: moved testbool() to object.c (now extern visible)
* stringobject.c: fix bugs in and rationalize string resize in formatstring()
* tokenizer.[ch]: fix non-working code for lines longer than BUFSIZ
* Stubs for faster implementation of local variables (not yet finished)
* Added function name to code object. Print it for code and function
objects. THIS MAKES THE .PYC FILE FORMAT INCOMPATIBLE (the version
number has changed accordingly)
* Print address of self for built-in methods
* New internal functions getattro and setattro (getattr/setattr with
string object arg)
* Replaced "dictobject" with more powerful "mappingobject"
* New per-type functio tp_hash to implement arbitrary object hashing,
and hashobject() to interface to it
* Added built-in functions hash(v) and hasattr(v, 'name')
* classobject: made some functions static that accidentally weren't;
added __hash__ special instance method to implement hash()
* Added proper comparison for built-in methods and functions
Yury Selivanov
Serhiy Storchaka
Victor Stinner
Georg Brandl
Antoine Pitrou
Benjamin Peterson
Brett Cannon
Guido van Rossum
Neil Schemenauer
Thomas Wouters
Martin v. Löwis
Tim Peters