Commit Graph

231 Commits

Author SHA1 Message Date
Martin v. Löwis 16c7f71380 Handle testlist_safe and or_test like testlist. 2006-02-27 16:11:03 +00:00
Martin v. Löwis 49a1302952 Alias non-terminals introduced for backwards compatibility. 2006-02-27 16:03:39 +00:00
Neal Norwitz 212a575242 Remove unused _callers member. No need for types, use isinstance 2005-11-25 03:19:29 +00:00
Neal Norwitz ef260c0803 Stop looping to do nothing, just pass. 2005-11-25 03:18:58 +00:00
Neal Norwitz d752f7d8e8 No need for types, use isinstance 2005-11-25 03:17:59 +00:00
Neal Norwitz f9232678ae Use sorted() builtin 2005-11-25 03:16:34 +00:00
Jeremy Hylton 3e0055f8c6 Merge ast-branch to head
This change implements a new bytecode compiler, based on a
transformation of the parse tree to an abstract syntax defined in
Parser/Python.asdl.

The compiler implementation is not complete, but it is in stable
enough shape to run the entire test suite excepting two disabled
tests.
2005-10-20 19:59:25 +00:00
Phillip J. Eby 0d6615fd29 PEP 342 implementation. Per Guido's comments, the generator throw()
method still needs to support string exceptions, and allow None for the
third argument.  Documentation updates are needed, too.
2005-08-02 00:46:46 +00:00
Georg Brandl f0af0e7a46 patch [ 1231538 ] Typo fix in compiler/transformer.py (WalkerEror) 2005-07-02 18:37:41 +00:00
Neil Schemenauer f36947032f Fix compiler.ast.flatten function so that it works on lists. 2005-06-02 05:55:20 +00:00
Brett Cannon f4189916e3 Flush out support for ``class B(): pass`` syntax by adding support to the
'parser' module and 'compiler' package.

Closes patch #1176012.  Thanks logistix.
2005-04-09 02:30:16 +00:00
Michael W. Hudson 05522ad795 Fix bug
[ 1057835 ] compiler.transformer, "from module import *"
2004-11-08 12:17:34 +00:00
Michael W. Hudson c054a8b1a9 This is jiwon's patch to fix:
[ 1042238 ] Lib/compiler chokes on certain genexps
2004-10-11 15:35:53 +00:00
Tim Peters 0e9980f75a Whitespace normalization. 2004-09-12 03:49:31 +00:00
Jeremy Hylton 8f00a24229 Remove debugging print (not triggered by test suite) and add XXX comment about how the code should raise a SyntaxError. 2004-09-07 15:36:48 +00:00
Jeremy Hylton 566d934745 compiler.transformer: correct lineno attribute when possible
SF patch #1015989

The basic idea of this patch is to compute lineno attributes for all AST nodes.  The actual
implementation lead to a lot of restructing and code cleanup.

The generated AST nodes now have an optional lineno argument to constructor.  Remove the
top-level asList(), since it didn't seem to serve any purpose.  Add an __iter__ to ast nodes.
Use isinstance() instead of explicit type tests.

Change transformer to use the new lineno attribute, which replaces three lines of code with one.
Use universal newlines so that we can get rid of special-case code for line endings.  Use
lookup_node() in a few more frequently called, but simple com_xxx methods().  Change string
exception to class exception.
2004-09-07 15:28:01 +00:00
Anthony Baxter 1a4ddaecc7 SF patch #1007189, multi-line imports, for instance:
"from blah import (foo, bar
baz, bongo)"
2004-08-31 10:07:13 +00:00
Tim Peters a45cacfc1c Whitespace normalization. 2004-08-20 03:47:14 +00:00
Michael W. Hudson 0ccff074cd This is Mark Russell's patch:
[ 1009560 ] Fix @decorator evaluation order

From the description:

Changes in this patch:

- Change Grammar/Grammar to require
newlines between adjacent decorators.

- Fix order of evaluation of decorators
in the C (compile.c) and python
(Lib/compiler/pycodegen.py) compilers

- Add better order of evaluation check
to test_decorators.py (test_eval_order)

- Update the decorator documentation in
the reference manual (improve description
of evaluation order and update syntax
description)

and the comment:

Used Brett's evaluation order (see
http://mail.python.org/pipermail/python-dev/2004-August/047835.html)

(I'm checking this in for Anthony who was having problems getting SF to
talk to him)
2004-08-17 17:29:16 +00:00
Jeremy Hylton 2876f5ad21 SF patch 836879.
Don't generate code for asserts in -O mode.
2004-08-07 19:21:56 +00:00
Tim Peters 6db15d7307 Whitespace normalization. 2004-08-04 02:36:18 +00:00
Anthony Baxter c2a5a63654 PEP-0318, @decorator-style. In Guido's words:
"@ seems the syntax that everybody can hate equally"
Implementation by Mark Russell, from SF #979728.
2004-08-02 06:10:11 +00:00
Michael W. Hudson 3f06189332 this is patch
[ 988698 ] compiler.transformer fix for (a, b) = 1, 2

fixing bug

[ 988613 ] compiler.transformer and tuple unpacking
2004-07-12 13:15:56 +00:00
Tim Peters 4e0e1b6a54 Whitespace normalization. 2004-07-07 20:54:48 +00:00
Raymond Hettinger 354433a59d SF patch #872326: Generator expression implementation
(Code contributed by Jiwon Seo.)

The documentation portion of the patch is being re-worked and will be
checked-in soon.  Likewise, PEP 289 will be updated to reflect Guido's
rationale for the design decisions on binding behavior (as described in
in his patch comments and in discussions on python-dev).

The test file, test_genexps.py, is written in doctest format and is
meant to exercise all aspects of the the patch.  Further additions are
welcome from everyone.  Please stress test this new feature as much as
possible before the alpha release.
2004-05-19 08:20:33 +00:00
Neil Schemenauer 4848557924 Remove unused instance attributes. 2004-03-21 15:18:50 +00:00
Jeremy Hylton ead21f596c Fix SF bug [ 788011 ] compiler.compileFile fails on csv.py
Bug fix candidate.
2003-08-28 02:09:26 +00:00
Andrew M. Kuchling b2f89ee71a Comment typo fixes 2003-08-11 16:20:39 +00:00
Michael W. Hudson 896e5164bb Jacob Hallen cornered me here at EuroPython and got me to look at
patch:

[ 750008 ] 'compiler' module bug with 'import foo.bar as baz'

which I'm now checking in.

after import foo.bar as baz, baz would refer to foo.
2003-06-27 12:32:39 +00:00
Raymond Hettinger ff41c48a77 SF patch #701494: more apply removals 2003-04-06 09:01:11 +00:00
Tim Peters 669454e9dc Whitespace normalization. 2003-03-07 17:30:48 +00:00
Michael W. Hudson 8d0ffe0126 Remove debugging prints. 2003-01-03 10:25:20 +00:00
Jeremy Hylton 1856541142 Replace all but one explicit emit('SET_LINENO') with call to set_lineno().
Remove broken code in visitDict().  I assume the code was trying to
add set lineno events for each line of a dict constructor, but I think
it was using the wrong object (node instead of k or v).
2002-12-31 18:26:17 +00:00
Jeremy Hylton accb62b28e SF patch [ 597919 ] compiler package and SET_LINENO
A variety of changes from Michael Hudson to get the compiler working
with 2.3.  The primary change is the handling of SET_LINENO:

# The set_lineno() function and the explicit emit() calls for
# SET_LINENO below are only used to generate the line number table.
# As of Python 2.3, the interpreter does not have a SET_LINENO
# instruction.  pyassem treats SET_LINENO opcodes as a special case.

A few other small changes:
 - Remove unused code from pycodegen and pyassem.
 - Fix error handling in parsermodule.  When PyParser_SimplerParseString()
   fails, it sets an exception with detailed info.  The parsermodule
   was clobbering that exception and replacing it was a generic
   "could not parse string" exception.  Keep the original exception.
2002-12-31 18:17:44 +00:00
Gustavo Niemeyer 78429a6aa6 Fixing bug
[#448679] Left to right

* Python/compile.c
  (com_dictmaker): Reordered evaluation of dictionaries to follow strict
  LTR evaluation.

* Lib/compiler/pycodegen.py
  (CodeGenerator.visitDict): Reordered evaluation of dictionaries to
  follow strict LTR evaluation.

* Doc/ref/ref5.tex
  Documented the general LTR evaluation order idea.

* Misc/NEWS
  Documented change in evaluation order of dictionaries.
2002-12-16 13:54:02 +00:00
Neal Norwitz a312c3ade7 Remove uses of string module and stat.ST_MODE 2002-06-06 18:30:10 +00:00
Jeremy Hylton 8d22b0409b Fix com_arglist() and update grammar fragment.
SF bug #522264 reported by Evelyn Mitchell.

The code included a comment about "STAR STAR" which was translated
into the code as the bogus attribute token.STARSTAR.  This name never
caused an attribute error because it was never retrieved.  The code
was based on an old version of the grammar that specified kwargs as
two tokens ('*' '*').  I checked as far back as 2.1 and didn't find
this production.

The fix is simple, because token.DOUBLESTAR is the only token
allowed.  Also update the grammar fragment in com_arglist().

XXX I'll bet lots of other grammar fragments in comments are out of
date, probably in this module and in compile.c.
2002-04-19 22:56:37 +00:00
Jeremy Hylton 16af557ae9 Remove more out-of-date comments and clarify explanation of visit(). 2002-04-18 16:26:40 +00:00
Jeremy Hylton 3b05c8ee4a Remove out-of-date explanation in doc strings,
as reported by Eric C. Newton.
2002-04-18 16:02:48 +00:00
Martin v. Löwis 51e234aa68 Patch #522279: transformer.py nodes shadows global. 2002-02-28 17:48:48 +00:00
Martin v. Löwis 0ac16ec149 Patch #521714: fix pychecker warnings in ast.py. 2002-02-23 22:35:33 +00:00
Tim Peters e4418609f7 Whitespace normalization. 2002-02-16 07:34:19 +00:00
Barry Warsaw 52acb49298 Merge of the release22 branch changes back into the trunk. 2001-12-21 20:04:22 +00:00
Jeremy Hylton 57911ae35a Fix [ #484645 ] little bug in pycodegen.py 2001-11-27 23:35:10 +00:00
Jeremy Hylton 6383c2d1a6 Fix SF bug #479186: compiler generates bad code for "del"
Fix by Neil Schemenauer.  Visit the Subscript node when trying to find
the operation for a statement.

XXX Not sure if there are other nodes that should be visited.
2001-11-09 16:24:34 +00:00
Tim Peters e0c446bb4a Whitespace normalization. 2001-10-18 21:57:37 +00:00
Jeremy Hylton 138d90eb73 Vastly improved stacksize calculation.
There are now no known cases where the compiler package computes a
stack depth lower than the one computed by the builtin compiler.  (To
achieve this state, we had to fix bugs in both compilers :-).

The chief change is to do the depth calculations with respect to basic
blocks.  The stack effect of block is calculated.  Then the flow graph
is traversed using breadth-first search to find the max weight path
through the graph.

Had to fix the StackDepthTracker to calculate the right info for
several opcodes: LOAD_ATTR, CALL_FUNCTION (and friends), MAKE_CLOSURE,
and DUP_TOPX.

XXX Still need to handle free variables in MAKE_CLOSURE.

XXX There are still a lot of places where the computed stack depth is
larger than for the builtin compiler.  These won't cause the
interpreter to overflow the frame, but they waste space.
2001-10-17 13:37:29 +00:00
Jeremy Hylton d114261608 Handle testlist_safe as if it were testlist. 2001-10-17 13:32:52 +00:00
Andrew M. Kuchling 264e8186d5 Fix comment typo 2001-09-27 04:18:36 +00:00
Jeremy Hylton 9dca36432e API change:
compile() becomes replacement for builtin compile()
compileFile() generates a .pyc from a .py
both are exported in __init__

compiler.parse() gets optional second argument to specify compilation
mode, e.g. single, eval, exec

Add AbstractCompileMode as parent class and Module, Expression, and
Interactive as concrete subclasses.  Each corresponds to a compilation
mode.

THe AbstractCompileMode instances in turn delegate to CodeGeneration
subclasses specialized for their particular functions --
ModuleCodeGenerator, ExpressionCodeGeneration,
InteractiveCodeGenerator.
2001-09-17 21:02:51 +00:00
Jeremy Hylton c8ed18a4e3 Re-created after change to astgen to calculate hardest_arg correctly 2001-09-17 20:17:02 +00:00
Jeremy Hylton 2e4cc7e0d8 Last set of change to get regression tests to pass
Remove the only test in the syntax module.  It ends up that the
transformer must handle this error case.

In the transformer, check for a list compression in com_assign_list()
by looking for a list_for node where a comma is expected.

In pycodegen.compile() re-raise the SyntaxError rather than catching
it and exiting
2001-09-17 19:33:48 +00:00
Jeremy Hylton 37c9351cf6 Handle more syntax errors.
Invoke compiler.syntax.check() after building AST.  If a SyntaxError
occurs, print the error and exit without generating a .pyc file.

Refactor code to use compiler.misc.set_filename() rather than passing
filename argument around to each CodeGenerator instance.
2001-09-17 18:03:55 +00:00
Jeremy Hylton 09392b77a4 Add utility to set filename attribute on all nodes 2001-09-17 18:02:21 +00:00
Jeremy Hylton aee0bfedcc support true division 2001-09-17 16:41:02 +00:00
Jeremy Hylton 1048aa933f Add code generator for yield stmt 2001-09-14 23:17:55 +00:00
Jeremy Hylton 6a9cac68b6 del no longer necessary now that new module is gone 2001-09-14 22:54:48 +00:00
Jeremy Hylton 1e99a77120 Various sundry changes for 2.2 compatibility
Remove the option to have nested scopes or old LGB scopes.  This has a
large impact on the code base, by removing the need for two variants
of each CodeGenerator.

Add a get_module() method to CodeGenerator objects, used to get the
future features for the current module.

Set CO_GENERATOR, CO_GENERATOR_ALLOWED, and CO_FUTURE_DIVISION flags
as appropriate.

Attempt to fix the value of nlocals in newCodeObject(), assuming that
nlocals is 0 if CO_NEWLOCALS is not defined.
2001-09-14 22:49:08 +00:00
Jeremy Hylton 652a22437a The object-being sliced in an assignment to a slice is referenced, not
bound.

When a Yield() node is visited, assign to the generator attribute of
the scope, not the visitor.
2001-09-14 22:45:57 +00:00
Jeremy Hylton 9ee78f7d61 the new new doesn't define CO_xxx as the old new did 2001-09-14 22:44:35 +00:00
Jeremy Hylton fff252d20d the names attribute of Global is not a node 2001-09-14 22:40:36 +00:00
Jeremy Hylton 71ebc3359b Fix _convert_NAME() so that it doesn't store locals for class bodies.
Fix list comp code generation -- emit GET_ITER instead of Const(0)
after the list.

Add CO_GENERATOR flag to generators.

Get CO_xxx flags from the new module
2001-08-30 20:25:55 +00:00
Jeremy Hylton f71b5fec43 spurious pop 2001-08-30 15:50:34 +00:00
Jeremy Hylton e4685ec57e Track the block stack more reasonably in order to handle continue in
try/except or try/finally.

Previous versions had only track SETUP_LOOP blocks and ignored the
exception part.  This meant that it allowed continue inside a
try/except but generated buggy code.  Now it does the right thing.
2001-08-29 22:30:09 +00:00
Jeremy Hylton 9263848fa1 Improve stack depth computation for try/except and try/finally
Add CONTINUE_LOOP to the list of unconditional transfers
2001-08-29 22:27:14 +00:00
Jeremy Hylton 4bd4dddd55 Add __getitem__() handler for use by visitContinue() 2001-08-29 22:26:35 +00:00
Jeremy Hylton 1936745668 Generate SET_LINENO for list and tuple literals when the open paren
starts a new line.

Also fix undetected typo in visitDict() -- uncovered by recent change
to add lineno attrs to atoms.
2001-08-29 20:57:43 +00:00
Jeremy Hylton 7845cf8d37 Make sure that atoms (Tuple, List, etc.) have lineno attributes 2001-08-29 20:56:30 +00:00
Jeremy Hylton 4ba9001f5c Fix off-by-one errors in code to find depth of stack.
XXX The code is still widely inaccurate, but most (all?) of the time
it's an overestimate.
2001-08-29 20:55:17 +00:00
Jeremy Hylton bf77c465bd Undo change from list to dict for handling varnames, consts, etc.
As the doc string for _lookupName() explains:

    This routine uses a list instead of a dictionary, because a
    dictionary can't store two different keys if the keys have the
    same value but different types, e.g. 2 and 2L.  The compiler
    must treat these two separately, so it does an explicit type
    comparison before comparing the values.
2001-08-29 19:45:33 +00:00
Jeremy Hylton 5a9ac97040 Change default() to use getChildNodes() instead of getChildren() 2001-08-29 18:17:22 +00:00
Jeremy Hylton 94afe32b5e Support // and //=
Generate SET_LINENO for del statements.

Define klass=1 for PyFlowGraph constructor for a class statement.  A
class has no varnames.
2001-08-29 18:14:39 +00:00
Jeremy Hylton 7abf520d6c Add support for // and //=.
Avoid if/elif/elif/else tests where the final else is supposed to
handle exactly one case instead of all other cases.  When the list of
operators is extended, the catchall else treats all new operators as
the last operator in the set of tests.  Instead, raise an exception if
an unexpected operator occurs.
2001-08-29 18:12:30 +00:00
Jeremy Hylton d4be10dc2c Add generator detection to symbol table.
Fix bug in handling of statements like "l[x:y] = 2".  The visitor was
treating this as assignments to l, x, and y!
2001-08-29 18:10:51 +00:00
Jeremy Hylton e4e9cd4c01 Modify name conversion to be (hopefully) a bit more efficient.
Use a dictionary instead of a list to map objects to their offsets in
a const/name tuple of a code object.

XXX The conversion is perhaps incomplete, in that we shouldn't have to
do the list2dict to start.
2001-08-29 18:09:50 +00:00
Jeremy Hylton 5477f529d6 Revise implementations of getChildren() and getChildNodes().
Add support for floor division (// and //=)

The implementation of getChildren() and getChildNodes() is intended to
be faster, because it avoids calling flatten() on every return value.
But it's not clear that it is a lot faster, because constructing a
tuple with just the right values ends up being slow.  (Too many
attribute lookups probably.)

The ast.txt file is much more complicated, with funny characters at
the ends of names (*, &, !) to indicate the types of each child node.

The astgen script is also much more complex, making me wonder if it's
still useful.
2001-08-29 18:08:02 +00:00
Jeremy Hylton da8db8ca18 Don't include doc string of class in its code child 2001-08-29 17:19:02 +00:00
Jeremy Hylton 2ac9c3eec5 Make sure the JUMP_ABSOLUTE and POP_BLOCK at the end of a for loop are
contiguous.
2001-08-28 17:28:33 +00:00
Jeremy Hylton 63db7b9ca1 XXX_NAME ops should affect varnames
varnames should list all the local variables (with arguments first).
The XXX_NAME ops typically occur at the module level and assignment
ops should create locals.
2001-08-28 16:36:12 +00:00
Jeremy Hylton f354575328 Generate FOR_ITER-based loops instead of old FOR_LOOP-based loops 2001-08-28 16:35:18 +00:00
Jeremy Hylton c59e220000 Handle private names
(Hard to believe these were never handled before)

Add misc.mangle() that mangles based on the rules in compile.c.
XXX Need to test the corner cases

Update CodeGenerator with a class_name attribute bound to None.  If a
particular instance is created within a class scope, the instance's
class_name is bound to that class's name.

Add mangle() method to CodeGenerator that mangles if the class_name
has a class_name in it.

Modify the FunctionCodeGenerator family to handle an extra argument--
the class_name.

Wrap all name ops and attrnames in calls to self.mangle()
2001-08-27 22:56:16 +00:00
Jeremy Hylton 80ea40d858 emit SET_LINENO for augmented assignments 2001-08-27 21:58:09 +00:00
Jeremy Hylton 2afff324ea Many changes -- bug fixes and sundry improvements
Make nested scopes enabled by default

Add is_constant_false() helper so that compiled code and symbols are
consistent with builtin compiler's handling of "if 0:"

Fix doc string handling to be consistent with recent change that
eliminates the doc string from the Module's node attribute.

Add fix to print handling from Evan & Shane.

Track change to visitor api by making "verbose" explicit.

Comment out setting CO_NESTED flag (it's unnecessary in 2.2).
2001-08-27 21:51:52 +00:00
Jeremy Hylton cd8a127e1a Fix for sibling nodes that define the same free variable
Evan Simpson's fix.  And his explanation:

    If you defined two nested functions in a row that refer to the
    same non-global variable, the second one will be generated as
    though the variable were global.
2001-08-27 21:06:35 +00:00
Jeremy Hylton 7e30c9bb5a Add lookup_name() to optimize use of stack frames
The use of com_node() introduces a lot of extra stack frames, enough
to cause a stack overflow compiling test.test_parser with the standard
interpreter recursionlimit.  The com_node() is a convenience function
that hides the dispatch details, but comes at a very high cost.  It is
more efficient to dispatch directly in the callers.  In these cases,
use lookup_node() and call the dispatched node directly.

Also handle yield_stmt in a way that will work with Python 2.1
(suggested by Shane Hathaway)
2001-08-27 21:02:51 +00:00
Jeremy Hylton 058a5adad0 Two changes to visitor API:
Remove _preorder as alias for dispatch and call dispatch directly.
    Add an extra optional argument to walk()

XXX Also comment out some code that does debugging prints.
2001-08-27 20:47:08 +00:00
Jeremy Hylton ab427b8cce Generate correct reprs for Mul, Add, etc. 2001-08-18 00:14:37 +00:00
Jeremy Hylton ec5bfd13ca Track removal of doc string from Module().nodes[0] 2001-08-18 00:07:46 +00:00
Jeremy Hylton 8548f9b183 Add Yield() node 2001-08-18 00:07:14 +00:00
Jeremy Hylton ec927348c2 Add Yield() statement handler
Fix Module() handler to avoid including the doc string in the AST
2001-08-18 00:04:31 +00:00
Jeremy Hylton 9272b14d62 Fix typo in astgen script 2001-08-14 21:18:30 +00:00
Jeremy Hylton 7713ac2ff1 Regenerated from new ast.txt and new astgen.py 2001-08-14 18:59:01 +00:00
Tim Peters 2a7f384122 SF bug 430991: wrong co_lnotab
Armin Rigo pointed out that the way the line-# table got built didn't work
for lines generating more than 255 bytes of bytecode.  Fixed as he
suggested, plus corresponding changes to pyassem.py, plus added some
long overdue docs about this subtle table to compile.c.

Bugfix candidate (line numbers may be off in tracebacks under -O).
2001-06-09 09:26:21 +00:00
Jeremy Hylton bb0bae6da2 Pop loop off the loop stack before handling the loop's else clause.
Otherwise, continue/break will attempt to affect the wrong loop.

A few more fiddles to get the SET_LINENOs consistent across compilers.
2001-04-12 21:54:41 +00:00
Jeremy Hylton 13d70944cb Use new _implicitNameOp() to generate name op code for list comprehensions.
Always emit a SET_LINENO 0 at the beginning of the module.  The
builtin compiler does this, and it's much easier to compare bytecode
generated by the two compilers if they both do.

Move the SET_LINENO inside the FOR_LOOP block for list
comprehensions.  Also for compat. with builtin compiler.
2001-04-12 21:04:43 +00:00
Jeremy Hylton 614e87f286 Add support for visitAssAttr to findOp(). 2001-04-12 20:24:26 +00:00
Jeremy Hylton 542b11acfd pyassem.py:
Fix annoying bugs in flow graph layout code.  In some cases the
    implicit control transfers weren't honored.  In other cases,
    JUMP_FORWARD instructions jumped backwards.

    Remove unused arg from nextBlock().

pycodegen.py

    Add optional force kwarg to set_lineno() that will emit a
    SET_LINENO even if it is the same as the previous lineno.

    Use explicit LOAD_FAST and STORE_FAST to access list comp implicit
    variables.  (The symbol table doesn't know about them.)
2001-04-12 20:21:39 +00:00
Jeremy Hylton bfb0cf822b Revise handling of tuple arguments so that the variables names match
those used by compile.c.  (test_grammar now depends on the names)
2001-04-12 17:33:34 +00:00
Jeremy Hylton 5c9aad6043 Only treat an AugAssign as def if its the target is a Name.
Fixes last bug found with test_scope.py.
2001-04-12 07:06:25 +00:00
Jeremy Hylton 3f76b7e6e4 Fix unpackSequence() to use _nameOp() rather than LOAD_FAST 2001-04-12 06:52:27 +00:00