Commit Graph

182 Commits

Author SHA1 Message Date
Jeremy Hylton d4be10dc2c Add generator detection to symbol table.
Fix bug in handling of statements like "l[x:y] = 2".  The visitor was
treating this as assignments to l, x, and y!
2001-08-29 18:10:51 +00:00
Jeremy Hylton e4e9cd4c01 Modify name conversion to be (hopefully) a bit more efficient.
Use a dictionary instead of a list to map objects to their offsets in
a const/name tuple of a code object.

XXX The conversion is perhaps incomplete, in that we shouldn't have to
do the list2dict to start.
2001-08-29 18:09:50 +00:00
Jeremy Hylton 5477f529d6 Revise implementations of getChildren() and getChildNodes().
Add support for floor division (// and //=)

The implementation of getChildren() and getChildNodes() is intended to
be faster, because it avoids calling flatten() on every return value.
But it's not clear that it is a lot faster, because constructing a
tuple with just the right values ends up being slow.  (Too many
attribute lookups probably.)

The ast.txt file is much more complicated, with funny characters at
the ends of names (*, &, !) to indicate the types of each child node.

The astgen script is also much more complex, making me wonder if it's
still useful.
2001-08-29 18:08:02 +00:00
Jeremy Hylton da8db8ca18 Don't include doc string of class in its code child 2001-08-29 17:19:02 +00:00
Jeremy Hylton 2ac9c3eec5 Make sure the JUMP_ABSOLUTE and POP_BLOCK at the end of a for loop are
contiguous.
2001-08-28 17:28:33 +00:00
Jeremy Hylton 63db7b9ca1 XXX_NAME ops should affect varnames
varnames should list all the local variables (with arguments first).
The XXX_NAME ops typically occur at the module level and assignment
ops should create locals.
2001-08-28 16:36:12 +00:00
Jeremy Hylton f354575328 Generate FOR_ITER-based loops instead of old FOR_LOOP-based loops 2001-08-28 16:35:18 +00:00
Jeremy Hylton c59e220000 Handle private names
(Hard to believe these were never handled before)

Add misc.mangle() that mangles based on the rules in compile.c.
XXX Need to test the corner cases

Update CodeGenerator with a class_name attribute bound to None.  If a
particular instance is created within a class scope, the instance's
class_name is bound to that class's name.

Add mangle() method to CodeGenerator that mangles if the class_name
has a class_name in it.

Modify the FunctionCodeGenerator family to handle an extra argument--
the class_name.

Wrap all name ops and attrnames in calls to self.mangle()
2001-08-27 22:56:16 +00:00
Jeremy Hylton 80ea40d858 emit SET_LINENO for augmented assignments 2001-08-27 21:58:09 +00:00
Jeremy Hylton 2afff324ea Many changes -- bug fixes and sundry improvements
Make nested scopes enabled by default

Add is_constant_false() helper so that compiled code and symbols are
consistent with builtin compiler's handling of "if 0:"

Fix doc string handling to be consistent with recent change that
eliminates the doc string from the Module's node attribute.

Add fix to print handling from Evan & Shane.

Track change to visitor api by making "verbose" explicit.

Comment out setting CO_NESTED flag (it's unnecessary in 2.2).
2001-08-27 21:51:52 +00:00
Jeremy Hylton cd8a127e1a Fix for sibling nodes that define the same free variable
Evan Simpson's fix.  And his explanation:

    If you defined two nested functions in a row that refer to the
    same non-global variable, the second one will be generated as
    though the variable were global.
2001-08-27 21:06:35 +00:00
Jeremy Hylton 7e30c9bb5a Add lookup_name() to optimize use of stack frames
The use of com_node() introduces a lot of extra stack frames, enough
to cause a stack overflow compiling test.test_parser with the standard
interpreter recursionlimit.  The com_node() is a convenience function
that hides the dispatch details, but comes at a very high cost.  It is
more efficient to dispatch directly in the callers.  In these cases,
use lookup_node() and call the dispatched node directly.

Also handle yield_stmt in a way that will work with Python 2.1
(suggested by Shane Hathaway)
2001-08-27 21:02:51 +00:00
Jeremy Hylton 058a5adad0 Two changes to visitor API:
Remove _preorder as alias for dispatch and call dispatch directly.
    Add an extra optional argument to walk()

XXX Also comment out some code that does debugging prints.
2001-08-27 20:47:08 +00:00
Jeremy Hylton cccc478577 Why didn't I think of Fred off the top of my head? 2001-08-27 20:44:20 +00:00
Jeremy Hylton 627673396a Add everyone I can think of off the top of my head 2001-08-27 20:43:16 +00:00
Jeremy Hylton 2a8ec79353 Revise regrtest to compile entire standard library.
The tests are run from a copy of the library directory, where
everything has been compiled by the compiler package.

Add a raw_input() call at the end of the script, so that I can check
the output before the temp directory with the compiled code is
removed.
2001-08-27 20:40:43 +00:00
Jeremy Hylton 33c2a627f1 Add -c option to continue if one file has a SyntaxError 2001-08-27 20:39:21 +00:00
Jeremy Hylton 241d69c11b Add a little introductory text.
Change several sections to subsections (part of the manual -> howto
transformation).

Flesh out discussion of assignment nodes (and delete statements).

Add an example of manipulating AST objects at a >>> prompt
2001-08-18 00:24:46 +00:00
Jeremy Hylton ab427b8cce Generate correct reprs for Mul, Add, etc. 2001-08-18 00:14:37 +00:00
Jeremy Hylton ec5bfd13ca Track removal of doc string from Module().nodes[0] 2001-08-18 00:07:46 +00:00
Jeremy Hylton 8548f9b183 Add Yield() node 2001-08-18 00:07:14 +00:00
Jeremy Hylton ec927348c2 Add Yield() statement handler
Fix Module() handler to avoid including the doc string in the AST
2001-08-18 00:04:31 +00:00
Fred Drake 4e6a3fe0f6 Remove spurious space before a period. 2001-08-15 18:48:10 +00:00
Fred Drake 834a85a235 Use the "howto" document class rather than the "manual" class.
Remove the module index; there aren't enough modules documented yet
for this to make sense.

Add a couple more index entries, fixed a few typos, and adjusted a few
more things for consistency.
2001-08-15 17:01:34 +00:00
Fred Drake 27413a5eda Properly mark attribute names in the table. 2001-08-15 16:47:10 +00:00
Fred Drake 3d05161c83 Change the generated markup so that attribute names are properly marked. 2001-08-15 16:46:22 +00:00
Fred Drake 42caf3f6a3 Fix small markup consistency nits. 2001-08-15 14:35:13 +00:00
Jeremy Hylton f5bf77c3f8 update simple explanation of parseFile() 2001-08-14 22:38:03 +00:00
Jeremy Hylton 76f42ac512 First day's progress on documentation 2001-08-14 22:04:44 +00:00
Jeremy Hylton 9272b14d62 Fix typo in astgen script 2001-08-14 21:18:30 +00:00
Jeremy Hylton 7713ac2ff1 Regenerated from new ast.txt and new astgen.py 2001-08-14 18:59:01 +00:00
Jeremy Hylton eef65908a1 Add getChildNodes() method to Node 2001-08-14 18:58:00 +00:00
Jeremy Hylton 34aeeffb62 Add Yield node 2001-08-14 18:57:43 +00:00
Andrew M. Kuchling 392aa1de1d Update e-mail address 2001-08-13 15:10:54 +00:00
Andrew M. Kuchling 564165e5c5 Added README to this directory 2001-08-08 19:10:11 +00:00
Tim Peters 2a7f384122 SF bug 430991: wrong co_lnotab
Armin Rigo pointed out that the way the line-# table got built didn't work
for lines generating more than 255 bytes of bytecode.  Fixed as he
suggested, plus corresponding changes to pyassem.py, plus added some
long overdue docs about this subtle table to compile.c.

Bugfix candidate (line numbers may be off in tracebacks under -O).
2001-06-09 09:26:21 +00:00
Jeremy Hylton bb0bae6da2 Pop loop off the loop stack before handling the loop's else clause.
Otherwise, continue/break will attempt to affect the wrong loop.

A few more fiddles to get the SET_LINENOs consistent across compilers.
2001-04-12 21:54:41 +00:00
Jeremy Hylton 13d70944cb Use new _implicitNameOp() to generate name op code for list comprehensions.
Always emit a SET_LINENO 0 at the beginning of the module.  The
builtin compiler does this, and it's much easier to compare bytecode
generated by the two compilers if they both do.

Move the SET_LINENO inside the FOR_LOOP block for list
comprehensions.  Also for compat. with builtin compiler.
2001-04-12 21:04:43 +00:00
Jeremy Hylton 614e87f286 Add support for visitAssAttr to findOp(). 2001-04-12 20:24:26 +00:00
Jeremy Hylton 542b11acfd pyassem.py:
Fix annoying bugs in flow graph layout code.  In some cases the
    implicit control transfers weren't honored.  In other cases,
    JUMP_FORWARD instructions jumped backwards.

    Remove unused arg from nextBlock().

pycodegen.py

    Add optional force kwarg to set_lineno() that will emit a
    SET_LINENO even if it is the same as the previous lineno.

    Use explicit LOAD_FAST and STORE_FAST to access list comp implicit
    variables.  (The symbol table doesn't know about them.)
2001-04-12 20:21:39 +00:00
Jeremy Hylton bfb0cf822b Revise handling of tuple arguments so that the variables names match
those used by compile.c.  (test_grammar now depends on the names)
2001-04-12 17:33:34 +00:00
Jeremy Hylton 5c9aad6043 Only treat an AugAssign as def if its the target is a Name.
Fixes last bug found with test_scope.py.
2001-04-12 07:06:25 +00:00
Jeremy Hylton 3f76b7e6e4 Fix unpackSequence() to use _nameOp() rather than LOAD_FAST 2001-04-12 06:52:27 +00:00
Jeremy Hylton 660cc77abb Inside a class scope always use LOAD_NAME, STORE_NAME, DEL_NAME 2001-04-12 06:49:00 +00:00
Jeremy Hylton 364f9b9e2f Preliminary support for nested scopes
XXX Still doesn't work right for classes
XXX Still doesn't do sufficient error checking
2001-04-12 06:40:42 +00:00
Jeremy Hylton 53ee2a94c7 Define constants for types of scopes 2001-04-12 06:39:24 +00:00
Jeremy Hylton 84ec1f9159 typo 2001-04-11 16:43:13 +00:00
Jeremy Hylton 01d12937df [finishing fix from earlier checkins]
Call set_lineno() in visitDiscard(), which will generate linenos for
discard statements, e.g. the statement "1/0"

Fixes SF bug #409587
2001-04-11 16:36:25 +00:00
Jeremy Hylton d91bbba89d Add support for extra (*) arguments to preorder.
Change default dispatch to use extended call syntax in place of apply.
2001-04-11 16:26:05 +00:00
Jeremy Hylton 9ab019bee7 Generate docstrings.
Fixes SF buf #217004

Add method fixDocstring() to CodeGenerator.  It converts the Discard
node containing the docstring into an assignment to __doc__.
2001-04-11 16:24:30 +00:00
Jeremy Hylton 4c1f42733c Add lineno attributes to Discard nodes 2001-04-11 16:22:26 +00:00
Jeremy Hylton 5af105eec9 Make sure the docstring is always entered as the first element in the
consts, even if it is None.

Simplify _lookupName() by removing lots of redundant tests.
2001-04-11 16:21:51 +00:00
Jeremy Hylton dbdb28e8be Add globals to list of names returned by get_names().
Fix func arg processing to handle args in tuples.
In test code, skip names beginning with '.'.
2001-04-09 20:11:59 +00:00
Jeremy Hylton f870c952f9 Add two arguments to Scope constructor, module scope and class name
Add mangling support
Add get_children() and add_child() methods to Scope
Skip nodes when If test is a false constant
Add test code that checks results against symtable module
2001-04-09 13:57:32 +00:00
Jeremy Hylton 8b966dcf03 Add preliminary module symbol table constructor 2001-04-09 04:35:35 +00:00
Jeremy Hylton 80e29bd139 Add support for future statements 2001-04-09 04:28:48 +00:00
Jeremy Hylton 42a0830713 Fix "import as" (has always skipping the as name)
Fix com_NEWLINE() so that is accepts arguments, which occurs for lines like:
    stmt; # note trailing semicolon
Add XXX about checking for assignment to list comps
2001-04-09 04:27:12 +00:00
Jeremy Hylton a384f737cc typo 2001-04-09 04:23:55 +00:00
Jeremy Hylton 4ebf3be407 a few small optimizations that seem to give a 5-10% speedup; the
further optimization of com_node makes the most difference.
2000-11-06 16:03:52 +00:00
Jeremy Hylton eefaeb78b3 move pruneNext method to correct object (doh!) 2000-11-06 03:47:39 +00:00
Jeremy Hylton 314e3fb215 Change the graph structure to contain the code generator object for
embedded code objects (e.g. functions) rather than the generated code
object.  This change means that the compiler generates code for
everything at the end, rather then generating code for each function
as it finds it.  Implementation note: _convert_LOAD_CONST in
pyassem.py must be change to call getCode().

Other changes follow.  Several changes creates extra edges between
basic blocks to reflect control flow for loops and exceptions.  These
missing edges had gone unnoticed because they do not affect the
current compilation process.

pyassem.py:
    Add _enable_debug() and _disable_debug() methods that print
    instructions and blocks to stdout as they are generated.

    Add edges between blocks for instructions like SETUP_LOOP,
    FOR_LOOP, etc.

    Add pruneNext to get rid of bogus edges remaining after
    unconditional transfer ops (e.g. JUMP_FORWARD)

    Change repr of Block to omit block length.

pycodegen.py:
    Make sure a new block is started after FOR_LOOP, etc.

    Change assert implementation to use RAISE_VARARGS 1 when there is
    no user-specified failure output.

misc.py:
    Implement __contains__ and copy for Set.
2000-11-06 03:43:11 +00:00
Jeremy Hylton a59ac0a7df If a function contains a doc string, remove the doc string node from
the function's body.

If assert is used without an error message, make the AST node None
rather than Name('None').
2000-11-06 03:33:52 +00:00
Jeremy Hylton 7cff7fe21f Many changes.
Reformatting -- long lines, "[ ]" -> "[]", a few indentation nits.

Replace calls to Node function (which constructed ast nodes) with
calls to actual constructors imported from ast module.

Optimize com_node (most frequently used method) for the common case --
the appropriate method is found in _dispatch.

Fix com_augassign to use class object's rather than node names
(rendered invalid by recent changes to ast)

Remove expensive tests for sequence-ness in com_stmt and
com_append_stmt. These tests should never fail; if they do, something
is really broken and exception will be raised elsewhere.

Fix com_stmt and com_append_stmt to use isinstance rather than
testing's type slot of ast node (this slot disappeared with recent
changes to ast).
2000-10-25 18:10:32 +00:00
Jeremy Hylton 628d289d12 Generated from rev 1.1 of ast.txt 2000-10-25 18:02:59 +00:00
Jeremy Hylton 66d2c1f7e5 Small optimizations in dispatch method: 1) lookup node's __class__ once
and store in local; 2) define _preorder to be dispatch (rather than
method that called dispatch).
2000-10-25 18:02:02 +00:00
Jeremy Hylton 821eee3321 Support for generation of ast.py from simple description of node
structure (ast.txt).  Usage is python astgen.py > ast.py.
2000-10-25 17:59:17 +00:00
Jeremy Hylton ac148b5fcd run the std regression test suite using bytecode produced by the compiler 2000-10-13 22:00:13 +00:00
Jeremy Hylton e7f710cc06 add -d option that dumps entire AST before compiling 2000-10-13 21:59:32 +00:00
Jeremy Hylton 9c048f9f65 Now supports entire Python 2.0 language and still supports Python
1.5.2.  The compiler generates code for the version of the interpreter
it is run under.

ast.py:
    Print and Printnl add dest attr for extended print
    new node AugAssign for augmented assignments
    new nodes ListComp, ListCompFor, and ListCompIf for list
        comprehensions

pyassem.py:
    add work around for string-Unicode comparison raising UnicodeError
        on comparison of two objects in code object's const table

pycodegen.py:
    define VERSION, the Python major version number
    get magic number using imp.get_magic() instead of hard coding
    implement list comprehensions, extended print, and augmented
        assignment; augmented assignment uses Delegator classes (see
        doc string)
    fix import and tuple unpacking for 1.5.2

transformer.py:
    various changes to support new 2.0 grammar and old 1.5 grammar
    add debug_tree helper than converts and symbol and token numbers
    to their names
2000-10-13 21:58:13 +00:00
Jeremy Hylton 4e1be72e6b Fix SF bug #116263: support for from .. import *
transformer.py: return '*', None from com_import_as_name
pycodegen.py: special case for name == '*'
pyassem.py: fix stack counting for IMPORT_ opcodes
2000-10-12 20:23:23 +00:00
Jeremy Hylton b769e80056 read in a .pyc file and disassemble the code objects 2000-10-09 14:35:24 +00:00
Jeremy Hylton 572bdce4b3 change 2-space indent to 4-space indent 2000-09-20 02:47:28 +00:00
Jeremy Hylton 9c36a41694 distutils setup script 2000-09-20 02:35:19 +00:00
Jeremy Hylton 92f3972090 patch by Neil Schemenauer to improve (fix?) line number generation 2000-09-01 20:47:37 +00:00
Jeremy Hylton 2051608616 Update magic number.
Fix import support to work with import as variant of Python 2.0.  The
grammar for import changed, requiring changes in transformer and code
generator, even to handle compilation of imports with as.
2000-09-01 20:33:26 +00:00
Thomas Wouters 46cc7c0f7b Bring Tools/compiler almost up to date. Specifically:
- fix tab space issues (SF patch #101167 by Neil Schemenauer)
- fix co_flags for classes to include CO_NEWLOCALS (SF patch #101145 by Neil)
- fix for merger of UNPACK_LIST and UNPACK_TUPLE into UNPACK_SEQUENCE,
  (SF patch #101168 by, well, Neil :)
- Adjust bytecode MAGIC to current bytecode.

TODO: teach compile.py about list comprehensions.
2000-08-12 20:32:46 +00:00
Jeremy Hylton 7daf04d9e0 replace most calls to emit 'SET_LINENO' will call to method set_lineno
based on bug report by Neil Schemenauer
2000-08-04 16:56:51 +00:00
Jeremy Hylton 8612f1c152 update my email address
fix com_call_function to cope with trailing comma in "f(a, b,)"
2000-08-04 16:54:54 +00:00
Greg Stein fd342bf453 add a bit more legal junk
(too lazy to paste in the whole BSD license tho; included by ref)
2000-08-03 17:39:13 +00:00
Jeremy Hylton ffe968b8fe This script demonstrates use of the visitor interface of the compiler
package.
2000-07-25 16:43:23 +00:00
Jeremy Hylton be317e615e patches from Mark Hammond
Attached is a set of diffs for the .py compiler that adds support
for the new extended call syntax.

compiler/ast.py:
CallFunc node gets 2 new children to support extended call syntax -
"star_args" (for "*args") and "dstar_args" (for "**args")

compiler/pyassem.py
It appear that self.lnotab is supposed to be responsible for
tracking line numbers, but self.firstlineno was still hanging
around.  Removed self.firstlineno completely.  NOTE - I didnt
actually test that the generated code has the correct line numbers!!

Stack depth tracking appeared a little broken - the checks never
made it beyond the "self.patterns" check - thus, the custom methods
were never called!  Fixed this.

(XXX Jeremy notes: I think this code is still broken because it
doesn't track stack effects across block bounaries.)

Added support for the new extended call syntax opcodes for depth
calculations.

compiler/pycodegen.py

Added support for the new extended call syntax opcodes.

compiler/transformer.py

Added support for the new extended call syntax.
2000-05-02 22:32:59 +00:00
Jeremy Hylton 0a4f1ff64e patches from Mark Hammond
compile.py:
On Windows, use 'nul' instead of '/dev/null'.

test.py:
Use double-quotes for the command-line, as Windows doesnt recognise
singles.
2000-05-02 22:29:46 +00:00
Jeremy Hylton 36cc6a2197 complete rewrite
code generator uses flowgraph as intermediate representation.  the old
    rep uses a list with explicit "StackRefs" to indicate the target
    of jumps.

pyassem converts flowgraph to bytecode, breaks up individual steps of
    generating bytecode
2000-03-16 20:06:59 +00:00
Jeremy Hylton f635abee3a simplify visitor walker class
- remove postorder
- remove protocol for automatically walking children based on visitor
  method return value; now only walks if there is no method
2000-03-16 20:04:16 +00:00
Jeremy Hylton b631b8ede5 fix list.append problems 2000-03-16 20:03:04 +00:00
Jeremy Hylton e4d6293383 change name of Set method: items -> elements (avoids confusion with
dict)
2000-03-16 20:02:38 +00:00
Jeremy Hylton 0c3208aa62 compiler command-line interface moved here from compiler.pycodegen 2000-03-06 19:13:21 +00:00
Jeremy Hylton f728f9a13e import compile function form pycodegen 2000-03-06 19:12:33 +00:00
Jeremy Hylton 7fab23e9cb rename compile.py to pycodegen.py
fix imports
remove parse functions and visitor code
track name change: Classdef to Class

add some comments and tweak order of visitXXX methods

get rid of if __name__ == "__main__ section
2000-03-06 19:10:54 +00:00
Jeremy Hylton 8c78341f45 add a doc string
import some useful functions from contained modules
2000-03-06 19:04:14 +00:00
Jeremy Hylton 9812e7bc6a fix import to refer to compiler package 2000-03-06 18:54:30 +00:00
Jeremy Hylton abd7ebf70b revise arguments for addCode method on lnotab. take several numbers
that are internally converted to chars, rather than taking a string.
2000-03-06 18:53:14 +00:00
Jeremy Hylton fa974a9d06 change node Classdef to Class
add doc string to transformer module
add two helper functions:
    parse(buf) -> AST
    parseFile(path) -> AST
2000-03-06 18:50:48 +00:00
Jeremy Hylton ed9586174d factor out the tree walking/visitor code that was in compile.py 2000-03-06 18:49:31 +00:00
Jeremy Hylton 772dd417f7 satisfy the tabnanny (thanks to MH for noticing the problem) 2000-02-21 22:46:00 +00:00
Jeremy Hylton efd0694a2d changes to _lookupName
- removed now (happily) unused second arg
- need to verify results of [].index are correct; for building consts,
  need to have same value and same type, e.g. 2 not the same as 2L
2000-02-17 22:58:54 +00:00
Jeremy Hylton 3ec7e2c4be the previous quick hack to fix def foo((x,y)) failed on some cases
(big surprise).  new solution is a little less hackish.

Code gen adds a TupleArg instance in the argument slot. The tuple arg
includes a copy of the names that it is responsble for binding.  The
PyAssembler uses this information to calculate the correct argcount.

all fix this wacky case: del (a, ((b,), c)), d
which is the same as: del a, b, c, d
(Can't wait for Guido to tell me why.)

solution uses findOp which walks a tree to find out whether it
contains OP_ASSIGN or OP_DELETE or ...
2000-02-17 22:09:35 +00:00
Jeremy Hylton 7708d697ee add varargs and kwargs flags to Lambda nodes 2000-02-17 22:06:20 +00:00
Jeremy Hylton 873bdc18e4 satisfy the tabnanny
fix broken references to filename var in generateXXX methods
2000-02-17 17:56:29 +00:00
Jeremy Hylton ad9a86fb1c support for arglists with implicit tuple unpacks
- added a number of support methods to generate code just before the
  body
- hack protocol for communicating number of args to PyAssembler

fix TryExcept generation for case where exception handler has no body
fix visitAssAttr
add comment about incomplete visitAssName

stop using the ExampleASTVisitor

change script invocation to accept a list of .py files (e.g. Lib/*.py)
2000-02-16 00:55:44 +00:00