Commit Graph

165 Commits

Author SHA1 Message Date
Jeremy Hylton 241d69c11b Add a little introductory text.
Change several sections to subsections (part of the manual -> howto
transformation).

Flesh out discussion of assignment nodes (and delete statements).

Add an example of manipulating AST objects at a >>> prompt
2001-08-18 00:24:46 +00:00
Jeremy Hylton ab427b8cce Generate correct reprs for Mul, Add, etc. 2001-08-18 00:14:37 +00:00
Jeremy Hylton ec5bfd13ca Track removal of doc string from Module().nodes[0] 2001-08-18 00:07:46 +00:00
Jeremy Hylton 8548f9b183 Add Yield() node 2001-08-18 00:07:14 +00:00
Jeremy Hylton ec927348c2 Add Yield() statement handler
Fix Module() handler to avoid including the doc string in the AST
2001-08-18 00:04:31 +00:00
Fred Drake 4e6a3fe0f6 Remove spurious space before a period. 2001-08-15 18:48:10 +00:00
Fred Drake 834a85a235 Use the "howto" document class rather than the "manual" class.
Remove the module index; there aren't enough modules documented yet
for this to make sense.

Add a couple more index entries, fixed a few typos, and adjusted a few
more things for consistency.
2001-08-15 17:01:34 +00:00
Fred Drake 27413a5eda Properly mark attribute names in the table. 2001-08-15 16:47:10 +00:00
Fred Drake 3d05161c83 Change the generated markup so that attribute names are properly marked. 2001-08-15 16:46:22 +00:00
Fred Drake 42caf3f6a3 Fix small markup consistency nits. 2001-08-15 14:35:13 +00:00
Jeremy Hylton f5bf77c3f8 update simple explanation of parseFile() 2001-08-14 22:38:03 +00:00
Jeremy Hylton 76f42ac512 First day's progress on documentation 2001-08-14 22:04:44 +00:00
Jeremy Hylton 9272b14d62 Fix typo in astgen script 2001-08-14 21:18:30 +00:00
Jeremy Hylton 7713ac2ff1 Regenerated from new ast.txt and new astgen.py 2001-08-14 18:59:01 +00:00
Jeremy Hylton eef65908a1 Add getChildNodes() method to Node 2001-08-14 18:58:00 +00:00
Jeremy Hylton 34aeeffb62 Add Yield node 2001-08-14 18:57:43 +00:00
Andrew M. Kuchling 392aa1de1d Update e-mail address 2001-08-13 15:10:54 +00:00
Andrew M. Kuchling 564165e5c5 Added README to this directory 2001-08-08 19:10:11 +00:00
Tim Peters 2a7f384122 SF bug 430991: wrong co_lnotab
Armin Rigo pointed out that the way the line-# table got built didn't work
for lines generating more than 255 bytes of bytecode.  Fixed as he
suggested, plus corresponding changes to pyassem.py, plus added some
long overdue docs about this subtle table to compile.c.

Bugfix candidate (line numbers may be off in tracebacks under -O).
2001-06-09 09:26:21 +00:00
Jeremy Hylton bb0bae6da2 Pop loop off the loop stack before handling the loop's else clause.
Otherwise, continue/break will attempt to affect the wrong loop.

A few more fiddles to get the SET_LINENOs consistent across compilers.
2001-04-12 21:54:41 +00:00
Jeremy Hylton 13d70944cb Use new _implicitNameOp() to generate name op code for list comprehensions.
Always emit a SET_LINENO 0 at the beginning of the module.  The
builtin compiler does this, and it's much easier to compare bytecode
generated by the two compilers if they both do.

Move the SET_LINENO inside the FOR_LOOP block for list
comprehensions.  Also for compat. with builtin compiler.
2001-04-12 21:04:43 +00:00
Jeremy Hylton 614e87f286 Add support for visitAssAttr to findOp(). 2001-04-12 20:24:26 +00:00
Jeremy Hylton 542b11acfd pyassem.py:
Fix annoying bugs in flow graph layout code.  In some cases the
    implicit control transfers weren't honored.  In other cases,
    JUMP_FORWARD instructions jumped backwards.

    Remove unused arg from nextBlock().

pycodegen.py

    Add optional force kwarg to set_lineno() that will emit a
    SET_LINENO even if it is the same as the previous lineno.

    Use explicit LOAD_FAST and STORE_FAST to access list comp implicit
    variables.  (The symbol table doesn't know about them.)
2001-04-12 20:21:39 +00:00
Jeremy Hylton bfb0cf822b Revise handling of tuple arguments so that the variables names match
those used by compile.c.  (test_grammar now depends on the names)
2001-04-12 17:33:34 +00:00
Jeremy Hylton 5c9aad6043 Only treat an AugAssign as def if its the target is a Name.
Fixes last bug found with test_scope.py.
2001-04-12 07:06:25 +00:00
Jeremy Hylton 3f76b7e6e4 Fix unpackSequence() to use _nameOp() rather than LOAD_FAST 2001-04-12 06:52:27 +00:00
Jeremy Hylton 660cc77abb Inside a class scope always use LOAD_NAME, STORE_NAME, DEL_NAME 2001-04-12 06:49:00 +00:00
Jeremy Hylton 364f9b9e2f Preliminary support for nested scopes
XXX Still doesn't work right for classes
XXX Still doesn't do sufficient error checking
2001-04-12 06:40:42 +00:00
Jeremy Hylton 53ee2a94c7 Define constants for types of scopes 2001-04-12 06:39:24 +00:00
Jeremy Hylton 84ec1f9159 typo 2001-04-11 16:43:13 +00:00
Jeremy Hylton 01d12937df [finishing fix from earlier checkins]
Call set_lineno() in visitDiscard(), which will generate linenos for
discard statements, e.g. the statement "1/0"

Fixes SF bug #409587
2001-04-11 16:36:25 +00:00
Jeremy Hylton d91bbba89d Add support for extra (*) arguments to preorder.
Change default dispatch to use extended call syntax in place of apply.
2001-04-11 16:26:05 +00:00
Jeremy Hylton 9ab019bee7 Generate docstrings.
Fixes SF buf #217004

Add method fixDocstring() to CodeGenerator.  It converts the Discard
node containing the docstring into an assignment to __doc__.
2001-04-11 16:24:30 +00:00
Jeremy Hylton 4c1f42733c Add lineno attributes to Discard nodes 2001-04-11 16:22:26 +00:00
Jeremy Hylton 5af105eec9 Make sure the docstring is always entered as the first element in the
consts, even if it is None.

Simplify _lookupName() by removing lots of redundant tests.
2001-04-11 16:21:51 +00:00
Jeremy Hylton dbdb28e8be Add globals to list of names returned by get_names().
Fix func arg processing to handle args in tuples.
In test code, skip names beginning with '.'.
2001-04-09 20:11:59 +00:00
Jeremy Hylton f870c952f9 Add two arguments to Scope constructor, module scope and class name
Add mangling support
Add get_children() and add_child() methods to Scope
Skip nodes when If test is a false constant
Add test code that checks results against symtable module
2001-04-09 13:57:32 +00:00
Jeremy Hylton 8b966dcf03 Add preliminary module symbol table constructor 2001-04-09 04:35:35 +00:00
Jeremy Hylton 80e29bd139 Add support for future statements 2001-04-09 04:28:48 +00:00
Jeremy Hylton 42a0830713 Fix "import as" (has always skipping the as name)
Fix com_NEWLINE() so that is accepts arguments, which occurs for lines like:
    stmt; # note trailing semicolon
Add XXX about checking for assignment to list comps
2001-04-09 04:27:12 +00:00
Jeremy Hylton a384f737cc typo 2001-04-09 04:23:55 +00:00
Jeremy Hylton 4ebf3be407 a few small optimizations that seem to give a 5-10% speedup; the
further optimization of com_node makes the most difference.
2000-11-06 16:03:52 +00:00
Jeremy Hylton eefaeb78b3 move pruneNext method to correct object (doh!) 2000-11-06 03:47:39 +00:00
Jeremy Hylton 314e3fb215 Change the graph structure to contain the code generator object for
embedded code objects (e.g. functions) rather than the generated code
object.  This change means that the compiler generates code for
everything at the end, rather then generating code for each function
as it finds it.  Implementation note: _convert_LOAD_CONST in
pyassem.py must be change to call getCode().

Other changes follow.  Several changes creates extra edges between
basic blocks to reflect control flow for loops and exceptions.  These
missing edges had gone unnoticed because they do not affect the
current compilation process.

pyassem.py:
    Add _enable_debug() and _disable_debug() methods that print
    instructions and blocks to stdout as they are generated.

    Add edges between blocks for instructions like SETUP_LOOP,
    FOR_LOOP, etc.

    Add pruneNext to get rid of bogus edges remaining after
    unconditional transfer ops (e.g. JUMP_FORWARD)

    Change repr of Block to omit block length.

pycodegen.py:
    Make sure a new block is started after FOR_LOOP, etc.

    Change assert implementation to use RAISE_VARARGS 1 when there is
    no user-specified failure output.

misc.py:
    Implement __contains__ and copy for Set.
2000-11-06 03:43:11 +00:00
Jeremy Hylton a59ac0a7df If a function contains a doc string, remove the doc string node from
the function's body.

If assert is used without an error message, make the AST node None
rather than Name('None').
2000-11-06 03:33:52 +00:00
Jeremy Hylton 7cff7fe21f Many changes.
Reformatting -- long lines, "[ ]" -> "[]", a few indentation nits.

Replace calls to Node function (which constructed ast nodes) with
calls to actual constructors imported from ast module.

Optimize com_node (most frequently used method) for the common case --
the appropriate method is found in _dispatch.

Fix com_augassign to use class object's rather than node names
(rendered invalid by recent changes to ast)

Remove expensive tests for sequence-ness in com_stmt and
com_append_stmt. These tests should never fail; if they do, something
is really broken and exception will be raised elsewhere.

Fix com_stmt and com_append_stmt to use isinstance rather than
testing's type slot of ast node (this slot disappeared with recent
changes to ast).
2000-10-25 18:10:32 +00:00
Jeremy Hylton 628d289d12 Generated from rev 1.1 of ast.txt 2000-10-25 18:02:59 +00:00
Jeremy Hylton 66d2c1f7e5 Small optimizations in dispatch method: 1) lookup node's __class__ once
and store in local; 2) define _preorder to be dispatch (rather than
method that called dispatch).
2000-10-25 18:02:02 +00:00
Jeremy Hylton 821eee3321 Support for generation of ast.py from simple description of node
structure (ast.txt).  Usage is python astgen.py > ast.py.
2000-10-25 17:59:17 +00:00
Jeremy Hylton ac148b5fcd run the std regression test suite using bytecode produced by the compiler 2000-10-13 22:00:13 +00:00
Jeremy Hylton e7f710cc06 add -d option that dumps entire AST before compiling 2000-10-13 21:59:32 +00:00
Jeremy Hylton 9c048f9f65 Now supports entire Python 2.0 language and still supports Python
1.5.2.  The compiler generates code for the version of the interpreter
it is run under.

ast.py:
    Print and Printnl add dest attr for extended print
    new node AugAssign for augmented assignments
    new nodes ListComp, ListCompFor, and ListCompIf for list
        comprehensions

pyassem.py:
    add work around for string-Unicode comparison raising UnicodeError
        on comparison of two objects in code object's const table

pycodegen.py:
    define VERSION, the Python major version number
    get magic number using imp.get_magic() instead of hard coding
    implement list comprehensions, extended print, and augmented
        assignment; augmented assignment uses Delegator classes (see
        doc string)
    fix import and tuple unpacking for 1.5.2

transformer.py:
    various changes to support new 2.0 grammar and old 1.5 grammar
    add debug_tree helper than converts and symbol and token numbers
    to their names
2000-10-13 21:58:13 +00:00
Jeremy Hylton 4e1be72e6b Fix SF bug #116263: support for from .. import *
transformer.py: return '*', None from com_import_as_name
pycodegen.py: special case for name == '*'
pyassem.py: fix stack counting for IMPORT_ opcodes
2000-10-12 20:23:23 +00:00
Jeremy Hylton b769e80056 read in a .pyc file and disassemble the code objects 2000-10-09 14:35:24 +00:00
Jeremy Hylton 572bdce4b3 change 2-space indent to 4-space indent 2000-09-20 02:47:28 +00:00
Jeremy Hylton 9c36a41694 distutils setup script 2000-09-20 02:35:19 +00:00
Jeremy Hylton 92f3972090 patch by Neil Schemenauer to improve (fix?) line number generation 2000-09-01 20:47:37 +00:00
Jeremy Hylton 2051608616 Update magic number.
Fix import support to work with import as variant of Python 2.0.  The
grammar for import changed, requiring changes in transformer and code
generator, even to handle compilation of imports with as.
2000-09-01 20:33:26 +00:00
Thomas Wouters 46cc7c0f7b Bring Tools/compiler almost up to date. Specifically:
- fix tab space issues (SF patch #101167 by Neil Schemenauer)
- fix co_flags for classes to include CO_NEWLOCALS (SF patch #101145 by Neil)
- fix for merger of UNPACK_LIST and UNPACK_TUPLE into UNPACK_SEQUENCE,
  (SF patch #101168 by, well, Neil :)
- Adjust bytecode MAGIC to current bytecode.

TODO: teach compile.py about list comprehensions.
2000-08-12 20:32:46 +00:00
Jeremy Hylton 7daf04d9e0 replace most calls to emit 'SET_LINENO' will call to method set_lineno
based on bug report by Neil Schemenauer
2000-08-04 16:56:51 +00:00
Jeremy Hylton 8612f1c152 update my email address
fix com_call_function to cope with trailing comma in "f(a, b,)"
2000-08-04 16:54:54 +00:00
Greg Stein fd342bf453 add a bit more legal junk
(too lazy to paste in the whole BSD license tho; included by ref)
2000-08-03 17:39:13 +00:00
Jeremy Hylton ffe968b8fe This script demonstrates use of the visitor interface of the compiler
package.
2000-07-25 16:43:23 +00:00
Jeremy Hylton be317e615e patches from Mark Hammond
Attached is a set of diffs for the .py compiler that adds support
for the new extended call syntax.

compiler/ast.py:
CallFunc node gets 2 new children to support extended call syntax -
"star_args" (for "*args") and "dstar_args" (for "**args")

compiler/pyassem.py
It appear that self.lnotab is supposed to be responsible for
tracking line numbers, but self.firstlineno was still hanging
around.  Removed self.firstlineno completely.  NOTE - I didnt
actually test that the generated code has the correct line numbers!!

Stack depth tracking appeared a little broken - the checks never
made it beyond the "self.patterns" check - thus, the custom methods
were never called!  Fixed this.

(XXX Jeremy notes: I think this code is still broken because it
doesn't track stack effects across block bounaries.)

Added support for the new extended call syntax opcodes for depth
calculations.

compiler/pycodegen.py

Added support for the new extended call syntax opcodes.

compiler/transformer.py

Added support for the new extended call syntax.
2000-05-02 22:32:59 +00:00
Jeremy Hylton 0a4f1ff64e patches from Mark Hammond
compile.py:
On Windows, use 'nul' instead of '/dev/null'.

test.py:
Use double-quotes for the command-line, as Windows doesnt recognise
singles.
2000-05-02 22:29:46 +00:00
Jeremy Hylton 36cc6a2197 complete rewrite
code generator uses flowgraph as intermediate representation.  the old
    rep uses a list with explicit "StackRefs" to indicate the target
    of jumps.

pyassem converts flowgraph to bytecode, breaks up individual steps of
    generating bytecode
2000-03-16 20:06:59 +00:00
Jeremy Hylton f635abee3a simplify visitor walker class
- remove postorder
- remove protocol for automatically walking children based on visitor
  method return value; now only walks if there is no method
2000-03-16 20:04:16 +00:00
Jeremy Hylton b631b8ede5 fix list.append problems 2000-03-16 20:03:04 +00:00
Jeremy Hylton e4d6293383 change name of Set method: items -> elements (avoids confusion with
dict)
2000-03-16 20:02:38 +00:00
Jeremy Hylton 0c3208aa62 compiler command-line interface moved here from compiler.pycodegen 2000-03-06 19:13:21 +00:00
Jeremy Hylton f728f9a13e import compile function form pycodegen 2000-03-06 19:12:33 +00:00
Jeremy Hylton 7fab23e9cb rename compile.py to pycodegen.py
fix imports
remove parse functions and visitor code
track name change: Classdef to Class

add some comments and tweak order of visitXXX methods

get rid of if __name__ == "__main__ section
2000-03-06 19:10:54 +00:00
Jeremy Hylton 8c78341f45 add a doc string
import some useful functions from contained modules
2000-03-06 19:04:14 +00:00
Jeremy Hylton 9812e7bc6a fix import to refer to compiler package 2000-03-06 18:54:30 +00:00
Jeremy Hylton abd7ebf70b revise arguments for addCode method on lnotab. take several numbers
that are internally converted to chars, rather than taking a string.
2000-03-06 18:53:14 +00:00
Jeremy Hylton fa974a9d06 change node Classdef to Class
add doc string to transformer module
add two helper functions:
    parse(buf) -> AST
    parseFile(path) -> AST
2000-03-06 18:50:48 +00:00
Jeremy Hylton ed9586174d factor out the tree walking/visitor code that was in compile.py 2000-03-06 18:49:31 +00:00
Jeremy Hylton 772dd417f7 satisfy the tabnanny (thanks to MH for noticing the problem) 2000-02-21 22:46:00 +00:00
Jeremy Hylton efd0694a2d changes to _lookupName
- removed now (happily) unused second arg
- need to verify results of [].index are correct; for building consts,
  need to have same value and same type, e.g. 2 not the same as 2L
2000-02-17 22:58:54 +00:00
Jeremy Hylton 3ec7e2c4be the previous quick hack to fix def foo((x,y)) failed on some cases
(big surprise).  new solution is a little less hackish.

Code gen adds a TupleArg instance in the argument slot. The tuple arg
includes a copy of the names that it is responsble for binding.  The
PyAssembler uses this information to calculate the correct argcount.

all fix this wacky case: del (a, ((b,), c)), d
which is the same as: del a, b, c, d
(Can't wait for Guido to tell me why.)

solution uses findOp which walks a tree to find out whether it
contains OP_ASSIGN or OP_DELETE or ...
2000-02-17 22:09:35 +00:00
Jeremy Hylton 7708d697ee add varargs and kwargs flags to Lambda nodes 2000-02-17 22:06:20 +00:00
Jeremy Hylton 873bdc18e4 satisfy the tabnanny
fix broken references to filename var in generateXXX methods
2000-02-17 17:56:29 +00:00
Jeremy Hylton ad9a86fb1c support for arglists with implicit tuple unpacks
- added a number of support methods to generate code just before the
  body
- hack protocol for communicating number of args to PyAssembler

fix TryExcept generation for case where exception handler has no body
fix visitAssAttr
add comment about incomplete visitAssName

stop using the ExampleASTVisitor

change script invocation to accept a list of .py files (e.g. Lib/*.py)
2000-02-16 00:55:44 +00:00
Jeremy Hylton 3d9f5e4de2 more robust assignment of lineno for keyword args
get the lineno from the name of the keyword arg

example of case that didn't work--
def foo(x, y, a = None,
	b = None):
2000-02-16 00:51:37 +00:00
Jeremy Hylton 2ce27b223b fix argcount generation for arg lists containing tuple unpacks
this is sort of a hack
2000-02-16 00:50:29 +00:00
Jeremy Hylton 65d4ea05d2 add flatten helper function 2000-02-16 00:49:47 +00:00
Jeremy Hylton 4f6bcd80fc finish first impl of code generator
add support for nodes TryExcept, TryFinally, Sliceobj
fix visitSubscript to properly handle x[a,b,c]
2000-02-15 23:45:26 +00:00
Jeremy Hylton 1ebba96871 fix creation of Ellipsis node 2000-02-15 23:43:19 +00:00
Jeremy Hylton 410e840c85 add a little debugging support when new.code raises SystemError 2000-02-15 21:59:50 +00:00
Jeremy Hylton d603dee4a3 add line numbers to nodes in the except clause (when possible) 2000-02-15 21:30:48 +00:00
Jeremy Hylton ca1ad182f2 tidy up tryexcept and tryfinally nodes 2000-02-15 21:30:00 +00:00
Jeremy Hylton 9e1cd688b5 edit a doc string
(real intent is to test out rsync install)
2000-02-14 23:57:56 +00:00
Jeremy Hylton a46fb3841b get rid of spurious print 2000-02-14 21:54:57 +00:00
Jeremy Hylton 126960b744 looks like everything is working except for try/except (pystone
compiles correctly)
2000-02-14 21:33:10 +00:00
Jeremy Hylton 17988d2a17 LeftShift & RightShift: fix reprs, change attr names to left and right
(so they are common with other binary ops)
2000-02-14 21:32:42 +00:00
Jeremy Hylton bf6267e6f8 rename several of the generic attribute names for nodes. new node attrs are:
Exec: expr, locals, globals
Dict: items
Assert: test, fail
2000-02-14 18:34:11 +00:00
Jeremy Hylton 42907790b6 (), [], and {} should not be represented as constant expressions, they
should be calls to BUILD_ ops for these types with no arguments
2000-02-14 18:32:46 +00:00
Jeremy Hylton a50581228e split compile.py into two files
add StackDepthFinder (and remove push/pop from CodeGen)
add several nodes, including Ellipsis, Bit&|^, Exec
2000-02-14 14:14:29 +00:00
Jeremy Hylton 3050d51571 change MODULE_NAMESPACE/FUNCTION_NAMESPACE stuff to have a single flag
named OPTIMIZED, which matches compile.c and makes more sense for
classes

revamp call signature for PythonVMCode.__init__; doesn't really matter
'cuz this code is going to be refactored out of existence

add generateClassCode and modify Func & Lambda generation

add support for nodes Classdef, Keyword,

fix CallFunc to generate right op arg when calling w/ keywords

add ugly hack to properly compute offsets when the same stack ref is
used multiple times
2000-02-12 00:12:38 +00:00
Jeremy Hylton 76d01b820c add support for Lambda nodes
change resolution of local name ops (LOAD_FAST).  i think it makes
sense now.  if it is an argument or a local var name that it used, it
must be in varnames.  if it is a local var name that is used, it must
also be in names
2000-02-11 20:27:07 +00:00