Commit Graph

226 Commits

Author SHA1 Message Date
Jeremy Hylton d91bbba89d Add support for extra (*) arguments to preorder.
Change default dispatch to use extended call syntax in place of apply.
2001-04-11 16:26:05 +00:00
Jeremy Hylton 9ab019bee7 Generate docstrings.
Fixes SF buf #217004

Add method fixDocstring() to CodeGenerator.  It converts the Discard
node containing the docstring into an assignment to __doc__.
2001-04-11 16:24:30 +00:00
Jeremy Hylton 4c1f42733c Add lineno attributes to Discard nodes 2001-04-11 16:22:26 +00:00
Jeremy Hylton 5af105eec9 Make sure the docstring is always entered as the first element in the
consts, even if it is None.

Simplify _lookupName() by removing lots of redundant tests.
2001-04-11 16:21:51 +00:00
Jeremy Hylton dbdb28e8be Add globals to list of names returned by get_names().
Fix func arg processing to handle args in tuples.
In test code, skip names beginning with '.'.
2001-04-09 20:11:59 +00:00
Jeremy Hylton f870c952f9 Add two arguments to Scope constructor, module scope and class name
Add mangling support
Add get_children() and add_child() methods to Scope
Skip nodes when If test is a false constant
Add test code that checks results against symtable module
2001-04-09 13:57:32 +00:00
Jeremy Hylton 8b966dcf03 Add preliminary module symbol table constructor 2001-04-09 04:35:35 +00:00
Jeremy Hylton 80e29bd139 Add support for future statements 2001-04-09 04:28:48 +00:00
Jeremy Hylton 42a0830713 Fix "import as" (has always skipping the as name)
Fix com_NEWLINE() so that is accepts arguments, which occurs for lines like:
    stmt; # note trailing semicolon
Add XXX about checking for assignment to list comps
2001-04-09 04:27:12 +00:00
Jeremy Hylton a384f737cc typo 2001-04-09 04:23:55 +00:00
Jeremy Hylton 4ebf3be407 a few small optimizations that seem to give a 5-10% speedup; the
further optimization of com_node makes the most difference.
2000-11-06 16:03:52 +00:00
Jeremy Hylton eefaeb78b3 move pruneNext method to correct object (doh!) 2000-11-06 03:47:39 +00:00
Jeremy Hylton 314e3fb215 Change the graph structure to contain the code generator object for
embedded code objects (e.g. functions) rather than the generated code
object.  This change means that the compiler generates code for
everything at the end, rather then generating code for each function
as it finds it.  Implementation note: _convert_LOAD_CONST in
pyassem.py must be change to call getCode().

Other changes follow.  Several changes creates extra edges between
basic blocks to reflect control flow for loops and exceptions.  These
missing edges had gone unnoticed because they do not affect the
current compilation process.

pyassem.py:
    Add _enable_debug() and _disable_debug() methods that print
    instructions and blocks to stdout as they are generated.

    Add edges between blocks for instructions like SETUP_LOOP,
    FOR_LOOP, etc.

    Add pruneNext to get rid of bogus edges remaining after
    unconditional transfer ops (e.g. JUMP_FORWARD)

    Change repr of Block to omit block length.

pycodegen.py:
    Make sure a new block is started after FOR_LOOP, etc.

    Change assert implementation to use RAISE_VARARGS 1 when there is
    no user-specified failure output.

misc.py:
    Implement __contains__ and copy for Set.
2000-11-06 03:43:11 +00:00
Jeremy Hylton a59ac0a7df If a function contains a doc string, remove the doc string node from
the function's body.

If assert is used without an error message, make the AST node None
rather than Name('None').
2000-11-06 03:33:52 +00:00
Jeremy Hylton 7cff7fe21f Many changes.
Reformatting -- long lines, "[ ]" -> "[]", a few indentation nits.

Replace calls to Node function (which constructed ast nodes) with
calls to actual constructors imported from ast module.

Optimize com_node (most frequently used method) for the common case --
the appropriate method is found in _dispatch.

Fix com_augassign to use class object's rather than node names
(rendered invalid by recent changes to ast)

Remove expensive tests for sequence-ness in com_stmt and
com_append_stmt. These tests should never fail; if they do, something
is really broken and exception will be raised elsewhere.

Fix com_stmt and com_append_stmt to use isinstance rather than
testing's type slot of ast node (this slot disappeared with recent
changes to ast).
2000-10-25 18:10:32 +00:00
Jeremy Hylton 628d289d12 Generated from rev 1.1 of ast.txt 2000-10-25 18:02:59 +00:00
Jeremy Hylton 66d2c1f7e5 Small optimizations in dispatch method: 1) lookup node's __class__ once
and store in local; 2) define _preorder to be dispatch (rather than
method that called dispatch).
2000-10-25 18:02:02 +00:00
Jeremy Hylton 9c048f9f65 Now supports entire Python 2.0 language and still supports Python
1.5.2.  The compiler generates code for the version of the interpreter
it is run under.

ast.py:
    Print and Printnl add dest attr for extended print
    new node AugAssign for augmented assignments
    new nodes ListComp, ListCompFor, and ListCompIf for list
        comprehensions

pyassem.py:
    add work around for string-Unicode comparison raising UnicodeError
        on comparison of two objects in code object's const table

pycodegen.py:
    define VERSION, the Python major version number
    get magic number using imp.get_magic() instead of hard coding
    implement list comprehensions, extended print, and augmented
        assignment; augmented assignment uses Delegator classes (see
        doc string)
    fix import and tuple unpacking for 1.5.2

transformer.py:
    various changes to support new 2.0 grammar and old 1.5 grammar
    add debug_tree helper than converts and symbol and token numbers
    to their names
2000-10-13 21:58:13 +00:00
Jeremy Hylton 4e1be72e6b Fix SF bug #116263: support for from .. import *
transformer.py: return '*', None from com_import_as_name
pycodegen.py: special case for name == '*'
pyassem.py: fix stack counting for IMPORT_ opcodes
2000-10-12 20:23:23 +00:00
Jeremy Hylton 572bdce4b3 change 2-space indent to 4-space indent 2000-09-20 02:47:28 +00:00
Jeremy Hylton 92f3972090 patch by Neil Schemenauer to improve (fix?) line number generation 2000-09-01 20:47:37 +00:00
Jeremy Hylton 2051608616 Update magic number.
Fix import support to work with import as variant of Python 2.0.  The
grammar for import changed, requiring changes in transformer and code
generator, even to handle compilation of imports with as.
2000-09-01 20:33:26 +00:00
Thomas Wouters 46cc7c0f7b Bring Tools/compiler almost up to date. Specifically:
- fix tab space issues (SF patch #101167 by Neil Schemenauer)
- fix co_flags for classes to include CO_NEWLOCALS (SF patch #101145 by Neil)
- fix for merger of UNPACK_LIST and UNPACK_TUPLE into UNPACK_SEQUENCE,
  (SF patch #101168 by, well, Neil :)
- Adjust bytecode MAGIC to current bytecode.

TODO: teach compile.py about list comprehensions.
2000-08-12 20:32:46 +00:00
Jeremy Hylton 7daf04d9e0 replace most calls to emit 'SET_LINENO' will call to method set_lineno
based on bug report by Neil Schemenauer
2000-08-04 16:56:51 +00:00
Jeremy Hylton 8612f1c152 update my email address
fix com_call_function to cope with trailing comma in "f(a, b,)"
2000-08-04 16:54:54 +00:00
Greg Stein fd342bf453 add a bit more legal junk
(too lazy to paste in the whole BSD license tho; included by ref)
2000-08-03 17:39:13 +00:00
Jeremy Hylton be317e615e patches from Mark Hammond
Attached is a set of diffs for the .py compiler that adds support
for the new extended call syntax.

compiler/ast.py:
CallFunc node gets 2 new children to support extended call syntax -
"star_args" (for "*args") and "dstar_args" (for "**args")

compiler/pyassem.py
It appear that self.lnotab is supposed to be responsible for
tracking line numbers, but self.firstlineno was still hanging
around.  Removed self.firstlineno completely.  NOTE - I didnt
actually test that the generated code has the correct line numbers!!

Stack depth tracking appeared a little broken - the checks never
made it beyond the "self.patterns" check - thus, the custom methods
were never called!  Fixed this.

(XXX Jeremy notes: I think this code is still broken because it
doesn't track stack effects across block bounaries.)

Added support for the new extended call syntax opcodes for depth
calculations.

compiler/pycodegen.py

Added support for the new extended call syntax opcodes.

compiler/transformer.py

Added support for the new extended call syntax.
2000-05-02 22:32:59 +00:00
Jeremy Hylton 36cc6a2197 complete rewrite
code generator uses flowgraph as intermediate representation.  the old
    rep uses a list with explicit "StackRefs" to indicate the target
    of jumps.

pyassem converts flowgraph to bytecode, breaks up individual steps of
    generating bytecode
2000-03-16 20:06:59 +00:00
Jeremy Hylton f635abee3a simplify visitor walker class
- remove postorder
- remove protocol for automatically walking children based on visitor
  method return value; now only walks if there is no method
2000-03-16 20:04:16 +00:00
Jeremy Hylton b631b8ede5 fix list.append problems 2000-03-16 20:03:04 +00:00
Jeremy Hylton e4d6293383 change name of Set method: items -> elements (avoids confusion with
dict)
2000-03-16 20:02:38 +00:00
Jeremy Hylton f728f9a13e import compile function form pycodegen 2000-03-06 19:12:33 +00:00
Jeremy Hylton 7fab23e9cb rename compile.py to pycodegen.py
fix imports
remove parse functions and visitor code
track name change: Classdef to Class

add some comments and tweak order of visitXXX methods

get rid of if __name__ == "__main__ section
2000-03-06 19:10:54 +00:00
Jeremy Hylton 8c78341f45 add a doc string
import some useful functions from contained modules
2000-03-06 19:04:14 +00:00
Jeremy Hylton 9812e7bc6a fix import to refer to compiler package 2000-03-06 18:54:30 +00:00
Jeremy Hylton abd7ebf70b revise arguments for addCode method on lnotab. take several numbers
that are internally converted to chars, rather than taking a string.
2000-03-06 18:53:14 +00:00
Jeremy Hylton fa974a9d06 change node Classdef to Class
add doc string to transformer module
add two helper functions:
    parse(buf) -> AST
    parseFile(path) -> AST
2000-03-06 18:50:48 +00:00
Jeremy Hylton ed9586174d factor out the tree walking/visitor code that was in compile.py 2000-03-06 18:49:31 +00:00
Jeremy Hylton 772dd417f7 satisfy the tabnanny (thanks to MH for noticing the problem) 2000-02-21 22:46:00 +00:00
Jeremy Hylton efd0694a2d changes to _lookupName
- removed now (happily) unused second arg
- need to verify results of [].index are correct; for building consts,
  need to have same value and same type, e.g. 2 not the same as 2L
2000-02-17 22:58:54 +00:00
Jeremy Hylton 3ec7e2c4be the previous quick hack to fix def foo((x,y)) failed on some cases
(big surprise).  new solution is a little less hackish.

Code gen adds a TupleArg instance in the argument slot. The tuple arg
includes a copy of the names that it is responsble for binding.  The
PyAssembler uses this information to calculate the correct argcount.

all fix this wacky case: del (a, ((b,), c)), d
which is the same as: del a, b, c, d
(Can't wait for Guido to tell me why.)

solution uses findOp which walks a tree to find out whether it
contains OP_ASSIGN or OP_DELETE or ...
2000-02-17 22:09:35 +00:00
Jeremy Hylton 7708d697ee add varargs and kwargs flags to Lambda nodes 2000-02-17 22:06:20 +00:00
Jeremy Hylton 873bdc18e4 satisfy the tabnanny
fix broken references to filename var in generateXXX methods
2000-02-17 17:56:29 +00:00
Jeremy Hylton ad9a86fb1c support for arglists with implicit tuple unpacks
- added a number of support methods to generate code just before the
  body
- hack protocol for communicating number of args to PyAssembler

fix TryExcept generation for case where exception handler has no body
fix visitAssAttr
add comment about incomplete visitAssName

stop using the ExampleASTVisitor

change script invocation to accept a list of .py files (e.g. Lib/*.py)
2000-02-16 00:55:44 +00:00
Jeremy Hylton 3d9f5e4de2 more robust assignment of lineno for keyword args
get the lineno from the name of the keyword arg

example of case that didn't work--
def foo(x, y, a = None,
	b = None):
2000-02-16 00:51:37 +00:00
Jeremy Hylton 2ce27b223b fix argcount generation for arg lists containing tuple unpacks
this is sort of a hack
2000-02-16 00:50:29 +00:00
Jeremy Hylton 65d4ea05d2 add flatten helper function 2000-02-16 00:49:47 +00:00
Jeremy Hylton 4f6bcd80fc finish first impl of code generator
add support for nodes TryExcept, TryFinally, Sliceobj
fix visitSubscript to properly handle x[a,b,c]
2000-02-15 23:45:26 +00:00
Jeremy Hylton 1ebba96871 fix creation of Ellipsis node 2000-02-15 23:43:19 +00:00
Jeremy Hylton 410e840c85 add a little debugging support when new.code raises SystemError 2000-02-15 21:59:50 +00:00
Jeremy Hylton d603dee4a3 add line numbers to nodes in the except clause (when possible) 2000-02-15 21:30:48 +00:00
Jeremy Hylton ca1ad182f2 tidy up tryexcept and tryfinally nodes 2000-02-15 21:30:00 +00:00
Jeremy Hylton 9e1cd688b5 edit a doc string
(real intent is to test out rsync install)
2000-02-14 23:57:56 +00:00
Jeremy Hylton a46fb3841b get rid of spurious print 2000-02-14 21:54:57 +00:00
Jeremy Hylton 126960b744 looks like everything is working except for try/except (pystone
compiles correctly)
2000-02-14 21:33:10 +00:00
Jeremy Hylton 17988d2a17 LeftShift & RightShift: fix reprs, change attr names to left and right
(so they are common with other binary ops)
2000-02-14 21:32:42 +00:00
Jeremy Hylton bf6267e6f8 rename several of the generic attribute names for nodes. new node attrs are:
Exec: expr, locals, globals
Dict: items
Assert: test, fail
2000-02-14 18:34:11 +00:00
Jeremy Hylton 42907790b6 (), [], and {} should not be represented as constant expressions, they
should be calls to BUILD_ ops for these types with no arguments
2000-02-14 18:32:46 +00:00
Jeremy Hylton a50581228e split compile.py into two files
add StackDepthFinder (and remove push/pop from CodeGen)
add several nodes, including Ellipsis, Bit&|^, Exec
2000-02-14 14:14:29 +00:00
Jeremy Hylton 3050d51571 change MODULE_NAMESPACE/FUNCTION_NAMESPACE stuff to have a single flag
named OPTIMIZED, which matches compile.c and makes more sense for
classes

revamp call signature for PythonVMCode.__init__; doesn't really matter
'cuz this code is going to be refactored out of existence

add generateClassCode and modify Func & Lambda generation

add support for nodes Classdef, Keyword,

fix CallFunc to generate right op arg when calling w/ keywords

add ugly hack to properly compute offsets when the same stack ref is
used multiple times
2000-02-12 00:12:38 +00:00
Jeremy Hylton 76d01b820c add support for Lambda nodes
change resolution of local name ops (LOAD_FAST).  i think it makes
sense now.  if it is an argument or a local var name that it used, it
must be in varnames.  if it is a local var name that is used, it must
also be in names
2000-02-11 20:27:07 +00:00
Jeremy Hylton 4eb504c2af add loop handling via
Loop object to handle StackRegs
loops stack attr on CodeGenreeator to hold the current loop object

add support for nodes While, Break, Continue
2000-02-10 20:55:50 +00:00
Jeremy Hylton 95b8614353 add an __len__ to Set and Stack 2000-02-10 20:54:27 +00:00
Jeremy Hylton 3e0910c10c add namespace attr to CodeGenerator, can be either MODULE_NAMESPACE or
FUNCTION_NAMESPACE.  initialize in __init__ and reset in
generateFunctionCode.

replace direct issue of STORE_FAST, STORE_GLOBAL, etc. with call to
storeName; same for loadName and deleteName

the new {store,load,delete}Name methods use the namespace attr and the
local variable stack to determine the correct bytecode to issue
2000-02-10 17:20:39 +00:00
Jeremy Hylton 5e0ce53e0e add ExampleASTVisitor:
* prints out examples of nodes that are handled by visitor.  simply a
  development convenience

remove NestedCodeGenerator -- it was bogus after all
replace with generateFunctionCode, a method to call to generate code
  for a function instead of a top-level module

fix impl of visitDiscard (most pop stack)
emit lineno for pass

handle the following new node types: Import, From, Getattr, Subscript,
Slice, AssAttr, AssTuple, Mod, Not, And, Or, List

LocalNameFinder: remove names declared as globals for locals

PythonVMCode: pass arg names to constructor, force varnames to contain
them all (even if they aren't referenced)

add -q option on command line to disable stdout
2000-02-10 00:47:08 +00:00
Jeremy Hylton 69926eaee0 add remove method to set 2000-02-10 00:43:22 +00:00
Jeremy Hylton 402456020b add optional verbose arg to walk function. it overrides the global
VERBOSE setting for the ASTVisitor

add getopt handling for one or more -v args

rename ForwardRef to StackRef, because it isn't necessarily directional

CodeGenerator:
* add assertStackEmpty method.  prints warning if stack is not empty
  when it should be
* define methods for AssName, UNARY_*, For

PythonVMCode:
* fix mix up between hasjrel and hasjabs for address calculation
2000-02-08 21:15:48 +00:00
Jeremy Hylton 156a975476 make all unary operators have a single child node called expr 2000-02-08 21:06:19 +00:00
Jeremy Hylton 1851901a05 add spaces to comparison names is not and not in to match dis 2000-02-08 19:58:33 +00:00
Jeremy Hylton 53187f32eb now produces valid pyc files for a least a trivial subset of the
language.

CodeGenerator:
* modify to track stack depth
* add emit method that call's PythonVMCode's makeCodeObject
* thread filenames through in hackish way
* set flags for code objects for modules and functions
  XXX the docs for the flags seem out of date and/or incomplete

PythonVMCode:
* add doc string describing the elements of a real code object

LineAddrTable:
* creates an lnotab (no quite correctly though)
2000-02-08 19:01:29 +00:00
Jeremy Hylton aa9d2d6123 constants from transformer 2000-02-08 18:57:51 +00:00
Jeremy Hylton 9605c11c4c move constants out of transformer so that they can be shared with ast
add varargs and kwargs attributes to Function nodes
2000-02-08 18:57:32 +00:00
Jeremy Hylton 0fdffcf916 rapid evolution towards producing real .pyc files (even though I don't
handle most of the language syntax yet)

create NestedCodeGenerator used to generator the separate code object
that needs to be passed as an argument to MAKE_FUNCTION when a def
stmt is found (probably useful for class too)

change CodeGenerator.visitFunction to use the NestedCG

add CompiledModule class to handle creation of .pyc (pretty minimal
for now)

add makeCodeObject method to PythonVMCode that replaces symbolic names
with indexes into slots of the code code.  the design of this
class will probably need to be revised.
2000-02-04 19:37:35 +00:00
Jeremy Hylton 8b6323d3ef checking in initial weekend's work
compile.py: ASTVisitor framework plus bits of a code generator that
    should be bug-for-buf compatible with compile.c

misc.py: Set and Stack helpers

test.py: a bit of simple sample code that compile.py will work on
2000-02-04 00:28:21 +00:00
Jeremy Hylton 106a02da94 make p2c a package 2000-02-04 00:25:34 +00:00
Jeremy Hylton f968e8545d three files from the p2c cvs tree. the message here indicates the
revision number the p2c cvs tree.

COPYRIGHT: 1.1
ast.py: 1.3
transformer.py: 1.11
2000-02-04 00:25:23 +00:00