Document the "st" API, to avoid confusion with the "new" AST.
Add a note about using the new AST module.
This commit is contained in:
Georg Brandl
parent
5f27af055b
commit
9cea511f9b
|
|
@ -24,6 +24,17 @@ from this. This is better than trying to parse and modify an arbitrary Python
|
|||
code fragment as a string because parsing is performed in a manner identical to
|
||||
the code forming the application. It is also faster.
|
||||
|
||||
.. note::
|
||||
|
||||
From Python 2.5 onward, it's much more convenient to cut in at the Abstract
|
||||
Syntax Tree (AST) generation and compilation stage, using the :mod:`ast`
|
||||
module.
|
||||
|
||||
The :mod:`parser` module exports the names documented here also with "st"
|
||||
replaced by "ast"; this is a legacy from the time when there was no other
|
||||
AST and has nothing to do with the AST found in Python 2.5. This is also the
|
||||
reason for the functions' keyword arguments being called *ast*, not *st*.
|
||||
|
||||
There are a few things to note about this module which are important to making
|
||||
use of the data structures created. This is not a tutorial on editing the parse
|
||||
trees for Python code, but some examples of using the :mod:`parser` module are
|
||||
|
|
@ -34,9 +45,9 @@ internal parser is required. For full information on the language syntax, refer
|
|||
to :ref:`reference-index`. The parser
|
||||
itself is created from a grammar specification defined in the file
|
||||
:file:`Grammar/Grammar` in the standard Python distribution. The parse trees
|
||||
stored in the AST objects created by this module are the actual output from the
|
||||
stored in the ST objects created by this module are the actual output from the
|
||||
internal parser when created by the :func:`expr` or :func:`suite` functions,
|
||||
described below. The AST objects created by :func:`sequence2ast` faithfully
|
||||
described below. The ST objects created by :func:`sequence2st` faithfully
|
||||
simulate those structures. Be aware that the values of the sequences which are
|
||||
considered "correct" will vary from one version of Python to another as the
|
||||
formal grammar for the language is revised. However, transporting code from one
|
||||
|
|
@ -46,7 +57,7 @@ migrating to an older version of the interpreter will not support more recent
|
|||
language constructs. The parse trees are not typically compatible from one
|
||||
version to another, whereas source code has always been forward-compatible.
|
||||
|
||||
Each element of the sequences returned by :func:`ast2list` or :func:`ast2tuple`
|
||||
Each element of the sequences returned by :func:`st2list` or :func:`st2tuple`
|
||||
has a simple form. Sequences representing non-terminal elements in the grammar
|
||||
always have a length greater than one. The first element is an integer which
|
||||
identifies a production in the grammar. These integers are given symbolic names
|
||||
|
|
@ -69,19 +80,19 @@ of the :keyword:`if` keyword above is representative. The various types of
|
|||
terminal symbols are defined in the C header file :file:`Include/token.h` and
|
||||
the Python module :mod:`token`.
|
||||
|
||||
The AST objects are not required to support the functionality of this module,
|
||||
The ST objects are not required to support the functionality of this module,
|
||||
but are provided for three purposes: to allow an application to amortize the
|
||||
cost of processing complex parse trees, to provide a parse tree representation
|
||||
which conserves memory space when compared to the Python list or tuple
|
||||
representation, and to ease the creation of additional modules in C which
|
||||
manipulate parse trees. A simple "wrapper" class may be created in Python to
|
||||
hide the use of AST objects.
|
||||
hide the use of ST objects.
|
||||
|
||||
The :mod:`parser` module defines functions for a few distinct purposes. The
|
||||
most important purposes are to create AST objects and to convert AST objects to
|
||||
most important purposes are to create ST objects and to convert ST objects to
|
||||
other representations such as parse trees and compiled code objects, but there
|
||||
are also functions which serve to query the type of parse tree represented by an
|
||||
AST object.
|
||||
ST object.
|
||||
|
||||
|
||||
.. seealso::
|
||||
|
|
@ -94,20 +105,20 @@ AST object.
|
|||
testing node values.
|
||||
|
||||
|
||||
.. _creating-asts:
|
||||
.. _creating-sts:
|
||||
|
||||
Creating AST Objects
|
||||
--------------------
|
||||
Creating ST Objects
|
||||
-------------------
|
||||
|
||||
AST objects may be created from source code or from a parse tree. When creating
|
||||
an AST object from source, different functions are used to create the ``'eval'``
|
||||
ST objects may be created from source code or from a parse tree. When creating
|
||||
an ST object from source, different functions are used to create the ``'eval'``
|
||||
and ``'exec'`` forms.
|
||||
|
||||
|
||||
.. function:: expr(source)
|
||||
|
||||
The :func:`expr` function parses the parameter *source* as if it were an input
|
||||
to ``compile(source, 'file.py', 'eval')``. If the parse succeeds, an AST object
|
||||
to ``compile(source, 'file.py', 'eval')``. If the parse succeeds, an ST object
|
||||
is created to hold the internal parse tree representation, otherwise an
|
||||
appropriate exception is thrown.
|
||||
|
||||
|
|
@ -115,22 +126,22 @@ and ``'exec'`` forms.
|
|||
.. function:: suite(source)
|
||||
|
||||
The :func:`suite` function parses the parameter *source* as if it were an input
|
||||
to ``compile(source, 'file.py', 'exec')``. If the parse succeeds, an AST object
|
||||
to ``compile(source, 'file.py', 'exec')``. If the parse succeeds, an ST object
|
||||
is created to hold the internal parse tree representation, otherwise an
|
||||
appropriate exception is thrown.
|
||||
|
||||
|
||||
.. function:: sequence2ast(sequence)
|
||||
.. function:: sequence2st(sequence)
|
||||
|
||||
This function accepts a parse tree represented as a sequence and builds an
|
||||
internal representation if possible. If it can validate that the tree conforms
|
||||
to the Python grammar and all nodes are valid node types in the host version of
|
||||
Python, an AST object is created from the internal representation and returned
|
||||
Python, an ST object is created from the internal representation and returned
|
||||
to the called. If there is a problem creating the internal representation, or
|
||||
if the tree cannot be validated, a :exc:`ParserError` exception is thrown. An
|
||||
AST object created this way should not be assumed to compile correctly; normal
|
||||
exceptions thrown by compilation may still be initiated when the AST object is
|
||||
passed to :func:`compileast`. This may indicate problems not related to syntax
|
||||
ST object created this way should not be assumed to compile correctly; normal
|
||||
exceptions thrown by compilation may still be initiated when the ST object is
|
||||
passed to :func:`compilest`. This may indicate problems not related to syntax
|
||||
(such as a :exc:`MemoryError` exception), but may also be due to constructs such
|
||||
as the result of parsing ``del f(0)``, which escapes the Python parser but is
|
||||
checked by the bytecode compiler.
|
||||
|
|
@ -142,31 +153,31 @@ and ``'exec'`` forms.
|
|||
symbols in the input tree.
|
||||
|
||||
|
||||
.. function:: tuple2ast(sequence)
|
||||
.. function:: tuple2st(sequence)
|
||||
|
||||
This is the same function as :func:`sequence2ast`. This entry point is
|
||||
This is the same function as :func:`sequence2st`. This entry point is
|
||||
maintained for backward compatibility.
|
||||
|
||||
|
||||
.. _converting-asts:
|
||||
.. _converting-sts:
|
||||
|
||||
Converting AST Objects
|
||||
----------------------
|
||||
Converting ST Objects
|
||||
---------------------
|
||||
|
||||
AST objects, regardless of the input used to create them, may be converted to
|
||||
ST objects, regardless of the input used to create them, may be converted to
|
||||
parse trees represented as list- or tuple- trees, or may be compiled into
|
||||
executable code objects. Parse trees may be extracted with or without line
|
||||
numbering information.
|
||||
|
||||
|
||||
.. function:: ast2list(ast[, line_info])
|
||||
.. function:: st2list(ast[, line_info])
|
||||
|
||||
This function accepts an AST object from the caller in *ast* and returns a
|
||||
This function accepts an ST object from the caller in *ast* and returns a
|
||||
Python list representing the equivalent parse tree. The resulting list
|
||||
representation can be used for inspection or the creation of a new parse tree in
|
||||
list form. This function does not fail so long as memory is available to build
|
||||
the list representation. If the parse tree will only be used for inspection,
|
||||
:func:`ast2tuple` should be used instead to reduce memory consumption and
|
||||
:func:`st2tuple` should be used instead to reduce memory consumption and
|
||||
fragmentation. When the list representation is required, this function is
|
||||
significantly faster than retrieving a tuple representation and converting that
|
||||
to nested lists.
|
||||
|
|
@ -177,29 +188,29 @@ numbering information.
|
|||
This information is omitted if the flag is false or omitted.
|
||||
|
||||
|
||||
.. function:: ast2tuple(ast[, line_info])
|
||||
.. function:: st2tuple(ast[, line_info])
|
||||
|
||||
This function accepts an AST object from the caller in *ast* and returns a
|
||||
This function accepts an ST object from the caller in *ast* and returns a
|
||||
Python tuple representing the equivalent parse tree. Other than returning a
|
||||
tuple instead of a list, this function is identical to :func:`ast2list`.
|
||||
tuple instead of a list, this function is identical to :func:`st2list`.
|
||||
|
||||
If *line_info* is true, line number information will be included for all
|
||||
terminal tokens as a third element of the list representing the token. This
|
||||
information is omitted if the flag is false or omitted.
|
||||
|
||||
|
||||
.. function:: compileast(ast[, filename='<ast>'])
|
||||
.. function:: compilest(ast[, filename='<syntax-tree>'])
|
||||
|
||||
.. index:: builtin: eval
|
||||
|
||||
The Python byte compiler can be invoked on an AST object to produce code objects
|
||||
The Python byte compiler can be invoked on an ST object to produce code objects
|
||||
which can be used as part of an :keyword:`exec` statement or a call to the
|
||||
built-in :func:`eval` function. This function provides the interface to the
|
||||
compiler, passing the internal parse tree from *ast* to the parser, using the
|
||||
source file name specified by the *filename* parameter. The default value
|
||||
supplied for *filename* indicates that the source was an AST object.
|
||||
supplied for *filename* indicates that the source was an ST object.
|
||||
|
||||
Compiling an AST object may result in exceptions related to compilation; an
|
||||
Compiling an ST object may result in exceptions related to compilation; an
|
||||
example would be a :exc:`SyntaxError` caused by the parse tree for ``del f(0)``:
|
||||
this statement is considered legal within the formal grammar for Python but is
|
||||
not a legal language construct. The :exc:`SyntaxError` raised for this
|
||||
|
|
@ -209,15 +220,15 @@ numbering information.
|
|||
tree.
|
||||
|
||||
|
||||
.. _querying-asts:
|
||||
.. _querying-sts:
|
||||
|
||||
Queries on AST Objects
|
||||
----------------------
|
||||
Queries on ST Objects
|
||||
---------------------
|
||||
|
||||
Two functions are provided which allow an application to determine if an AST was
|
||||
Two functions are provided which allow an application to determine if an ST was
|
||||
created as an expression or a suite. Neither of these functions can be used to
|
||||
determine if an AST was created from source code via :func:`expr` or
|
||||
:func:`suite` or from a parse tree via :func:`sequence2ast`.
|
||||
determine if an ST was created from source code via :func:`expr` or
|
||||
:func:`suite` or from a parse tree via :func:`sequence2st`.
|
||||
|
||||
|
||||
.. function:: isexpr(ast)
|
||||
|
|
@ -227,19 +238,19 @@ determine if an AST was created from source code via :func:`expr` or
|
|||
When *ast* represents an ``'eval'`` form, this function returns true, otherwise
|
||||
it returns false. This is useful, since code objects normally cannot be queried
|
||||
for this information using existing built-in functions. Note that the code
|
||||
objects created by :func:`compileast` cannot be queried like this either, and
|
||||
objects created by :func:`compilest` cannot be queried like this either, and
|
||||
are identical to those created by the built-in :func:`compile` function.
|
||||
|
||||
|
||||
.. function:: issuite(ast)
|
||||
|
||||
This function mirrors :func:`isexpr` in that it reports whether an AST object
|
||||
This function mirrors :func:`isexpr` in that it reports whether an ST object
|
||||
represents an ``'exec'`` form, commonly known as a "suite." It is not safe to
|
||||
assume that this function is equivalent to ``not isexpr(ast)``, as additional
|
||||
syntactic fragments may be supported in the future.
|
||||
|
||||
|
||||
.. _ast-errors:
|
||||
.. _st-errors:
|
||||
|
||||
Exceptions and Error Handling
|
||||
-----------------------------
|
||||
|
|
@ -255,12 +266,12 @@ function for information about the exceptions it can raise.
|
|||
generally produced for validation failures rather than the built in
|
||||
:exc:`SyntaxError` thrown during normal parsing. The exception argument is
|
||||
either a string describing the reason of the failure or a tuple containing a
|
||||
sequence causing the failure from a parse tree passed to :func:`sequence2ast`
|
||||
and an explanatory string. Calls to :func:`sequence2ast` need to be able to
|
||||
sequence causing the failure from a parse tree passed to :func:`sequence2st`
|
||||
and an explanatory string. Calls to :func:`sequence2st` need to be able to
|
||||
handle either type of exception, while calls to other functions in the module
|
||||
will only need to be aware of the simple string values.
|
||||
|
||||
Note that the functions :func:`compileast`, :func:`expr`, and :func:`suite` may
|
||||
Note that the functions :func:`compilest`, :func:`expr`, and :func:`suite` may
|
||||
throw exceptions which are normally thrown by the parsing and compilation
|
||||
process. These include the built in exceptions :exc:`MemoryError`,
|
||||
:exc:`OverflowError`, :exc:`SyntaxError`, and :exc:`SystemError`. In these
|
||||
|
|
@ -268,49 +279,49 @@ cases, these exceptions carry all the meaning normally associated with them.
|
|||
Refer to the descriptions of each function for detailed information.
|
||||
|
||||
|
||||
.. _ast-objects:
|
||||
.. _st-objects:
|
||||
|
||||
AST Objects
|
||||
-----------
|
||||
ST Objects
|
||||
----------
|
||||
|
||||
Ordered and equality comparisons are supported between AST objects. Pickling of
|
||||
AST objects (using the :mod:`pickle` module) is also supported.
|
||||
Ordered and equality comparisons are supported between ST objects. Pickling of
|
||||
ST objects (using the :mod:`pickle` module) is also supported.
|
||||
|
||||
|
||||
.. data:: ASTType
|
||||
.. data:: STType
|
||||
|
||||
The type of the objects returned by :func:`expr`, :func:`suite` and
|
||||
:func:`sequence2ast`.
|
||||
:func:`sequence2st`.
|
||||
|
||||
AST objects have the following methods:
|
||||
ST objects have the following methods:
|
||||
|
||||
|
||||
.. method:: AST.compile([filename])
|
||||
.. method:: ST.compile([filename])
|
||||
|
||||
Same as ``compileast(ast, filename)``.
|
||||
Same as ``compilest(st, filename)``.
|
||||
|
||||
|
||||
.. method:: AST.isexpr()
|
||||
.. method:: ST.isexpr()
|
||||
|
||||
Same as ``isexpr(ast)``.
|
||||
Same as ``isexpr(st)``.
|
||||
|
||||
|
||||
.. method:: AST.issuite()
|
||||
.. method:: ST.issuite()
|
||||
|
||||
Same as ``issuite(ast)``.
|
||||
Same as ``issuite(st)``.
|
||||
|
||||
|
||||
.. method:: AST.tolist([line_info])
|
||||
.. method:: ST.tolist([line_info])
|
||||
|
||||
Same as ``ast2list(ast, line_info)``.
|
||||
Same as ``st2list(st, line_info)``.
|
||||
|
||||
|
||||
.. method:: AST.totuple([line_info])
|
||||
.. method:: ST.totuple([line_info])
|
||||
|
||||
Same as ``ast2tuple(ast, line_info)``.
|
||||
Same as ``st2tuple(st, line_info)``.
|
||||
|
||||
|
||||
.. _ast-examples:
|
||||
.. _st-examples:
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
|
@ -338,27 +349,27 @@ to the code ::
|
|||
10
|
||||
|
||||
The equivalent operation using the :mod:`parser` module is somewhat longer, and
|
||||
allows the intermediate internal parse tree to be retained as an AST object::
|
||||
allows the intermediate internal parse tree to be retained as an ST object::
|
||||
|
||||
>>> import parser
|
||||
>>> ast = parser.expr('a + 5')
|
||||
>>> code = ast.compile('file.py')
|
||||
>>> st = parser.expr('a + 5')
|
||||
>>> code = st.compile('file.py')
|
||||
>>> a = 5
|
||||
>>> eval(code)
|
||||
10
|
||||
|
||||
An application which needs both AST and code objects can package this code into
|
||||
An application which needs both ST and code objects can package this code into
|
||||
readily available functions::
|
||||
|
||||
import parser
|
||||
|
||||
def load_suite(source_string):
|
||||
ast = parser.suite(source_string)
|
||||
return ast, ast.compile()
|
||||
st = parser.suite(source_string)
|
||||
return st, st.compile()
|
||||
|
||||
def load_expression(source_string):
|
||||
ast = parser.expr(source_string)
|
||||
return ast, ast.compile()
|
||||
st = parser.expr(source_string)
|
||||
return st, st.compile()
|
||||
|
||||
|
||||
Information Discovery
|
||||
|
|
@ -412,8 +423,8 @@ tuples. ::
|
|||
|
||||
>>> import parser
|
||||
>>> import pprint
|
||||
>>> ast = parser.suite(open('docstring.py').read())
|
||||
>>> tup = ast.totuple()
|
||||
>>> st = parser.suite(open('docstring.py').read())
|
||||
>>> tup = st.totuple()
|
||||
>>> pprint.pprint(tup)
|
||||
(257,
|
||||
(264,
|
||||
|
|
@ -670,8 +681,8 @@ file :file:`example.py`.) ::
|
|||
|
||||
source = open(fileName).read()
|
||||
basename = os.path.basename(os.path.splitext(fileName)[0])
|
||||
ast = parser.suite(source)
|
||||
return ModuleInfo(ast.totuple(), basename)
|
||||
st = parser.suite(source)
|
||||
return ModuleInfo(st.totuple(), basename)
|
||||
|
||||
This provides an easy-to-use interface to the documentation of a module. If
|
||||
information is required which is not extracted by the code of this example, the
|
||||
|
|
|
|||
Loading…
Reference in New Issue