Commit Graph

304 Commits

Author SHA1 Message Date
Martin v. Löwis 8d0701daf1 Stop generating empty arrays. 2006-02-26 23:40:20 +00:00
Martin v. Löwis ce1d5d2527 Fix iterating over cmpop_ty lists. 2006-02-26 20:51:25 +00:00
Martin v. Löwis bd260da900 Generate code to recursively copy an AST into
a tree of Python objects. Expose this through compile().
2006-02-26 19:42:26 +00:00
Martin v. Löwis f5adf1eb72 Use Py_ssize_t to count the length. 2006-02-16 14:35:38 +00:00
Martin v. Löwis a87c445177 Remove C99ism. 2006-02-15 19:56:18 +00:00
Martin v. Löwis 18e165558b Merge ssize_t branch. 2006-02-15 17:27:45 +00:00
Jeremy Hylton c960f26044 Improved handling of syntax errors.
Expand set of errors caught in set_context().  Some new errors, some
old error messages changed for consistency.

Fixed error checking in generator expression code.  The first set of
tests were impossible condition given the grammar.  In general, the
ast code uses REQ() for those sanity checks.

Fix some error handling for augmented assignments.  As comments in the
code explain, set_context() ought to work here, but I got unexpected
crashes when I tried it.  Should come back to this.

Add note to Grammar that yield expression is a special case.

Add doctest cases for SyntaxErrors raised by ast.c.
2006-01-27 15:18:39 +00:00
Neal Norwitz 46aae198ad Revert previous checkin, the check is for <, not ==. i is unsed in non-debug builds, but is used in debug builds 2006-01-08 02:06:01 +00:00
Neal Norwitz 056a2d6582 Fix icc warnings. This couldn't have been correct since i is checked
for 2 different values without changing.  I think this was the intent.
The unused warning only occurs when not building in debug mode.
2006-01-08 01:10:34 +00:00
Tim Peters 536cf99536 Whitespace normalization. 2005-12-25 23:18:31 +00:00
Neal Norwitz 30b5c5d011 Fix SF bug #1072182, problems with signed characters.
Most of these can be backported.
2005-12-19 06:05:18 +00:00
Neal Norwitz db83eb3170 Fix Bug #1378022, UTF-8 files with a leading BOM crashed the interpreter.
Needs backport.
2005-12-18 05:29:30 +00:00
Neal Norwitz adb69fcdff Merge from ast-arena. This reduces the code in Python/ast.c by ~300 lines,
simplifies a lot of error handling code, and fixes many memory leaks.
2005-12-17 20:54:49 +00:00
Armin Rigo 6b1793ff88 When regenerating files like Python-ast.h, take care that the generated
comment based on 'sys.args[0]' does not depend on the path.  For Python
builds from a remote directory ("/path/to/configure; make") the previous
logic used to include the "/path/to" portion in Python-ast.h.  Then svn
would consider this file to be locally modified.
2005-12-14 18:05:14 +00:00
Neal Norwitz 897ff817d5 SF #1373150, diffs in working copy after a build
Strip off leading dots and slash so the generated files are the same regardless
of whether you configure in the checkout directory or build.

If anyone configures in a different directory, we might want a cleaner
approach using os.path.*().  Hopefully this is good enough.
2005-12-11 21:18:22 +00:00
Neal Norwitz dee2fd5448 Fix some more memory leaks.
Call error_ret() in decode_str().  It was called in some other places,
but seemed inconsistent.  It is safe to call PyTokenizer_Free() after
calling error_ret().
2005-11-16 05:12:59 +00:00
Neal Norwitz 56c6561d30 Prevent unlikely memory leak, tok should always be freed when parsetok() returns 2005-11-15 04:54:23 +00:00
Neal Norwitz 19b0f40cb1 Thou shalt not lie, there are really 5 types now 2005-11-15 04:52:16 +00:00
Neal Norwitz 7b5a604d24 Whoops, checkin consistent versions of *all* files to stop polluting
a bunch of names
2005-11-13 19:14:20 +00:00
Neal Norwitz 6576bd844f Prevent name pollution by making lots of internal functions static. 2005-11-13 18:41:28 +00:00
Martin v. Löwis ab0f947a21 Remove .cvsignore files, as they live in svn:ignore
properties now.
2005-10-30 22:01:41 +00:00
Neal Norwitz a34584be3b Use PyErr_NoMemory() instead of rolling our own.
Get rid of "int i" unused warnings from Python-ast.c which we are generating.
2005-10-23 18:59:17 +00:00
Neal Norwitz c0d5faa9b4 Free coding spec (cs) if there was an error to prevent mem leak. Maybe backport candidate 2005-10-21 06:05:33 +00:00
Jeremy Hylton 3e0055f8c6 Merge ast-branch to head
This change implements a new bytecode compiler, based on a
transformation of the parse tree to an abstract syntax defined in
Parser/Python.asdl.

The compiler implementation is not complete, but it is in stable
enough shape to run the entire test suite excepting two disabled
tests.
2005-10-20 19:59:25 +00:00
Neal Norwitz 40d3781416 - Fix segfault with invalid coding.
- SF Bug #772896, unknown encoding results in MemoryError, which is not helpful

I will only backport the segfault fix.  I'll let Anthony decide if he wants
the other changes backported.  I will do the backport if asked.
2005-10-02 01:48:49 +00:00
Walter Dörwald c1f5fff2b7 Apply SF patch #1101726: Fix buffer overrun in tokenizer.c when a source file
with a PEP 263 encoding declaration results in long decoded line.
2005-07-12 21:53:43 +00:00
Michael W. Hudson e3afc598bc In a threads-disabled build, typing Ctrl-C into a raw_input() crashed,
because (essentially) I didn't realise that PY_BEGIN/END_ALLOW_THREADS
actually expanded to nothing under a no-threads build, so if you somehow
NULLed out the threadstate (e.g. by calling PyThread_SaveThread) it would
stay NULLed when you return to Python.  Argh!

Backport candidate.
2005-04-07 10:11:19 +00:00
Martin v. Löwis 4bf108d74f Patch #802188: better parser error message for non-EOL following line cont. 2005-03-03 11:45:45 +00:00
Anthony Baxter 9ceaa72ebe Patch #975056 - fixes for restartable signals on *BSD. In addition,
a few remaining calls to signal() were converted to PyOS_setsig().
2004-10-13 14:48:50 +00:00
Hye-Shik Chang 7df44b384a SF #941229: Decode source code with sys.stdin.encoding in interactive
modes like non-interactive modes.  This allows for non-latin-1 users
to write unicode strings directly and sets Japanese users free from
weird manual escaping <wink> in shift_jis environments.
(Reviewed by Martin v. Loewis)
2004-08-04 17:36:41 +00:00
Anthony Baxter c2a5a63654 PEP-0318, @decorator-style. In Guido's words:
"@ seems the syntax that everybody can hate equally"
Implementation by Mark Russell, from SF #979728.
2004-08-02 06:10:11 +00:00
Michael W. Hudson 2384990603 PyThreadState_Swap(NULL) didn't do what I thought it did. Fixes
[ 987287 ] Python 2.4a1, interpreter hanging on Keyboard Interrupt
2004-07-08 15:28:26 +00:00
Neal Norwitz 4ded4b52c4 Pass the flags along, rather than ignoring them. Backport candidate 2004-07-08 01:54:07 +00:00
Tim Peters b7e898a0e2 "#if WITH_THREAD" is incorrect; must be #ifdef instead; WITH_THREAD
isn't always set to an integer value when it's defined.
2004-07-07 20:42:07 +00:00
Michael W. Hudson 30ea2f223f This closes patch:
[ 960406 ] unblock signals in threads

although the changes do not correspond exactly to any patch attached to
that report.

Non-main threads no longer have all signals masked.

A different interface to readline is used.

The handling of signals inside calls to PyOS_Readline is now rather
different.

These changes are all a bit scary!  Review and cross-platform testing
much appreciated.
2004-07-07 17:44:12 +00:00
Andrew M. Kuchling 93b4b88e96 [Patch #974633] Check PyObject_MALLOC return for error 2004-06-29 14:03:04 +00:00
Guido van Rossum c69661725a Fix for SF 780407.
Change %08l to %p to print a pointer.
Will backport to 2.3.
2004-03-20 22:34:14 +00:00
Skip Montanaro db6080507d Remove support for --without-universal-newlines (see PEP 11). 2004-02-07 13:53:46 +00:00
Guido van Rossum bd311d8e4e Remove a "temporary" piece of code that was probably unneeded since
mid 1990.  Remove an untrue XXX comment.
2003-12-15 06:06:24 +00:00
Jack Jansen 6402357d40 Talk about old code: removed a reference to THINK_C. 2003-11-20 13:33:48 +00:00
Jack Jansen eddc1449ba Getting rid of all the code inside #ifdef macintosh too. 2003-11-20 01:44:59 +00:00
Jack Jansen fb2765666f Getting rid of support for the ancient Apple MPW compiler. 2003-11-19 15:24:47 +00:00
Tim Peters 8ab2ec432a SF bug 751956: graminit.[ch] don't build on windows
A change from Duncan Booth, to deal with changes in the way pgen gets
built.  Note that graminit.[ch] aren't normally built on Windows (they're
obtained from CVS).
2003-06-17 00:05:53 +00:00
Martin v. Löwis a94568a753 Patch #734231: Update RiscOS support. In particular, correct
riscospath.extsep, and use os.extsep throughout.
2003-05-10 07:36:56 +00:00
Martin v. Löwis c16f3bd8a3 Patch #708495: Port more stuff to OpenVMS. 2003-05-03 09:14:54 +00:00
Guido van Rossum a649e9cb7a Remove file that was moved to ../Include 2003-04-29 14:14:58 +00:00
Guido van Rossum d3ab37f1df Changes from Jonathan Riehl to allow his pgen extension (PEP 269) to
work.  This includes some more code that used to be part of pgen in
the main parser; I'm okay with that.  I'll see if the Windows build
needs work next.
2003-04-17 14:55:42 +00:00
Barry Warsaw edaa071eb4 compile_atom(): Neal's last checkin removing the setting of i broke
the build, so I'm restoring it.  I'm not sure what Neal's intent was,
since the line following the one he removed was "REQN(i, 1)" so i is
obviously used. ;)
2003-02-28 15:27:40 +00:00
Neal Norwitz 5f16a3178c Remove setting i since it isn't used. Found in unrelated bug 690012. 2003-02-28 03:16:07 +00:00
Marc-André Lemburg 1fb1400d08 Add URL for PEP to the source code encoding warning.
Remove the usage of PyErr_WarnExplicit() since this could cause
sensitive information from the source files to appear in e.g.
log files.
2003-02-17 18:31:57 +00:00
Guido van Rossum 4b499dd3fb - Finally fixed the bug in compile() and exec where a string ending
with an indented code block but no newline would raise SyntaxError.
  This would have been a four-line change in parsetok.c...  Except
  codeop.py depends on this behavior, so a compilation flag had to be
  invented that causes the tokenizer to revert to the old behavior;
  this required extra changes to 2 .h files, 2 .c files, and 2 .py
  files.  (Fixes SF bug #501622.)
2003-02-13 22:07:59 +00:00
Just van Rossum f032f86e9e patch 680474 that fixes bug 679880: compile/eval/exec refused utf-8 bom
mark. Added unit test.
2003-02-09 20:38:48 +00:00
Mark Hammond a2e303c32d Fix [ 665014 ] files with long lines and an encoding crash.
Ensure that the 'size' arg is correctly passed to the encoding reader to
prevent buffer overflows.
2003-01-14 23:15:22 +00:00
Martin v. Löwis 95292d6caa Constify filenames and scripts. Fixes #651362. 2002-12-11 14:04:59 +00:00
Neal Norwitz e08e1bc80a Fix compiler warning on HP-UX.
Cast param to isalnum() to int.
2002-11-02 20:43:25 +00:00
Skip Montanaro 35f63a860b Change PyOS_Readline declaration to match the recent change to myreadline.c
(see Patch 512981).  I changed stdin to sys_stdin in the body of the
function, but did not change stderr to sys_stdout, though I suspect that may
be the correct course.  I don't know the code involved well enough to judge.
2002-10-27 01:59:21 +00:00
Martin v. Löwis 566f6afe9a Patch #512981: Update readline input stream on sys.stdin/out change. 2002-10-26 14:39:10 +00:00
Tim Peters 17db21ffd0 Removed reliance on gcc/C99 extension. 2002-09-03 15:39:58 +00:00
Martin v. Löwis f62a89b1e0 Ignore encoding declarations inside strings. Fixes #603509. 2002-09-03 11:52:44 +00:00
Guido van Rossum 84b2bed435 Squash a few calls to the hideously expensive PyObject_CallObject(o,a)
-- replace then with slightly faster PyObject_Call(o,a,NULL).  (The
difference is that the latter requires a to be a tuple; the former
allows other values and wraps them in a tuple if necessary; it
involves two more levels of C function calls to accomplish all that.)
2002-08-16 17:01:09 +00:00
Skip Montanaro 118ec70ea2 provide less mysterious error messages when seeing end-of-line in
single-quoted strings or end-of-file in triple-quoted strings.
closes patch 586561.
2002-08-15 01:20:16 +00:00
Martin v. Löwis 2863c10a86 Use Py_FatalError instead of abort. 2002-08-07 15:18:57 +00:00
Martin v. Löwis 019934b3cc Fix PEP 263 code --without-unicode. Fixes #591943. 2002-08-07 12:33:18 +00:00
Jack Jansen cf0a2cfdb2 Added a cast to shut up a compiler warning. 2002-08-05 14:14:05 +00:00
Martin v. Löwis 725bb233b9 Add 1 to lineno in deprecation warning. Fixes #590888. 2002-08-05 01:49:16 +00:00
Martin v. Löwis 1ee99d31d9 Make pgen compile with pydebug. Duplicate normalized names, as it may
be longer than the old string.
2002-08-04 20:10:29 +00:00
Martin v. Löwis cd280fb59c Group statements properly. 2002-08-04 18:28:44 +00:00
Tim Peters 2c3f9c6f04 Repaired a fatal compiler error in the debug build: it's not clear what
this was trying to assert, but the name it referenced didn't exist.
2002-08-04 17:58:34 +00:00
Tim Peters 919603b27a Squash compiler wng about signed-vs-unsigned mismatch. 2002-08-04 17:56:42 +00:00
Martin v. Löwis 00f1e3f5a5 Patch #534304: Implement phase 1 of PEP 263. 2002-08-04 17:29:52 +00:00
Andrew MacIntyre 80d4e2acf5 SF patch #578297:
Change the parser and compiler to use PyMalloc.

Only the files implementing processes that will request memory
allocations small enough for PyMalloc to be a win have been
changed, which are:-
 - Python/compile.c
 - Parser/acceler.c
 - Parser/node.c
 - Parser/parsetok.c

This augments the aggressive overallocation strategy implemented by
Tim Peters in PyNode_AddChild() [Parser/node.c], in reducing the
impact of platform malloc()/realloc()/free() corner case behaviour.
Such corner cases are known to be triggered by test_longexp and
test_import.

Jeremy Hylton, in accepting this patch, recommended this as a
bugfix candidate for 2.2.  While the changes to Python/compile.c
and Parser/node.c backport easily (and could go in), the changes
to Parser/acceler.c and Parser/parsetok.c require other not
insignificant changes as a result of the differences in the memory
APIs between 2.3 and 2.2, which I'm not in a position to work
through at the moment.  This is a pity, as the Parser/parsetok.c
changes are the most important after the Parser/node.c changes, due
to the size of the memory requests involved and their frequency.
2002-08-04 06:28:21 +00:00
Tim Peters e561dc231e XXXROUNDUP(): Turns out this fixed Andrew MacIntyre's memory-mgmt
disaster too, so this change is here to stay.  Beefed up the comments
and added some stats Andrew reported.  Also a small change to the
macro body, to make it obvious how XXXROUNDUP(0) ends up returning 0.
See SF patch 578297 for context.

Not a bugfix candidate, as the functional changes here have already
been backported to the 2.2 line (this patch just improves clarity).
2002-07-15 17:58:03 +00:00
Mark Hammond 2f10cb8fa5 Fix bug 439992 - [win32] KeyboardInterrupt Not Caught.
This gets us closer to consistent Ctrl+C behaviour on NT and Win9x.  NT now reliably generates KeyboardInterrupt exceptions for NT when a file IO operation was aborted.  Bugfix candidate
2002-07-14 23:12:29 +00:00
Jeremy Hylton 7b4c8e485c I trust the parser accelators are getting added :-). 2002-07-11 15:43:37 +00:00
Thomas Heller 6b17abf6c0 Fix SF Bug 564931: compile() traceback must include filename. 2002-07-09 09:23:27 +00:00
Tim Peters 7c321a80f9 The Py_REF_DEBUG/COUNT_ALLOCS/Py_TRACE_REFS macro minefield: added
more trivial lexical helper macros so that uses of these guys expand
to nothing at all when they're not enabled.  This should help sub-
standard compilers that can't do a good job of optimizing away the
previous "(void)0" expressions.

Py_DECREF:  There's only one definition of this now.  Yay!  That
was that last one in the family defined multiple times in an #ifdef
maze.

Py_FatalError():  Changed the char* signature to const char*.

_Py_NegativeRefcount():  New helper function for the Py_REF_DEBUG
expansion of Py_DECREF.  Calling an external function cuts down on
the volume of generated code.  The previous inline expansion of abort()
didn't work as intended on Windows (the program often kept going, and
the error msg scrolled off the screen unseen).  _Py_NegativeRefcount
calls Py_FatalError instead, which captures our best knowledge of
how to abort effectively across platforms.
2002-07-09 02:57:01 +00:00
Tim Peters 623fdb9884 PyNode_AddChild() and fancy_roundup(): Be paranoid about int overflow. 2002-07-08 19:11:07 +00:00
Tim Peters 755ebea23b PyNode_AddChild(): Do aggressive over-allocation when the number of
children gets large, to avoid severe platform realloc() degeneration
in extreme cases (like test_longexp).

Bugfix candidate.

This was doing extremely timid over-allocation, just rounding up to the
nearest multiple of 3.  Now so long as the number of children is <= 128,
it rounds up to a multiple of 4 but via a much faster method.  When the
number of children exceeds 128, though, and more space is needed, it
doubles the capacity.  This is aggressive over-allocation.

SF patch <http://www.python.org/sf/578297> has Andrew MacIntyre using
PyMalloc in the parser to overcome platform malloc problems in
test_longexp on OS/2 EMX.  Jack Jansen notes there that it didn't help
him on the Mac, because the Mac has problems with frequent ever-growing
reallocs, not just with gazillions of teensy mallocs.  Win98 has no
visible problems with test_longexp, but I tried boosting the test-case
size and soon got "senseless" MemoryErrors out of it, and soon after
crashed the OS:  as I've seen in many other contexts before, while the
Win98 realloc remains zippy in bad cases, it leads to extreme
fragmentation of user address space, to the point that the OS barfs.

I don't yet know whether this fixes Jack's Mac problems, but it does cure
Win98's problems when boosting the test case size.  It also speeds
test_longexp in its unaltered state.
2002-07-08 06:32:09 +00:00
Guido van Rossum 79e39bd4d0 Add definition of Py_IgnoreEnvironmentFlag (needed at least in debug
mode).
2002-05-31 14:32:07 +00:00
Neal Norwitz 5352d8c0c6 Py_IgnoreEnvironmentFlag should be extern, since it is declared in pythonrun.c 2002-05-31 13:11:40 +00:00
Tim Peters 1ccb3eeb7b Link with the right C library! This has always been wrong (& my fault). 2002-05-23 19:25:59 +00:00
Neil Schemenauer 44c4c6dbd6 Py_GETENV is used by obmalloc and needs Py_IgnoreEnvironmentFlag. Provide it. 2002-04-22 03:29:32 +00:00
Tim Peters 1c6192662d Update the Windows makefile for 2.3. 2002-04-22 00:39:44 +00:00
Jack Jansen 7b8c7546eb Mass checkin of universal newline support.
Highlights: import and friends will understand any of \r, \n and \r\n
as end of line. Python file input will do the same if you use mode 'U'.
Everything can be disabled by configuring with --without-universal-newlines.

See PEP278 for details.
2002-04-14 20:12:41 +00:00
Neil Schemenauer c24ea08644 Disable the parser hacks that enabled the "yield" keyword using a future
statement.
2002-03-22 23:53:36 +00:00
Neil Schemenauer c155dd4ca9 Disable the parser hacks that allowed the "yield" keyword to be enabled
by a future statement.
2002-03-22 23:38:11 +00:00
Martin v. Löwis cdc4451222 Include <unistd.h> in Python.h. Fixes #500924. 2002-01-12 11:05:12 +00:00
Martin v. Löwis 16eff6f77b Initialize err_ret with filename if available. Fixes #498828. 2002-01-05 21:40:08 +00:00
Tim Peters 88e138c842 PyGrammar_LabelRepr(): sprintf -> PyOS_snprintf. 2001-12-04 03:36:01 +00:00
Tim Peters 1ca1296157 The parser doesn't need its own implementation of assert, and having its
own interfered with including Python.h.  Remove Python's assert.h.
2001-12-04 03:18:48 +00:00
Barry Warsaw 5947af5ac3 Reverting last change so we don't have to think about the assert macro
redefinition problem.
2001-11-28 21:34:34 +00:00
Barry Warsaw b97c969fee PyGrammar_LabelRepr(): Conversion of sprintf() to PyOS_snprintf() for
buffer overrun avoidance.
2001-11-28 21:04:25 +00:00
Guido van Rossum 0c156a5130 Patch from SF bug #472956: UMR when there is a syntax error (Neal Norwitz)
perrdetail.token is unitialized when there is a syntax
    error in a file.
2001-10-20 14:27:56 +00:00
Tim Peters eba84cdacb An MSVC makefile to rebuild the grammar files (graminit.[ch]) manually.
Ugly, but it works.
2001-10-13 20:16:17 +00:00
Guido van Rossum 25dfe2c095 Silence parser generator output. 2001-09-11 16:43:16 +00:00
Tim Peters d507dab91f SF patch #455966: Allow leading 0 in float/imag literals.
Consequences for Jython still unknown (but raised on Jython-Dev).
2001-08-30 20:51:59 +00:00
Tim Peters 9aa70d93aa SF bug [#455775] float parsing discrepancy.
PyTokenizer_Get:  error if exponent contains no digits (3e, 2.0e+, ...).
2001-08-27 19:19:28 +00:00
Guido van Rossum 4668b000a1 Implement PEP 238 in its (almost) full glory.
This introduces:

- A new operator // that means floor division (the kind of division
  where 1/2 is 0).

- The "future division" statement ("from __future__ import division)
  which changes the meaning of the / operator to implement "true
  division" (where 1/2 is 0.5).

- New overloadable operators __truediv__ and __floordiv__.

- New slots in the PyNumberMethods struct for true and floor division,
  new abstract APIs for them, new opcodes, and so on.

I emphasize that without the future division statement, the semantics
of / will remain unchanged until Python 3.0.

Not yet implemented are warnings (default off) when / is used with int
or long arguments.

This has been on display since 7/31 as SF patch #443474.

Flames to /dev/null.
2001-08-08 05:00:18 +00:00
Guido van Rossum 3c033230ec Fis SF bug #442647: not all forms of legal future statements were
parsed correctly.  Now they are.
2001-07-19 15:27:45 +00:00
Guido van Rossum da62ecc9aa Add a really stupid warning about 'yield' used as an identifier.
This is really stupid because it cannot be suppressed or altered using
the warning framework; that's because the warning framework is built
on Python interpreter internals, and the parser generator doesn't have
access to any of those (you cannot use anything of type PyObject * in
the parser).

But it's better than nothing, and implementing a proper check for this
appears to require modifying compile.c in a dozen places, for which I
don't have the stamina today.  I promise we'll do better in 2.2a2.

At least it tells you the filename and line number (unlike the first
hack I considered :-).
2001-07-17 16:53:11 +00:00