This is my patch

[ 587993 ] SET_LINENO killer

Remove SET_LINENO.  Tracing is now supported by inspecting co_lnotab.

Many sundry changes to document and adapt to this change.

Doc/lib/libdis.tex

@@ -23,17 +23,15 @@ the following command can be used to get the disassembly of
 
 \begin{verbatim}
 >>> dis.dis(myfunc)
-          0 SET_LINENO          1
+  2           0 LOAD_GLOBAL              0 (len)
-
+              3 LOAD_FAST                0 (alist)
-          3 SET_LINENO          2
+              6 CALL_FUNCTION            1
-          6 LOAD_GLOBAL         0 (len)
+              9 RETURN_VALUE        
-          9 LOAD_FAST           0 (alist)
+             10 RETURN_NONE
-         12 CALL_FUNCTION       1
-         15 RETURN_VALUE   
-         16 LOAD_CONST          0 (None)
-         19 RETURN_VALUE   
 \end{verbatim}
 
+(The ``2'' is a line number).
+
 The \module{dis} module defines the following functions and constants:
 
 \begin{funcdesc}{dis}{\optional{bytesource}}
@@ -56,6 +54,7 @@ Disassembles a code object, indicating the last instruction if \var{lasti}
 was provided.  The output is divided in the following columns:
 
 \begin{enumerate}
+\item the line number, for the first instruction of each line
 \item the current instruction, indicated as \samp{-->},
 \item a labelled instruction, indicated with \samp{>\code{>}},
 \item the address of the instruction,
@@ -402,6 +401,14 @@ is evaluated, the locals are passed to the class definition.
 Returns with TOS to the caller of the function.
 \end{opcodedesc}
 
+\begin{opcodedesc}{RETURN_NONE}{}
+Returns \constant{None} to the caller of the function.  This opcode is
+generated as the last opcode of every function and only then, for
+reasons to do with tracing support.  See the comments in the function
+\cfunction{maybe_call_line_trace} in \file{Python/ceval.c} for the
+gory details.  \versionadded{2.3}.
+\end{opcodedesc}
+
 \begin{opcodedesc}{YIELD_VALUE}{}
 Pops \code{TOS} and yields it from a generator.
 \end{opcodedesc}
@@ -621,7 +628,7 @@ free variable storage.
 \end{opcodedesc}
 
 \begin{opcodedesc}{SET_LINENO}{lineno}
-Sets the current line number to \var{lineno}.
+This opcode is obsolete.
 \end{opcodedesc}
 
 \begin{opcodedesc}{RAISE_VARARGS}{argc}

Doc/lib/libtraceback.tex

@@ -118,10 +118,10 @@ A shorthand for \code{format_list(extract_stack(\var{f}, \var{limit}))}.
 
 \begin{funcdesc}{tb_lineno}{tb}
 This function returns the current line number set in the traceback
-object.  This is normally the same as the \code{\var{tb}.tb_lineno}
+object.  This function was necessary because in versions of Python
-field of the object, but when optimization is used (the -O flag) this
+prior to 2.3 when the \programopt{O} flag was passed to Python the
-field is not updated correctly; this function calculates the correct
+\code{\var{tb}.tb_lineno} was not updated correctly.  This function
-value.
+has no use in versions past 2.3.
 \end{funcdesc}
 
 

Doc/tut/tut.tex

@@ -2340,12 +2340,11 @@ Some tips for experts:
 
 \item
 When the Python interpreter is invoked with the \programopt{-O} flag,
-optimized code is generated and stored in \file{.pyo} files.
+optimized code is generated and stored in \file{.pyo} files.  The
-The optimizer currently doesn't help much; it only removes
+optimizer currently doesn't help much; it only removes
-\keyword{assert} statements and \code{SET_LINENO} instructions.
+\keyword{assert} statements.  When \programopt{-O} is used, \emph{all}
-When \programopt{-O} is used, \emph{all} bytecode is optimized;
+bytecode is optimized; \code{.pyc} files are ignored and \code{.py}
-\code{.pyc} files are ignored and \code{.py} files are compiled to
+files are compiled to optimized bytecode.
-optimized bytecode.
 
 \item
 Passing two \programopt{-O} flags to the Python interpreter

Doc/whatsnew/whatsnew23.tex

@@ -658,7 +658,6 @@ In 2.3, you get this:
 
 \end{itemize}
 
-
 %======================================================================
 \section{New and Improved Modules}
 
@@ -987,9 +986,9 @@ allocate objects, and \cfunction{PyObject_GC_Del} to deallocate them.
 when running Python's \file{configure} script.  (Contributed by Ondrej
 Palkovsky.)
 
-\item The  \csimplemacro{DL_EXPORT} and \csimplemacro{DL_IMPORT} macros are now
+\item The \csimplemacro{DL_EXPORT} and \csimplemacro{DL_IMPORT} macros
-deprecated.  Initialization functions for Python extension modules
+are now deprecated.  Initialization functions for Python extension
-should now be declared using the new macro
+modules should now be declared using the new macro
 \csimplemacro{PyMODINIT_FUNC}, while the Python core will generally
 use the \csimplemacro{PyAPI_FUNC} and \csimplemacro{PyAPI_DATA}
 macros.
@@ -1076,6 +1075,29 @@ Finally, there are various miscellaneous fixes:
 \item The tools used to build the documentation now work under Cygwin
 as well as \UNIX.
 
+\item The \code{SET_LINENO} opcode has been removed.  Back in the
+mists of time, this opcode was needed to produce line numbers in
+tracebacks and support trace functions (for, e.g., \module{pdb}).
+Since Python 1.5, the line numbers in tracebacks have been computed
+using a different mechanism that works with ``python -O''.  For Python
+2.3 Michael Hudson implemented a similar scheme to determine when to
+call the trace function, removing the need for \code{SET_LINENO}
+entirely.
+
+Python code will be hard pushed to notice a difference from this
+change, apart from a slight speed up when python is run without
+\programopt{-O}.
+
+C extensions that access the \member{f_lineno} field of frame objects
+should instead call \code{PyCode_Addr2Line(f->f_code, f->f_lasti)}.
+This will have the added effect of making the code work as desired
+under ``python -O'' in earlier versions of Python.
+
+To make tracing work as expected, it was found necessary to add a new
+opcode, \cdata{RETURN_NONE}, to the VM.  If you want to know why, read
+the comments in the function \cfunction{maybe_call_line_trace} in
+\file{Python/ceval.c}.
+
 \end{itemize}
 
 

Include/opcode.h

@@ -71,6 +71,9 @@ extern "C" {
 #define INPLACE_OR	79
 #define BREAK_LOOP	80
 
+#define RETURN_NONE	81 /* *only* for function epilogues 
+			      -- see comments in 
+			      ceval.c:maybe_call_line_trace for why */
 #define LOAD_LOCALS	82
 #define RETURN_VALUE	83
 #define IMPORT_STAR	84
@@ -119,8 +122,6 @@ extern "C" {
 #define STORE_FAST	125	/* Local variable number */
 #define DELETE_FAST	126	/* Local variable number */
 
-#define SET_LINENO	127	/* Current line number */
-
 #define RAISE_VARARGS	130	/* Number of raise arguments (1, 2 or 3) */
 /* CALL_FUNCTION_XXX opcodes defined below depend on this definition */
 #define CALL_FUNCTION	131	/* #args + (#kwargs<<8) */

Lib/dis.py

@@ -55,6 +55,20 @@ def distb(tb=None):
 def disassemble(co, lasti=-1):
     """Disassemble a code object."""
     code = co.co_code
+
+    byte_increments = [ord(c) for c in co.co_lnotab[0::2]]
+    line_increments = [ord(c) for c in co.co_lnotab[1::2]]
+    table_length = len(byte_increments) # == len(line_increments)
+
+    lineno = co.co_firstlineno
+    table_index = 0
+    while (table_index < table_length
+           and byte_increments[table_index] == 0):
+        lineno += line_increments[table_index]
+        table_index += 1
+    addr = 0
+    line_incr = 0
+    
     labels = findlabels(code)
     n = len(code)
     i = 0
@@ -63,7 +77,23 @@ def disassemble(co, lasti=-1):
     while i < n:
         c = code[i]
         op = ord(c)
-        if op == SET_LINENO and i > 0: print # Extra blank line
+
+        if i >= addr:
+            lineno += line_incr
+            while table_index < table_length:
+                addr += byte_increments[table_index]
+                line_incr = line_increments[table_index]
+                table_index += 1
+                if line_incr:
+                    break
+            else:
+                addr = sys.maxint
+            if i > 0:
+                print
+            print "%3d"%lineno,
+        else:
+            print '   ',
+
         if i == lasti: print '-->',
         else: print '   ',
         if i in labels: print '>>',
@@ -224,6 +254,7 @@ def_op('INPLACE_XOR', 78)
 def_op('INPLACE_OR', 79)
 def_op('BREAK_LOOP', 80)
 
+def_op('RETURN_NONE', 81)
 def_op('LOAD_LOCALS', 82)
 def_op('RETURN_VALUE', 83)
 def_op('IMPORT_STAR', 84)
@@ -277,9 +308,6 @@ haslocal.append(125)
 def_op('DELETE_FAST', 126)      # Local variable number
 haslocal.append(126)
 
-def_op('SET_LINENO', 127)       # Current line number
-SET_LINENO = 127
-
 def_op('RAISE_VARARGS', 130)    # Number of raise arguments (1, 2, or 3)
 def_op('CALL_FUNCTION', 131)    # #args + (#kwargs << 8)
 def_op('MAKE_FUNCTION', 132)    # Number of args with default values

Lib/inspect.py

@@ -711,7 +711,7 @@ def getframeinfo(frame, context=1):
         raise TypeError, 'arg is not a frame or traceback object'
 
     filename = getsourcefile(frame) or getfile(frame)
-    lineno = getlineno(frame)
+    lineno = frame.f_lineno
     if context > 0:
         start = lineno - 1 - context//2
         try:
@@ -730,18 +730,8 @@ def getframeinfo(frame, context=1):
 
 def getlineno(frame):
     """Get the line number from a frame object, allowing for optimization."""
-    # Written by Marc-André Lemburg; revised by Jim Hugunin and Fredrik Lundh.
+    # FrameType.f_lineno is now a descriptor that grovels co_lnotab
-    lineno = frame.f_lineno
+    return frame.f_lineno
-    code = frame.f_code
-    if hasattr(code, 'co_lnotab'):
-        table = code.co_lnotab
-        lineno = code.co_firstlineno
-        addr = 0
-        for i in range(0, len(table), 2):
-            addr = addr + ord(table[i])
-            if addr > frame.f_lasti: break
-            lineno = lineno + ord(table[i+1])
-    return lineno
 
 def getouterframes(frame, context=1):
     """Get a list of records for a frame and all higher (calling) frames.

Lib/pdb.py

@@ -105,7 +105,13 @@ class Pdb(bdb.Bdb, cmd.Cmd):
                 if len(line) > 0 and line[0] != '#':
                     self.onecmd(line)
 
-    # Override Bdb methods (except user_call, for now)
+    # Override Bdb methods 
+
+    def user_call(self, frame, argument_list):
+        """This method is called when there is the remote possibility
+        that we ever need to stop in this function."""
+        print '--Call--'
+        self.interaction(frame, None)
 
     def user_line(self, frame):
         """This function is called when we stop or break at this line."""

Lib/test/test_hotshot.py

@@ -91,10 +91,8 @@ class HotShotTestCase(unittest.TestCase):
         f_lineno = f.func_code.co_firstlineno
         g_lineno = g.func_code.co_firstlineno
         events = [(ENTER, ("test_hotshot", g_lineno, "g")),
-                  (LINE,  ("test_hotshot", g_lineno, "g")),
                   (LINE,  ("test_hotshot", g_lineno+1, "g")),
                   (ENTER, ("test_hotshot", f_lineno, "f")),
-                  (LINE,  ("test_hotshot", f_lineno, "f")),
                   (LINE,  ("test_hotshot", f_lineno+1, "f")),
                   (LINE,  ("test_hotshot", f_lineno+2, "f")),
                   (EXIT,  ("test_hotshot", f_lineno, "f")),

Lib/traceback.py

@@ -59,7 +59,7 @@ def print_tb(tb, limit=None, file=None):
     n = 0
     while tb is not None and (limit is None or n < limit):
         f = tb.tb_frame
-        lineno = tb_lineno(tb)
+        lineno = tb.tb_lineno
         co = f.f_code
         filename = co.co_filename
         name = co.co_name
@@ -92,7 +92,7 @@ def extract_tb(tb, limit = None):
     n = 0
     while tb is not None and (limit is None or n < limit):
         f = tb.tb_frame
-        lineno = tb_lineno(tb)
+        lineno = tb.tb_lineno
         co = f.f_code
         filename = co.co_filename
         name = co.co_name
@@ -263,7 +263,7 @@ def extract_stack(f=None, limit = None):
     list = []
     n = 0
     while f is not None and (limit is None or n < limit):
-        lineno = f.f_lineno     # XXX Too bad if -O is used
+        lineno = f.f_lineno
         co = f.f_code
         filename = co.co_filename
         name = co.co_name
@@ -279,23 +279,6 @@ def extract_stack(f=None, limit = None):
 def tb_lineno(tb):
     """Calculate correct line number of traceback given in tb.
 
-    Even works with -O on.
+    Obsolete in 2.3.
     """
-    # Coded by Marc-Andre Lemburg from the example of PyCode_Addr2Line()
+    return tb.tb_lineno
-    # in compile.c.
-    # Revised version by Jim Hugunin to work with JPython too.
-
-    c = tb.tb_frame.f_code
-    if not hasattr(c, 'co_lnotab'):
-        return tb.tb_lineno
-
-    tab = c.co_lnotab
-    line = c.co_firstlineno
-    stopat = tb.tb_lasti
-    addr = 0
-    for i in range(0, len(tab), 2):
-        addr = addr + ord(tab[i])
-        if addr > stopat:
-            break
-        line = line + ord(tab[i+1])
-    return line

Misc/NEWS

@@ -57,6 +57,11 @@ Type/class unification and new-style classes
 
 Core and builtins
 
+- SET_LINENO is gone.  co_lnotab is now consulted to determine when to
+  call the trace function.  C code that accessed f_lineno should call
+  PyCode_Addr2Line instead (f_lineno is still there, but not kept up
+  to date).
+
 - There's a new warning category, FutureWarning.  This is used to warn
   about a number of situations where the value or sign of an integer
   result will change in Python 2.4 as a result of PEP 237 (integer

Modules/_hotshot.c

@@ -152,7 +152,7 @@ logreader_tp_iter(LogReaderObject *self)
  * Low bits:    Opcode:        Meaning:
  *       0x00         ENTER     enter a frame
  *       0x01          EXIT     exit a frame
- *       0x02        LINENO     SET_LINENO instruction was executed
+ *       0x02        LINENO     execution moved onto a different line
  *       0x03         OTHER     more bits are needed to deecode
  *
  * If the type is OTHER, the record is not packed so tightly, and the
@@ -888,9 +888,12 @@ tracer_callback(ProfilerObject *self, PyFrameObject *frame, int what,
 
     case PyTrace_LINE:
         if (self->linetimings)
-            return pack_lineno_tdelta(self, frame->f_lineno, get_tdelta(self));
+            return pack_lineno_tdelta(self, PyCode_Addr2Line(frame->f_code, 
+							     frame->f_lasti),
+				      get_tdelta(self));
         else
-            return pack_lineno(self, frame->f_lineno);
+            return pack_lineno(self, PyCode_Addr2Line(frame->f_code,
+						      frame->f_lasti));
 
     default:
         /* ignore PyTrace_EXCEPTION */
@@ -1227,8 +1230,8 @@ PyDoc_STRVAR(profiler_object__doc__,
 "\n"
 "closed:       True if the profiler has already been closed.\n"
 "frametimings: True if ENTER/EXIT events collect timing information.\n"
-"lineevents:   True if SET_LINENO events are reported to the profiler.\n"
+"lineevents:   True if line events are reported to the profiler.\n"
-"linetimings:  True if SET_LINENO events collect timing information.");
+"linetimings:  True if line events collect timing information.");
 
 static PyTypeObject ProfilerType = {
     PyObject_HEAD_INIT(NULL)

Objects/frameobject.c

@@ -16,7 +16,6 @@ static PyMemberDef frame_memberlist[] = {
 	{"f_builtins",	T_OBJECT,	OFF(f_builtins),RO},
 	{"f_globals",	T_OBJECT,	OFF(f_globals),	RO},
 	{"f_lasti",	T_INT,		OFF(f_lasti),	RO},
-	{"f_lineno",	T_INT,		OFF(f_lineno),	RO},
 	{"f_restricted",T_INT,		OFF(f_restricted),RO},
 	{"f_trace",	T_OBJECT,	OFF(f_trace)},
 	{"f_exc_type",	T_OBJECT,	OFF(f_exc_type)},
@@ -33,8 +32,19 @@ frame_getlocals(PyFrameObject *f, void *closure)
 	return f->f_locals;
 }
 
+static PyObject *
+frame_getlineno(PyFrameObject *f, void *closure)
+{
+	int lineno;
+
+	lineno = PyCode_Addr2Line(f->f_code, f->f_lasti);
+
+	return PyInt_FromLong(lineno);
+}
+
 static PyGetSetDef frame_getsetlist[] = {
 	{"f_locals",	(getter)frame_getlocals, NULL, NULL},
+	{"f_lineno",	(getter)frame_getlineno, NULL, NULL},
 	{0}
 };
 
@@ -306,7 +316,7 @@ PyFrame_New(PyThreadState *tstate, PyCodeObject *code, PyObject *globals,
 	f->f_exc_type = f->f_exc_value = f->f_exc_traceback = NULL;
 	f->f_tstate = tstate;
 
-	f->f_lasti = 0;
+	f->f_lasti = -1;
 	f->f_lineno = code->co_firstlineno;
 	f->f_restricted = (builtins != tstate->interp->builtins);
 	f->f_iblock = 0;

Python/ceval.c

@@ -51,6 +51,9 @@ static int call_trace(Py_tracefunc, PyObject *, PyFrameObject *,
 static void call_trace_protected(Py_tracefunc, PyObject *,
 				 PyFrameObject *, int);
 static void call_exc_trace(Py_tracefunc, PyObject *, PyFrameObject *);
+static void maybe_call_line_trace(int, Py_tracefunc, PyObject *, 
+				  PyFrameObject *, int *, int *);
+
 static PyObject *apply_slice(PyObject *, PyObject *, PyObject *);
 static int assign_slice(PyObject *, PyObject *,
 			PyObject *, PyObject *);
@@ -499,6 +502,16 @@ eval_frame(PyFrameObject *f)
 	PyObject *retval = NULL;	/* Return value */
 	PyThreadState *tstate = PyThreadState_GET();
 	PyCodeObject *co;
+
+	/* when tracing we set things up so that 
+
+               not (instr_lb <= current_bytecode_offset < instr_ub)
+
+	   is true when the line being executed has changed.  The 
+           initial values are such as to make this false the first
+           time it is tested. */
+	int instr_ub = -1, instr_lb = 0;
+
 	unsigned char *first_instr;
 	PyObject *names;
 	PyObject *consts;
@@ -586,7 +599,12 @@ eval_frame(PyFrameObject *f)
 	fastlocals = f->f_localsplus;
 	freevars = f->f_localsplus + f->f_nlocals;
 	_PyCode_GETCODEPTR(co, &first_instr);
-	next_instr = first_instr + f->f_lasti;
+	if (f->f_lasti < 0) {
+		next_instr = first_instr;
+	}
+	else {
+		next_instr = first_instr + f->f_lasti;
+	}
 	stack_pointer = f->f_stacktop;
 	assert(stack_pointer != NULL);
 	f->f_stacktop = NULL;	/* remains NULL unless yield suspends frame */
@@ -637,8 +655,9 @@ eval_frame(PyFrameObject *f)
 	w = NULL;
 
 	for (;;) {
-		assert(stack_pointer >= f->f_valuestack);	/* else underflow */
+		assert(stack_pointer >= f->f_valuestack); /* else underflow */
-		assert(STACK_LEVEL() <= f->f_stacksize);	/* else overflow */
+		assert(STACK_LEVEL() <= f->f_stacksize);  /* else overflow */
+
 		/* Do periodic things.  Doing this every time through
 		   the loop would add too much overhead, so we do it
 		   only every Nth instruction.  We also do it if
@@ -658,8 +677,8 @@ eval_frame(PyFrameObject *f)
 #if !defined(HAVE_SIGNAL_H) || defined(macintosh)
 			/* If we have true signals, the signal handler
 			   will call Py_AddPendingCall() so we don't
-			   have to call sigcheck().  On the Mac and
+			   have to call PyErr_CheckSignals().  On the 
-			   DOS, alas, we have to call it. */
+			   Mac and DOS, alas, we have to call it. */
 			if (PyErr_CheckSignals()) {
 				why = WHY_EXCEPTION;
 				goto on_error;
@@ -686,9 +705,7 @@ eval_frame(PyFrameObject *f)
 	fast_next_opcode:
 		/* Extract opcode and argument */
 
-#if defined(Py_DEBUG) || defined(LLTRACE)
 		f->f_lasti = INSTR_OFFSET();
-#endif
 
 		opcode = NEXTOP();
 		if (HAS_ARG(opcode))
@@ -708,15 +725,26 @@ eval_frame(PyFrameObject *f)
 		if (lltrace) {
 			if (HAS_ARG(opcode)) {
 				printf("%d: %d, %d\n",
-					(int) (INSTR_OFFSET() - 3),
+				       f->f_lasti, opcode, oparg);
-					opcode, oparg);
 			}
 			else {
 				printf("%d: %d\n",
-					(int) (INSTR_OFFSET() - 1), opcode);
+				       f->f_lasti, opcode);
 			}
 		}
 #endif
+
+		/* line-by-line tracing support */
+
+		if (tstate->c_tracefunc != NULL && !tstate->tracing) {
+			/* see maybe_call_line_trace
+			   for expository comments */
+			maybe_call_line_trace(opcode, 
+					      tstate->c_tracefunc,
+					      tstate->c_traceobj,
+					      f, &instr_lb, &instr_ub);
+		}
+
 		/* Main switch on opcode */
 
 		switch (opcode) {
@@ -728,26 +756,6 @@ eval_frame(PyFrameObject *f)
 
 		/* case STOP_CODE: this is an error! */
 
-		case SET_LINENO:
-#ifdef LLTRACE
-			if (lltrace)
-				printf("--- %s:%d \n", filename, oparg);
-#endif
-			f->f_lineno = oparg;
-			if (tstate->c_tracefunc == NULL || tstate->tracing)
-				goto fast_next_opcode;
-			/* Trace each line of code reached */
-			f->f_lasti = INSTR_OFFSET();
-			/* Inline call_trace() for performance: */
-			tstate->tracing++;
-			tstate->use_tracing = 0;
-			err = (tstate->c_tracefunc)(tstate->c_traceobj, f,
-						    PyTrace_LINE, Py_None);
-			tstate->use_tracing = (tstate->c_tracefunc
-					       || tstate->c_profilefunc);
-			tstate->tracing--;
-			break;
-
 		case LOAD_FAST:
 			x = GETLOCAL(oparg);
 			if (x != NULL) {
@@ -1504,9 +1512,17 @@ eval_frame(PyFrameObject *f)
 			why = WHY_RETURN;
 			break;
 
+		case RETURN_NONE:
+			retval = Py_None;
+			Py_INCREF(retval);
+			why = WHY_RETURN;
+			break;
+
 		case YIELD_VALUE:
 			retval = POP();
 			f->f_stacktop = stack_pointer;
+			/* abuse the lasti field: here it points to 
+			   the *next* instruction */
 			f->f_lasti = INSTR_OFFSET();
 			why = WHY_YIELD;
 			break;
@@ -1954,7 +1970,6 @@ eval_frame(PyFrameObject *f)
 		    int n = na + 2 * nk;
 		    PyObject **pfunc = stack_pointer - n - 1;
 		    PyObject *func = *pfunc;
-		    f->f_lasti = INSTR_OFFSET() - 3; /* For tracing */
 
 		    /* Always dispatch PyCFunction first, because
 		       these are presumed to be the most frequent
@@ -2022,7 +2037,6 @@ eval_frame(PyFrameObject *f)
 			    n++;
 		    pfunc = stack_pointer - n - 1;
 		    func = *pfunc;
-		    f->f_lasti = INSTR_OFFSET() - 3; /* For tracing */
 
 		    if (PyMethod_Check(func)
 			&& PyMethod_GET_SELF(func) != NULL) {
@@ -2134,7 +2148,8 @@ eval_frame(PyFrameObject *f)
 		default:
 			fprintf(stderr,
 				"XXX lineno: %d, opcode: %d\n",
-				f->f_lineno, opcode);
+				PyCode_Addr2Line(f->f_code, f->f_lasti),
+				opcode);
 			PyErr_SetString(PyExc_SystemError, "unknown opcode");
 			why = WHY_EXCEPTION;
 			break;
@@ -2189,9 +2204,6 @@ eval_frame(PyFrameObject *f)
 		/* Log traceback info if this is a real exception */
 
 		if (why == WHY_EXCEPTION) {
-			f->f_lasti = INSTR_OFFSET() - 1;
-			if (HAS_ARG(opcode))
-				f->f_lasti -= 2;
 			PyTraceBack_Here(f);
 
 			if (tstate->c_tracefunc != NULL)
@@ -2875,6 +2887,125 @@ call_trace(Py_tracefunc func, PyObject *obj, PyFrameObject *frame,
 	return result;
 }
 
+static void
+maybe_call_line_trace(int opcode, Py_tracefunc func, PyObject *obj, 
+		      PyFrameObject *frame, int *instr_lb, int *instr_ub)
+{
+	/* The theory of SET_LINENO-less tracing.
+	   
+	   In a nutshell, we use the co_lnotab field of the code object
+	   to tell when execution has moved onto a different line.
+
+	   As mentioned above, the basic idea is so set things up so
+	   that
+
+	         *instr_lb <= frame->f_lasti < *instr_ub
+
+	   is true so long as execution does not change lines.
+
+	   This is all fairly simple.  Digging the information out of
+	   co_lnotab takes some work, but is conceptually clear.
+
+	   Somewhat harder to explain is why we don't call the line
+	   trace function when executing a POP_TOP or RETURN_NONE
+	   opcodes.  An example probably serves best.
+
+	   Consider this code:
+
+	   1: def f(a):
+	   2:     if a:
+	   3:        print 1
+	   4:     else:
+	   5:        print 2
+
+	   which compiles to this:
+
+	   2           0 LOAD_FAST                0 (a)
+		       3 JUMP_IF_FALSE            9 (to 15)
+		       6 POP_TOP             
+
+	   3           7 LOAD_CONST               1 (1)
+		      10 PRINT_ITEM          
+		      11 PRINT_NEWLINE       
+		      12 JUMP_FORWARD             6 (to 21)
+		 >>   15 POP_TOP             
+
+	   5          16 LOAD_CONST               2 (2)
+		      19 PRINT_ITEM          
+		      20 PRINT_NEWLINE       
+		 >>   21 RETURN_NONE         
+
+	   If a is false, execution will jump to instruction at offset
+	   15 and the co_lnotab will claim that execution has moved to
+	   line 3.  This is at best misleading.  In this case we could
+	   associate the POP_TOP with line 4, but that doesn't make
+	   sense in all cases (I think).
+
+	   On the other hand, if a is true, execution will jump from
+	   instruction offset 12 to offset 21.  Then the co_lnotab would
+	   imply that execution has moved to line 5, which is again
+	   misleading.
+
+	   This is why it is important that RETURN_NONE is *only* used
+	   for the "falling off the end of the function" form of
+	   returning None -- using it for code like
+
+	   1: def f():
+	   2:     return
+
+	   would, once again, lead to misleading tracing behaviour.
+
+	   It is also worth mentioning that getting tracing behaviour
+	   right is the *entire* motivation for adding the RETURN_NONE
+	   opcode.
+	*/
+
+	if (opcode != POP_TOP && opcode != RETURN_NONE &&
+	    (frame->f_lasti < *instr_lb || frame->f_lasti > *instr_ub)) {
+		PyCodeObject* co = frame->f_code;
+		int size, addr;
+		unsigned char* p;
+
+		call_trace(func, obj, frame, PyTrace_LINE, Py_None);
+
+		size = PyString_Size(co->co_lnotab) / 2;
+		p = (unsigned char*)PyString_AsString(co->co_lnotab);
+
+		/* possible optimization: if f->f_lasti == instr_ub
+		   (likely to be a common case) then we already know
+		   instr_lb -- if we stored the matching value of p
+		   somwhere we could skip the first while loop. */
+
+		addr = 0;
+
+		/* see comments in compile.c for the description of
+		   co_lnotab.  A point to remember: increments to p
+		   should come in pairs -- although we don't care about
+		   the line increments here, treating them as byte
+		   increments gets confusing, to say the least. */
+
+		while (size >= 0) {
+			if (addr + *p > frame->f_lasti)
+				break;
+			addr += *p++;
+			p++;
+			--size;
+		}
+		*instr_lb = addr;
+		if (size > 0) {
+			while (--size >= 0) {
+				addr += *p++;
+				if (*p++)
+					break;
+			}
+			*instr_ub = addr;
+		}
+		else {
+			*instr_ub = INT_MAX;
+		}
+	}
+}
+
 void
 PyEval_SetProfile(Py_tracefunc func, PyObject *arg)
 {

Python/compile.c

@@ -407,9 +407,10 @@ PyCode_New(int argcount, int nlocals, int stacksize, int flags,
 
 /* All about c_lnotab.
 
-c_lnotab is an array of unsigned bytes disguised as a Python string.  In -O
+c_lnotab is an array of unsigned bytes disguised as a Python string.  Since
-mode, SET_LINENO opcodes aren't generated, and bytecode offsets are mapped
+version 2.3, SET_LINENO opcodes are never generated and bytecode offsets are
-to source code line #s (when needed for tracebacks) via c_lnotab instead.
+mapped to source code line #s via c_lnotab instead.
+
 The array is conceptually a list of
     (bytecode offset increment, line number increment)
 pairs.  The details are important and delicate, best illustrated by example:
@@ -830,11 +831,6 @@ static void
 com_addoparg(struct compiling *c, int op, int arg)
 {
 	int extended_arg = arg >> 16;
-	if (op == SET_LINENO) {
-		com_set_lineno(c, arg);
-		if (Py_OptimizeFlag)
-			return;
-	}
 	if (extended_arg){
 		com_addbyte(c, EXTENDED_ARG);
 		com_addint(c, extended_arg);
@@ -1738,7 +1734,7 @@ com_call_function(struct compiling *c, node *n)
 			  break;
 			if (ch->n_lineno != lineno) {
 				lineno = ch->n_lineno;
-				com_addoparg(c, SET_LINENO, lineno);
+				com_set_lineno(c, lineno);
 			}
 			com_argument(c, ch, &keywords);
 			if (keywords == NULL)
@@ -3168,7 +3164,7 @@ com_if_stmt(struct compiling *c, node *n)
 			continue;
 		}
 		if (i > 0)
-			com_addoparg(c, SET_LINENO, ch->n_lineno);
+			com_set_lineno(c, ch->n_lineno);
 		com_node(c, ch);
 		com_addfwref(c, JUMP_IF_FALSE, &a);
 		com_addbyte(c, POP_TOP);
@@ -3195,7 +3191,7 @@ com_while_stmt(struct compiling *c, node *n)
 	com_addfwref(c, SETUP_LOOP, &break_anchor);
 	block_push(c, SETUP_LOOP);
 	c->c_begin = c->c_nexti;
-	com_addoparg(c, SET_LINENO, n->n_lineno);
+	com_set_lineno(c, n->n_lineno);
 	com_node(c, CHILD(n, 1));
 	com_addfwref(c, JUMP_IF_FALSE, &anchor);
 	com_addbyte(c, POP_TOP);
@@ -3228,7 +3224,7 @@ com_for_stmt(struct compiling *c, node *n)
 	com_node(c, CHILD(n, 3));
 	com_addbyte(c, GET_ITER);
 	c->c_begin = c->c_nexti;
-	com_addoparg(c, SET_LINENO, n->n_lineno);
+	com_set_lineno(c, n->n_lineno);
 	com_addfwref(c, FOR_ITER, &anchor);
 	com_push(c, 1);
 	com_assign(c, CHILD(n, 1), OP_ASSIGN, NULL);
@@ -3339,7 +3335,7 @@ com_try_except(struct compiling *c, node *n)
 		}
 		except_anchor = 0;
 		com_push(c, 3); /* tb, val, exc pushed by exception */
-		com_addoparg(c, SET_LINENO, ch->n_lineno);
+		com_set_lineno(c, ch->n_lineno);
 		if (NCH(ch) > 1) {
 			com_addbyte(c, DUP_TOP);
 			com_push(c, 1);
@@ -3401,7 +3397,7 @@ com_try_finally(struct compiling *c, node *n)
 	com_push(c, 3);
 	com_backpatch(c, finally_anchor);
 	ch = CHILD(n, NCH(n)-1);
-	com_addoparg(c, SET_LINENO, ch->n_lineno);
+	com_set_lineno(c, ch->n_lineno);
 	com_node(c, ch);
 	com_addbyte(c, END_FINALLY);
 	block_pop(c, END_FINALLY);
@@ -3727,7 +3723,7 @@ com_node(struct compiling *c, node *n)
 
 	case simple_stmt:
 		/* small_stmt (';' small_stmt)* [';'] NEWLINE */
-		com_addoparg(c, SET_LINENO, n->n_lineno);
+		com_set_lineno(c, n->n_lineno);
 		{
 			int i;
 			for (i = 0; i < NCH(n)-1; i += 2)
@@ -3736,7 +3732,7 @@ com_node(struct compiling *c, node *n)
 		break;
 	
 	case compound_stmt:
-		com_addoparg(c, SET_LINENO, n->n_lineno);
+		com_set_lineno(c, n->n_lineno);
 		n = CHILD(n, 0);
 		goto loop;
 
@@ -3990,10 +3986,7 @@ compile_funcdef(struct compiling *c, node *n)
 	c->c_infunction = 1;
 	com_node(c, CHILD(n, 4));
 	c->c_infunction = 0;
-	com_addoparg(c, LOAD_CONST, com_addconst(c, Py_None));
+	com_addbyte(c, RETURN_NONE);
-	com_push(c, 1);
-	com_addbyte(c, RETURN_VALUE);
-	com_pop(c, 1);
 }
 
 static void
@@ -4050,7 +4043,7 @@ compile_classdef(struct compiling *c, node *n)
 static void
 compile_node(struct compiling *c, node *n)
 {
-	com_addoparg(c, SET_LINENO, n->n_lineno);
+	com_set_lineno(c, n->n_lineno);
 	
 	switch (TYPE(n)) {
 	
@@ -4060,19 +4053,13 @@ compile_node(struct compiling *c, node *n)
 		n = CHILD(n, 0);
 		if (TYPE(n) != NEWLINE)
 			com_node(c, n);
-		com_addoparg(c, LOAD_CONST, com_addconst(c, Py_None));
+		com_addbyte(c, RETURN_NONE);
-		com_push(c, 1);
-		com_addbyte(c, RETURN_VALUE);
-		com_pop(c, 1);
 		c->c_interactive--;
 		break;
 	
 	case file_input: /* A whole file, or built-in function exec() */
 		com_file_input(c, n);
-		com_addoparg(c, LOAD_CONST, com_addconst(c, Py_None));
+		com_addbyte(c, RETURN_NONE);
-		com_push(c, 1);
-		com_addbyte(c, RETURN_VALUE);
-		com_pop(c, 1);
 		break;
 	
 	case eval_input: /* Built-in function input() */

Python/frozen.c

@@ -13,12 +13,12 @@
 
 static unsigned char M___hello__[] = {
 	99,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,
-	0,115,15,0,0,0,127,0,0,127,1,0,100,0,0,71,
+	0,115,9,0,0,0,100,0,0,71,72,100,1,0,83,40,
-	72,100,1,0,83,40,2,0,0,0,115,14,0,0,0,72,
+	2,0,0,0,115,14,0,0,0,72,101,108,108,111,32,119,
-	101,108,108,111,32,119,111,114,108,100,46,46,46,78,40,0,
+	111,114,108,100,46,46,46,78,40,0,0,0,0,40,0,0,
-	0,0,0,40,0,0,0,0,40,0,0,0,0,40,0,0,
+	0,0,40,0,0,0,0,40,0,0,0,0,115,8,0,0,
-	0,0,115,8,0,0,0,104,101,108,108,111,46,112,121,115,
+	0,104,101,108,108,111,46,112,121,115,1,0,0,0,63,1,
-	1,0,0,0,63,1,0,0,0,115,0,0,0,0,
+	0,0,0,115,0,0,0,0,
 };
 
 #define SIZE (int)sizeof(M___hello__)

Python/import.c

@@ -49,6 +49,9 @@ extern time_t PyOS_GetLastModificationTime(char *, FILE *);
        algorithm relying on the above scheme. Perhaps we should simply
        start counting in increments of 10 from now on ?!
 
+       MWH, 2002-08-03: Removed SET_LINENO.  Couldn't be bothered figuring
+       out the MAGIC schemes, so just incremented it by 10.
+
    Known values:
        Python 1.5:   20121
        Python 1.5.1: 20121
@@ -60,8 +63,9 @@ extern time_t PyOS_GetLastModificationTime(char *, FILE *);
        Python 2.1.2: 60202
        Python 2.2:   60717
        Python 2.3a0: 62011
+       Python 2.3a0: 62021
 */
-#define MAGIC (62011 | ((long)'\r'<<16) | ((long)'\n'<<24))
+#define MAGIC (62021 | ((long)'\r'<<16) | ((long)'\n'<<24))
 
 /* Magic word as global; note that _PyImport_Init() can change the
    value of this global to accommodate for alterations of how the

Python/traceback.c

@@ -103,8 +103,7 @@ PyTypeObject PyTraceBack_Type = {
 };
 
 static tracebackobject *
-newtracebackobject(tracebackobject *next, PyFrameObject *frame, int lasti,
+newtracebackobject(tracebackobject *next, PyFrameObject *frame)
-		   int lineno)
 {
 	tracebackobject *tb;
 	if ((next != NULL && !PyTraceBack_Check(next)) ||
@@ -118,8 +117,9 @@ newtracebackobject(tracebackobject *next, PyFrameObject *frame, int lasti,
 		tb->tb_next = next;
 		Py_XINCREF(frame);
 		tb->tb_frame = frame;
-		tb->tb_lasti = lasti;
+		tb->tb_lasti = frame->f_lasti;
-		tb->tb_lineno = lineno;
+		tb->tb_lineno = PyCode_Addr2Line(frame->f_code, 
+						 frame->f_lasti);
 		PyObject_GC_Track(tb);
 	}
 	return tb;
@@ -130,8 +130,7 @@ PyTraceBack_Here(PyFrameObject *frame)
 {
 	PyThreadState *tstate = frame->f_tstate;
 	tracebackobject *oldtb = (tracebackobject *) tstate->curexc_traceback;
-	tracebackobject *tb = newtracebackobject(oldtb,
+	tracebackobject *tb = newtracebackobject(oldtb, frame);
-				frame, frame->f_lasti, frame->f_lineno);
 	if (tb == NULL)
 		return -1;
 	tstate->curexc_traceback = (PyObject *)tb;

Tools/scripts/trace.py

@@ -370,41 +370,29 @@ class CoverageResults:
             except IOError, err:
                 sys.stderr.write("cannot save counts files because %s" % err)
 
-# Given a code string, return the SET_LINENO information
+def _find_LINENO_from_code(code):
-def _find_LINENO_from_string(co_code):
+    """return the numbers of the lines containing the source code that
-    """return all of the SET_LINENO information from a code string"""
+    was compiled into code"""
-    import dis
     linenos = {}
 
-    # This code was filched from the `dis' module then modified
+    line_increments = [ord(c) for c in code.co_lnotab[1::2]]
-    n = len(co_code)
+    table_length = len(line_increments)
-    i = 0
+
-    prev_op = None
+    lineno = code.co_first_lineno
-    prev_lineno = 0
+
-    while i < n:
+    for li in line_increments:
-        c = co_code[i]
+        linenos[lineno] = 1
-        op = ord(c)
+        lineno += li
-        if op == dis.SET_LINENO:
+    linenos[lineno] = 1
-            if prev_op == op:
+
-                # two SET_LINENO in a row, so the previous didn't
-                # indicate anything.  This occurs with triple
-                # quoted strings (?).  Remove the old one.
-                del linenos[prev_lineno]
-            prev_lineno = ord(co_code[i+1]) + ord(co_code[i+2])*256
-            linenos[prev_lineno] = 1
-        if op >= dis.HAVE_ARGUMENT:
-            i = i + 3
-        else:
-            i = i + 1
-        prev_op = op
     return linenos
 
 def _find_LINENO(code):
-    """return all of the SET_LINENO information from a code object"""
+    """return all of the lineno information from a code object"""
     import types
 
     # get all of the lineno information from the code of this scope level
-    linenos = _find_LINENO_from_string(code.co_code)
+    linenos = _find_LINENO_from_code(code)
 
     # and check the constants for references to other code objects
     for c in code.co_consts:
@@ -416,9 +404,6 @@ def _find_LINENO(code):
 def find_executable_linenos(filename):
     """return a dict of the line numbers from executable statements in a file
 
-    Works by finding all of the code-like objects in the module then searching
-    the byte code for 'SET_LINENO' terms (so this won't work one -O files).
-
     """
     import parser
 
@@ -428,10 +413,6 @@ def find_executable_linenos(filename):
     ast = parser.suite(prog)
     code = parser.compileast(ast, filename)
 
-    # The only way I know to find line numbers is to look for the
-    # SET_LINENO instructions.  Isn't there some way to get it from
-    # the AST?
-
     return _find_LINENO(code)
 
 ### XXX because os.path.commonprefix seems broken by my way of thinking...