Building MacPython-OS9 from source

This document explains how to build MacPython-OS9 from source. This is necessary if you want to make modifications to the Python core. Building Python is not something to be undertaken lightly, you need a reasonable working knowledge of the CodeWarrior development environment, a good net connection and probably quite some time too.

Note that if you only want to build new extension modules you don't need to build Python from source, see the note on extending Python.

The information density in this file is high, so you should probably print it and read it at your leasure. Most things are explained only once (and probably in the wrong place:-).

First a warning: this information may become outdated if a new CodeWarrior is released after MacPython. The MacPython homepage will hopefully have updated instructions in that case. These instructions are for CW7, it is rumoured you may encounter some problems with newer versions of CodeWarrior.
I am interested in feedback on this document, send your comments to the Mac Python Special Interest Group.

What you need.

The following things you definitely need: The MacPython project files are configured to include a plethora of optional modules, and these modules need a number of extra packages. To use the project files as-is you have to download these packages too. Python has all such modules as dynamically loaded modules, so if you don't need a certain package it suffices to just refrain from builing the extension module. Here are the locations for the various things you need:

Setting Up

Now that you have collected everything you should start with building the various parts. If you don't want to fix access paths try to set things up as follows: 
Top-level-folder:
	GUSI2
	imglibs
		jpeg
		netpbm
			libtiff
		zlib
		png
	gdbm
	Python
		Modules
		...
		Mac
			Modules
			Build
			...
If your setup of the libraries is exactly the same as mine (which is not very likely, unless you happen to work from the same CVS repository) you can use the project buildlibs.prj in the :Mac:Build folder to build all needed libraries in one fell swoop, otherwise you will have to build the libraries one by one.

First build GUSI, the Carbon variant.

Next, in libjpeg, pbmplus, zlib, libpng, gdbm, andlibtiff you build all projects. Usually the projects are in "mac" subfolders, sometimes they are in the main folder. Tcl/tk is a special case, see below.

The organization of the Python source tree

Time for a short break, while we have a look at the organization of the Python source tree. At the top level, we find the following folders:
Demo
Demo programs that are not Mac-specific. Some of these may not work.
Extensions
Extensions to the interpreter that are not Mac-specific. Contains the img, Imaging and Numerical extensions in this distribution.
Grammar
The Python grammar. Included for reference only, you cannot build the parser on a Mac.
Include
Machine-independent header files.
Modules
Machine-independent optional modules. Not all of these will work on the Mac.
Lib
Machine-independent modules in Python.
Lib:lib-dynload
This is where the dynamically-loaded plugin modules live.
Lib:plat-mac
This is where most of the Mac-specific modules live. The modules here are available both in MacPython-OS9 and MacPython-OSX.
Objects
Machine-independent code for various object types. Most of these are not really optional: the interpreter will not function without them.
Parser
The Python parser (machine-independent).
Python
The core interpreter. Most files are machine-independent, some are unix-specific and not used on the Mac.
Tools
Tools for python developers. Contains modulator which builds skeleton C extension modules, bgen which generates complete interface modules from information in C header files and freeze which is used to turn Python scripts into real applications (used by MacFreeze and BuildApplication) There are some readme files, but more documentation is sorely needed.
The mac-specific stuff lives in the Mac folder:
Build
This is where the project files live and where you build the libraries, shared libraries, executables and plugin modules. All the resulting binaries, except for intermedeate results, are deposited in the toplevel folder or the :Lib:lib-dynload folder (for plugin modules).
Compat
Unix-compatability routines. Most of these are not used anymore, since GUSI provides a rather complete emulation, but you may need these if you are trying to build a non-GUSI python.
Demo
Mac-specific demo programs, some of them annotated.
Include
Mac-specific but compiler-independent include files.
Lib
MacPython-OS9 specific standard modules which are not shared with MacPython-OSX.
Modules
Mac-specific builtin modules. Theoretically these are all optional, but some are rather essential (like macosmodule). A lot of these modules are generated with bgen, in which case the bgen input files are included so you can attempt to regenerate them or extend them.
MPW
MPW-specific files. These have not been used or kept up-to-date for a long time, so use at your own risk.
mwerks
Mwerks-specific sources and headers. Contains glue code for Pythons shared-library architecture, a replacement for malloc and a directory with various projects for building variations on the Python interpreter. The mwerks_*.h files here are the option-setting files for the various interpreters and such, comparable to the unix command-line -D options to the compiler. Each project uses the correct option file as its "prefix file" in the "C/C++ language" settings. Disabling optional modules (for the 68K interpreter), building non-GUSI interpreters and various other things are accomplished by modifying these files (and possibly changing the list of files included in the project window, of course).
OSX
Specific to MacPython-OSX, not used by MacPython-OS9.
OSXResources
Specific to MacPython-OSX, not used by MacPython-OS9.
Python
Mac-specific parts of the core interpreter.
Resources
Resource files needed to build the interpreter.
Scripts
A collection of various mac-specific Python scripts. Some are essential, some are useful but few are documented, so you will have to use your imagination to work them out.
Tools
A collection of tools, usually bigger than those in the scripts folder. The important ones here are the IDE and macfreeze. The IDE is built with the buildIDE.py script, which puts the resulting applet in the toplevel folder. Macfreeze is usually invoked through the BuildApplication script, but for more control over the freezing process you can run the main script here.
Unsupported
Modules that are not supported any longer but may still work with a little effort.

Building the PPC interpreter

First you optionally build the external libraries with buildlibs.prj.

Then, the fullbuild script can be used to build everything, but you need a fully-functional interpreter before you can use it (and one that isn't rebuilt in the process: you cannot rebuild a running program). You could copy the interpreter to a different place and use that to run fullbuild. The PythonStandSmall.prj project builds an interpreter that is suited to this, and it can also come in handy if you need to debug things (which is easier in a static program).

In case you want to build by hand, or in case the fullbuild script does not work, here is a breakdown of the various projects.

The projects for interpreter and core library are linked together, so building the PythonInterpreter target in PythonInterpreter.prj will result in the whole core being built, but not the extension modules.

You will get about 100 warnings on "missing prototype" for the various module init routines, ignore these. You will also get numerous warnings on functions from GUSI which override functions from MSL, ignore these too.

Here is a breakdown of the projects:

PythonCore
The shared library that contains the bulk of the interpreter and its resources. It is a good idea to immedeately put an alias to this shared library in the Extensions folder of your system folder. Do exactly that: put an alias there, copying or moving the file will cause you grief later if you rebuild the library and forget to copy it to the extensions folder again. The ConfigurePythonXXX applets will also do this.
PythonInterpeter
The interpreter. This is basically a routine to call out to the shared library.

Plugin projects
Each plugin module has a separate project, and these can be rebuilt on the fly. Fullbuild (or actually it's little helper genpluginprojects) takes care of this.
After creating the alias to PythonCore you remove any old Python XXXX Preferences file from the Preferences folder (if you had python installed on your system before) and run the interpreter once to create the correct preferences file.

Next, you have to build the extension modules. If you don't use fullbuild simply open each project and build it.

Finally, you must build the standard applets: EditPythonPrefs, BuildApplet, etc. For the N-th time: fullbuild does this for you, but you can also manually drag/drop them onto BuildApplet.

You are all set now, and should read the release notes and ReadMe file from the Mac folder. Rebuilding .exp files is no longer needed since CodeWarrior 7.

Using the CVS source archive

It is possible (and probably best) to access the Python sources through remote CVS. The advantage of this is that you get the very latest sources, so any bug fixed or new features will be immedeately available. This is also the disadvantage, of course: as this is the same tree as is used for development it may sometimes be a little less stable.

The CVS client of choice is Alexandre Parenteau's MacCVS. It can be obtained through the WinCVS homepage. MacCVS uses Internet Config to set file types correctly based on the filename extension. In the maccvs preferences you should also set (in the "binary files" section) "use mac encoding: applesingle" and (in the "text files" section) "use ISO latin 1 conversion".

It is a good idea to disable Quicktime Exchange in the Quicktime control panel if you are on OS9 or before. Quicktime Exchange will magically map some extensions to filetypes, and this can seriously hinder you if, for instance, .bmp is not a Windows bitmap file.

The Python sources are checked out from the main Python CVS archive on sourceforge.net, see the Source access via CVS page for details. When you check the sources out you will get something like Python:dist:src, and under that the Modules, Lib, Mac etc hierarchy. The src folder can be renamed to Python, and is what this document refers to as the "toplevel Python folder".

The CVS repository does not contain all the projects for the plugin modules, these are built with fullbuild.py normally. For this reason it is probably a good idea to first build PythonStandSmall.prj, which builds a fairly minimal interpreter, and then follow the fullbuild instructions.

Odds and ends

Some remarks that I could not fit in elsewhere: