Building Mac Python from source

This document explains how to build MacPython from source. This is necessary if you want to write extension modules for 68K Python, and is also necessary if you want to make modifications to the Python core. Building Python is not something to be undertaken lightly, you need a reasonable working knowledge of the CodeWarrior development environment, a good net connection and probably quite some time too.

The information density in this file is high, so you should probably print it and read it at your leasure. Most things are explained only once (and probably in the wrong place:-).

First a warning: this information may become outdated if a new CodeWarrior is released after MacPython. The MacPython homepage will hopefully have updated instructions in that case.

I am very interested in feedback on this document, send your comments to the Mac Python Special Interest Group.

What you need.

The following things you definitely need:

You need a MacPython source distribution, of course. You can obtain one from ftp://ftp.cwi.nl/pub/jack/python/mac or from the companion webpage at http://www.cwi.nl/~jack/macpython.html (which has up-to-date links to the other packages needed too) and possibly also from the standard python.org ftp site. Everything you need is also included in the standard Python source distribution, but the organization is different. Look in directory Mac/mwerks/projects for the project files and related stuff.
An alternative is to check the sources straight out of the CVS repository, see below. Most of the packages mentioned here are also available through CVS. Check the section on CVS repository use below.
You need MetroWerks CodeWarrior. The current distribution has been built with CodeWarrior Pro 4. Ordering information is available on the MetroWerks homepage. Building Python with MPW or Think/Symantec C is probably impossible without major surgery.
You need GUSI, the Grand Unified Socket Interface, by Matthias Neeracher. The original CWGUSI is obtainable from ftp://sunsite.cnlab-switch.ch/software/platform/macos/src. At the moment Python is built with a slightly modified version of GUSI, these modifications are available in folder Python:Mac:GUSI-mods.
The modified GUSI is also in the MacPython cvs source repository, in the directory lib-src/CWGUSI. However, some files contain slashes in their names, something CVS seriously frowns upon, and each slash has been replaced by "_s_". There is a script Mac:scripts:fixgusidir.py which you should run after checking CWGUSI out

The MacPython project files are configured to include a plethora of optional modules, and these modules need a number of extra packages. To use the project files as-is you have to download these packages too. PPC and CFM68K Python have all such modules as dynamically loaded modules, so if you don't need a certain package it suffices to just refrain from builing the extension module. For static 68K Python things are a bit more complicated: you have to edit the interpreter project file to remove the reference to the module (and the libraries it uses), and edit the Mac:mwerks:mwerks_nonshared_config.h file to remove the USE_... line. Here are the locations for the various things you need:

Tcl and Tk are in a sad state on the Mac, the standard source distributions simply don't compile, so I have created a distribution especially for use with MacPython. See the section on building Tcl/Tk Python below.
Waste, a TextEdit replacement written by Marco Piovanelli, <piovanel@kagi.com>. Python was built using version 1.3, which you can obtain from <http://www.boingo.com/waste> and various other places.
Gdbm library for the Mac. Available from Jack's Mac software page at http://www.cwi.nl/~jack/macsoftware.html and ftp://ftp.cwi.nl/pub/jack/mac. Also in the MacPython cvs repository at lib-src/gdbm.
JPEG library by the Independent JPEG Group. A version including Mac projects can be found at Jack's page mentioned above. The most recent JPEG library can always be obtained from ftp://ftp.uu.net/graphics/jpeg/. Again, also in the MacPython cvs repository at lib-src/jpeg.
The netpbm/pbmplus, libtiff, zlib and png libraries. The netpbm distribution (which includes libtiff) is generally available on Internet ftp servers. For Python pbmplus, an older incarnation of netpbm, is functionally identical to netpbm, since Python only uses the library and not the complete applications. A distribution with correct projects and library source only is available from, you guessed it, Jack's Mac software page mentioned above. And, guessed it again, in the MacPython cvs repository at lib-src/netpbm, etc. The only gotcha is that libtiff lives in lib-src/netpbm/libtiff, for historical reasons.

Setting Up

Now that you have collected everything you should start with building the various parts. If you don't want to fix access paths try to set things up as follows:

Top-level-folder:
	CWGUSI
	imglibs
		jpeg
		netpbm
			libtiff
		zlib
		png
	gdbm
	Python
		Modules
		...
		Mac
			Modules
			Build
			...
	Tcl/Tk Folder
		tcl8.0
		tk8.0
		MoreFiles 1.4.3
	Waste 1.3 distribution (if you want waste)

If your setup of the libraries is exactly the same as mine (which is not very likely, unless you happen to work from the same CVS repository) you can use the project buildlibs.prj in the :Mac:build.mac folder to build all needed libraries in one fell swoop, otherwise you will have to build the libraries one by one.

First build GUSI. If you didn't get the python-specific GUSI you have to move the files from the "CWGUSI-mods" to the right place in the CWGUSI distribution folder. Build the MSL version for your platform (ppc, 68k, cfm68k).

Note: always rebuild the CWGUSI libraries, even if you have checked them out from the CVS repository.

Next, in MoreFiles, libjpeg, pbmplus, zlib, libpng, gdbm, andlibtiff you build all projects. Usually the projects are in "mac" subfolders, sometimes they are in the main folder. Tcl/tk is a special case, see below. Of course, if you are only interested in one of static 68K, CFM68K or PPC you can skip building the other libraries.

Building Tcl/Tk

The Tcl/Tk 8.3.0 source distribution does not work on the Mac. I have created an archive of the sources that I used to build _tkinter for MacPython, you can obtain this from ftp://ftp.cwi.nl/pub/jack/python/mac/tcltk830src-for-python.sit. Only the libraries needed to build _tkinter for PPC have been fixed.

Note that if you use a different release of Tcl and Tk than the ones I have used you may have to adapt the Python tkresources.rsrc file. This is easiest done by building SimpleTk and copying the TEXT, ICON and CRSR resources from it to tkresources.rsrc. This allows the _tkinter module to work without an installed Tk/Tcl on your machine.

Also note that the _tkinter.ppc.slb that is normally distributed in the PlugIns folder is the one from the Imaging extension, which has some extra features needed by PIL (and which features should not hinder normal operation). Build first the Tcl library, then SimpleTcl (test it by typing ls -l in the window you get) then the Tk library, then SimpleTk (which can again be tested with ls -l). If this all worked you are all set to try building Python.

Building Waste

You do not need to build the Waste libraries, as Python includes the source modules themselves.

The organization of the Python source tree

Time for a short break, while we have a look at the organization of the Python source tree. At the top level, we find the following folders:

Demo: Demo programs that are not Mac-specific. Some of these may not work, the file README-Mac has some details.
Extensions: Extensions to the interpreter that are not Mac-specific. Contains the img, Imaging and Numeric extensions in this distribution. Nowadays, the extensions are all built in their own folders (unlike in older distributions, where img was incorporated in the main build procedure).
Grammar: The Python grammar. Included for reference only, you cannot build the parser on a Mac.
Include: Machine-independent header files.
Modules: Machine-independent optional modules. Not all of these will work on the Mac.
Objects: Machine-independent code for various objects. Most of these are not really optional: the interpreter will not function without them.
Parser: The Python parser (machine-independent).
Python: The core interpreter. Most files are machine-independent, some are unix-specific and not used on the Mac.
Tools: Tools for python developers. Contains modulator which builds skeleton C extension modules and bgen which generates complete interface modules from information in C header files. There are some readme files, but more documentation is sorely needed.

All the mac-specific stuff lives in the Mac folder:

Build: This is where the project files live and where you build the libraries, shared libraries, executables and plugin modules. All the resulting binaries, except for intermedeate results, are deposited in the toplevel folder or the Mac:PlugIns folder (for plugin modules).
Compat: Unix-compatability routines. Some of these are not used anymore, since CWGUSI provides a rather complete emulation, but you may need these if you are trying to build a non-GUSI python.
Demo: Mac-specific demo programs, some of them annotated.
Include: Mac-specific but compiler-independent include files.
Lib: Mac-specific standard modules. The toolbox folder contains modules specifically needed with various MacOS toolbox interface modules.
Modules: Mac-specific builtin modules. Theoretically these are all optional, but some are rather essential (like macmodule). A lot of these modules are generated with bgen, in which case the bgen input files are included so you can attempt to regenerate them or extend them.
MPW: MPW-specific files. These have not been used or kept up-to-date for a long time, so use at your own risk.
mwerks: Mwerks-specific sources and headers. Contains glue code for Pythons shared-library architecture, a replacement for malloc and a directory with various projects for building variations on the Python interpreter. The mwerks_*.h files here are the option-setting files for the various interpreters and such, comparable to the unix command-line -D options to the compiler. Each project uses the correct option file as its "prefix file" in the "C/C++ language" settings. Disabling optional modules (for the 68K interpreter), building non-GUSI interpreters and various other things are accomplished by modifying these files (and possibly changing the list of files included in the project window, of course).
PlugIns: This is where the PPC and CFM68K dynamically-loaded plugin modules live.
Python: Mac-specific parts of the core interpreter.
Resources: Resource files needed to build the interpreter.
Scripts: A collection of various mac-specific Python scripts. Some are essential, some are useful but few are documented, so you will have to use your imagination to work them out.
Tools: A collection of tools, usually bigger than those in the scripts folder. The important ones here are the IDE and macfreeze. The IDE is built with the buildIDE.py script, which puts the resulting applet in the toplevel folder. Macfreeze is usually invoked through the BuildApplication script, but for more control over the freezing process you can run the main script here.
Unsupported: Modules that are not supported any longer but may still work with a little effort.

Building the 68K interpreter

If you have all the optional libraries mentioned above loaded building Python for 68K macs is a breeze: in the Mac:Build folder you build the libraries with buildlibs.prj and then the interpreter with PythonStandalone.prj.

If you were previously running another copy of this Python release, from a binary installer for instance, you should first remove the Python XXX preferences file from your preference folder. Next, run the interpreter, in the toplevel folder. This will create a correct initial preferences file. You are now all set, and your tree should be completely compatible with a binary-only distribution. Read the release notes (Relnotes-somethingorother) and ReadMe in the Mac folder.

If something goes wrong you may end up with a garbled preferences file. Removing it from the system folder and running Python once again will re-create it.

Building the PPC and CFM68K interpreter

First you optionally build the external libraries with buildlibs.prj. Next, the projects for interpreter, core library and applet skeleton are all linked together, so building the fat target in PythonEngine.prj will result in everything being built. The resulting applications and fat shared library are deposited in the main Python folder. Finally, you build all the plugins with the plugins.prj project. For completeness sake here is a breakdown of the projects:

PythonCore (with subprojects PythonCorePPC and PythonCoreCFM68K): The shared library that contains the bulk of the interpreter and its resources. It is a good idea to immedeately put an alias to this shared library in the Extensions folder of your system folder. Do exactly that: put an alias there, copying or moving the file will cause you grief later if you rebuild the library and forget to copy it to the extensions folder again. The InstallPython applet will also do this, along with creating the plugin aliases.
PythonInterpeter: The interpreter. This is basically a routine to call out to the shared library. Unlike in previous releases the same program is used for creating applets (for which formerly PythonApplet was used).
Plugin projects: Each plugin module has a separate project. The Plugins.prj project tries to build them all, but is known to be flakey under CW Pro 4.

After creating the alias to PythonCore you remove any old Python XXX Preferences file from the Preferences folder (if you had python installed on your system before) and run the interpreter once to create the correct preferences file.

Next, you have to build the extension modules. The PlugIns.ppc project has all the other projects as subprojects and builds everything (but see the gotcha above).

Finally, you must build the standard applets: EditPythonPrefs, BuildApplet, etc. This is easiest done with the fullbuild script from Mac:scripts.

Actually, the fullbuild script can be used to build everything, but you need a fully-functional interpreter before you can use it (and one that isn't rebuilt in the process: you cannot rebuild a running program). You could copy the 68K interpreter to a different place and use that to run fullbuild, or use the standalone PPC python for this. I tend to keep a standalone interpreter in a safe place for this use only.

You are all set now, and should read the release notes and ReadMe file from the Mac folder.

Rebuilding `.exp` files for PPC and CFM68K

Occasionally it may be necessary to rebuild your PythonCore .exp file, a file that controls which symbols are exported by your PythonCore shared library. Rebuild it if you get unexpected undefined symbols when you are building a plugin module.

Rebuilding the .exp file is done by first removing the file and removing the reference to it in the project (in the "config" section). Next, build PythonCore. This will create a new .exp file. Edit this file to remove the references to the symbols __initialize, __terminate, setjmp, longjmp, vec_longjmp, main and (for PPC) __ptmf_null or (for CFM68K) __start and dummy_init_routine. Next, add the .exp file to the project again and rebuild PythonCore.

This rather convoluted procedure is needed to ensure that plugin modules don't accidentally link with those entrypoints from PythonCore, which will not work because those routines have to be in the same code fragment as they are used from.

Using the CVS source archive

It is possible to access the Python sources through remote CVS. The advantage of this is that you get the very latest sources, so any bug fixed or new features will be immedeately available. This is also the disadvantage, of course: as this is the same tree as is used for development it may sometimes be a little less stable.

The CVS client of choice is Alexandre Parenteau's MacCVS. It can be obtained through the WinCVS homepage. MacCVS uses Internet Config to set file types correctly based on the filename extension. In the maccvs preferences you should also set (in the "binary files" section) "use mac encoding: applesingle" and (in the "text files" section) "use ISO latin 1 conversion".

It is also a good idea to disable Quicktime Exchange in the Quicktime control panel. Quicktime Exchange will magically map some extensions to filetypes, and this can seriously hinder you if, for instance, .bmp is not a Windows bitmap file.

The machine-independent Python sources are checked out from the main Python CVS archive, see the Source access via CVS page for details. When you check the sources out you will get something like Python:dist:src, and under that the Modules, Lib, etc hierarchy. The src folder should be renamed to Python, and is what this document refers to as the "toplevel Python folder".

Next, in a separate folder, you check out the mac-specific sources. You then move the Mac folder from this checkout (the only folder with anything in it) to the Python source folder. Note that the checking out in a separate folder and moving is necessary, due to the way cvs works. The CVS path to use for the mac stuff can be found at the MacPython homepage. Finally, you check out the external libraries needed in the parent of the toplevel Python folder. The CVS path for these libraries is also mentioned at the MacPython homepage.

Neither of the pages mentioned above contains the passwords for the CVS sites, for obvious reasons, but they do contain instructions on how to obtain the passwords.

You should end up with a folder structure as described at the top of this document.

Note that while the Mac folder is now a subfolder of your toplevel Python folder this does not mean that they "act as one" as far as CVS is concerned. To update all your sources you have to do a "cvs update" in the toplevel Python folder and another one in the Mac folder. This is again a cvs problem: it cannot deal with subpackages coming from different repositories.

Odds and ends

Some remarks that I could not fit in elsewhere:

It may be possible to use the PythonCore shared library to embed Python in another program, if your program can live with using GUSI for I/O. Use PythonCore in stead of your MSL C library (or, at the very least, link it before the normal C library). Let me know whether this works.
It is possible to build PPC extension modules without building a complete Python. The binary distribution installer can optionally install all the needed folders. A template for a dynamic module can be found in xx.prj.
The Python shared library architecture is a variant of the architecture described as "application with shared libraries and dropins" in the MetroWerks "Targeting MacOS" documentation. The Python Application and applet-template use the MSL AppRuntime.Lib runtime library (with properly set CFM initialization and termination routines). PythonCore uses MSL Runtime.Lib, which is really intended for standalone programs but which we fool into working by providing a dummy main program. It is linked statically into PythonCore (and exported to the applications and plugins) so we do not have to distribute yet another shared library. Plugin modules use MSL ShlibRuntime.Lib (not the dropin runtime: modules are never unloaded) and obtain the rest from PythonCore. PythonCore uses a non-standard initialization entry point, __initialize_with_resources, to be able to obtain resources from the library file later on. Plugins can do the same (_tkinter does) or use the standard __initialize entry point.