cpython/Mac/Demo/building.html

<HTML>
<HEAD>
<TITLE>Building MacPython-OS9 from source</TITLE>
</HEAD>
<BODY>
<H1>Building MacPython-OS9 from source</H1>
<HR>

This document explains how to build MacPython-OS9 from source. This is
necessary if you want to make modifications to the Python core. Building
Python is not something to be undertaken lightly, you need a reasonable
working knowledge of the CodeWarrior development environment, a good net
connection and probably quite some time too. <p>

Note that if you only want to build new extension modules you don't need to
build Python from source, see the <a href="#extending">note on extending Python</a>.<p>

The information density in this file is high, so you should probably
print it and read it at your leasure. Most things are explained only
once (and probably in the wrong place:-). <p>

<blockquote>
First a warning: this information may become outdated if a new CodeWarrior is
released after MacPython. The 
<a href="http://www.cwi.nl/~jack/macpython.html">MacPython homepage</a> will
hopefully have updated instructions in that case. These instructions are for CW7,
it is rumoured you may encounter some problems with newer versions of CodeWarrior.
</blockquote>

I am interested in feedback on this document, send your
comments to the <A
HREF="http://www.python.org/sigs/pythonmac-sig/">Mac Python Special
Interest Group</A>.

<H2>What you need.</H2>

The following things you definitely need:

<UL>

<LI> You need a MacPython source distribution, of course. You can
obtain one via <A HREF="http://www.cwi.nl/~jack/macpython.html">
http://www.cwi.nl/~jack/macpython.html</A>  (which has up-to-date links
to the other packages needed too) and possibly also from the standard
<A HREF="ftp://ftp.python.org/pub/python/mac">python.org ftp
site</A>. <BR>

A better alternative is to check the sources straight out of the CVS
repository, see below. Most of the packages mentioned here are also
available through CVS. Check the section on <a href="#cvs">CVS
repository use</a> below.

<LI> You need MetroWerks CodeWarrior. The current distribution has
been built with CodeWarrior Pro 7.1. Ordering information is
available on the <A HREF="http://www.metrowerks.com/">MetroWerks
homepage</A>. Building Python with MPW, Think/Symantec C or the OSX
developer tools is impossible without major surgery.

<LI> You need GUSI version 2, the Grand Unified Socket Interface, by
Matthias Neeracher. The original GUSI is obtainable from <A
HREF="ftp://gusi.sourceforge.net/pub/gusi/">
ftp://gusi.sourceforge.net/pub/gusi/</A>. At
the moment Python is built with a modified version of GUSI
with Carbon adaptations, so it may be better to check the <A
HREF="http://www.cwi.nl/~jack/macpython.html">MacPython homepage</A>
for a GUSI that is most easily used for building Python. 

</UL>

<A NAME="optional">The MacPython project files are configured to
include a plethora of optional modules</A>, and these modules need a
number of extra packages. To use the project files as-is you have to
download these packages too. Python has all such modules as
dynamically loaded modules, so if you don't need a certain package it
suffices to just refrain from builing the extension module.
Here are the locations for the various things
you need:

<UL>

<LI> Waste, a TextEdit replacement written by Marco Piovanelli, <A
HREF="mailto:piovanel@kagi.com">&lt;piovanel@kagi.com&gt;</A>.  Python
was built using version 2.0, which is included in the CodeWarrior
package. You can also obtain it from <A
HREF="http://www.merzwaren.com/waste">&lt;http://www.merzwaren.com/waste&gt;</A>
and various other places.

<LI> Gdbm library for the Mac. Available from Jack's Mac software page at
<A HREF="http://www.cwi.nl/~jack/macsoftware.html">
http://www.cwi.nl/~jack/macsoftware.html</A> and <A HREF="ftp://ftp.cwi.nl/pub/jack/mac">
ftp://ftp.cwi.nl/pub/jack/mac</A>. 

<LI> JPEG library by the Independent JPEG Group. A version including
Mac projects can be found at Jack's page mentioned above. 
The most recent JPEG library can always be obtained from <A
HREF="ftp://ftp.uu.net/graphics/jpeg/">ftp://ftp.uu.net/graphics/jpeg/</A>. 

<LI> The netpbm/pbmplus, libtiff, zlib and png libraries. The netpbm distribution
(which includes libtiff) is generally available on Internet ftp
servers. For Python pbmplus, an older incarnation of netpbm, is
functionally identical to netpbm, since Python only uses the library
and not the complete applications. A distribution with correct
projects and library source only is available from, you guessed it, Jack's Mac software
page mentioned above. 

</UL>

<H2>Setting Up</H2>

Now that you have collected everything you should start with building
the various parts.  If you don't want to fix
access paths try to set things up as follows:

<PRE>
Top-level-folder:
	GUSI2
	imglibs
		jpeg
		netpbm
			libtiff
		zlib
		png
	gdbm
	Python
		Modules
		...
		Mac
			Modules
			Build
			...
</PRE>

If your setup of the libraries is exactly the same as mine (which is
not very likely, unless you happen to work from the same CVS
repository) you can use the project <code>buildlibs.prj</code> in the
<code>:Mac:Build</code> folder to build all needed libraries in one
fell swoop, otherwise you will have to build the libraries one by
one. <p>

First build GUSI, the Carbon variant.
 <p>

Next, in
<code>libjpeg</code>, <code>pbmplus</code>,
<code>zlib</code>, <code>libpng</code>, <code>gdbm</code>,
and<code>libtiff</code> you build all projects. Usually the projects are in "mac"
subfolders, sometimes they are in the main folder. Tcl/tk is a special
case, see below.

<H2>The organization of the Python source tree</H2>

Time for a short break, while we have a look at the organization of
the Python source tree.  At the top level, we find the following
folders:

<DL>
<DT> Demo
<DD> Demo programs that are not Mac-specific. Some of these may not
work.

<DT> Extensions
<DD> Extensions to the interpreter that are not Mac-specific. Contains
the <code>img</code>, <code>Imaging</code> and <code>Numerical</code> extensions
in this distribution.

<DT> Grammar
<DD> The Python grammar. Included for reference only, you cannot build
the parser on a Mac.

<DT> Include
<DD> Machine-independent header files.

<DT> Modules
<DD> Machine-independent optional modules. Not all of these will work
on the Mac.

<DT> Lib
<DD> Machine-independent modules in Python.

<DT> Lib:lib-dynload
<DD> This is where the dynamically-loaded plugin modules live.

<DT> Lib:plat-mac
<DD> This is where most of the Mac-specific modules live. The modules here
are available both in MacPython-OS9 and MacPython-OSX.

<DT> Objects
<DD> Machine-independent code for various object types. Most of these are
not really optional: the interpreter will not function without them.

<DT> Parser
<DD> The Python parser (machine-independent).

<DT> Python
<DD> The core interpreter. Most files are machine-independent, some
are unix-specific and not used on the Mac.

<DT> Tools
<DD> Tools for python developers. Contains <code>modulator</code> which
builds skeleton C extension modules, <code>bgen</code> which generates
complete interface modules from information in C header files and
<code>freeze</code> which is used to turn Python scripts into real
applications (used by MacFreeze and BuildApplication) There are some
readme files, but more documentation is sorely needed.

</DL>

The mac-specific stuff lives in the <code>Mac</code> folder:
<DL>
<DT> Build
<DD> This is where the project files live and where you build the
libraries, shared libraries, executables and plugin modules. All the
resulting binaries, except for intermedeate results, are deposited in
the toplevel folder or the :Lib:lib-dynload folder (for plugin modules).

<DT> Compat
<DD> Unix-compatability routines. Most of these are not used anymore,
since GUSI provides a rather complete emulation, but you may need
these if you are trying to build a non-GUSI python.

<DT> Demo
<DD> Mac-specific demo programs, some of them annotated.

<DT> Include
<DD> Mac-specific but compiler-independent include files. 

<DT> Lib
<DD> MacPython-OS9 specific standard modules which are not shared with
MacPython-OSX.

<DT> Modules
<DD> Mac-specific builtin modules. Theoretically these are all
optional, but some are rather essential (like
<code>macosmodule</code>). A lot of these modules are generated with
<code>bgen</code>, in which case the bgen input files are included so
you can attempt to regenerate them or extend them.

<DT> MPW
<DD> MPW-specific files. These have not been used or kept up-to-date
for a long time, so use at your own risk.

<DT> mwerks
<DD> Mwerks-specific sources and headers. Contains glue code for
Pythons shared-library architecture, a replacement for
<code>malloc</code> and a directory with various projects for building
variations on the Python interpreter. The <code>mwerks_*.h</code>
files here are the option-setting files for the various interpreters
and such, comparable to the unix command-line <code>-D</code> options
to the compiler. Each project uses the correct option file as its
"prefix file" in the "C/C++ language" settings. Disabling optional
modules (for the 68K interpreter), building non-GUSI interpreters and
various other things are accomplished by modifying these files (and
possibly changing the list of files included in the project window, of
course).

<DT> OSX
<DD> Specific to MacPython-OSX, not used by MacPython-OS9.

<DT> OSXResources
<DD> Specific to MacPython-OSX, not used by MacPython-OS9.

<DT> Python
<DD> Mac-specific parts of the core interpreter.

<DT> Resources
<DD> Resource files needed to build the interpreter.

<DT> Scripts
<DD> A collection of various mac-specific Python scripts. Some are
essential, some are useful but few are documented, so you will have to
use your imagination to work them out.

<DT> Tools
<DD> A collection of tools, usually bigger than those in the scripts
folder. The important ones here are the IDE and macfreeze. The IDE is built
with the buildIDE.py script, which puts the resulting applet in the toplevel
folder. Macfreeze is usually invoked through the BuildApplication script,
but for more control over the freezing process you can run the main script here.


<DT> Unsupported
<DD> Modules that are not supported any longer but may still work with a little effort.
</DL>

<H2>Building the PPC interpreter</H2>

First you optionally build the external libraries with buildlibs.prj. <p>

Then, the <code>fullbuild</code> script can be used to build
everything, but you need a fully-functional interpreter before you can
use it (and one that isn't rebuilt in the process: you cannot rebuild
a running program). You could copy the interpreter to a different
place and use that to run fullbuild. The <code>PythonStandSmall.prj</code>
project builds an interpreter that is suited to this, and it can also come
in handy if you need to debug things (which is easier in a static program). <p>

In case you want to build by hand, or in case the <code>fullbuild</code>
script does not work, here is a breakdown of the various projects. <p>

The projects for interpreter and core library are linked together, so
building the PythonInterpreter target
in <code>PythonInterpreter.prj</code> 
will result in the whole core being built, but not the extension modules.  <p>

You will get about 100 warnings on "missing prototype" for the various module init
routines, ignore these. You will also get numerous warnings on functions from GUSI which
override functions from MSL, ignore these too. <p>

Here is a breakdown of the projects:

<DL>

<DT> PythonCore
<DD> The shared library that contains the bulk of the interpreter and
its resources.
It is a good idea to immedeately put an alias to this
shared library in the <code>Extensions</code> folder of your system
folder.  Do exactly that: put an <em>alias</em> there, copying or
moving the file will cause you grief later if you rebuild the library and
forget to copy it to the extensions folder again. The ConfigurePythonXXX applets
will also do this. <br>

<DT> PythonInterpeter
<DD> The interpreter. This is basically a routine to call out to the
shared library. <p>

<DT> Plugin projects
<DD> Each plugin module has a separate project, and these can be rebuilt on
the fly. Fullbuild (or actually it's little helper genpluginprojects) takes
care of this.
</DL>

After creating the alias to <code>PythonCore</code> you remove any old
<code>Python XXXX Preferences</code> file from the <code>Preferences</code> folder
(if you had python installed on your system before) and run the interpreter once
to create the correct preferences file.  <p>

Next, you have to build the extension modules.
If you don't use fullbuild simply open each project and build it.
<p>

Finally, you must build the standard applets:
<code>EditPythonPrefs</code>, <code>BuildApplet</code>, etc. For the N-th time:
fullbuild does this for you, but you can also manually drag/drop them onto
BuildApplet. <p>

You are all set now, and should read the release notes and
<code>ReadMe</code> file from the <code>Mac</code> folder.

Rebuilding .exp files is no longer needed since CodeWarrior 7.

<H2><a name="cvs">Using the CVS source archive</a></H2>

It is possible (and probably best) to access the Python sources through remote CVS. The
advantage of this is that you get the very latest sources, so any bug
fixed or new features will be immedeately available. This is also the
disadvantage, of course: as this is the same tree as is used for
development it may sometimes be a little less stable. <p>

The CVS client of choice is Alexandre Parenteau's MacCVS. It can be
obtained through the <a href="http://www.wincvs.org">WinCVS
homepage</a>. MacCVS uses Internet Config to set file types correctly
based on the filename extension. In the maccvs preferences you should
also set (in the "binary files" section) "use mac encoding:
applesingle" and (in the "text files" section) "use ISO latin 1
conversion". <p>

It is a good idea to disable Quicktime Exchange in the Quicktime control
panel if you are on OS9 or before. Quicktime Exchange will magically map
some extensions to filetypes, and this can seriously hinder you if, for
instance, <code>.bmp</code> is not a Windows bitmap file. <p>

The Python sources are checked out from the main
Python CVS archive on sourceforge.net, see the <a
href="http://www.python.org/download/cvs.html">Source access via
CVS</a> page for details. When you check the sources out you will get
something like <code>Python:dist:src</code>, and under that the
<code>Modules</code>, <code>Lib</code>, <code>Mac</code> etc hierarchy. The
<code>src</code> folder can be renamed to <code>Python</code>, and
is what this document refers to as the "toplevel Python folder". <P>

The CVS repository does not contain all the projects for the plugin modules,
these are built with <code>fullbuild.py</code> normally. For this reason
it is probably a good idea to first build <code>PythonStandSmall.prj</code>,
which builds a fairly minimal interpreter, and then follow the
fullbuild instructions</a>.

<H2>Odds and ends</H2>

Some remarks that I could not fit in elsewhere:

<UL>

<LI> It may be possible to use the <code>PythonCore</code> shared
library to embed Python in another program, if your program can live
with using GUSI for I/O. Use PythonCore in stead of your MSL C library
(or, at the very least, link it before the normal C library). Ask for help
on PythonMac-SIG if you have problems with this.

<LI> <a name="extending"></a>It is possible to build PPC extension
modules without building a complete Python. The binary distribution
installer can optionally install all the needed folders (the develop
option). A template for a dynamic module can be found in
<code>xx.prj</code>.

<LI> The Python shared library architecture is a variant of the architecture
described as "application with shared libraries and dropins" in the MetroWerks
"Targeting MacOS" documentation. The Python Application and applet-template use
the <code>MSL AppRuntime.Lib</code> runtime library (with properly set CFM
initialization and termination routines). PythonCore uses <code>MSL Runtime.Lib</code>,
which is really intended for standalone programs but which we fool into working by
providing a dummy main program.
It is linked statically into PythonCore (and exported to the applications and plugins)
so we do not have to distribute yet another shared library. Plugin modules use
<code>MSL ShlibRuntime.Lib</code> (not the dropin runtime: modules are never unloaded)
and obtain the rest from PythonCore. PythonCore uses a
non-standard initialization entry point, <code>__initialize_with_resources</code>, to
be able to obtain resources from the library file later on. Plugins can do the same
(_tkinter does) or use the standard <code>__initialize</code> entry point.


</UL>
</BODY>
</HTML>