diff --git a/Doc/contents.rst b/Doc/contents.rst index e938fcd93d1..e9d1771a85f 100644 --- a/Doc/contents.rst +++ b/Doc/contents.rst @@ -11,7 +11,7 @@ library/index.rst extending/index.rst c-api/index.rst - distutils/index.rst + packaging/index.rst install/index.rst documenting/index.rst howto/index.rst diff --git a/Doc/distutils/index.rst b/Doc/distutils/index.rst index ace8280894e..5fa25a6d58a 100644 --- a/Doc/distutils/index.rst +++ b/Doc/distutils/index.rst @@ -29,3 +29,16 @@ very little overhead for build/release/install mechanics. extending.rst commandref.rst apiref.rst + +Another document describes how to install modules and extensions packaged +following the above guidelines: + +.. toctree:: + + install.rst + + +.. seealso:: + + :ref:`packaging-index` and :ref:`packaging-install-index` + Documentation of Packaging, the new version of Distutils. diff --git a/Doc/install/index.rst b/Doc/distutils/install.rst similarity index 100% rename from Doc/install/index.rst rename to Doc/distutils/install.rst diff --git a/Doc/install/install.rst b/Doc/install/install.rst new file mode 100644 index 00000000000..8067544caa2 --- /dev/null +++ b/Doc/install/install.rst @@ -0,0 +1,1029 @@ +.. highlightlang:: none + +==================================== +Installing Python projects: overwiew +==================================== + +.. _packaging_packaging-intro: + +Introduction +============ + +Although Python's extensive standard library covers many programming needs, +there often comes a time when you need to add new functionality to your Python +installation in the form of third-party modules. This might be necessary to +support your own programming, or to support an application that you want to use +and that happens to be written in Python. + +In the past, there was little support for adding third-party modules to an +existing Python installation. With the introduction of the Python Distribution +Utilities (Distutils for short) in Python 2.0, this changed. However, not all +problems were solved; end-users had to rely on ``easy_install`` or +``pip`` to download third-party modules from PyPI, uninstall distributions or do +other maintenance operations. Packaging is a more complete replacement for +Distutils, in the standard library, with a backport named Distutils2 available +for older Python versions. + +This document is aimed primarily at people who need to install third-party +Python modules: end-users and system administrators who just need to get some +Python application running, and existing Python programmers who want to add +new goodies to their toolbox. You don't need to know Python to read this +document; there will be some brief forays into using Python's interactive mode +to explore your installation, but that's it. If you're looking for information +on how to distribute your own Python modules so that others may use them, see +the :ref:`packaging-index` manual. + + +.. _packaging-trivial-install: + +Best case: trivial installation +------------------------------- + +In the best case, someone will have prepared a special version of the module +distribution you want to install that is targeted specifically at your platform +and can be installed just like any other software on your platform. For example, +the module's developer might make an executable installer available for Windows +users, an RPM package for users of RPM-based Linux systems (Red Hat, SuSE, +Mandrake, and many others), a Debian package for users of Debian and derivative +systems, and so forth. + +In that case, you would use the standard system tools to download and install +the specific installer for your platform and its dependencies. + +Of course, things will not always be that easy. You might be interested in a +module whose distribution doesn't have an easy-to-use installer for your +platform. In that case, you'll have to start with the source distribution +released by the module's author/maintainer. Installing from a source +distribution is not too hard, as long as the modules are packaged in the +standard way. The bulk of this document addresses the building and installing +of modules from standard source distributions. + + +.. _packaging-distutils: + +The Python standard: Distutils +------------------------------ + +If you download a source distribution of a module, it will be obvious whether +it was packaged and distributed using Distutils. First, the distribution's name +and version number will be featured prominently in the name of the downloaded +archive, e.g. :file:`foo-1.0.tar.gz` or :file:`widget-0.9.7.zip`. Next, the +archive will unpack into a similarly-named directory: :file:`foo-1.0` or +:file:`widget-0.9.7`. Additionally, the distribution may contain a +:file:`setup.cfg` file and a file named :file:`README.txt` ---or possibly just +:file:`README`--- explaining that building and installing the module +distribution is a simple matter of issuing the following command at your shell's +prompt:: + + python setup.py install + +Third-party projects have extended Distutils to work around its limitations or +add functionality. After some years of near-inactivity in Distutils, a new +maintainer has started to standardize good ideas in PEPs and implement them in a +new, improved version of Distutils, called Distutils2 or Packaging. + + +.. _packaging-new-standard: + +The new standard: Packaging +--------------------------- + +The rules described in the first paragraph above apply to Packaging-based +projects too: a source distribution will have a name like +:file:`widget-0.9.7.zip`. One of the main differences with Distutils is that +distributions no longer have a :file:`setup.py` script; it used to cause a +number of issues. Now there is a unique script installed with Python itself:: + + pysetup install widget-0.9.7.zip + +Running this command is enough to build and install projects (Python modules or +packages, scripts or whole applications), without even having to unpack the +archive. It is also compatible with Distutils-based distributions. + +Unless you have to perform non-standard installations or customize the build +process, you can stop reading this manual ---the above command is everything you +need to get out of it. + +With :program:`pysetup`, you won't even have to manually download a distribution +before installing it; see :ref:`packaging-pysetup`. + + +.. _packaging-standard-install: + +Standard build and install +========================== + +As described in section :ref:`packaging-new-standard`, building and installing +a module distribution using Packaging usually comes down to one simple +command:: + + pysetup run install_dist + +How you actually run this command depends on the platform and the command line +interface it provides: + +* **Unix**: Use a shell prompt. +* **Windows**: Open a command prompt ("DOS console") or use :command:`Powershell`. +* **OS X**: Open a :command:`Terminal`. + + +.. _packaging-platform-variations: + +Platform variations +------------------- + +The setup command is meant to be run from the root directory of the source +distribution, i.e. the top-level subdirectory that the module source +distribution unpacks into. For example, if you've just downloaded a module +source distribution :file:`foo-1.0.tar.gz` onto a Unix system, the normal +steps to follow are these:: + + gunzip -c foo-1.0.tar.gz | tar xf - # unpacks into directory foo-1.0 + cd foo-1.0 + pysetup run install_dist + +On Windows, you'd probably download :file:`foo-1.0.zip`. If you downloaded the +archive file to :file:`C:\\Temp`, then it would unpack into +:file:`C:\\Temp\\foo-1.0`. To actually unpack the archive, you can use either +an archive manipulator with a graphical user interface (such as WinZip or 7-Zip) +or a command-line tool (such as :program:`unzip`, :program:`pkunzip` or, again, +:program:`7z`). Then, open a command prompt window ("DOS box" or +Powershell), and run:: + + cd c:\Temp\foo-1.0 + pysetup run install_dist + + +.. _packaging-splitting-up: + +Splitting the job up +-------------------- + +Running ``pysetup run install_dist`` builds and installs all modules in one go. If you +prefer to work incrementally ---especially useful if you want to customize the +build process, or if things are going wrong--- you can use the setup script to +do one thing at a time. This is a valuable tool when different users will perform +separately the build and install steps. For example, you might want to build a +module distribution and hand it off to a system administrator for installation +(or do it yourself, but with super-user or admin privileges). + +For example, to build everything in one step and then install everything +in a second step, you aptly invoke two distinct Packaging commands:: + + pysetup run build + pysetup run install_dist + +If you do this, you will notice that invoking the :command:`install_dist` command +first runs the :command:`build` command, which ---in this case--- quickly +notices it can spare itself the work, since everything in the :file:`build` +directory is up-to-date. + +You may often ignore this ability to divide the process in steps if all you do +is installing modules downloaded from the Internet, but it's very handy for +more advanced tasks. If you find yourself in the need for distributing your own +Python modules and extensions, though, you'll most likely run many individual +Packaging commands. + + +.. _packaging-how-build-works: + +How building works +------------------ + +As implied above, the :command:`build` command is responsible for collecting +and placing the files to be installed into a *build directory*. By default, +this is :file:`build`, under the distribution root. If you're excessively +concerned with speed, or want to keep the source tree pristine, you can specify +a different build directory with the :option:`--build-base` option. For example:: + + pysetup run build --build-base /tmp/pybuild/foo-1.0 + +(Or you could do this permanently with a directive in your system or personal +Packaging configuration file; see section :ref:`packaging-config-files`.) +In the usual case, however, all this is unnecessary. + +The build tree's default layout looks like so:: + + --- build/ --- lib/ + or + --- build/ --- lib./ + temp./ + +where ```` expands to a brief description of the current OS/hardware +platform and Python version. The first form, with just a :file:`lib` directory, +is used for pure module distributions (module distributions that +include only pure Python modules). If a module distribution contains any +extensions (modules written in C/C++), then the second form, with two ```` +directories, is used. In that case, the :file:`temp.{plat}` directory holds +temporary files generated during the compile/link process which are not intended +to be installed. In either case, the :file:`lib` (or :file:`lib.{plat}`) directory +contains all Python modules (pure Python and extensions) to be installed. + +In the future, more directories will be added to handle Python scripts, +documentation, binary executables, and whatever else is required to install +Python modules and applications. + + +.. _packaging-how-install-works: + +How installation works +---------------------- + +After the :command:`build` command is run (whether explicitly or by the +:command:`install_dist` command on your behalf), the work of the :command:`install_dist` +command is relatively simple: all it has to do is copy the contents of +:file:`build/lib` (or :file:`build/lib.{plat}`) to the installation directory +of your choice. + +If you don't choose an installation directory ---i.e., if you just run +``pysetup run install_dist``\ --- then the :command:`install_dist` command +installs to the standard location for third-party Python modules. This location +varies by platform and depending on how you built/installed Python itself. On +Unix (and Mac OS X, which is also Unix-based), it also depends on whether the +module distribution being installed is pure Python or contains extensions +("non-pure"): + ++-----------------+-----------------------------------------------------+--------------------------------------------------+-------+ +| Platform | Standard installation location | Default value | Notes | ++=================+=====================================================+==================================================+=======+ +| Unix (pure) | :file:`{prefix}/lib/python{X.Y}/site-packages` | :file:`/usr/local/lib/python{X.Y}/site-packages` | \(1) | ++-----------------+-----------------------------------------------------+--------------------------------------------------+-------+ +| Unix (non-pure) | :file:`{exec-prefix}/lib/python{X.Y}/site-packages` | :file:`/usr/local/lib/python{X.Y}/site-packages` | \(1) | ++-----------------+-----------------------------------------------------+--------------------------------------------------+-------+ +| Windows | :file:`{prefix}\\Lib\\site-packages` | :file:`C:\\Python{XY}\\Lib\\site-packages` | \(2) | ++-----------------+-----------------------------------------------------+--------------------------------------------------+-------+ + +Notes: + +(1) + Most Linux distributions include Python as a standard part of the system, so + :file:`{prefix}` and :file:`{exec-prefix}` are usually both :file:`/usr` on + Linux. If you build Python yourself on Linux (or any Unix-like system), the + default :file:`{prefix}` and :file:`{exec-prefix}` are :file:`/usr/local`. + +(2) + The default installation directory on Windows was :file:`C:\\Program + Files\\Python` under Python 1.6a1, 1.5.2, and earlier. + +:file:`{prefix}` and :file:`{exec-prefix}` stand for the directories that Python +is installed to, and where it finds its libraries at run-time. They are always +the same under Windows, and very often the same under Unix and Mac OS X. You +can find out what your Python installation uses for :file:`{prefix}` and +:file:`{exec-prefix}` by running Python in interactive mode and typing a few +simple commands. + +.. TODO link to Doc/using instead of duplicating + +To start the interactive Python interpreter, you need to follow a slightly +different recipe for each platform. Under Unix, just type :command:`python` at +the shell prompt. Under Windows (assuming the Python executable is on your +:envvar:`PATH`, which is the usual case), you can choose :menuselection:`Start --> Run`, +type ``python`` and press ``enter``. Alternatively, you can simply execute +:command:`python` at a command prompt ("DOS console" or Powershell). + +Once the interpreter is started, you type Python code at the prompt. For +example, on my Linux system, I type the three Python statements shown below, +and get the output as shown, to find out my :file:`{prefix}` and :file:`{exec-prefix}`:: + + Python 3.3 (r32:88445, Apr 2 2011, 10:43:54) + Type "help", "copyright", "credits" or "license" for more information. + >>> import sys + >>> sys.prefix + '/usr' + >>> sys.exec_prefix + '/usr' + +If you don't want to install modules to the standard location, or if you don't +have permission to write there, then you need to read about alternate +installations in section :ref:`packaging-alt-install`. If you want to customize your +installation directories more heavily, see section :ref:`packaging-custom-install`. + + +.. _packaging-alt-install: + +Alternate installation +====================== + +Often, it is necessary or desirable to install modules to a location other than +the standard location for third-party Python modules. For example, on a Unix +system you might not have permission to write to the standard third-party module +directory. Or you might wish to try out a module before making it a standard +part of your local Python installation. This is especially true when upgrading +a distribution already present: you want to make sure your existing base of +scripts still works with the new version before actually upgrading. + +The Packaging :command:`install_dist` command is designed to make installing module +distributions to an alternate location simple and painless. The basic idea is +that you supply a base directory for the installation, and the +:command:`install_dist` command picks a set of directories (called an *installation +scheme*) under this base directory in which to install files. The details +differ across platforms, so read whichever of the following sections applies to +you. + + +.. _packaging-alt-install-prefix: + +Alternate installation: the home scheme +--------------------------------------- + +The idea behind the "home scheme" is that you build and maintain a personal +stash of Python modules. This scheme's name is derived from the concept of a +"home" directory on Unix, since it's not unusual for a Unix user to make their +home directory have a layout similar to :file:`/usr/` or :file:`/usr/local/`. +In spite of its name's origin, this scheme can be used by anyone, regardless +of the operating system. + +Installing a new module distribution in this way is as simple as :: + + pysetup run install_dist --home + +where you can supply any directory you like for the :option:`--home` option. On +Unix, lazy typists can just type a tilde (``~``); the :command:`install_dist` command +will expand this to your home directory:: + + pysetup run install_dist --home ~ + +The :option:`--home` option defines the base directory for the installation. +Under it, files are installed to the following directories: + ++------------------------------+---------------------------+-----------------------------+ +| Type of file | Installation Directory | Override option | ++==============================+===========================+=============================+ +| pure module distribution | :file:`{home}/lib/python` | :option:`--install-purelib` | ++------------------------------+---------------------------+-----------------------------+ +| non-pure module distribution | :file:`{home}/lib/python` | :option:`--install-platlib` | ++------------------------------+---------------------------+-----------------------------+ +| scripts | :file:`{home}/bin` | :option:`--install-scripts` | ++------------------------------+---------------------------+-----------------------------+ +| data | :file:`{home}/share` | :option:`--install-data` | ++------------------------------+---------------------------+-----------------------------+ + + +.. _packaging-alt-install-home: + +Alternate installation: Unix (the prefix scheme) +------------------------------------------------ + +The "prefix scheme" is useful when you wish to use one Python installation to +run the build command, but install modules into the third-party module directory +of a different Python installation (or something that looks like a different +Python installation). If this sounds a trifle unusual, it is ---that's why the +"home scheme" comes first. However, there are at least two known cases where the +prefix scheme will be useful. + +First, consider that many Linux distributions put Python in :file:`/usr`, rather +than the more traditional :file:`/usr/local`. This is entirely appropriate, +since in those cases Python is part of "the system" rather than a local add-on. +However, if you are installing Python modules from source, you probably want +them to go in :file:`/usr/local/lib/python2.{X}` rather than +:file:`/usr/lib/python2.{X}`. This can be done with :: + + pysetup run install_dist --prefix /usr/local + +Another possibility is a network filesystem where the name used to write to a +remote directory is different from the name used to read it: for example, the +Python interpreter accessed as :file:`/usr/local/bin/python` might search for +modules in :file:`/usr/local/lib/python2.{X}`, but those modules would have to +be installed to, say, :file:`/mnt/{@server}/export/lib/python2.{X}`. This could +be done with :: + + pysetup run install_dist --prefix=/mnt/@server/export + +In either case, the :option:`--prefix` option defines the installation base, and +the :option:`--exec-prefix` option defines the platform-specific installation +base, which is used for platform-specific files. (Currently, this just means +non-pure module distributions, but could be expanded to C libraries, binary +executables, etc.) If :option:`--exec-prefix` is not supplied, it defaults to +:option:`--prefix`. Files are installed as follows: + ++------------------------------+-----------------------------------------------------+-----------------------------+ +| Type of file | Installation Directory | Override option | ++==============================+=====================================================+=============================+ +| pure module distribution | :file:`{prefix}/lib/python{X.Y}/site-packages` | :option:`--install-purelib` | ++------------------------------+-----------------------------------------------------+-----------------------------+ +| non-pure module distribution | :file:`{exec-prefix}/lib/python{X.Y}/site-packages` | :option:`--install-platlib` | ++------------------------------+-----------------------------------------------------+-----------------------------+ +| scripts | :file:`{prefix}/bin` | :option:`--install-scripts` | ++------------------------------+-----------------------------------------------------+-----------------------------+ +| data | :file:`{prefix}/share` | :option:`--install-data` | ++------------------------------+-----------------------------------------------------+-----------------------------+ + +There is no requirement that :option:`--prefix` or :option:`--exec-prefix` +actually point to an alternate Python installation; if the directories listed +above do not already exist, they are created at installation time. + +Incidentally, the real reason the prefix scheme is important is simply that a +standard Unix installation uses the prefix scheme, but with :option:`--prefix` +and :option:`--exec-prefix` supplied by Python itself as ``sys.prefix`` and +``sys.exec_prefix``. Thus, you might think you'll never use the prefix scheme, +but every time you run ``pysetup run install_dist`` without any other +options, you're using it. + +Note that installing extensions to an alternate Python installation doesn't have +anything to do with how those extensions are built: in particular, extensions +will be compiled using the Python header files (:file:`Python.h` and friends) +installed with the Python interpreter used to run the build command. It is +therefore your responsibility to ensure compatibility between the interpreter +intended to run extensions installed in this way and the interpreter used to +build these same extensions. To avoid problems, it is best to make sure that +the two interpreters are the same version of Python (possibly different builds, +or possibly copies of the same build). (Of course, if your :option:`--prefix` +and :option:`--exec-prefix` don't even point to an alternate Python installation, +this is immaterial.) + + +.. _packaging-alt-install-windows: + +Alternate installation: Windows (the prefix scheme) +--------------------------------------------------- + +Windows has a different and vaguer notion of home directories than Unix, and +since its standard Python installation is simpler, the :option:`--prefix` option +has traditionally been used to install additional packages to arbitrary +locations. :: + + pysetup run install_dist --prefix "\Temp\Python" + +to install modules to the :file:`\\Temp\\Python` directory on the current drive. + +The installation base is defined by the :option:`--prefix` option; the +:option:`--exec-prefix` option is unsupported under Windows. Files are +installed as follows: + ++------------------------------+---------------------------+-----------------------------+ +| Type of file | Installation Directory | Override option | ++==============================+===========================+=============================+ +| pure module distribution | :file:`{prefix}` | :option:`--install-purelib` | ++------------------------------+---------------------------+-----------------------------+ +| non-pure module distribution | :file:`{prefix}` | :option:`--install-platlib` | ++------------------------------+---------------------------+-----------------------------+ +| scripts | :file:`{prefix}\\Scripts` | :option:`--install-scripts` | ++------------------------------+---------------------------+-----------------------------+ +| data | :file:`{prefix}\\Data` | :option:`--install-data` | ++------------------------------+---------------------------+-----------------------------+ + + +.. _packaging-custom-install: + +Custom installation +=================== + +Sometimes, the alternate installation schemes described in section +:ref:`packaging-alt-install` just don't do what you want. You might want to tweak +just one or two directories while keeping everything under the same base +directory, or you might want to completely redefine the installation scheme. +In either case, you're creating a *custom installation scheme*. + +You probably noticed the column of "override options" in the tables describing +the alternate installation schemes above. Those options are how you define a +custom installation scheme. These override options can be relative, absolute, +or explicitly defined in terms of one of the installation base directories. +(There are two installation base directories, and they are normally the same +---they only differ when you use the Unix "prefix scheme" and supply different +:option:`--prefix` and :option:`--exec-prefix` options.) + +For example, say you're installing a module distribution to your home directory +under Unix, but you want scripts to go in :file:`~/scripts` rather than +:file:`~/bin`. As you might expect, you can override this directory with the +:option:`--install-scripts` option and, in this case, it makes most sense to supply +a relative path, which will be interpreted relative to the installation base +directory (in our example, your home directory):: + + pysetup run install_dist --home ~ --install-scripts scripts + +Another Unix example: suppose your Python installation was built and installed +with a prefix of :file:`/usr/local/python`. Thus, in a standard installation, +scripts will wind up in :file:`/usr/local/python/bin`. If you want them in +:file:`/usr/local/bin` instead, you would supply this absolute directory for +the :option:`--install-scripts` option:: + + pysetup run install_dist --install-scripts /usr/local/bin + +This command performs an installation using the "prefix scheme", where the +prefix is whatever your Python interpreter was installed with ---in this case, +:file:`/usr/local/python`. + +If you maintain Python on Windows, you might want third-party modules to live in +a subdirectory of :file:`{prefix}`, rather than right in :file:`{prefix}` +itself. This is almost as easy as customizing the script installation directory +---you just have to remember that there are two types of modules to worry about, +pure modules and non-pure modules (i.e., modules from a non-pure distribution). +For example:: + + pysetup run install_dist --install-purelib Site --install-platlib Site + +.. XXX Nothing is installed right under prefix in windows, is it?? + +The specified installation directories are relative to :file:`{prefix}`. Of +course, you also have to ensure that these directories are in Python's module +search path, such as by putting a :file:`.pth` file in :file:`{prefix}`. See +section :ref:`packaging-search-path` to find out how to modify Python's search path. + +If you want to define an entire installation scheme, you just have to supply all +of the installation directory options. Using relative paths is recommended here. +For example, if you want to maintain all Python module-related files under +:file:`python` in your home directory, and you want a separate directory for +each platform that you use your home directory from, you might define the +following installation scheme:: + + pysetup run install_dist --home ~ \ + --install-purelib python/lib \ + --install-platlib python/'lib.$PLAT' \ + --install-scripts python/scripts \ + --install-data python/data + +or, equivalently, :: + + pysetup run install_dist --home ~/python \ + --install-purelib lib \ + --install-platlib 'lib.$PLAT' \ + --install-scripts scripts \ + --install-data data + +``$PLAT`` doesn't need to be defined as an environment variable ---it will also +be expanded by Packaging as it parses your command line options, just as it +does when parsing your configuration file(s). (More on that later.) + +Obviously, specifying the entire installation scheme every time you install a +new module distribution would be very tedious. To spare you all that work, you +can store it in a Packaging configuration file instead (see section +:ref:`packaging-config-files`), like so:: + + [install_dist] + install-base = $HOME + install-purelib = python/lib + install-platlib = python/lib.$PLAT + install-scripts = python/scripts + install-data = python/data + +or, equivalently, :: + + [install_dist] + install-base = $HOME/python + install-purelib = lib + install-platlib = lib.$PLAT + install-scripts = scripts + install-data = data + +Note that these two are *not* equivalent if you override their installation +base directory when running the setup script. For example, :: + + pysetup run install_dist --install-base /tmp + +would install pure modules to :file:`/tmp/python/lib` in the first case, and +to :file:`/tmp/lib` in the second case. (For the second case, you'd probably +want to supply an installation base of :file:`/tmp/python`.) + +You may have noticed the use of ``$HOME`` and ``$PLAT`` in the sample +configuration file. These are Packaging configuration variables, which +bear a strong resemblance to environment variables. In fact, you can use +environment variables in configuration files on platforms that have such a notion, but +Packaging additionally defines a few extra variables that may not be in your +environment, such as ``$PLAT``. Of course, on systems that don't have +environment variables, such as Mac OS 9, the configuration variables supplied by +the Packaging are the only ones you can use. See section :ref:`packaging-config-files` +for details. + +.. XXX which vars win out eventually in case of clash env or Packaging? + +.. XXX need some Windows examples---when would custom installation schemes be + needed on those platforms? + + +.. XXX Move this section to Doc/using + +.. _packaging-search-path: + +Modifying Python's search path +------------------------------ + +When the Python interpreter executes an :keyword:`import` statement, it searches +for both Python code and extension modules along a search path. A default value +for this path is configured into the Python binary when the interpreter is built. +You can obtain the search path by importing the :mod:`sys` module and printing +the value of ``sys.path``. :: + + $ python + Python 2.2 (#11, Oct 3 2002, 13:31:27) + [GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-112)] on linux2 + Type "help", "copyright", "credits" or "license" for more information. + >>> import sys + >>> sys.path + ['', '/usr/local/lib/python2.3', '/usr/local/lib/python2.3/plat-linux2', + '/usr/local/lib/python2.3/lib-tk', '/usr/local/lib/python2.3/lib-dynload', + '/usr/local/lib/python2.3/site-packages'] + >>> + +The null string in ``sys.path`` represents the current working directory. + +The expected convention for locally installed packages is to put them in the +:file:`{...}/site-packages/` directory, but you may want to choose a different +location for some reason. For example, if your site kept by convention all web +server-related software under :file:`/www`. Add-on Python modules might then +belong in :file:`/www/python`, and in order to import them, this directory would +have to be added to ``sys.path``. There are several ways to solve this problem. + +The most convenient way is to add a path configuration file to a directory +that's already on Python's path, usually to the :file:`.../site-packages/` +directory. Path configuration files have an extension of :file:`.pth`, and each +line must contain a single path that will be appended to ``sys.path``. (Because +the new paths are appended to ``sys.path``, modules in the added directories +will not override standard modules. This means you can't use this mechanism for +installing fixed versions of standard modules.) + +Paths can be absolute or relative, in which case they're relative to the +directory containing the :file:`.pth` file. See the documentation of +the :mod:`site` module for more information. + +A slightly less convenient way is to edit the :file:`site.py` file in Python's +standard library, and modify ``sys.path``. :file:`site.py` is automatically +imported when the Python interpreter is executed, unless the :option:`-S` switch +is supplied to suppress this behaviour. So you could simply edit +:file:`site.py` and add two lines to it:: + + import sys + sys.path.append('/www/python/') + +However, if you reinstall the same major version of Python (perhaps when +upgrading from 3.3 to 3.3.1, for example) :file:`site.py` will be overwritten by +the stock version. You'd have to remember that it was modified and save a copy +before doing the installation. + +Alternatively, there are two environment variables that can modify ``sys.path``. +:envvar:`PYTHONHOME` sets an alternate value for the prefix of the Python +installation. For example, if :envvar:`PYTHONHOME` is set to ``/www/python``, +the search path will be set to ``['', '/www/python/lib/pythonX.Y/', +'/www/python/lib/pythonX.Y/plat-linux2', ...]``. + +The :envvar:`PYTHONPATH` variable can be set to a list of paths that will be +added to the beginning of ``sys.path``. For example, if :envvar:`PYTHONPATH` is +set to ``/www/python:/opt/py``, the search path will begin with +``['/www/python', '/opt/py']``. (Note that directories must exist in order to +be added to ``sys.path``; the :mod:`site` module removes non-existent paths.) + +Finally, ``sys.path`` is just a regular Python list, so any Python application +can modify it by adding or removing entries. + + +.. _packaging-config-files: + +Configuration files for Packaging +================================= + +As mentioned above, you can use configuration files to store personal or site +preferences for any option supported by any Packaging command. Depending on your +platform, you can use one of two or three possible configuration files. These +files will be read before parsing the command-line, so they take precedence over +default values. In turn, the command-line will override configuration files. +Lastly, if there are multiple configuration files, values from files read +earlier will be overridden by values from files read later. + +.. XXX "one of two or three possible..." seems wrong info. Below always 3 files + are indicated in the tables. + + +.. _packaging-config-filenames: + +Location and names of configuration files +----------------------------------------- + +The name and location of the configuration files vary slightly across +platforms. On Unix and Mac OS X, these are the three configuration files listed +in the order they are processed: + ++--------------+----------------------------------------------------------+-------+ +| Type of file | Location and filename | Notes | ++==============+==========================================================+=======+ +| system | :file:`{prefix}/lib/python{ver}/packaging/packaging.cfg` | \(1) | ++--------------+----------------------------------------------------------+-------+ +| personal | :file:`$HOME/.pydistutils.cfg` | \(2) | ++--------------+----------------------------------------------------------+-------+ +| local | :file:`setup.cfg` | \(3) | ++--------------+----------------------------------------------------------+-------+ + +Similarly, the configuration files on Windows ---also listed in the order they +are processed--- are these: + ++--------------+-------------------------------------------------+-------+ +| Type of file | Location and filename | Notes | ++==============+=================================================+=======+ +| system | :file:`{prefix}\\Lib\\packaging\\packaging.cfg` | \(4) | ++--------------+-------------------------------------------------+-------+ +| personal | :file:`%HOME%\\pydistutils.cfg` | \(5) | ++--------------+-------------------------------------------------+-------+ +| local | :file:`setup.cfg` | \(3) | ++--------------+-------------------------------------------------+-------+ + +On all platforms, the *personal* file can be temporarily disabled by +means of the `--no-user-cfg` option. + +Notes: + +(1) + Strictly speaking, the system-wide configuration file lives in the directory + where Packaging is installed. + +(2) + On Unix, if the :envvar:`HOME` environment variable is not defined, the + user's home directory will be determined with the :func:`getpwuid` function + from the standard :mod:`pwd` module. Packaging uses the + :func:`os.path.expanduser` function to do this. + +(3) + I.e., in the current directory (usually the location of the setup script). + +(4) + (See also note (1).) Python's default installation prefix is + :file:`C:\\Python`, so the system configuration file is normally + :file:`C:\\Python\\Lib\\packaging\\packaging.cfg`. + +(5) + On Windows, if the :envvar:`HOME` environment variable is not defined, + :envvar:`USERPROFILE` then :envvar:`HOMEDRIVE` and :envvar:`HOMEPATH` will + be tried. Packaging uses the :func:`os.path.expanduser` function to do this. + + +.. _packaging-config-syntax: + +Syntax of configuration files +----------------------------- + +All Packaging configuration files share the same syntax. Options defined in +them are grouped into sections, and each Packaging command gets its own section. +Additionally, there's a ``global`` section for options that affect every command. +Sections consist of one or more lines containing a single option specified as +``option = value``. + +For example, here's a complete configuration file that forces all commands to +run quietly by default:: + + [global] + verbose = 0 + +If this was the system configuration file, it would affect all processing +of any Python module distribution by any user on the current system. If it was +installed as your personal configuration file (on systems that support them), +it would affect only module distributions processed by you. Lastly, if it was +used as the :file:`setup.cfg` for a particular module distribution, it would +affect that distribution only. + +.. XXX "(on systems that support them)" seems wrong info + +If you wanted to, you could override the default "build base" directory and +make the :command:`build\*` commands always forcibly rebuild all files with +the following:: + + [build] + build-base = blib + force = 1 + +which corresponds to the command-line arguments:: + + pysetup run build --build-base blib --force + +except that including the :command:`build` command on the command-line means +that command will be run. Including a particular command in configuration files +has no such implication; it only means that if the command is run, the options +for it in the configuration file will apply. (This is also true if you run +other commands that derive values from it.) + +You can find out the complete list of options for any command using the +:option:`--help` option, e.g.:: + + pysetup run build --help + +and you can find out the complete list of global options by using +:option:`--help` without a command:: + + pysetup run --help + +See also the "Reference" section of the "Distributing Python Modules" manual. + +.. XXX no links to the relevant section exist. + + +.. _packaging-building-ext: + +Building extensions: tips and tricks +==================================== + +Whenever possible, Packaging tries to use the configuration information made +available by the Python interpreter used to run `pysetup`. +For example, the same compiler and linker flags used to compile Python will also +be used for compiling extensions. Usually this will work well, but in +complicated situations this might be inappropriate. This section discusses how +to override the usual Packaging behaviour. + + +.. _packaging-tweak-flags: + +Tweaking compiler/linker flags +------------------------------ + +Compiling a Python extension written in C or C++ will sometimes require +specifying custom flags for the compiler and linker in order to use a particular +library or produce a special kind of object code. This is especially true if the +extension hasn't been tested on your platform, or if you're trying to +cross-compile Python. + +.. TODO update to new setup.cfg + +In the most general case, the extension author might have foreseen that +compiling the extensions would be complicated, and provided a :file:`Setup` file +for you to edit. This will likely only be done if the module distribution +contains many separate extension modules, or if they often require elaborate +sets of compiler flags in order to work. + +A :file:`Setup` file, if present, is parsed in order to get a list of extensions +to build. Each line in a :file:`Setup` describes a single module. Lines have +the following structure:: + + module ... [sourcefile ...] [cpparg ...] [library ...] + + +Let's examine each of the fields in turn. + +* *module* is the name of the extension module to be built, and should be a + valid Python identifier. You can't just change this in order to rename a module + (edits to the source code would also be needed), so this should be left alone. + +* *sourcefile* is anything that's likely to be a source code file, at least + judging by the filename. Filenames ending in :file:`.c` are assumed to be + written in C, filenames ending in :file:`.C`, :file:`.cc`, and :file:`.c++` are + assumed to be C++, and filenames ending in :file:`.m` or :file:`.mm` are assumed + to be in Objective C. + +* *cpparg* is an argument for the C preprocessor, and is anything starting with + :option:`-I`, :option:`-D`, :option:`-U` or :option:`-C`. + +* *library* is anything ending in :file:`.a` or beginning with :option:`-l` or + :option:`-L`. + +If a particular platform requires a special library on your platform, you can +add it by editing the :file:`Setup` file and running ``pysetup run build``. +For example, if the module defined by the line :: + + foo foomodule.c + +must be linked with the math library :file:`libm.a` on your platform, simply add +:option:`-lm` to the line:: + + foo foomodule.c -lm + +Arbitrary switches intended for the compiler or the linker can be supplied with +the :option:`-Xcompiler` *arg* and :option:`-Xlinker` *arg* options:: + + foo foomodule.c -Xcompiler -o32 -Xlinker -shared -lm + +The next option after :option:`-Xcompiler` and :option:`-Xlinker` will be +appended to the proper command line, so in the above example the compiler will +be passed the :option:`-o32` option, and the linker will be passed +:option:`-shared`. If a compiler option requires an argument, you'll have to +supply multiple :option:`-Xcompiler` options; for example, to pass ``-x c++`` +the :file:`Setup` file would have to contain ``-Xcompiler -x -Xcompiler c++``. + +Compiler flags can also be supplied through setting the :envvar:`CFLAGS` +environment variable. If set, the contents of :envvar:`CFLAGS` will be added to +the compiler flags specified in the :file:`Setup` file. + + +.. _packaging-non-ms-compilers: + +Using non-Microsoft compilers on Windows +---------------------------------------- + +.. sectionauthor:: Rene Liebscher + + + +Borland/CodeGear C++ +^^^^^^^^^^^^^^^^^^^^ + +This subsection describes the necessary steps to use Packaging with the Borland +C++ compiler version 5.5. First you have to know that Borland's object file +format (OMF) is different from the format used by the Python version you can +download from the Python or ActiveState Web site. (Python is built with +Microsoft Visual C++, which uses COFF as the object file format.) For this +reason, you have to convert Python's library :file:`python25.lib` into the +Borland format. You can do this as follows: + +.. Should we mention that users have to create cfg-files for the compiler? +.. see also http://community.borland.com/article/0,1410,21205,00.html + +:: + + coff2omf python25.lib python25_bcpp.lib + +The :file:`coff2omf` program comes with the Borland compiler. The file +:file:`python25.lib` is in the :file:`Libs` directory of your Python +installation. If your extension uses other libraries (zlib, ...) you have to +convert them too. + +The converted files have to reside in the same directories as the normal +libraries. + +How does Packaging manage to use these libraries with their changed names? If +the extension needs a library (eg. :file:`foo`) Packaging checks first if it +finds a library with suffix :file:`_bcpp` (eg. :file:`foo_bcpp.lib`) and then +uses this library. In the case it doesn't find such a special library it uses +the default name (:file:`foo.lib`.) [#]_ + +To let Packaging compile your extension with Borland, C++ you now have to +type:: + + pysetup run build --compiler bcpp + +If you want to use the Borland C++ compiler as the default, you could specify +this in your personal or system-wide configuration file for Packaging (see +section :ref:`packaging-config-files`.) + + +.. seealso:: + + `C++Builder Compiler `_ + Information about the free C++ compiler from Borland, including links to the + download pages. + + `Creating Python Extensions Using Borland's Free Compiler `_ + Document describing how to use Borland's free command-line C++ compiler to build + Python. + + +GNU C / Cygwin / MinGW +^^^^^^^^^^^^^^^^^^^^^^ + +This section describes the necessary steps to use Packaging with the GNU C/C++ +compilers in their Cygwin and MinGW distributions. [#]_ For a Python interpreter +that was built with Cygwin, everything should work without any of these +following steps. + +Not all extensions can be built with MinGW or Cygwin, but many can. Extensions +most likely to not work are those that use C++ or depend on Microsoft Visual C +extensions. + +To let Packaging compile your extension with Cygwin, you have to type:: + + pysetup run build --compiler=cygwin + +and for Cygwin in no-cygwin mode [#]_ or for MinGW, type:: + + pysetup run build --compiler=mingw32 + +If you want to use any of these options/compilers as default, you should +consider writing it in your personal or system-wide configuration file for +Packaging (see section :ref:`packaging-config-files`.) + +Older Versions of Python and MinGW +"""""""""""""""""""""""""""""""""" +The following instructions only apply if you're using a version of Python +inferior to 2.4.1 with a MinGW inferior to 3.0.0 (with +:file:`binutils-2.13.90-20030111-1`). + +These compilers require some special libraries. This task is more complex than +for Borland's C++, because there is no program to convert the library. First +you have to create a list of symbols which the Python DLL exports. (You can find +a good program for this task at +http://www.emmestech.com/software/pexports-0.43/download_pexports.html). + +.. I don't understand what the next line means. --amk + (inclusive the references on data structures.) + +:: + + pexports python25.dll > python25.def + +The location of an installed :file:`python25.dll` will depend on the +installation options and the version and language of Windows. In a "just for +me" installation, it will appear in the root of the installation directory. In +a shared installation, it will be located in the system directory. + +Then you can create from these information an import library for gcc. :: + + /cygwin/bin/dlltool --dllname python25.dll --def python25.def --output-lib libpython25.a + +The resulting library has to be placed in the same directory as +:file:`python25.lib`. (Should be the :file:`libs` directory under your Python +installation directory.) + +If your extension uses other libraries (zlib,...) you might have to convert +them too. The converted files have to reside in the same directories as the +normal libraries do. + + +.. seealso:: + + `Building Python modules on MS Windows platform with MinGW `_ + Information about building the required libraries for the MinGW + environment. + + +.. rubric:: Footnotes + +.. [#] This also means you could replace all existing COFF-libraries with + OMF-libraries of the same name. + +.. [#] Check http://sources.redhat.com/cygwin/ and http://www.mingw.org/ for + more information. + +.. [#] Then you have no POSIX emulation available, but you also don't need + :file:`cygwin1.dll`. diff --git a/Doc/install/pysetup-config.rst b/Doc/install/pysetup-config.rst new file mode 100644 index 00000000000..0ce9022f806 --- /dev/null +++ b/Doc/install/pysetup-config.rst @@ -0,0 +1,44 @@ +.. _packaging-pysetup-config: + +===================== +Pysetup Configuration +===================== + +Pysetup supports two configuration files: :file:`.pypirc` and :file:`packaging.cfg`. + +.. FIXME integrate with configfile instead of duplicating + +Configuring indexes +------------------- + +You can configure additional indexes in :file:`.pypirc` to be used for index-related +operations. By default, all configured index-servers and package-servers will be used +in an additive fashion. To limit operations to specific indexes, use the :option:`--index` +and :option:`--package-server options`:: + + $ pysetup install --index pypi --package-server django some.project + +Adding indexes to :file:`.pypirc`:: + + [packaging] + index-servers = + pypi + other + + package-servers = + django + + [pypi] + repository: + username: + password: + + [other] + repository: + username: + password: + + [django] + repository: + username: + password: diff --git a/Doc/install/pysetup-servers.rst b/Doc/install/pysetup-servers.rst new file mode 100644 index 00000000000..ddaaa5b033b --- /dev/null +++ b/Doc/install/pysetup-servers.rst @@ -0,0 +1,61 @@ +.. _packaging-pysetup-servers: + +=============== +Package Servers +=============== + +Pysetup supports installing Python packages from *Package Servers* in addition +to PyPI indexes and mirrors. + +Package Servers are simple directory listings of Python distributions. Directories +can be served via HTTP or a local file system. This is useful when you want to +dump source distributions in a directory and not worry about the full index structure. + +Serving distributions from Apache +--------------------------------- +:: + + $ mkdir -p /var/www/html/python/distributions + $ cp *.tar.gz /var/www/html/python/distributions/ + + + ServerAdmin webmaster@domain.com + DocumentRoot "/var/www/html/python" + ServerName python.example.org + ErrorLog logs/python.example.org-error.log + CustomLog logs/python.example.org-access.log common + Options Indexes FollowSymLinks MultiViews + DirectoryIndex index.html index.htm + + + Options Indexes FollowSymLinks MultiViews + Order allow,deny + Allow from all + + + +Add the Apache based distribution server to :file:`.pypirc`:: + + [packaging] + package-servers = + apache + + [apache] + repository: http://python.example.org/distributions/ + + +Serving distributions from a file system +---------------------------------------- +:: + + $ mkdir -p /data/python/distributions + $ cp *.tar.gz /data/python/distributions/ + +Add the directory to :file:`.pypirc`:: + + [packaging] + package-servers = + local + + [local] + repository: file:///data/python/distributions/ diff --git a/Doc/install/pysetup.rst b/Doc/install/pysetup.rst new file mode 100644 index 00000000000..b88c8e1780f --- /dev/null +++ b/Doc/install/pysetup.rst @@ -0,0 +1,163 @@ +.. _packaging-pysetup: + +================ +Pysetup Tutorial +================ + +Getting started +--------------- + +Pysetup is a simple script that supports the following features: + +- install, remove, list, and verify Python packages; +- search for available packages on PyPI or any *Simple Index*; +- verify installed packages (md5sum, installed files, version). + + +Finding out what's installed +---------------------------- + +Pysetup makes it easy to find out what Python packages are installed:: + + $ pysetup search virtualenv + virtualenv 1.6 at /opt/python3.3/lib/python3.3/site-packages/virtualenv-1.6-py3.3.egg-info + + $ pysetup search --all + pyverify 0.8.1 at /opt/python3.3/lib/python3.3/site-packages/pyverify-0.8.1.dist-info + virtualenv 1.6 at /opt/python3.3/lib/python3.3/site-packages/virtualenv-1.6-py3.3.egg-info + wsgiref 0.1.2 at /opt/python3.3/lib/python3.3/wsgiref.egg-info + ... + + +Installing a distribution +------------------------- + +Pysetup can install a Python project from the following sources: + +- PyPI and Simple Indexes; +- source directories containing a valid :file:`setup.py` or :file:`setup.cfg`; +- distribution source archives (:file:`project-1.0.tar.gz`, :file:`project-1.0.zip`); +- HTTP (http://host/packages/project-1.0.tar.gz). + + +Installing from PyPI and Simple Indexes:: + + $ pysetup install project + $ pysetup install project==1.0 + +Installing from a distribution source archive:: + + $ pysetup install project-1.0.tar.gz + +Installing from a source directory containing a valid :file:`setup.py` or +:file:`setup.cfg`:: + + $ cd path/to/source/directory + $ pysetup install + + $ pysetup install path/to/source/directory + +Installing from HTTP:: + + $ pysetup install http://host/packages/project-1.0.tar.gz + + +Retrieving metadata +------------------- + +You can gather metadata from two sources, a project's source directory or an +installed distribution. The `pysetup metadata` command can retrieve one or +more metadata fields using the `-f` option and a metadata field as the +argument. :: + + $ pysetup metadata virtualenv -f version -f name + Version: + 1.6 + Name: + virtualenv + + $ pysetup metadata virtualenv --all + Metadata-Version: + 1.0 + Name: + virtualenv + Version: + 1.6 + Platform: + UNKNOWN + Summary: + Virtual Python Environment builder + ... + +.. seealso:: + + There are three metadata versions, 1.0, 1.1, and 1.2. The following PEPs + describe specifics of the field names, and their semantics and usage. 1.0 + :PEP:`241`, 1.1 :PEP:`314`, and 1.2 :PEP:`345` + + +Removing a distribution +----------------------- + +You can remove one or more installed distributions using the `pysetup remove` +command:: + + $ pysetup remove virtualenv + removing 'virtualenv': + /opt/python3.3/lib/python3.3/site-packages/virtualenv-1.6-py3.3.egg-info/dependency_links.txt + /opt/python3.3/lib/python3.3/site-packages/virtualenv-1.6-py3.3.egg-info/entry_points.txt + /opt/python3.3/lib/python3.3/site-packages/virtualenv-1.6-py3.3.egg-info/not-zip-safe + /opt/python3.3/lib/python3.3/site-packages/virtualenv-1.6-py3.3.egg-info/PKG-INFO + /opt/python3.3/lib/python3.3/site-packages/virtualenv-1.6-py3.3.egg-info/SOURCES.txt + /opt/python3.3/lib/python3.3/site-packages/virtualenv-1.6-py3.3.egg-info/top_level.txt + Proceed (y/n)? y + success: removed 6 files and 1 dirs + +The optional '-y' argument auto confirms, skipping the conformation prompt:: + + $ pysetup remove virtualenv -y + + +Getting help +------------ + +All pysetup actions take the `-h` and `--help` options which prints the commands +help string to stdout. :: + + $ pysetup remove -h + Usage: pysetup remove dist [-y] + or: pysetup remove --help + + Uninstall a Python package. + + positional arguments: + dist installed distribution name + + optional arguments: + -y auto confirm package removal + +Getting a list of all pysetup actions and global options:: + + $ pysetup --help + Usage: pysetup [options] action [action_options] + + Actions: + run: Run one or several commands + metadata: Display the metadata of a project + install: Install a project + remove: Remove a project + search: Search for a project + graph: Display a graph + create: Create a Project + + To get more help on an action, use: + + pysetup action --help + + Global options: + --verbose (-v) run verbosely (default) + --quiet (-q) run quietly (turns verbosity off) + --dry-run (-n) don't actually do anything + --help (-h) show detailed help message + --no-user-cfg ignore pydistutils.cfg in your home directory + --version Display the version diff --git a/Doc/library/depgraph-output.png b/Doc/library/depgraph-output.png new file mode 100644 index 00000000000..960bb1b5639 Binary files /dev/null and b/Doc/library/depgraph-output.png differ diff --git a/Doc/library/distutils.rst b/Doc/library/distutils.rst index 238b79dac2b..f1a0f6b915b 100644 --- a/Doc/library/distutils.rst +++ b/Doc/library/distutils.rst @@ -27,3 +27,8 @@ This package is discussed in two separate chapters: modules into an existing Python installation. You do not need to be a Python programmer to read this manual. + +.. toctree:: + :hidden: + + ../distutils/index diff --git a/Doc/library/packaging-misc.rst b/Doc/library/packaging-misc.rst new file mode 100644 index 00000000000..5e562473e2e --- /dev/null +++ b/Doc/library/packaging-misc.rst @@ -0,0 +1,27 @@ +.. temporary file for modules that don't need a dedicated file yet + +:mod:`packaging.errors` --- Packaging exceptions +================================================ + +.. module:: packaging.errors + :synopsis: Packaging exceptions. + + +Provides exceptions used by the Packaging modules. Note that Packaging modules +may raise standard exceptions; in particular, SystemExit is usually raised for +errors that are obviously the end-user's fault (e.g. bad command-line arguments). + +This module is safe to use in ``from ... import *`` mode; it only exports +symbols whose names start with ``Packaging`` and end with ``Error``. + + +:mod:`packaging.manifest` --- The Manifest class +================================================ + +.. module:: packaging.manifest + :synopsis: The Manifest class, used for poking about the file system and + building lists of files. + + +This module provides the :class:`Manifest` class, used for poking about the +filesystem and building lists of files. diff --git a/Doc/library/packaging.command.rst b/Doc/library/packaging.command.rst new file mode 100644 index 00000000000..98835c07561 --- /dev/null +++ b/Doc/library/packaging.command.rst @@ -0,0 +1,111 @@ +:mod:`packaging.command` --- Standard Packaging commands +======================================================== + +.. module:: packaging.command + :synopsis: Standard packaging commands. + + +This subpackage contains one module for each standard Packaging command, such as +:command:`build` or :command:`upload`. Each command is implemented as a +separate module, with the command name as the name of the module and of the +class defined therein. + + + +:mod:`packaging.command.cmd` --- Abstract base class for Packaging commands +=========================================================================== + +.. module:: packaging.command.cmd + :synopsis: Abstract base class for commands. + + +This module supplies the abstract base class :class:`Command`. This class is +subclassed by the modules in the packaging.command subpackage. + + +.. class:: Command(dist) + + Abstract base class for defining command classes, the "worker bees" of the + Packaging. A useful analogy for command classes is to think of them as + subroutines with local variables called *options*. The options are declared + in :meth:`initialize_options` and defined (given their final values) in + :meth:`finalize_options`, both of which must be defined by every command + class. The distinction between the two is necessary because option values + might come from the outside world (command line, config file, ...), and any + options dependent on other options must be computed after these outside + influences have been processed --- hence :meth:`finalize_options`. The body + of the subroutine, where it does all its work based on the values of its + options, is the :meth:`run` method, which must also be implemented by every + command class. + + The class constructor takes a single argument *dist*, a + :class:`~packaging.dist.Distribution` instance. + + +Creating a new Packaging command +-------------------------------- + +This section outlines the steps to create a new Packaging command. + +.. XXX the following paragraph is focused on the stdlib; expand it to document + how to write and register a command in third-party projects + +A new command lives in a module in the :mod:`packaging.command` package. There +is a sample template in that directory called :file:`command_template`. Copy +this file to a new module with the same name as the new command you're +implementing. This module should implement a class with the same name as the +module (and the command). So, for instance, to create the command +``peel_banana`` (so that users can run ``setup.py peel_banana``), you'd copy +:file:`command_template` to :file:`packaging/command/peel_banana.py`, then edit +it so that it's implementing the class :class:`peel_banana`, a subclass of +:class:`Command`. It must define the following methods: + +.. method:: Command.initialize_options() + + Set default values for all the options that this command supports. Note that + these defaults may be overridden by other commands, by the setup script, by + config files, or by the command line. Thus, this is not the place to code + dependencies between options; generally, :meth:`initialize_options` + implementations are just a bunch of ``self.foo = None`` assignments. + + +.. method:: Command.finalize_options() + + Set final values for all the options that this command supports. This is + always called as late as possible, i.e. after any option assignments from the + command line or from other commands have been done. Thus, this is the place + to to code option dependencies: if *foo* depends on *bar*, then it is safe to + set *foo* from *bar* as long as *foo* still has the same value it was + assigned in :meth:`initialize_options`. + + +.. method:: Command.run() + + A command's raison d'etre: carry out the action it exists to perform, + controlled by the options initialized in :meth:`initialize_options`, + customized by other commands, the setup script, the command line, and config + files, and finalized in :meth:`finalize_options`. All terminal output and + filesystem interaction should be done by :meth:`run`. + + +Command classes may define this attribute: + + +.. attribute:: Command.sub_commands + + *sub_commands* formalizes the notion of a "family" of commands, + e.g. ``install_dist`` as the parent with sub-commands ``install_lib``, + ``install_headers``, etc. The parent of a family of commands defines + *sub_commands* as a class attribute; it's a list of 2-tuples ``(command_name, + predicate)``, with *command_name* a string and *predicate* a function, a + string or ``None``. *predicate* is a method of the parent command that + determines whether the corresponding command is applicable in the current + situation. (E.g. ``install_headers`` is only applicable if we have any C + header files to install.) If *predicate* is ``None``, that command is always + applicable. + + *sub_commands* is usually defined at the *end* of a class, because + predicates can be methods of the class, so they must already have been + defined. The canonical example is the :command:`install_dist` command. + +.. XXX document how to add a custom command to another one's subcommands diff --git a/Doc/library/packaging.compiler.rst b/Doc/library/packaging.compiler.rst new file mode 100644 index 00000000000..dac62632a4f --- /dev/null +++ b/Doc/library/packaging.compiler.rst @@ -0,0 +1,672 @@ +:mod:`packaging.compiler` --- Compiler classes +============================================== + +.. module:: packaging.compiler + :synopsis: Compiler classes to build C/C++ extensions or libraries. + + +This subpackage contains an abstract base class representing a compiler and +concrete implementations for common compilers. The compiler classes should not +be instantiated directly, but created using the :func:`new_compiler` factory +function. Compiler types provided by Packaging are listed in +:ref:`packaging-standard-compilers`. + + +Public functions +---------------- + +.. function:: new_compiler(plat=None, compiler=None, verbose=0, dry_run=0, force=0) + + Factory function to generate an instance of some + :class:`~.ccompiler.CCompiler` subclass for the requested platform or + compiler type. + + If no argument is given for *plat* and *compiler*, the default compiler type + for the platform (:attr:`os.name`) will be used: ``'unix'`` for Unix and + Mac OS X, ``'msvc'`` for Windows. + + If *plat* is given, it must be one of ``'posix'``, ``'darwin'`` or ``'nt'``. + An invalid value will not raise an exception but use the default compiler + type for the current platform. + + .. XXX errors should never pass silently; this behavior is particularly + harmful when a compiler type is given as first argument + + If *compiler* is given, *plat* will be ignored, allowing you to get for + example a ``'unix'`` compiler object under Windows or an ``'msvc'`` compiler + under Unix. However, not all compiler types can be instantiated on every + platform. + + +.. function:: customize_compiler(compiler) + + Do any platform-specific customization of a CCompiler instance. Mainly + needed on Unix to plug in the information that varies across Unices and is + stored in CPython's Makefile. + + +.. function:: gen_lib_options(compiler, library_dirs, runtime_library_dirs, libraries) + + Generate linker options for searching library directories and linking with + specific libraries. *libraries* and *library_dirs* are, respectively, lists + of library names (not filenames!) and search directories. Returns a list of + command-line options suitable for use with some compiler (depending on the + two format strings passed in). + + +.. function:: gen_preprocess_options(macros, include_dirs) + + Generate C preprocessor options (:option:`-D`, :option:`-U`, :option:`-I`) as + used by at least two types of compilers: the typical Unix compiler and Visual + C++. *macros* is the usual thing, a list of 1- or 2-tuples, where ``(name,)`` + means undefine (:option:`-U`) macro *name*, and ``(name, value)`` means + define (:option:`-D`) macro *name* to *value*. *include_dirs* is just a list + of directory names to be added to the header file search path (:option:`-I`). + Returns a list of command-line options suitable for either Unix compilers or + Visual C++. + + +.. function:: get_default_compiler(osname, platform) + + Determine the default compiler to use for the given platform. + + *osname* should be one of the standard Python OS names (i.e. the ones + returned by ``os.name``) and *platform* the common value returned by + ``sys.platform`` for the platform in question. + + The default values are ``os.name`` and ``sys.platform``. + + +.. function:: set_compiler(location) + + Add or change a compiler + + +.. function:: show_compilers() + + Print list of available compilers (used by the :option:`--help-compiler` + options to :command:`build`, :command:`build_ext`, :command:`build_clib`). + + +.. _packaging-standard-compilers: + +Standard compilers +------------------ + +Concrete subclasses of :class:`~.ccompiler.CCompiler` are provided in submodules +of the :mod:`packaging.compiler` package. You do not need to import them, using +:func:`new_compiler` is the public API to use. This table documents the +standard compilers; be aware that they can be replaced by other classes on your +platform. + +=============== ======================================================== ======= +name description notes +=============== ======================================================== ======= +``'unix'`` typical Unix-style command-line C compiler [#]_ +``'msvc'`` Microsoft compiler [#]_ +``'bcpp'`` Borland C++ compiler +``'cygwin'`` Cygwin compiler (Windows port of GCC) +``'mingw32'`` Mingw32 port of GCC (same as Cygwin in no-Cygwin mode) +=============== ======================================================== ======= + + +.. [#] The Unix compiler class assumes this behavior: + + * macros defined with :option:`-Dname[=value]` + + * macros undefined with :option:`-Uname` + + * include search directories specified with :option:`-Idir` + + * libraries specified with :option:`-llib` + + * library search directories specified with :option:`-Ldir` + + * compile handled by :program:`cc` (or similar) executable with + :option:`-c` option: compiles :file:`.c` to :file:`.o` + + * link static library handled by :program:`ar` command (possibly with + :program:`ranlib`) + + * link shared library handled by :program:`cc` :option:`-shared` + + +.. [#] On Windows, extension modules typically need to be compiled with the same + compiler that was used to compile CPython (for example Microsoft Visual + Studio .NET 2003 for CPython 2.4 and 2.5). The AMD64 and Itanium + binaries are created using the Platform SDK. + + Under the hood, there are actually two different subclasses of + :class:`~.ccompiler.CCompiler` defined: one is compatible with MSVC 2005 + and 2008, the other works with older versions. This should not be a + concern for regular use of the functions in this module. + + Packaging will normally choose the right compiler, linker etc. on its + own. To override this choice, the environment variables + *DISTUTILS_USE_SDK* and *MSSdk* must be both set. *MSSdk* indicates that + the current environment has been setup by the SDK's ``SetEnv.Cmd`` + script, or that the environment variables had been registered when the + SDK was installed; *DISTUTILS_USE_SDK* indicates that the user has made + an explicit choice to override the compiler selection done by Packaging. + + .. TODO document the envvars in Doc/using and the man page + + +:mod:`packaging.compiler.ccompiler` --- CCompiler base class +============================================================ + +.. module:: packaging.compiler.ccompiler + :synopsis: Abstract CCompiler class. + + +This module provides the abstract base class for the :class:`CCompiler` +classes. A :class:`CCompiler` instance can be used for all the compile and +link steps needed to build a single project. Methods are provided to set +options for the compiler --- macro definitions, include directories, link path, +libraries and the like. + +.. class:: CCompiler([verbose=0, dry_run=0, force=0]) + + The abstract base class :class:`CCompiler` defines the interface that must be + implemented by real compiler classes. The class also has some utility + methods used by several compiler classes. + + The basic idea behind a compiler abstraction class is that each instance can + be used for all the compile/link steps in building a single project. Thus, + attributes common to all of those compile and link steps --- include + directories, macros to define, libraries to link against, etc. --- are + attributes of the compiler instance. To allow for variability in how + individual files are treated, most of those attributes may be varied on a + per-compilation or per-link basis. + + The constructor for each subclass creates an instance of the Compiler object. + Flags are *verbose* (show verbose output), *dry_run* (don't actually execute + the steps) and *force* (rebuild everything, regardless of dependencies). All + of these flags default to ``0`` (off). Note that you probably don't want to + instantiate :class:`CCompiler` or one of its subclasses directly - use the + :func:`packaging.CCompiler.new_compiler` factory function instead. + + The following methods allow you to manually alter compiler options for the + instance of the Compiler class. + + + .. method:: CCompiler.add_include_dir(dir) + + Add *dir* to the list of directories that will be searched for header + files. The compiler is instructed to search directories in the order in + which they are supplied by successive calls to :meth:`add_include_dir`. + + + .. method:: CCompiler.set_include_dirs(dirs) + + Set the list of directories that will be searched to *dirs* (a list of + strings). Overrides any preceding calls to :meth:`add_include_dir`; + subsequent calls to :meth:`add_include_dir` add to the list passed to + :meth:`set_include_dirs`. This does not affect any list of standard + include directories that the compiler may search by default. + + + .. method:: CCompiler.add_library(libname) + + Add *libname* to the list of libraries that will be included in all links + driven by this compiler object. Note that *libname* should *not* be the + name of a file containing a library, but the name of the library itself: + the actual filename will be inferred by the linker, the compiler, or the + compiler class (depending on the platform). + + The linker will be instructed to link against libraries in the order they + were supplied to :meth:`add_library` and/or :meth:`set_libraries`. It is + perfectly valid to duplicate library names; the linker will be instructed + to link against libraries as many times as they are mentioned. + + + .. method:: CCompiler.set_libraries(libnames) + + Set the list of libraries to be included in all links driven by this + compiler object to *libnames* (a list of strings). This does not affect + any standard system libraries that the linker may include by default. + + + .. method:: CCompiler.add_library_dir(dir) + + Add *dir* to the list of directories that will be searched for libraries + specified to :meth:`add_library` and :meth:`set_libraries`. The linker + will be instructed to search for libraries in the order they are supplied + to :meth:`add_library_dir` and/or :meth:`set_library_dirs`. + + + .. method:: CCompiler.set_library_dirs(dirs) + + Set the list of library search directories to *dirs* (a list of strings). + This does not affect any standard library search path that the linker may + search by default. + + + .. method:: CCompiler.add_runtime_library_dir(dir) + + Add *dir* to the list of directories that will be searched for shared + libraries at runtime. + + + .. method:: CCompiler.set_runtime_library_dirs(dirs) + + Set the list of directories to search for shared libraries at runtime to + *dirs* (a list of strings). This does not affect any standard search path + that the runtime linker may search by default. + + + .. method:: CCompiler.define_macro(name[, value=None]) + + Define a preprocessor macro for all compilations driven by this compiler + object. The optional parameter *value* should be a string; if it is not + supplied, then the macro will be defined without an explicit value and the + exact outcome depends on the compiler used (XXX true? does ANSI say + anything about this?) + + + .. method:: CCompiler.undefine_macro(name) + + Undefine a preprocessor macro for all compilations driven by this compiler + object. If the same macro is defined by :meth:`define_macro` and + undefined by :meth:`undefine_macro` the last call takes precedence + (including multiple redefinitions or undefinitions). If the macro is + redefined/undefined on a per-compilation basis (i.e. in the call to + :meth:`compile`), then that takes precedence. + + + .. method:: CCompiler.add_link_object(object) + + Add *object* to the list of object files (or analogues, such as explicitly + named library files or the output of "resource compilers") to be included + in every link driven by this compiler object. + + + .. method:: CCompiler.set_link_objects(objects) + + Set the list of object files (or analogues) to be included in every link + to *objects*. This does not affect any standard object files that the + linker may include by default (such as system libraries). + + The following methods implement methods for autodetection of compiler + options, providing some functionality similar to GNU :program:`autoconf`. + + + .. method:: CCompiler.detect_language(sources) + + Detect the language of a given file, or list of files. Uses the instance + attributes :attr:`language_map` (a dictionary), and :attr:`language_order` + (a list) to do the job. + + + .. method:: CCompiler.find_library_file(dirs, lib[, debug=0]) + + Search the specified list of directories for a static or shared library file + *lib* and return the full path to that file. If *debug* is true, look for a + debugging version (if that makes sense on the current platform). Return + ``None`` if *lib* wasn't found in any of the specified directories. + + + .. method:: CCompiler.has_function(funcname [, includes=None, include_dirs=None, libraries=None, library_dirs=None]) + + Return a boolean indicating whether *funcname* is supported on the current + platform. The optional arguments can be used to augment the compilation + environment by providing additional include files and paths and libraries and + paths. + + + .. method:: CCompiler.library_dir_option(dir) + + Return the compiler option to add *dir* to the list of directories searched for + libraries. + + + .. method:: CCompiler.library_option(lib) + + Return the compiler option to add *dir* to the list of libraries linked into the + shared library or executable. + + + .. method:: CCompiler.runtime_library_dir_option(dir) + + Return the compiler option to add *dir* to the list of directories searched for + runtime libraries. + + + .. method:: CCompiler.set_executables(**args) + + Define the executables (and options for them) that will be run to perform the + various stages of compilation. The exact set of executables that may be + specified here depends on the compiler class (via the 'executables' class + attribute), but most will have: + + +--------------+------------------------------------------+ + | attribute | description | + +==============+==========================================+ + | *compiler* | the C/C++ compiler | + +--------------+------------------------------------------+ + | *linker_so* | linker used to create shared objects and | + | | libraries | + +--------------+------------------------------------------+ + | *linker_exe* | linker used to create binary executables | + +--------------+------------------------------------------+ + | *archiver* | static library creator | + +--------------+------------------------------------------+ + + On platforms with a command line (Unix, DOS/Windows), each of these is a string + that will be split into executable name and (optional) list of arguments. + (Splitting the string is done similarly to how Unix shells operate: words are + delimited by spaces, but quotes and backslashes can override this. See + :func:`packaging.util.split_quoted`.) + + The following methods invoke stages in the build process. + + + .. method:: CCompiler.compile(sources[, output_dir=None, macros=None, include_dirs=None, debug=0, extra_preargs=None, extra_postargs=None, depends=None]) + + Compile one or more source files. Generates object files (e.g. transforms a + :file:`.c` file to a :file:`.o` file.) + + *sources* must be a list of filenames, most likely C/C++ files, but in reality + anything that can be handled by a particular compiler and compiler class (e.g. + an ``'msvc'`` compiler` can handle resource files in *sources*). Return a list of + object filenames, one per source filename in *sources*. Depending on the + implementation, not all source files will necessarily be compiled, but all + corresponding object filenames will be returned. + + If *output_dir* is given, object files will be put under it, while retaining + their original path component. That is, :file:`foo/bar.c` normally compiles to + :file:`foo/bar.o` (for a Unix implementation); if *output_dir* is *build*, then + it would compile to :file:`build/foo/bar.o`. + + *macros*, if given, must be a list of macro definitions. A macro definition is + either a ``(name, value)`` 2-tuple or a ``(name,)`` 1-tuple. The former defines + a macro; if the value is ``None``, the macro is defined without an explicit + value. The 1-tuple case undefines a macro. Later + definitions/redefinitions/undefinitions take precedence. + + *include_dirs*, if given, must be a list of strings, the directories to add to + the default include file search path for this compilation only. + + *debug* is a boolean; if true, the compiler will be instructed to output debug + symbols in (or alongside) the object file(s). + + *extra_preargs* and *extra_postargs* are implementation-dependent. On platforms + that have the notion of a command line (e.g. Unix, DOS/Windows), they are most + likely lists of strings: extra command-line arguments to prepend/append to the + compiler command line. On other platforms, consult the implementation class + documentation. In any event, they are intended as an escape hatch for those + occasions when the abstract compiler framework doesn't cut the mustard. + + *depends*, if given, is a list of filenames that all targets depend on. If a + source file is older than any file in depends, then the source file will be + recompiled. This supports dependency tracking, but only at a coarse + granularity. + + Raises :exc:`CompileError` on failure. + + + .. method:: CCompiler.create_static_lib(objects, output_libname[, output_dir=None, debug=0, target_lang=None]) + + Link a bunch of stuff together to create a static library file. The "bunch of + stuff" consists of the list of object files supplied as *objects*, the extra + object files supplied to :meth:`add_link_object` and/or + :meth:`set_link_objects`, the libraries supplied to :meth:`add_library` and/or + :meth:`set_libraries`, and the libraries supplied as *libraries* (if any). + + *output_libname* should be a library name, not a filename; the filename will be + inferred from the library name. *output_dir* is the directory where the library + file will be put. XXX defaults to what? + + *debug* is a boolean; if true, debugging information will be included in the + library (note that on most platforms, it is the compile step where this matters: + the *debug* flag is included here just for consistency). + + *target_lang* is the target language for which the given objects are being + compiled. This allows specific linkage time treatment of certain languages. + + Raises :exc:`LibError` on failure. + + + .. method:: CCompiler.link(target_desc, objects, output_filename[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None]) + + Link a bunch of stuff together to create an executable or shared library file. + + The "bunch of stuff" consists of the list of object files supplied as *objects*. + *output_filename* should be a filename. If *output_dir* is supplied, + *output_filename* is relative to it (i.e. *output_filename* can provide + directory components if needed). + + *libraries* is a list of libraries to link against. These are library names, + not filenames, since they're translated into filenames in a platform-specific + way (e.g. *foo* becomes :file:`libfoo.a` on Unix and :file:`foo.lib` on + DOS/Windows). However, they can include a directory component, which means the + linker will look in that specific directory rather than searching all the normal + locations. + + *library_dirs*, if supplied, should be a list of directories to search for + libraries that were specified as bare library names (i.e. no directory + component). These are on top of the system default and those supplied to + :meth:`add_library_dir` and/or :meth:`set_library_dirs`. *runtime_library_dirs* + is a list of directories that will be embedded into the shared library and used + to search for other shared libraries that \*it\* depends on at run-time. (This + may only be relevant on Unix.) + + *export_symbols* is a list of symbols that the shared library will export. + (This appears to be relevant only on Windows.) + + *debug* is as for :meth:`compile` and :meth:`create_static_lib`, with the + slight distinction that it actually matters on most platforms (as opposed to + :meth:`create_static_lib`, which includes a *debug* flag mostly for form's + sake). + + *extra_preargs* and *extra_postargs* are as for :meth:`compile` (except of + course that they supply command-line arguments for the particular linker being + used). + + *target_lang* is the target language for which the given objects are being + compiled. This allows specific linkage time treatment of certain languages. + + Raises :exc:`LinkError` on failure. + + + .. method:: CCompiler.link_executable(objects, output_progname[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, debug=0, extra_preargs=None, extra_postargs=None, target_lang=None]) + + Link an executable. *output_progname* is the name of the file executable, while + *objects* are a list of object filenames to link in. Other arguments are as for + the :meth:`link` method. + + + .. method:: CCompiler.link_shared_lib(objects, output_libname[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None]) + + Link a shared library. *output_libname* is the name of the output library, + while *objects* is a list of object filenames to link in. Other arguments are + as for the :meth:`link` method. + + + .. method:: CCompiler.link_shared_object(objects, output_filename[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None]) + + Link a shared object. *output_filename* is the name of the shared object that + will be created, while *objects* is a list of object filenames to link in. + Other arguments are as for the :meth:`link` method. + + + .. method:: CCompiler.preprocess(source[, output_file=None, macros=None, include_dirs=None, extra_preargs=None, extra_postargs=None]) + + Preprocess a single C/C++ source file, named in *source*. Output will be written + to file named *output_file*, or *stdout* if *output_file* not supplied. + *macros* is a list of macro definitions as for :meth:`compile`, which will + augment the macros set with :meth:`define_macro` and :meth:`undefine_macro`. + *include_dirs* is a list of directory names that will be added to the default + list, in the same way as :meth:`add_include_dir`. + + Raises :exc:`PreprocessError` on failure. + + The following utility methods are defined by the :class:`CCompiler` class, for + use by the various concrete subclasses. + + + .. method:: CCompiler.executable_filename(basename[, strip_dir=0, output_dir='']) + + Returns the filename of the executable for the given *basename*. Typically for + non-Windows platforms this is the same as the basename, while Windows will get + a :file:`.exe` added. + + + .. method:: CCompiler.library_filename(libname[, lib_type='static', strip_dir=0, output_dir='']) + + Returns the filename for the given library name on the current platform. On Unix + a library with *lib_type* of ``'static'`` will typically be of the form + :file:`liblibname.a`, while a *lib_type* of ``'dynamic'`` will be of the form + :file:`liblibname.so`. + + + .. method:: CCompiler.object_filenames(source_filenames[, strip_dir=0, output_dir='']) + + Returns the name of the object files for the given source files. + *source_filenames* should be a list of filenames. + + + .. method:: CCompiler.shared_object_filename(basename[, strip_dir=0, output_dir='']) + + Returns the name of a shared object file for the given file name *basename*. + + + .. method:: CCompiler.execute(func, args[, msg=None, level=1]) + + Invokes :func:`packaging.util.execute` This method invokes a Python function + *func* with the given arguments *args*, after logging and taking into account + the *dry_run* flag. XXX see also. + + + .. method:: CCompiler.spawn(cmd) + + Invokes :func:`packaging.util.spawn`. This invokes an external process to run + the given command. XXX see also. + + + .. method:: CCompiler.mkpath(name[, mode=511]) + + Invokes :func:`packaging.dir_util.mkpath`. This creates a directory and any + missing ancestor directories. XXX see also. + + + .. method:: CCompiler.move_file(src, dst) + + Invokes :meth:`packaging.file_util.move_file`. Renames *src* to *dst*. XXX see + also. + + +:mod:`packaging.compiler.extension` --- The Extension class +=========================================================== + +.. module:: packaging.compiler.extension + :synopsis: Class used to represent C/C++ extension modules. + + +This module provides the :class:`Extension` class, used to represent C/C++ +extension modules. + +.. class:: Extension + + The Extension class describes a single C or C++ extension module. It accepts + the following keyword arguments in its constructor + + +------------------------+--------------------------------+---------------------------+ + | argument name | value | type | + +========================+================================+===========================+ + | *name* | the full name of the | string | + | | extension, including any | | + | | packages --- i.e. *not* a | | + | | filename or pathname, but | | + | | Python dotted name | | + +------------------------+--------------------------------+---------------------------+ + | *sources* | list of source filenames, | string | + | | relative to the distribution | | + | | root (where the setup script | | + | | lives), in Unix form (slash- | | + | | separated) for portability. | | + | | Source files may be C, C++, | | + | | SWIG (.i), platform-specific | | + | | resource files, or whatever | | + | | else is recognized by the | | + | | :command:`build_ext` command | | + | | as source for a Python | | + | | extension. | | + +------------------------+--------------------------------+---------------------------+ + | *include_dirs* | list of directories to search | string | + | | for C/C++ header files (in | | + | | Unix form for portability) | | + +------------------------+--------------------------------+---------------------------+ + | *define_macros* | list of macros to define; each | (string, string) tuple or | + | | macro is defined using a | (name, ``None``) | + | | 2-tuple ``(name, value)``, | | + | | where *value* is | | + | | either the string to define it | | + | | to or ``None`` to define it | | + | | without a particular value | | + | | (equivalent of ``#define FOO`` | | + | | in source or :option:`-DFOO` | | + | | on Unix C compiler command | | + | | line) | | + +------------------------+--------------------------------+---------------------------+ + | *undef_macros* | list of macros to undefine | string | + | | explicitly | | + +------------------------+--------------------------------+---------------------------+ + | *library_dirs* | list of directories to search | string | + | | for C/C++ libraries at link | | + | | time | | + +------------------------+--------------------------------+---------------------------+ + | *libraries* | list of library names (not | string | + | | filenames or paths) to link | | + | | against | | + +------------------------+--------------------------------+---------------------------+ + | *runtime_library_dirs* | list of directories to search | string | + | | for C/C++ libraries at run | | + | | time (for shared extensions, | | + | | this is when the extension is | | + | | loaded) | | + +------------------------+--------------------------------+---------------------------+ + | *extra_objects* | list of extra files to link | string | + | | with (e.g. object files not | | + | | implied by 'sources', static | | + | | library that must be | | + | | explicitly specified, binary | | + | | resource files, etc.) | | + +------------------------+--------------------------------+---------------------------+ + | *extra_compile_args* | any extra platform- and | string | + | | compiler-specific information | | + | | to use when compiling the | | + | | source files in 'sources'. For | | + | | platforms and compilers where | | + | | a command line makes sense, | | + | | this is typically a list of | | + | | command-line arguments, but | | + | | for other platforms it could | | + | | be anything. | | + +------------------------+--------------------------------+---------------------------+ + | *extra_link_args* | any extra platform- and | string | + | | compiler-specific information | | + | | to use when linking object | | + | | files together to create the | | + | | extension (or to create a new | | + | | static Python interpreter). | | + | | Similar interpretation as for | | + | | 'extra_compile_args'. | | + +------------------------+--------------------------------+---------------------------+ + | *export_symbols* | list of symbols to be exported | string | + | | from a shared extension. Not | | + | | used on all platforms, and not | | + | | generally necessary for Python | | + | | extensions, which typically | | + | | export exactly one symbol: | | + | | ``init`` + extension_name. | | + +------------------------+--------------------------------+---------------------------+ + | *depends* | list of files that the | string | + | | extension depends on | | + +------------------------+--------------------------------+---------------------------+ + | *language* | extension language (i.e. | string | + | | ``'c'``, ``'c++'``, | | + | | ``'objc'``). Will be detected | | + | | from the source extensions if | | + | | not provided. | | + +------------------------+--------------------------------+---------------------------+ diff --git a/Doc/library/packaging.database.rst b/Doc/library/packaging.database.rst new file mode 100644 index 00000000000..2a8c39f92c4 --- /dev/null +++ b/Doc/library/packaging.database.rst @@ -0,0 +1,324 @@ +:mod:`packaging.database` --- Database of installed distributions +================================================================= + +.. module:: packaging.database + :synopsis: Functions to query and manipulate installed distributions. + + +This module provides an implementation of :PEP:`376`. It was originally +intended to land in :mod:`pkgutil`, but with the inclusion of Packaging in the +standard library, it was thought best to include it in a submodule of +:mod:`packaging`, leaving :mod:`pkgutil` to deal with imports. + +Installed Python distributions are represented by instances of +:class:`Distribution`, or :class:`EggInfoDistribution` for legacy egg formats. +Most functions also provide an extra argument ``use_egg_info`` to take legacy +distributions into account. + + +Classes representing installed distributions +-------------------------------------------- + +.. class:: Distribution(path) + + Class representing an installed distribution. It is different from + :class:`packaging.dist.Distribution` which holds the list of files, the + metadata and options during the run of a Packaging command. + + Instantiate with the *path* to a ``.dist-info`` directory. Instances can be + compared and sorted. Other available methods are: + + .. XXX describe how comparison works + + .. method:: get_distinfo_file(path, binary=False) + + Return a read-only file object for a file located at + :file:`{project-version}.dist-info/path}`. *path* should be a + ``'/'``-separated path relative to the ``.dist-info`` directory or an + absolute path; if it is an absolute path and doesn't start with the path + to the :file:`.dist-info` directory, a :class:`PackagingError` is raised. + + If *binary* is ``True``, the file is opened in binary mode. + + .. method:: get_resource_path(relative_path) + + .. TODO + + .. method:: list_distinfo_files(local=False) + + Return an iterator over all files located in the :file:`.dist-info` + directory. If *local* is ``True``, each returned path is transformed into + a local absolute path, otherwise the raw value found in the :file:`RECORD` + file is returned. + + .. method:: list_installed_files(local=False) + + Iterate over the files installed with the distribution and registered in + the :file:`RECORD` file and yield a tuple ``(path, md5, size)`` for each + line. If *local* is ``True``, the returned path is transformed into a + local absolute path, otherwise the raw value is returned. + + A local absolute path is an absolute path in which occurrences of ``'/'`` + have been replaced by :data:`os.sep`. + + .. method:: uses(path) + + Check whether *path* was installed by this distribution (i.e. if the path + is present in the :file:`RECORD` file). *path* can be a local absolute + path or a relative ``'/'``-separated path. Returns a boolean. + + Available attributes: + + .. attribute:: metadata + + Instance of :class:`packaging.metadata.Metadata` filled with the contents + of the :file:`{project-version}.dist-info/METADATA` file. + + .. attribute:: name + + Shortcut for ``metadata['Name']``. + + .. attribute:: version + + Shortcut for ``metadata['Version']``. + + .. attribute:: requested + + Boolean indicating whether this distribution was requested by the user of + automatically installed as a dependency. + + +.. class:: EggInfoDistribution(path) + + Class representing a legacy distribution. It is compatible with distutils' + and setuptools' :file:`.egg-info` and :file:`.egg` files and directories. + + .. FIXME should be named EggDistribution + + Instantiate with the *path* to an egg file or directory. Instances can be + compared and sorted. Other available methods are: + + .. method:: list_installed_files(local=False) + + .. method:: uses(path) + + Available attributes: + + .. attribute:: metadata + + Instance of :class:`packaging.metadata.Metadata` filled with the contents + of the :file:`{project-version}.egg-info/PKG-INFO` or + :file:`{project-version}.egg` file. + + .. attribute:: name + + Shortcut for ``metadata['Name']``. + + .. attribute:: version + + Shortcut for ``metadata['Version']``. + + +Functions to work with the database +----------------------------------- + +.. function:: get_distribution(name, use_egg_info=False, paths=None) + + Return an instance of :class:`Distribution` or :class:`EggInfoDistribution` + for the first installed distribution matching *name*. Egg distributions are + considered only if *use_egg_info* is true; if both a dist-info and an egg + file are found, the dist-info prevails. The directories to be searched are + given in *paths*, which defaults to :data:`sys.path`. Return ``None`` if no + matching distribution is found. + + .. FIXME param should be named use_egg + + +.. function:: get_distributions(use_egg_info=False, paths=None) + + Return an iterator of :class:`Distribution` instances for all installed + distributions found in *paths* (defaults to :data:`sys.path`). If + *use_egg_info* is true, also return instances of :class:`EggInfoDistribution` + for legacy distributions found. + + +.. function:: get_file_users(path) + + Return an iterator over all distributions using *path*, a local absolute path + or a relative ``'/'``-separated path. + + .. XXX does this work with prefixes or full file path only? + + +.. function:: obsoletes_distribution(name, version=None, use_egg_info=False) + + Return an iterator over all distributions that declare they obsolete *name*. + *version* is an optional argument to match only specific releases (see + :mod:`packaging.version`). If *use_egg_info* is true, legacy egg + distributions will be considered as well. + + +.. function:: provides_distribution(name, version=None, use_egg_info=False) + + Return an iterator over all distributions that declare they provide *name*. + *version* is an optional argument to match only specific releases (see + :mod:`packaging.version`). If *use_egg_info* is true, legacy egg + distributions will be considered as well. + + +Utility functions +----------------- + +.. function:: distinfo_dirname(name, version) + + Escape *name* and *version* into a filename-safe form and return the + directory name built from them, for example + :file:`{safename}-{safeversion}.dist-info.` In *name*, runs of + non-alphanumeric characters are replaced with one ``'_'``; in *version*, + spaces become dots, and runs of other non-alphanumeric characters (except + dots) a replaced by one ``'-'``. + + .. XXX wth spaces in version numbers? + +For performance purposes, the list of distributions is being internally +cached. Caching is enabled by default, but you can control it with these +functions: + +.. function:: clear_cache() + + Clear the cache. + +.. function:: disable_cache() + + Disable the cache, without clearing it. + +.. function:: enable_cache() + + Enable the internal cache, without clearing it. + + +Examples +-------- + +Print all information about a distribution +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Given a path to a ``.dist-info`` distribution, we shall print out all +information that can be obtained using functions provided in this module:: + + import sys + import packaging.database + + path = input() + # first create the Distribution instance + try: + dist = packaging.database.Distribution(path) + except IOError: + sys.exit('No such distribution') + + print('Information about %r' % dist.name) + print() + + print('Files') + print('=====') + for path, md5, size in dist.list_installed_files(): + print('* Path: %s' % path) + print(' Hash %s, Size: %s bytes' % (md5, size)) + print() + + print('Metadata') + print('========') + for key, value in dist.metadata.items(): + print('%20s: %s' % (key, value)) + print() + + print('Extra') + print('=====') + if dist.requested: + print('* It was installed by user request') + else: + print('* It was installed as a dependency') + +If we save the script above as ``print_info.py``, we can use it to extract +information from a :file:`.dist-info` directory. By typing in the console: + +.. code-block:: sh + + $ echo /tmp/choxie/choxie-2.0.0.9.dist-info | python3 print_info.py + +we get the following output: + +.. code-block:: none + + Information about 'choxie' + + Files + ===== + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9/truffles.py + Hash 5e052db6a478d06bad9ae033e6bc08af, Size: 111 bytes + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9/choxie/chocolate.py + Hash ac56bf496d8d1d26f866235b95f31030, Size: 214 bytes + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9/choxie/__init__.py + Hash 416aab08dfa846f473129e89a7625bbc, Size: 25 bytes + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9.dist-info/INSTALLER + Hash d41d8cd98f00b204e9800998ecf8427e, Size: 0 bytes + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9.dist-info/METADATA + Hash 696a209967fef3c8b8f5a7bb10386385, Size: 225 bytes + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9.dist-info/REQUESTED + Hash d41d8cd98f00b204e9800998ecf8427e, Size: 0 bytes + * Path: ../tmp/distutils2/tests/fake_dists/choxie-2.0.0.9.dist-info/RECORD + Hash None, Size: None bytes + + Metadata + ======== + Metadata-Version: 1.2 + Name: choxie + Version: 2.0.0.9 + Platform: [] + Supported-Platform: UNKNOWN + Summary: Chocolate with a kick! + Description: UNKNOWN + Keywords: [] + Home-page: UNKNOWN + Author: UNKNOWN + Author-email: UNKNOWN + Maintainer: UNKNOWN + Maintainer-email: UNKNOWN + License: UNKNOWN + Classifier: [] + Download-URL: UNKNOWN + Obsoletes-Dist: ['truffles (<=0.8,>=0.5)', 'truffles (<=0.9,>=0.6)'] + Project-URL: [] + Provides-Dist: ['truffles (1.0)'] + Requires-Dist: ['towel-stuff (0.1)'] + Requires-Python: UNKNOWN + Requires-External: [] + + Extra + ===== + * It was installed as a dependency + + +Find out obsoleted distributions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Now, we take tackle a different problem, we are interested in finding out +which distributions have been obsoleted. This can be easily done as follows:: + + import packaging.database + + # iterate over all distributions in the system + for dist in packaging.database.get_distributions(): + name, version = dist.name, dist.version + # find out which distributions obsolete this name/version combination + replacements = packaging.database.obsoletes_distribution(name, version) + if replacements: + print('%r %s is obsoleted by' % (name, version), + ', '.join(repr(r.name) for r in replacements)) + +This is how the output might look like: + +.. code-block:: none + + 'strawberry' 0.6 is obsoleted by 'choxie' + 'grammar' 1.0a4 is obsoleted by 'towel-stuff' diff --git a/Doc/library/packaging.depgraph.rst b/Doc/library/packaging.depgraph.rst new file mode 100644 index 00000000000..c384788e9b6 --- /dev/null +++ b/Doc/library/packaging.depgraph.rst @@ -0,0 +1,199 @@ +:mod:`packaging.depgraph` --- Dependency graph builder +====================================================== + +.. module:: packaging.depgraph + :synopsis: Graph builder for dependencies between releases. + + +This module provides the means to analyse the dependencies between various +distributions and to create a graph representing these dependency relationships. +In this document, "distribution" refers to an instance of +:class:`packaging.database.Distribution` or +:class:`packaging.database.EggInfoDistribution`. + +.. XXX terminology problem with dist vs. release: dists are installed, but deps + use releases + +.. XXX explain how to use it with dists not installed: Distribution can only be + instantiated with a path, but this module is useful for remote dist too + +.. XXX functions should accept and return iterators, not lists + + +The :class:`DependencyGraph` class +---------------------------------- + +.. class:: DependencyGraph + + Represent a dependency graph between releases. The nodes are distribution + instances; the edge model dependencies. An edge from ``a`` to ``b`` means + that ``a`` depends on ``b``. + + .. method:: add_distribution(distribution) + + Add *distribution* to the graph. + + .. method:: add_edge(x, y, label=None) + + Add an edge from distribution *x* to distribution *y* with the given + *label* (string). + + .. method:: add_missing(distribution, requirement) + + Add a missing *requirement* (string) for the given *distribution*. + + .. method:: repr_node(dist, level=1) + + Print a subgraph starting from *dist*. *level* gives the depth of the + subgraph. + + Direct access to the graph nodes and edges is provided through these + attributes: + + .. attribute:: adjacency_list + + Dictionary mapping distributions to a list of ``(other, label)`` tuples + where ``other`` is a distribution and the edge is labeled with ``label`` + (i.e. the version specifier, if such was provided). + + .. attribute:: reverse_list + + Dictionary mapping distributions to a list of predecessors. This allows + efficient traversal. + + .. attribute:: missing + + Dictionary mapping distributions to a list of requirements that were not + provided by any distribution. + + +Auxiliary functions +------------------- + +.. function:: dependent_dists(dists, dist) + + Recursively generate a list of distributions from *dists* that are dependent + on *dist*. + + .. XXX what does member mean here: "dist is a member of *dists* for which we + are interested" + +.. function:: generate_graph(dists) + + Generate a :class:`DependencyGraph` from the given list of distributions. + + .. XXX make this alternate constructor a DepGraph classmethod or rename; + 'generate' can suggest it creates a file or an image, use 'make' + +.. function:: graph_to_dot(graph, f, skip_disconnected=True) + + Write a DOT output for the graph to the file-like object *f*. + + If *skip_disconnected* is true, all distributions that are not dependent on + any other distribution are skipped. + + .. XXX why is this not a DepGraph method? + + +Example Usage +------------- + +Depict all dependenciess in the system +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +First, we shall generate a graph of all the distributions on the system +and then create an image out of it using the tools provided by +`Graphviz `_:: + + from packaging.database import get_distributions + from packaging.depgraph import generate_graph + + dists = list(get_distributions()) + graph = generate_graph(dists) + +It would be interesting to print out the missing requirements. This can be done +as follows:: + + for dist, reqs in graph.missing.items(): + if reqs: + reqs = ' ,'.join(repr(req) for req in reqs) + print('Missing dependencies for %r: %s' % (dist.name, reqs)) + +Example output is: + +.. code-block:: none + + Missing dependencies for 'TurboCheetah': 'Cheetah' + Missing dependencies for 'TurboGears': 'ConfigObj', 'DecoratorTools', 'RuleDispatch' + Missing dependencies for 'jockey': 'PyKDE4.kdecore', 'PyKDE4.kdeui', 'PyQt4.QtCore', 'PyQt4.QtGui' + Missing dependencies for 'TurboKid': 'kid' + Missing dependencies for 'TurboJson: 'DecoratorTools', 'RuleDispatch' + +Now, we proceed with generating a graphical representation of the graph. First +we write it to a file, and then we generate a PNG image using the +:program:`dot` command-line tool:: + + from packaging.depgraph import graph_to_dot + with open('output.dot', 'w') as f: + # only show the interesting distributions, skipping the disconnected ones + graph_to_dot(graph, f, skip_disconnected=True) + +We can create the final picture using: + +.. code-block:: sh + + $ dot -Tpng output.dot > output.png + +An example result is: + +.. figure:: depgraph-output.png + :alt: Example PNG output from packaging.depgraph and dot + +If you want to include egg distributions as well, then the code requires only +one change, namely the line:: + + dists = list(packaging.database.get_distributions()) + +has to be replaced with:: + + dists = list(packaging.database.get_distributions(use_egg_info=True)) + +On many platforms, a richer graph is obtained because at the moment most +distributions are provided in the egg rather than the new standard +``.dist-info`` format. + +.. XXX missing image + + An example of a more involved graph for illustrative reasons can be seen + here: + + .. image:: depgraph_big.png + + +List all dependent distributions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We will list all distributions that are dependent on some given distibution. +This time, egg distributions will be considered as well:: + + import sys + from packaging.database import get_distribution, get_distributions + from packaging.depgraph import dependent_dists + + dists = list(get_distributions(use_egg_info=True)) + dist = get_distribution('bacon', use_egg_info=True) + if dist is None: + sys.exit('No such distribution in the system') + + deps = dependent_dists(dists, dist) + deps = ', '.join(repr(x.name) for x in deps) + print('Distributions depending on %r: %s' % (dist.name, deps)) + +And this is example output: + +.. with the dependency relationships as in the previous section + (depgraph_big) + +.. code-block:: none + + Distributions depending on 'bacon': 'towel-stuff', 'choxie', 'grammar' diff --git a/Doc/library/packaging.dist.rst b/Doc/library/packaging.dist.rst new file mode 100644 index 00000000000..fb05b69c706 --- /dev/null +++ b/Doc/library/packaging.dist.rst @@ -0,0 +1,102 @@ +:mod:`packaging.dist` --- The Distribution class +================================================ + +.. module:: packaging.dist + :synopsis: Core Distribution class. + + +This module provides the :class:`Distribution` class, which represents the +module distribution being built/packaged/distributed/installed. + +.. class:: Distribution(arguments) + + A :class:`Distribution` describes how to build, package, distribute and + install a Python project. + + The arguments accepted by the constructor are laid out in the following + table. Some of them will end up in a metadata object, the rest will become + data attributes of the :class:`Distribution` instance. + + .. TODO improve constructor to take a Metadata object + named params? + (i.e. Distribution(metadata, cmdclass, py_modules, etc) + .. TODO also remove obsolete(?) script_name, etc. parameters? see what + py2exe and other tools need + + +--------------------+--------------------------------+-------------------------------------------------------------+ + | argument name | value | type | + +====================+================================+=============================================================+ + | *name* | The name of the project | string | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *version* | The version number of the | See :mod:`packaging.version` | + | | release | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *summary* | A single line describing the | a string | + | | project | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *description* | Longer description of the | a string | + | | project | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *author* | The name of the project author | a string | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *author_email* | The email address of the | a string | + | | project author | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *maintainer* | The name of the current | a string | + | | maintainer, if different from | | + | | the author | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *maintainer_email* | The email address of the | | + | | current maintainer, if | | + | | different from the author | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *home_page* | A URL for the proejct | a URL | + | | (homepage) | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *download_url* | A URL to download the project | a URL | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *packages* | A list of Python packages that | a list of strings | + | | packaging will manipulate | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *py_modules* | A list of Python modules that | a list of strings | + | | packaging will manipulate | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *scripts* | A list of standalone scripts | a list of strings | + | | to be built and installed | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *ext_modules* | A list of Python extensions to | A list of instances of | + | | be built | :class:`packaging.compiler.extension.Extension` | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *classifiers* | A list of categories for the | The list of available | + | | distribution | categorizations is at | + | | | http://pypi.python.org/pypi?:action=list_classifiers. | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *distclass* | the :class:`Distribution` | A subclass of | + | | class to use | :class:`packaging.dist.Distribution` | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *script_name* | The name of the setup.py | a string | + | | script - defaults to | | + | | ``sys.argv[0]`` | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *script_args* | Arguments to supply to the | a list of strings | + | | setup script | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *options* | default options for the setup | a string | + | | script | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *license* | The license for the | a string; should be used when there is no suitable License | + | | distribution | classifier, or to specify a classifier | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *keywords* | Descriptive keywords | a list of strings; used by catalogs | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *platforms* | Platforms compatible with this | a list of strings; should be used when there is no | + | | distribution | suitable Platform classifier | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *cmdclass* | A mapping of command names to | a dictionary | + | | :class:`Command` subclasses | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *data_files* | A list of data files to | a list | + | | install | | + +--------------------+--------------------------------+-------------------------------------------------------------+ + | *package_dir* | A mapping of Python packages | a dictionary | + | | to directory names | | + +--------------------+--------------------------------+-------------------------------------------------------------+ diff --git a/Doc/library/packaging.fancy_getopt.rst b/Doc/library/packaging.fancy_getopt.rst new file mode 100644 index 00000000000..2c693411c75 --- /dev/null +++ b/Doc/library/packaging.fancy_getopt.rst @@ -0,0 +1,75 @@ +:mod:`packaging.fancy_getopt` --- Wrapper around the getopt module +================================================================== + +.. module:: packaging.fancy_getopt + :synopsis: Additional getopt functionality. + + +.. warning:: + This module is deprecated and will be replaced with :mod:`optparse`. + +This module provides a wrapper around the standard :mod:`getopt` module that +provides the following additional features: + +* short and long options are tied together + +* options have help strings, so :func:`fancy_getopt` could potentially create a + complete usage summary + +* options set attributes of a passed-in object + +* boolean options can have "negative aliases" --- e.g. if :option:`--quiet` is + the "negative alias" of :option:`--verbose`, then :option:`--quiet` on the + command line sets *verbose* to false. + +.. function:: fancy_getopt(options, negative_opt, object, args) + + Wrapper function. *options* is a list of ``(long_option, short_option, + help_string)`` 3-tuples as described in the constructor for + :class:`FancyGetopt`. *negative_opt* should be a dictionary mapping option names + to option names, both the key and value should be in the *options* list. + *object* is an object which will be used to store values (see the :meth:`getopt` + method of the :class:`FancyGetopt` class). *args* is the argument list. Will use + ``sys.argv[1:]`` if you pass ``None`` as *args*. + + +.. class:: FancyGetopt([option_table=None]) + + The option_table is a list of 3-tuples: ``(long_option, short_option, + help_string)`` + + If an option takes an argument, its *long_option* should have ``'='`` appended; + *short_option* should just be a single character, no ``':'`` in any case. + *short_option* should be ``None`` if a *long_option* doesn't have a + corresponding *short_option*. All option tuples must have long options. + +The :class:`FancyGetopt` class provides the following methods: + + +.. method:: FancyGetopt.getopt([args=None, object=None]) + + Parse command-line options in args. Store as attributes on *object*. + + If *args* is ``None`` or not supplied, uses ``sys.argv[1:]``. If *object* is + ``None`` or not supplied, creates a new :class:`OptionDummy` instance, stores + option values there, and returns a tuple ``(args, object)``. If *object* is + supplied, it is modified in place and :func:`getopt` just returns *args*; in + both cases, the returned *args* is a modified copy of the passed-in *args* list, + which is left untouched. + + .. TODO and args returned are? + + +.. method:: FancyGetopt.get_option_order() + + Returns the list of ``(option, value)`` tuples processed by the previous run of + :meth:`getopt` Raises :exc:`RuntimeError` if :meth:`getopt` hasn't been called + yet. + + +.. method:: FancyGetopt.generate_help([header=None]) + + Generate help text (a list of strings, one per suggested line of output) from + the option table for this :class:`FancyGetopt` object. + + If supplied, prints the supplied *header* at the top of the help. diff --git a/Doc/library/packaging.install.rst b/Doc/library/packaging.install.rst new file mode 100644 index 00000000000..b619a98e2eb --- /dev/null +++ b/Doc/library/packaging.install.rst @@ -0,0 +1,112 @@ +:mod:`packaging.install` --- Installation tools +=============================================== + +.. module:: packaging.install + :synopsis: Download and installation building blocks + + +Packaging provides a set of tools to deal with downloads and installation of +distributions. Their role is to download the distribution from indexes, resolve +the dependencies, and provide a safe way to install distributions. An operation +that fails will cleanly roll back, not leave half-installed distributions on the +system. Here's the basic process followed: + +#. Move all distributions that will be removed to a temporary location. + +#. Install all the distributions that will be installed in a temporary location. + +#. If the installation fails, move the saved distributions back to their + location and delete the installed distributions. + +#. Otherwise, move the installed distributions to the right location and delete + the temporary locations. + +This is a higher-level module built on :mod:`packaging.database` and +:mod:`packaging.pypi`. + + +Public functions +---------------- + +.. function:: get_infos(requirements, index=None, installed=None, \ + prefer_final=True) + + Return information about what's going to be installed and upgraded. + *requirements* is a string string containing the requirements for this + project, for example ``'FooBar 1.1'`` or ``'BarBaz (<1.2)'``. + + .. XXX are requirements comma-separated? + + If you want to use another index than the main PyPI, give its URI as *index* + argument. + + *installed* is a list of already installed distributions used to find + satisfied dependencies, obsoleted distributions and eventual conflicts. + + By default, alpha, beta and candidate versions are not picked up. Set + *prefer_final* to false to accept them too. + + The results are returned in a dictionary containing all the information + needed to perform installation of the requirements with the + :func:`install_from_infos` function: + + >>> get_install_info("FooBar (<=1.2)") + {'install': [], 'remove': [], 'conflict': []} + + .. TODO should return tuple or named tuple, not dict + .. TODO use "predicate" or "requirement" consistently in version and here + .. FIXME "info" cannot be plural in English, s/infos/info/ + + +.. function:: install(project) + + +.. function:: install_dists(dists, path, paths=None) + + Safely install all distributions provided in *dists* into *path*. *paths* is + a list of paths where already-installed distributions will be looked for to + find satisfied dependencies and conflicts (default: :data:`sys.path`). + Returns a list of installed dists. + + .. FIXME dists are instances of what? + + +.. function:: install_from_infos(install_path=None, install=[], remove=[], \ + conflicts=[], paths=None) + + Safely install and remove given distributions. This function is designed to + work with the return value of :func:`get_infos`: *install*, *remove* and + *conflicts* should be list of distributions returned by :func:`get_infos`. + If *install* is not empty, *install_path* must be given to specify the path + where the distributions should be installed. *paths* is a list of paths + where already-installed distributions will be looked for (default: + :data:`sys.path`). + + This function is a very basic installer; if *conflicts* is not empty, the + system will be in a conflicting state after the function completes. It is a + building block for more sophisticated installers with conflict resolution + systems. + + .. TODO document typical value for install_path + .. TODO document integration with default schemes, esp. user site-packages + + +.. function:: install_local_project(path) + + Install a distribution from a source directory, which must contain either a + Packaging-compliant :file:`setup.cfg` file or a legacy Distutils + :file:`setup.py` script (in which case Distutils will be used under the hood + to perform the installation). + + +.. function:: remove(project_name, paths=None, auto_confirm=True) + + Remove one distribution from the system. + + .. FIXME this is the only function using "project" instead of dist/release + +.. + Example usage + -------------- + + Get the scheme of what's gonna be installed if we install "foobar": diff --git a/Doc/library/packaging.metadata.rst b/Doc/library/packaging.metadata.rst new file mode 100644 index 00000000000..332d69df4f7 --- /dev/null +++ b/Doc/library/packaging.metadata.rst @@ -0,0 +1,122 @@ +:mod:`packaging.metadata` --- Metadata handling +=============================================== + +.. module:: packaging.metadata + :synopsis: Class holding the metadata of a release. + + +.. TODO use sphinx-autogen to generate basic doc from the docstrings + +.. class:: Metadata + + This class can read and write metadata files complying with any of the + defined versions: 1.0 (:PEP:`241`), 1.1 (:PEP:`314`) and 1.2 (:PEP:`345`). It + implements methods to parse Metadata files and write them, and a mapping + interface to its contents. + + The :PEP:`345` implementation supports the micro-language for the environment + markers, and displays warnings when versions that are supposed to be + :PEP:`386`-compliant are violating the specification. + + +Reading metadata +---------------- + +The :class:`Metadata` class can be instantiated +with the path of the metadata file, and provides a dict-like interface to the +values:: + + >>> from packaging.metadata import Metadata + >>> metadata = Metadata('PKG-INFO') + >>> metadata.keys()[:5] + ('Metadata-Version', 'Name', 'Version', 'Platform', 'Supported-Platform') + >>> metadata['Name'] + 'CLVault' + >>> metadata['Version'] + '0.5' + >>> metadata['Requires-Dist'] + ["pywin32; sys.platform == 'win32'", "Sphinx"] + + +The fields that support environment markers can be automatically ignored if +the object is instantiated using the ``platform_dependent`` option. +:class:`Metadata` will interpret in this case +the markers and will automatically remove the fields that are not compliant +with the running environment. Here's an example under Mac OS X. The win32 +dependency we saw earlier is ignored:: + + >>> from packaging.metadata import Metadata + >>> metadata = Metadata('PKG-INFO', platform_dependent=True) + >>> metadata['Requires-Dist'] + ['Sphinx'] + + +If you want to provide your own execution context, let's say to test the +metadata under a particular environment that is not the current environment, +you can provide your own values in the ``execution_context`` option, which +is the dict that may contain one or more keys of the context the micro-language +expects. + +Here's an example, simulating a win32 environment:: + + >>> from packaging.metadata import Metadata + >>> context = {'sys.platform': 'win32'} + >>> metadata = Metadata('PKG-INFO', platform_dependent=True, + ... execution_context=context) + ... + >>> metadata['Requires-Dist'] = ["pywin32; sys.platform == 'win32'", + ... "Sphinx"] + ... + >>> metadata['Requires-Dist'] + ['pywin32', 'Sphinx'] + + +Writing metadata +---------------- + +Writing metadata can be done using the ``write`` method:: + + >>> metadata.write('/to/my/PKG-INFO') + +The class will pick the best version for the metadata, depending on the values +provided. If all the values provided exist in all versions, the class will +use :attr:`PKG_INFO_PREFERRED_VERSION`. It is set by default to 1.0, the most +widespread version. + + +Conflict checking and best version +---------------------------------- + +Some fields in :PEP:`345` have to comply with the version number specification +defined in :PEP:`386`. When they don't comply, a warning is emitted:: + + >>> from packaging.metadata import Metadata + >>> metadata = Metadata() + >>> metadata['Requires-Dist'] = ['Funky (Groovie)'] + "Funky (Groovie)" is not a valid predicate + >>> metadata['Requires-Dist'] = ['Funky (1.2)'] + +See also :mod:`packaging.version`. + + +.. TODO talk about check() + + +:mod:`packaging.markers` --- Environment markers +================================================ + +.. module:: packaging.markers + :synopsis: Micro-language for environment markers + + +This is an implementation of environment markers `as defined in PEP 345 +`_. It is used +for some metadata fields. + +.. function:: interpret(marker, execution_context=None) + + Interpret a marker and return a boolean result depending on the environment. + Example: + + >>> interpret("python_version > '1.0'") + True diff --git a/Doc/library/packaging.pypi.dist.rst b/Doc/library/packaging.pypi.dist.rst new file mode 100644 index 00000000000..aaaaab7fb8d --- /dev/null +++ b/Doc/library/packaging.pypi.dist.rst @@ -0,0 +1,114 @@ +:mod:`packaging.pypi.dist` --- Classes representing query results +================================================================= + +.. module:: packaging.pypi.dist + :synopsis: Classes representing the results of queries to indexes. + + +Information coming from the indexes is held in instances of the classes defined +in this module. + +Keep in mind that each project (eg. FooBar) can have several releases +(eg. 1.1, 1.2, 1.3), and each of these releases can be provided in multiple +distributions (eg. a source distribution, a binary one, etc). + + +ReleaseInfo +----------- + +Each release has a project name, version, metadata, and related distributions. + +This information is stored in :class:`ReleaseInfo` +objects. + +.. class:: ReleaseInfo + + +DistInfo +--------- + +:class:`DistInfo` is a simple class that contains +information related to distributions; mainly the URLs where distributions +can be found. + +.. class:: DistInfo + + +ReleasesList +------------ + +The :mod:`~packaging.pypi.dist` module provides a class which works +with lists of :class:`ReleaseInfo` classes; +used to filter and order results. + +.. class:: ReleasesList + + +Example usage +------------- + +Build a list of releases and order them +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Assuming we have a list of releases:: + + >>> from packaging.pypi.dist import ReleasesList, ReleaseInfo + >>> fb10 = ReleaseInfo("FooBar", "1.0") + >>> fb11 = ReleaseInfo("FooBar", "1.1") + >>> fb11a = ReleaseInfo("FooBar", "1.1a1") + >>> ReleasesList("FooBar", [fb11, fb11a, fb10]) + >>> releases.sort_releases() + >>> releases.get_versions() + ['1.1', '1.1a1', '1.0'] + >>> releases.add_release("1.2a1") + >>> releases.get_versions() + ['1.1', '1.1a1', '1.0', '1.2a1'] + >>> releases.sort_releases() + ['1.2a1', '1.1', '1.1a1', '1.0'] + >>> releases.sort_releases(prefer_final=True) + >>> releases.get_versions() + ['1.1', '1.0', '1.2a1', '1.1a1'] + + +Add distribution related information to releases +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It's easy to add distribution information to releases:: + + >>> from packaging.pypi.dist import ReleasesList, ReleaseInfo + >>> r = ReleaseInfo("FooBar", "1.0") + >>> r.add_distribution("sdist", url="http://example.org/foobar-1.0.tar.gz") + >>> r.dists + {'sdist': FooBar 1.0 sdist} + >>> r['sdist'].url + {'url': 'http://example.org/foobar-1.0.tar.gz', 'hashname': None, 'hashval': + None, 'is_external': True} + + +Getting attributes from the dist objects +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To abstract querying information returned from the indexes, attributes and +release information can be retrieved directly from dist objects. + +For instance, if you have a release instance that does not contain the metadata +attribute, it can be fetched by using the "fetch_metadata" method:: + + >>> r = Release("FooBar", "1.1") + >>> print r.metadata + None # metadata field is actually set to "None" + >>> r.fetch_metadata() + + +.. XXX add proper roles to these constructs + + +It's possible to retrieve a project's releases (`fetch_releases`), +metadata (`fetch_metadata`) and distributions (`fetch_distributions`) using +a similar work flow. + +.. XXX what is possible? + +Internally, this is possible because while retrieving information about +projects, releases or distributions, a reference to the client used is +stored which can be accessed using the objects `_index` attribute. diff --git a/Doc/library/packaging.pypi.rst b/Doc/library/packaging.pypi.rst new file mode 100644 index 00000000000..93b61c93368 --- /dev/null +++ b/Doc/library/packaging.pypi.rst @@ -0,0 +1,53 @@ +:mod:`packaging.pypi` --- Interface to projects indexes +======================================================= + +.. module:: packaging.pypi + :synopsis: Low-level and high-level APIs to query projects indexes. + + +Packaging queries PyPI to get information about projects or download them. The +low-level facilities used internally are also part of the public API designed to +be used by other tools. + +The :mod:`packaging.pypi` package provides those facilities, which can be +used to access information about Python projects registered at indexes, the +main one being PyPI, located ad http://pypi.python.org/. + +There is two ways to retrieve data from these indexes: a screen-scraping +interface called the "simple API", and XML-RPC. The first one uses HTML pages +located under http://pypi.python.org/simple/, the second one makes XML-RPC +requests to http://pypi.python.org/pypi/. All functions and classes also work +with other indexes such as mirrors, which typically implement only the simple +interface. + +Packaging provides a class that wraps both APIs to provide full query and +download functionality: :class:`packaging.pypi.client.ClientWrapper`. If you +want more control, you can use the underlying classes +:class:`packaging.pypi.simple.Crawler` and :class:`packaging.pypi.xmlrpc.Client` +to connect to one specific interface. + + +:mod:`packaging.pypi.client` --- High-level query API +===================================================== + +.. module:: packaging.pypi.client + :synopsis: Wrapper around :mod;`packaging.pypi.xmlrpc` and + :mod:`packaging.pypi.simple` to query indexes. + + +This module provides a high-level API to query indexes and search +for releases and distributions. The aim of this module is to choose the best +way to query the API automatically, either using XML-RPC or the simple index, +with a preference toward the latter. + +.. class:: ClientWrapper + + Instances of this class will use the simple interface or XML-RPC requests to + query indexes and return :class:`packaging.pypi.dist.ReleaseInfo` and + :class:`packaging.pypi.dist.ReleasesList` objects. + + .. method:: find_projects + + .. method:: get_release + + .. method:: get_releases diff --git a/Doc/library/packaging.pypi.simple.rst b/Doc/library/packaging.pypi.simple.rst new file mode 100644 index 00000000000..ea5edca9c12 --- /dev/null +++ b/Doc/library/packaging.pypi.simple.rst @@ -0,0 +1,157 @@ +:mod:`packaging.pypi.simple` --- Crawler using the PyPI "simple" interface +========================================================================== + +.. module:: packaging.pypi.simple + :synopsis: Crawler using the screen-scraping "simple" interface to fetch info + and distributions. + + +`packaging.pypi.simple` can process Python Package Indexes and provides +useful information about distributions. It also can crawl local indexes, for +instance. + +You should use `packaging.pypi.simple` for: + + * Search distributions by name and versions. + * Process index external pages. + * Download distributions by name and versions. + +And should not be used for: + + * Things that will end up in too long index processing (like "finding all + distributions with a specific version, no matters the name") + + +API +--- + +.. class:: Crawler + + +Usage Exemples +--------------- + +To help you understand how using the `Crawler` class, here are some basic +usages. + +Request the simple index to get a specific distribution +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Supposing you want to scan an index to get a list of distributions for +the "foobar" project. You can use the "get_releases" method for that. +The get_releases method will browse the project page, and return +:class:`ReleaseInfo` objects for each found link that rely on downloads. :: + + >>> from packaging.pypi.simple import Crawler + >>> crawler = Crawler() + >>> crawler.get_releases("FooBar") + [, ] + + +Note that you also can request the client about specific versions, using version +specifiers (described in `PEP 345 +`_):: + + >>> client.get_releases("FooBar < 1.2") + [, ] + + +`get_releases` returns a list of :class:`ReleaseInfo`, but you also can get the +best distribution that fullfil your requirements, using "get_release":: + + >>> client.get_release("FooBar < 1.2") + + + +Download distributions +^^^^^^^^^^^^^^^^^^^^^^ + +As it can get the urls of distributions provided by PyPI, the `Crawler` +client also can download the distributions and put it for you in a temporary +destination:: + + >>> client.download("foobar") + /tmp/temp_dir/foobar-1.2.tar.gz + + +You also can specify the directory you want to download to:: + + >>> client.download("foobar", "/path/to/my/dir") + /path/to/my/dir/foobar-1.2.tar.gz + + +While downloading, the md5 of the archive will be checked, if not matches, it +will try another time, then if fails again, raise `MD5HashDoesNotMatchError`. + +Internally, that's not the Crawler which download the distributions, but the +`DistributionInfo` class. Please refer to this documentation for more details. + + +Following PyPI external links +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The default behavior for packaging is to *not* follow the links provided +by HTML pages in the "simple index", to find distributions related +downloads. + +It's possible to tell the PyPIClient to follow external links by setting the +`follow_externals` attribute, on instantiation or after:: + + >>> client = Crawler(follow_externals=True) + +or :: + + >>> client = Crawler() + >>> client.follow_externals = True + + +Working with external indexes, and mirrors +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The default `Crawler` behavior is to rely on the Python Package index stored +on PyPI (http://pypi.python.org/simple). + +As you can need to work with a local index, or private indexes, you can specify +it using the index_url parameter:: + + >>> client = Crawler(index_url="file://filesystem/path/") + +or :: + + >>> client = Crawler(index_url="http://some.specific.url/") + + +You also can specify mirrors to fallback on in case the first index_url you +provided doesnt respond, or not correctly. The default behavior for +`Crawler` is to use the list provided by Python.org DNS records, as +described in the :PEP:`381` about mirroring infrastructure. + +If you don't want to rely on these, you could specify the list of mirrors you +want to try by specifying the `mirrors` attribute. It's a simple iterable:: + + >>> mirrors = ["http://first.mirror","http://second.mirror"] + >>> client = Crawler(mirrors=mirrors) + + +Searching in the simple index +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It's possible to search for projects with specific names in the package index. +Assuming you want to find all projects containing the "distutils" keyword:: + + >>> c.search_projects("distutils") + [, , , , , , ] + + +You can also search the projects starting with a specific text, or ending with +that text, using a wildcard:: + + >>> c.search_projects("distutils*") + [, , ] + + >>> c.search_projects("*distutils") + [, , , , ] diff --git a/Doc/library/packaging.pypi.xmlrpc.rst b/Doc/library/packaging.pypi.xmlrpc.rst new file mode 100644 index 00000000000..0253d68089e --- /dev/null +++ b/Doc/library/packaging.pypi.xmlrpc.rst @@ -0,0 +1,143 @@ +:mod:`packaging.pypi.xmlrpc` --- Crawler using the PyPI XML-RPC interface +========================================================================= + +.. module:: packaging.pypi.xmlrpc + :synopsis: Client using XML-RPC requests to fetch info and distributions. + + +Indexes can be queried using XML-RPC calls, and Packaging provides a simple +way to interface with XML-RPC. + +You should **use** XML-RPC when: + +* Searching the index for projects **on other fields than project + names**. For instance, you can search for projects based on the + author_email field. +* Searching all the versions that have existed for a project. +* you want to retrieve METADATAs information from releases or + distributions. + + +You should **avoid using** XML-RPC method calls when: + +* Retrieving the last version of a project +* Getting the projects with a specific name and version. +* The simple index can match your needs + + +When dealing with indexes, keep in mind that the index queries will always +return you :class:`packaging.pypi.dist.ReleaseInfo` and +:class:`packaging.pypi.dist.ReleasesList` objects. + +Some methods here share common APIs with the one you can find on +:class:`packaging.pypi.simple`, internally, :class:`packaging.pypi.client` +is inherited by :class:`Client` + + +API +--- + +.. class:: Client + + +Usage examples +-------------- + +Use case described here are use case that are not common to the other clients. +If you want to see all the methods, please refer to API or to usage examples +described in :class:`packaging.pypi.client.Client` + + +Finding releases +^^^^^^^^^^^^^^^^ + +It's a common use case to search for "things" within the index. We can +basically search for projects by their name, which is the most used way for +users (eg. "give me the last version of the FooBar project"). + +This can be accomplished using the following syntax:: + + >>> client = xmlrpc.Client() + >>> client.get_release("Foobar (<= 1.3)) + + >>> client.get_releases("FooBar (<= 1.3)") + [FooBar 1.1, FooBar 1.1.1, FooBar 1.2, FooBar 1.2.1] + + +And we also can find for specific fields:: + + >>> client.search_projects(field=value) + + +You could specify the operator to use, default is "or":: + + >>> client.search_projects(field=value, operator="and") + + +The specific fields you can search are: + +* name +* version +* author +* author_email +* maintainer +* maintainer_email +* home_page +* license +* summary +* description +* keywords +* platform +* download_url + + +Getting metadata information +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +XML-RPC is a prefered way to retrieve metadata information from indexes. +It's really simple to do so:: + + >>> client = xmlrpc.Client() + >>> client.get_metadata("FooBar", "1.1") + + + +Assuming we already have a :class:`packaging.pypi.ReleaseInfo` object defined, +it's possible to pass it to the xmlrpc client to retrieve and complete its +metadata:: + + >>> foobar11 = ReleaseInfo("FooBar", "1.1") + >>> client = xmlrpc.Client() + >>> returned_release = client.get_metadata(release=foobar11) + >>> returned_release + + + +Get all the releases of a project +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To retrieve all the releases for a project, you can build them using +`get_releases`:: + + >>> client = xmlrpc.Client() + >>> client.get_releases("FooBar") + [, , ] + + +Get information about distributions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Indexes have information about projects, releases **and** distributions. +If you're not familiar with those, please refer to the documentation of +:mod:`packaging.pypi.dist`. + +It's possible to retrieve information about distributions, e.g "what are the +existing distributions for this release ? How to retrieve them ?":: + + >>> client = xmlrpc.Client() + >>> release = client.get_distributions("FooBar", "1.1") + >>> release.dists + {'sdist': , 'bdist': } + +As you see, this does not return a list of distributions, but a release, +because a release can be used like a list of distributions. diff --git a/Doc/library/packaging.rst b/Doc/library/packaging.rst new file mode 100644 index 00000000000..5bf2c06ad9d --- /dev/null +++ b/Doc/library/packaging.rst @@ -0,0 +1,78 @@ +:mod:`packaging` --- Packaging support +====================================== + +.. module:: packaging + :synopsis: Packaging system and building blocks for other packaging systems. +.. sectionauthor:: Fred L. Drake, Jr. , distutils and packaging + contributors + + +The :mod:`packaging` package provides support for building, packaging, +distributing and installing additional projects into a Python installation. +Projects may include Python modules, extension modules, packages and scripts. +:mod:`packaging` also provides building blocks for other packaging systems +that are not tied to the command system. + +This manual is the reference documentation for those standalone building +blocks and for extending Packaging. If you're looking for the user-centric +guides to install a project or package your own code, head to `See also`__. + + +Building blocks +--------------- + +.. toctree:: + :maxdepth: 2 + :numbered: + + packaging-misc + packaging.version + packaging.metadata + packaging.database + packaging.depgraph + packaging.pypi + packaging.pypi.dist + packaging.pypi.simple + packaging.pypi.xmlrpc + packaging.install + + +The command machinery +--------------------- + +.. toctree:: + :maxdepth: 2 + :numbered: + + packaging.dist + packaging.command + packaging.compiler + packaging.fancy_getopt + + +Other utilities +---------------- + +.. toctree:: + :maxdepth: 2 + :numbered: + + packaging.util + packaging.tests.pypi_server + +.. XXX missing: compat config create (dir_util) run pypi.{base,mirrors} + + +.. __: + +.. seealso:: + + :ref:`packaging-index` + The manual for developers of Python projects who want to package and + distribute them. This describes how to use :mod:`packaging` to make + projects easily found and added to an existing Python installation. + + :ref:`packaging-install-index` + A user-centered manual which includes information on adding projects + into an existing Python installation. You do not need to be a Python + programmer to read this manual. diff --git a/Doc/library/packaging.tests.pypi_server.rst b/Doc/library/packaging.tests.pypi_server.rst new file mode 100644 index 00000000000..f3b77203f4c --- /dev/null +++ b/Doc/library/packaging.tests.pypi_server.rst @@ -0,0 +1,105 @@ +:mod:`packaging.tests.pypi_server` --- PyPI mock server +======================================================= + +.. module:: packaging.tests.pypi_server + :synopsis: Mock server used to test PyPI-related modules and commands. + + +When you are testing code that works with Packaging, you might find these tools +useful. + + +The mock server +--------------- + +.. class:: PyPIServer + + PyPIServer is a class that implements an HTTP server running in a separate + thread. All it does is record the requests for further inspection. The recorded + data is available under ``requests`` attribute. The default + HTTP response can be overridden with the ``default_response_status``, + ``default_response_headers`` and ``default_response_data`` attributes. + + By default, when accessing the server with urls beginning with `/simple/`, + the server also record your requests, but will look for files under + the `/tests/pypiserver/simple/` path. + + You can tell the sever to serve static files for other paths. This could be + accomplished by using the `static_uri_paths` parameter, as below:: + + server = PyPIServer(static_uri_paths=["first_path", "second_path"]) + + + You need to create the content that will be served under the + `/tests/pypiserver/default` path. If you want to serve content from another + place, you also can specify another filesystem path (which needs to be under + `tests/pypiserver/`. This will replace the default behavior of the server, and + it will not serve content from the `default` dir :: + + server = PyPIServer(static_filesystem_paths=["path/to/your/dir"]) + + + If you just need to add some paths to the existing ones, you can do as shown, + keeping in mind that the server will always try to load paths in reverse order + (e.g here, try "another/super/path" then the default one) :: + + server = PyPIServer(test_static_path="another/super/path") + server = PyPIServer("another/super/path") + # or + server.static_filesystem_paths.append("another/super/path") + + + As a result of what, in your tests, while you need to use the PyPIServer, in + order to isolates the test cases, the best practice is to place the common files + in the `default` folder, and to create a directory for each specific test case:: + + server = PyPIServer(static_filesystem_paths = ["default", "test_pypi_server"], + static_uri_paths=["simple", "external"]) + + +Base class and decorator for tests +---------------------------------- + +.. class:: PyPIServerTestCase + + ``PyPIServerTestCase`` is a test case class with setUp and tearDown methods that + take care of a single PyPIServer instance attached as a ``pypi`` attribute on + the test class. Use it as one of the base classes in your test case:: + + + class UploadTestCase(PyPIServerTestCase): + + def test_something(self): + cmd = self.prepare_command() + cmd.ensure_finalized() + cmd.repository = self.pypi.full_address + cmd.run() + + environ, request_data = self.pypi.requests[-1] + self.assertEqual(request_data, EXPECTED_REQUEST_DATA) + + +.. decorator:: use_pypi_server + + You also can use a decorator for your tests, if you do not need the same server + instance along all you test case. So, you can specify, for each test method, + some initialisation parameters for the server. + + For this, you need to add a `server` parameter to your method, like this:: + + class SampleTestCase(TestCase): + + @use_pypi_server() + def test_something(self, server): + ... + + + The decorator will instantiate the server for you, and run and stop it just + before and after your method call. You also can pass the server initializer, + just like this:: + + class SampleTestCase(TestCase): + + @use_pypi_server("test_case_name") + def test_something(self, server): + ... diff --git a/Doc/library/packaging.util.rst b/Doc/library/packaging.util.rst new file mode 100644 index 00000000000..b95d5b528ec --- /dev/null +++ b/Doc/library/packaging.util.rst @@ -0,0 +1,186 @@ +:mod:`packaging.util` --- Miscellaneous utility functions +========================================================= + +.. module:: packaging.util + :synopsis: Miscellaneous utility functions. + + +This module contains various helpers for the other modules. + +.. XXX a number of functions are missing, but the module may be split first + (it's ginormous right now, some things could go to compat for example) + +.. function:: get_platform() + + Return a string that identifies the current platform. This is used mainly to + distinguish platform-specific build directories and platform-specific built + distributions. Typically includes the OS name and version and the + architecture (as supplied by 'os.uname()'), although the exact information + included depends on the OS; e.g. for IRIX the architecture isn't particularly + important (IRIX only runs on SGI hardware), but for Linux the kernel version + isn't particularly important. + + Examples of returned values: + + * ``linux-i586`` + * ``linux-alpha`` + * ``solaris-2.6-sun4u`` + * ``irix-5.3`` + * ``irix64-6.2`` + + For non-POSIX platforms, currently just returns ``sys.platform``. + + For Mac OS X systems the OS version reflects the minimal version on which + binaries will run (that is, the value of ``MACOSX_DEPLOYMENT_TARGET`` + during the build of Python), not the OS version of the current system. + + For universal binary builds on Mac OS X the architecture value reflects + the univeral binary status instead of the architecture of the current + processor. For 32-bit universal binaries the architecture is ``fat``, + for 64-bit universal binaries the architecture is ``fat64``, and + for 4-way universal binaries the architecture is ``universal``. Starting + from Python 2.7 and Python 3.2 the architecture ``fat3`` is used for + a 3-way universal build (ppc, i386, x86_64) and ``intel`` is used for + a univeral build with the i386 and x86_64 architectures + + Examples of returned values on Mac OS X: + + * ``macosx-10.3-ppc`` + + * ``macosx-10.3-fat`` + + * ``macosx-10.5-universal`` + + * ``macosx-10.6-intel`` + + .. XXX reinvention of platform module? + + +.. function:: convert_path(pathname) + + Return 'pathname' as a name that will work on the native filesystem, i.e. + split it on '/' and put it back together again using the current directory + separator. Needed because filenames in the setup script are always supplied + in Unix style, and have to be converted to the local convention before we + can actually use them in the filesystem. Raises :exc:`ValueError` on + non-Unix-ish systems if *pathname* either starts or ends with a slash. + + +.. function:: change_root(new_root, pathname) + + Return *pathname* with *new_root* prepended. If *pathname* is relative, this + is equivalent to ``os.path.join(new_root,pathname)`` Otherwise, it requires + making *pathname* relative and then joining the two, which is tricky on + DOS/Windows. + + +.. function:: check_environ() + + Ensure that 'os.environ' has all the environment variables we guarantee that + users can use in config files, command-line options, etc. Currently this + includes: + + * :envvar:`HOME` - user's home directory (Unix only) + * :envvar:`PLAT` - description of the current platform, including hardware + and OS (see :func:`get_platform`) + + +.. function:: find_executable(executable, path=None) + + Search the path for a given executable name. + + +.. function:: subst_vars(s, local_vars) + + Perform shell/Perl-style variable substitution on *s*. Every occurrence of + ``$`` followed by a name is considered a variable, and variable is + substituted by the value found in the *local_vars* dictionary, or in + ``os.environ`` if it's not in *local_vars*. *os.environ* is first + checked/augmented to guarantee that it contains certain values: see + :func:`check_environ`. Raise :exc:`ValueError` for any variables not found + in either *local_vars* or ``os.environ``. + + Note that this is not a fully-fledged string interpolation function. A valid + ``$variable`` can consist only of upper and lower case letters, numbers and + an underscore. No { } or ( ) style quoting is available. + + +.. function:: split_quoted(s) + + Split a string up according to Unix shell-like rules for quotes and + backslashes. In short: words are delimited by spaces, as long as those spaces + are not escaped by a backslash, or inside a quoted string. Single and double + quotes are equivalent, and the quote characters can be backslash-escaped. + The backslash is stripped from any two-character escape sequence, leaving + only the escaped character. The quote characters are stripped from any + quoted string. Returns a list of words. + + .. TODO Should probably be moved into the standard library. + + +.. function:: execute(func, args[, msg=None, verbose=0, dry_run=0]) + + Perform some action that affects the outside world (for instance, writing to + the filesystem). Such actions are special because they are disabled by the + *dry_run* flag. This method takes care of all that bureaucracy for you; + all you have to do is supply the function to call and an argument tuple for + it (to embody the "external action" being performed), and an optional message + to print. + + +.. function:: newer(source, target) + + Return true if *source* exists and is more recently modified than *target*, + or if *source* exists and *target* doesn't. Return false if both exist and + *target* is the same age or newer than *source*. Raise + :exc:`PackagingFileError` if *source* does not exist. + + +.. function:: strtobool(val) + + Convert a string representation of truth to true (1) or false (0). + + True values are ``y``, ``yes``, ``t``, ``true``, ``on`` and ``1``; false + values are ``n``, ``no``, ``f``, ``false``, ``off`` and ``0``. Raises + :exc:`ValueError` if *val* is anything else. + +.. TODO Add :term: markup to bytecode when merging into the stdlib + +.. function:: byte_compile(py_files[, optimize=0, force=0, prefix=None, base_dir=None, verbose=1, dry_run=0, direct=None]) + + Byte-compile a collection of Python source files to either :file:`.pyc` or + :file:`.pyo` files in the same directory. *py_files* is a list of files to + compile; any files that don't end in :file:`.py` are silently skipped. + *optimize* must be one of the following: + + * ``0`` - don't optimize (generate :file:`.pyc`) + * ``1`` - normal optimization (like ``python -O``) + * ``2`` - extra optimization (like ``python -OO``) + + If *force* is true, all files are recompiled regardless of timestamps. + + The source filename encoded in each bytecode file defaults to the filenames + listed in *py_files*; you can modify these with *prefix* and *basedir*. + *prefix* is a string that will be stripped off of each source filename, and + *base_dir* is a directory name that will be prepended (after *prefix* is + stripped). You can supply either or both (or neither) of *prefix* and + *base_dir*, as you wish. + + If *dry_run* is true, doesn't actually do anything that would affect the + filesystem. + + Byte-compilation is either done directly in this interpreter process with the + standard :mod:`py_compile` module, or indirectly by writing a temporary + script and executing it. Normally, you should let :func:`byte_compile` + figure out to use direct compilation or not (see the source for details). + The *direct* flag is used by the script generated in indirect mode; unless + you know what you're doing, leave it set to ``None``. + + +.. function:: rfc822_escape(header) + + Return a version of *header* escaped for inclusion in an :rfc:`822` header, by + ensuring there are 8 spaces space after each newline. Note that it does no + other modification of the string. + + .. TODO this _can_ be replaced diff --git a/Doc/library/packaging.version.rst b/Doc/library/packaging.version.rst new file mode 100644 index 00000000000..f36cdaba10c --- /dev/null +++ b/Doc/library/packaging.version.rst @@ -0,0 +1,104 @@ +:mod:`packaging.version` --- Version number classes +=================================================== + +.. module:: packaging.version + :synopsis: Classes that represent project version numbers. + + +This module contains classes and functions useful to deal with version numbers. +It's an implementation of version specifiers `as defined in PEP 345 +`_. + + +Version numbers +--------------- + +.. class:: NormalizedVersion(self, s, error_on_huge_major_num=True) + + A specific version of a distribution, as described in PEP 345. *s* is a + string object containing the version number (for example ``'1.2b1'``), + *error_on_huge_major_num* a boolean specifying whether to consider an + apparent use of a year or full date as the major version number an error. + + The rationale for the second argument is that there were projects using years + or full dates as version numbers, which could cause problems with some + packaging systems sorting. + + Instances of this class can be compared and sorted:: + + >>> NormalizedVersion('1.2b1') < NormalizedVersion('1.2') + True + + :class:`NormalizedVersion` is used internally by :class:`VersionPredicate` to + do its work. + + +.. class:: IrrationalVersionError + + Exception raised when an invalid string is given to + :class:`NormalizedVersion`. + + >>> NormalizedVersion("irrational_version_number") + ... + IrrationalVersionError: irrational_version_number + + +.. function:: suggest_normalized_version(s) + + Before standardization in PEP 386, various schemes were in use. Packaging + provides a function to try to convert any string to a valid, normalized + version:: + + >>> suggest_normalized_version('2.1-rc1') + 2.1c1 + + + If :func:`suggest_normalized_version` can't make sense of the given string, + it will return ``None``:: + + >>> print(suggest_normalized_version('not a version')) + None + + +Version predicates +------------------ + +.. class:: VersionPredicate(predicate) + + This class deals with the parsing of field values like + ``ProjectName (>=version)``. + + .. method:: match(version) + + Test if a version number matches the predicate: + + >>> version = VersionPredicate("ProjectName (<1.2, >1.0)") + >>> version.match("1.2.1") + False + >>> version.match("1.1.1") + True + + +Validation helpers +------------------ + +If you want to use :term:`LBYL`-style checks instead of instantiating the +classes and catching :class:`IrrationalVersionError` and :class:`ValueError`, +you can use these functions: + +.. function:: is_valid_version(predicate) + + Check whether the given string is a valid version number. Example of valid + strings: ``'1.2'``, ``'4.2.0.dev4'``, ``'2.5.4.post2'``. + + +.. function:: is_valid_versions(predicate) + + Check whether the given string is a valid value for specifying multiple + versions, such as in the Requires-Python field. Example: ``'2.7, >=3.2'``. + + +.. function:: is_valid_predicate(predicate) + + Check whether the given string is a valid version predicate. Examples: + ``'some.project == 4.5, <= 4.7'``, ``'speciallib (> 1.0, != 1.4.2, < 2.0)'``. diff --git a/Doc/library/python.rst b/Doc/library/python.rst index b67fbfc2816..07eadb4faa2 100644 --- a/Doc/library/python.rst +++ b/Doc/library/python.rst @@ -25,4 +25,5 @@ overview: inspect.rst site.rst fpectl.rst + packaging.rst distutils.rst diff --git a/Doc/packaging/builtdist.rst b/Doc/packaging/builtdist.rst new file mode 100644 index 00000000000..b52c2085b72 --- /dev/null +++ b/Doc/packaging/builtdist.rst @@ -0,0 +1,307 @@ +.. _packaging-built-dist: + +**************************** +Creating Built Distributions +**************************** + +A "built distribution" is what you're probably used to thinking of either as a +"binary package" or an "installer" (depending on your background). It's not +necessarily binary, though, because it might contain only Python source code +and/or byte-code; and we don't call it a package, because that word is already +spoken for in Python. (And "installer" is a term specific to the world of +mainstream desktop systems.) + +A built distribution is how you make life as easy as possible for installers of +your module distribution: for users of RPM-based Linux systems, it's a binary +RPM; for Windows users, it's an executable installer; for Debian-based Linux +users, it's a Debian package; and so forth. Obviously, no one person will be +able to create built distributions for every platform under the sun, so the +Distutils are designed to enable module developers to concentrate on their +specialty---writing code and creating source distributions---while an +intermediary species called *packagers* springs up to turn source distributions +into built distributions for as many platforms as there are packagers. + +Of course, the module developer could be his own packager; or the packager could +be a volunteer "out there" somewhere who has access to a platform which the +original developer does not; or it could be software periodically grabbing new +source distributions and turning them into built distributions for as many +platforms as the software has access to. Regardless of who they are, a packager +uses the setup script and the :command:`bdist` command family to generate built +distributions. + +As a simple example, if I run the following command in the Distutils source +tree:: + + python setup.py bdist + +then the Distutils builds my module distribution (the Distutils itself in this +case), does a "fake" installation (also in the :file:`build` directory), and +creates the default type of built distribution for my platform. The default +format for built distributions is a "dumb" tar file on Unix, and a simple +executable installer on Windows. (That tar file is considered "dumb" because it +has to be unpacked in a specific location to work.) + +Thus, the above command on a Unix system creates +:file:`Distutils-1.0.{plat}.tar.gz`; unpacking this tarball from the right place +installs the Distutils just as though you had downloaded the source distribution +and run ``python setup.py install``. (The "right place" is either the root of +the filesystem or Python's :file:`{prefix}` directory, depending on the options +given to the :command:`bdist_dumb` command; the default is to make dumb +distributions relative to :file:`{prefix}`.) + +Obviously, for pure Python distributions, this isn't any simpler than just +running ``python setup.py install``\ ---but for non-pure distributions, which +include extensions that would need to be compiled, it can mean the difference +between someone being able to use your extensions or not. And creating "smart" +built distributions, such as an executable installer for +Windows, is far more convenient for users even if your distribution doesn't +include any extensions. + +The :command:`bdist` command has a :option:`--formats` option, similar to the +:command:`sdist` command, which you can use to select the types of built +distribution to generate: for example, :: + + python setup.py bdist --format=zip + +would, when run on a Unix system, create :file:`Distutils-1.0.{plat}.zip`\ +---again, this archive would be unpacked from the root directory to install the +Distutils. + +The available formats for built distributions are: + ++-------------+------------------------------+---------+ +| Format | Description | Notes | ++=============+==============================+=========+ +| ``gztar`` | gzipped tar file | (1),(3) | +| | (:file:`.tar.gz`) | | ++-------------+------------------------------+---------+ +| ``ztar`` | compressed tar file | \(3) | +| | (:file:`.tar.Z`) | | ++-------------+------------------------------+---------+ +| ``tar`` | tar file (:file:`.tar`) | \(3) | ++-------------+------------------------------+---------+ +| ``zip`` | zip file (:file:`.zip`) | (2),(4) | ++-------------+------------------------------+---------+ +| ``wininst`` | self-extracting ZIP file for | \(4) | +| | Windows | | ++-------------+------------------------------+---------+ +| ``msi`` | Microsoft Installer. | | ++-------------+------------------------------+---------+ + + +Notes: + +(1) + default on Unix + +(2) + default on Windows + +(3) + requires external utilities: :program:`tar` and possibly one of :program:`gzip`, + :program:`bzip2`, or :program:`compress` + +(4) + requires either external :program:`zip` utility or :mod:`zipfile` module (part + of the standard Python library since Python 1.6) + +You don't have to use the :command:`bdist` command with the :option:`--formats` +option; you can also use the command that directly implements the format you're +interested in. Some of these :command:`bdist` "sub-commands" actually generate +several similar formats; for instance, the :command:`bdist_dumb` command +generates all the "dumb" archive formats (``tar``, ``ztar``, ``gztar``, and +``zip``). The :command:`bdist` sub-commands, and the formats generated by +each, are: + ++--------------------------+-----------------------+ +| Command | Formats | ++==========================+=======================+ +| :command:`bdist_dumb` | tar, ztar, gztar, zip | ++--------------------------+-----------------------+ +| :command:`bdist_wininst` | wininst | ++--------------------------+-----------------------+ +| :command:`bdist_msi` | msi | ++--------------------------+-----------------------+ + +The following sections give details on the individual :command:`bdist_\*` +commands. + + +.. _packaging-creating-dumb: + +Creating dumb built distributions +================================= + +.. XXX Need to document absolute vs. prefix-relative packages here, but first + I have to implement it! + + +.. _packaging-creating-wininst: + +Creating Windows Installers +=========================== + +Executable installers are the natural format for binary distributions on +Windows. They display a nice graphical user interface, display some information +about the module distribution to be installed taken from the metadata in the +setup script, let the user select a few options, and start or cancel the +installation. + +Since the metadata is taken from the setup script, creating Windows installers +is usually as easy as running:: + + python setup.py bdist_wininst + +or the :command:`bdist` command with the :option:`--formats` option:: + + python setup.py bdist --formats=wininst + +If you have a pure module distribution (only containing pure Python modules and +packages), the resulting installer will be version independent and have a name +like :file:`foo-1.0.win32.exe`. These installers can even be created on Unix +platforms or Mac OS X. + +If you have a non-pure distribution, the extensions can only be created on a +Windows platform, and will be Python version dependent. The installer filename +will reflect this and now has the form :file:`foo-1.0.win32-py2.0.exe`. You +have to create a separate installer for every Python version you want to +support. + +.. TODO Add :term: markup to bytecode when merging into the stdlib + +The installer will try to compile pure modules into bytecode after installation +on the target system in normal and optimizing mode. If you don't want this to +happen for some reason, you can run the :command:`bdist_wininst` command with +the :option:`--no-target-compile` and/or the :option:`--no-target-optimize` +option. + +By default the installer will display the cool "Python Powered" logo when it is +run, but you can also supply your own 152x261 bitmap which must be a Windows +:file:`.bmp` file with the :option:`--bitmap` option. + +The installer will also display a large title on the desktop background window +when it is run, which is constructed from the name of your distribution and the +version number. This can be changed to another text by using the +:option:`--title` option. + +The installer file will be written to the "distribution directory" --- normally +:file:`dist/`, but customizable with the :option:`--dist-dir` option. + +.. _packaging-cross-compile-windows: + +Cross-compiling on Windows +========================== + +Starting with Python 2.6, packaging is capable of cross-compiling between +Windows platforms. In practice, this means that with the correct tools +installed, you can use a 32bit version of Windows to create 64bit extensions +and vice-versa. + +To build for an alternate platform, specify the :option:`--plat-name` option +to the build command. Valid values are currently 'win32', 'win-amd64' and +'win-ia64'. For example, on a 32bit version of Windows, you could execute:: + + python setup.py build --plat-name=win-amd64 + +to build a 64bit version of your extension. The Windows Installers also +support this option, so the command:: + + python setup.py build --plat-name=win-amd64 bdist_wininst + +would create a 64bit installation executable on your 32bit version of Windows. + +To cross-compile, you must download the Python source code and cross-compile +Python itself for the platform you are targetting - it is not possible from a +binary installtion of Python (as the .lib etc file for other platforms are +not included.) In practice, this means the user of a 32 bit operating +system will need to use Visual Studio 2008 to open the +:file:`PCBuild/PCbuild.sln` solution in the Python source tree and build the +"x64" configuration of the 'pythoncore' project before cross-compiling +extensions is possible. + +Note that by default, Visual Studio 2008 does not install 64bit compilers or +tools. You may need to reexecute the Visual Studio setup process and select +these tools (using Control Panel->[Add/Remove] Programs is a convenient way to +check or modify your existing install.) + +.. _packaging-postinstallation-script: + +The Postinstallation script +--------------------------- + +Starting with Python 2.3, a postinstallation script can be specified with the +:option:`--install-script` option. The basename of the script must be +specified, and the script filename must also be listed in the scripts argument +to the setup function. + +This script will be run at installation time on the target system after all the +files have been copied, with ``argv[1]`` set to :option:`-install`, and again at +uninstallation time before the files are removed with ``argv[1]`` set to +:option:`-remove`. + +The installation script runs embedded in the windows installer, every output +(``sys.stdout``, ``sys.stderr``) is redirected into a buffer and will be +displayed in the GUI after the script has finished. + +Some functions especially useful in this context are available as additional +built-in functions in the installation script. + +.. currentmodule:: bdist_wininst-postinst-script + +.. function:: directory_created(path) + file_created(path) + + These functions should be called when a directory or file is created by the + postinstall script at installation time. It will register *path* with the + uninstaller, so that it will be removed when the distribution is uninstalled. + To be safe, directories are only removed if they are empty. + + +.. function:: get_special_folder_path(csidl_string) + + This function can be used to retrieve special folder locations on Windows like + the Start Menu or the Desktop. It returns the full path to the folder. + *csidl_string* must be one of the following strings:: + + "CSIDL_APPDATA" + + "CSIDL_COMMON_STARTMENU" + "CSIDL_STARTMENU" + + "CSIDL_COMMON_DESKTOPDIRECTORY" + "CSIDL_DESKTOPDIRECTORY" + + "CSIDL_COMMON_STARTUP" + "CSIDL_STARTUP" + + "CSIDL_COMMON_PROGRAMS" + "CSIDL_PROGRAMS" + + "CSIDL_FONTS" + + If the folder cannot be retrieved, :exc:`OSError` is raised. + + Which folders are available depends on the exact Windows version, and probably + also the configuration. For details refer to Microsoft's documentation of the + c:function:`SHGetSpecialFolderPath` function. + + +.. function:: create_shortcut(target, description, filename[, arguments[, workdir[, iconpath[, iconindex]]]]) + + This function creates a shortcut. *target* is the path to the program to be + started by the shortcut. *description* is the description of the shortcut. + *filename* is the title of the shortcut that the user will see. *arguments* + specifies the command-line arguments, if any. *workdir* is the working directory + for the program. *iconpath* is the file containing the icon for the shortcut, + and *iconindex* is the index of the icon in the file *iconpath*. Again, for + details consult the Microsoft documentation for the :class:`IShellLink` + interface. + + +Vista User Access Control (UAC) +=============================== + +Starting with Python 2.6, bdist_wininst supports a :option:`--user-access-control` +option. The default is 'none' (meaning no UAC handling is done), and other +valid values are 'auto' (meaning prompt for UAC elevation if Python was +installed for all users) and 'force' (meaning always prompt for elevation). diff --git a/Doc/packaging/commandhooks.rst b/Doc/packaging/commandhooks.rst new file mode 100644 index 00000000000..8dc233b7583 --- /dev/null +++ b/Doc/packaging/commandhooks.rst @@ -0,0 +1,31 @@ +============= +Command hooks +============= + +Packaging provides a way of extending its commands by the use of pre- and +post- command hooks. The hooks are simple Python functions (or any callable +objects) and are specified in the config file using their full qualified names. +The pre-hooks are run after the command is finalized (its options are +processed), but before it is run. The post-hooks are run after the command +itself. Both types of hooks receive an instance of the command object. + +Sample usage of hooks +===================== + +Firstly, you need to make sure your hook is present in the path. This is usually +done by dropping them to the same folder where `setup.py` file lives :: + + # file: myhooks.py + def my_install_hook(install_cmd): + print "Oh la la! Someone is installing my project!" + +Then, you need to point to it in your `setup.cfg` file, under the appropriate +command section :: + + [install_dist] + pre-hook.project = myhooks.my_install_hook + +The hooks defined in different config files (system-wide, user-wide and +package-wide) do not override each other as long as they are specified with +different aliases (additional names after the dot). The alias in the example +above is ``project``. diff --git a/Doc/packaging/commandref.rst b/Doc/packaging/commandref.rst new file mode 100644 index 00000000000..0d7447dc4cb --- /dev/null +++ b/Doc/packaging/commandref.rst @@ -0,0 +1,349 @@ +.. _packaging-command-reference: + +***************** +Command Reference +***************** + +This reference briefly documents all standard Packaging commands and some of +their options. + +.. FIXME does not work: Use pysetup run --help-commands to list all + standard and extra commands availavble on your system, with their + description. Use pysetup run --help to get help about the options + of one command. + + +Preparing distributions +======================= + +:command:`check` +---------------- + +Perform some tests on the metadata of a distribution. + +For example, it verifies that all required metadata fields are provided in the +:file:`setup.cfg` file. + +.. TODO document reST checks + + +:command:`test` +--------------- + +Run a test suite. + +When doing test-driven development, or running automated builds that need +testing before they are installed for downloading or use, it's often useful to +be able to run a project's unit tests without actually installing the project +anywhere. The :command:`test` command runs project's unit tests without +actually installing it, by temporarily putting the project's source on +:data:`sys.path`, after first running :command:`build_ext -i` to ensure that any +C extensions are built. + +You can use this command in one of two ways: either by specifying a +unittest-compatible test suite for your project (or any callable that returns +it) or by passing a test runner function that will run your tests and display +results in the console. Both options take a Python dotted name in the form +``package.module.callable`` to specify the object to use. + +If none of these options are specified, Packaging will try to perform test +discovery using either unittest (for Python 3.2 and higher) or unittest2 (for +older versions, if installed). + +.. this is a pseudo-command name used to disambiguate the options in indexes and + links +.. program:: packaging test + +.. cmdoption:: --suite=NAME, -s NAME + + Specify the test suite (or module, class, or method) to be run. The default + for this option can be set by in the project's :file:`setup.cfg` file:: + + .. code-block:: cfg + + [test] + suite = mypackage.tests.get_all_tests + +.. cmdoption:: --runner=NAME, -r NAME + + Specify the test runner to be called. + + +:command:`config` +----------------- + +Perform distribution configuration. + + +The build step +============== + +This step is mainly useful to compile C/C++ libraries or extension modules. The +build commands can be run manually to check for syntax errors or packaging +issues (for example if the addition of a new source file was forgotten in the +:file:`setup.cfg` file), and is also run automatically by commands which need +it. Packaging checks the mtime of source and built files to avoid re-building +if it's not necessary. + + +:command:`build` +---------------- + +Build all files of a distribution, delegating to the other :command:`build_*` +commands to do the work. + + +:command:`build_clib` +--------------------- + +Build C libraries. + + +:command:`build_ext` +-------------------- + +Build C/C++ extension modules. + + +:command:`build_py` +------------------- + +Build the Python modules (just copy them to the build directory) and +byte-compile them to .pyc files. + + +:command:`build_scripts` +------------------------ +Build the scripts (just copy them to the build directory and adjust their +shebang if they're Python scripts). + + +:command:`clean` +---------------- + +Clean the build tree of the release. + +.. program:: packaging clean + +.. cmdoption:: --all, -a + + Remove build directories for modules, scripts, etc., not only temporary build + by-products. + + +Creating source and built distributions +======================================= + +:command:`sdist` +---------------- + +Build a source distribution for a release. + +It is recommended that you always build and upload a source distribution. Users +of OSes with easy access to compilers and users of advanced packaging tools will +prefer to compile from source rather than using pre-built distributions. For +Windows users, providing a binary installer is also recommended practice. + + +:command:`bdist` +---------------- + +Build a binary distribution for a release. + +This command will call other :command:`bdist_*` commands to create one or more +distributions depending on the options given. The default is to create a +.tar.gz archive on Unix and a zip archive on Windows or OS/2. + +.. program:: packaging bdist + +.. cmdoption:: --formats + + Binary formats to build (comma-separated list). + +.. cmdoption:: --show-formats + + Dump list of available formats. + + +:command:`bdist_dumb` +--------------------- + +Build a "dumb" installer, a simple archive of files that could be unpacked under +``$prefix`` or ``$exec_prefix``. + + +:command:`bdist_wininst` +------------------------ + +Build a Windows installer. + + +:command:`bdist_msi` +-------------------- + +Build a `Microsoft Installer`_ (.msi) file. + +.. _Microsoft Installer: http://msdn.microsoft.com/en-us/library/cc185688(VS.85).aspx + +In most cases, the :command:`bdist_msi` installer is a better choice than the +:command:`bdist_wininst` installer, because it provides better support for Win64 +platforms, allows administrators to perform non-interactive installations, and +allows installation through group policies. + + +Publishing distributions +======================== + +:command:`register` +------------------- + +This command registers the current release with the Python Package Index. This +is described in more detail in :PEP:`301`. + +.. TODO explain user and project registration with the web UI + + +:command:`upload` +----------------- + +Upload source and/or binary distributions to PyPI. + +The distributions have to be built on the same command line as the +:command:`upload` command; see :ref:`packaging-package-upload` for more info. + +.. program:: packaging upload + +.. cmdoption:: --sign, -s + + Sign each uploaded file using GPG (GNU Privacy Guard). The ``gpg`` program + must be available for execution on the system ``PATH``. + +.. cmdoption:: --identity=NAME, -i NAME + + Specify the identity or key name for GPG to use when signing. The value of + this option will be passed through the ``--local-user`` option of the + ``gpg`` program. + +.. cmdoption:: --show-response + + Display the full response text from server; this is useful for debugging + PyPI problems. + +.. cmdoption:: --repository=URL, -r URL + + The URL of the repository to upload to. Defaults to + http://pypi.python.org/pypi (i.e., the main PyPI installation). + +.. cmdoption:: --upload-docs + + Also run :command:`upload_docs`. Mainly useful as a default value in + :file:`setup.cfg` (on the command line, it's shorter to just type both + commands). + + +:command:`upload_docs` +---------------------- + +Upload HTML documentation to PyPI. + +PyPI now supports publishing project documentation at a URI of the form +``http://packages.python.org/``. :command:`upload_docs` will create +the necessary zip file out of a documentation directory and will post to the +repository. + +Note that to upload the documentation of a project, the corresponding version +must already be registered with PyPI, using the :command:`register` command --- +just like with :command:`upload`. + +Assuming there is an ``Example`` project with documentation in the subdirectory +:file:`docs`, for example:: + + Example/ + example.py + setup.cfg + docs/ + build/ + html/ + index.html + tips_tricks.html + conf.py + index.txt + tips_tricks.txt + +You can simply specify the directory with the HTML files in your +:file:`setup.cfg` file: + +.. code-block:: cfg + + [upload_docs] + upload-dir = docs/build/html + + +.. program:: packaging upload_docs + +.. cmdoption:: --upload-dir + + The directory to be uploaded to the repository. By default documentation + is searched for in ``docs`` (or ``doc``) directory in project root. + +.. cmdoption:: --show-response + + Display the full response text from server; this is useful for debugging + PyPI problems. + +.. cmdoption:: --repository=URL, -r URL + + The URL of the repository to upload to. Defaults to + http://pypi.python.org/pypi (i.e., the main PyPI installation). + + +The install step +================ + +These commands are used by end-users of a project using :program:`pysetup` or +another compatible installer. Each command will run the corresponding +:command:`build_*` command and then move the built files to their destination on +the target system. + + +:command:`install_dist` +----------------------- + +Install a distribution, delegating to the other :command:`install_*` commands to +do the work. + +.. program:: packaging install_dist + +.. cmdoption:: --user + + Install in user site-packages directory (see :PEP:`370`). + + +:command:`install_data` +----------------------- + +Install data files. + + +:command:`install_distinfo` +--------------------------- + +Install files recording details of the installation as specified in :PEP:`376`. + + +:command:`install_headers` +-------------------------- + +Install C/C++ header files. + + +:command:`install_lib` +---------------------- + +Install C library files. + + +:command:`install_scripts` +-------------------------- + +Install scripts. diff --git a/Doc/packaging/configfile.rst b/Doc/packaging/configfile.rst new file mode 100644 index 00000000000..825b5cb5589 --- /dev/null +++ b/Doc/packaging/configfile.rst @@ -0,0 +1,125 @@ +.. _packaging-setup-config: + +************************************ +Writing the Setup Configuration File +************************************ + +Often, it's not possible to write down everything needed to build a distribution +*a priori*: you may need to get some information from the user, or from the +user's system, in order to proceed. As long as that information is fairly +simple---a list of directories to search for C header files or libraries, for +example---then providing a configuration file, :file:`setup.cfg`, for users to +edit is a cheap and easy way to solicit it. Configuration files also let you +provide default values for any command option, which the installer can then +override either on the command line or by editing the config file. + +The setup configuration file is a useful middle-ground between the setup script +---which, ideally, would be opaque to installers [#]_---and the command line to +the setup script, which is outside of your control and entirely up to the +installer. In fact, :file:`setup.cfg` (and any other Distutils configuration +files present on the target system) are processed after the contents of the +setup script, but before the command line. This has several useful +consequences: + +.. If you have more advanced needs, such as determining which extensions to + build based on what capabilities are present on the target system, then you + need the Distutils auto-configuration facility. This started to appear in + Distutils 0.9 but, as of this writing, isn't mature or stable enough yet + for real-world use. + +* installers can override some of what you put in :file:`setup.py` by editing + :file:`setup.cfg` + +* you can provide non-standard defaults for options that are not easily set in + :file:`setup.py` + +* installers can override anything in :file:`setup.cfg` using the command-line + options to :file:`setup.py` + +The basic syntax of the configuration file is simple:: + + [command] + option = value + ... + +where *command* is one of the Distutils commands (e.g. :command:`build_py`, +:command:`install_dist`), and *option* is one of the options that command supports. +Any number of options can be supplied for each command, and any number of +command sections can be included in the file. Blank lines are ignored, as are +comments, which run from a ``'#'`` character until the end of the line. Long +option values can be split across multiple lines simply by indenting the +continuation lines. + +You can find out the list of options supported by a particular command with the +universal :option:`--help` option, e.g. :: + + > python setup.py --help build_ext + [...] + Options for 'build_ext' command: + --build-lib (-b) directory for compiled extension modules + --build-temp (-t) directory for temporary files (build by-products) + --inplace (-i) ignore build-lib and put compiled extensions into the + source directory alongside your pure Python modules + --include-dirs (-I) list of directories to search for header files + --define (-D) C preprocessor macros to define + --undef (-U) C preprocessor macros to undefine + --swig-opts list of SWIG command-line options + [...] + +.. XXX do we want to support ``setup.py --help metadata``? + +Note that an option spelled :option:`--foo-bar` on the command line is spelled +:option:`foo_bar` in configuration files. + +For example, say you want your extensions to be built "in-place"---that is, you +have an extension :mod:`pkg.ext`, and you want the compiled extension file +(:file:`ext.so` on Unix, say) to be put in the same source directory as your +pure Python modules :mod:`pkg.mod1` and :mod:`pkg.mod2`. You can always use the +:option:`--inplace` option on the command line to ensure this:: + + python setup.py build_ext --inplace + +But this requires that you always specify the :command:`build_ext` command +explicitly, and remember to provide :option:`--inplace`. An easier way is to +"set and forget" this option, by encoding it in :file:`setup.cfg`, the +configuration file for this distribution:: + + [build_ext] + inplace = 1 + +This will affect all builds of this module distribution, whether or not you +explicitly specify :command:`build_ext`. If you include :file:`setup.cfg` in +your source distribution, it will also affect end-user builds---which is +probably a bad idea for this option, since always building extensions in-place +would break installation of the module distribution. In certain peculiar cases, +though, modules are built right in their installation directory, so this is +conceivably a useful ability. (Distributing extensions that expect to be built +in their installation directory is almost always a bad idea, though.) + +Another example: certain commands take options that vary from project to +project but not depending on the installation system, for example, +:command:`test` needs to know where your test suite is located and what test +runner to use; likewise, :command:`upload_docs` can find HTML documentation in +a :file:`doc` or :file:`docs` directory, but needs an option to find files in +:file:`docs/build/html`. Instead of having to type out these options each +time you want to run the command, you can put them in the project's +:file:`setup.cfg`:: + + [test] + suite = packaging.tests + + [upload_docs] + upload-dir = docs/build/html + + +.. seealso:: + + :ref:`packaging-config-syntax` in "Installing Python Projects" + More information on the configuration files is available in the manual for + system administrators. + + +.. rubric:: Footnotes + +.. [#] This ideal probably won't be achieved until auto-configuration is fully + supported by the Distutils. diff --git a/Doc/packaging/examples.rst b/Doc/packaging/examples.rst new file mode 100644 index 00000000000..594ade01d74 --- /dev/null +++ b/Doc/packaging/examples.rst @@ -0,0 +1,334 @@ +.. _packaging-examples: + +******** +Examples +******** + +This chapter provides a number of basic examples to help get started with +Packaging. + + +.. _packaging-pure-mod: + +Pure Python distribution (by module) +==================================== + +If you're just distributing a couple of modules, especially if they don't live +in a particular package, you can specify them individually using the +:option:`py_modules` option in the setup script. + +In the simplest case, you'll have two files to worry about: a setup script and +the single module you're distributing, :file:`foo.py` in this example:: + + / + setup.py + foo.py + +(In all diagrams in this section, ** will refer to the distribution root +directory.) A minimal setup script to describe this situation would be:: + + from packaging.core import setup + setup(name='foo', + version='1.0', + py_modules=['foo']) + +Note that the name of the distribution is specified independently with the +:option:`name` option, and there's no rule that says it has to be the same as +the name of the sole module in the distribution (although that's probably a good +convention to follow). However, the distribution name is used to generate +filenames, so you should stick to letters, digits, underscores, and hyphens. + +Since :option:`py_modules` is a list, you can of course specify multiple +modules, e.g. if you're distributing modules :mod:`foo` and :mod:`bar`, your +setup might look like this:: + + / + setup.py + foo.py + bar.py + +and the setup script might be :: + + from packaging.core import setup + setup(name='foobar', + version='1.0', + py_modules=['foo', 'bar']) + +You can put module source files into another directory, but if you have enough +modules to do that, it's probably easier to specify modules by package rather +than listing them individually. + + +.. _packaging-pure-pkg: + +Pure Python distribution (by package) +===================================== + +If you have more than a couple of modules to distribute, especially if they are +in multiple packages, it's probably easier to specify whole packages rather than +individual modules. This works even if your modules are not in a package; you +can just tell the Distutils to process modules from the root package, and that +works the same as any other package (except that you don't have to have an +:file:`__init__.py` file). + +The setup script from the last example could also be written as :: + + from packaging.core import setup + setup(name='foobar', + version='1.0', + packages=['']) + +(The empty string stands for the root package.) + +If those two files are moved into a subdirectory, but remain in the root +package, e.g.:: + + / + setup.py + src/ + foo.py + bar.py + +then you would still specify the root package, but you have to tell the +Distutils where source files in the root package live:: + + from packaging.core import setup + setup(name='foobar', + version='1.0', + package_dir={'': 'src'}, + packages=['']) + +More typically, though, you will want to distribute multiple modules in the same +package (or in sub-packages). For example, if the :mod:`foo` and :mod:`bar` +modules belong in package :mod:`foobar`, one way to lay out your source tree is + +:: + + / + setup.py + foobar/ + __init__.py + foo.py + bar.py + +This is in fact the default layout expected by the Distutils, and the one that +requires the least work to describe in your setup script:: + + from packaging.core import setup + setup(name='foobar', + version='1.0', + packages=['foobar']) + +If you want to put modules in directories not named for their package, then you +need to use the :option:`package_dir` option again. For example, if the +:file:`src` directory holds modules in the :mod:`foobar` package:: + + / + setup.py + src/ + __init__.py + foo.py + bar.py + +an appropriate setup script would be :: + + from packaging.core import setup + setup(name='foobar', + version='1.0', + package_dir={'foobar': 'src'}, + packages=['foobar']) + +Or, you might put modules from your main package right in the distribution +root:: + + / + setup.py + __init__.py + foo.py + bar.py + +in which case your setup script would be :: + + from packaging.core import setup + setup(name='foobar', + version='1.0', + package_dir={'foobar': ''}, + packages=['foobar']) + +(The empty string also stands for the current directory.) + +If you have sub-packages, they must be explicitly listed in :option:`packages`, +but any entries in :option:`package_dir` automatically extend to sub-packages. +(In other words, the Distutils does *not* scan your source tree, trying to +figure out which directories correspond to Python packages by looking for +:file:`__init__.py` files.) Thus, if the default layout grows a sub-package:: + + / + setup.py + foobar/ + __init__.py + foo.py + bar.py + subfoo/ + __init__.py + blah.py + +then the corresponding setup script would be :: + + from packaging.core import setup + setup(name='foobar', + version='1.0', + packages=['foobar', 'foobar.subfoo']) + +(Again, the empty string in :option:`package_dir` stands for the current +directory.) + + +.. _packaging-single-ext: + +Single extension module +======================= + +Extension modules are specified using the :option:`ext_modules` option. +:option:`package_dir` has no effect on where extension source files are found; +it only affects the source for pure Python modules. The simplest case, a +single extension module in a single C source file, is:: + + / + setup.py + foo.c + +If the :mod:`foo` extension belongs in the root package, the setup script for +this could be :: + + from packaging.core import setup, Extension + setup(name='foobar', + version='1.0', + ext_modules=[Extension('foo', ['foo.c'])]) + +If the extension actually belongs in a package, say :mod:`foopkg`, then + +With exactly the same source tree layout, this extension can be put in the +:mod:`foopkg` package simply by changing the name of the extension:: + + from packaging.core import setup, Extension + setup(name='foobar', + version='1.0', + packages=['foopkg'], + ext_modules=[Extension('foopkg.foo', ['foo.c'])]) + + +Checking metadata +================= + +The ``check`` command allows you to verify if your project's metadata +meets the minimum requirements to build a distribution. + +To run it, just call it using your :file:`setup.py` script. If something is +missing, ``check`` will display a warning. + +Let's take an example with a simple script:: + + from packaging.core import setup + + setup(name='foobar') + +.. TODO configure logging StreamHandler to match this output + +Running the ``check`` command will display some warnings:: + + $ python setup.py check + running check + warning: check: missing required metadata: version, home_page + warning: check: missing metadata: either (author and author_email) or + (maintainer and maintainer_email) must be supplied + + +If you use the reStructuredText syntax in the ``long_description`` field and +`Docutils `_ is installed you can check if +the syntax is fine with the ``check`` command, using the ``restructuredtext`` +option. + +For example, if the :file:`setup.py` script is changed like this:: + + from packaging.core import setup + + desc = """\ + Welcome to foobar! + =============== + + This is the description of the ``foobar`` project. + """ + + setup(name='foobar', + version='1.0', + author=u'Tarek Ziadé', + author_email='tarek@ziade.org', + summary='Foobar utilities' + description=desc, + home_page='http://example.com') + +Where the long description is broken, ``check`` will be able to detect it +by using the :mod:`docutils` parser:: + + $ python setup.py check --restructuredtext + running check + warning: check: Title underline too short. (line 2) + warning: check: Could not finish the parsing. + + +.. _packaging-reading-metadata: + +Reading the metadata +==================== + +The :func:`packaging.core.setup` function provides a command-line interface +that allows you to query the metadata fields of a project through the +:file:`setup.py` script of a given project:: + + $ python setup.py --name + foobar + +This call reads the ``name`` metadata by running the +:func:`packaging.core.setup` function. When a source or binary +distribution is created with Distutils, the metadata fields are written +in a static file called :file:`PKG-INFO`. When a Distutils-based project is +installed in Python, the :file:`PKG-INFO` file is copied alongside the modules +and packages of the distribution under :file:`NAME-VERSION-pyX.X.egg-info`, +where ``NAME`` is the name of the project, ``VERSION`` its version as defined +in the Metadata, and ``pyX.X`` the major and minor version of Python like +``2.7`` or ``3.2``. + +You can read back this static file, by using the +:class:`packaging.dist.Metadata` class and its +:func:`read_pkg_file` method:: + + >>> from packaging.metadata import Metadata + >>> metadata = Metadata() + >>> metadata.read_pkg_file(open('distribute-0.6.8-py2.7.egg-info')) + >>> metadata.name + 'distribute' + >>> metadata.version + '0.6.8' + >>> metadata.description + 'Easily download, build, install, upgrade, and uninstall Python packages' + +Notice that the class can also be instantiated with a metadata file path to +loads its values:: + + >>> pkg_info_path = 'distribute-0.6.8-py2.7.egg-info' + >>> Metadata(pkg_info_path).name + 'distribute' + + +.. XXX These comments have been here for at least ten years. Write the + sections or delete the comments (we can maybe ask Greg Ward about + the planned contents). (Unindent to make them section titles) + + .. multiple-ext:: + + Multiple extension modules + ========================== + + Putting it all together + ======================= diff --git a/Doc/packaging/extending.rst b/Doc/packaging/extending.rst new file mode 100644 index 00000000000..f2d38631716 --- /dev/null +++ b/Doc/packaging/extending.rst @@ -0,0 +1,95 @@ +.. _extending-packaging: + +******************* +Extending Distutils +******************* + +Distutils can be extended in various ways. Most extensions take the form of new +commands or replacements for existing commands. New commands may be written to +support new types of platform-specific packaging, for example, while +replacements for existing commands may be made to modify details of how the +command operates on a package. + +Most extensions of the packaging are made within :file:`setup.py` scripts that +want to modify existing commands; many simply add a few file extensions that +should be copied into packages in addition to :file:`.py` files as a +convenience. + +Most packaging command implementations are subclasses of the +:class:`packaging.cmd.Command` class. New commands may directly inherit from +:class:`Command`, while replacements often derive from :class:`Command` +indirectly, directly subclassing the command they are replacing. Commands are +required to derive from :class:`Command`. + +.. .. _extend-existing: + Extending existing commands + =========================== + + +.. .. _new-commands: + Writing new commands + ==================== + + +Integrating new commands +======================== + +There are different ways to integrate new command implementations into +packaging. The most difficult is to lobby for the inclusion of the new features +in packaging itself, and wait for (and require) a version of Python that +provides that support. This is really hard for many reasons. + +The most common, and possibly the most reasonable for most needs, is to include +the new implementations with your :file:`setup.py` script, and cause the +:func:`packaging.core.setup` function use them:: + + from packaging.core import setup + from packaging.command.build_py import build_py as _build_py + + class build_py(_build_py): + """Specialized Python source builder.""" + + # implement whatever needs to be different... + + setup(..., cmdclass={'build_py': build_py}) + +This approach is most valuable if the new implementations must be used to use a +particular package, as everyone interested in the package will need to have the +new command implementation. + +Beginning with Python 2.4, a third option is available, intended to allow new +commands to be added which can support existing :file:`setup.py` scripts without +requiring modifications to the Python installation. This is expected to allow +third-party extensions to provide support for additional packaging systems, but +the commands can be used for anything packaging commands can be used for. A new +configuration option, :option:`command_packages` (command-line option +:option:`--command-packages`), can be used to specify additional packages to be +searched for modules implementing commands. Like all packaging options, this +can be specified on the command line or in a configuration file. This option +can only be set in the ``[global]`` section of a configuration file, or before +any commands on the command line. If set in a configuration file, it can be +overridden from the command line; setting it to an empty string on the command +line causes the default to be used. This should never be set in a configuration +file provided with a package. + +This new option can be used to add any number of packages to the list of +packages searched for command implementations; multiple package names should be +separated by commas. When not specified, the search is only performed in the +:mod:`packaging.command` package. When :file:`setup.py` is run with the option +:option:`--command-packages` :option:`distcmds,buildcmds`, however, the packages +:mod:`packaging.command`, :mod:`distcmds`, and :mod:`buildcmds` will be searched +in that order. New commands are expected to be implemented in modules of the +same name as the command by classes sharing the same name. Given the example +command-line option above, the command :command:`bdist_openpkg` could be +implemented by the class :class:`distcmds.bdist_openpkg.bdist_openpkg` or +:class:`buildcmds.bdist_openpkg.bdist_openpkg`. + + +Adding new distribution types +============================= + +Commands that create distributions (files in the :file:`dist/` directory) need +to add ``(command, filename)`` pairs to ``self.distribution.dist_files`` so that +:command:`upload` can upload it to PyPI. The *filename* in the pair contains no +path information, only the name of the file itself. In dry-run mode, pairs +should still be added to represent what would have been created. diff --git a/Doc/packaging/index.rst b/Doc/packaging/index.rst new file mode 100644 index 00000000000..1b597d744a7 --- /dev/null +++ b/Doc/packaging/index.rst @@ -0,0 +1,45 @@ +.. _packaging-index: + +############################## + Distributing Python Projects +############################## + +:Authors: Greg Ward, Anthony Baxter and Packaging contributors +:Email: distutils-sig@python.org +:Release: |version| +:Date: |today| + +This document describes Packaging for Python authors, describing how to use the +module to make Python applications, packages or modules easily available to a +wider audience with very little overhead for build/release/install mechanics. + +.. toctree:: + :maxdepth: 2 + :numbered: + + tutorial + setupcfg + introduction + setupscript + configfile + sourcedist + builtdist + packageindex + uploading + examples + extending + commandhooks + commandref + + +.. seealso:: + + :ref:`packaging-install-index` + A user-centered manual which includes information on adding projects + into an existing Python installation. You do not need to be a Python + programmer to read this manual. + + :mod:`packaging` + A library reference for developers of packaging tools wanting to use + standalone building blocks like :mod:`~packaging.version` or + :mod:`~packaging.metadata`, or extend Packaging itself. diff --git a/Doc/packaging/introduction.rst b/Doc/packaging/introduction.rst new file mode 100644 index 00000000000..db0ffbb96fd --- /dev/null +++ b/Doc/packaging/introduction.rst @@ -0,0 +1,193 @@ +.. _packaging-intro: + +***************************** +An Introduction to Packaging +***************************** + +This document covers using Packaging to distribute your Python modules, +concentrating on the role of developer/distributor. If you're looking for +information on installing Python modules you should refer to the +:ref:`packaging-install-index` chapter. + +Throughout this documentation, the terms "Distutils", "the Distutils" and +"Packaging" will be used interchangeably. + +.. _packaging-concepts: + +Concepts & Terminology +====================== + +Using Distutils is quite simple both for module developers and for +users/administrators installing third-party modules. As a developer, your +responsibilities (apart from writing solid, well-documented and well-tested +code, of course!) are: + +* writing a setup script (:file:`setup.py` by convention) + +* (optional) writing a setup configuration file + +* creating a source distribution + +* (optional) creating one or more "built" (binary) distributions of your + project + +All of these tasks are covered in this document. + +Not all module developers have access to multiple platforms, so one cannot +expect them to create buildt distributions for every platform. To remedy +this, it is hoped that intermediaries called *packagers* will arise to address +this need. Packagers take source distributions released by module developers, +build them on one or more platforms and release the resulting built +distributions. Thus, users on a greater range of platforms will be able to +install the most popular Python modules in the most natural way for their +platform without having to run a setup script or compile a single line of code. + + +.. _packaging-simple-example: + +A Simple Example +================ + +A setup script is usually quite simple, although since it's written in Python +there are no arbitrary limits to what you can do with it, though you should be +careful about putting expensive operations in your setup script. +Unlike, say, Autoconf-style configure scripts the setup script may be run +multiple times in the course of building and installing a module +distribution. + +If all you want to do is distribute a module called :mod:`foo`, contained in a +file :file:`foo.py`, then your setup script can be as simple as:: + + from packaging.core import setup + setup(name='foo', + version='1.0', + py_modules=['foo']) + +Some observations: + +* most information that you supply to the Distutils is supplied as keyword + arguments to the :func:`setup` function + +* those keyword arguments fall into two categories: package metadata (name, + version number, etc.) and information about what's in the package (a list + of pure Python modules in this case) + +* modules are specified by module name, not filename (the same will hold true + for packages and extensions) + +* it's recommended that you supply a little more metadata than we have in the + example. In particular your name, email address and a URL for the + project if appropriate (see section :ref:`packaging-setup-script` for an example) + +To create a source distribution for this module you would create a setup +script, :file:`setup.py`, containing the above code and run:: + + python setup.py sdist + +which will create an archive file (e.g., tarball on Unix, ZIP file on Windows) +containing your setup script :file:`setup.py`, and your module :file:`foo.py`. +The archive file will be named :file:`foo-1.0.tar.gz` (or :file:`.zip`), and +will unpack into a directory :file:`foo-1.0`. + +If an end-user wishes to install your :mod:`foo` module all he has to do is +download :file:`foo-1.0.tar.gz` (or :file:`.zip`), unpack it, and from the +:file:`foo-1.0` directory run :: + + python setup.py install + +which will copy :file:`foo.py` to the appropriate directory for +third-party modules in their Python installation. + +This simple example demonstrates some fundamental concepts of Distutils. +First, both developers and installers have the same basic user interface, i.e. +the setup script. The difference is which Distutils *commands* they use: the +:command:`sdist` command is almost exclusively for module developers, while +:command:`install` is more often used by installers (although some developers +will want to install their own code occasionally). + +If you want to make things really easy for your users, you can create more +than one built distributions for them. For instance, if you are running on a +Windows machine and want to make things easy for other Windows users, you can +create an executable installer (the most appropriate type of built distribution +for this platform) with the :command:`bdist_wininst` command. For example:: + + python setup.py bdist_wininst + +will create an executable installer, :file:`foo-1.0.win32.exe`, in the current +directory. You can find out what distribution formats are available at any time +by running :: + + python setup.py bdist --help-formats + + +.. _packaging-python-terms: + +General Python terminology +========================== + +If you're reading this document, you probably have a good idea of what Python +modules, extensions and so forth are. Nevertheless, just to be sure that +everyone is on the same page, here's a quick overview of Python terms: + +module + The basic unit of code reusability in Python: a block of code imported by + some other code. Three types of modules are important to us here: pure + Python modules, extension modules and packages. + +pure Python module + A module written in Python and contained in a single :file:`.py` file (and + possibly associated :file:`.pyc` and/or :file:`.pyo` files). Sometimes + referred to as a "pure module." + +extension module + A module written in the low-level language of the Python implementation: C/C++ + for Python, Java for Jython. Typically contained in a single dynamically + loaded pre-compiled file, e.g. a shared object (:file:`.so`) file for Python + extensions on Unix, a DLL (given the :file:`.pyd` extension) for Python + extensions on Windows, or a Java class file for Jython extensions. Note that + currently Distutils only handles C/C++ extensions for Python. + +package + A module that contains other modules, typically contained in a directory of + the filesystem and distinguished from other directories by the presence of a + file :file:`__init__.py`. + +root package + The root of the hierarchy of packages. (This isn't really a package, + since it doesn't have an :file:`__init__.py` file. But... we have to + call it something, right?) The vast majority of the standard library is + in the root package, as are many small standalone third-party modules that + don't belong to a larger module collection. Unlike regular packages, + modules in the root package can be found in many directories: in fact, + every directory listed in ``sys.path`` contributes modules to the root + package. + + +.. _packaging-term: + +Distutils-specific terminology +============================== + +The following terms apply more specifically to the domain of distributing Python +modules using Distutils: + +module distribution + A collection of Python modules distributed together as a single downloadable + resource and meant to be installed all as one. Examples of some well-known + module distributions are NumPy, SciPy, PIL (the Python Imaging + Library) or mxBase. (Module distributions would be called a *package*, + except that term is already taken in the Python context: a single module + distribution may contain zero, one, or many Python packages.) + +pure module distribution + A module distribution that contains only pure Python modules and packages. + Sometimes referred to as a "pure distribution." + +non-pure module distribution + A module distribution that contains at least one extension module. Sometimes + referred to as a "non-pure distribution." + +distribution root + The top-level directory of your source tree (or source distribution). The + directory where :file:`setup.py` exists. Generally :file:`setup.py` will + be run from this directory. diff --git a/Doc/packaging/packageindex.rst b/Doc/packaging/packageindex.rst new file mode 100644 index 00000000000..cd1d5986312 --- /dev/null +++ b/Doc/packaging/packageindex.rst @@ -0,0 +1,104 @@ +.. _packaging-package-index: + +********************************** +Registering with the Package Index +********************************** + +The Python Package Index (PyPI) holds metadata describing distributions +packaged with packaging. The packaging command :command:`register` is used to +submit your distribution's metadata to the index. It is invoked as follows:: + + python setup.py register + +Distutils will respond with the following prompt:: + + running register + We need to know who you are, so please choose either: + 1. use your existing login, + 2. register as a new user, + 3. have the server generate a new password for you (and email it to you), or + 4. quit + Your selection [default 1]: + +Note: if your username and password are saved locally, you will not see this +menu. + +If you have not registered with PyPI, then you will need to do so now. You +should choose option 2, and enter your details as required. Soon after +submitting your details, you will receive an email which will be used to confirm +your registration. + +Once you are registered, you may choose option 1 from the menu. You will be +prompted for your PyPI username and password, and :command:`register` will then +submit your metadata to the index. + +You may submit any number of versions of your distribution to the index. If you +alter the metadata for a particular version, you may submit it again and the +index will be updated. + +PyPI holds a record for each (name, version) combination submitted. The first +user to submit information for a given name is designated the Owner of that +name. They may submit changes through the :command:`register` command or through +the web interface. They may also designate other users as Owners or Maintainers. +Maintainers may edit the package information, but not designate other Owners or +Maintainers. + +By default PyPI will list all versions of a given package. To hide certain +versions, the Hidden property should be set to yes. This must be edited through +the web interface. + + +.. _packaging-pypirc: + +The .pypirc file +================ + +The format of the :file:`.pypirc` file is as follows:: + + [packaging] + index-servers = + pypi + + [pypi] + repository: + username: + password: + +The *packaging* section defines a *index-servers* variable that lists the +name of all sections describing a repository. + +Each section describing a repository defines three variables: + +- *repository*, that defines the url of the PyPI server. Defaults to + ``http://www.python.org/pypi``. +- *username*, which is the registered username on the PyPI server. +- *password*, that will be used to authenticate. If omitted the user + will be prompt to type it when needed. + +If you want to define another server a new section can be created and +listed in the *index-servers* variable:: + + [packaging] + index-servers = + pypi + other + + [pypi] + repository: + username: + password: + + [other] + repository: http://example.com/pypi + username: + password: + +:command:`register` can then be called with the -r option to point the +repository to work with:: + + python setup.py register -r http://example.com/pypi + +For convenience, the name of the section that describes the repository +may also be used:: + + python setup.py register -r other diff --git a/Doc/packaging/setupcfg.rst b/Doc/packaging/setupcfg.rst new file mode 100644 index 00000000000..be6c8c9d5ff --- /dev/null +++ b/Doc/packaging/setupcfg.rst @@ -0,0 +1,648 @@ +.. highlightlang:: cfg + +******************************************* +Specification of the :file:`setup.cfg` file +******************************************* + +.. :version: 1.0 + +This document describes the :file:`setup.cfg`, an ini-style configuration file +(compatible with :class:`configparser.RawConfigParser`) configuration file used +by Packaging to replace the :file:`setup.py` file. + +Each section contains a description of its options. + +- Options that are marked *multi* can have multiple values, one value per + line. +- Options that are marked *optional* can be omitted. +- Options that are marked *environ* can use environment markers, as described + in :PEP:`345`. + + +The sections are: + +global + Global options not related to one command. + +metadata + Name, version and other information defined by :PEP:`345`. + +files + Modules, scripts, data, documentation and other files to include in the + distribution. + +command sections + Options given for specific commands, identical to those that can be given + on the command line. + + +Global options +============== + +Contains global options for Packaging. This section is shared with Distutils. + + +commands + Defined Packaging command. A command is defined by its fully + qualified name. *optional*, *multi* + + Examples:: + + [global] + commands = + package.setup.CustomSdistCommand + package.setup.BdistDeb + +compilers + Defined Packaging compiler. A compiler is defined by its fully + qualified name. *optional*, *multi* + + Example:: + + [global] + compilers = + hotcompiler.SmartCCompiler + +setup_hook + defines a callable that will be called right after the + :file:`setup.cfg` file is read. The callable receives the configuration + in form of a mapping and can make some changes to it. *optional* + + Example:: + + [global] + setup_hook = package.setup.customize_dist + + +Metadata +======== + +The metadata section contains the metadata for the project as described in +:PEP:`345`. Field names are case-insensitive. + +Fields: + +name + Name of the project. + +version + Version of the project. Must comply with :PEP:`386`. + +platform + Platform specification describing an operating system + supported by the distribution which is not listed in the "Operating System" + Trove classifiers (:PEP:`301`). *optional*, *multi* + +supported-platform + Binary distributions containing a PKG-INFO file will + use the Supported-Platform field in their metadata to specify the OS and + CPU for which the binary distribution was compiled. The semantics of + the Supported-Platform field are free form. *optional*, *multi* + +summary + A one-line summary of what the distribution does. + (Used to be called *description* in Distutils1.) + +description + A longer description. (Used to be called *long_description* + in Distutils1.) A file can be provided in the *description-file* field. + *optional* + +description-file + path to a text file that will be used for the + **description** field. *optional* + +keywords + A list of additional keywords to be used to assist searching + for the distribution in a larger catalog. Comma or space-separated. + *optional* + +home-page + The URL for the distribution's home page. + +download-url + The URL from which this version of the distribution + can be downloaded. *optional* + +author + Author's name. *optional* + +author-email + Author's e-mail. *optional* + +maintainer + Maintainer's name. *optional* + +maintainer-email + Maintainer's e-mail. *optional* + +license + A text indicating the term of uses, when a trove classifier does + not match. *optional*. + +classifiers + Classification for the distribution, as described in PEP 301. + *optional*, *multi*, *environ* + +requires-dist + name of another packaging project required as a dependency. + The format is *name (version)* where version is an optional + version declaration, as described in PEP 345. *optional*, *multi*, *environ* + +provides-dist + name of another packaging project contained within this + distribution. Same format than *requires-dist*. *optional*, *multi*, + *environ* + +obsoletes-dist + name of another packaging project this version obsoletes. + Same format than *requires-dist*. *optional*, *multi*, *environ* + +requires-python + Specifies the Python version the distribution requires. + The value is a version number, as described in PEP 345. + *optional*, *multi*, *environ* + +requires-externals + a dependency in the system. This field is free-form, + and just a hint for downstream maintainers. *optional*, *multi*, + *environ* + +project-url + A label, followed by a browsable URL for the project. + "label, url". The label is limited to 32 signs. *optional*, *multi* + + +Example:: + + [metadata] + name = pypi2rpm + version = 0.1 + author = Tarek Ziadé + author-email = tarek@ziade.org + summary = Script that transforms an sdist archive into a RPM package + description-file = README + home-page = http://bitbucket.org/tarek/pypi2rpm/wiki/Home + project-url: + Repository, http://bitbucket.org/tarek/pypi2rpm/ + RSS feed, https://bitbucket.org/tarek/pypi2rpm/rss + classifier = + Development Status :: 3 - Alpha + License :: OSI Approved :: Mozilla Public License 1.1 (MPL 1.1) + +You should not give any explicit value for metadata-version: it will be guessed +from the fields present in the file. + + +Files +===== + +This section describes the files included in the project. + +packages_root + the root directory containing all packages and modules + (default: current directory). *optional* + +packages + a list of packages the project includes *optional*, *multi* + +modules + a list of packages the project includes *optional*, *multi* + +scripts + a list of scripts the project includes *optional*, *multi* + +extra_files + a list of patterns to include extra files *optional*, + *multi* + +Example:: + + [files] + packages_root = src + packages = + pypi2rpm + pypi2rpm.command + + scripts = + pypi2rpm/pypi2rpm.py + + extra_files = + setup.py + README + + +.. Note:: + The :file:`setup.cfg` configuration file is included by default. Contrary to + Distutils, :file:`README` (or :file:`README.txt`) and :file:`setup.py` are + not included by default. + + +Resources +--------- + +This section describes the files used by the project which must not be installed +in the same place that python modules or libraries, they are called +**resources**. They are for example documentation files, script files, +databases, etc... + +For declaring resources, you must use this notation:: + + source = destination + +Data-files are declared in the **resources** field in the **file** section, for +example:: + + [files] + resources = + source1 = destination1 + source2 = destination2 + +The **source** part of the declaration are relative paths of resources files +(using unix path separator **/**). For example, if you've this source tree:: + + foo/ + doc/ + doc.man + scripts/ + foo.sh + +Your setup.cfg will look like:: + + [files] + resources = + doc/doc.man = destination_doc + scripts/foo.sh = destination_scripts + +The final paths where files will be placed are composed by : **source** + +**destination**. In the previous example, **doc/doc.man** will be placed in +**destination_doc/doc/doc.man** and **scripts/foo.sh** will be placed in +**destination_scripts/scripts/foo.sh**. (If you want more control on the final +path, take a look at base_prefix_). + +The **destination** part of resources declaration are paths with categories. +Indeed, it's generally a bad idea to give absolute path as it will be cross +incompatible. So, you must use resources categories in your **destination** +declaration. Categories will be replaced by their real path at the installation +time. Using categories is all benefit, your declaration will be simpler, cross +platform and it will allow packager to place resources files where they want +without breaking your code. + +Categories can be specified by using this syntax:: + + {category} + +Default categories are: + +* config +* appdata +* appdata.arch +* appdata.persistent +* appdata.disposable +* help +* icon +* scripts +* doc +* info +* man + +A special category also exists **{distribution.name}** that will be replaced by +the name of the distribution, but as most of the defaults categories use them, +so it's not necessary to add **{distribution.name}** into your destination. + +If you use categories in your declarations, and you are encouraged to do, final +path will be:: + + source + destination_expanded + +.. _example_final_path: + +For example, if you have this setup.cfg:: + + [metadata] + name = foo + + [files] + resources = + doc/doc.man = {doc} + +And if **{doc}** is replaced by **{datadir}/doc/{distribution.name}**, final +path will be:: + + {datadir}/doc/foo/doc/doc.man + +Where {datafir} category will be platform-dependent. + + +More control on source part +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Glob syntax +""""""""""" + +When you declare source file, you can use a glob-like syntax to match multiples file, for example:: + + scripts/* = {script} + +Will match all the files in the scripts directory and placed them in the script category. + +Glob tokens are: + + * ``*``: match all files. + * ``?``: match any character. + * ``**``: match any level of tree recursion (even 0). + * ``{}``: will match any part separated by comma (example: ``{sh,bat}``). + +.. TODO Add examples + +Order of declaration +"""""""""""""""""""" + +The order of declaration is important if one file match multiple rules. The last +rules matched by file is used, this is useful if you have this source tree:: + + foo/ + doc/ + index.rst + setup.rst + documentation.txt + doc.tex + README + +And you want all the files in the doc directory to be placed in {doc} category, +but README must be placed in {help} category, instead of listing all the files +one by one, you can declare them in this way:: + + [files] + resources = + doc/* = {doc} + doc/README = {help} + +Exclude +""""""" + +You can exclude some files of resources declaration by giving no destination, it +can be useful if you have a non-resources file in the same directory of +resources files:: + + foo/ + doc/ + RELEASES + doc.tex + documentation.txt + docu.rst + +Your **files** section will be:: + + [files] + resources = + doc/* = {doc} + doc/RELEASES = + +More control on destination part +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. _base_prefix: + +Defining a base prefix +"""""""""""""""""""""" + +When you define your resources, you can have more control of how the final path +is compute. + +By default, the final path is:: + + destination + source + +This can generate long paths, for example (example_final_path_):: + + {datadir}/doc/foo/doc/doc.man + +When you declare your source, you can use whitespace to split the source in +**prefix** **suffix**. So, for example, if you have this source:: + + docs/ doc.man + +The **prefix** is "docs/" and the **suffix** is "doc.html". + +.. note:: + + Separator can be placed after a path separator or replace it. So these two + sources are equivalent:: + + docs/ doc.man + docs doc.man + +.. note:: + + Glob syntax is working the same way with standard source and splitted source. + So these rules:: + + docs/* + docs/ * + docs * + + Will match all the files in the docs directory. + +When you use splitted source, the final path is compute in this way:: + + destination + prefix + +So for example, if you have this setup.cfg:: + + [metadata] + name = foo + + [files] + resources = + doc/ doc.man = {doc} + +And if **{doc}** is replaced by **{datadir}/doc/{distribution.name}**, final +path will be:: + + {datadir}/doc/foo/doc.man + + +Overwriting paths for categories +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This part is intended for system administrators or downstream OS packagers. + +The real paths of categories are registered in the *sysconfig.cfg* file +installed in your python installation. This file uses an ini format too. +The content of the file is organized into several sections: + +* globals: Standard categories's paths. +* posix_prefix: Standard paths for categories and installation paths for posix + system. +* other ones XXX + +Standard categories paths are platform independent, they generally refers to +other categories, which are platform dependent. :mod:`sysconfig` will choose +these category from sections matching os.name. For example:: + + doc = {datadir}/doc/{distribution.name} + +It refers to datadir category, which can be different between platforms. In +posix system, it may be:: + + datadir = /usr/share + +So the final path will be:: + + doc = /usr/share/doc/{distribution.name} + +The platform-dependent categories are: + +* confdir +* datadir +* libdir +* base + + +Defining extra categories +^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. TODO + + +Examples +^^^^^^^^ + +These examples are incremental but work unitarily. + +Resources in root dir +""""""""""""""""""""" + +Source tree:: + + babar-1.0/ + README + babar.sh + launch.sh + babar.py + +:file:`setup.cfg`:: + + [files] + resources = + README = {doc} + *.sh = {scripts} + +So babar.sh and launch.sh will be placed in {scripts} directory. + +Now let's move all the scripts into a scripts directory. + +Resources in sub-directory +"""""""""""""""""""""""""" + +Source tree:: + + babar-1.1/ + README + scripts/ + babar.sh + launch.sh + LAUNCH + babar.py + +:file:`setup.cfg`:: + + [files] + resources = + README = {doc} + scripts/ LAUNCH = {doc} + scripts/ *.sh = {scripts} + +It's important to use the separator after scripts/ to install all the shell +scripts into {scripts} instead of {scripts}/scripts. + +Now let's add some docs. + +Resources in multiple sub-directories +""""""""""""""""""""""""""""""""""""" + +Source tree:: + + babar-1.2/ + README + scripts/ + babar.sh + launch.sh + LAUNCH + docs/ + api + man + babar.py + +:file:`setup.cfg`:: + + [files] + resources = + README = {doc} + scripts/ LAUNCH = {doc} + scripts/ *.sh = {scripts} + doc/ * = {doc} + doc/ man = {man} + +You want to place all the file in the docs script into {doc} category, instead +of man, which must be placed into {man} category, we will use the order of +declaration of globs to choose the destination, the last glob that match the +file is used. + +Now let's add some scripts for windows users. + +Complete example +"""""""""""""""" + +Source tree:: + + babar-1.3/ + README + doc/ + api + man + scripts/ + babar.sh + launch.sh + babar.bat + launch.bat + LAUNCH + +:file:`setup.cfg`:: + + [files] + resources = + README = {doc} + scripts/ LAUNCH = {doc} + scripts/ *.{sh,bat} = {scripts} + doc/ * = {doc} + doc/ man = {man} + +We use brace expansion syntax to place all the shell and batch scripts into +{scripts} category. + + +Command sections +================ + +To pass options to commands without having to type them on the command line +for each invocation, you can write them in the :file:`setup.cfg` file, in a +section named after the command. Example:: + + [sdist] + # special function to add custom files + manifest-builders = package.setup.list_extra_files + + [build] + use-2to3 = True + + [build_ext] + inplace = on + + [check] + strict = on + all = on + +Option values given in the configuration file can be overriden on the command +line. See :ref:`packaging-setup-config` for more information. diff --git a/Doc/packaging/setupscript.rst b/Doc/packaging/setupscript.rst new file mode 100644 index 00000000000..3a1a98b374d --- /dev/null +++ b/Doc/packaging/setupscript.rst @@ -0,0 +1,689 @@ +.. _packaging-setup-script: + +************************ +Writing the Setup Script +************************ + +The setup script is the center of all activity in building, distributing, and +installing modules using Distutils. The main purpose of the setup script is +to describe your module distribution to Distutils, so that the various +commands that operate on your modules do the right thing. As we saw in section +:ref:`packaging-simple-example`, the setup script consists mainly of a +call to :func:`setup` where the most information is supplied as +keyword arguments to :func:`setup`. + +Here's a slightly more involved example, which we'll follow for the next couple +of sections: a setup script that could be used for Packaging itself:: + + #!/usr/bin/env python + + from packaging.core import setup, find_packages + + setup(name='Packaging', + version='1.0', + summary='Python Distribution Utilities', + keywords=['packaging', 'packaging'], + author=u'Tarek Ziadé', + author_email='tarek@ziade.org', + home_page='http://bitbucket.org/tarek/packaging/wiki/Home', + license='PSF', + packages=find_packages()) + + +There are only two differences between this and the trivial one-file +distribution presented in section :ref:`packaging-simple-example`: more +metadata and the specification of pure Python modules by package rather than +by module. This is important since Ristutils consist of a couple of dozen +modules split into (so far) two packages; an explicit list of every module +would be tedious to generate and difficult to maintain. For more information +on the additional metadata, see section :ref:`packaging-metadata`. + +Note that any pathnames (files or directories) supplied in the setup script +should be written using the Unix convention, i.e. slash-separated. The +Distutils will take care of converting this platform-neutral representation into +whatever is appropriate on your current platform before actually using the +pathname. This makes your setup script portable across operating systems, which +of course is one of the major goals of the Distutils. In this spirit, all +pathnames in this document are slash-separated. + +This, of course, only applies to pathnames given to Distutils functions. If +you, for example, use standard Python functions such as :func:`glob.glob` or +:func:`os.listdir` to specify files, you should be careful to write portable +code instead of hardcoding path separators:: + + glob.glob(os.path.join('mydir', 'subdir', '*.html')) + os.listdir(os.path.join('mydir', 'subdir')) + + +.. _packaging-listing-packages: + +Listing whole packages +====================== + +The :option:`packages` option tells the Distutils to process (build, distribute, +install, etc.) all pure Python modules found in each package mentioned in the +:option:`packages` list. In order to do this, of course, there has to be a +correspondence between package names and directories in the filesystem. The +default correspondence is the most obvious one, i.e. package :mod:`packaging` is +found in the directory :file:`packaging` relative to the distribution root. +Thus, when you say ``packages = ['foo']`` in your setup script, you are +promising that the Distutils will find a file :file:`foo/__init__.py` (which +might be spelled differently on your system, but you get the idea) relative to +the directory where your setup script lives. If you break this promise, the +Distutils will issue a warning but still process the broken package anyways. + +If you use a different convention to lay out your source directory, that's no +problem: you just have to supply the :option:`package_dir` option to tell the +Distutils about your convention. For example, say you keep all Python source +under :file:`lib`, so that modules in the "root package" (i.e., not in any +package at all) are in :file:`lib`, modules in the :mod:`foo` package are in +:file:`lib/foo`, and so forth. Then you would put :: + + package_dir = {'': 'lib'} + +in your setup script. The keys to this dictionary are package names, and an +empty package name stands for the root package. The values are directory names +relative to your distribution root. In this case, when you say ``packages = +['foo']``, you are promising that the file :file:`lib/foo/__init__.py` exists. + +Another possible convention is to put the :mod:`foo` package right in +:file:`lib`, the :mod:`foo.bar` package in :file:`lib/bar`, etc. This would be +written in the setup script as :: + + package_dir = {'foo': 'lib'} + +A ``package: dir`` entry in the :option:`package_dir` dictionary implicitly +applies to all packages below *package*, so the :mod:`foo.bar` case is +automatically handled here. In this example, having ``packages = ['foo', +'foo.bar']`` tells the Distutils to look for :file:`lib/__init__.py` and +:file:`lib/bar/__init__.py`. (Keep in mind that although :option:`package_dir` +applies recursively, you must explicitly list all packages in +:option:`packages`: the Distutils will *not* recursively scan your source tree +looking for any directory with an :file:`__init__.py` file.) + + +.. _packaging-listing-modules: + +Listing individual modules +========================== + +For a small module distribution, you might prefer to list all modules rather +than listing packages---especially the case of a single module that goes in the +"root package" (i.e., no package at all). This simplest case was shown in +section :ref:`packaging-simple-example`; here is a slightly more involved +example:: + + py_modules = ['mod1', 'pkg.mod2'] + +This describes two modules, one of them in the "root" package, the other in the +:mod:`pkg` package. Again, the default package/directory layout implies that +these two modules can be found in :file:`mod1.py` and :file:`pkg/mod2.py`, and +that :file:`pkg/__init__.py` exists as well. And again, you can override the +package/directory correspondence using the :option:`package_dir` option. + + +.. _packaging-describing-extensions: + +Describing extension modules +============================ + +Just as writing Python extension modules is a bit more complicated than writing +pure Python modules, describing them to the Distutils is a bit more complicated. +Unlike pure modules, it's not enough just to list modules or packages and expect +the Distutils to go out and find the right files; you have to specify the +extension name, source file(s), and any compile/link requirements (include +directories, libraries to link with, etc.). + +.. XXX read over this section + +All of this is done through another keyword argument to :func:`setup`, the +:option:`ext_modules` option. :option:`ext_modules` is just a list of +:class:`Extension` instances, each of which describes a single extension module. +Suppose your distribution includes a single extension, called :mod:`foo` and +implemented by :file:`foo.c`. If no additional instructions to the +compiler/linker are needed, describing this extension is quite simple:: + + Extension('foo', ['foo.c']) + +The :class:`Extension` class can be imported from :mod:`packaging.core` along +with :func:`setup`. Thus, the setup script for a module distribution that +contains only this one extension and nothing else might be:: + + from packaging.core import setup, Extension + setup(name='foo', + version='1.0', + ext_modules=[Extension('foo', ['foo.c'])]) + +The :class:`Extension` class (actually, the underlying extension-building +machinery implemented by the :command:`build_ext` command) supports a great deal +of flexibility in describing Python extensions, which is explained in the +following sections. + + +Extension names and packages +---------------------------- + +The first argument to the :class:`Extension` constructor is always the name of +the extension, including any package names. For example, :: + + Extension('foo', ['src/foo1.c', 'src/foo2.c']) + +describes an extension that lives in the root package, while :: + + Extension('pkg.foo', ['src/foo1.c', 'src/foo2.c']) + +describes the same extension in the :mod:`pkg` package. The source files and +resulting object code are identical in both cases; the only difference is where +in the filesystem (and therefore where in Python's namespace hierarchy) the +resulting extension lives. + +If you have a number of extensions all in the same package (or all under the +same base package), use the :option:`ext_package` keyword argument to +:func:`setup`. For example, :: + + setup(..., + ext_package='pkg', + ext_modules=[Extension('foo', ['foo.c']), + Extension('subpkg.bar', ['bar.c'])]) + +will compile :file:`foo.c` to the extension :mod:`pkg.foo`, and :file:`bar.c` to +:mod:`pkg.subpkg.bar`. + + +Extension source files +---------------------- + +The second argument to the :class:`Extension` constructor is a list of source +files. Since the Distutils currently only support C, C++, and Objective-C +extensions, these are normally C/C++/Objective-C source files. (Be sure to use +appropriate extensions to distinguish C++\ source files: :file:`.cc` and +:file:`.cpp` seem to be recognized by both Unix and Windows compilers.) + +However, you can also include SWIG interface (:file:`.i`) files in the list; the +:command:`build_ext` command knows how to deal with SWIG extensions: it will run +SWIG on the interface file and compile the resulting C/C++ file into your +extension. + +.. XXX SWIG support is rough around the edges and largely untested! + +This warning notwithstanding, options to SWIG can be currently passed like +this:: + + setup(..., + ext_modules=[Extension('_foo', ['foo.i'], + swig_opts=['-modern', '-I../include'])], + py_modules=['foo']) + +Or on the command line like this:: + + > python setup.py build_ext --swig-opts="-modern -I../include" + +On some platforms, you can include non-source files that are processed by the +compiler and included in your extension. Currently, this just means Windows +message text (:file:`.mc`) files and resource definition (:file:`.rc`) files for +Visual C++. These will be compiled to binary resource (:file:`.res`) files and +linked into the executable. + + +Preprocessor options +-------------------- + +Three optional arguments to :class:`Extension` will help if you need to specify +include directories to search or preprocessor macros to define/undefine: +``include_dirs``, ``define_macros``, and ``undef_macros``. + +For example, if your extension requires header files in the :file:`include` +directory under your distribution root, use the ``include_dirs`` option:: + + Extension('foo', ['foo.c'], include_dirs=['include']) + +You can specify absolute directories there; if you know that your extension will +only be built on Unix systems with X11R6 installed to :file:`/usr`, you can get +away with :: + + Extension('foo', ['foo.c'], include_dirs=['/usr/include/X11']) + +You should avoid this sort of non-portable usage if you plan to distribute your +code: it's probably better to write C code like :: + + #include + +If you need to include header files from some other Python extension, you can +take advantage of the fact that header files are installed in a consistent way +by the Distutils :command:`install_header` command. For example, the Numerical +Python header files are installed (on a standard Unix installation) to +:file:`/usr/local/include/python1.5/Numerical`. (The exact location will differ +according to your platform and Python installation.) Since the Python include +directory---\ :file:`/usr/local/include/python1.5` in this case---is always +included in the search path when building Python extensions, the best approach +is to write C code like :: + + #include + +.. TODO check if it's d2.sysconfig or the new sysconfig module now + +If you must put the :file:`Numerical` include directory right into your header +search path, though, you can find that directory using the Distutils +:mod:`packaging.sysconfig` module:: + + from packaging.sysconfig import get_python_inc + incdir = os.path.join(get_python_inc(plat_specific=1), 'Numerical') + setup(..., + Extension(..., include_dirs=[incdir])) + +Even though this is quite portable---it will work on any Python installation, +regardless of platform---it's probably easier to just write your C code in the +sensible way. + +You can define and undefine preprocessor macros with the ``define_macros`` and +``undef_macros`` options. ``define_macros`` takes a list of ``(name, value)`` +tuples, where ``name`` is the name of the macro to define (a string) and +``value`` is its value: either a string or ``None``. (Defining a macro ``FOO`` +to ``None`` is the equivalent of a bare ``#define FOO`` in your C source: with +most compilers, this sets ``FOO`` to the string ``1``.) ``undef_macros`` is +just a list of macros to undefine. + +For example:: + + Extension(..., + define_macros=[('NDEBUG', '1'), + ('HAVE_STRFTIME', None)], + undef_macros=['HAVE_FOO', 'HAVE_BAR']) + +is the equivalent of having this at the top of every C source file:: + + #define NDEBUG 1 + #define HAVE_STRFTIME + #undef HAVE_FOO + #undef HAVE_BAR + + +Library options +--------------- + +You can also specify the libraries to link against when building your extension, +and the directories to search for those libraries. The ``libraries`` option is +a list of libraries to link against, ``library_dirs`` is a list of directories +to search for libraries at link-time, and ``runtime_library_dirs`` is a list of +directories to search for shared (dynamically loaded) libraries at run-time. + +For example, if you need to link against libraries known to be in the standard +library search path on target systems :: + + Extension(..., + libraries=['gdbm', 'readline']) + +If you need to link with libraries in a non-standard location, you'll have to +include the location in ``library_dirs``:: + + Extension(..., + library_dirs=['/usr/X11R6/lib'], + libraries=['X11', 'Xt']) + +(Again, this sort of non-portable construct should be avoided if you intend to +distribute your code.) + +.. XXX Should mention clib libraries here or somewhere else! + + +Other options +------------- + +There are still some other options which can be used to handle special cases. + +The :option:`optional` option is a boolean; if it is true, +a build failure in the extension will not abort the build process, but +instead simply not install the failing extension. + +The :option:`extra_objects` option is a list of object files to be passed to the +linker. These files must not have extensions, as the default extension for the +compiler is used. + +:option:`extra_compile_args` and :option:`extra_link_args` can be used to +specify additional command-line options for the respective compiler and linker +command lines. + +:option:`export_symbols` is only useful on Windows. It can contain a list of +symbols (functions or variables) to be exported. This option is not needed when +building compiled extensions: Distutils will automatically add ``initmodule`` +to the list of exported symbols. + +The :option:`depends` option is a list of files that the extension depends on +(for example header files). The build command will call the compiler on the +sources to rebuild extension if any on this files has been modified since the +previous build. + +Relationships between Distributions and Packages +================================================ + +.. FIXME rewrite to update to PEP 345 (but without dist/release confusion) + +A distribution may relate to packages in three specific ways: + +#. It can require packages or modules. + +#. It can provide packages or modules. + +#. It can obsolete packages or modules. + +These relationships can be specified using keyword arguments to the +:func:`packaging.core.setup` function. + +Dependencies on other Python modules and packages can be specified by supplying +the *requires* keyword argument to :func:`setup`. The value must be a list of +strings. Each string specifies a package that is required, and optionally what +versions are sufficient. + +To specify that any version of a module or package is required, the string +should consist entirely of the module or package name. Examples include +``'mymodule'`` and ``'xml.parsers.expat'``. + +If specific versions are required, a sequence of qualifiers can be supplied in +parentheses. Each qualifier may consist of a comparison operator and a version +number. The accepted comparison operators are:: + + < > == + <= >= != + +These can be combined by using multiple qualifiers separated by commas (and +optional whitespace). In this case, all of the qualifiers must be matched; a +logical AND is used to combine the evaluations. + +Let's look at a bunch of examples: + ++-------------------------+----------------------------------------------+ +| Requires Expression | Explanation | ++=========================+==============================================+ +| ``==1.0`` | Only version ``1.0`` is compatible | ++-------------------------+----------------------------------------------+ +| ``>1.0, !=1.5.1, <2.0`` | Any version after ``1.0`` and before ``2.0`` | +| | is compatible, except ``1.5.1`` | ++-------------------------+----------------------------------------------+ + +Now that we can specify dependencies, we also need to be able to specify what we +provide that other distributions can require. This is done using the *provides* +keyword argument to :func:`setup`. The value for this keyword is a list of +strings, each of which names a Python module or package, and optionally +identifies the version. If the version is not specified, it is assumed to match +that of the distribution. + +Some examples: + ++---------------------+----------------------------------------------+ +| Provides Expression | Explanation | ++=====================+==============================================+ +| ``mypkg`` | Provide ``mypkg``, using the distribution | +| | version | ++---------------------+----------------------------------------------+ +| ``mypkg (1.1)`` | Provide ``mypkg`` version 1.1, regardless of | +| | the distribution version | ++---------------------+----------------------------------------------+ + +A package can declare that it obsoletes other packages using the *obsoletes* +keyword argument. The value for this is similar to that of the *requires* +keyword: a list of strings giving module or package specifiers. Each specifier +consists of a module or package name optionally followed by one or more version +qualifiers. Version qualifiers are given in parentheses after the module or +package name. + +The versions identified by the qualifiers are those that are obsoleted by the +distribution being described. If no qualifiers are given, all versions of the +named module or package are understood to be obsoleted. + +.. _packaging-installing-scripts: + +Installing Scripts +================== + +So far we have been dealing with pure and non-pure Python modules, which are +usually not run by themselves but imported by scripts. + +Scripts are files containing Python source code, intended to be started from the +command line. Scripts don't require Distutils to do anything very complicated. +The only clever feature is that if the first line of the script starts with +``#!`` and contains the word "python", the Distutils will adjust the first line +to refer to the current interpreter location. By default, it is replaced with +the current interpreter location. The :option:`--executable` (or :option:`-e`) +option will allow the interpreter path to be explicitly overridden. + +The :option:`scripts` option simply is a list of files to be handled in this +way. From the PyXML setup script:: + + setup(..., + scripts=['scripts/xmlproc_parse', 'scripts/xmlproc_val']) + +All the scripts will also be added to the ``MANIFEST`` file if no template is +provided. See :ref:`packaging-manifest`. + +.. _packaging-installing-package-data: + +Installing Package Data +======================= + +Often, additional files need to be installed into a package. These files are +often data that's closely related to the package's implementation, or text files +containing documentation that might be of interest to programmers using the +package. These files are called :dfn:`package data`. + +Package data can be added to packages using the ``package_data`` keyword +argument to the :func:`setup` function. The value must be a mapping from +package name to a list of relative path names that should be copied into the +package. The paths are interpreted as relative to the directory containing the +package (information from the ``package_dir`` mapping is used if appropriate); +that is, the files are expected to be part of the package in the source +directories. They may contain glob patterns as well. + +The path names may contain directory portions; any necessary directories will be +created in the installation. + +For example, if a package should contain a subdirectory with several data files, +the files can be arranged like this in the source tree:: + + setup.py + src/ + mypkg/ + __init__.py + module.py + data/ + tables.dat + spoons.dat + forks.dat + +The corresponding call to :func:`setup` might be:: + + setup(..., + packages=['mypkg'], + package_dir={'mypkg': 'src/mypkg'}, + package_data={'mypkg': ['data/*.dat']}) + + +All the files that match ``package_data`` will be added to the ``MANIFEST`` +file if no template is provided. See :ref:`packaging-manifest`. + + +.. _packaging-additional-files: + +Installing Additional Files +=========================== + +The :option:`data_files` option can be used to specify additional files needed +by the module distribution: configuration files, message catalogs, data files, +anything which doesn't fit in the previous categories. + +:option:`data_files` specifies a sequence of (*directory*, *files*) pairs in the +following way:: + + setup(..., + data_files=[('bitmaps', ['bm/b1.gif', 'bm/b2.gif']), + ('config', ['cfg/data.cfg']), + ('/etc/init.d', ['init-script'])]) + +Note that you can specify the directory names where the data files will be +installed, but you cannot rename the data files themselves. + +Each (*directory*, *files*) pair in the sequence specifies the installation +directory and the files to install there. If *directory* is a relative path, it +is interpreted relative to the installation prefix (Python's ``sys.prefix`` for +pure-Python packages, ``sys.exec_prefix`` for packages that contain extension +modules). Each file name in *files* is interpreted relative to the +:file:`setup.py` script at the top of the package source distribution. No +directory information from *files* is used to determine the final location of +the installed file; only the name of the file is used. + +You can specify the :option:`data_files` options as a simple sequence of files +without specifying a target directory, but this is not recommended, and the +:command:`install_dist` command will print a warning in this case. To install data +files directly in the target directory, an empty string should be given as the +directory. + +All the files that match ``data_files`` will be added to the ``MANIFEST`` file +if no template is provided. See :ref:`packaging-manifest`. + + + +.. _packaging-metadata: + +Metadata reference +================== + +The setup script may include additional metadata beyond the name and version. +This table describes required and additional information: + +.. TODO synchronize with setupcfg; link to it (but don't remove it, it's a + useful summary) + ++----------------------+---------------------------+-----------------+--------+ +| Meta-Data | Description | Value | Notes | ++======================+===========================+=================+========+ +| ``name`` | name of the project | short string | \(1) | ++----------------------+---------------------------+-----------------+--------+ +| ``version`` | version of this release | short string | (1)(2) | ++----------------------+---------------------------+-----------------+--------+ +| ``author`` | project author's name | short string | \(3) | ++----------------------+---------------------------+-----------------+--------+ +| ``author_email`` | email address of the | email address | \(3) | +| | project author | | | ++----------------------+---------------------------+-----------------+--------+ +| ``maintainer`` | project maintainer's name | short string | \(3) | ++----------------------+---------------------------+-----------------+--------+ +| ``maintainer_email`` | email address of the | email address | \(3) | +| | project maintainer | | | ++----------------------+---------------------------+-----------------+--------+ +| ``home_page`` | home page for the project | URL | \(1) | ++----------------------+---------------------------+-----------------+--------+ +| ``summary`` | short description of the | short string | | +| | project | | | ++----------------------+---------------------------+-----------------+--------+ +| ``description`` | longer description of the | long string | \(5) | +| | project | | | ++----------------------+---------------------------+-----------------+--------+ +| ``download_url`` | location where the | URL | | +| | project may be downloaded | | | ++----------------------+---------------------------+-----------------+--------+ +| ``classifiers`` | a list of classifiers | list of strings | \(4) | ++----------------------+---------------------------+-----------------+--------+ +| ``platforms`` | a list of platforms | list of strings | | ++----------------------+---------------------------+-----------------+--------+ +| ``license`` | license for the release | short string | \(6) | ++----------------------+---------------------------+-----------------+--------+ + +Notes: + +(1) + These fields are required. + +(2) + It is recommended that versions take the form *major.minor[.patch[.sub]]*. + +(3) + Either the author or the maintainer must be identified. + +(4) + The list of classifiers is available from the `PyPI website + `_. See also :mod:`packaging.create`. + +(5) + The ``description`` field is used by PyPI when you are registering a + release, to build its PyPI page. + +(6) + The ``license`` field is a text indicating the license covering the + distribution where the license is not a selection from the "License" Trove + classifiers. See the ``Classifier`` field. Notice that + there's a ``licence`` distribution option which is deprecated but still + acts as an alias for ``license``. + +'short string' + A single line of text, not more than 200 characters. + +'long string' + Multiple lines of plain text in reStructuredText format (see + http://docutils.sf.net/). + +'list of strings' + See below. + +In Python 2.x, "string value" means a unicode object. If a byte string (str or +bytes) is given, it has to be valid ASCII. + +.. TODO move this section to the version document, keep a summary, add a link + +Encoding the version information is an art in itself. Python projects generally +adhere to the version format *major.minor[.patch][sub]*. The major number is 0 +for initial, experimental releases of software. It is incremented for releases +that represent major milestones in a project. The minor number is incremented +when important new features are added to the project. The patch number +increments when bug-fix releases are made. Additional trailing version +information is sometimes used to indicate sub-releases. These are +"a1,a2,...,aN" (for alpha releases, where functionality and API may change), +"b1,b2,...,bN" (for beta releases, which only fix bugs) and "pr1,pr2,...,prN" +(for final pre-release release testing). Some examples: + +0.1.0 + the first, experimental release of a project + +1.0.1a2 + the second alpha release of the first patch version of 1.0 + +:option:`classifiers` are specified in a Python list:: + + setup(..., + classifiers=[ + 'Development Status :: 4 - Beta', + 'Environment :: Console', + 'Environment :: Web Environment', + 'Intended Audience :: End Users/Desktop', + 'Intended Audience :: Developers', + 'Intended Audience :: System Administrators', + 'License :: OSI Approved :: Python Software Foundation License', + 'Operating System :: MacOS :: MacOS X', + 'Operating System :: Microsoft :: Windows', + 'Operating System :: POSIX', + 'Programming Language :: Python', + 'Topic :: Communications :: Email', + 'Topic :: Office/Business', + 'Topic :: Software Development :: Bug Tracking', + ]) + + +Debugging the setup script +========================== + +Sometimes things go wrong, and the setup script doesn't do what the developer +wants. + +Distutils catches any exceptions when running the setup script, and print a +simple error message before the script is terminated. The motivation for this +behaviour is to not confuse administrators who don't know much about Python and +are trying to install a project. If they get a big long traceback from deep +inside the guts of Distutils, they may think the project or the Python +installation is broken because they don't read all the way down to the bottom +and see that it's a permission problem. + +.. FIXME DISTUTILS_DEBUG is dead, document logging/warnings here + +On the other hand, this doesn't help the developer to find the cause of the +failure. For this purpose, the DISTUTILS_DEBUG environment variable can be set +to anything except an empty string, and Packaging will now print detailed +information about what it is doing, and prints the full traceback in case an +exception occurs. diff --git a/Doc/packaging/sourcedist.rst b/Doc/packaging/sourcedist.rst new file mode 100644 index 00000000000..0cd4df3f815 --- /dev/null +++ b/Doc/packaging/sourcedist.rst @@ -0,0 +1,273 @@ +.. _packaging-source-dist: + +****************************** +Creating a Source Distribution +****************************** + +As shown in section :ref:`packaging-simple-example`, you use the :command:`sdist` command +to create a source distribution. In the simplest case, :: + + python setup.py sdist + +(assuming you haven't specified any :command:`sdist` options in the setup script +or config file), :command:`sdist` creates the archive of the default format for +the current platform. The default format is a gzip'ed tar file +(:file:`.tar.gz`) on Unix, and ZIP file on Windows. + +You can specify as many formats as you like using the :option:`--formats` +option, for example:: + + python setup.py sdist --formats=gztar,zip + +to create a gzipped tarball and a zip file. The available formats are: + ++-----------+-------------------------+---------+ +| Format | Description | Notes | ++===========+=========================+=========+ +| ``zip`` | zip file (:file:`.zip`) | (1),(3) | ++-----------+-------------------------+---------+ +| ``gztar`` | gzip'ed tar file | \(2) | +| | (:file:`.tar.gz`) | | ++-----------+-------------------------+---------+ +| ``bztar`` | bzip2'ed tar file | | +| | (:file:`.tar.bz2`) | | ++-----------+-------------------------+---------+ +| ``ztar`` | compressed tar file | \(4) | +| | (:file:`.tar.Z`) | | ++-----------+-------------------------+---------+ +| ``tar`` | tar file (:file:`.tar`) | | ++-----------+-------------------------+---------+ + +Notes: + +(1) + default on Windows + +(2) + default on Unix + +(3) + requires either external :program:`zip` utility or :mod:`zipfile` module (part + of the standard Python library since Python 1.6) + +(4) + requires the :program:`compress` program. Notice that this format is now + pending for deprecation and will be removed in the future versions of Python. + +When using any ``tar`` format (``gztar``, ``bztar``, ``ztar`` or +``tar``) under Unix, you can specify the ``owner`` and ``group`` names +that will be set for each member of the archive. + +For example, if you want all files of the archive to be owned by root:: + + python setup.py sdist --owner=root --group=root + + +.. _packaging-manifest: + +Specifying the files to distribute +================================== + +If you don't supply an explicit list of files (or instructions on how to +generate one), the :command:`sdist` command puts a minimal default set into the +source distribution: + +* all Python source files implied by the :option:`py_modules` and + :option:`packages` options + +* all C source files mentioned in the :option:`ext_modules` or + :option:`libraries` options + +* scripts identified by the :option:`scripts` option + See :ref:`packaging-installing-scripts`. + +* anything that looks like a test script: :file:`test/test\*.py` (currently, the + Packaging don't do anything with test scripts except include them in source + distributions, but in the future there will be a standard for testing Python + module distributions) + +* the configuration file :file:`setup.cfg` + +* all files that matches the ``package_data`` metadata. + See :ref:`packaging-installing-package-data`. + +* all files that matches the ``data_files`` metadata. + See :ref:`packaging-additional-files`. + +Contrary to Distutils, :file:`README` (or :file:`README.txt`) and +:file:`setup.py` are not included by default. + +Sometimes this is enough, but usually you will want to specify additional files +to distribute. The typical way to do this is to write a *manifest template*, +called :file:`MANIFEST.in` by default. The manifest template is just a list of +instructions for how to generate your manifest file, :file:`MANIFEST`, which is +the exact list of files to include in your source distribution. The +:command:`sdist` command processes this template and generates a manifest based +on its instructions and what it finds in the filesystem. + +If you prefer to roll your own manifest file, the format is simple: one filename +per line, regular files (or symlinks to them) only. If you do supply your own +:file:`MANIFEST`, you must specify everything: the default set of files +described above does not apply in this case. + +:file:`MANIFEST` files start with a comment indicating they are generated. +Files without this comment are not overwritten or removed. + +See :ref:`packaging-manifest-template` section for a syntax reference. + + +.. _packaging-manifest-options: + +Manifest-related options +======================== + +The normal course of operations for the :command:`sdist` command is as follows: + +* if the manifest file, :file:`MANIFEST` doesn't exist, read :file:`MANIFEST.in` + and create the manifest + +* if neither :file:`MANIFEST` nor :file:`MANIFEST.in` exist, create a manifest + with just the default file set + +* if either :file:`MANIFEST.in` or the setup script (:file:`setup.py`) are more + recent than :file:`MANIFEST`, recreate :file:`MANIFEST` by reading + :file:`MANIFEST.in` + +* use the list of files now in :file:`MANIFEST` (either just generated or read + in) to create the source distribution archive(s) + +There are a couple of options that modify this behaviour. First, use the +:option:`--no-defaults` and :option:`--no-prune` to disable the standard +"include" and "exclude" sets. + +Second, you might just want to (re)generate the manifest, but not create a +source distribution:: + + python setup.py sdist --manifest-only + +:option:`-o` is a shortcut for :option:`--manifest-only`. + + +.. _packaging-manifest-template: + +The MANIFEST.in template +======================== + +A :file:`MANIFEST.in` file can be added in a project to define the list of +files to include in the distribution built by the :command:`sdist` command. + +When :command:`sdist` is run, it will look for the :file:`MANIFEST.in` file +and interpret it to generate the :file:`MANIFEST` file that contains the +list of files that will be included in the package. + +This mechanism can be used when the default list of files is not enough. +(See :ref:`packaging-manifest`). + +Principle +--------- + +The manifest template has one command per line, where each command specifies a +set of files to include or exclude from the source distribution. For an +example, let's look at the Packaging' own manifest template:: + + include *.txt + recursive-include examples *.txt *.py + prune examples/sample?/build + +The meanings should be fairly clear: include all files in the distribution root +matching :file:`\*.txt`, all files anywhere under the :file:`examples` directory +matching :file:`\*.txt` or :file:`\*.py`, and exclude all directories matching +:file:`examples/sample?/build`. All of this is done *after* the standard +include set, so you can exclude files from the standard set with explicit +instructions in the manifest template. (Or, you can use the +:option:`--no-defaults` option to disable the standard set entirely.) + +The order of commands in the manifest template matters: initially, we have the +list of default files as described above, and each command in the template adds +to or removes from that list of files. Once we have fully processed the +manifest template, we remove files that should not be included in the source +distribution: + +* all files in the Packaging "build" tree (default :file:`build/`) + +* all files in directories named :file:`RCS`, :file:`CVS`, :file:`.svn`, + :file:`.hg`, :file:`.git`, :file:`.bzr` or :file:`_darcs` + +Now we have our complete list of files, which is written to the manifest for +future reference, and then used to build the source distribution archive(s). + +You can disable the default set of included files with the +:option:`--no-defaults` option, and you can disable the standard exclude set +with :option:`--no-prune`. + +Following the Packaging' own manifest template, let's trace how the +:command:`sdist` command builds the list of files to include in the Packaging +source distribution: + +#. include all Python source files in the :file:`packaging` and + :file:`packaging/command` subdirectories (because packages corresponding to + those two directories were mentioned in the :option:`packages` option in the + setup script---see section :ref:`packaging-setup-script`) + +#. include :file:`README.txt`, :file:`setup.py`, and :file:`setup.cfg` (standard + files) + +#. include :file:`test/test\*.py` (standard files) + +#. include :file:`\*.txt` in the distribution root (this will find + :file:`README.txt` a second time, but such redundancies are weeded out later) + +#. include anything matching :file:`\*.txt` or :file:`\*.py` in the sub-tree + under :file:`examples`, + +#. exclude all files in the sub-trees starting at directories matching + :file:`examples/sample?/build`\ ---this may exclude files included by the + previous two steps, so it's important that the ``prune`` command in the manifest + template comes after the ``recursive-include`` command + +#. exclude the entire :file:`build` tree, and any :file:`RCS`, :file:`CVS`, + :file:`.svn`, :file:`.hg`, :file:`.git`, :file:`.bzr` and :file:`_darcs` + directories + +Just like in the setup script, file and directory names in the manifest template +should always be slash-separated; the Packaging will take care of converting +them to the standard representation on your platform. That way, the manifest +template is portable across operating systems. + +Commands +-------- + +The manifest template commands are: + ++-------------------------------------------+-----------------------------------------------+ +| Command | Description | ++===========================================+===============================================+ +| :command:`include pat1 pat2 ...` | include all files matching any of the listed | +| | patterns | ++-------------------------------------------+-----------------------------------------------+ +| :command:`exclude pat1 pat2 ...` | exclude all files matching any of the listed | +| | patterns | ++-------------------------------------------+-----------------------------------------------+ +| :command:`recursive-include dir pat1 pat2 | include all files under *dir* matching any of | +| ...` | the listed patterns | ++-------------------------------------------+-----------------------------------------------+ +| :command:`recursive-exclude dir pat1 pat2 | exclude all files under *dir* matching any of | +| ...` | the listed patterns | ++-------------------------------------------+-----------------------------------------------+ +| :command:`global-include pat1 pat2 ...` | include all files anywhere in the source tree | +| | matching --- & any of the listed patterns | ++-------------------------------------------+-----------------------------------------------+ +| :command:`global-exclude pat1 pat2 ...` | exclude all files anywhere in the source tree | +| | matching --- & any of the listed patterns | ++-------------------------------------------+-----------------------------------------------+ +| :command:`prune dir` | exclude all files under *dir* | ++-------------------------------------------+-----------------------------------------------+ +| :command:`graft dir` | include all files under *dir* | ++-------------------------------------------+-----------------------------------------------+ + +The patterns here are Unix-style "glob" patterns: ``*`` matches any sequence of +regular filename characters, ``?`` matches any single regular filename +character, and ``[range]`` matches any of the characters in *range* (e.g., +``a-z``, ``a-zA-Z``, ``a-f0-9_.``). The definition of "regular filename +character" is platform-specific: on Unix it is anything except slash; on Windows +anything except backslash or colon. diff --git a/Doc/packaging/tutorial.rst b/Doc/packaging/tutorial.rst new file mode 100644 index 00000000000..04f41e519de --- /dev/null +++ b/Doc/packaging/tutorial.rst @@ -0,0 +1,112 @@ +================== +Packaging tutorial +================== + +Welcome to the Packaging tutorial! We will learn how to use Packaging +to package your project. + +.. TODO merge with introduction.rst + + +Getting started +--------------- + +Packaging works with the *setup.cfg* file. It contains all the metadata for +your project, as defined in PEP 345, but also declare what your project +contains. + +Let's say you have a project called *CLVault* containing one package called +*clvault*, and a few scripts inside. You can use the *pysetup* script to create +a *setup.cfg* file for the project. The script will ask you a few questions:: + + $ mkdir CLVault + $ cd CLVault + $ pysetup create + Project name [CLVault]: + Current version number: 0.1 + Package description: + >Command-line utility to store and retrieve passwords + Author name: Tarek Ziade + Author e-mail address: tarek@ziade.org + Project Home Page: http://bitbucket.org/tarek/clvault + Do you want to add a package ? (y/n): y + Package name: clvault + Do you want to add a package ? (y/n): n + Do you want to set Trove classifiers? (y/n): y + Please select the project status: + + 1 - Planning + 2 - Pre-Alpha + 3 - Alpha + 4 - Beta + 5 - Production/Stable + 6 - Mature + 7 - Inactive + + Status: 3 + What license do you use: GPL + Matching licenses: + + 1) License :: OSI Approved :: GNU General Public License (GPL) + 2) License :: OSI Approved :: GNU Library or Lesser General Public License (LGPL) + + Type the number of the license you wish to use or ? to try again:: 1 + Do you want to set other trove identifiers (y/n) [n]: n + Wrote "setup.cfg". + + +A setup.cfg file is created, containing the metadata of your project and the +list of the packages it contains:: + + $ cat setup.cfg + [metadata] + name = CLVault + version = 0.1 + author = Tarek Ziade + author_email = tarek@ziade.org + description = Command-line utility to store and retrieve passwords + home_page = http://bitbucket.org/tarek/clvault + + classifier = Development Status :: 3 - Alpha + License :: OSI Approved :: GNU General Public License (GPL) + + [files] + packages = clvault + + +Our project will depend on the *keyring* project. Let's add it in the +[metadata] section:: + + [metadata] + ... + requires_dist = + keyring + + +Running commands +---------------- + +You can run useful commands on your project once the setup.cfg file is ready: + +- sdist: creates a source distribution +- register: register your project to PyPI +- upload: upload the distribution to PyPI +- install_dist: install it + +All commands are run using the run script:: + + $ pysetup run install_dist + $ pysetup run sdist + $ pysetup run upload + +If you want to push a source distribution of your project to PyPI, do:: + + $ pysetup run sdist register upload + + +Installing the project +---------------------- + +The project can be installed by manually running the packaging install command:: + + $ pysetup run install_dist diff --git a/Doc/packaging/uploading.rst b/Doc/packaging/uploading.rst new file mode 100644 index 00000000000..297518bb837 --- /dev/null +++ b/Doc/packaging/uploading.rst @@ -0,0 +1,80 @@ +.. _packaging-package-upload: + +*************************************** +Uploading Packages to the Package Index +*************************************** + +The Python Package Index (PyPI) not only stores the package info, but also the +package data if the author of the package wishes to. The packaging command +:command:`upload` pushes the distribution files to PyPI. + +The command is invoked immediately after building one or more distribution +files. For example, the command :: + + python setup.py sdist bdist_wininst upload + +will cause the source distribution and the Windows installer to be uploaded to +PyPI. Note that these will be uploaded even if they are built using an earlier +invocation of :file:`setup.py`, but that only distributions named on the command +line for the invocation including the :command:`upload` command are uploaded. + +The :command:`upload` command uses the username, password, and repository URL +from the :file:`$HOME/.pypirc` file (see section :ref:`packaging-pypirc` for more on this +file). If a :command:`register` command was previously called in the same +command, and if the password was entered in the prompt, :command:`upload` will +reuse the entered password. This is useful if you do not want to store a clear +text password in the :file:`$HOME/.pypirc` file. + +You can specify another PyPI server with the :option:`--repository=*url*` +option:: + + python setup.py sdist bdist_wininst upload -r http://example.com/pypi + +See section :ref:`packaging-pypirc` for more on defining several servers. + +You can use the :option:`--sign` option to tell :command:`upload` to sign each +uploaded file using GPG (GNU Privacy Guard). The :program:`gpg` program must +be available for execution on the system :envvar:`PATH`. You can also specify +which key to use for signing using the :option:`--identity=*name*` option. + +Other :command:`upload` options include :option:`--repository=` or +:option:`--repository=
` where *url* is the url of the server and +*section* the name of the section in :file:`$HOME/.pypirc`, and +:option:`--show-response` (which displays the full response text from the PyPI +server for help in debugging upload problems). + +PyPI package display +==================== + +The ``description`` field plays a special role at PyPI. It is used by +the server to display a home page for the registered package. + +If you use the `reStructuredText `_ +syntax for this field, PyPI will parse it and display an HTML output for +the package home page. + +The ``description`` field can be filled from a text file located in the +project:: + + from packaging.core import setup + + fp = open('README.txt') + try: + description = fp.read() + finally: + fp.close() + + setup(name='Packaging', + description=description) + +In that case, :file:`README.txt` is a regular reStructuredText text file located +in the root of the package besides :file:`setup.py`. + +To prevent registering broken reStructuredText content, you can use the +:program:`rst2html` program that is provided by the :mod:`docutils` package +and check the ``description`` from the command line:: + + $ python setup.py --description | rst2html.py > output.html + +:mod:`docutils` will display a warning if there's something wrong with your +syntax. diff --git a/Doc/tools/sphinxext/indexcontent.html b/Doc/tools/sphinxext/indexcontent.html index d5e17cd62b1..10f9d266ef8 100644 --- a/Doc/tools/sphinxext/indexcontent.html +++ b/Doc/tools/sphinxext/indexcontent.html @@ -20,9 +20,9 @@ tutorial for C/C++ programmers

Python/C API
reference for C/C++ programmers

-

Installing Python Modules
+

Installing Python Projects
information for installers & sys-admins

-

Distributing Python Modules
+

Distributing Python Projects
sharing modules with others

Documenting Python
guide for documentation authors