mirror of https://github.com/python/cpython
254 lines
10 KiB
TeX
254 lines
10 KiB
TeX
\section{\module{urlparse} ---
|
|
Parse URLs into components}
|
|
\declaremodule{standard}{urlparse}
|
|
|
|
\modulesynopsis{Parse URLs into components.}
|
|
|
|
\index{WWW}
|
|
\index{World Wide Web}
|
|
\index{URL}
|
|
\indexii{URL}{parsing}
|
|
\indexii{relative}{URL}
|
|
|
|
|
|
This module defines a standard interface to break Uniform Resource
|
|
Locator (URL) strings up in components (addressing scheme, network
|
|
location, path etc.), to combine the components back into a URL
|
|
string, and to convert a ``relative URL'' to an absolute URL given a
|
|
``base URL.''
|
|
|
|
The module has been designed to match the Internet RFC on Relative
|
|
Uniform Resource Locators (and discovered a bug in an earlier
|
|
draft!). It supports the following URL schemes:
|
|
\code{file}, \code{ftp}, \code{gopher}, \code{hdl}, \code{http},
|
|
\code{https}, \code{imap}, \code{mailto}, \code{mms}, \code{news},
|
|
\code{nntp}, \code{prospero}, \code{rsync}, \code{rtsp}, \code{rtspu},
|
|
\code{sftp}, \code{shttp}, \code{sip}, \code{sips}, \code{snews}, \code{svn},
|
|
\code{svn+ssh}, \code{telnet}, \code{wais}.
|
|
|
|
\versionadded[Support for the \code{sftp} and \code{sips} schemes]{2.5}
|
|
|
|
The \module{urlparse} module defines the following functions:
|
|
|
|
\begin{funcdesc}{urlparse}{urlstring\optional{,
|
|
default_scheme\optional{, allow_fragments}}}
|
|
Parse a URL into six components, returning a 6-tuple. This
|
|
corresponds to the general structure of a URL:
|
|
\code{\var{scheme}://\var{netloc}/\var{path};\var{parameters}?\var{query}\#\var{fragment}}.
|
|
Each tuple item is a string, possibly empty.
|
|
The components are not broken up in smaller parts (for example, the network
|
|
location is a single string), and \% escapes are not expanded.
|
|
The delimiters as shown above are not part of the result,
|
|
except for a leading slash in the \var{path} component, which is
|
|
retained if present. For example:
|
|
|
|
\begin{verbatim}
|
|
>>> from urlparse import urlparse
|
|
>>> o = urlparse('http://www.cwi.nl:80/%7Eguido/Python.html')
|
|
>>> o
|
|
('http', 'www.cwi.nl:80', '/%7Eguido/Python.html', '', '', '')
|
|
>>> o.scheme
|
|
'http'
|
|
>>> o.port
|
|
80
|
|
>>> o.geturl()
|
|
'http://www.cwi.nl:80/%7Eguido/Python.html'
|
|
\end{verbatim}
|
|
|
|
If the \var{default_scheme} argument is specified, it gives the
|
|
default addressing scheme, to be used only if the URL does not
|
|
specify one. The default value for this argument is the empty string.
|
|
|
|
If the \var{allow_fragments} argument is false, fragment identifiers
|
|
are not allowed, even if the URL's addressing scheme normally does
|
|
support them. The default value for this argument is \constant{True}.
|
|
|
|
The return value is actually an instance of a subclass of
|
|
\pytype{tuple}. This class has the following additional read-only
|
|
convenience attributes:
|
|
|
|
\begin{tableiv}{l|c|l|c}{member}{Attribute}{Index}{Value}{Value if not present}
|
|
\lineiv{scheme} {0} {URL scheme specifier} {empty string}
|
|
\lineiv{netloc} {1} {Network location part} {empty string}
|
|
\lineiv{path} {2} {Hierarchical path} {empty string}
|
|
\lineiv{params} {3} {Parameters for last path element} {empty string}
|
|
\lineiv{query} {4} {Query component} {empty string}
|
|
\lineiv{fragment}{5} {Fragment identifier} {empty string}
|
|
\lineiv{username}{ } {User name} {\constant{None}}
|
|
\lineiv{password}{ } {Password} {\constant{None}}
|
|
\lineiv{hostname}{ } {Host name (lower case)} {\constant{None}}
|
|
\lineiv{port} { } {Port number as integer, if present} {\constant{None}}
|
|
\end{tableiv}
|
|
|
|
See section~\ref{urlparse-result-object}, ``Results of
|
|
\function{urlparse()} and \function{urlsplit()},'' for more
|
|
information on the result object.
|
|
|
|
\versionchanged[Added attributes to return value]{2.5}
|
|
\end{funcdesc}
|
|
|
|
\begin{funcdesc}{urlunparse}{parts}
|
|
Construct a URL from a tuple as returned by \code{urlparse()}.
|
|
The \var{parts} argument can be any six-item iterable.
|
|
This may result in a slightly different, but equivalent URL, if the
|
|
URL that was parsed originally had unnecessary delimiters (for example,
|
|
a ? with an empty query; the RFC states that these are equivalent).
|
|
\end{funcdesc}
|
|
|
|
\begin{funcdesc}{urlsplit}{urlstring\optional{,
|
|
default_scheme\optional{, allow_fragments}}}
|
|
This is similar to \function{urlparse()}, but does not split the
|
|
params from the URL. This should generally be used instead of
|
|
\function{urlparse()} if the more recent URL syntax allowing
|
|
parameters to be applied to each segment of the \var{path} portion of
|
|
the URL (see \rfc{2396}) is wanted. A separate function is needed to
|
|
separate the path segments and parameters. This function returns a
|
|
5-tuple: (addressing scheme, network location, path, query, fragment
|
|
identifier).
|
|
|
|
The return value is actually an instance of a subclass of
|
|
\pytype{tuple}. This class has the following additional read-only
|
|
convenience attributes:
|
|
|
|
\begin{tableiv}{l|c|l|c}{member}{Attribute}{Index}{Value}{Value if not present}
|
|
\lineiv{scheme} {0} {URL scheme specifier} {empty string}
|
|
\lineiv{netloc} {1} {Network location part} {empty string}
|
|
\lineiv{path} {2} {Hierarchical path} {empty string}
|
|
\lineiv{query} {3} {Query component} {empty string}
|
|
\lineiv{fragment} {4} {Fragment identifier} {empty string}
|
|
\lineiv{username} { } {User name} {\constant{None}}
|
|
\lineiv{password} { } {Password} {\constant{None}}
|
|
\lineiv{hostname} { } {Host name (lower case)} {\constant{None}}
|
|
\lineiv{port} { } {Port number as integer, if present} {\constant{None}}
|
|
\end{tableiv}
|
|
|
|
See section~\ref{urlparse-result-object}, ``Results of
|
|
\function{urlparse()} and \function{urlsplit()},'' for more
|
|
information on the result object.
|
|
|
|
\versionadded{2.2}
|
|
\versionchanged[Added attributes to return value]{2.5}
|
|
\end{funcdesc}
|
|
|
|
\begin{funcdesc}{urlunsplit}{parts}
|
|
Combine the elements of a tuple as returned by \function{urlsplit()}
|
|
into a complete URL as a string.
|
|
The \var{parts} argument can be any five-item iterable.
|
|
This may result in a slightly different, but equivalent URL, if the
|
|
URL that was parsed originally had unnecessary delimiters (for example,
|
|
a ? with an empty query; the RFC states that these are equivalent).
|
|
\versionadded{2.2}
|
|
\end{funcdesc}
|
|
|
|
\begin{funcdesc}{urljoin}{base, url\optional{, allow_fragments}}
|
|
Construct a full (``absolute'') URL by combining a ``base URL''
|
|
(\var{base}) with another URL (\var{url}). Informally, this
|
|
uses components of the base URL, in particular the addressing scheme,
|
|
the network location and (part of) the path, to provide missing
|
|
components in the relative URL. For example:
|
|
|
|
\begin{verbatim}
|
|
>>> from urlparse import urljoin
|
|
>>> urljoin('http://www.cwi.nl/%7Eguido/Python.html', 'FAQ.html')
|
|
'http://www.cwi.nl/%7Eguido/FAQ.html'
|
|
\end{verbatim}
|
|
|
|
The \var{allow_fragments} argument has the same meaning and default as
|
|
for \function{urlparse()}.
|
|
|
|
\note{If \var{url} is an absolute URL (that is, starting with \code{//}
|
|
or \code{scheme://}, the \var{url}'s host name and/or scheme
|
|
will be present in the result. For example:}
|
|
|
|
\begin{verbatim}
|
|
>>> urljoin('http://www.cwi.nl/%7Eguido/Python.html',
|
|
... '//www.python.org/%7Eguido')
|
|
'http://www.python.org/%7Eguido'
|
|
\end{verbatim}
|
|
|
|
If you do not want that behavior, preprocess
|
|
the \var{url} with \function{urlsplit()} and \function{urlunsplit()},
|
|
removing possible \emph{scheme} and \emph{netloc} parts.
|
|
\end{funcdesc}
|
|
|
|
\begin{funcdesc}{urldefrag}{url}
|
|
If \var{url} contains a fragment identifier, returns a modified
|
|
version of \var{url} with no fragment identifier, and the fragment
|
|
identifier as a separate string. If there is no fragment identifier
|
|
in \var{url}, returns \var{url} unmodified and an empty string.
|
|
\end{funcdesc}
|
|
|
|
|
|
\begin{seealso}
|
|
\seerfc{1738}{Uniform Resource Locators (URL)}{
|
|
This specifies the formal syntax and semantics of absolute
|
|
URLs.}
|
|
\seerfc{1808}{Relative Uniform Resource Locators}{
|
|
This Request For Comments includes the rules for joining an
|
|
absolute and a relative URL, including a fair number of
|
|
``Abnormal Examples'' which govern the treatment of border
|
|
cases.}
|
|
\seerfc{2396}{Uniform Resource Identifiers (URI): Generic Syntax}{
|
|
Document describing the generic syntactic requirements for
|
|
both Uniform Resource Names (URNs) and Uniform Resource
|
|
Locators (URLs).}
|
|
\end{seealso}
|
|
|
|
|
|
\subsection{Results of \function{urlparse()} and \function{urlsplit()}
|
|
\label{urlparse-result-object}}
|
|
|
|
The result objects from the \function{urlparse()} and
|
|
\function{urlsplit()} functions are subclasses of the \pytype{tuple}
|
|
type. These subclasses add the attributes described in those
|
|
functions, as well as provide an additional method:
|
|
|
|
\begin{methoddesc}[ParseResult]{geturl}{}
|
|
Return the re-combined version of the original URL as a string.
|
|
This may differ from the original URL in that the scheme will always
|
|
be normalized to lower case and empty components may be dropped.
|
|
Specifically, empty parameters, queries, and fragment identifiers
|
|
will be removed.
|
|
|
|
The result of this method is a fixpoint if passed back through the
|
|
original parsing function:
|
|
|
|
\begin{verbatim}
|
|
>>> import urlparse
|
|
>>> url = 'HTTP://www.Python.org/doc/#'
|
|
|
|
>>> r1 = urlparse.urlsplit(url)
|
|
>>> r1.geturl()
|
|
'http://www.Python.org/doc/'
|
|
|
|
>>> r2 = urlparse.urlsplit(r1.geturl())
|
|
>>> r2.geturl()
|
|
'http://www.Python.org/doc/'
|
|
\end{verbatim}
|
|
|
|
\versionadded{2.5}
|
|
\end{methoddesc}
|
|
|
|
The following classes provide the implementations of the parse results::
|
|
|
|
\begin{classdesc*}{BaseResult}
|
|
Base class for the concrete result classes. This provides most of
|
|
the attribute definitions. It does not provide a \method{geturl()}
|
|
method. It is derived from \class{tuple}, but does not override the
|
|
\method{__init__()} or \method{__new__()} methods.
|
|
\end{classdesc*}
|
|
|
|
|
|
\begin{classdesc}{ParseResult}{scheme, netloc, path, params, query, fragment}
|
|
Concrete class for \function{urlparse()} results. The
|
|
\method{__new__()} method is overridden to support checking that the
|
|
right number of arguments are passed.
|
|
\end{classdesc}
|
|
|
|
|
|
\begin{classdesc}{SplitResult}{scheme, netloc, path, query, fragment}
|
|
Concrete class for \function{urlsplit()} results. The
|
|
\method{__new__()} method is overridden to support checking that the
|
|
right number of arguments are passed.
|
|
\end{classdesc}
|