2007-08-15 11:28:01 -03:00
|
|
|
|
2008-05-20 03:08:38 -03:00
|
|
|
:mod:`HTMLParser` --- Simple HTML and XHTML parser
|
|
|
|
|
==================================================
|
2008-05-17 18:23:02 -03:00
|
|
|
|
2008-05-20 03:08:38 -03:00
|
|
|
.. module:: HTMLParser
|
2007-08-15 11:28:01 -03:00
|
|
|
:synopsis: A simple parser that can handle HTML and XHTML.
|
|
|
|
|
|
2008-05-17 18:23:02 -03:00
|
|
|
.. note::
|
2008-05-20 04:21:58 -03:00
|
|
|
|
|
|
|
|
The :mod:`HTMLParser` module has been renamed to :mod:`html.parser` in Python
|
2011-10-28 08:20:08 -03:00
|
|
|
3. The :term:`2to3` tool will automatically adapt imports when converting
|
|
|
|
|
your sources to Python 3.
|
2008-05-17 18:23:02 -03:00
|
|
|
|
2007-08-15 11:28:01 -03:00
|
|
|
|
|
|
|
|
.. versionadded:: 2.2
|
|
|
|
|
|
|
|
|
|
.. index::
|
|
|
|
|
single: HTML
|
|
|
|
|
single: XHTML
|
|
|
|
|
|
2011-08-18 21:14:03 -03:00
|
|
|
**Source code:** :source:`Lib/HTMLParser.py`
|
|
|
|
|
|
|
|
|
|
--------------
|
|
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
This module defines a class :class:`.HTMLParser` which serves as the basis for
|
2007-08-15 11:28:01 -03:00
|
|
|
parsing text files formatted in HTML (HyperText Mark-up Language) and XHTML.
|
|
|
|
|
Unlike the parser in :mod:`htmllib`, this parser is not based on the SGML parser
|
|
|
|
|
in :mod:`sgmllib`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
.. class:: HTMLParser()
|
|
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
An :class:`.HTMLParser` instance is fed HTML data and calls handler methods
|
|
|
|
|
when start tags, end tags, text, comments, and other markup elements are
|
|
|
|
|
encountered. The user should subclass :class:`.HTMLParser` and override its
|
|
|
|
|
methods to implement the desired behavior.
|
2007-08-15 11:28:01 -03:00
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
The :class:`.HTMLParser` class is instantiated without arguments.
|
2007-08-15 11:28:01 -03:00
|
|
|
|
|
|
|
|
Unlike the parser in :mod:`htmllib`, this parser does not check that end tags
|
|
|
|
|
match start tags or call the end-tag handler for elements which are closed
|
|
|
|
|
implicitly by closing an outer element.
|
|
|
|
|
|
|
|
|
|
An exception is defined as well:
|
|
|
|
|
|
|
|
|
|
.. exception:: HTMLParseError
|
|
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
:class:`.HTMLParser` is able to handle broken markup, but in some cases it
|
|
|
|
|
might raise this exception when it encounters an error while parsing.
|
|
|
|
|
This exception provides three attributes: :attr:`msg` is a brief
|
|
|
|
|
message explaining the error, :attr:`lineno` is the number of the line on
|
|
|
|
|
which the broken construct was detected, and :attr:`offset` is the number of
|
2007-08-15 11:28:01 -03:00
|
|
|
characters into the line at which the construct starts.
|
|
|
|
|
|
|
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
Example HTML Parser Application
|
|
|
|
|
-------------------------------
|
2007-08-15 11:28:01 -03:00
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
As a basic example, below is a simple HTML parser that uses the
|
|
|
|
|
:class:`.HTMLParser` class to print out start tags, end tags and data
|
|
|
|
|
as they are encountered::
|
|
|
|
|
|
|
|
|
|
from HTMLParser import HTMLParser
|
|
|
|
|
|
|
|
|
|
# create a subclass and override the handler methods
|
|
|
|
|
class MyHTMLParser(HTMLParser):
|
|
|
|
|
def handle_starttag(self, tag, attrs):
|
|
|
|
|
print "Encountered a start tag:", tag
|
|
|
|
|
def handle_endtag(self, tag):
|
|
|
|
|
print "Encountered an end tag :", tag
|
|
|
|
|
def handle_data(self, data):
|
|
|
|
|
print "Encountered some data :", data
|
|
|
|
|
|
|
|
|
|
# instantiate the parser and fed it some HTML
|
|
|
|
|
parser = MyHTMLParser()
|
|
|
|
|
parser.feed('<html><head><title>Test</title></head>'
|
|
|
|
|
'<body><h1>Parse me!</h1></body></html>')
|
|
|
|
|
|
|
|
|
|
The output will then be::
|
|
|
|
|
|
|
|
|
|
Encountered a start tag: html
|
|
|
|
|
Encountered a start tag: head
|
|
|
|
|
Encountered a start tag: title
|
|
|
|
|
Encountered some data : Test
|
|
|
|
|
Encountered an end tag : title
|
|
|
|
|
Encountered an end tag : head
|
|
|
|
|
Encountered a start tag: body
|
|
|
|
|
Encountered a start tag: h1
|
|
|
|
|
Encountered some data : Parse me!
|
|
|
|
|
Encountered an end tag : h1
|
|
|
|
|
Encountered an end tag : body
|
|
|
|
|
Encountered an end tag : html
|
2007-08-15 11:28:01 -03:00
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
|
|
|
|
|
:class:`.HTMLParser` Methods
|
|
|
|
|
----------------------------
|
|
|
|
|
|
|
|
|
|
:class:`.HTMLParser` instances have the following methods:
|
2007-08-15 11:28:01 -03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
.. method:: HTMLParser.feed(data)
|
|
|
|
|
|
|
|
|
|
Feed some text to the parser. It is processed insofar as it consists of
|
|
|
|
|
complete elements; incomplete data is buffered until more data is fed or
|
2011-12-19 01:15:26 -04:00
|
|
|
:meth:`close` is called. *data* can be either :class:`unicode` or
|
|
|
|
|
:class:`str`, but passing :class:`unicode` is advised.
|
2007-08-15 11:28:01 -03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
.. method:: HTMLParser.close()
|
|
|
|
|
|
|
|
|
|
Force processing of all buffered data as if it were followed by an end-of-file
|
|
|
|
|
mark. This method may be redefined by a derived class to define additional
|
|
|
|
|
processing at the end of the input, but the redefined version should always call
|
2012-02-17 19:46:04 -04:00
|
|
|
the :class:`.HTMLParser` base class method :meth:`close`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
.. method:: HTMLParser.reset()
|
|
|
|
|
|
|
|
|
|
Reset the instance. Loses all unprocessed data. This is called implicitly at
|
|
|
|
|
instantiation time.
|
2007-08-15 11:28:01 -03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
.. method:: HTMLParser.getpos()
|
|
|
|
|
|
|
|
|
|
Return current line number and offset.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
.. method:: HTMLParser.get_starttag_text()
|
|
|
|
|
|
|
|
|
|
Return the text of the most recently opened start tag. This should not normally
|
|
|
|
|
be needed for structured processing, but may be useful in dealing with HTML "as
|
|
|
|
|
deployed" or for re-generating input with minimal changes (whitespace between
|
|
|
|
|
attributes can be preserved, etc.).
|
|
|
|
|
|
|
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
The following methods are called when data or markup elements are encountered
|
|
|
|
|
and they are meant to be overridden in a subclass. The base class
|
|
|
|
|
implementations do nothing (except for :meth:`~HTMLParser.handle_startendtag`):
|
|
|
|
|
|
|
|
|
|
|
2007-08-15 11:28:01 -03:00
|
|
|
.. method:: HTMLParser.handle_starttag(tag, attrs)
|
|
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
This method is called to handle the start of a tag (e.g. ``<div id="main">``).
|
2007-08-15 11:28:01 -03:00
|
|
|
|
|
|
|
|
The *tag* argument is the name of the tag converted to lower case. The *attrs*
|
|
|
|
|
argument is a list of ``(name, value)`` pairs containing the attributes found
|
|
|
|
|
inside the tag's ``<>`` brackets. The *name* will be translated to lower case,
|
|
|
|
|
and quotes in the *value* have been removed, and character and entity references
|
2012-02-17 19:46:04 -04:00
|
|
|
have been replaced.
|
|
|
|
|
|
|
|
|
|
For instance, for the tag ``<A HREF="http://www.cwi.nl/">``, this method
|
|
|
|
|
would be called as ``handle_starttag('a', [('href', 'http://www.cwi.nl/')])``.
|
2007-08-15 11:28:01 -03:00
|
|
|
|
|
|
|
|
.. versionchanged:: 2.6
|
2012-02-17 19:46:04 -04:00
|
|
|
All entity references from :mod:`htmlentitydefs` are now replaced in the
|
|
|
|
|
attribute values.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
.. method:: HTMLParser.handle_endtag(tag)
|
|
|
|
|
|
|
|
|
|
This method is called to handle the end tag of an element (e.g. ``</div>``).
|
|
|
|
|
|
|
|
|
|
The *tag* argument is the name of the tag converted to lower case.
|
2007-08-15 11:28:01 -03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
.. method:: HTMLParser.handle_startendtag(tag, attrs)
|
|
|
|
|
|
|
|
|
|
Similar to :meth:`handle_starttag`, but called when the parser encounters an
|
2011-10-28 08:20:08 -03:00
|
|
|
XHTML-style empty tag (``<img ... />``). This method may be overridden by
|
2007-08-15 11:28:01 -03:00
|
|
|
subclasses which require this particular lexical information; the default
|
2011-10-28 08:20:08 -03:00
|
|
|
implementation simply calls :meth:`handle_starttag` and :meth:`handle_endtag`.
|
2007-08-15 11:28:01 -03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
.. method:: HTMLParser.handle_data(data)
|
|
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
This method is called to process arbitrary data (e.g. text nodes and the
|
|
|
|
|
content of ``<script>...</script>`` and ``<style>...</style>``).
|
2007-08-15 11:28:01 -03:00
|
|
|
|
|
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
.. method:: HTMLParser.handle_entityref(name)
|
2007-08-15 11:28:01 -03:00
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
This method is called to process a named character reference of the form
|
|
|
|
|
``&name;`` (e.g. ``>``), where *name* is a general entity reference
|
|
|
|
|
(e.g. ``'gt'``).
|
2007-08-15 11:28:01 -03:00
|
|
|
|
|
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
.. method:: HTMLParser.handle_charref(name)
|
2007-08-15 11:28:01 -03:00
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
This method is called to process decimal and hexadecimal numeric character
|
|
|
|
|
references of the form ``&#NNN;`` and ``&#xNNN;``. For example, the decimal
|
|
|
|
|
equivalent for ``>`` is ``>``, whereas the hexadecimal is ``>``;
|
|
|
|
|
in this case the method will receive ``'62'`` or ``'x3E'``.
|
2007-08-15 11:28:01 -03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
.. method:: HTMLParser.handle_comment(data)
|
|
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
This method is called when a comment is encountered (e.g. ``<!--comment-->``).
|
2007-08-15 11:28:01 -03:00
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
For example, the comment ``<!-- comment -->`` will cause this method to be
|
|
|
|
|
called with the argument ``' comment '``.
|
2007-08-15 11:28:01 -03:00
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
The content of Internet Explorer conditional comments (condcoms) will also be
|
|
|
|
|
sent to this method, so, for ``<!--[if IE 9]>IE9-specific content<![endif]-->``,
|
|
|
|
|
this method will receive ``'[if IE 9]>IE-specific content<![endif]'``.
|
2007-08-15 11:28:01 -03:00
|
|
|
|
2010-08-01 18:10:57 -03:00
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
.. method:: HTMLParser.handle_decl(decl)
|
2010-08-01 18:10:57 -03:00
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
This method is called to handle an HTML doctype declaration (e.g.
|
|
|
|
|
``<!DOCTYPE html>``).
|
2010-08-01 18:10:57 -03:00
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
The *decl* parameter will be the entire contents of the declaration inside
|
|
|
|
|
the ``<!...>`` markup (e.g. ``'DOCTYPE html'``).
|
2007-08-15 11:28:01 -03:00
|
|
|
|
|
|
|
|
|
|
|
|
|
.. method:: HTMLParser.handle_pi(data)
|
|
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
This method is called when a processing instruction is encountered. The *data*
|
|
|
|
|
parameter will contain the entire processing instruction. For example, for the
|
2007-08-15 11:28:01 -03:00
|
|
|
processing instruction ``<?proc color='red'>``, this method would be called as
|
2012-02-17 19:46:04 -04:00
|
|
|
``handle_pi("proc color='red'")``.
|
2007-08-15 11:28:01 -03:00
|
|
|
|
|
|
|
|
.. note::
|
|
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
The :class:`.HTMLParser` class uses the SGML syntactic rules for processing
|
2007-08-15 11:28:01 -03:00
|
|
|
instructions. An XHTML processing instruction using the trailing ``'?'`` will
|
|
|
|
|
cause the ``'?'`` to be included in *data*.
|
|
|
|
|
|
|
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
.. method:: HTMLParser.unknown_decl(data)
|
2007-08-15 11:28:01 -03:00
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
This method is called when an unrecognized declaration is read by the parser.
|
|
|
|
|
|
|
|
|
|
The *data* parameter will be the entire contents of the declaration inside
|
|
|
|
|
the ``<![...]>`` markup. It is sometimes useful to be overridden by a
|
|
|
|
|
derived class.
|
2007-08-15 11:28:01 -03:00
|
|
|
|
2012-02-17 19:46:04 -04:00
|
|
|
|
|
|
|
|
.. _htmlparser-examples:
|
|
|
|
|
|
|
|
|
|
Examples
|
|
|
|
|
--------
|
|
|
|
|
|
|
|
|
|
The following class implements a parser that will be used to illustrate more
|
|
|
|
|
examples::
|
2007-08-15 11:28:01 -03:00
|
|
|
|
2008-05-20 03:08:38 -03:00
|
|
|
from HTMLParser import HTMLParser
|
2012-02-17 19:46:04 -04:00
|
|
|
from htmlentitydefs import name2codepoint
|
2007-08-15 11:28:01 -03:00
|
|
|
|
|
|
|
|
class MyHTMLParser(HTMLParser):
|
|
|
|
|
def handle_starttag(self, tag, attrs):
|
2012-02-17 19:46:04 -04:00
|
|
|
print "Start tag:", tag
|
|
|
|
|
for attr in attrs:
|
|
|
|
|
print " attr:", attr
|
2007-08-15 11:28:01 -03:00
|
|
|
def handle_endtag(self, tag):
|
2012-02-17 19:46:04 -04:00
|
|
|
print "End tag :", tag
|
2011-10-28 08:14:34 -03:00
|
|
|
def handle_data(self, data):
|
2012-02-17 19:46:04 -04:00
|
|
|
print "Data :", data
|
|
|
|
|
def handle_comment(self, data):
|
|
|
|
|
print "Comment :", data
|
|
|
|
|
def handle_entityref(self, name):
|
|
|
|
|
c = unichr(name2codepoint[name])
|
|
|
|
|
print "Named ent:", c
|
|
|
|
|
def handle_charref(self, name):
|
|
|
|
|
if name.startswith('x'):
|
|
|
|
|
c = unichr(int(name[1:], 16))
|
|
|
|
|
else:
|
|
|
|
|
c = unichr(int(name))
|
|
|
|
|
print "Num ent :", c
|
|
|
|
|
def handle_decl(self, data):
|
|
|
|
|
print "Decl :", data
|
2007-08-15 11:28:01 -03:00
|
|
|
|
2011-10-28 08:14:34 -03:00
|
|
|
parser = MyHTMLParser()
|
2012-02-17 19:46:04 -04:00
|
|
|
|
|
|
|
|
Parsing a doctype::
|
|
|
|
|
|
|
|
|
|
>>> parser.feed('<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" '
|
|
|
|
|
... '"http://www.w3.org/TR/html4/strict.dtd">')
|
|
|
|
|
Decl : DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"
|
|
|
|
|
|
|
|
|
|
Parsing an element with a few attributes and a title::
|
|
|
|
|
|
|
|
|
|
>>> parser.feed('<img src="python-logo.png" alt="The Python logo">')
|
|
|
|
|
Start tag: img
|
|
|
|
|
attr: ('src', 'python-logo.png')
|
|
|
|
|
attr: ('alt', 'The Python logo')
|
|
|
|
|
>>>
|
|
|
|
|
>>> parser.feed('<h1>Python</h1>')
|
|
|
|
|
Start tag: h1
|
|
|
|
|
Data : Python
|
|
|
|
|
End tag : h1
|
|
|
|
|
|
|
|
|
|
The content of ``script`` and ``style`` elements is returned as is, without
|
|
|
|
|
further parsing::
|
|
|
|
|
|
|
|
|
|
>>> parser.feed('<style type="text/css">#python { color: green }</style>')
|
|
|
|
|
Start tag: style
|
|
|
|
|
attr: ('type', 'text/css')
|
|
|
|
|
Data : #python { color: green }
|
|
|
|
|
End tag : style
|
|
|
|
|
>>>
|
|
|
|
|
>>> parser.feed('<script type="text/javascript">'
|
|
|
|
|
... 'alert("<strong>hello!</strong>");</script>')
|
|
|
|
|
Start tag: script
|
|
|
|
|
attr: ('type', 'text/javascript')
|
|
|
|
|
Data : alert("<strong>hello!</strong>");
|
|
|
|
|
End tag : script
|
|
|
|
|
|
|
|
|
|
Parsing comments::
|
|
|
|
|
|
|
|
|
|
>>> parser.feed('<!-- a comment -->'
|
|
|
|
|
... '<!--[if IE 9]>IE-specific content<![endif]-->')
|
|
|
|
|
Comment : a comment
|
|
|
|
|
Comment : [if IE 9]>IE-specific content<![endif]
|
|
|
|
|
|
|
|
|
|
Parsing named and numeric character references and converting them to the
|
|
|
|
|
correct char (note: these 3 references are all equivalent to ``'>'``)::
|
|
|
|
|
|
|
|
|
|
>>> parser.feed('>>>')
|
|
|
|
|
Named ent: >
|
|
|
|
|
Num ent : >
|
|
|
|
|
Num ent : >
|
|
|
|
|
|
|
|
|
|
Feeding incomplete chunks to :meth:`~HTMLParser.feed` works, but
|
|
|
|
|
:meth:`~HTMLParser.handle_data` might be called more than once::
|
|
|
|
|
|
|
|
|
|
>>> for chunk in ['<sp', 'an>buff', 'ered ', 'text</s', 'pan>']:
|
|
|
|
|
... parser.feed(chunk)
|
|
|
|
|
...
|
|
|
|
|
Start tag: span
|
|
|
|
|
Data : buff
|
|
|
|
|
Data : ered
|
|
|
|
|
Data : text
|
|
|
|
|
End tag : span
|
|
|
|
|
|
|
|
|
|
Parsing invalid HTML (e.g. unquoted attributes) also works::
|
|
|
|
|
|
|
|
|
|
>>> parser.feed('<p><a class=link href=#main>tag soup</p ></a>')
|
|
|
|
|
Start tag: p
|
|
|
|
|
Start tag: a
|
|
|
|
|
attr: ('class', 'link')
|
|
|
|
|
attr: ('href', '#main')
|
|
|
|
|
Data : tag soup
|
|
|
|
|
End tag : p
|
|
|
|
|
End tag : a
|
