Commit Graph

139 Commits

Author SHA1 Message Date
Georg Brandl 6aab16e9f6 Move "httpresponses" dict from urllib2 to httplib where it belongs. 2006-02-17 19:17:25 +00:00
Georg Brandl 531cebad4c Bug #902075: urllib2 now handles "host:port" proxy specifications
Can/should this be backported?
2006-01-21 07:20:56 +00:00
Tim Peters 9e34c04732 Whitespace normalization (via reindent.py). 2005-08-26 15:20:46 +00:00
Georg Brandl e1b13d2019 Bug #735248: Fix urllib2.parse_http_list. 2005-08-24 22:20:32 +00:00
Georg Brandl a5a2088dd7 Bug 1016563: Bug in urllib2 proxy auth 2005-08-24 20:29:32 +00:00
Tim Peters f5f32b4712 Whitespace normalization. 2005-07-17 23:16:17 +00:00
Georg Brandl 5c5fe2f445 RFE [ 1216944 ] Add Error Code Dictionary to urllib2 2005-07-14 06:40:47 +00:00
Georg Brandl 8d457c78b4 bug [ 1175848 ] poorly named variable in urllib2.py 2005-06-26 22:01:35 +00:00
Raymond Hettinger dbecd93b72 Replace list of constants with tuples of constants. 2005-02-06 06:57:08 +00:00
Raymond Hettinger f7bf02ded5 SF patch #1028908
(John J Lee)

Minor code clarification and simplification.
2005-02-05 14:37:06 +00:00
Johannes Gijsbers cdd625a770 Patch #1095362: replace hardcoded test for POST/GET with call to get_method,
removing some duplication and gaining some flexibility in the process.
2005-01-09 05:51:49 +00:00
Jeremy Hylton b300ae3a79 Two small changes, SF bug #974757 and SF patch #1037974.
Pass the full URL to find_user_password(), in particular so that hosts
with port numbers can be looked up.

Also specify the digest algorithm, even if it's MD5.  Titus Brown
verified that this fixes a problem with LiveJournal.
2004-12-22 14:27:19 +00:00
Anthony Baxter 3dd9e46161 Added a usegmt flag to email.Utils.formatdate - this allows it to be
used to replace rfc822.formatdate for protocols like HTTP (where 'GMT' must
be the timezone string).
2004-10-11 13:53:08 +00:00
Andrew M. Kuchling 33ad28b68d Use multi-line import 2004-08-31 11:38:12 +00:00
Tim Peters 9ca3f8551a Whitespace normalization. 2004-08-08 01:05:14 +00:00
Jeremy Hylton 5d9c3031c8 Fix urllib2.urlopen() handling of chunked content encoding.
The change to use the newer httplib interface admitted the possibility
that we'd get an HTTP/1.1 chunked response, but the code didn't handle
it correctly.  The raw socket object can't be pass to addinfourl(),
because it would read the undecoded response.  Instead, addinfourl()
must call HTTPResponse.read(), which will handle the decoding.

One extra wrinkle is that the HTTPReponse object can't be passed to
addinfourl() either, because it doesn't implement readline() or
readlines().  As a quick hack, use socket._fileobject(), which
implements those methods on top of a read buffer.  (suggested by mwh)

Finally, add some tests based on test_urllibnet.

Thanks to Andrew Sawyers for originally reporting the chunked problem.
2004-08-07 17:40:50 +00:00
Martin v. Löwis 65a7975f16 Patch #994595: Recognize Basic auth even if other schemes are offered.
Will backport to 2.3.
2004-08-03 12:59:55 +00:00
Kurt B. Kaiser 3f7cb5d9f5 Patch [ 972332 ] urllib2 FTPHandler bugs / John J. Lee
Modified Files:
	urllib2.py test/test_urllib2.py
2004-07-11 17:14:13 +00:00
Andrew M. Kuchling 85064ffd76 [Patch #988602] Move the urllib2 tests into the test framework 2004-07-10 19:46:40 +00:00
Andrew M. Kuchling f9ea7c067a [Patch #988504] Fix HTTP error handling via a patch from John J. Lee 2004-07-10 15:34:34 +00:00
Andrew M. Kuchling 08c08bb3d1 [Bug #912845] urllib2 only checks for a 200 return code, but 206 is also legal if a Range: header was supplied.
(Actually, should the first 'if' statement be modified to allow any 2xx status code?)
2004-06-29 13:19:19 +00:00
Neal Norwitz 1cdd3632a0 Cleanup: the in operator already returns a bool, no need to bool() it again
Cleanup: use condition to be consistent with code above
CookieJar is in cookielib
2004-06-07 03:49:50 +00:00
Martin v. Löwis 2a6ba9097e Patch #963318: Add support for client-side cookie management. 2004-05-31 18:22:40 +00:00
Mark Hammond c533c986e8 Fix [ 738973 ] urllib2 CacheFTPHandler doesn't work on multiple dirs, as
implemented in patch [ 851736 ].
2004-05-10 07:35:33 +00:00
Martin v. Löwis 8b3e871a19 Patch #944110: Properly process empty passwords. Fixes #944082.
Backported to 2.3.
2004-05-06 01:41:26 +00:00
Andrew M. Kuchling 22ab06e4de Fix comment typo 2004-04-06 19:43:03 +00:00
Jeremy Hylton b3ee6f9921 Fix two bugs in the new do_open() implementation for HTTPHandler.
Invoke the standard error handlers for non-200 responses.

Always supply a "Connection: close" header to prevent the server from
leaving the connection open.  Downstream users of the socket may
attempt recv()/read() with no arguments, which would block if the
connection were kept open.
2004-02-24 19:40:35 +00:00
Martin v. Löwis a79449e7a2 Patch #711838: Allow non-anonymous ftp urls in urllib2.
Backported to 2.3.
2004-02-15 21:19:18 +00:00
Martin v. Löwis 7db04e7c48 Patch #817379: Allow for absolute ftp paths.
Backported to 2.3.
2004-02-15 20:51:39 +00:00
Tim Peters 58eb11cf62 Whitespace normalization. 2004-01-18 20:29:55 +00:00
Jeremy Hylton 023518a6a8 Rewrite AbstractHTTPHandler to use modern httplib interface.
The chief benefit of this change is that requests will now use
HTTP/1.1 instead of HTTP/1.0.  Bump the module version number as part
of the change.

There are two possible incompatibilities that we'll need to watch out
for when we get to an alpha release.  We may get a different class of
exceptions out of httplib, and the do_open() method changed its
signature.  The latter is only important if anyone actually subclasses
AbstractHTTPHandler.
2003-12-17 18:52:16 +00:00
Jeremy Hylton dce391cb39 Remove __del__ methods to avoid creating uncollectable cyclic trash.
Keep close() methods for backwards compatibility.

Does any call close() explicitly?
2003-12-15 16:08:48 +00:00
Jeremy Hylton c1be59f40a SF patch 852995: add processors feature to urllib2
John J. Lee writes: "the patch makes it possible to implement
functionality like HTTP cookie handling, Refresh handling,
etc. etc. using handler objects. At the moment urllib2's handler
objects aren't quite up to the job, which results in a lot of
cut-n-paste and subclassing. I believe the changes are
backwards-compatible, with the exception of people who've
reimplemented build_opener()'s functionality -- those people would
need to call opener.add_handler(HTTPErrorProcessor).

The main change is allowing handlers to implement
methods like:

http_request(request)
http_response(request, response)

In addition to the usual

http_open(request)
http_error{_*}(...)
"

Note that the change isn't well documented at least in part because
handlers aren't well documented at all.  Need to fix this.

Add a bunch of new tests.  It appears that none of these tests
actually use the network, so they don't need to be guarded by a
resource flag.
2003-12-14 05:27:34 +00:00
Jeremy Hylton fcefd0d2a5 Apply patch 823328 -- support for rfc 2617 digestion authentication.
The patch was tweaked slightly.  It's get a different mechanism for
generating the cnonce which uses /dev/urandom when possible to
generate less-easily-guessed random input.

Also rearrange the imports so that they are alphabetical and
duplicates are eliminated.

Add a few XXX comments about things left undone and things that could
be improved.
2003-10-21 18:07:07 +00:00
Walter Dörwald f0dfc7ac5c Fix a bunch of typos in documentation, docstrings and comments.
(From SF patch #810751)
2003-10-20 14:01:56 +00:00
Jeremy Hylton 0a4a50dd85 SF patch [ 816787 ] urllib2.URLError don't calll IOError.__init__
Not sure if this fix is great, but it's probably a small improvement.
2003-10-06 05:15:13 +00:00
Martin v. Löwis 162f081fb3 Patch 549151, rev4: redirect posts for 301 also. Will backport to 2.2. 2003-07-12 07:33:32 +00:00
Jeremy Hylton aefae5570d Fix wrong header name. The framework looks for auth_header. 2003-07-10 13:30:12 +00:00
Brett Cannon 783eaf4774 Change all header strings to be as if they were capitalize()'ed. Also call
capitalize in AbstractHTTPHandler before inserting headers into HTTP instance.

Closes bug #649742, again.
2003-06-17 21:52:34 +00:00
Tim Peters f545baa0cc Whitespace normalization. 2003-06-15 23:26:30 +00:00
Martin v. Löwis e3b67bcb7e Patch #754340: Fix typo in redirect_request. Will backport to 2.2. 2003-06-14 05:51:25 +00:00
Gustavo Niemeyer 9556fba685 - urllib2.py now knows how to order proxy classes, so the user doesn't
have to insert it in front of other classes, nor do dirty tricks like
  inserting a "dummy" HTTPHandler after a ProxyHandler when building an
  opener with proxy support.
2003-06-07 17:53:08 +00:00
Raymond Hettinger 4ec4fa208e iteritems() should not have been used for self.timeout which changes during the loop. 2003-05-23 08:51:51 +00:00
Brett Cannon df0d87a922 To be on the safe side, backed out any questionable iteritem changes and set back to item calls. 2003-05-18 02:25:07 +00:00
Brett Cannon c8b188a9e2 Revert some changes back to dict.items made in a previous patch. 2003-05-17 19:51:26 +00:00
Brett Cannon 24b2bc3654 Remove assert that checked if a parameter was an instance of Request. Closes patch #639139. 2003-05-12 07:31:53 +00:00
Brett Cannon 86503b131b Change Request.add_header to call string.capitalize in order to normalize
headers and not have any dependency on case.  Closes patch #649742.

Also changed all instances of dict.items to dict.iteritems where appropriate.
2003-05-12 07:29:42 +00:00
Jeremy Hylton 0389295dcd Better fix for newurl as suggested by Jim Jewett in SF bug #730963. 2003-05-05 04:09:13 +00:00
Jeremy Hylton f6b444ede4 Add comment explaining try/except for endheaders(). 2003-05-05 01:47:13 +00:00
Jeremy Hylton 828023b6b5 Repair redirect handling and raise URLError on host-not-found.
The latest changes to the redirect handler couldn't possibly have been
tested, because they did not compute a newurl and failed with a
NameError.  The __name__ == "__main__": block has a test for
redirects.

Also, fix SF bug 723831.  A urlopen() that failed because the host was
not found raised a socket.gaierror unlike earlier versions of
urllib2.  The problem is that httplib actually establishes the
connection at a different point starting with Python 2.2.  Move the
try/except to endheaders(), which is where the connection gets
established.
2003-05-04 23:44:49 +00:00
Raymond Hettinger 024aaa1bfe SF Patch 549151: urllib2 POSTs on redirect
(contributed by John J Lee)
2003-04-24 15:32:12 +00:00
Andrew M. Kuchling aca49b065b Fix NameError exception ('name' undefined) 2002-11-06 15:40:05 +00:00
Jeremy Hylton e190663742 Typo in docstring 2002-10-11 17:27:55 +00:00
Jeremy Hylton 96f1129de8 Fix for SF bug #599836: Don't duplicate headers.
If the request object has a header, it should override the default
header provided by the OpenerDirector.
2002-10-11 17:26:46 +00:00
Neal Norwitz 853ddd5cb9 SF #614596, fix for urllib2.AbstractBasicAuthHandler, John Williams (johnw42)
Make the regex case insensitive for some web sites which use Realm.
2002-10-09 23:17:04 +00:00
Fred Drake a87a521b83 Remove ugly irregular spaces from in front of some comments. 2002-08-13 13:59:55 +00:00
Tim Peters c411dbaeee Whitespace normalization. 2002-07-16 21:35:23 +00:00
Jeremy Hylton 144dea3e05 Fix from SF patch #527518: proxy config with user+pass authentication.
Bug fix candidate.
2002-07-07 16:57:35 +00:00
Jeremy Hylton 40bbae3b03 Fix HTTPError __init__ for cases where fp is None.
The HTTPError class tries to act as a regular response objects for
HTTP protocol errors that include full responses.  If the failure is
more basic, like no valid response, the __init__ choked when it tried
to initialize its superclasses in addinfourl hierarchy that requires a
valid response.

The solution isn't elegant but seems to be effective.  Do not
initialize the base classes if there isn't a file object containing
the response.  In this case, user code expecting to use the addinfourl
methods will fail; but it was going to fail anyway.

It might be cleaner to factor out HTTPError into two classes, only one
of which inherits from addinfourl.  Not sure that the extra complexity
would lead to any improved functionality, though.

Partial fix for SF bug # 563665.

Bug fix candidate for 2.1 and 2.2.
2002-06-03 16:53:00 +00:00
Walter Dörwald 65230a2de7 Remove uses of the string and types modules:
x in string.whitespace => x.isspace()
type(x) in types.StringTypes => isinstance(x, basestring)
isinstance(x, types.StringTypes) => isinstance(x, basestring)
type(x) is types.StringType => isinstance(x, str)
type(x) == types.StringType => isinstance(x, str)
string.split(x, ...) => x.split(...)
string.join(x, y) => y.join(x)
string.zfill(x, ...) => x.zfill(...)
string.count(x, ...) => x.count(...)
hasattr(types, "UnicodeType") => try: unicode except NameError:
type(x) != types.TupleTuple => not isinstance(x, tuple)
isinstance(x, types.TupleType) => isinstance(x, tuple)
type(x) is types.IntType => isinstance(x, int)

Do not mention the string module in the rlcompleter docstring.

This partially applies SF patch http://www.python.org/sf/562373
(with basestring instead of string). (It excludes the changes to
unittest.py and does not change the os.stat stuff.)
2002-06-03 15:58:32 +00:00
Raymond Hettinger 54f0222547 SF 563203. Replaced 'has_key()' with 'in'. 2002-06-01 14:18:47 +00:00
Michael W. Hudson 3704644acb Use types.StringTypes instead of (types.StringType, types.UnicodeType) --
works better (at all!) in --disable-unicode builds.

Bugfix candidate, probably.
2002-05-20 14:30:06 +00:00
Guido van Rossum 8ca162f417 Partial introduction of bools where appropriate. 2002-04-07 06:36:23 +00:00
Martin v. Löwis 9d3eba87d6 Patch #525870: Avoid duplicate stat calls, use st_ attributes. 2002-03-18 08:37:19 +00:00
Greg Ward 2e250b4378 Fix bug #511786 (2.2.1 candidate): ensure that custom-supplied headers
are preserved for redirected requests.
2002-02-11 20:46:10 +00:00
Jeremy Hylton 52a17becbc Fix SF bug 468948 & 451295: urllib2 authentication problems
Fix contributed by Jeffrey C. Ollie.

I haven't tested the fix because the situation is non-trivial to
reproduce.

The basic solution is to get rid of the __current_realm attribute of
authentication handlers.  Instead, prevent infinite retries by
checking for the presence of an Authenticate: header in the request
object that exactly matches the Authenticate: header that would be
added.

The problem prevent authentication from working correctly in the
presence of retries.

Ollie mentioned that digest authentication has the same problem and I
applied the same solution there.
2001-11-09 16:46:51 +00:00
Fred Drake d5214b04f5 Clean up one comment, fix typos in others. 2001-11-08 17:19:29 +00:00
Jeremy Hylton 8b78b99647 Fix [ #465502 ] urllib2: urlopen unicode problem
When checking for strings use,
!         if isinstance(uri, (types.StringType, types.UnicodeType)):

Also get rid of some dodgy code that tried to guess whether attributes
were callable or not.
2001-10-09 16:18:45 +00:00
Jeremy Hylton 6d8c1aabff Add content-type header to ftp URLs (SF patch #454553)
Modify rfc822.formatdate() to always generate English names,
regardless of locale.  This is required by RFC 1123.

In open_local_file() of urllib and urllib2, use new formatdate() from
rfc822.
2001-08-27 20:16:53 +00:00
Guido van Rossum 833a8d8641 SF patch #454553 by Walter Dörwald: add content-type to FTP URLs, like
for urllib.
2001-08-24 13:10:13 +00:00
Jeremy Hylton d5d8fc559c Replace all type comparisons with isinstance() calls 2001-08-11 21:44:46 +00:00
Tim Peters ab9ba27dc0 Whitespace normalization. 2001-08-09 21:40:30 +00:00
Jeremy Hylton 54e99e8b3b Fix SF bug [ #447370 ] typo in urllib2.py
Also fix another bug caught by pychecker-- HTTPError() raised when
redirect limit exceed did not pass an fp object.  Had to change method
to keep fp object around until it's certain that the error won't be
raised.

Remove useless line in do_proxy().
2001-08-07 21:12:25 +00:00
Martin v. Löwis 02d893cfae Patch #444359: Remove unused imports. 2001-08-02 07:15:29 +00:00
Fred Drake ec3dfdee6a Only write out one blank line before the request data.
This closes SF patch #419459.
2001-07-04 05:18:29 +00:00
Jeremy Hylton 78cae61ad4 Raise useful exception when called with URL for which request type
cannot be determined.

Pseudo-fix for SF bug #420724
2001-05-09 15:49:24 +00:00
Guido van Rossum 2d996c0704 Fix typo (missing "req." prefix on error_302_dict) found by Neil
Norwitz's PyChecker.
2001-04-15 13:08:01 +00:00
Moshe Zadka 7667680d70 Idiotic braino caused HTTP openers to ignore proxies.
This fixes 413135
2001-04-11 07:44:53 +00:00
Moshe Zadka d3f193fe9d * Fixing the password-proxy bug
* Not sending content-type and content-length twice
2001-03-20 13:14:28 +00:00
Tim Peters 30edd2387d Whitespace normalization. 2001-03-16 08:29:48 +00:00
Moshe Zadka 8a18e99008 Checking in patch 404826 -- urllib2 enhancements and documentations.
(please not that the library reference does *not* include the
urllib2 documnetation -- that will wiat for Fred)
2001-03-01 08:40:42 +00:00
Eric S. Raymond b08b2d3166 String method conversion. 2001-02-09 11:10:16 +00:00
Tim Peters e119006e7d Whitespace normalization. Top level of Lib now fixed-point for reindent.py! 2001-01-15 03:34:38 +00:00
Jeremy Hylton 73574eefe5 two fixes for redirects:
- don't close the fp, since that appears to also close the socket
    - join the original url with the redirect reponse to deal with
      relative redirect URL

wrap two socket ops in try/except to turn them into URLErrors, so that
client code need only catch one exception.

in HTTPError.__del__ only close fp if fp is not None

style changes:
    - use f(*args) instead of apply(f, args)
    - use __super_init instead of super.__init__(self, ...)
2000-10-12 18:54:18 +00:00
Thomas Wouters 7e47402264 Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either
comments, docstrings or error messages. I fixed two minor things in
test_winreg.py ("didn't" -> "Didn't" and "Didnt" -> "Didn't").

There is a minor style issue involved: Guido seems to have preferred English
grammar (behaviour, honour) in a couple places. This patch changes that to
American, which is the more prominent style in the source. I prefer English
myself, so if English is preferred, I'd be happy to supply a patch myself ;)
2000-07-16 12:04:32 +00:00
Jeremy Hylton a05e293a21 typos fixed by Rob Hooft 2000-06-28 14:48:01 +00:00
Fred Drake 13a2c279c5 Untabify to pass the -tt test. 2000-02-10 17:17:14 +00:00
Guido van Rossum e7b146fb3b The third and final doc-string sweep by Ka-Ping Yee.
The attached patches update the standard library so that all modules
have docstrings beginning with one-line summaries.

A new docstring was added to formatter.  The docstring for os.py
was updated to mention nt, os2, ce in addition to posix, dos, mac.
2000-02-04 15:28:42 +00:00
Jeremy Hylton 6d7e47b8ea EXPERIMENTAL
An extensible library for opening URLs using a variety protocols.
Intended as a replacement for urllib.
2000-01-20 18:19:08 +00:00