Commit Graph

202 Commits

Author SHA1 Message Date
Guido van Rossum ba3113807d Promote the server version from a local variable to a class variable,
so that a subclass can override it.

This partly addresses Bug #112634 -- but the documentation is still
wrong, since it suggests that you can set self.version *after* calling
the base class __init__.  In fact it must be done *before*.

I'll fix that too.
2000-08-24 16:18:04 +00:00
Fred Drake 316a793a58 Randall Hopper <aa8vb@yahoo.com>>:
Make it easier to use HTTP POST with urlretrieve().
2000-08-24 01:01:26 +00:00
Skip Montanaro 79f1c1778d * added doc strings to urlopen and unquote_plus
* fixed type in doc string for quote
2000-08-22 03:00:52 +00:00
Fred Drake 567ca8e732 Patch from Paul Schreiber <paul@commerceflow.com>:
Patch description
-----------------
This addresses four issues:

(1) usernames and passwords in urls with special characters are now
    decoded properly. i.e. http://foo%2C:bar@www.whatever.com/

(2) Basic Auth support has been added to HTTPS, like it was in HTTP.

(3) Version 1.92 sent the POSTed data, but did not deal with errors
    (HTTP responses other than 200) properly. HTTPS now behaves the
    same way HTTP does.

(4) made URL-checking beahve the same way with HTTPS as it does with
    HTTP (changed == to !=).
2000-08-21 21:42:42 +00:00
Mark Hammond 4f570b9239 Patch #100873 - Use registry to find proxies for urllib on Win32
Note that this patch looks worse than it is - an existing function (getproxies() for all platforms other than Win/Mac) has been moved, renamed and indentation changed, but the body of that function is identical.  Windows now allows the environment variables to override the registry.
2000-07-26 07:04:38 +00:00
Thomas Wouters 7e47402264 Spelling fixes supplied by Rob W. W. Hooft. All these are fixes in either
comments, docstrings or error messages. I fixed two minor things in
test_winreg.py ("didn't" -> "Didn't" and "Didnt" -> "Didn't").

There is a minor style issue involved: Guido seems to have preferred English
grammar (behaviour, honour) in a couple places. This patch changes that to
American, which is the more prominent style in the source. I prefer English
myself, so if English is preferred, I'd be happy to supply a patch myself ;)
2000-07-16 12:04:32 +00:00
Fred Drake 9e94afd18d Fix bug #314, reported by Craig Allen <cba@mediaone.net>:
splittype():  Always lower-case the URL scheme; these are supposed to be
               normalized according to RFC 1738 anyway.
2000-07-01 07:03:30 +00:00
Andrew M. Kuchling 7ad4792307 Comment out an apparent debug print 2000-06-10 01:41:48 +00:00
Guido van Rossum c580dae6da Fix a problem reported by Oleg Broytmann, who complains that very
often, ftp URLs hang in the final close.  Further analysis suggests
that this is because the close hook in addclosehook() calls the hook
before acually closing the connection.  The hook, in this case, waits
for the '226 Transfer complete' status from the server on the command
socket.  However, more and more ftp servers only send this status when
the data socket has actually been closed -- causing a deadlock.

The fix is simple: in addclosehook.close(), call addbase.close()
*before* calling the closehook.
2000-05-24 13:21:46 +00:00
Andrew M. Kuchling 43c5af026f Fix to previous patch: send the request data when it's provided 2000-04-24 14:17:06 +00:00
Andrew M. Kuchling 141e9894b7 Fixed bug reported by JP Calderone: https:// URL's didn't work.
The fix also adds support for POSTing to an https URL
2000-04-23 02:53:11 +00:00
Guido van Rossum e7b146fb3b The third and final doc-string sweep by Ka-Ping Yee.
The attached patches update the standard library so that all modules
have docstrings beginning with one-line summaries.

A new docstring was added to formatter.  The docstring for os.py
was updated to mention nt, os2, ce in addition to posix, dos, mac.
2000-02-04 15:28:42 +00:00
Guido van Rossum 3c8baedaf8 Sjoerd Mullender writes:
Fixed a TypeError: not enough arguments; expected 4, got 3.
When authentication is needed, the default http_error_401 method calls
retry_http_basic_auth.  The default version of that method expected a
data argument which wasn't provided, so now we provide the argument if
it was given and we also made the data argument optional.

Also changed other calls where data was optional to not pass data if
it was not passed to the calling method (in line with other similar
occurances).
2000-02-01 23:36:55 +00:00
Guido van Rossum 09c8b6c3e4 OpenSSL support. This is based on patches for a version of SSLeay by
Brian E Gallew, which were improved and adapted to OpenSSL 0.9.4 by
Laszlo Kovacs of HP.  Both have kindly given permission to include
the patches in the Python distribution.  Final formatting by GvR.
1999-12-07 21:37:17 +00:00
Guido van Rossum 5e006a3cc3 Patches by Michael Reilly to correctly deal with ftp URLs of the form
ftp://user@host//root/path: the double slash in the pathname means to
go to the root directory even if the initial directory isn't the root.
1999-08-18 17:40:33 +00:00
Guido van Rossum 3427c1f71b Sjoerd Mullender:
In splithost, accept empty host part in URLs.  This is required for
file URLs that can have an empty host part.  For such URLs, we should
not return the initial 2 slashes as part of the file name.
1999-07-01 23:20:56 +00:00
Guido van Rossum 336a201d4f Sjoerd Mullender writes:
Urllib makes the URL of the opened file available through the geturl
method of the returned object.  For local files, this consists of
file: plus the name of the file.  This results in an invalid URL if
the file name was relative.  This patch fixes this so that the
returned URL is just a relative URL in that case.  When the file name
is absolute, the URL returned is of the form file:///absolute/path.

[I guess that a URL of the form "file:foo.html" is illegal...  GvR]
1999-06-24 15:27:36 +00:00
Guido van Rossum 0dee4ee0f8 Updated lagging version#. Also added some comments about how quote()
and quote_plus() can be optimized tenfold.
1999-06-09 15:14:50 +00:00
Guido van Rossum 3527f59457 Hack so that if a 302 or 301 redirect contains a relative URL, the
right thing "just happens" (basejoin() with old URL).
1999-03-29 20:23:41 +00:00
Guido van Rossum 3764595c98 Yet another patch by Sjoerd Mullender:
Don't convert URLs to URLs using pathname2url.
1999-03-15 16:16:29 +00:00
Guido van Rossum 367ac80d3b From: Sjoerd Mullender
The filename to URL conversion didn't properly quote special
characters.
The URL to filename didn't properly unquote special chatacters.
1999-03-12 14:31:10 +00:00
Guido van Rossum 29aab7582f open_http also had the 'data is None' test backwards. don't call with the
extra argument if data is None.
1999-03-09 19:31:21 +00:00
Jeremy Hylton b30f52a471 http_error had the 'data is None' test backwards. don't call with the
extra argument if data is None.
1999-02-25 16:14:58 +00:00
Jeremy Hylton f90b002e31 change indentation from 8 spaces to 4 spaces 1999-02-25 16:12:12 +00:00
Jeremy Hylton 547c3f1c13 pleasing the tabnanny 1999-02-25 15:59:54 +00:00
Jeremy Hylton dbc8364e1f When performing a POST request, i.e. when the second argument to
urlopen is used to specify form data, make sure the second argument is
threaded through all of the http_error_NNN calls.  This allows error
handlers like the redirect and authorization handlers to properly
re-start the connection.
1999-02-24 18:42:38 +00:00
Guido van Rossum 4505895e68 As Des Barry points out, we need to call pathname2url(file) in two
calls to addinfourl() in open_file().
1999-02-22 19:01:42 +00:00
Guido van Rossum ed52a20c6e In open_ftp(), check that retrlen is not None before using it in a %d format! 1999-02-16 15:10:12 +00:00
Guido van Rossum 33add0a95a Sjoerd Mullender:
File names with "funny" characters get translated wrong by
pathname2url (any variety).  E.g. the (Unix) file "/ufs/sjoerd/#tmp"
gets translated into "/ufs/sjoerd/#tmp" which, when interpreted as a
URL is file "/ufs/sjoerd/" with fragment ID "tmp".

Here's an easy fix.  (An alternative fix would be to change the
various implementations of pathname2url and url2pathname to include
calls to quote and unquote.

[The main problem is with the normal use of URLs:
	url = url2pathname(file)
	transmit url
	url, tag = splittag(url)
	urlopen(url)
]

In addition, this patch fixes some uses of unquote:
- the host part of URLs should be unquoted
- the file path in the FTP URL should be unquoted before it is split
  into components.
- because of the latter, I removed all unquoting from ftpwrapper,
  and moved it to the caller, but that is not essential
1998-12-18 15:25:22 +00:00
Guido van Rossum 9ab96d40eb Changes by Eric Raymond:
1. Generate a correct Content-Length header visible through the info() method
   if a request to open an FTP URL gets a length in the response to RETR.

2. Take a third argument to urlretrieve() that makes it possible to progress-
   meter an urlretrieve call (this is what I needed the above change for).
   See the second patch band below for details.

3. To avoid spurious errors, I commented out the gopher test.  The target
   document no longer exists.
1998-09-28 14:07:00 +00:00
Guido van Rossum 4163e708ed On the Mac, use Internet Config to find the proxies (Jack Jansen).
Also added two XXX comments about lingering thread unsafeness.
1998-08-06 13:39:09 +00:00
Guido van Rossum 810a3396d1 Speed up the implementation of quote().
Fix the implementation of quote_plus().  (It wouldn't treat '+' in the
original data right.)

Add urlencode(dict) which is handy to create the data for sending a
POST request with urlopen().
1998-07-22 21:33:23 +00:00
Guido van Rossum c94f16f156 Oops! Of course, Tim is right -- when the item is not a hex number,
the '%' should be put back in.
1998-06-29 00:42:54 +00:00
Guido van Rossum 52e86ad05b Speed-up unquote(), inspired by post from Daniel Walton. 1998-06-28 23:49:35 +00:00
Guido van Rossum 2349015a87 Rewrite the (test) main program so that when used as a script, it can
retrieve one or more URLs to stdout.  Use -t to run the self-test.
1998-06-25 02:39:00 +00:00
Guido van Rossum ae9ee7329d Use the getpass module instead of having platform-specific echo on/off
code here.
1998-06-12 14:21:13 +00:00
Guido van Rossum e0c0da98d8 Patches to make the proxy code work again. (Why does that always break
as soon as I change things even just a little bit? :-)  Even works
when accessing a password-protected page through the proxy.  Prompted
by complaints from, and correct operation verified by, Nigel O'Brian.
1998-05-05 13:58:13 +00:00
Guido van Rossum 0eae8fba81 Feeble attempt at making urlopen more robust -- don't call splituser()
when splithost() returned no useable host, to avoid calling
splituser() on None.
1998-04-27 15:19:17 +00:00
Guido van Rossum c74521acc4 Oops -- remove some debug print statements! 1998-04-11 01:18:35 +00:00
Guido van Rossum 0454b51282 Oops, pulled over by the tab police :-) 1998-04-03 15:57:58 +00:00
Guido van Rossum b5916ab065 Change by Sjoerd (with minor reformatting):
guess the mime type of a local file.

Change suggested by Sjoerd (with different implementation):
  when retrieve() creates a temporary file, preserve the suffix.

Corrollary of the first change:
  also return the mime type of a local file in retrieve().
1998-04-03 15:56:16 +00:00
Guido van Rossum a08fabad72 A few lines were indented using spaces instead of tabs -- fix them. 1998-03-30 17:17:24 +00:00
Guido van Rossum 7e7ca0ba17 A few lines were indented using spaces instead of tabs -- fix them. 1998-03-26 21:01:39 +00:00
Guido van Rossum 6d4d1c2a25 Added support for "data" URL, by Sjoerd Mullender. 1998-03-12 14:32:55 +00:00
Guido van Rossum 8a666e7c56 Fix a horrible race condition -- various routines were storing the
most recently opened URL in self.openedurl of the URLopener instance.
This doesn't really work if multiple threads share the same opener
instance!

Fix: openedurl was actually simply the type prefix (e.g. "http:")
followed by the rest of the URL; since the rest of the URL is
available and the type is effectively determined by where you are in
the code, I can reconstruct the full URL easily, e.g. "http:" + url.
1998-02-13 01:39:16 +00:00
Guido van Rossum 03710d2a40 Two suggested features by Sjoerd:
- use the tempcache in the open() method, too.

- use the "unwrap"ped url as key for the tempcache.
1998-02-05 16:22:27 +00:00
Guido van Rossum c5d8fed261 (1) Use matchobj.groups(), not matchbj.group() to get all groups.
(2) Provisional hack to avoid dying when trying to turn echo on or off
on Macs, where os.system() doesn't exist.
1998-02-05 16:21:28 +00:00
Guido van Rossum e7579624ef Fix bad new bug in ftp code -- the test for existing file using NLST
would set the transfer to text mode instead of the specified mode.
1998-01-19 22:26:54 +00:00
Guido van Rossum d499004860 Solve two annoying problems with ftp URLs for Jack: when repeated
retrieving files from the same host and directory, you had to close
the previous instance before opening a new one; and retrieving a
non-existent file would return an empty file.  (The latter fix relies
on maybe an undocumented property of NLST -- NLST of a file returns
just that file, while NLST of a non-existent file returns nothing.  A
side effect, unfortunately, seems to be that now ftp-retrieving an
*empty* directory may fail.  Ah well.)
1997-12-28 04:21:20 +00:00
Guido van Rossum c0f29c2d31 When a port is specified in an ftp:// URL, must convert it to a number! 1997-12-02 20:26:21 +00:00
Guido van Rossum 83600050d0 Assert that the proxies object passed in to the URLopener constructor
is indeed a dictionary (or a mapping).
1997-11-18 15:50:39 +00:00
Guido van Rossum 036309b13e This should hopefully finally clean up the remaining __del__ related
problems with this module, even if an instance of a derived class is
kept alive longer than the urllib module itself...
1997-10-27 18:56:19 +00:00
Fred Drake 654451dc54 splitpasswd(): The parameter is named "user", not "host". 1997-10-14 13:30:57 +00:00
Guido van Rossum 332e14437c Use sys.exc_info() where needed.
Use "re" module, making it threadsafe.
1997-09-29 23:23:46 +00:00
Guido van Rossum 10499324cf Initialize self.__tempfiles to [] in the constructor (else it remains
a shared class variable -- but each instance will attempt to clean it
up entirely ob cleanup).
1997-09-08 02:16:33 +00:00
Guido van Rossum 2b3fd76cc7 One patch from Sjoerd and one from Jack.
Sjoerd: add separate administration of temporary files created y
URLopener.retrieve() so cleanup can properly remove them.  The old
code removed everything in tempcache which was a bad idea if the user
had passed a non-temp file into it.  (I added a line to delete the
tempcache in cleanup() -- it still seems to make sense.)

Jack: in basejoin(), interpret relative paths starting in "../".  This
is necessary if the server uses symbolic links.
1997-09-03 22:36:15 +00:00
Guido van Rossum ab0abdcef8 Explicitly close the socket and temp file in URLopener.retrieve(), so
that multiple retrievals using the same connection will work.

This leaves open the more general problem that after
    f = urlopen("ftp://...")
f must be closed before another retrieval from the same host should be
attempted.
1997-08-26 19:06:40 +00:00
Guido van Rossum b6784dc53f Interpret three slashes in file: URL as local file (for Netscape on
Windows/Mac).
1997-08-20 23:34:01 +00:00
Guido van Rossum f668d17e01 Clear the ftp cache when it contains more than 10 entries. 1997-06-06 21:11:11 +00:00
Guido van Rossum 2966b32205 Catch *all* errors that ftplib can raise (ftplib.all_errors) rather
than just the four.

Also folded some long lines.
1997-06-06 17:44:07 +00:00
Guido van Rossum c24751b57a Add Host: header to URL request. 1997-06-03 14:34:19 +00:00
Guido van Rossum 1aec3f087e lowercase proxies env variables, for Windows. 1997-05-28 15:37:19 +00:00
Guido van Rossum ab0d1afdf3 spliturl() should not throw away everything past first newline 1997-04-16 15:17:06 +00:00
Guido van Rossum 54a1d0bc69 (Jack+Guido:) interpret ../ in basejoin() according to RFC1808. 1997-04-11 19:09:51 +00:00
Guido van Rossum c511aee0e3 Open files in binary mode. 1997-04-11 19:01:48 +00:00
Guido van Rossum fd79566d0f Fix two small bugs with proxies. 1997-04-02 05:46:35 +00:00
Guido van Rossum 838cb28290 Put a new, more useful, set of references in the leading comment. 1997-02-10 17:51:56 +00:00
Guido van Rossum d23d9409f3 Move the 'import os' in URLopener.cleanup() to inside the block
guarded by 'if self.tempcache', to reduce the likelihood of this
causing an exception when invoked during __del__...
1997-01-30 15:54:58 +00:00
Guido van Rossum 0564e12367 Added quote_plus() and unquote_plus(), to do space/plus substitutions
for form field values.
1996-12-13 14:47:36 +00:00
Guido van Rossum bd01374da8 Add optional data argument to urlopen() and open_http(), to trigger POST. 1996-12-10 16:00:28 +00:00
Guido van Rossum 29e77816f5 Added safeguard against failure in __del__. 1996-11-27 19:39:58 +00:00
Guido van Rossum 3c8484e866 When re-raising an exception raised by a module used internally as
IOError, keep the traceback.
1996-11-20 22:02:24 +00:00
Guido van Rossum c5d7e80739 Fix the way the Authorization header is sent (how could this have worked?). 1996-11-11 19:01:17 +00:00
Guido van Rossum 5b1b33c7f4 Fix another case where... 1996-10-22 13:28:37 +00:00
Guido van Rossum b030bc026e Fix some cases where self.openedurl wasn't set. 1996-10-10 16:01:16 +00:00
Guido van Rossum e6ad8913e2 One fix by sjoerd and one suggested by him. Bumped __version__ to 1.5.
Fix 1: add a method geturl() to the file-like object returned by urlopen().
Fix 2: treat http error 301 the same as error 302.
1996-09-10 17:02:56 +00:00
Guido van Rossum 78c9637075 Bump exposed __version__ to 1.4.
Correctly handle a URL containing user:password@host when using a proxy.
1996-08-26 18:09:59 +00:00
Guido van Rossum f8abb38737 Slightly faster (un)quoting. 1996-08-26 15:56:12 +00:00
Guido van Rossum 2281d35578 add nturl2path 1996-06-26 19:47:37 +00:00
Guido van Rossum 84a00a80a2 Change defn of splitnport() to only accept valid digit strings. 1996-06-17 17:11:40 +00:00
Guido van Rossum 53725a2858 Added splitnport(), which is like splitport() but returns a numeric port,
is forgiving about semi-numeric port numbers, and allows you to specify
a default port (default is -1, None returned for nonnumeric port).
1996-06-13 19:12:35 +00:00
Guido van Rossum a7e4b28422 Support optional filename argument for retrieve() and urlretrieve(),
to specify where it should go (if specified, even local files will be
copied into the given file).
1996-06-11 00:16:27 +00:00
Guido van Rossum 71ac945321 move mac url2path conversion to separate module 1996-03-21 16:31:41 +00:00
Guido van Rossum 442e7202f8 Added proxy handling; upped version.
(Proxy handling uses <proto>_proxy environment variables by default.)
1996-03-20 15:33:11 +00:00
Jack Jansen 0d12ead05c Try to normalize urls referring to local files (code copied from posixpath) 1996-02-14 16:05:20 +00:00
Guido van Rossum 8c8a02a258 speed up unquote() by using atoi() instead of eval() 1996-01-26 17:41:44 +00:00
Jack Jansen e8ea21b0fe Added pathname2url and url2pathname methods (only correct for unix and
mac, so far)
1995-12-21 15:43:53 +00:00
Jack Jansen dc3e3f69db Fixed local file access for macintosh 1995-12-15 13:22:13 +00:00
Sjoerd Mullender e0371b8415 Fixed basejoin. There were two main problems:
- basejoin('some/url', '#name') would strip the last component of
  some/url and resturn some/#name.
- basejoin('file.html', 'relative/path') would return something like
  file:/relative/path, making a relative path into an absolute one.
These bugs are fixed by some drastic changes.  No scheme is added when
none is present (i.e. it works as replacement for posix.joinpath).
If a scheme is present in the second argument, it is returned
unprocessed.  No hostname are added in this case.  If no scheme is
present, the scheme of the first argument, if present, is used.
The algorithm is commented profusely.
Also fixed a typo in a comment.
1995-11-10 10:36:07 +00:00
Guido van Rossum ca44540bc8 support overriding how to open unknown url types 1995-08-29 19:19:12 +00:00
Sjoerd Mullender 9c4585acfa Removed addbase.__del__ because it can't work.
If code keeps a reference to self.fp or any of its methods, you don't
want to close self.fp just because no explicit reference is kept to self.
1995-08-15 11:33:39 +00:00
Guido van Rossum 30642ab29f changed version :-) 1995-08-10 19:44:54 +00:00
Guido van Rossum bbb0a05972 use mimetools; add error handling and authentication 1995-08-04 04:29:05 +00:00
Guido van Rossum 6cb15a0572 add User-agent hdr; read and close the file upon http error 1995-06-22 19:00:13 +00:00
Guido van Rossum a1124700f8 Add hacks for switching protocol and path but leaving host unchanged 1994-12-30 17:18:59 +00:00
Guido van Rossum fa59e83813 Fix bug if tmpcache is None 1994-09-21 11:36:19 +00:00
Guido van Rossum 3bb5448767 New way of generating .pyc files, thanks to Sjoerd.
urllib.py: '+' is not always safe (even though the RFC says so :-( )
whrandom.py: throw away top bits of time to avoid overflow on Mac
(where times can be negative)
1994-08-29 10:52:58 +00:00
Guido van Rossum 7aeb4b9ce8 * Lib/linecache.py: don't crash on empty filename
* Lib/macpath.py: don't return trailing colon for dirname()
	(XXX won't do for volume names -- but otherwise glob(':*:*.py')
	loops forever)
	* Lib/traceback.py: print SyntaxError correctly
	* Lib/stat.py: moved to posixstat.py; added macstat.py which has
	the constants for the Mac; and created new stat.py which includes
	the right one
	* Lib/urllib.py: fix caching bug (by disabling the cache)
1994-08-23 13:32:20 +00:00
Guido van Rossum 3f9a6ec9e6 * Lib/rfc822.py: fix two bugs: error in readheaders interpreting
regex.match() result, and wrong logic in getfirstmatchingheader()
	when the same header occurs twice consecutively
1994-08-12 13:16:50 +00:00
Guido van Rossum 7c395db298 * Lib/urllib.py: implemented new quoting rules; added splituser,
splitpasswd, splitattr, splitvalue; new ftp syntax (user:passwd,
cwd to each subdir, type={a,i,d})
1994-07-04 22:14:49 +00:00
Guido van Rossum 590b289672 Added tests for missing host to open_http and open_gopher 1994-04-18 09:39:56 +00:00
Guido van Rossum 7c6ebb572e Renamed urlopen.py to urllib.py. 1994-03-22 12:05:32 +00:00