Commit Graph

74 Commits

Author SHA1 Message Date
Benjamin Peterson 42aa93b8ff
closes bpo-31650: PEP 552 (Deterministic pycs) implementation (#4575)
Python now supports checking bytecode cache up-to-dateness with a hash of the
source contents rather than volatile source metadata. See the PEP for details.

While a fairly straightforward idea, quite a lot of code had to be modified due
to the pervasiveness of pyc implementation details in the codebase. Changes in
this commit include:

- The core changes to importlib to understand how to read, validate, and
  regenerate hash-based pycs.

- Support for generating hash-based pycs in py_compile and compileall.

- Modifications to our siphash implementation to support passing a custom
  key. We then expose it to importlib through _imp.

- Updates to all places in the interpreter, standard library, and tests that
  manually generate or parse pyc files to grok the new format.

- Support in the interpreter command line code for long options like
  --check-hash-based-pycs.

- Tests and documentation for all of the above.
2017-12-09 10:26:52 -08:00
Oren Milman db60a5bfa5 bpo-31781: Prevent crashes when calling methods of an uninitialized zipimport.zipimporter object (GH-3986) 2017-10-20 13:42:35 -07:00
Oren Milman 01c6a8859e bpo-31602: Fix an assertion failure in zipimporter.get_source() in case of a bad zlib.decompress() (GH-3784)
While a rare potential failure (it requires swapping out zlib.decompress() itself and forcing it to return a non-bytes object), this change prevents a potential C-level assertion failure and instead substitutes it with an exception.

Thanks to Oren Milman for the patch.
2017-09-29 11:34:31 -07:00
Oren Milman 631fdee6e6 bpo-31291: Fixed an assertion failure in zipimport.zipimporter.get_data() (#3226)
if pathname.replace('/', '\\') returns non-string.
2017-08-29 20:40:15 +03:00
Berker Peksag 8ee9edbf45 Issue #28131: Merge from 3.5 2016-09-14 08:10:45 +03:00
Berker Peksag 4aa74c429c Issue #28131: Fix a regression in zipimport's compile_source()
zipimport should use the same optimization level as the interpreter.
2016-09-14 08:09:48 +03:00
Steve Dower ff70fc2206 Issue #25758: Prevents zipimport from unnecessarily encoding a filename (patch by Eryk Sun) 2016-09-09 17:33:37 -07:00
Steve Dower 8dcc48ee3b Issue #25758: Prevents zipimport from unnecessarily encoding a filename (patch by Eryk Sun) 2016-09-09 17:27:33 -07:00
Serhiy Storchaka febc332056 Issue #26754: Undocumented support of general bytes-like objects
as path in compile() and similar functions is now deprecated.
2016-08-06 23:29:29 +03:00
Serhiy Storchaka f95de0e8cc Issue #26754: PyUnicode_FSDecoder() accepted a filename argument encoded as
an iterable of integers. Now only strings and byte-like objects are accepted.
2016-06-18 13:56:16 +03:00
Serhiy Storchaka 9305d83425 Issue #26754: PyUnicode_FSDecoder() accepted a filename argument encoded as
an iterable of integers. Now only strings and byte-like objects are accepted.
2016-06-18 13:53:36 +03:00
Brett Cannon 43cfd8240c Merge for issue #17633 2016-01-15 11:23:46 -08:00
Brett Cannon 56aae8f304 Issue #17633: Improve support for namespace packages with zipimport.
Previously zipimport mistakenly limited namespace support to only the
top-level of the zipfile when it should have supported an arbitrary
depth.

Thanks to Phil Connel for the bug report and initial patch and Mike
Romberg for the final patch.
2016-01-15 11:22:19 -08:00
Robert Collins 9e1bf4968f Issue #8585: improved tests for zipimporter2. Patch from Mark Lawrence. 2015-07-23 02:57:56 +12:00
Brett Cannon f299abdafa Issue #23731: Implement PEP 488.
The concept of .pyo files no longer exists. Now .pyc files have an
optional `opt-` tag which specifies if any extra optimizations beyond
the peepholer were applied.
2015-04-13 14:21:02 -04:00
Victor Stinner 57d516bd6b Issue #23696: Remove test on ZipImportError.__context__ because the context is
None on Windows.

When the file is not readable, the error occurs at open on UNIX. On Windows,
the error only occurs at the first operation on the open file. It would require
to many changes to set __context__ to an OSError for all file operations, for a
little benefit (__context__ is almost never used).
2015-03-20 13:48:36 +01:00
Victor Stinner fbd6f9ed12 Issue #23696: Chain ZipImportError to the OSError 2015-03-20 10:52:25 +01:00
Benjamin Peterson ec91cd6040 merge backout for #20621 2014-02-16 14:17:28 -05:00
Benjamin Peterson eb43736de2 backout 2807a5f011e4 for causing #20621 2014-02-16 14:12:57 -05:00
Gregory P. Smith 2e385e2592 Fixes issue #19081: When a zipimport .zip file in sys.path being imported from
is modified during the lifetime of the Python process after zipimport has
already cached the zip's table of contents we detect this and recover
rather than read bad data from the .zip (causing odd import errors).
2014-01-07 18:34:23 -08:00
Gregory P. Smith 2bcbc14117 Fixes Issue #19081: When a zipimport .zip file in sys.path being imported from
is modified during the lifetime of the Python process after zipimport has
already cached the zip's table of contents we detect this and recover
rather than read bad data from the .zip (causing odd import errors).
2014-01-07 18:30:07 -08:00
Zachary Ware 101d9e7250 Issue 19572: More silently skipped tests explicitly skipped. 2013-12-08 00:44:27 -06:00
Zachary Ware 9fe6d86709 Issue 19572: More silently skipped tests explicitly skipped. 2013-12-08 00:20:35 -06:00
Brett Cannon 9529fbfd36 Issue #17177: Stop using imp in a bunch of tests 2013-06-15 17:11:25 -04:00
Brett Cannon 638ce0779b Move code from test_importhooks into test_zipimport. 2013-06-12 15:57:01 -04:00
Serhiy Storchaka 0e6b7b5cd2 Issue #8745: Small speed up zipimport on Windows. Patch by Catalin Iacob. 2013-02-16 17:43:45 +02:00
Andrew Svetlov f7a17b48d7 Replace IOError with OSError (#16715) 2012-12-25 16:47:37 +02:00
R David Murray a9719057a2 #8942: add test for __path__ contents in zipimport tests
Patch by Tatiana Al-Chueyr.
2012-03-12 21:16:33 -04:00
Antoine Pitrou 5136ac0ca2 Issue #13645: pyc files now contain the size of the corresponding source
code, to avoid timestamp collisions (especially on filesystems with a low
timestamp resolution) when checking for freshness of the bytecode.
2012-01-13 18:52:16 +01:00
Victor Stinner bf816223df Issue #12451: Add support.create_empty_file()
We don't need to create a temporary buffered binary or text file object just to
create an empty file.

Replace also os.fdopen(handle).close() by os.close(handle).
2011-06-30 23:25:47 +02:00
Victor Stinner 21809a6938 (Merge 3.2) Issue #12124: zipimport doesn't keep a reference to
zlib.decompress() anymore to be able to unload the module.
2011-05-20 00:22:39 +02:00
Victor Stinner 4445ec81c1 (Merge 3.1) Issue #12124: zipimport doesn't keep a reference to
zlib.decompress() anymore to be able to unload the module.
2011-05-20 00:18:58 +02:00
Victor Stinner 4925cde1cc Issue #12124: zipimport doesn't keep a reference to zlib.decompress() anymore
to be able to unload the module.
2011-05-20 00:16:09 +02:00
Ezio Melotti 975077a889 Refactor a few tests to use the new requires_zlib decorator. 2011-05-19 22:03:22 +03:00
Ezio Melotti 19f2aeba67 Merged revisions 86596 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/py3k

........
  r86596 | ezio.melotti | 2010-11-20 21:04:17 +0200 (Sat, 20 Nov 2010) | 1 line

  #9424: Replace deprecated assert* methods in the Python test suite.
........
2010-11-21 01:30:29 +00:00
Ezio Melotti b3aedd4862 #9424: Replace deprecated assert* methods in the Python test suite. 2010-11-20 19:04:17 +00:00
Victor Stinner 36e791179c test_zipimport: fix test name 2010-08-17 00:44:11 +00:00
Victor Stinner 2460a43a65 Issue #9425: read_directory() is fully unicode compliant
zipimport is now able to load a module with an unencodable filename.
2010-08-16 17:54:28 +00:00
Barry Warsaw 28a691b7fd PEP 3147 2010-04-17 00:19:56 +00:00
Georg Brandl 89fad14944 Merged revisions 78018,78035-78040,78042-78043,78046,78048-78052,78054,78059,78075-78080 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r78018 | georg.brandl | 2010-02-06 11:08:21 +0100 (Sa, 06 Feb 2010) | 1 line

  #7864: make deprecation notices a bit clearer.
........
  r78035 | georg.brandl | 2010-02-06 23:44:17 +0100 (Sa, 06 Feb 2010) | 1 line

  Fix duplicate import.
........
  r78036 | georg.brandl | 2010-02-06 23:49:47 +0100 (Sa, 06 Feb 2010) | 1 line

  Remove unused import.
........
  r78037 | georg.brandl | 2010-02-06 23:59:15 +0100 (Sa, 06 Feb 2010) | 1 line

  No need to assign the results of expressions used only for side effects.
........
  r78038 | georg.brandl | 2010-02-07 00:02:29 +0100 (So, 07 Feb 2010) | 1 line

  Add a missing import.
........
  r78039 | georg.brandl | 2010-02-07 00:06:24 +0100 (So, 07 Feb 2010) | 1 line

  Add missing imports.
........
  r78040 | georg.brandl | 2010-02-07 00:08:00 +0100 (So, 07 Feb 2010) | 1 line

  Fix a few UnboundLocalErrors in test_long.
........
  r78042 | georg.brandl | 2010-02-07 00:12:12 +0100 (So, 07 Feb 2010) | 1 line

  Add missing import.
........
  r78043 | georg.brandl | 2010-02-07 00:12:19 +0100 (So, 07 Feb 2010) | 1 line

  Remove duplicate test method.
........
  r78046 | georg.brandl | 2010-02-07 00:18:00 +0100 (So, 07 Feb 2010) | 1 line

  Fix various missing import/unbound name errors.
........
  r78048 | georg.brandl | 2010-02-07 00:23:45 +0100 (So, 07 Feb 2010) | 1 line

  We heard you like test failures so we put unbound locals in your test so that you can fail while you fail.
........
  r78049 | georg.brandl | 2010-02-07 00:33:33 +0100 (So, 07 Feb 2010) | 1 line

  Fix import/access for some identifiers.  _TestSharedCTypes does not seem to be executed?
........
  r78050 | georg.brandl | 2010-02-07 00:34:10 +0100 (So, 07 Feb 2010) | 1 line

  Fix more unbound locals in code paths that do not seem to be used.
........
  r78051 | georg.brandl | 2010-02-07 00:53:52 +0100 (So, 07 Feb 2010) | 1 line

  Add missing import when running these tests standalone.
........
  r78052 | georg.brandl | 2010-02-07 00:54:04 +0100 (So, 07 Feb 2010) | 1 line

  Add missing import when running these tests standalone.
........
  r78054 | georg.brandl | 2010-02-07 00:58:25 +0100 (So, 07 Feb 2010) | 1 line

  Add missing import.
........
  r78059 | georg.brandl | 2010-02-07 12:34:15 +0100 (So, 07 Feb 2010) | 1 line

  Use "regexp" consistently.
........
  r78075 | georg.brandl | 2010-02-07 13:16:12 +0100 (So, 07 Feb 2010) | 1 line

  Fix another duplicated test method.
........
  r78076 | georg.brandl | 2010-02-07 13:19:43 +0100 (So, 07 Feb 2010) | 1 line

  Fix wrong usage of "except X, Y:".
........
  r78077 | georg.brandl | 2010-02-07 13:25:50 +0100 (So, 07 Feb 2010) | 1 line

  Fix two redefined test methods.
........
  r78078 | georg.brandl | 2010-02-07 13:27:06 +0100 (So, 07 Feb 2010) | 1 line

  Fix a redefined test method.
........
  r78079 | georg.brandl | 2010-02-07 13:34:26 +0100 (So, 07 Feb 2010) | 1 line

  Add a minimal test for fnmatchcase().
........
  r78080 | georg.brandl | 2010-02-07 13:55:12 +0100 (So, 07 Feb 2010) | 1 line

  Remove duplicate test method.
........
2010-03-14 10:23:39 +00:00
Benjamin Peterson 577473fe68 use assert[Not]In where appropriate
A patch from Dave Malcolm.
2010-01-19 00:09:57 +00:00
Ezio Melotti 78ea2023d8 Merged revisions 74754 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r74754 | ezio.melotti | 2009-09-12 17:43:43 +0300 (Sat, 12 Sep 2009) | 1 line

  #6026 - fix tests that failed without zlib
........
2009-09-12 18:41:20 +00:00
Georg Brandl ab91fdef1f Merged revisions 73715 via svnmerge from
svn+ssh://svn.python.org/python/branches/py3k

........
  r73715 | benjamin.peterson | 2009-07-01 01:06:06 +0200 (Mi, 01 Jul 2009) | 1 line

  convert old fail* assertions to assert*
........
2009-08-13 08:51:18 +00:00
Benjamin Peterson c9c0f201fe convert old fail* assertions to assert* 2009-06-30 23:06:06 +00:00
Nick Coghlan 9a1d6e3022 Merged revisions 69425 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r69425 | nick.coghlan | 2009-02-08 13:17:00 +1000 (Sun, 08 Feb 2009) | 1 line

  Issue #4512 closeout: Make ZipImport.get_filename() a public method
........
2009-02-08 03:37:27 +00:00
Nick Coghlan f088e5e6cc Merged revisions 67750-67751 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r67750 | nick.coghlan | 2008-12-14 20:54:50 +1000 (Sun, 14 Dec 2008) | 1 line

  Fix several issues relating to access to source code inside zipfiles. Initial work by Alexander Belopolsky. See Misc/NEWS in this checkin for details.
........
  r67751 | nick.coghlan | 2008-12-14 21:09:40 +1000 (Sun, 14 Dec 2008) | 1 line

  Add file that was missed from r67750
........
2008-12-14 11:50:48 +00:00
Benjamin Peterson ee8712cda4 #2621 rename test.test_support to test.support 2008-05-20 21:35:26 +00:00
Alexandre Vassalotti 8ae3e055a5 Merged revisions 63066-63076,63079,63081-63085,63087-63097,63099,63101-63104 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r63066 | georg.brandl | 2008-05-11 10:56:04 -0400 (Sun, 11 May 2008) | 2 lines

  #2709 followup: better description of Tk's pros and cons.
........
  r63067 | georg.brandl | 2008-05-11 11:05:13 -0400 (Sun, 11 May 2008) | 2 lines

  #1326: document and test zipimporter.archive and zipimporter.prefix.
........
  r63068 | georg.brandl | 2008-05-11 11:07:39 -0400 (Sun, 11 May 2008) | 2 lines

  #2816: clarify error messages for EOF while scanning strings.
........
  r63069 | georg.brandl | 2008-05-11 11:17:41 -0400 (Sun, 11 May 2008) | 3 lines

  #2787: Flush stdout after writing test name, helpful when running
  hanging or long-running tests. Patch by Adam Olsen.
........
  r63070 | georg.brandl | 2008-05-11 11:20:16 -0400 (Sun, 11 May 2008) | 3 lines

  #2803: fix wrong invocation of heappush in seldom-reached code.
  Thanks to Matt Harden.
........
  r63073 | benjamin.peterson | 2008-05-11 12:38:07 -0400 (Sun, 11 May 2008) | 2 lines

  broaden .bzrignore
........
  r63076 | andrew.kuchling | 2008-05-11 15:15:52 -0400 (Sun, 11 May 2008) | 1 line

  Add message to test assertion
........
  r63083 | andrew.kuchling | 2008-05-11 16:08:33 -0400 (Sun, 11 May 2008) | 1 line

  Try setting HOME env.var to fix test on Win32
........
  r63092 | georg.brandl | 2008-05-11 16:53:55 -0400 (Sun, 11 May 2008) | 2 lines

  #2809 followup: even better split docstring.
........
  r63094 | georg.brandl | 2008-05-11 17:03:42 -0400 (Sun, 11 May 2008) | 4 lines

  - #2250: Exceptions raised during evaluation of names in rlcompleter's
    ``Completer.complete()`` method are now caught and ignored.
........
  r63095 | georg.brandl | 2008-05-11 17:16:37 -0400 (Sun, 11 May 2008) | 2 lines

  Clarify os.strerror()s exception behavior.
........
  r63097 | georg.brandl | 2008-05-11 17:34:10 -0400 (Sun, 11 May 2008) | 2 lines

  #2535: remove duplicated method.
........
  r63104 | alexandre.vassalotti | 2008-05-11 19:04:27 -0400 (Sun, 11 May 2008) | 2 lines

  Moved the Queue module stub in lib-old.
........
2008-05-16 00:41:41 +00:00
Christian Heimes 7f044315f4 Merged revisions 59774-59783 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r59774 | georg.brandl | 2008-01-06 16:41:50 +0100 (Sun, 06 Jan 2008) | 2 lines

  #1501: document that 0**0 == 1.
........
  r59775 | georg.brandl | 2008-01-06 16:48:20 +0100 (Sun, 06 Jan 2008) | 2 lines

  #759525: document that dir() doesn't return metaclass attrs when given a class as arg.
........
  r59776 | georg.brandl | 2008-01-06 16:55:26 +0100 (Sun, 06 Jan 2008) | 2 lines

  #1615275: clarify return object types of different tempfile factories.
........
  r59777 | georg.brandl | 2008-01-06 17:01:26 +0100 (Sun, 06 Jan 2008) | 2 lines

  #1727024: document that Popen.returncode is set by Popen.poll/wait.
........
  r59778 | georg.brandl | 2008-01-06 17:04:56 +0100 (Sun, 06 Jan 2008) | 2 lines

  #1686390: add example for csv.Sniffer use.
........
  r59779 | georg.brandl | 2008-01-06 17:12:39 +0100 (Sun, 06 Jan 2008) | 2 lines

  #1559684: document that shutil.copy* doesn't copy all metadata on Posix and Windows too.
........
  r59780 | georg.brandl | 2008-01-06 17:17:56 +0100 (Sun, 06 Jan 2008) | 2 lines

  #1582: document __reversed__, patch by Mark Russell.
........
  r59781 | georg.brandl | 2008-01-06 17:22:56 +0100 (Sun, 06 Jan 2008) | 2 lines

  #1499: Document compile() exceptions.
........
  r59782 | georg.brandl | 2008-01-06 17:49:50 +0100 (Sun, 06 Jan 2008) | 2 lines

  #1325: Add docs and tests for zipimporter.archive and zipimporter.prefix.
........
2008-01-06 17:05:40 +00:00
Guido van Rossum 254348e201 Rename buffer -> bytearray. 2007-11-21 19:29:53 +00:00