Unconditional forcing of ``CHECKED_HASH`` invalidation was introduced in
3.7.0 in bpo-29708. The change is bad, as it unconditionally overrides
*invalidation_mode*, even if it was passed as an explicit argument to
``py_compile.compile()`` or ``compileall``. An environment variable
should *never* override an explicit argument to a library function.
That change leads to multiple test failures if the ``SOURCE_DATE_EPOCH``
environment variable is set.
This changes ``py_compile.compile()`` to only look at
``SOURCE_DATE_EPOCH`` if no explicit *invalidation_mode* was specified.
I also made various relevant tests run with explicit control over the
value of ``SOURCE_DATE_EPOCH``.
While looking at this, I noticed that ``zipimport`` does not work
with hash-based .pycs _at all_, though I left the fixes for
subsequent commits.
(cherry picked from commit a6b3ec5b6d)
Co-authored-by: Elvis Pranskevichus <elvis@magic.io>
To support reproducible builds, the setting of of SOURCE_DATE_EPOCH triggers the py_compile module -- and by extension, compileall -- to forcibly compile with hash-based .pyc files. This eliminates the possibility of timestamp-based .pyc files which vary between builds.
Python now supports checking bytecode cache up-to-dateness with a hash of the
source contents rather than volatile source metadata. See the PEP for details.
While a fairly straightforward idea, quite a lot of code had to be modified due
to the pervasiveness of pyc implementation details in the codebase. Changes in
this commit include:
- The core changes to importlib to understand how to read, validate, and
regenerate hash-based pycs.
- Support for generating hash-based pycs in py_compile and compileall.
- Modifications to our siphash implementation to support passing a custom
key. We then expose it to importlib through _imp.
- Updates to all places in the interpreter, standard library, and tests that
manually generate or parse pyc files to grok the new format.
- Support in the interpreter command line code for long options like
--check-hash-based-pycs.
- Tests and documentation for all of the above.
The concept of .pyo files no longer exists. Now .pyc files have an
optional `opt-` tag which specifies if any extra optimizations beyond
the peepholer were applied.
argument `-` which tells it to read files to compile from stdin. Each line
is read on demand and the named file is compiled immediately. (Original
patch by Piotr Ożarowski).
svn+ssh://pythondev@svn.python.org/python/trunk
........
r79070 | benjamin.peterson | 2010-03-18 17:44:54 -0500 (Thu, 18 Mar 2010) | 1 line
these lines can now be dispensed with
........
svn+ssh://pythondev@svn.python.org/python/trunk
........
r61239 | andrew.kuchling | 2008-03-05 01:44:41 +0100 (Wed, 05 Mar 2008) | 1 line
Add more items; add fragmentary notes
........
r61240 | amaury.forgeotdarc | 2008-03-05 02:50:33 +0100 (Wed, 05 Mar 2008) | 13 lines
Issue#2238: some syntax errors from *args or **kwargs expressions
would give bogus error messages, because of untested exceptions::
>>> f(**g(1=2))
XXX undetected error
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'int' object is not iterable
instead of the expected SyntaxError: keyword can't be an expression
Will backport.
........
r61241 | neal.norwitz | 2008-03-05 06:10:48 +0100 (Wed, 05 Mar 2008) | 3 lines
Remove the files/dirs after closing the DB so the tests work on Windows.
Patch from Trent Nelson. Also simplified removing a file by using test_support.
........
r61242 | neal.norwitz | 2008-03-05 06:14:18 +0100 (Wed, 05 Mar 2008) | 3 lines
Get this test to pass even when there is no sound card in the system.
Patch from Trent Nelson. (I can't test this.)
........
r61243 | neal.norwitz | 2008-03-05 06:20:44 +0100 (Wed, 05 Mar 2008) | 3 lines
Catch OSError when trying to remove a file in case removal fails. This
should prevent a failure in tearDown masking any real test failure.
........
r61244 | neal.norwitz | 2008-03-05 06:38:06 +0100 (Wed, 05 Mar 2008) | 5 lines
Make the timeout longer to give slow machines a chance to pass the test
before timing out. This doesn't change the duration of the test under
normal circumstances. This is targetted at fixing the spurious failures
on the FreeBSD buildbot primarily.
........
r61245 | neal.norwitz | 2008-03-05 06:49:03 +0100 (Wed, 05 Mar 2008) | 1 line
Tabs -> spaces
........
r61246 | neal.norwitz | 2008-03-05 06:50:20 +0100 (Wed, 05 Mar 2008) | 1 line
Use -u urlfetch to run more tests
........
r61247 | neal.norwitz | 2008-03-05 06:51:20 +0100 (Wed, 05 Mar 2008) | 1 line
test_smtplib sometimes reports leaks too, suppress it
........
r61248 | jeffrey.yasskin | 2008-03-05 07:19:56 +0100 (Wed, 05 Mar 2008) | 5 lines
Fix test_socketserver on Windows after r61099 added several signal.alarm()
calls (which don't exist on non-Unix platforms).
Thanks to Trent Nelson for the report and patch.
........
r61249 | georg.brandl | 2008-03-05 08:10:35 +0100 (Wed, 05 Mar 2008) | 2 lines
Fix some rst.
........
r61252 | thomas.heller | 2008-03-05 15:53:39 +0100 (Wed, 05 Mar 2008) | 2 lines
News entry for yesterdays commit.
........
r61253 | thomas.heller | 2008-03-05 16:34:29 +0100 (Wed, 05 Mar 2008) | 3 lines
Issue 1872: Changed the struct module typecode from 't' to '?', for
compatibility with PEP3118.
........
r61254 | skip.montanaro | 2008-03-05 17:41:09 +0100 (Wed, 05 Mar 2008) | 4 lines
Elaborate on the role of the altinstall target when installing multiple
versions.
........
r61255 | georg.brandl | 2008-03-05 20:31:44 +0100 (Wed, 05 Mar 2008) | 2 lines
#2239: PYTHONPATH delimiter is os.pathsep.
........
r61256 | raymond.hettinger | 2008-03-05 21:59:58 +0100 (Wed, 05 Mar 2008) | 1 line
C implementation of itertools.permutations().
........
r61257 | raymond.hettinger | 2008-03-05 22:04:32 +0100 (Wed, 05 Mar 2008) | 1 line
Small code cleanup.
........
r61260 | martin.v.loewis | 2008-03-05 23:24:31 +0100 (Wed, 05 Mar 2008) | 2 lines
cd PCbuild only after deleting all pyc files.
........
r61261 | raymond.hettinger | 2008-03-06 02:15:52 +0100 (Thu, 06 Mar 2008) | 1 line
Add examples.
........
r61262 | andrew.kuchling | 2008-03-06 02:36:27 +0100 (Thu, 06 Mar 2008) | 1 line
Add two items
........
r61263 | georg.brandl | 2008-03-06 07:47:18 +0100 (Thu, 06 Mar 2008) | 2 lines
#1725737: ignore other VC directories other than CVS and SVN's too.
........
r61264 | martin.v.loewis | 2008-03-06 07:55:22 +0100 (Thu, 06 Mar 2008) | 4 lines
Patch #2232: os.tmpfile might fail on Windows if the user has no
permission to create files in the root directory.
Will backport to 2.5.
........
r61269 | georg.brandl | 2008-03-06 08:19:15 +0100 (Thu, 06 Mar 2008) | 2 lines
Expand on re.split behavior with captured expressions.
........
r61270 | georg.brandl | 2008-03-06 08:22:09 +0100 (Thu, 06 Mar 2008) | 2 lines
Little clarification of assignments.
........
r61271 | georg.brandl | 2008-03-06 08:31:34 +0100 (Thu, 06 Mar 2008) | 2 lines
Add isinstance/issubclass to tutorial.
........
r61272 | georg.brandl | 2008-03-06 08:34:52 +0100 (Thu, 06 Mar 2008) | 2 lines
Add missing NEWS entry for r61263.
........
r61273 | georg.brandl | 2008-03-06 08:41:16 +0100 (Thu, 06 Mar 2008) | 2 lines
#2225: return nonzero status code from py_compile if not all files could be compiled.
........
r61274 | georg.brandl | 2008-03-06 08:43:02 +0100 (Thu, 06 Mar 2008) | 2 lines
#2220: handle matching failure more gracefully.
........
r61275 | georg.brandl | 2008-03-06 08:45:52 +0100 (Thu, 06 Mar 2008) | 2 lines
Bug #2220: handle rlcompleter attribute match failure more gracefully.
........
r61278 | martin.v.loewis | 2008-03-06 14:49:47 +0100 (Thu, 06 Mar 2008) | 1 line
Rely on x64 platform configuration when building _bsddb on AMD64.
........
r61279 | martin.v.loewis | 2008-03-06 14:50:28 +0100 (Thu, 06 Mar 2008) | 1 line
Update db-4.4.20 build procedure.
........
r61285 | raymond.hettinger | 2008-03-06 21:52:01 +0100 (Thu, 06 Mar 2008) | 1 line
More tests.
........
r61286 | raymond.hettinger | 2008-03-06 23:51:36 +0100 (Thu, 06 Mar 2008) | 1 line
Issue 2246: itertools grouper object did not participate in GC (should be backported).
........
r61288 | raymond.hettinger | 2008-03-07 02:33:20 +0100 (Fri, 07 Mar 2008) | 1 line
Tweak recipes and tests
........
r61289 | jeffrey.yasskin | 2008-03-07 07:22:15 +0100 (Fri, 07 Mar 2008) | 5 lines
Progress on issue #1193577 by adding a polling .shutdown() method to
SocketServers. The core of the patch was written by Pedro Werneck, but any bugs
are mine. I've also rearranged the code for timeouts in order to avoid
interfering with the shutdown poll.
........
r61290 | nick.coghlan | 2008-03-07 15:13:28 +0100 (Fri, 07 Mar 2008) | 1 line
Speed up with statements by storing the __exit__ method on the stack instead of in a temp variable (bumps the magic number for pyc files)
........
r61298 | andrew.kuchling | 2008-03-07 22:09:23 +0100 (Fri, 07 Mar 2008) | 1 line
Grammar fix
........
r61303 | georg.brandl | 2008-03-08 10:54:06 +0100 (Sat, 08 Mar 2008) | 2 lines
#2253: fix continue vs. finally docs.
........
r61304 | marc-andre.lemburg | 2008-03-08 11:01:43 +0100 (Sat, 08 Mar 2008) | 3 lines
Add new name for Mandrake: Mandriva.
........
r61305 | georg.brandl | 2008-03-08 11:05:24 +0100 (Sat, 08 Mar 2008) | 2 lines
#1533486: fix types in refcount intro.
........
r61312 | facundo.batista | 2008-03-08 17:50:27 +0100 (Sat, 08 Mar 2008) | 5 lines
Issue 1106316. post_mortem()'s parameter, traceback, is now
optional: it defaults to the traceback of the exception that is currently
being handled.
........
r61313 | jeffrey.yasskin | 2008-03-08 19:26:54 +0100 (Sat, 08 Mar 2008) | 2 lines
Add tests for with and finally performance to pybench.
........
r61314 | jeffrey.yasskin | 2008-03-08 21:08:21 +0100 (Sat, 08 Mar 2008) | 2 lines
Fix pybench for pythons < 2.6, tested back to 2.3.
........
r61317 | jeffrey.yasskin | 2008-03-08 22:35:15 +0100 (Sat, 08 Mar 2008) | 3 lines
Well that was dumb. platform.python_implementation returns a function, not a
string.
........
r61329 | georg.brandl | 2008-03-09 16:11:39 +0100 (Sun, 09 Mar 2008) | 2 lines
#2249: document assertTrue and assertFalse.
........
r61332 | neal.norwitz | 2008-03-09 20:03:42 +0100 (Sun, 09 Mar 2008) | 4 lines
Introduce a lock to fix a race condition which caused an exception in the test.
Some buildbots were consistently failing (e.g., amd64).
Also remove a couple of semi-colons.
........
r61344 | raymond.hettinger | 2008-03-11 01:19:07 +0100 (Tue, 11 Mar 2008) | 1 line
Add recipe to docs.
........
r61350 | guido.van.rossum | 2008-03-11 22:18:06 +0100 (Tue, 11 Mar 2008) | 3 lines
Fix the overflows in expandtabs(). "This time for sure!"
(Exploit at request.)
........
r61351 | raymond.hettinger | 2008-03-11 22:37:46 +0100 (Tue, 11 Mar 2008) | 1 line
Improve docs for itemgetter(). Show that it works with slices.
........
r61363 | georg.brandl | 2008-03-13 08:15:56 +0100 (Thu, 13 Mar 2008) | 2 lines
#2265: fix example.
........
r61364 | georg.brandl | 2008-03-13 08:17:14 +0100 (Thu, 13 Mar 2008) | 2 lines
#2270: fix typo.
........
r61365 | georg.brandl | 2008-03-13 08:21:41 +0100 (Thu, 13 Mar 2008) | 2 lines
#1720705: add docs about import/threading interaction, wording by Nick.
........
r61366 | andrew.kuchling | 2008-03-13 12:07:35 +0100 (Thu, 13 Mar 2008) | 1 line
Add class decorators
........
r61367 | raymond.hettinger | 2008-03-13 17:43:17 +0100 (Thu, 13 Mar 2008) | 1 line
Add 2-to-3 support for the itertools moved to builtins or renamed.
........
r61368 | raymond.hettinger | 2008-03-13 17:43:59 +0100 (Thu, 13 Mar 2008) | 1 line
Consistent tense.
........
r61369 | raymond.hettinger | 2008-03-13 20:03:51 +0100 (Thu, 13 Mar 2008) | 1 line
Issue 2274: Add heapq.heappushpop().
........
r61370 | raymond.hettinger | 2008-03-13 20:33:34 +0100 (Thu, 13 Mar 2008) | 1 line
Simplify the nlargest() code using heappushpop().
........
r61371 | brett.cannon | 2008-03-13 21:27:00 +0100 (Thu, 13 Mar 2008) | 4 lines
Move test_thread over to unittest. Commits GHOP 237.
Thanks Benjamin Peterson for the patch.
........
r61372 | brett.cannon | 2008-03-13 21:33:10 +0100 (Thu, 13 Mar 2008) | 4 lines
Move test_tokenize to doctest.
Done as GHOP 238 by Josip Dzolonga.
........
r61373 | brett.cannon | 2008-03-13 21:47:41 +0100 (Thu, 13 Mar 2008) | 4 lines
Convert test_contains, test_crypt, and test_select to unittest.
Patch from GHOP 294 by David Marek.
........
r61374 | brett.cannon | 2008-03-13 22:02:16 +0100 (Thu, 13 Mar 2008) | 4 lines
Move test_gdbm to use unittest.
Closes issue #1960. Thanks Giampaolo Rodola.
........
r61375 | brett.cannon | 2008-03-13 22:09:28 +0100 (Thu, 13 Mar 2008) | 4 lines
Convert test_fcntl to unittest.
Closes issue #2055. Thanks Giampaolo Rodola.
........
r61376 | raymond.hettinger | 2008-03-14 06:03:44 +0100 (Fri, 14 Mar 2008) | 1 line
Leave heapreplace() unchanged.
........
r61378 | martin.v.loewis | 2008-03-14 14:56:09 +0100 (Fri, 14 Mar 2008) | 2 lines
Patch #2284: add -x64 option to rt.bat.
........
r61379 | martin.v.loewis | 2008-03-14 14:57:59 +0100 (Fri, 14 Mar 2008) | 2 lines
Use -x64 flag.
........
r61382 | brett.cannon | 2008-03-14 15:03:10 +0100 (Fri, 14 Mar 2008) | 2 lines
Remove a bad test.
........
r61383 | mark.dickinson | 2008-03-14 15:23:37 +0100 (Fri, 14 Mar 2008) | 9 lines
Issue 705836: Fix struct.pack(">f", 1e40) to behave consistently
across platforms: it should now raise OverflowError on all
platforms. (Previously it raised OverflowError only on
non IEEE 754 platforms.)
Also fix the (already existing) test for this behaviour
so that it actually raises TestFailed instead of just
referencing it.
........
r61387 | thomas.heller | 2008-03-14 22:06:21 +0100 (Fri, 14 Mar 2008) | 1 line
Remove unneeded initializer.
........
r61388 | martin.v.loewis | 2008-03-14 22:19:28 +0100 (Fri, 14 Mar 2008) | 2 lines
Run debug version, cd to PCbuild.
........
r61392 | georg.brandl | 2008-03-15 00:10:34 +0100 (Sat, 15 Mar 2008) | 2 lines
Remove obsolete paragraph. #2288.
........
r61395 | georg.brandl | 2008-03-15 01:20:19 +0100 (Sat, 15 Mar 2008) | 2 lines
Fix lots of broken links in the docs, found by Sphinx' external link checker.
........
r61396 | skip.montanaro | 2008-03-15 03:32:49 +0100 (Sat, 15 Mar 2008) | 1 line
note that fork and forkpty raise OSError on failure
........
r61402 | skip.montanaro | 2008-03-15 17:04:45 +0100 (Sat, 15 Mar 2008) | 1 line
add %f format to datetime - issue 1158
........
r61403 | skip.montanaro | 2008-03-15 17:07:11 +0100 (Sat, 15 Mar 2008) | 2 lines
.
........
The zipfile module now does all I/O in binary mode using bytes.
(Maybe we should support wrapping a TextIOWrapper around it
when text mode reading is requested?)
Even the password is a bytes array now.
Had to fix py_compile.py to use bytes while I was at it.
The _struct needed a patch to support bytes, str8 and str
for the 's' and 'p' formats.
changed to use universal newlines.
Remove all imports from the compile() function; these are
now done at the top of the module ("Python normal form"),
and define a helper based on the platform instead of
testing the platform in the compile() function.
Highlights: import and friends will understand any of \r, \n and \r\n
as end of line. Python file input will do the same if you use mode 'U'.
Everything can be disabled by configuring with --without-universal-newlines.
See PEP278 for details.
who writes:
Here is batch 2, as a big collection of CVS context diffs.
Along with moving comments into docstrings, i've added a
couple of missing docstrings and attempted to make sure more
module docstrings begin with a one-line summary.
I did not add docstrings to the methods in profile.py for
fear of upsetting any careful optimizations there, though
i did move class documentation into class docstrings.
The convention i'm using is to leave credits/version/copyright
type of stuff in # comments, and move the rest of the descriptive
stuff about module usage into module docstrings. Hope this is
okay.
there's a syntax error. (In particular, display the correct
filename). This changes the API: if there's a syntax error, the
function now returns normally after dumping the error to sys.stderr.
I changed Sjoerd's use of string.join(string.split(...)) with
string.replace().
Miss Islington (bot)
Bernhard M. Wiedemann
Benjamin Peterson
Eric Snow
Berker Peksag
Brett Cannon
Andrew Svetlov
Antoine Pitrou
Meador Inge
Georg Brandl
Victor Stinner
Matthias Klose
Barry Warsaw
Christian Heimes
Guido van Rossum
Jack Jansen
Tim Peters
Martin v. Löwis
Fred Drake
Raymond Hettinger
Skip Montanaro
Eric S. Raymond