Fix compileall.compile_dir() ddir= behavior on sub-packages.
Fixes compileall.compile_dir's ddir parameter and compileall command
line flag `-d` to no longer write the wrong pathname to the generated
pyc file for submodules beneath the root of the directory tree being
compiled. This fixes a regression introduced with Python 3.5.
Tests backported from GH 02673352b5, the
implementation is different due to intervening code changes. But still
quiet simple.
Why was the bug ever introduced? The refactoring to add parallel
execution kept the ddir -> dfile computations but discarded the results
instead of sending them to compile_file(). This fixes that. Lack of tests
meant this went unnoticed.
Importing ProcessPoolExecutor may hang or cause an error when the import
accesses urandom on a low resource platform
https://bugs.python.org/issue29877
Unconditional forcing of ``CHECKED_HASH`` invalidation was introduced in
3.7.0 in bpo-29708. The change is bad, as it unconditionally overrides
*invalidation_mode*, even if it was passed as an explicit argument to
``py_compile.compile()`` or ``compileall``. An environment variable
should *never* override an explicit argument to a library function.
That change leads to multiple test failures if the ``SOURCE_DATE_EPOCH``
environment variable is set.
This changes ``py_compile.compile()`` to only look at
``SOURCE_DATE_EPOCH`` if no explicit *invalidation_mode* was specified.
I also made various relevant tests run with explicit control over the
value of ``SOURCE_DATE_EPOCH``.
While looking at this, I noticed that ``zipimport`` does not work
with hash-based .pycs _at all_, though I left the fixes for
subsequent commits.
Python now supports checking bytecode cache up-to-dateness with a hash of the
source contents rather than volatile source metadata. See the PEP for details.
While a fairly straightforward idea, quite a lot of code had to be modified due
to the pervasiveness of pyc implementation details in the codebase. Changes in
this commit include:
- The core changes to importlib to understand how to read, validate, and
regenerate hash-based pycs.
- Support for generating hash-based pycs in py_compile and compileall.
- Modifications to our siphash implementation to support passing a custom
key. We then expose it to importlib through _imp.
- Updates to all places in the interpreter, standard library, and tests that
manually generate or parse pyc files to grok the new format.
- Support in the interpreter command line code for long options like
--check-hash-based-pycs.
- Tests and documentation for all of the above.
Before the argparse conversion, compileall would (sometimes) accept multiple
paths when -d was specified. Afterward, it does not. The corresponding check
in the original code claimed to prevent multiple *directories* from being
specified...but it didn't really work even to do that. So this patch fixes
the regression by invoking the consenting adults rule: if you specify a
combination of arguments to compileall that produces files with inconsistent
destdirs (which you could do before), it is on you.
Patch by Jake Garver.
The concept of .pyo files no longer exists. Now .pyc files have an
optional `opt-` tag which specifies if any extra optimizations beyond
the peepholer were applied.
quiet parameters of compile_{dir, file, path} functions now have
a multilevel value.
Also, -q option of the CLI now have a multilevel value.
Patch by Thomas Kluyver.
Both compileall.compile_dir() and the CLI for compileall now allow for
specifying how many workers to use (or 0 to use all CPUs).
Thanks to Claudiu Popa for the patch.
Before the introduction of filename arguments to compileall it gave semi useful
messages about not being able to 'list' names that weren't valid directories.
This fix restores that behavior. In addition to the test for this case, the
patch also adds a test for the default behavior of compileall when no arguments
are provided, and fixes a bug in one of the previously added tests.
* Patch contributed by Arfrever Frehtes Taifersar Arahesis.
* Test added by Barry
Also, improve Makefile's deletion of __pycache__ directories so e.g. 'make
distclean' doesn't fail if no __pycache__ directories exist.
svn+ssh://pythondev@svn.python.org/python/trunk
........
r69481 | brett.cannon | 2009-02-09 18:07:38 -0800 (Mon, 09 Feb 2009) | 4 lines
compileall used the ctime of bytecode and source to determine if the bytecode
should be recreated. This created a timing hole. Fixed by just doing what
import does; check the mtime and magic number.
........
There's one major and one minor category still unfixed:
doctests are the major category (and I hope to be able to augment the
refactoring tool to refactor bona fide doctests soon);
other code generating print statements in strings is the minor category.
(Oh, and I don't know if the compiler package works.)
Gregory P. Smith
Antoine Pitrou
Dustin Spicuzza
Elvis Pranskevichus
Benjamin Peterson
Martin Panter
Berker Peksag
Brett Cannon
R David Murray
Victor Stinner
Andrew Svetlov
Éric Araujo
Victor Stinner
Georg Brandl
Barry Warsaw
Martin v. Löwis
Matthias Klose
Mark Dickinson
Brett Cannon
Christian Heimes
Guido van Rossum
Raymond Hettinger
Thomas Heller