compileall is now able to use hardlinks to prevent duplicates in a
case when .pyc files for different optimization levels have the same content.
Co-authored-by: Miro Hrončok <miro@hroncok.cz>
Co-authored-by: Victor Stinner <vstinner@python.org>
Fix compileall.compile_dir() ddir= behavior on sub-packages.
Fixes compileall.compile_dir's ddir parameter and compileall command
line flag `-d` to no longer write the wrong pathname to the generated
pyc file for submodules beneath the root of the directory tree being
compiled. This fixes a regression introduced with Python 3.5.
Also marks the _new_ in 3.9 from PR #16012 parameters to compile_dir as keyword only (as that is the only way they will be used) and fixes an omission of them in one place from the docs.
* Raise the limit of maximum path depth to actual recursion limit
* Add posibilities to adjust a path compiled in .pyc file.
Now, you can:
- Strip a part of path from a beggining of path into compiled file
example "-s /test /test/build/real/test.py" → "build/real/test.py"
- Append some new path to a beggining of path into compiled file
example "-p /boo real/test.py" → "/boo/real/test.py"
You can also use both options in the same time. In that case,
striping is done before appending.
* Add a possibility to specify multiple optimization levels
Each optimization level then leads to separated compiled file.
Use `action='append'` instead of `nargs='+'` for the -o option.
Instead of `-o 0 1 2`, specify `-o 0 -o 1 -o 2`. It's more to type,
but much more explicit.
* Add a symlinks limitation feature
This feature allows us to limit byte-compilation of symbolic
links if they are pointing outside specified dir (build root
for example).
Unconditional forcing of ``CHECKED_HASH`` invalidation was introduced in
3.7.0 in bpo-29708. The change is bad, as it unconditionally overrides
*invalidation_mode*, even if it was passed as an explicit argument to
``py_compile.compile()`` or ``compileall``. An environment variable
should *never* override an explicit argument to a library function.
That change leads to multiple test failures if the ``SOURCE_DATE_EPOCH``
environment variable is set.
This changes ``py_compile.compile()`` to only look at
``SOURCE_DATE_EPOCH`` if no explicit *invalidation_mode* was specified.
I also made various relevant tests run with explicit control over the
value of ``SOURCE_DATE_EPOCH``.
While looking at this, I noticed that ``zipimport`` does not work
with hash-based .pycs _at all_, though I left the fixes for
subsequent commits.
In some development setups it is inconvenient or impossible to write bytecode
caches to the code tree, but the bytecode caches are still useful. The
PYTHONPYCACHEPREFIX environment variable allows specifying an alternate
location for cached bytecode files, within which a directory tree mirroring the code
tree will be created. This cache tree is then used (for both reading and writing)
instead of the local `__pycache__` subdirectory within each source directory.
Exposed at runtime as sys.pycache_prefix (defaulting to None), and can
be set from the CLI as "-X pycache_prefix=path".
Patch by Carl Meyer.
Python now supports checking bytecode cache up-to-dateness with a hash of the
source contents rather than volatile source metadata. See the PEP for details.
While a fairly straightforward idea, quite a lot of code had to be modified due
to the pervasiveness of pyc implementation details in the codebase. Changes in
this commit include:
- The core changes to importlib to understand how to read, validate, and
regenerate hash-based pycs.
- Support for generating hash-based pycs in py_compile and compileall.
- Modifications to our siphash implementation to support passing a custom
key. We then expose it to importlib through _imp.
- Updates to all places in the interpreter, standard library, and tests that
manually generate or parse pyc files to grok the new format.
- Support in the interpreter command line code for long options like
--check-hash-based-pycs.
- Tests and documentation for all of the above.
The concept of .pyo files no longer exists. Now .pyc files have an
optional `opt-` tag which specifies if any extra optimizations beyond
the peepholer were applied.
quiet parameters of compile_{dir, file, path} functions now have
a multilevel value.
Also, -q option of the CLI now have a multilevel value.
Patch by Thomas Kluyver.
Both compileall.compile_dir() and the CLI for compileall now allow for
specifying how many workers to use (or 0 to use all CPUs).
Thanks to Claudiu Popa for the patch.
:option: is used to create a link to an option of python, not to mark
up any instance of any arbitrary command-line option. These were
changed to ````.
For modules which do have a command-line interface, lists of options
have been properly marked up with the program/cmdoption directives
combo. Options defined in such blocks can be linked to with :option:
later in the same file, they won’t link to an option of python.
Finally, the markup of command-line fragments in optparse.rst has
been cleaned to use ``x`` instead of ``"x"``, keeping that latter
form for actual Python strings.
Patch by Eli Bendersky and Éric Araujo.
Lumír 'Frenzy' Balhar
Shantanu
Gregory P. Smith
Lumír 'Frenzy' Balhar
Hai Shi
Antoine Pitrou
NAKAMURA Osamu
Elvis Pranskevichus
Carl Meyer
Benjamin Peterson
Berker Peksag
Terry Jan Reedy
Brett Cannon
Serhiy Storchaka
Yury Selivanov
Georg Brandl
Éric Araujo
R. David Murray
Barry Warsaw