Commit Graph

77 Commits

Author SHA1 Message Date
Shantanu b4fe02f595
gh-117205: Increase chunksize when compiling pyc in parallel (#117206) 2024-04-03 15:24:24 -07:00
hetmankp 3726cb0f14
gh-105931: Fix surprising compileall stripdir behaviour (GH-108671)
Before, the '-s STRIPDIR' option on
compileall lead to some surprising results as it only strips away
path components that match, but leaves alone the non-matching ones
interspersed in between.

For example, with: python -m compileall -s/path/to/another/src
/path/to/build/src/file.py

The resulting written path will be: build/file.py

This fix only strips directories that are a fully matching prefix of the
source path. If a stripdir is provided that is not a valid prefix, a
warning will be displayed (which can be silenced with '-qq').
2023-10-23 13:55:39 +00:00
Gregory P. Smith 0ca67e6313
GH-84559: Deprecate fork being the multiprocessing default. (#100618)
This starts the process. Users who don't specify their own start method
and use the default on platforms where it is 'fork' will see a
DeprecationWarning upon multiprocessing.Pool() construction or upon
multiprocessing.Process.start() or concurrent.futures.ProcessPool use.

See the related issue and documentation within this change for details.
2023-02-02 15:50:35 -08:00
Filipe Laíns 1ecfd1ebf1
bpo-40447: accept all path-like objects in compileall.compile_file (#19883)
Signed-off-by: Filipe Laíns <lains@archlinux.org>
Signed-off-by: Filipe Laíns <lains@riseup.net>
Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>
Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
2022-12-23 14:19:16 -06:00
Vishal Pandey 87a18deda4
Fixed documentation typo in compileall.py (GH-29912) 2021-12-05 00:38:17 +09:00
Daniel Hahler 74cc2453ae
Fix missing space with help for `-m compileall -o` (GH-27591) 2021-09-18 00:28:09 +02:00
Ammar Askar bb21e28fd0
bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int (GH-19708)
Co-authored-by: Stéphane Wirtel <stephane@wirtel.be>
2021-08-24 12:13:32 +03:00
Stefan Hoelzl 80f0707629
bpo-44666: Use default encoding as fallback for compile_file (GH-27236)
When sys.stdout.encoding is None compile_file will fall back to
sys.getdefaultencoding to encode/decode error messages.

Co-authored-by: Stefan Hoelzl <stefan.hoelzl@posteo.de>
Co-authored-by: Mickaël Schoentgen <contact@tiger-222.fr>
2021-07-30 18:38:42 +02:00
Inada Naoki 80017752ba
bpo-43651: Fix test_compileall with PEP 597 (GH-25128) 2021-04-02 09:01:57 +09:00
Asheesh Laroia bf2e7e55d7
bpo-40692: Run more test_concurrent_futures tests (GH-20239)
In the case of multiprocessing.synchronize() being missing, the
test_concurrent_futures test suite now skips only the tests that
require multiprocessing.synchronize().

Validate that multiprocessing.synchronize exists as part of
_check_system_limits(), allowing ProcessPoolExecutor to raise
NotImplementedError during __init__, rather than crashing with
ImportError during __init__ when creating a lock imported from
multiprocessing.synchronize.

Use _check_system_limits() to disable tests of
ProcessPoolExecutor on systems without multiprocessing.synchronize.

Running the test suite without multiprocessing.synchronize reveals
that Lib/compileall.py crashes when it uses a ProcessPoolExecutor.
Therefore, change Lib/compileall.py to call _check_system_limits()
before creating the ProcessPoolExecutor.

Note that both Lib/compileall.py and Lib/test/test_compileall.py
were attempting to sanity-check ProcessPoolExecutor by expecting
ImportError. In multiprocessing.resource_tracker, sem_unlink() is also absent
on platforms where POSIX semaphores aren't available. Avoid using
sem_unlink() if it, too, does not exist.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2021-02-08 03:15:51 +00:00
Lumír 'Frenzy' Balhar e77d428856
bpo-40495: compileall option to hardlink duplicate pyc files (GH-19901)
compileall is now able to use hardlinks to prevent duplicates in a
case when .pyc files for different optimization levels have the same content.

Co-authored-by: Miro Hrončok <miro@hroncok.cz>
Co-authored-by: Victor Stinner <vstinner@python.org>
2020-05-14 16:17:22 +02:00
Jelle Zijlstra efb8dd5b3e
compileall: Fix typos in docstring (GH-19810) 2020-04-30 16:09:11 +05:30
Gregory P. Smith 02673352b5
bpo-39769: Fix compileall ddir for subpkgs. (GH-18676)
Fix compileall.compile_dir() ddir= behavior on sub-packages.

Fixes compileall.compile_dir's ddir parameter and compileall command
line flag `-d` to no longer write the wrong pathname to the generated
pyc file for submodules beneath the root of the directory tree being
compiled.  This fixes a regression introduced with Python 3.5.

Also marks the _new_ in 3.9 from PR #16012 parameters to compile_dir as keyword only (as that is the only way they will be used) and fixes an omission of them in one place from the docs.
2020-02-28 17:28:37 -08:00
Victor Stinner eb1dda2b56 bpo-38470: Fix test_compileall.test_compile_dir_maxlevels() (GH-16789)
Fix test_compile_dir_maxlevels() on Windows without long path
support: only create 3 subdirectories instead of between 20 and 100
subdirectories.

Fix also compile_dir() to use the current sys.getrecursionlimit()
value as the default maxlevels value, rather than using
sys.getrecursionlimit() value read at startup.
2019-10-15 11:26:13 +02:00
Lumír 'Frenzy' Balhar 8e7bb991de bpo-38112: Compileall improvements (GH-16012)
* Raise the limit of maximum path depth to actual  recursion limit

* Add posibilities to adjust a path compiled in .pyc  file.

Now, you can:
- Strip a part of path from a beggining of path into compiled file
   example "-s /test /test/build/real/test.py" → "build/real/test.py"
- Append some new path to a beggining of path into compiled file
   example "-p /boo real/test.py" → "/boo/real/test.py"

You can also use both options in the same time. In that case,
striping is done before appending.

* Add a possibility to specify multiple optimization levels

Each optimization level then leads to separated compiled file.
Use `action='append'` instead of `nargs='+'` for the -o option.
Instead of `-o 0 1 2`, specify `-o 0 -o 1 -o 2`. It's more to type,
but much more explicit.

* Add a symlinks limitation feature

This feature allows us to limit byte-compilation of symbolic
links if they are pointing outside specified dir (build root
for example).
2019-09-26 08:28:26 +02:00
Antoine Pitrou 1a2dd82f56
bpo-36786: Run compileall in parallel during "make install" (GH-13078) 2019-05-15 23:45:18 +02:00
Dustin Spicuzza 1d817e4c82 bpo-29877: compileall: import ProcessPoolExecutor only when needed (GH-4856)
Importing ProcessPoolExecutor may hang or cause an error when the import
accesses urandom on a low resource platform


https://bugs.python.org/issue29877
2018-11-23 09:06:55 -08:00
Elvis Pranskevichus a6b3ec5b6d bpo-34022: Stop forcing of hash-based invalidation with SOURCE_DATE_EPOCH (GH-9607)
Unconditional forcing of ``CHECKED_HASH`` invalidation was introduced in
3.7.0 in bpo-29708.  The change is bad, as it unconditionally overrides
*invalidation_mode*, even if it was passed as an explicit argument to
``py_compile.compile()`` or ``compileall``.  An environment variable
should *never* override an explicit argument to a library function.
That change leads to multiple test failures if the ``SOURCE_DATE_EPOCH``
environment variable is set.

This changes ``py_compile.compile()`` to only look at
``SOURCE_DATE_EPOCH`` if no explicit *invalidation_mode* was specified.
I also made various relevant tests run with explicit control over the
value of ``SOURCE_DATE_EPOCH``.

While looking at this, I noticed that ``zipimport`` does not work
with hash-based .pycs _at all_, though I left the fixes for
subsequent commits.
2018-10-10 18:43:14 +02:00
Benjamin Peterson 42aa93b8ff
closes bpo-31650: PEP 552 (Deterministic pycs) implementation (#4575)
Python now supports checking bytecode cache up-to-dateness with a hash of the
source contents rather than volatile source metadata. See the PEP for details.

While a fairly straightforward idea, quite a lot of code had to be modified due
to the pervasiveness of pyc implementation details in the codebase. Changes in
this commit include:

- The core changes to importlib to understand how to read, validate, and
  regenerate hash-based pycs.

- Support for generating hash-based pycs in py_compile and compileall.

- Modifications to our siphash implementation to support passing a custom
  key. We then expose it to importlib through _imp.

- Updates to all places in the interpreter, standard library, and tests that
  manually generate or parse pyc files to grok the new format.

- Support in the interpreter command line code for long options like
  --check-hash-based-pycs.

- Tests and documentation for all of the above.
2017-12-09 10:26:52 -08:00
Martin Panter cee93c3b73 Issue #28485: Merge single-threading fix from 3.5 into 3.6 2016-11-05 01:35:25 +00:00
Martin Panter 88281ceed0 Issue #28485: Check for negative workers even without ProcessPoolExecutor
This matches the documentation, and passes the test suite when multithreading
is disabled.
2016-11-05 01:11:36 +00:00
Berker Peksag 812a2b65af Issue #28226: compileall now supports pathlib 2016-10-01 00:54:18 +03:00
Brett Cannon 1e3c3e906c Issue #25768: Make compileall functions return booleans and document
the return values as well as test them.

Thanks to Nicholas Chammas for the bug report and initial patch.
2015-12-27 13:17:04 -08:00
R David Murray 939614c48c Merge: #24903: Remove misleading error message to fix regression. 2015-12-04 23:04:37 -05:00
R David Murray ced699b4c7 #24903: Remove misleading error message to fix regression.
Before the argparse conversion, compileall would (sometimes) accept multiple
paths when -d was specified.  Afterward, it does not.  The corresponding check
in the original code claimed to prevent multiple *directories* from being
specified...but it didn't really work even to do that.  So this patch fixes
the regression by invoking the consenting adults rule: if you specify a
combination of arguments to compileall that produces files with inconsistent
destdirs (which you could do before), it is on you.

Patch by Jake Garver.
2015-12-04 22:54:38 -05:00
Berker Peksag d86ef05a02 Issue #23917: Fall back to sequential compilation when ProcessPoolExecutor doesn't exist.
Patch by Claudiu Popa.
2015-04-22 09:39:19 +03:00
Brett Cannon f299abdafa Issue #23731: Implement PEP 488.
The concept of .pyo files no longer exists. Now .pyc files have an
optional `opt-` tag which specifies if any extra optimizations beyond
the peepholer were applied.
2015-04-13 14:21:02 -04:00
Berker Peksag 6554b86b1f Issue #21338: Add silent mode for compileall.
quiet parameters of compile_{dir, file, path} functions now have
a multilevel value.

Also, -q option of the CLI now have a multilevel value.

Patch by Thomas Kluyver.
2014-10-15 11:10:57 +03:00
Brett Cannon f1a8df0ac9 Issue #16104: Allow compileall to do parallel bytecode compilation.
Both compileall.compile_dir() and the CLI for compileall now allow for
specifying how many workers to use (or 0 to use all CPUs).

Thanks to Claudiu Popa for the patch.
2014-09-12 10:39:48 -04:00
Benjamin Peterson 344ff4ab2b allow recursion depth to be specified (closes #19628)
Patch from Claudiu Popa.
2014-08-19 16:13:26 -05:00
Victor Stinner 7fa767e517 Issue #20976: pyflakes: Remove unused imports 2014-03-20 09:16:38 +01:00
R David Murray 755d5ea12a Merge: #19532: make compileall with no file/dir args respect -f and -q. 2013-12-15 20:56:00 -05:00
R David Murray 8a1d1e647e #19532: make compileall with no file/dir args respect -f and -q.
Patch by Vajrasky Kok.
2013-12-15 20:49:38 -05:00
Brett Cannon 7822e123c4 Issue #17177: stop using imp for compileall. 2013-06-14 23:04:02 -04:00
Andrew Svetlov f7a17b48d7 Replace IOError with OSError (#16715) 2012-12-25 16:47:37 +02:00
Andrew Svetlov ad28c7f9da Issue #16706: get rid of os.error 2012-12-18 22:02:39 +02:00
Andrew Svetlov 3438fa496d Get rig of EnvironmentError (#16705) 2012-12-17 23:35:18 +02:00
Éric Araujo 3b371cfeb8 #10454: a few edits to compileall help messages 2011-09-01 20:00:33 +02:00
Victor Stinner 530712625b Issue #11169: compileall module uses repr() to format filenames and paths to
escape surrogate characters and show spaces.
2011-05-11 00:36:28 +02:00
Georg Brandl 4543846517 #11132: pass optimize parameter to recursive call in compileall.compile_dir(). Reviewed by Eric A. 2011-02-07 12:36:54 +00:00
Éric Araujo 413d7b4f24 Nits: use a real boolean, make one docstring more similar to the other ones 2010-12-23 18:44:31 +00:00
R. David Murray 94f58c3a65 #10454: clarify the compileall docs and help messages. 2010-12-17 16:29:07 +00:00
R. David Murray 5317e9cd8d #10719: restore messages generated on invalid compileall args
Before the introduction of filename arguments to compileall it gave semi useful
messages about not being able to 'list' names that weren't valid directories.
This fix restores that behavior.  In addition to the test for this case, the
patch also adds a test for the default behavior of compileall when no arguments
are provided, and fixes a bug in one of the previously added tests.
2010-12-16 19:08:51 +00:00
R. David Murray 95333e3aa9 More comprehensive compileall cli tests, and fixes. 2010-12-14 22:32:50 +00:00
Georg Brandl 8334fd9285 Add an "optimize" parameter to compile() to control the optimization level, and provide an interface to it in py_compile, compileall and PyZipFile. 2010-12-04 10:26:46 +00:00
Éric Araujo 2e579f0a87 Fix typos and style in compileall. 2010-11-20 21:53:02 +00:00
R. David Murray 650f147298 #10453: compileall now uses argparse instead of getopt, so -h works.
Patch by Michele Orrù.
2010-11-20 21:18:51 +00:00
Benjamin Peterson 25216baf32 Create __pycache__ dir when the pyc path is explicitly given
Patch from Arfrever Frehtes Taifersar Arahesis.
2010-05-08 19:52:21 +00:00
Barry Warsaw c8a99de751 Bug 8563 - compileall.compile_file() creates empty __pycache__ directories in
data directories where there is no source.

Fix by: Arfrever Frehtes Taifersar Arahesis (Arfrever)
Test by: Barry
2010-04-29 18:43:10 +00:00
Barry Warsaw c04317fdc4 Bug 8527 - multiple compileall calls produce cascading __pycache__ directories.
* Patch contributed by Arfrever Frehtes Taifersar Arahesis.
* Test added by Barry

Also, improve Makefile's deletion of __pycache__ directories so e.g. 'make
distclean' doesn't fail if no __pycache__ directories exist.
2010-04-26 15:59:03 +00:00