Commit Graph

69 Commits

Author SHA1 Message Date
Inada Naoki 80017752ba
bpo-43651: Fix test_compileall with PEP 597 (GH-25128) 2021-04-02 09:01:57 +09:00
Asheesh Laroia bf2e7e55d7
bpo-40692: Run more test_concurrent_futures tests (GH-20239)
In the case of multiprocessing.synchronize() being missing, the
test_concurrent_futures test suite now skips only the tests that
require multiprocessing.synchronize().

Validate that multiprocessing.synchronize exists as part of
_check_system_limits(), allowing ProcessPoolExecutor to raise
NotImplementedError during __init__, rather than crashing with
ImportError during __init__ when creating a lock imported from
multiprocessing.synchronize.

Use _check_system_limits() to disable tests of
ProcessPoolExecutor on systems without multiprocessing.synchronize.

Running the test suite without multiprocessing.synchronize reveals
that Lib/compileall.py crashes when it uses a ProcessPoolExecutor.
Therefore, change Lib/compileall.py to call _check_system_limits()
before creating the ProcessPoolExecutor.

Note that both Lib/compileall.py and Lib/test/test_compileall.py
were attempting to sanity-check ProcessPoolExecutor by expecting
ImportError. In multiprocessing.resource_tracker, sem_unlink() is also absent
on platforms where POSIX semaphores aren't available. Avoid using
sem_unlink() if it, too, does not exist.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2021-02-08 03:15:51 +00:00
Lumír 'Frenzy' Balhar e77d428856
bpo-40495: compileall option to hardlink duplicate pyc files (GH-19901)
compileall is now able to use hardlinks to prevent duplicates in a
case when .pyc files for different optimization levels have the same content.

Co-authored-by: Miro Hrončok <miro@hroncok.cz>
Co-authored-by: Victor Stinner <vstinner@python.org>
2020-05-14 16:17:22 +02:00
Jelle Zijlstra efb8dd5b3e
compileall: Fix typos in docstring (GH-19810) 2020-04-30 16:09:11 +05:30
Gregory P. Smith 02673352b5
bpo-39769: Fix compileall ddir for subpkgs. (GH-18676)
Fix compileall.compile_dir() ddir= behavior on sub-packages.

Fixes compileall.compile_dir's ddir parameter and compileall command
line flag `-d` to no longer write the wrong pathname to the generated
pyc file for submodules beneath the root of the directory tree being
compiled.  This fixes a regression introduced with Python 3.5.

Also marks the _new_ in 3.9 from PR #16012 parameters to compile_dir as keyword only (as that is the only way they will be used) and fixes an omission of them in one place from the docs.
2020-02-28 17:28:37 -08:00
Victor Stinner eb1dda2b56 bpo-38470: Fix test_compileall.test_compile_dir_maxlevels() (GH-16789)
Fix test_compile_dir_maxlevels() on Windows without long path
support: only create 3 subdirectories instead of between 20 and 100
subdirectories.

Fix also compile_dir() to use the current sys.getrecursionlimit()
value as the default maxlevels value, rather than using
sys.getrecursionlimit() value read at startup.
2019-10-15 11:26:13 +02:00
Lumír 'Frenzy' Balhar 8e7bb991de bpo-38112: Compileall improvements (GH-16012)
* Raise the limit of maximum path depth to actual  recursion limit

* Add posibilities to adjust a path compiled in .pyc  file.

Now, you can:
- Strip a part of path from a beggining of path into compiled file
   example "-s /test /test/build/real/test.py" → "build/real/test.py"
- Append some new path to a beggining of path into compiled file
   example "-p /boo real/test.py" → "/boo/real/test.py"

You can also use both options in the same time. In that case,
striping is done before appending.

* Add a possibility to specify multiple optimization levels

Each optimization level then leads to separated compiled file.
Use `action='append'` instead of `nargs='+'` for the -o option.
Instead of `-o 0 1 2`, specify `-o 0 -o 1 -o 2`. It's more to type,
but much more explicit.

* Add a symlinks limitation feature

This feature allows us to limit byte-compilation of symbolic
links if they are pointing outside specified dir (build root
for example).
2019-09-26 08:28:26 +02:00
Antoine Pitrou 1a2dd82f56
bpo-36786: Run compileall in parallel during "make install" (GH-13078) 2019-05-15 23:45:18 +02:00
Dustin Spicuzza 1d817e4c82 bpo-29877: compileall: import ProcessPoolExecutor only when needed (GH-4856)
Importing ProcessPoolExecutor may hang or cause an error when the import
accesses urandom on a low resource platform


https://bugs.python.org/issue29877
2018-11-23 09:06:55 -08:00
Elvis Pranskevichus a6b3ec5b6d bpo-34022: Stop forcing of hash-based invalidation with SOURCE_DATE_EPOCH (GH-9607)
Unconditional forcing of ``CHECKED_HASH`` invalidation was introduced in
3.7.0 in bpo-29708.  The change is bad, as it unconditionally overrides
*invalidation_mode*, even if it was passed as an explicit argument to
``py_compile.compile()`` or ``compileall``.  An environment variable
should *never* override an explicit argument to a library function.
That change leads to multiple test failures if the ``SOURCE_DATE_EPOCH``
environment variable is set.

This changes ``py_compile.compile()`` to only look at
``SOURCE_DATE_EPOCH`` if no explicit *invalidation_mode* was specified.
I also made various relevant tests run with explicit control over the
value of ``SOURCE_DATE_EPOCH``.

While looking at this, I noticed that ``zipimport`` does not work
with hash-based .pycs _at all_, though I left the fixes for
subsequent commits.
2018-10-10 18:43:14 +02:00
Benjamin Peterson 42aa93b8ff
closes bpo-31650: PEP 552 (Deterministic pycs) implementation (#4575)
Python now supports checking bytecode cache up-to-dateness with a hash of the
source contents rather than volatile source metadata. See the PEP for details.

While a fairly straightforward idea, quite a lot of code had to be modified due
to the pervasiveness of pyc implementation details in the codebase. Changes in
this commit include:

- The core changes to importlib to understand how to read, validate, and
  regenerate hash-based pycs.

- Support for generating hash-based pycs in py_compile and compileall.

- Modifications to our siphash implementation to support passing a custom
  key. We then expose it to importlib through _imp.

- Updates to all places in the interpreter, standard library, and tests that
  manually generate or parse pyc files to grok the new format.

- Support in the interpreter command line code for long options like
  --check-hash-based-pycs.

- Tests and documentation for all of the above.
2017-12-09 10:26:52 -08:00
Martin Panter cee93c3b73 Issue #28485: Merge single-threading fix from 3.5 into 3.6 2016-11-05 01:35:25 +00:00
Martin Panter 88281ceed0 Issue #28485: Check for negative workers even without ProcessPoolExecutor
This matches the documentation, and passes the test suite when multithreading
is disabled.
2016-11-05 01:11:36 +00:00
Berker Peksag 812a2b65af Issue #28226: compileall now supports pathlib 2016-10-01 00:54:18 +03:00
Brett Cannon 1e3c3e906c Issue #25768: Make compileall functions return booleans and document
the return values as well as test them.

Thanks to Nicholas Chammas for the bug report and initial patch.
2015-12-27 13:17:04 -08:00
R David Murray 939614c48c Merge: #24903: Remove misleading error message to fix regression. 2015-12-04 23:04:37 -05:00
R David Murray ced699b4c7 #24903: Remove misleading error message to fix regression.
Before the argparse conversion, compileall would (sometimes) accept multiple
paths when -d was specified.  Afterward, it does not.  The corresponding check
in the original code claimed to prevent multiple *directories* from being
specified...but it didn't really work even to do that.  So this patch fixes
the regression by invoking the consenting adults rule: if you specify a
combination of arguments to compileall that produces files with inconsistent
destdirs (which you could do before), it is on you.

Patch by Jake Garver.
2015-12-04 22:54:38 -05:00
Berker Peksag d86ef05a02 Issue #23917: Fall back to sequential compilation when ProcessPoolExecutor doesn't exist.
Patch by Claudiu Popa.
2015-04-22 09:39:19 +03:00
Brett Cannon f299abdafa Issue #23731: Implement PEP 488.
The concept of .pyo files no longer exists. Now .pyc files have an
optional `opt-` tag which specifies if any extra optimizations beyond
the peepholer were applied.
2015-04-13 14:21:02 -04:00
Berker Peksag 6554b86b1f Issue #21338: Add silent mode for compileall.
quiet parameters of compile_{dir, file, path} functions now have
a multilevel value.

Also, -q option of the CLI now have a multilevel value.

Patch by Thomas Kluyver.
2014-10-15 11:10:57 +03:00
Brett Cannon f1a8df0ac9 Issue #16104: Allow compileall to do parallel bytecode compilation.
Both compileall.compile_dir() and the CLI for compileall now allow for
specifying how many workers to use (or 0 to use all CPUs).

Thanks to Claudiu Popa for the patch.
2014-09-12 10:39:48 -04:00
Benjamin Peterson 344ff4ab2b allow recursion depth to be specified (closes #19628)
Patch from Claudiu Popa.
2014-08-19 16:13:26 -05:00
Victor Stinner 7fa767e517 Issue #20976: pyflakes: Remove unused imports 2014-03-20 09:16:38 +01:00
R David Murray 755d5ea12a Merge: #19532: make compileall with no file/dir args respect -f and -q. 2013-12-15 20:56:00 -05:00
R David Murray 8a1d1e647e #19532: make compileall with no file/dir args respect -f and -q.
Patch by Vajrasky Kok.
2013-12-15 20:49:38 -05:00
Brett Cannon 7822e123c4 Issue #17177: stop using imp for compileall. 2013-06-14 23:04:02 -04:00
Andrew Svetlov f7a17b48d7 Replace IOError with OSError (#16715) 2012-12-25 16:47:37 +02:00
Andrew Svetlov ad28c7f9da Issue #16706: get rid of os.error 2012-12-18 22:02:39 +02:00
Andrew Svetlov 3438fa496d Get rig of EnvironmentError (#16705) 2012-12-17 23:35:18 +02:00
Éric Araujo 3b371cfeb8 #10454: a few edits to compileall help messages 2011-09-01 20:00:33 +02:00
Victor Stinner 530712625b Issue #11169: compileall module uses repr() to format filenames and paths to
escape surrogate characters and show spaces.
2011-05-11 00:36:28 +02:00
Georg Brandl 4543846517 #11132: pass optimize parameter to recursive call in compileall.compile_dir(). Reviewed by Eric A. 2011-02-07 12:36:54 +00:00
Éric Araujo 413d7b4f24 Nits: use a real boolean, make one docstring more similar to the other ones 2010-12-23 18:44:31 +00:00
R. David Murray 94f58c3a65 #10454: clarify the compileall docs and help messages. 2010-12-17 16:29:07 +00:00
R. David Murray 5317e9cd8d #10719: restore messages generated on invalid compileall args
Before the introduction of filename arguments to compileall it gave semi useful
messages about not being able to 'list' names that weren't valid directories.
This fix restores that behavior.  In addition to the test for this case, the
patch also adds a test for the default behavior of compileall when no arguments
are provided, and fixes a bug in one of the previously added tests.
2010-12-16 19:08:51 +00:00
R. David Murray 95333e3aa9 More comprehensive compileall cli tests, and fixes. 2010-12-14 22:32:50 +00:00
Georg Brandl 8334fd9285 Add an "optimize" parameter to compile() to control the optimization level, and provide an interface to it in py_compile, compileall and PyZipFile. 2010-12-04 10:26:46 +00:00
Éric Araujo 2e579f0a87 Fix typos and style in compileall. 2010-11-20 21:53:02 +00:00
R. David Murray 650f147298 #10453: compileall now uses argparse instead of getopt, so -h works.
Patch by Michele Orrù.
2010-11-20 21:18:51 +00:00
Benjamin Peterson 25216baf32 Create __pycache__ dir when the pyc path is explicitly given
Patch from Arfrever Frehtes Taifersar Arahesis.
2010-05-08 19:52:21 +00:00
Barry Warsaw c8a99de751 Bug 8563 - compileall.compile_file() creates empty __pycache__ directories in
data directories where there is no source.

Fix by: Arfrever Frehtes Taifersar Arahesis (Arfrever)
Test by: Barry
2010-04-29 18:43:10 +00:00
Barry Warsaw c04317fdc4 Bug 8527 - multiple compileall calls produce cascading __pycache__ directories.
* Patch contributed by Arfrever Frehtes Taifersar Arahesis.
* Test added by Barry

Also, improve Makefile's deletion of __pycache__ directories so e.g. 'make
distclean' doesn't fail if no __pycache__ directories exist.
2010-04-26 15:59:03 +00:00
Barry Warsaw 28a691b7fd PEP 3147 2010-04-17 00:19:56 +00:00
Benjamin Peterson 886af966d8 Merged revisions 78966,78970,79018,79026-79027,79055,79156,79159,79163-79164,79173,79176,79194,79208,79212 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r78966 | florent.xicluna | 2010-03-14 10:20:59 -0500 (Sun, 14 Mar 2010) | 2 lines

  Do not hardcode Expat version.  It's possible to build Python with --with-system-expat option.
........
  r78970 | benjamin.peterson | 2010-03-14 21:58:24 -0500 (Sun, 14 Mar 2010) | 1 line

  this little exception dance is pointless
........
  r79018 | collin.winter | 2010-03-16 22:04:01 -0500 (Tue, 16 Mar 2010) | 1 line

  Delete unused import.
........
  r79026 | vinay.sajip | 2010-03-17 10:05:57 -0500 (Wed, 17 Mar 2010) | 1 line

  Issue #8162: logging: Clarified docstring and documentation for disable function.
........
  r79027 | collin.winter | 2010-03-17 12:36:16 -0500 (Wed, 17 Mar 2010) | 1 line

  Avoid hardcoding refcounts in tests.
........
  r79055 | benjamin.peterson | 2010-03-18 16:30:48 -0500 (Thu, 18 Mar 2010) | 1 line

  remove installation of deleted test/output dir
........
  r79156 | florent.xicluna | 2010-03-20 17:21:02 -0500 (Sat, 20 Mar 2010) | 2 lines

  Cleanup test_struct using check_warnings.
........
  r79159 | florent.xicluna | 2010-03-20 17:26:42 -0500 (Sat, 20 Mar 2010) | 2 lines

  Cleanup test_tarfile, and use check_warnings.
........
  r79163 | michael.foord | 2010-03-20 19:53:39 -0500 (Sat, 20 Mar 2010) | 1 line

  A faulty load_tests in a test module no longer halts test discovery. A placeholder test, that reports the failure, is created instead.
........
  r79164 | michael.foord | 2010-03-20 19:55:58 -0500 (Sat, 20 Mar 2010) | 1 line

  Change order of arguments in a unittest function.
........
  r79173 | georg.brandl | 2010-03-21 04:09:38 -0500 (Sun, 21 Mar 2010) | 1 line

  Document that GzipFile supports iteration.
........
  r79176 | georg.brandl | 2010-03-21 04:17:41 -0500 (Sun, 21 Mar 2010) | 1 line

  Introduce copy by slicing, used in later chapters.
........
  r79194 | florent.xicluna | 2010-03-21 06:58:11 -0500 (Sun, 21 Mar 2010) | 2 lines

  Use assertRaises and add a specific warning filter.
........
  r79208 | andrew.kuchling | 2010-03-21 13:47:12 -0500 (Sun, 21 Mar 2010) | 1 line

  Add items
........
  r79212 | georg.brandl | 2010-03-21 14:01:38 -0500 (Sun, 21 Mar 2010) | 1 line

  Fix plural.
........
2010-03-21 23:13:07 +00:00
Martin v. Löwis 4b00307425 Issue #6716/2: Backslash-replace error output in compilall. 2010-03-16 13:19:21 +00:00
Matthias Klose c33b902ebb Merged revisions 78983,78985 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r78983 | matthias.klose | 2010-03-15 18:44:12 +0100 (Mo, 15 Mär 2010) | 2 lines

  - Issue #8140: extend compileall to compile single files. Add -i option.
........
  r78985 | matthias.klose | 2010-03-15 19:00:01 +0100 (Mo, 15 Mär 2010) | 2 lines

  - Fix typo in Lib/compileall.py(__all__).
........
2010-03-16 00:36:26 +00:00
Mark Dickinson 769ba47bb2 Merged revisions 71750 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r71750 | mark.dickinson | 2009-04-19 18:10:47 +0100 (Sun, 19 Apr 2009) | 3 lines

  Automatic conversion of floats to integers for struct.pack integer codes
  is deprecated.  Use an explicit int() instead.
........
2009-04-19 17:14:11 +00:00
Brett Cannon befb14fe27 Merged revisions 69481 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r69481 | brett.cannon | 2009-02-09 18:07:38 -0800 (Mon, 09 Feb 2009) | 4 lines

  compileall used the ctime of bytecode and source to determine if the bytecode
  should be recreated. This created a timing hole. Fixed by just doing what
  import does; check the mtime and magic number.
........
2009-02-10 02:10:16 +00:00
Christian Heimes 7864476afa Merged revisions 61209-61214,61217-61222,61224-61226,61233-61237 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r61209 | georg.brandl | 2008-03-03 21:37:55 +0100 (Mon, 03 Mar 2008) | 2 lines

  There are now sixteen isfoo functions.
........
  r61210 | georg.brandl | 2008-03-03 21:39:00 +0100 (Mon, 03 Mar 2008) | 2 lines

  15 -> 16, the 2nd
........
  r61211 | georg.brandl | 2008-03-03 22:22:47 +0100 (Mon, 03 Mar 2008) | 2 lines

  Actually import itertools.
........
  r61212 | georg.brandl | 2008-03-03 22:31:50 +0100 (Mon, 03 Mar 2008) | 2 lines

  Expand a bit on genexp scopes.
........
  r61213 | raymond.hettinger | 2008-03-03 23:04:55 +0100 (Mon, 03 Mar 2008) | 1 line

  Remove dependency on itertools -- a simple genexp suffices.
........
  r61214 | raymond.hettinger | 2008-03-03 23:19:58 +0100 (Mon, 03 Mar 2008) | 1 line

  Issue 2226: Callable checked for the wrong abstract method.
........
  r61217 | andrew.kuchling | 2008-03-04 01:40:32 +0100 (Tue, 04 Mar 2008) | 1 line

  Typo fix
........
  r61218 | andrew.kuchling | 2008-03-04 02:30:10 +0100 (Tue, 04 Mar 2008) | 1 line

  Grammar fix; markup fix
........
  r61219 | andrew.kuchling | 2008-03-04 02:47:38 +0100 (Tue, 04 Mar 2008) | 1 line

  Fix sentence fragment
........
  r61220 | andrew.kuchling | 2008-03-04 02:48:26 +0100 (Tue, 04 Mar 2008) | 1 line

  Typo fix
........
  r61221 | andrew.kuchling | 2008-03-04 02:49:37 +0100 (Tue, 04 Mar 2008) | 1 line

  Add versionadded tags
........
  r61222 | andrew.kuchling | 2008-03-04 02:50:32 +0100 (Tue, 04 Mar 2008) | 1 line

  Thesis night results: add various items
........
  r61224 | raymond.hettinger | 2008-03-04 05:17:08 +0100 (Tue, 04 Mar 2008) | 1 line

  Beef-up docs and tests for itertools.  Fix-up end-case for product().
........
  r61225 | georg.brandl | 2008-03-04 08:25:54 +0100 (Tue, 04 Mar 2008) | 2 lines

  Fix some patch attributions.
........
  r61226 | georg.brandl | 2008-03-04 08:33:30 +0100 (Tue, 04 Mar 2008) | 2 lines

  #2230: document that PyArg_* leaves addresses alone on error.
........
  r61233 | neal.norwitz | 2008-03-04 17:22:46 +0100 (Tue, 04 Mar 2008) | 3 lines

  Close the file before trying to remove the directory so it works on Windows.
  As reported by Trent Nelson on python-dev.
........
  r61234 | thomas.heller | 2008-03-04 21:09:11 +0100 (Tue, 04 Mar 2008) | 9 lines

  Merged changes from libffi3-branch.

  The bundled libffi copy is now in sync with the recently released
  libffi3.0.4 version, apart from some small changes to
  Modules/_ctypes/libffi/configure.ac.

  I gave up on using libffi3 files on os x.
  Instead, static configuration with files from pyobjc is used.
........
  r61235 | thomas.heller | 2008-03-04 21:21:42 +0100 (Tue, 04 Mar 2008) | 1 line

  Try to fix the build for PY_LINUX.
........
  r61236 | fred.drake | 2008-03-04 22:14:04 +0100 (Tue, 04 Mar 2008) | 2 lines

  fix typo
........
  r61237 | raymond.hettinger | 2008-03-04 23:29:44 +0100 (Tue, 04 Mar 2008) | 1 line

  Fix refleak in chain().
........
2008-03-04 23:39:23 +00:00
Guido van Rossum b7d3e65b3b Improve error handling; don't die from unicode errors or syntax errors. 2007-07-15 13:01:48 +00:00