Commit Graph

68 Commits

Author SHA1 Message Date
Victor Stinner cc057ff522
bpo-45205: Make test_compileall quiet (GH-28356)
Make test_compileall quiet: test_year_2038_mtime_compilation() and
test_larger_than_32_bit_times() of test_compileall no longer log
"Compiling ..." messages to stdout.
2021-09-15 20:20:45 +02:00
Ammar Askar bb21e28fd0
bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int (GH-19708)
Co-authored-by: Stéphane Wirtel <stephane@wirtel.be>
2021-08-24 12:13:32 +03:00
Stefan Hoelzl 80f0707629
bpo-44666: Use default encoding as fallback for compile_file (GH-27236)
When sys.stdout.encoding is None compile_file will fall back to
sys.getdefaultencoding to encode/decode error messages.

Co-authored-by: Stefan Hoelzl <stefan.hoelzl@posteo.de>
Co-authored-by: Mickaël Schoentgen <contact@tiger-222.fr>
2021-07-30 18:38:42 +02:00
Inada Naoki 80017752ba
bpo-43651: Fix test_compileall with PEP 597 (GH-25128) 2021-04-02 09:01:57 +09:00
Asheesh Laroia bf2e7e55d7
bpo-40692: Run more test_concurrent_futures tests (GH-20239)
In the case of multiprocessing.synchronize() being missing, the
test_concurrent_futures test suite now skips only the tests that
require multiprocessing.synchronize().

Validate that multiprocessing.synchronize exists as part of
_check_system_limits(), allowing ProcessPoolExecutor to raise
NotImplementedError during __init__, rather than crashing with
ImportError during __init__ when creating a lock imported from
multiprocessing.synchronize.

Use _check_system_limits() to disable tests of
ProcessPoolExecutor on systems without multiprocessing.synchronize.

Running the test suite without multiprocessing.synchronize reveals
that Lib/compileall.py crashes when it uses a ProcessPoolExecutor.
Therefore, change Lib/compileall.py to call _check_system_limits()
before creating the ProcessPoolExecutor.

Note that both Lib/compileall.py and Lib/test/test_compileall.py
were attempting to sanity-check ProcessPoolExecutor by expecting
ImportError. In multiprocessing.resource_tracker, sem_unlink() is also absent
on platforms where POSIX semaphores aren't available. Avoid using
sem_unlink() if it, too, does not exist.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2021-02-08 03:15:51 +00:00
Hai Shi 883bc63833
bpo-40275: Use new test.support helper submodules in tests (GH-21314) 2020-07-06 11:12:49 +02:00
Serhiy Storchaka 700cfa8c90
bpo-41069: Make TESTFN and the CWD for tests containing non-ascii characters. (GH-21035) 2020-06-25 17:56:31 +03:00
Lumír 'Frenzy' Balhar e77d428856
bpo-40495: compileall option to hardlink duplicate pyc files (GH-19901)
compileall is now able to use hardlinks to prevent duplicates in a
case when .pyc files for different optimization levels have the same content.

Co-authored-by: Miro Hrončok <miro@hroncok.cz>
Co-authored-by: Victor Stinner <vstinner@python.org>
2020-05-14 16:17:22 +02:00
Victor Stinner b1e11c31c5
bpo-40443: Remove unused imports in tests (GH-19804) 2020-04-30 02:21:30 +02:00
Gregory P. Smith 02673352b5
bpo-39769: Fix compileall ddir for subpkgs. (GH-18676)
Fix compileall.compile_dir() ddir= behavior on sub-packages.

Fixes compileall.compile_dir's ddir parameter and compileall command
line flag `-d` to no longer write the wrong pathname to the generated
pyc file for submodules beneath the root of the directory tree being
compiled.  This fixes a regression introduced with Python 3.5.

Also marks the _new_ in 3.9 from PR #16012 parameters to compile_dir as keyword only (as that is the only way they will be used) and fixes an omission of them in one place from the docs.
2020-02-28 17:28:37 -08:00
Victor Stinner eb1dda2b56 bpo-38470: Fix test_compileall.test_compile_dir_maxlevels() (GH-16789)
Fix test_compile_dir_maxlevels() on Windows without long path
support: only create 3 subdirectories instead of between 20 and 100
subdirectories.

Fix also compile_dir() to use the current sys.getrecursionlimit()
value as the default maxlevels value, rather than using
sys.getrecursionlimit() value read at startup.
2019-10-15 11:26:13 +02:00
Petr Viktorin 3d984a1fd0
compileall tests: Use shorter name for long_path test (GH-16419)
Apparently, the path needs to be limited to 260 characters on
(some versions of) Windows.
2019-09-26 15:16:32 +02:00
Petr Viktorin 4267c989e7
bpo-38112: compileall: Skip long path path on Windows if the path can't be created (GH-16414)
This avoids the buildbot failure on Windows:
```
FileNotFoundError: [WinError 206] The filename or extension is too long: 'd:\\temp\\tmp5r3z438t\\long\\1\\2\\3\\4\\5\\6\\7\\8\\9\\10\\11\\12\\13\\14\\15\\16\\17\\18\\19\\20\\21\\22\\23\\24\\25\\26\\27\\28\\29\\30\\31\\32\\33\\34\\35\\36\\37\\38\\39\\40\\41\\42\\43\\44\\45\\46\\47\\48\\49\\50\\51\\52\\53\\54\\55\\56\\57\\58\\59\\60\\61\\62\\63\\64\\65\\66\\67\\68\\69\\70\\71\\72\\73\\74\\75\\76\\77\\78'
```
Creates a path that's long but avoids OS restrictions.

https://bugs.python.org/issue38112
2019-09-26 11:53:51 +02:00
Lumír 'Frenzy' Balhar 8e7bb991de bpo-38112: Compileall improvements (GH-16012)
* Raise the limit of maximum path depth to actual  recursion limit

* Add posibilities to adjust a path compiled in .pyc  file.

Now, you can:
- Strip a part of path from a beggining of path into compiled file
   example "-s /test /test/build/real/test.py" → "build/real/test.py"
- Append some new path to a beggining of path into compiled file
   example "-p /boo real/test.py" → "/boo/real/test.py"

You can also use both options in the same time. In that case,
striping is done before appending.

* Add a possibility to specify multiple optimization levels

Each optimization level then leads to separated compiled file.
Use `action='append'` instead of `nargs='+'` for the -o option.
Instead of `-o 0 1 2`, specify `-o 0 -o 1 -o 2`. It's more to type,
but much more explicit.

* Add a symlinks limitation feature

This feature allows us to limit byte-compilation of symbolic
links if they are pointing outside specified dir (build root
for example).
2019-09-26 08:28:26 +02:00
Min ho Kim c4cacc8c5e Fix typos in comments, docs and test names (#15018)
* Fix typos in comments, docs and test names

* Update test_pyparse.py

account for change in string length

* Apply suggestion: splitable -> splittable

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Apply suggestion: splitable -> splittable

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Apply suggestion: Dealloccte -> Deallocate

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Update posixmodule checksum.

* Reverse idlelib changes.
2019-07-30 18:16:13 -04:00
Antoine Pitrou 1a2dd82f56
bpo-36786: Run compileall in parallel during "make install" (GH-13078) 2019-05-15 23:45:18 +02:00
Anthony Sottile 8377cd4fcd Clean up code which checked presence of os.{stat,lstat,chmod} (#11643) 2019-02-25 23:32:27 +01:00
Dustin Spicuzza 1d817e4c82 bpo-29877: compileall: import ProcessPoolExecutor only when needed (GH-4856)
Importing ProcessPoolExecutor may hang or cause an error when the import
accesses urandom on a low resource platform


https://bugs.python.org/issue29877
2018-11-23 09:06:55 -08:00
Elvis Pranskevichus a6b3ec5b6d bpo-34022: Stop forcing of hash-based invalidation with SOURCE_DATE_EPOCH (GH-9607)
Unconditional forcing of ``CHECKED_HASH`` invalidation was introduced in
3.7.0 in bpo-29708.  The change is bad, as it unconditionally overrides
*invalidation_mode*, even if it was passed as an explicit argument to
``py_compile.compile()`` or ``compileall``.  An environment variable
should *never* override an explicit argument to a library function.
That change leads to multiple test failures if the ``SOURCE_DATE_EPOCH``
environment variable is set.

This changes ``py_compile.compile()`` to only look at
``SOURCE_DATE_EPOCH`` if no explicit *invalidation_mode* was specified.
I also made various relevant tests run with explicit control over the
value of ``SOURCE_DATE_EPOCH``.

While looking at this, I noticed that ``zipimport`` does not work
with hash-based .pycs _at all_, though I left the fixes for
subsequent commits.
2018-10-10 18:43:14 +02:00
Benjamin Peterson 42aa93b8ff
closes bpo-31650: PEP 552 (Deterministic pycs) implementation (#4575)
Python now supports checking bytecode cache up-to-dateness with a hash of the
source contents rather than volatile source metadata. See the PEP for details.

While a fairly straightforward idea, quite a lot of code had to be modified due
to the pervasiveness of pyc implementation details in the codebase. Changes in
this commit include:

- The core changes to importlib to understand how to read, validate, and
  regenerate hash-based pycs.

- Support for generating hash-based pycs in py_compile and compileall.

- Modifications to our siphash implementation to support passing a custom
  key. We then expose it to importlib through _imp.

- Updates to all places in the interpreter, standard library, and tests that
  manually generate or parse pyc files to grok the new format.

- Support in the interpreter command line code for long options like
  --check-hash-based-pycs.

- Tests and documentation for all of the above.
2017-12-09 10:26:52 -08:00
Berker Peksag d8e9713cf1 Issue #28226: Fix test_compileall on Windows 2016-10-01 02:44:37 +03:00
Berker Peksag 812a2b65af Issue #28226: compileall now supports pathlib 2016-10-01 00:54:18 +03:00
Berker Peksag 408b78c06f Issue #27322: Set sys.path to a temp dir in test_compile_path 2016-09-28 17:38:53 +03:00
Christian Heimes 6d6602ced2 Issue #27322: back out the commit. needs to be addressed after beta1. 2016-09-12 15:22:25 +02:00
Christian Heimes 139652100e Issue #27322: skip test_compile_path when sys.path is not writeable. 2016-09-12 15:08:32 +02:00
Victor Stinner 9def284387 subprocess._optim_args_from_interpreter_flags()
Issue #26100:

* Add subprocess._optim_args_from_interpreter_flags()
* Add test.support.optim_args_from_interpreter_flags()
* Use new functions in distutils, test_cmd_line_script, test_compileall and
  test_inspect

The change enables test_details() test of test_inspect when -O or -OO command
line option is used.
2016-01-18 12:15:08 +01:00
Victor Stinner c437d0cb4e Fix test_compilepath() of test_compileall
Issue #26101: Exclude Lib/test/ from sys.path in test_compilepath(). The
directory contains invalid Python files like Lib/test/badsyntax_pep3120.py,
whereas the test ensures that all files can be compiled.
2016-01-18 11:25:50 +01:00
Brett Cannon 1e3c3e906c Issue #25768: Make compileall functions return booleans and document
the return values as well as test them.

Thanks to Nicholas Chammas for the bug report and initial patch.
2015-12-27 13:17:04 -08:00
R David Murray 939614c48c Merge: #24903: Remove misleading error message to fix regression. 2015-12-04 23:04:37 -05:00
R David Murray ced699b4c7 #24903: Remove misleading error message to fix regression.
Before the argparse conversion, compileall would (sometimes) accept multiple
paths when -d was specified.  Afterward, it does not.  The corresponding check
in the original code claimed to prevent multiple *directories* from being
specified...but it didn't really work even to do that.  So this patch fixes
the regression by invoking the consenting adults rule: if you specify a
combination of arguments to compileall that produces files with inconsistent
destdirs (which you could do before), it is on you.

Patch by Jake Garver.
2015-12-04 22:54:38 -05:00
Brett Cannon 89065d9fc7 Issue #25099: Skip relevant tests in test_compileall when an entry on
sys.path has an unwritable __pycache__ directory.

This typically comes up when someone runs the test suite from an
administrative install of Python on Windows where the user does not
have write permissions to the stdlib's directory.

Thanks to Zachary Ware and Matthias Klose for reporting bugs related
to this issue.
(grafted from 34bbd537b3e688dfbb6498e9083445a6a72fc4b1)
2015-10-09 15:09:43 -07:00
Brett Cannon 65ed750375 Issue #25099: Skip relevant tests in test_compileall when an entry on
sys.path has an unwritable __pycache__ directory.

This typically comes up when someone runs the test suite from an
administrative install of Python on Windows where the user does not
have write permissions to the stdlib's directory.

Thanks to Zachary Ware and Matthias Klose for reporting bugs related
to this issue.
2015-10-09 15:09:43 -07:00
Berker Peksag ce643913a9 Issue #9517: Move script_helper to the support package.
Patch by Christie Wilson.
2015-05-06 06:33:17 +03:00
Berker Peksag d86ef05a02 Issue #23917: Fall back to sequential compilation when ProcessPoolExecutor doesn't exist.
Patch by Claudiu Popa.
2015-04-22 09:39:19 +03:00
Brett Cannon f299abdafa Issue #23731: Implement PEP 488.
The concept of .pyo files no longer exists. Now .pyc files have an
optional `opt-` tag which specifies if any extra optimizations beyond
the peepholer were applied.
2015-04-13 14:21:02 -04:00
Berker Peksag 6554b86b1f Issue #21338: Add silent mode for compileall.
quiet parameters of compile_{dir, file, path} functions now have
a multilevel value.

Also, -q option of the CLI now have a multilevel value.

Patch by Thomas Kluyver.
2014-10-15 11:10:57 +03:00
Brett Cannon f1a8df0ac9 Issue #16104: Allow compileall to do parallel bytecode compilation.
Both compileall.compile_dir() and the CLI for compileall now allow for
specifying how many workers to use (or 0 to use all CPUs).

Thanks to Claudiu Popa for the patch.
2014-09-12 10:39:48 -04:00
Benjamin Peterson 344ff4ab2b allow recursion depth to be specified (closes #19628)
Patch from Claudiu Popa.
2014-08-19 16:13:26 -05:00
R David Murray 755d5ea12a Merge: #19532: make compileall with no file/dir args respect -f and -q. 2013-12-15 20:56:00 -05:00
R David Murray 8a1d1e647e #19532: make compileall with no file/dir args respect -f and -q.
Patch by Vajrasky Kok.
2013-12-15 20:49:38 -05:00
Serhiy Storchaka 43767638a9 Issue #18702: All skipped tests now reported as skipped. 2013-11-03 21:31:38 +02:00
Serhiy Storchaka 7908068627 Issue #18702: All skipped tests now reported as skipped. 2013-11-03 21:31:18 +02:00
Victor Stinner e8785ff82a Close #18754: Run Python child processes in isolated more in the test suite. 2013-10-12 14:44:01 +02:00
Brett Cannon 7822e123c4 Issue #17177: stop using imp for compileall. 2013-06-14 23:04:02 -04:00
Benjamin Peterson a820c7ca70 fix test_compileall when run with -O[O] 2012-09-25 11:42:35 -04:00
R David Murray ee1a7cb4a4 #11873: another try at fixing the regex, courtesy of Victor Stinner 2011-07-01 14:55:43 -04:00
R David Murray a3bb2f5514 #11873: fix test regex so it covers windows os.sep as well. 2011-07-01 11:51:50 -04:00
Victor Stinner 530712625b Issue #11169: compileall module uses repr() to format filenames and paths to
escape surrogate characters and show spaces.
2011-05-11 00:36:28 +02:00
R David Murray 66a18fd801 #11873: Improve test regex so random directory names don't cause test to fail 2011-05-05 11:52:44 -04:00
Georg Brandl 4543846517 #11132: pass optimize parameter to recursive call in compileall.compile_dir(). Reviewed by Eric A. 2011-02-07 12:36:54 +00:00