Commit Graph

49 Commits

Author SHA1 Message Date
Benjamin Peterson 42aa93b8ff
closes bpo-31650: PEP 552 (Deterministic pycs) implementation (#4575)
Python now supports checking bytecode cache up-to-dateness with a hash of the
source contents rather than volatile source metadata. See the PEP for details.

While a fairly straightforward idea, quite a lot of code had to be modified due
to the pervasiveness of pyc implementation details in the codebase. Changes in
this commit include:

- The core changes to importlib to understand how to read, validate, and
  regenerate hash-based pycs.

- Support for generating hash-based pycs in py_compile and compileall.

- Modifications to our siphash implementation to support passing a custom
  key. We then expose it to importlib through _imp.

- Updates to all places in the interpreter, standard library, and tests that
  manually generate or parse pyc files to grok the new format.

- Support in the interpreter command line code for long options like
  --check-hash-based-pycs.

- Tests and documentation for all of the above.
2017-12-09 10:26:52 -08:00
Berker Peksag d8e9713cf1 Issue #28226: Fix test_compileall on Windows 2016-10-01 02:44:37 +03:00
Berker Peksag 812a2b65af Issue #28226: compileall now supports pathlib 2016-10-01 00:54:18 +03:00
Berker Peksag 408b78c06f Issue #27322: Set sys.path to a temp dir in test_compile_path 2016-09-28 17:38:53 +03:00
Christian Heimes 6d6602ced2 Issue #27322: back out the commit. needs to be addressed after beta1. 2016-09-12 15:22:25 +02:00
Christian Heimes 139652100e Issue #27322: skip test_compile_path when sys.path is not writeable. 2016-09-12 15:08:32 +02:00
Victor Stinner 9def284387 subprocess._optim_args_from_interpreter_flags()
Issue #26100:

* Add subprocess._optim_args_from_interpreter_flags()
* Add test.support.optim_args_from_interpreter_flags()
* Use new functions in distutils, test_cmd_line_script, test_compileall and
  test_inspect

The change enables test_details() test of test_inspect when -O or -OO command
line option is used.
2016-01-18 12:15:08 +01:00
Victor Stinner c437d0cb4e Fix test_compilepath() of test_compileall
Issue #26101: Exclude Lib/test/ from sys.path in test_compilepath(). The
directory contains invalid Python files like Lib/test/badsyntax_pep3120.py,
whereas the test ensures that all files can be compiled.
2016-01-18 11:25:50 +01:00
Brett Cannon 1e3c3e906c Issue #25768: Make compileall functions return booleans and document
the return values as well as test them.

Thanks to Nicholas Chammas for the bug report and initial patch.
2015-12-27 13:17:04 -08:00
R David Murray 939614c48c Merge: #24903: Remove misleading error message to fix regression. 2015-12-04 23:04:37 -05:00
R David Murray ced699b4c7 #24903: Remove misleading error message to fix regression.
Before the argparse conversion, compileall would (sometimes) accept multiple
paths when -d was specified.  Afterward, it does not.  The corresponding check
in the original code claimed to prevent multiple *directories* from being
specified...but it didn't really work even to do that.  So this patch fixes
the regression by invoking the consenting adults rule: if you specify a
combination of arguments to compileall that produces files with inconsistent
destdirs (which you could do before), it is on you.

Patch by Jake Garver.
2015-12-04 22:54:38 -05:00
Brett Cannon 89065d9fc7 Issue #25099: Skip relevant tests in test_compileall when an entry on
sys.path has an unwritable __pycache__ directory.

This typically comes up when someone runs the test suite from an
administrative install of Python on Windows where the user does not
have write permissions to the stdlib's directory.

Thanks to Zachary Ware and Matthias Klose for reporting bugs related
to this issue.
(grafted from 34bbd537b3e688dfbb6498e9083445a6a72fc4b1)
2015-10-09 15:09:43 -07:00
Brett Cannon 65ed750375 Issue #25099: Skip relevant tests in test_compileall when an entry on
sys.path has an unwritable __pycache__ directory.

This typically comes up when someone runs the test suite from an
administrative install of Python on Windows where the user does not
have write permissions to the stdlib's directory.

Thanks to Zachary Ware and Matthias Klose for reporting bugs related
to this issue.
2015-10-09 15:09:43 -07:00
Berker Peksag ce643913a9 Issue #9517: Move script_helper to the support package.
Patch by Christie Wilson.
2015-05-06 06:33:17 +03:00
Berker Peksag d86ef05a02 Issue #23917: Fall back to sequential compilation when ProcessPoolExecutor doesn't exist.
Patch by Claudiu Popa.
2015-04-22 09:39:19 +03:00
Brett Cannon f299abdafa Issue #23731: Implement PEP 488.
The concept of .pyo files no longer exists. Now .pyc files have an
optional `opt-` tag which specifies if any extra optimizations beyond
the peepholer were applied.
2015-04-13 14:21:02 -04:00
Berker Peksag 6554b86b1f Issue #21338: Add silent mode for compileall.
quiet parameters of compile_{dir, file, path} functions now have
a multilevel value.

Also, -q option of the CLI now have a multilevel value.

Patch by Thomas Kluyver.
2014-10-15 11:10:57 +03:00
Brett Cannon f1a8df0ac9 Issue #16104: Allow compileall to do parallel bytecode compilation.
Both compileall.compile_dir() and the CLI for compileall now allow for
specifying how many workers to use (or 0 to use all CPUs).

Thanks to Claudiu Popa for the patch.
2014-09-12 10:39:48 -04:00
Benjamin Peterson 344ff4ab2b allow recursion depth to be specified (closes #19628)
Patch from Claudiu Popa.
2014-08-19 16:13:26 -05:00
R David Murray 755d5ea12a Merge: #19532: make compileall with no file/dir args respect -f and -q. 2013-12-15 20:56:00 -05:00
R David Murray 8a1d1e647e #19532: make compileall with no file/dir args respect -f and -q.
Patch by Vajrasky Kok.
2013-12-15 20:49:38 -05:00
Serhiy Storchaka 43767638a9 Issue #18702: All skipped tests now reported as skipped. 2013-11-03 21:31:38 +02:00
Serhiy Storchaka 7908068627 Issue #18702: All skipped tests now reported as skipped. 2013-11-03 21:31:18 +02:00
Victor Stinner e8785ff82a Close #18754: Run Python child processes in isolated more in the test suite. 2013-10-12 14:44:01 +02:00
Brett Cannon 7822e123c4 Issue #17177: stop using imp for compileall. 2013-06-14 23:04:02 -04:00
Benjamin Peterson a820c7ca70 fix test_compileall when run with -O[O] 2012-09-25 11:42:35 -04:00
R David Murray ee1a7cb4a4 #11873: another try at fixing the regex, courtesy of Victor Stinner 2011-07-01 14:55:43 -04:00
R David Murray a3bb2f5514 #11873: fix test regex so it covers windows os.sep as well. 2011-07-01 11:51:50 -04:00
Victor Stinner 530712625b Issue #11169: compileall module uses repr() to format filenames and paths to
escape surrogate characters and show spaces.
2011-05-11 00:36:28 +02:00
R David Murray 66a18fd801 #11873: Improve test regex so random directory names don't cause test to fail 2011-05-05 11:52:44 -04:00
Georg Brandl 4543846517 #11132: pass optimize parameter to recursive call in compileall.compile_dir(). Reviewed by Eric A. 2011-02-07 12:36:54 +00:00
R. David Murray e0436bcb24 Make test_compileall more robust by using -S to keep sys.path minimized.
Try this again, hopefully the right way this time.

Arfrever Taifersar Arahesis reported that test_compileall failed during Gentoo
install because it was tyring to write .pyc files to a read-only system
directory during test_no_args_compiles_path.  Having the tests call python
with -S should eliminate the system directories from the path.
2010-12-21 18:24:33 +00:00
R. David Murray 92812028f2 Revert incorrect patch made at the wrong time. 2010-12-20 19:04:51 +00:00
R. David Murray bf9004483d Make test_compileall more robust by using -S to keep sys.path minimized.
Arfrever Taifersar Arahesis reported that test_compileall failed during Gentoo
install because it was tyring to write .pyc files to a read-only system
directory during test_no_args_compiles_path.  Having subprocess call python
with -S should eliminate the system directories from the path.
2010-12-20 18:08:59 +00:00
R. David Murray 5317e9cd8d #10719: restore messages generated on invalid compileall args
Before the introduction of filename arguments to compileall it gave semi useful
messages about not being able to 'list' names that weren't valid directories.
This fix restores that behavior.  In addition to the test for this case, the
patch also adds a test for the default behavior of compileall when no arguments
are provided, and fixes a bug in one of the previously added tests.
2010-12-16 19:08:51 +00:00
R. David Murray 95333e3aa9 More comprehensive compileall cli tests, and fixes. 2010-12-14 22:32:50 +00:00
Georg Brandl 8334fd9285 Add an "optimize" parameter to compile() to control the optimization level, and provide an interface to it in py_compile, compileall and PyZipFile. 2010-12-04 10:26:46 +00:00
Éric Araujo 31717e8a55 #10453 follow-up: Fix test_quiet on Windows, thanks to Stephan Krah. 2010-11-26 00:39:59 +00:00
Éric Araujo 5cb823d353 Fix one compileall test (#10453). Patch by Michele Orrù. 2010-11-22 02:42:43 +00:00
Éric Araujo a491cedaf4 Try to get more useful output from failing buildbot 2010-11-21 02:19:09 +00:00
R. David Murray 650f147298 #10453: compileall now uses argparse instead of getopt, so -h works.
Patch by Michele Orrù.
2010-11-20 21:18:51 +00:00
Georg Brandl 1463a3f833 #9964: fix running test_compileall under -O and -OO. 2010-10-14 07:42:27 +00:00
Barry Warsaw c8a99de751 Bug 8563 - compileall.compile_file() creates empty __pycache__ directories in
data directories where there is no source.

Fix by: Arfrever Frehtes Taifersar Arahesis (Arfrever)
Test by: Barry
2010-04-29 18:43:10 +00:00
Barry Warsaw c04317fdc4 Bug 8527 - multiple compileall calls produce cascading __pycache__ directories.
* Patch contributed by Arfrever Frehtes Taifersar Arahesis.
* Test added by Barry

Also, improve Makefile's deletion of __pycache__ directories so e.g. 'make
distclean' doesn't fail if no __pycache__ directories exist.
2010-04-26 15:59:03 +00:00
Barry Warsaw 28a691b7fd PEP 3147 2010-04-17 00:19:56 +00:00
Martin v. Löwis 4b00307425 Issue #6716/2: Backslash-replace error output in compilall. 2010-03-16 13:19:21 +00:00
Matthias Klose c33b902ebb Merged revisions 78983,78985 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r78983 | matthias.klose | 2010-03-15 18:44:12 +0100 (Mo, 15 Mär 2010) | 2 lines

  - Issue #8140: extend compileall to compile single files. Add -i option.
........
  r78985 | matthias.klose | 2010-03-15 19:00:01 +0100 (Mo, 15 Mär 2010) | 2 lines

  - Fix typo in Lib/compileall.py(__all__).
........
2010-03-16 00:36:26 +00:00
Georg Brandl 1b37e8728c Merged revisions 78093 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r78093 | georg.brandl | 2010-02-07 18:03:15 +0100 (So, 07 Feb 2010) | 1 line

  Remove unused imports in test modules.
........
2010-03-14 10:45:50 +00:00
Brett Cannon befb14fe27 Merged revisions 69481 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r69481 | brett.cannon | 2009-02-09 18:07:38 -0800 (Mon, 09 Feb 2009) | 4 lines

  compileall used the ctime of bytecode and source to determine if the bytecode
  should be recreated. This created a timing hole. Fixed by just doing what
  import does; check the mtime and magic number.
........
2009-02-10 02:10:16 +00:00