In debug mode, PyObject_GC_Track() now calls tp_traverse() of the
object type to ensure that the object is valid: test that objects
visited by tp_traverse() are valid.
Fix pyexpat.c: only track the parser in the GC once the parser is
fully initialized.
Prior to 3.7, re.escape escaped many characters that don't have
special meaning in Python, but that use to require escaping in other
tools and languages. This commit aims to make it clear which characters
were, but are no longer escaped.
bpo-36389, bpo-38376: The _PyObject_CheckConsistency() function is
now also available in release mode. For example, it can be used to
debug a crash in the visit_decref() function of the GC.
Modify the following functions to also work in release mode:
* _PyDict_CheckConsistency()
* _PyObject_CheckConsistency()
* _PyType_CheckConsistency()
* _PyUnicode_CheckConsistency()
Other changes:
* _PyMem_IsPtrFreed(ptr) now also returns 1 if ptr is NULL
(equals to 0).
* _PyBytesWriter_CheckConsistency() now returns 1 and is only used
with assert().
* Reorder _PyObject_Dump() to write safe fields first, and only
attempt to render repr() at the end.
On Windows use UTF-16 (or UTF-32 for 32-bit Tcl_UniChar) with the
"surrogatepass" error handler for converting to/from Tcl Unicode objects.
On Linux use UTF-8 with the "surrogateescape" error handler for converting
to/from Tcl String objects.
Converting strings from Tcl to Python and back now never fails
(except MemoryError).
On FreeBSD, Python no longer calls fedisableexcept() at startup to
control the floating point control mode. The call became useless
since FreeBSD 6: it became the default mode.
For now, we'll rely on the fact that the config structures aren't covered by the stable ABI.
We may revisit this in the future if we further explore the idea of offering a stable embedding API.
(cherry picked from commit bdace21b76)
Important work originally done by @emilyemorehouse two years ago and nearly ready to go in.
This bug has affected many people and in some cases has been a dealbreaker to the adoption of the otherwise wonderful pathlib and PEP519. https://stackoverflow.com/questions/33625931/copy-file-with-pathlib-in-python.
This adds the outstanding test request from that PR @vstinner (https://github.com/python/cpython/pull/5393).
Test fails without the change, passes with it, along with every other test in test_shutil.
Some variants were experimented with to make the one line change and the most performant one was picked.
# Added Test for PathLike directory destination, the current fail case
```
Lib/test/test_shutil.py::TestMove::test_move_file_pathlike FAILED [100%]
============================================================== FAILURES ===============================================================
__________________________________________________ TestMove.test_move_file_pathlike ___________________________________________________
self = <test.test_shutil.TestMove testMethod=test_move_file_pathlike>
def test_move_file_pathlike(self):
# Move a file to another location on the same filesystem.
src = pathlib.Path(self.src_file)
> self._check_move_file(src, self.dst_dir, self.dst_file)
Lib/test/test_shutil.py:1563:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Lib/test/test_shutil.py:1545: in _check_move_file
shutil.move(src, dst)
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py:562: in move
real_dst = os.path.join(dst, _basename(src))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
path = PosixPath('/var/folders/r2/psq74t5x3nbfzlph8bh2pvdw0000gn/T/tmp9ie0wh9_/foo')
def _basename(path):
# A basename() variant which first strips the trailing slash, if present.
# Thus we always get the last component of the path, even for directories.
sep = os.path.sep + (os.path.altsep or '')
> return os.path.basename(path.rstrip(sep))
E AttributeError: 'PosixPath' object has no attribute 'rstrip'
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/shutil.py:526: AttributeError
============================================== 1 failed, 102 deselected in 0.30 seconds ===============================================
```
After change:
```
========================================================= test session starts =========================================================
platform darwin -- Python 3.7.4, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /Users/maxwellmckinnon/.venvs/TA3.7/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/maxwellmckinnon/dev/cpython
plugins: cov-2.7.1, mock-1.10.4
collected 103 items / 102 deselected / 1 selected
Lib/test/test_shutil.py::TestMove::test_move_file_pathlike PASSED [100%]
============================================== 1 passed, 102 deselected in 0.06 seconds ===============================================
```
Running all the tests in test_shutil.py
```
╰─ pytest Lib/test/test_shutil.py -v
========================================================= test session starts =========================================================
platform darwin -- Python 3.7.4, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /Users/maxwellmckinnon/.venvs/TA3.7/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/maxwellmckinnon/dev/cpython
plugins: cov-2.7.1, mock-1.10.4
collected 103 items
Lib/test/test_shutil.py::TestShutil::test_chown PASSED [ 0%]
Lib/test/test_shutil.py::TestShutil::test_copy PASSED [ 1%]
...
Lib/test/test_shutil.py::TermsizeTests::test_stty_match SKIPPED [ 99%]
Lib/test/test_shutil.py::PublicAPITests::test_module_all_attribute PASSED [100%]
================================================ 96 passed, 7 skipped in 1.25 seconds =================================================
```
# Performance Considerations
Is it considered poor form to get rid of _basename altogether and make use of pathlib in the move function? I'm not sure if the idea is for all these modules to strictly avoid circular dependencies. They are already using os.path which is just as much a citizen in 3.8 as pathlib right?
e.g.
`real_dst = os.path.join(dst, _basename(src))`
becomes
`real_dst = Path(dst) / Path(src).name`
I've looked around and familiarized myself, and I now think importing pathlib here is fine. My only remaining concern is that of performance.
Here's the performance difference for this step.
```
In [46]: %timeit real_dst = os.path.join("a/b/c", _basename('b/'))
2.71 µs ± 62.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In [47]: %timeit real_dst = Path("a/b/c") / Path('b/').name
12.4 µs ± 65.3 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
```
Is 10us significant or insignificant compared to the least expensive operation this function will do? I don't know. Let's find out.
```
In [55]: %timeit os.rename('/tmp/a/a.txt', '/tmp/a/b.txt'); os.rename('/tmp/a/b.txt', '/tmp/a/a.txt')
124 µs ± 2.18 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
```
62us to rename. 10us seems significant enough that we wouldn't want to favor the Path sugar suggestion. 16% speed decrease from adding the 10us.
What do people think? I was hoping to get to use pathlib.Path here, but I suspect for this low level move, it should be as fast as possible, and 16% is not worth one line of sugary code to me.
https://bugs.python.org/issue32689
Automerge-Triggered-By: @gvanrossum
Fix a bug due to the interaction of weakrefs and the cyclic garbage
collector. We must clear any weakrefs in garbage in order to prevent
their callbacks from executing and causing a crash.
Fix warnings options priority: PyConfig.warnoptions has the highest
priority, as stated in the PEP 587.
* Document options order in PyConfig.warnoptions documentation.
* Make PyWideStringList_INIT macro private: replace "Py" prefix
with "_Py".
* test_embed: add test_init_warnoptions().
Document that lnotab can contain invalid bytecode offsets (because of
terrible reasons that are difficult to fix). Make dis.findlinestarts()
ignore invalid offsets in lnotab. All other uses of lnotab in CPython
(various reimplementations of addr2line or line2addr in Python, C and gdb)
already ignore this, because they take an address to look for, instead.
Add tests for the result of dis.findlinestarts() on wacky constructs in
test_peepholer.py, because it's the easiest place to add them.
* bpo-38216: Allow bypassing input validation
* bpo-36274: Also allow the URL encoding to be overridden.
* bpo-38216, bpo-36274: Add tests demonstrating a hook for overriding validation, test demonstrating override encoding, and a test to capture expectation of the interface for the URL.
* Call with skip_host to avoid tripping on the host checking in the URL.
* Remove obsolete comment.
* Make _prepare_path_encoding its own attr.
This makes overriding just that simpler.
Also, don't use the := operator to make backporting easier.
* Add a news entry.
* _prepare_path_encoding -> _encode_prepared_path()
* Once again separate the path validation and request encoding, drastically simplifying the behavior. Drop the guarantee that all processing happens in _prepare_path.
Add a new struct_size field to PyPreConfig and PyConfig structures to
allow to modify these structures in the future without breaking the
backward compatibility.
* Replace private _config_version field with public struct_size field
in PyPreConfig and PyConfig.
* Public PyPreConfig_InitIsolatedConfig() and
PyPreConfig_InitPythonConfig()
return type becomes PyStatus, instead of void.
* Internal _PyConfig_InitCompatConfig(),
_PyPreConfig_InitCompatConfig(), _PyPreConfig_InitFromConfig(),
_PyPreConfig_InitFromPreConfig() return type becomes PyStatus,
instead of void.
* Remove _Py_CONFIG_VERSION
* Update the Initialization Configuration documentation.
test_ssl now handles disabled TLS/SSL versions better. OpenSSL's crypto
policy and run-time settings are recognized and tests for disabled versions
are skipped.
Signed-off-by: Christian Heimes <christian@python.org>
https://bugs.python.org/issue38275
* Raise the limit of maximum path depth to actual recursion limit
* Add posibilities to adjust a path compiled in .pyc file.
Now, you can:
- Strip a part of path from a beggining of path into compiled file
example "-s /test /test/build/real/test.py" → "build/real/test.py"
- Append some new path to a beggining of path into compiled file
example "-p /boo real/test.py" → "/boo/real/test.py"
You can also use both options in the same time. In that case,
striping is done before appending.
* Add a possibility to specify multiple optimization levels
Each optimization level then leads to separated compiled file.
Use `action='append'` instead of `nargs='+'` for the -o option.
Instead of `-o 0 1 2`, specify `-o 0 -o 1 -o 2`. It's more to type,
but much more explicit.
* Add a symlinks limitation feature
This feature allows us to limit byte-compilation of symbolic
links if they are pointing outside specified dir (build root
for example).
* Updated _hashopenssl.c to be PEP 384 compliant
* Remove refleak test from test_hashlib. The updated type no longer accepts random arguments to __init__.
The private keys for test_ssl were encrypted with 3DES in traditional
PKCS#5 format. 3DES and the digest algorithm of PKCS#5 are blocked by
some strict crypto policies. Use PKCS#8 format with AES256 encryption
instead.
Signed-off-by: Christian Heimes <christian@python.org>
https://bugs.python.org/issue38271
Automerge-Triggered-By: @tiran
Make it easier to run and test Python on systems with restrict crypto policies:
* add requires_hashdigest to test.support to check if a hash digest algorithm is available and working
* avoid MD5 in test_hmac
* replace MD5 with SHA256 in test_tarfile
* mark network tests that require MD5 for MD5-based digest auth or CRAM-MD5
https://bugs.python.org/issue38270
The fix in PR 13261 handled the underlying issue about the spec for specific methods not being applied correctly, but it didn't fix the issue that was causing the misleading error message.
The code currently grabs a list of responses from _call_matcher (which may include exceptions). But it doesn't reach inside the list when checking if the result is an exception. This results in a misleading error message when one of the provided calls does not match the spec.
https://bugs.python.org/issue36871
Automerge-Triggered-By: @gpshead
Py_SetPath() now sets sys.executable to the program full path
(Py_GetProgramFullPath()), rather than to the program name
(Py_GetProgramName()).
Fix also memory leaks in pathconfig_set_from_config().
Multiprocessing test test_mymanager() now also expects -SIGTERM, not
only exitcode 0.
bpo-30356: BaseManager._finalize_manager() sends SIGTERM to the
manager process if it takes longer than 1 second to stop, which
happens on slow buildbots.
* _PyConfig_InitPathConfig() now starts by copying the global path
configuration, and then override values set in PyConfig.
* _PyPathConfig_Calculate() implementations no longer override
_PyPathConfig fields which are already computed. For example,
if _PyPathConfig.prefix is not NULL, leave it unchanged.
* If Py_SetPath() has been called, _PyConfig_InitPathConfig() doesn't
call _PyPathConfig_Calculate() anymore.
* _PyPathConfig_Calculate() no longer uses PyConfig,
except to initialize PyCalculatePath structure.
* pathconfig_calculate(): remove useless temporary
"_PyPathConfig new_config" variable.
* calculate_module_search_path(): remove hack to workaround memory
allocation failure, call Py_FatalError() instead.
* Fix get_program_full_path(): handle memory allocation failure.
* If Py_SetPath() has been called, _PyConfig_InitPathConfig() now
uses its value.
* Py_Initialize() now longer copies path configuration from PyConfig
to the global path configuration (_Py_path_config).
Mention frame.f_trace in sys.settrace docs, as well as the fact you still
need to call `sys.settrace` to enable the tracing machinery before setting
`frame.f_trace` will have any effect.
* Check intersection of two sets explicitly
Comparing ``len(a) > ``len(a - b)`` is essentially looking for an
intersection between the two sets. If set ``b`` does not intersect ``a``
then ``len(a - b)`` will be equal to ``len(a)``. This logic is more
clearly expressed as ``a & b``.
* Change while/pop to a for-loop
Copying the list, then repeatedly popping the first element was
unnecessarily slow. I also cleaned up a couple other inefficiencies.
There's no need to unpack a tuple, then re-pack and append it. The list
can be created with the first element instead of empty. Secondly, the
``endswith`` method returns a bool, so there's no need for an if-
statement to set ``encoding`` to True or False.
* Use set.intersection to check for intersections
``a.intersection(b)`` method is more clear of purpose than ``not
a.isdisjoint(b)`` and avoids an unnecessary set construction that ``a &
set(b)`` performs.
* Use not isdisjoint instead of intersection
While it reads slightly worse, the isdisjoint method will stop when it
finds a counterexample and returns a bool, rather than looping over the
entire iterable and constructing a new set.
Even when the helper is not started yet.
This behavior follows conventional generator one.
There is no reason for `async_generator_athrow` to handle `gen.throw()` differently.
https://bugs.python.org/issue38013
In the format string for assert_called the evaluation order is incorrect and hence for mock's without name, 'None' is printed whereas it should be 'mock' like for other messages. The error message is ("Expected '%s' to have been called." % self._mock_name or 'mock').
* Convert select module to PEP-384
Summary: Do the necessary versions to be Pyro-compatible, including migrating `PyType_Ready` to `PyType_FromSpec` and moving static data into a new `_selectstate` struct.
* 📜🤖 Added by blurb_it.
* Fixup Mac OS/X build
In ArgumentClinic, value "NULL" should now be used only for unrepresentable default values
(like in the optional third parameter of getattr). "None" should be used if None is accepted
as argument and passing None has the same effect as not passing the argument at all.
* Fix a crash in comparing with float (and maybe other crashes).
* They are now never equal to strings and non-integer numbers.
* Comparison with a large number no longer raises OverflowError.
* Arbitrary exceptions no longer silenced in constructors and comparisons.
* TypeError raised in the constructor contains now the name of the type.
* Accept only ChannelID and int-like objects in channel functions.
* Accept only InterpreterId, int-like objects and str in the InterpreterId constructor.
* Accept int-like objects, not just int in interpreter related functions.
https://bugs.python.org/issue34706
Specifically in the case of a class that does not override its
constructor signature inherited from object.
These are Buck Evan @bukzor's changes cherrypicked from GH-9344.
* Flip equality to use mock calls' __eq__
* bpo-37555: Regression test demonstrating assert_has_calls not working with ANY and spec_set
Co-authored-by: Neal Finne <neal@nealfinne.com>
* Revert "Flip equality to use mock calls' __eq__"
This reverts commit 94ddf54c5a.
* bpo-37555: Add regression tests for mock ANY ordering issues
Add regression tests for whether __eq__ is order agnostic on _Call and _CallList, which is useful for comparisons involving ANY, especially if the ANY comparison is to a class not defaulting __eq__ to NotImplemented.
Co-authored-by: Neal Finne <neal@nealfinne.com>
* bpo-37555: Fix _CallList and _Call order sensitivity
_Call and _CallList depend on ordering to correctly process that an object being compared to ANY with __eq__ should return True. This fix updates the comparison to check both a == b and b == a and return True if either condition is met, fixing situations from the tests in the previous two commits where assertEqual would not be commutative if checking _Call or _CallList objects. This seems like a reasonable fix considering that the Python data model specifies that if an object doesn't know how to compare itself to another object it should return NotImplemented, and that on getting NotImplemented from a == b, it should try b == a, implying that good behavior for __eq__ is commutative. This also flips the order of comparison in _CallList's __contains__ method, guaranteeing ANY will be on the left and have it's __eq__ called for equality checking, fixing the interaction between assert_has_calls and ANY.
Co-author: Neal Finne <neal@neal.finne.com>
* bpo-37555: Ensure _call_matcher returns _Call object
* Adding ACK and news entry
* bpo-37555: Replacing __eq__ with == to sidestep NotImplemented
bool(NotImplemented) returns True, so it's necessary to use ==
instead of __eq__ in this comparison.
* bpo-37555: cleaning up changes unnecessary to the final product
* bpo-37555: Fixed call on bound arguments to respect args and kwargs
* Revert "bpo-37555: Add regression tests for mock ANY ordering issues"
This reverts commit 49c5310ad4.
* Revert "bpo-37555: cleaning up changes unnecessary to the final product"
This reverts commit 18e964ba01.
* Revert "bpo-37555: Replacing __eq__ with == to sidestep NotImplemented"
This reverts commit f295eaca5b.
* Revert "bpo-37555: Fix _CallList and _Call order sensitivity"
This reverts commit 874fb697b8.
* Updated NEWS.d
* bpo-37555: Add tests checking every function using _call_matcher both with and without spec
* bpo-37555: Ensure all assert methods using _call_matcher are actually passing calls
* Remove AnyCompare and use call objects everywhere.
* Revert "Remove AnyCompare and use call objects everywhere."
This reverts commit 24973c0b32.
* Check for exception in assert_any_await
Handle time comparison for cookies with `expires` attribute when `CookieJar.make_cookies` is called.
Co-authored-by: Demian Brecht <demianbrecht@gmail.com>
https://bugs.python.org/issue12144
Automerge-Triggered-By: @asvetlov
- Migrate `Random_Type` to `PyType_FromSpec`
- To simulate an old use of `PyLong_Type.tp_as_number->nb_absolute`, I added
code to the module init function to stash `int.__abs__` for later
use. Ideally we'd use `PyType_GetSlot()` instead, but it doesn't currently
work for static types in CPython, and implementing it just for this case
doesn't seem worth it.
- Do exact check for long and dispatch to PyNumber_Absolute, use vector call when not exact.
The usedforsecurity keyword only argument added to the hash constructors is useful for FIPS builds and similar restrictive environment with non-technical requirements that legacy algorithms be forbidden by their implementations without being explicitly annotated as not being used for any security related purposes. Linux distros with FIPS support benefit from this being standard rather than making up their own way(s) to do it.
Contributed and Signed-off-by: Christian Heimes christian@python.org
* subprocess: Add user, group and extra_groups paremeters to subprocess.Popen
This adds a `user` parameter to the Popen constructor that will call
setreuid() in the child before calling exec(). This allows processes
running as root to safely drop privileges before running the subprocess
without having to use a preexec_fn.
This also adds a `group` parameter that will call setregid() in
the child process before calling exec().
Finally an `extra_groups` parameter was added that will call
setgroups() to set the supplimental groups.
* 1. add test case with wrong behavior
* 2. fix bug when max_length == -1
* 3. allow b"" as valid input data for decompress_buf()
* 4. when max_length >= 0, let needs_input mechanism works
* add more asserts to test case
* Add a note to the PyModule_AddObject docs.
* Correct example usages of PyModule_AddObject.
* Whitespace.
* Clean up wording.
* 📜🤖 Added by blurb_it.
* First code review.
* Add < 0 in the tests with PyModule_AddObject
Antonio Gutierrez
Victor Stinner
James Abel
Ricardo Bánffy
Serhiy Storchaka
Dong-hee Na
Andrei Troie
nde
Joannah Nanjekye
Steve Dower
Victor Stinner
Giampaolo Rodola
Maxwell A McKinnon
Neil Schemenauer
Yury Selivanov
Yury Selivanov
Lisa Roach
Andrew Svetlov
Brandt Bucher
T. Wouters
Jason R. Coombs
Jesús Cea
Michael Felt
Christian Heimes
Lumír 'Frenzy' Balhar
Benjamin Peterson
PatrikKopkan
Samuel Freilich
Ammar Askar
Ram Rachum
Michael Selik
Eddie Elizondo
t k
Kyle Stanley
Abraham Toriz Cruz
Terry Jan Reedy
Ned Deily
Dino Viehland
plokmijnuhby
Gregory P. Smith
Elizabeth Uselton
Zackery Spytz
Doyle Rowland
Ashwin Ramaswami
Xtreak
Rémi Lapeyre
Dino Viehland
Jakub Kulík
Patrick McLean
Daniel Hahler
Sergey Fedoseev
animalize
Berker Peksag
Hai Shi
Divij Rajkumar
bggardner