When compiling 3rd party C extensions, the linker flags used by the
compiler for the interpreter and the stdlib modules, will get
leaked into distutils. In order to avoid that, the PY_CORE_LDFLAGS
and PY_LDFLAGS_NODIST are introduced to keep those flags separated.
Fix a race condition in check_interrupted_write() of test_io:
create directly the thread with SIGALRM signal blocked,
rather than blocking the signal later from the thread. Previously, it
was possible that the thread gets the signal before the signal is
blocked.
Rename test.bisect module to test.bisect_cmd to avoid conflict with
bisect module when running directly a test like
"./python Lib/test/test_xmlrpc.py".
TextTestRunner of unittest.runner now uses time.perf_counter() rather
than time.time() to measure the execution time of a test: time.time()
can go backwards, whereas time.perf_counter() is monotonic.
Similar change made in libregrtest, pprint and random.
Replace time.time() with time.monotonic() in tests to measure time
delta.
test_zipfile64: display progress every minute (60 secs) rather than
every 5 minutes (5*60 seconds).
bpo-34279, bpo-35412: support.run_unittest() no longer raises
TestDidNotRun if a test result contains skipped tests. The
exception is now only raised if no test have been run and no test
have been skipped.
The following test fails if a different process creates or removes
a file on the same disk partition between the two lines:
usage = shutil.disk_usage(os.path.dirname(__file__))
self.assertEqual(usage, shutil.disk_usage(__file__))
Only test that disk_usage() succeed on a filename, but don't check
the result. Add also tests on the fields type (must be int).
The length check for AF_ALG salg_name and salg_type had a off-by-one
error. The code assumed that both values are not necessarily NULL
terminated. However the Kernel code for alg_bind() ensures that the last
byte of both strings are NULL terminated.
Signed-off-by: Christian Heimes <christian@python.org>
Fix xml.dom.minidom cloneNode() on a document with an entity: pass
the correct arguments to the user data handler of an entity (fix an
old copy/paste mistake).
Bug spotted and fix proposed by Charalampos Stratakis, initial
reproducer written by Petr Viktorin.
Co-Authored-By: Charalampos Stratakis <cstratak@redhat.com>
Co-Authored-By: Petr Viktorin <encukou@gmail.com>
Replace os.popen() with subprocess.check_output() in the platform module:
* platform.uname() (its _syscmd_ver() helper function) now redirects
stderr to DEVNULL.
* Remove platform.DEV_NULL.
* _syscmd_uname() and _syscmd_file() no longer catch AttributeError.
The "except AttributeError:" was only needed in Python 2, when
os.popen() was not always available. In Python 3,
subprocess.check_output() is always available.
Join 3 pools in these tests:
* test.test_multiprocessing_spawn.WithProcessesTestPool.test_context
* test.test_multiprocessing_spawn.WithProcessesTestPool.test_traceback
When running test_bdb.py as a script, `import test_module` would be
importing the existing Lib/test/test_modules.py instead of the
tempcwd/test_module.py module which was dynamically created by
test_bdb.py itself.
* posixpath.expanduser() now returns the input path unchanged if
the HOME environment variable is not set and pwd.getpwuid() raises
KeyError (the current user identifier doesn't exist in the password
database).
* Add test_no_home_directory() to test_site.
platform.libc_ver() now uses os.confstr('CS_GNU_LIBC_VERSION') if
available and the *executable* parameter is not set. The default
value of the libc_ver() *executable* parameter becomes None.
Quick benchmark on Fedora 29:
python3 -m perf command ./python -S -c 'import platform; platform.libc_ver()'
94.9 ms +- 4.3 ms -> 33.2 ms +- 1.4 ms: 2.86x faster (-65%)
On Travis CI, FTP tests of test_urllib2net randomly fail with "425
Security: Bad IP connecting".
test.pythoninfo now also logs TRAVIS environment variable.
Modify asyncio tests to utilize the certificates from the test directory
instead of its own set, as they are the same and with each update they had
to be updated as well.
* Replace testInheritFlags() with two tests:
testInheritFlagsBlocking() and testInheritFlagsTimeout()
to test different default socket timeout. Moreover, the test now
checks sock.gettimeout() rather than a functional test on recv().
* Replace time.time() with time.monotonic()
* Add socket_setdefaulttimeout() context manager to restore the
default timeout when the test completes.
* Remove testConnect(): accept() wasn't blocking and testAccept()
already tests non-blocking accept().
* Remove accept() functional test from testInitNonBlocking():
already tested by testAccept()
* Rewrite testSetBlocking() with a new assert_sock_timeout() method
* Use addCleanup() and context manager to close sockets
* Replace assertTrue(x < y) with assertLess(x, y)
testAccept() and testRecv() of test_socket.NonBlockingTCPTests have a
race condition: time.sleep() is used as a weak synchronization
primitive and the tests fail randomly on slow buildbots.
Use a reliable threading.Event to fix these tests.
Other changes:
* Replace send() with sendall()
* Expect specific BlockingIOError rather than generic OSError
* Add a timeout to select() in testAccept() and testRecv()
* Use addCleanup() to close sockets
* Use assertRaises()
Remove platform.popen() function, it was deprecated since Python 3.3:
use os.popen() instead.
Rename also the "Removed" section to "API and Feature Removals"
of What's New in Python 3.8.
Fix memory leak in PyUnicode_EncodeLocale() and
PyUnicode_EncodeFSDefault() on error handling.
Changes:
* Fix unicode_encode_locale() error handling
* Fix test_codecs.LocaleCodecTest
Fix WithThreadsTestPool.test_wrapped_exception()
of test_multiprocessing_fork: join the pool.
WithThreadsTestPool.test_del_pool() is now also decorated
with @support.reap_threads.
Fix mktime() overflow error in test_email: run
test_localtime_daylight_true_dst_true() and
test_localtime_daylight_false_dst_true() with a specific timezone.
The lineno and col_offset attributes of AST nodes for list comprehensions,
generator expressions and tuples are now point to the opening parenthesis or
square brace. For tuples without parenthesis they point to the position
of the first item.
Regression introduced in e3ce695 and 25b804a, where the old parameter
update_tryorder to _synthesize was first ignored, then given the opposite
value in the attempt to fix bpo-31014.
There are some same consts in a module. This commit merges them into
single instance. It reduces number of objects in memory after loading modules.
https://bugs.python.org/issue34100
Importing ProcessPoolExecutor may hang or cause an error when the import
accesses urandom on a low resource platform
https://bugs.python.org/issue29877
The "-I" command line option (run Python in isolated mode) is now
also copied by the multiprocessing and distutils modules when
spawning child processes. Previously, only -E and -s options (enabled
by -I) were copied.
subprocess._args_from_interpreter_flags() now copies the -I flag.
Call setitimer() before each test method, instead of once per test
case, to ensure that signals are sent in each test method.
Previously, only the first method of a testcase class got signals.
Changes:
* Replace setUpClass() with setUp() and replace tearDownClass() with
tearDown().
* tearDown() now ensures that at least one signal has been sent.
* Replace support.run_unittest() with unittest.main() which has
a nicer CLI and automatically discover test cases.
Fixes assertion failures in _datetimemodule.c
introduced in the previous fix (see bpo-31752).
Rather of trying to handle an int subclass as exact int,
let it to use overridden special methods, but check the
result of divmod().
* The _PySys_EndInit() function now copies the
config->module_search_path list, so config is longer modified when
sys.path is updated.
* config->warnoptions list and config->xoptions dict are also copied
* test_embed: InitConfigTests now also tests
main_config['module_search_path']
* Fix _Py_InitializeMainInterpreter(): don't use config->warnoptions
but sys.warnoptions to decide if the warnings module should
be imported at startup.
test_embed.InitConfigTests tests more configuration variables.
Changes:
* InitConfigTests tests more core configuration variables:
* base_exec_prefix
* base_prefix
* exec_prefix
* home
* legacy_windows_fs_encoding
* legacy_windows_stdio
* module_search_path_env
* prefix
* "_testembed init_from_config" tests more variables:
* argv
* warnoptions
* xoptions
* InitConfigTests: add check_global_config(), check_core_config() and
check_main_config() subfunctions to cleanup the code. Move also
constants at the class level (ex: COPY_MAIN_CONFIG).
* Fix _PyCoreConfig_AsDict(): don't set stdio_encoding twice
* Use more macros in _PyCoreConfig_AsDict() and
_PyMainInterpreterConfig_AsDict() to reduce code duplication.
* Other minor cleanups.
* Fix _PyCoreConfig_SetGlobalConfig(): set also Py_FrozenFlag
* Fix _PyCoreConfig_AsDict(): export also xoptions
* Add _Py_GetGlobalVariablesAsDict() and _testcapi.get_global_config()
* test.pythoninfo: dump also global configuration variables
* _testembed now serializes global, core and main configurations
using JSON to reuse _Py_GetGlobalVariablesAsDict(),
_PyCoreConfig_AsDict() and _PyMainInterpreterConfig_AsDict(),
rather than duplicating code.
* test_embed.InitConfigTests now test much more configuration
variables
* Fix _PyMainInterpreterConfig_Copy():
copy 'install_signal_handlers' attribute
* Add _PyMainInterpreterConfig_AsDict()
* Add unit tests on the main interpreter configuration
to test_embed.InitConfigTests
* test.pythoninfo: log also main_config
If tracemalloc is not tracing Python memory allocations,
_PyMem_DumpTraceback() now suggests to enable tracemalloc
to get the traceback where the memory block has been allocated.
The warnings module now suggests to enable tracemalloc if the source
is specified, tracemalloc module is available, but tracemalloc is not
tracing memory allocations.
Fix an off by one error in the peephole optimizer when checking for unreachable code beyond a return.
Do a bounds check within find_op so it can return before going past the end as a safety measure.
7db3c48833 (diff-a33329ae6ae0bb295d742f0caf93c137)
introduced this off by one error while fixing another one nearby.
This bug was shipped in all Python 3.6 and 3.7 releases.
The included unittest won't fail unless you do a clang msan build.
The call to `_untrack_reader` is performed too soon, causing the protocol
to forget about the reader before `connection_lost` can run and feed the
EOF to the reader. See bpo-35065.
Current support for hash-based bytecode files in `zipimport` is rather
sparse, which leads to test failures when the test suite is ran with
the ``SOURCE_DATE_EPOCH`` environment variable set.
This teaches zipimport to handle hash-based pycs properly.
Without setting mtime, time.time() will be used as the timestamp which will
end up in the compressed data and each invocation of the compress() function
will vary over time.
The test depended on '/usr/share/zoneinfo/posixrules' or equivalent
because it set TZ without explicit DST transition rules. At least
on OpenSUSE Tumbleweed that file is linked to '/etc/localtime',
making the test fail with certain local timezones,
such as 'Europe/Moscow' which doesn't have DST transitions since 2011.
Two kind of mistakes:
1. Missed space. After concatenating there is no space between words.
2. Missed comma. Causes unintentional concatenating in a list of strings.
The new option in the CLI of the profile module allow to profile
executable modules. This change follows the same implementation as the
one already present in `cProfile`.
As the argument is now present on both modules, move the tests to the
common test case to be run with profile as well.
This implements getstate and setstate for the cjkcodecs multibyte incremental encoders/decoders, primarily to fix issues with seek/tell.
The encoder getstate/setstate is slightly tricky as the "state" is pending bytes + MultibyteCodec_State but only an integer can be returned. The approach I've taken is to encode this data into a long, similar to how .tell() encodes a "cookie_type" as a long.
https://bugs.python.org/issue33578
Some FreeBSD buildbots fail to run this test as the eof was not being received by the server if the size is not big enough. This behaviour only appears if the client is using TLS1.3.
After commit d0f49d2f50, the output of the
test suite is always buffered as the test output needs to be included in
the JUnit file in same cases (as when a test fails). This has the
consequence that printing or using debuggers (like pdb) in the test
suite does not result in a good user experience anymore.
This commit modifies the test suite runner so it only captures the test
output when the JUnit file is requested to fix the regression so prints
and debuggers are usable again.
The list() constructor isn't taking full advantage of known input
lengths or length hints. This commit makes the constructor
pre-size and not over-allocate when the input size is known (the
input collection implements __len__). One on the main advantages is
that this provides 12% difference in memory savings due to the difference
between overallocating and allocating exactly the input size.
For efficiency purposes and to avoid a performance regression for small
generators and collections, the size of the input object is calculated using
__len__ and not __length_hint__, as the later is considerably slower.
inspect.isfunction() processes both inspect.isfunction(func) and
inspect.isfunction(partial(func, arg)) correctly but some other functions in the
inspect module (iscoroutinefunction, isgeneratorfunction and isasyncgenfunction)
lack this functionality. This commits adds a new check in the mentioned functions
in the inspect module so they can work correctly with arbitrarily nested partial
functions.
Changes:
* Add _PyObject_AssertFailed() function.
* Add _PyObject_ASSERT() and _PyObject_ASSERT_WITH_MSG() macros.
* gc_decref(): replace assert() with _PyObject_ASSERT_WITH_MSG() to
dump the faulty object if the assertion fails.
_PyObject_AssertFailed() calls:
* _PyMem_DumpTraceback(): try to log the traceback where the object
memory has been allocated if tracemalloc is enabled.
* _PyObject_Dump(): log repr(obj).
* Py_FatalError(): log the current Python traceback.
_PyObject_AssertFailed() uses _PyObject_IsFreed() heuristic to check
if the object memory has been freed by a debug hook on Python memory
allocators.
Initial patch written by David Malcolm.
Co-Authored-By: David Malcolm <dmalcolm@redhat.com>
* Add Py_STATIC_INLINE() macro to declare a "static inline" function.
If the compiler supports it, try to always inline the function even if no
optimization level was specified.
* Modify pydtrace.h to use Py_STATIC_INLINE() when WITH_DTRACE is
not defined.
* Add an unit test on Py_DECREF() to make sure that
_Py_NegativeRefcount() reports the correct filename.
tracemalloc now tries to update the traceback when an object is
reused from a "free list" (optimization for faster object creation,
used by the builtin list type for example).
Changes:
* Add _PyTraceMalloc_NewReference() function which tries to update
the Python traceback of a Python object.
* _Py_NewReference() now calls _PyTraceMalloc_NewReference().
* Add an unit test.
* Use _PyUnicode_Copy in sanitize_isoformat_str
* Use repr in fromisoformat error message
This reverses commit 67b74a98b2 per Serhiy Storchaka's suggestion:
I suggested to use %R in the error message because including the raw
string can be confusing in the case of empty string, or string
containing trailing whitespaces, invisible or unprintable characters.
We agree that it is better to change both the C and pure Python versions
to use repr.
* Retain non-sanitized dtstr for error printing
This does not create an extra string, it just holds on to a reference to
the original input string for purposes of creating the error message.
* PEP 7 fixes to from_isoformat
* Separate handling of Unicode and other errors
In the initial implementation, errors other than encoding errors would
both raise an error indicating an invalid format, which would not be
true for errors like MemoryError.
* Drop needs_decref from _sanitize_isoformat_str
Instead _sanitize_isoformat_str returns a new reference, even to the
original string.
For builtin types with builtin subclasses, help() on the type now shows up
to 4 of the subclasses. This partially replaces the exception hierarchy
information previously displayed in Python 2.7.
path_error() uses GetLastError() on Windows, but some os functions
are implemented via CRT APIs which report errors via errno.
This may result in raising OSError with invalid error code (such
as zero).
Introduce posix_path_error() function and use it where appropriate.
If buffering=1 is specified for open() in binary mode, it is silently
treated as buffering=-1 (i.e., the default buffer size).
Coupled with the fact that line buffering is always supported in Python 2,
such behavior caused several issues (e.g., bpo-10344, bpo-21332).
Warn that line buffering is not supported if open() is called with
binary mode and buffering=1.
The reprlib code was copied here instead of importing reprlib. I'm not sure if we really need to avoid the import, but since I expect dataclasses to be more common that reprlib, it seems wise. Plus, the code is small.
Adding `max_num_fields` to `cgi.FieldStorage` to make DOS attacks harder by
limiting the number of `MiniFieldStorage` objects created by `FieldStorage`.
It is now guarantied that children of xml.etree.ElementTree.Element
are Elements (at least in C implementation). Previously methods
__setitem__(), __setstate__() and __deepcopy__() could be used for
adding non-Element children.
When runnint test_logging with --huntrleaks after commit
18fb1fb943, test_out_of_order fails
to raise ValueError due to the fact that the new test
test_out_of_order_with_dollar_style mutates the out_of_order
dictionary. Even if the test copies the dictionary first, the mutation
is done in a very deep level so the original one is also affected.
Fix test_cmd_line_script.test_nonexisting_script(): the test must not
rely on sys.executable, since main.c uses config->program which can
be different than sys.executable in many cases (for example, on macOS
when using the framework).
Allow annotated global names in the module namespace after the symbol is
declared as global. Previously, only symbols annotated before they are declared
as global (i.e. inside a function) were allowed. This change allows symbols to be
declared as global before the annotation happens in the global scope.
With macOS framework builds, test case test_nonexisting_script in
test_nonexisting_script fails because the test case assumes that
the file name in sys.executable will appear in the error message.
For macOS framework builds, sys.executable is the file name of the
stub launcher and its file name bears no relationship to the file
name of the actual python executable. For now, skip the test in
this case.
Unconditional forcing of ``CHECKED_HASH`` invalidation was introduced in
3.7.0 in bpo-29708. The change is bad, as it unconditionally overrides
*invalidation_mode*, even if it was passed as an explicit argument to
``py_compile.compile()`` or ``compileall``. An environment variable
should *never* override an explicit argument to a library function.
That change leads to multiple test failures if the ``SOURCE_DATE_EPOCH``
environment variable is set.
This changes ``py_compile.compile()`` to only look at
``SOURCE_DATE_EPOCH`` if no explicit *invalidation_mode* was specified.
I also made various relevant tests run with explicit control over the
value of ``SOURCE_DATE_EPOCH``.
While looking at this, I noticed that ``zipimport`` does not work
with hash-based .pycs _at all_, though I left the fixes for
subsequent commits.
Tests involving sending signals to the semaphore_tracker will not fail anymore due to
the fact that running the test suite with -Werror propagates warnings as errors.
Fix a missing assertion when the semaphore_tracker is expected to die.
When Python is built with the intel control-flow protection flags,
-mcet -fcf-protection, gdb is not able to read the stack without
actually jumping inside the function. This means an extra
'next' command is required to make the $pc (program counter)
enter the function and make the stack of the function exposed to gdb.
Co-Authored-By: Marcel Plch <gmarcel.plch@gmail.com>
(cherry picked from commit 9b7c74ca32)
The C implementation of asyncio.Task currently fails to perform the
cancellation cleanup correctly in the following scenario.
async def task1():
async def task2():
await task3 # task3 is never cancelled
asyncio.current_task().cancel()
await asyncio.create_task(task2())
The actuall error is a hardcoded call to `future_cancel()` instead of
calling the `cancel()` method of a future-like object.
Thanks to Vladimir Matveev for noticing the code discrepancy and to
Yury Selivanov for coming up with a pathological scenario.
Fix a reference issue inside multiprocessing.Pool that caused the pool to remain alive if it was deleted without being closed or terminated explicitly.
* Compiling a string annotation containing a lambda with keyword-only
argument without default value caused a crash.
* Remove the final "*" (it is incorrect syntax) in the representation of
lambda without *args and keyword-only arguments when compile from AST.
* Improve the representation of lambda without arguments.
The waiting is pretty normal for any asyncio program, logging its time just adds
a noise to logs without any useful information provided.
https://bugs.python.org/issue34849
Improvements:
1. Include the number of valid data characters in the error message.
2. Mention "number of data characters" rather than "length".
https://bugs.python.org/issue34736
Report the filename to the exception when raising {gdbm,dbm.ndbm}.error in
dbm.gnu.open() and dbm.ndbm.open() functions, so it gets printed when the
exception is raised, and can also be obtained by the filename attribute of the
exception object.
After some failures in AMD64 FreeBSD CURRENT Debug 3.x buildbots
regarding tests in test_socket that are using
testFDPassSeparateMinSpace(), FreeBDS revision 337423 was pointed
out to be the reason the test started to fail.
A close examination of the manpage for cmsg_space(3) reveals that
the number of file descriptors needs to be taken into account when
using CMSG_LEN().
This commit fixes tests in test_socket to use correctly CMSG_LEN, taking
into account the number of FDs.
When dict subclass overrides order (`__iter__()`, `keys()`, and `items()`), `dict(o)`
should use it instead of dict ordering.
https://bugs.python.org/issue34320
Serhiy Storchaka
stratakis
Victor Stinner
Vladimir Matveev
Zackery Spytz
Vajrasky Kok
Dima Tisnek
Xiang Zhang
Andrew Svetlov
Pablo Galindo
Steve Dower
Christian Heimes
Ned Deily
native-api
Alex H
Ismo Toijala
INADA Naoki
Zhiming Wang
Lisa Roach
Dustin Spicuzza
nierob
Srinivas Thatiparthy (శ్రీనివాస్ తాటిపర్తి)
Gregory P. Smith
Vincent Michel
Elvis Pranskevichus
Cheryl Sabella
Diego Rojas
HongWeipeng
guoci
Alexey Izbyshev
Rémi Lapeyre
Mario Corchero
Max Bélanger
Julien Palard
Lysandros Nikolaou
Stéphane Wirtel
Christopher Thorne
Raymond Hettinger
jdemeyer
Andrés Delfino
Tim Graham
madman-bob
matthewbelisle-wf
Paul Ganssle
Sanyam Khurana
Sergey Fedoseev
Srinivas Thatiparthy (శ్రీనివాస్ తాటిపర్తి)
BNMetrics
Mayank Asthana
twisteroid ambassador
Romuald Brunet
Paul Bailey
Ethan Furman
Chih-Hsuan Yen
tzickel
Yury Selivanov
Benjamin Peterson
Tal Einat
Zsolt Cserna
Bjorn Andersson
Joe Pamer
Ammar Askar