and there are non-ascii characters in the header.
(cherry picked from commit feac6cd775)
Co-authored-by: Abhilash Raj <maxking@users.noreply.github.com>
Move the check for dead conditionals (if 0) to the peephole optimizer
and make sure that the code block is still compiled to report any
existing syntax errors within.
(cherry picked from commit af8646c805)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
This PR proposes a solution to [bpo-35545](https://bugs.python.org/issue35545) by adding an optional `flowinfo` and `scopeid` to `asyncio.base_events._ipaddr_info` to carry the full address information into `_ipaddr_info` and avoid discarding IPv6 specific information.
Changelog entry & regression tests to come.
https://bugs.python.org/issue35545
(cherry picked from commit ac8eb8f36b)
Co-authored-by: Erwan Le Pape <lepaperwan@users.noreply.github.com>
The final addition (cur += step) may overflow, so use size_t for "cur".
"cur" is always positive (even for negative steps), so it is safe to use
size_t here.
Co-Authored-By: Martin Panter <vadmium+py@gmail.com>
(cherry picked from commit 14514d9084)
Co-authored-by: Zackery Spytz <zspytz@gmail.com>
When using multiprocessing (-jN option), worker processes now create
their temporary directory inside the temporary directory of the
main process. So the main process is able to remove temporary
directories of worker processes even if they crash or when they are
killed by regrtest on KeyboardInterrupt (CTRL+c).
Rework also how multiprocessing arguments are parsed in main.py.
(cherry picked from commit 3c93153f7d)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
Extended attributes can only be set on user-writeable files, but shutil previously
first chmod()ed the destination file to the source's permissions and then tried to
copy xattrs. This will cause failures if attempting to copy read-only files with
xattrs, as occurs with Git clones on Lustre FS.
(cherry picked from commit 79efbb7193)
Co-authored-by: Olexa Bilaniuk <obilaniu@users.noreply.github.com>
"python3 -m test -jN ..." now continues the execution of next tests
when a worker process crash (CHILD_ERROR state). Previously, the test
suite stopped immediately. Use --failfast to stop at the first error.
Moreover, --forever now also implies --failfast.
(cherry picked from commit b0917df329)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
* A pointer in `PyInterpreterState_New()` could have been `NULL` when being dereferenced.
* Memory was leaked in `PyInterpreterState_New()` when taking some error-handling code path.
(cherry picked from commit 95d630e)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
Instead of attempting to acquire and release them all across fork
which was leading to deadlocks in some applications that had chained
their own handlers while holding multiple locks.
(cherry picked from commit 64aa6d2000)
Co-authored-by: Gregory P. Smith <greg@krypto.org> [Google LLC]
When the future returned by shield is cancelled, its completion callback of the
inner future is not removed. This makes the callback list of inner inner future
grow each time a shield is created and cancelled.
This change unregisters the callback from the inner future when the outer
future is cancelled.
https://bugs.python.org/issue35125
(cherry picked from commit b35acc5b3a)
Co-authored-by: Romain Picard <romain.picard@oakbits.com>
*Moved from python/asyncioGH-493.*
This PR fixes issue python/asyncioGH-480, as explained in [this comment](https://github.com/python/asyncio/issues/480GH-issuecomment-278703828).
The `_SelectorDatagramTransport.sendto` method has to be modified ~~so `_sock.sendto` is used in all cases (because it is tricky to reliably tell if the socket is connected or not). Could that be an issue for connected sockets?~~ *EDIT* ... so `_sock.send` is used only if `_sock` is connected.
It also protects `socket.getsockname` against `OSError` in `_SelectorTransport`. This might happen on Windows if the socket is not connected (e.g. for UDP broadcasting).
https://bugs.python.org/issue31922
(cherry picked from commit 63deaa5b70)
Co-authored-by: Vincent Michel <vxgmichel@gmail.com>
Disallow control chars in http URLs in urllib.urlopen. This addresses a potential security problem for applications that do not sanity check their URLs where http request headers could be injected.
Disable https related urllib tests on a build without ssl (GH-13032)
These tests require an SSL enabled build. Skip these tests when python is built without SSL to fix test failures.
Use http.client.InvalidURL instead of ValueError as the new error case's exception. (GH-13044)
Backport Co-Authored-By: Miro Hrončok <miro@hroncok.cz>
unittest.mock.mock_open() results now respects the argument of read([size])
Co-Authored-By: remilapeyre <remi.lapeyre@henki.fr>
(cherry picked from commit 11a8832c98)
Co-authored-by: Rémi Lapeyre <remi.lapeyre@henki.fr>
Modern Linux distros such as Debian Buster have default OpenSSL system
configurations that reject connections to servers with weak certificates
by default. This causes our test suite run with external networking
resources enabled to skip these tests when they encounter such a failure.
Fixing the network servers is a separate issue.
(cherry picked from commit 2cc0223f43)
Co-authored-by: Gregory P. Smith <greg@krypto.org>
QueueHandler.prepare() now makes a copy of the record before modifying and enqueueing it, to avoid affecting other handlers in the chain.
(cherry picked from commit da6424e96a)
Co-authored-by: Manjusaka <lizheao940510@gmail.com>
Under some conditions the earlier fix for bpo-18075, "Infinite recursion
tests triggering a segfault on Mac OS X", now causes failures on macOS
when attempting to change stack limit with resource.setrlimit
resource.RLIMIT_STACK, like regrtest does when running the test suite.
The reverted change had specified a non-default stack size when linking
the python executable on macOS. As of macOS 10.14.4, the previous
code causes a hard failure when running tests, although similar
failures had been seen under some conditions under some earlier
systems. For now, revert the original change and resume using
the default stack size when linking the interpreter.
(cherry picked from commit 883dfc668f)
Co-authored-by: Ned Deily <nad@python.org>
* Clean up code which checked presence of os.{stat,lstat,chmod} (GH-11643)
(cherry picked from commit 8377cd4fcd)
* bpo-36725: regrtest: add TestResult type (GH-12960)
* Add TestResult and MultiprocessResult types to ensure that results
always have the same fields.
* runtest() now handles KeyboardInterrupt
* accumulate_result() and format_test_result() now takes a TestResult
* cleanup_test_droppings() is now called by runtest() and mark the
test as ENV_CHANGED if the test leaks support.TESTFN file.
* runtest() now includes code "around" the test in the test timing
* Add print_warning() in test.libregrtest.utils to standardize how
libregrtest logs warnings to ease parsing the test output.
* support.unload() is now called with abstest rather than test_name
* Rename 'test' variable/parameter to 'test_name'
* dash_R(): remove unused the_module parameter
* Remove unused imports
(cherry picked from commit 4d29983185)
* bpo-36725: Refactor regrtest multiprocessing code (GH-12961)
Rewrite run_tests_multiprocess() function as a new MultiprocessRunner
class with multiple methods to better report errors and stop
immediately when needed.
Changes:
* Worker processes are now killed immediately if tests are
interrupted or if a test does crash (CHILD_ERROR): worker
processes are killed.
* Rewrite how errors in a worker thread are reported to
the main thread. No longer ignore BaseException or parsing errors
silently.
* Remove 'finished' variable: use worker.is_alive() instead
* Always compute omitted tests. Add Regrtest.get_executed() method.
(cherry picked from commit 3cde440f20)
* bpo-36719: regrtest always detect uncollectable objects (GH-12951)
regrtest now always detects uncollectable objects. Previously, the
check was only enabled by --findleaks. The check now also works with
-jN/--multiprocess N.
--findleaks becomes a deprecated alias to --fail-env-changed.
(cherry picked from commit 75120d2205)
* bpo-34060: Report system load when running test suite for Windows (GH-8357)
While Windows exposes the system processor queue length, the raw value
used for load calculations on Unix systems, it does not provide an API
to access the averaged value. Hence to calculate the load we must track
and average it ourselves. We can't use multiprocessing or a thread to
read it in the background while the tests run since using those would
conflict with test_multiprocessing and test_xxsubprocess.
Thus, we use Window's asynchronous IO API to run the tracker in the
background with it sampling at the correct rate. When we wish to access
the load we check to see if there's new data on the stream, if there is,
we update our load values.
(cherry picked from commit e16467af0b)
* bpo-36719: Fix regrtest re-run (GH-12964)
Properly handle a test which fail but then pass.
Add test_rerun_success() unit test.
(cherry picked from commit 837acc1957)
* bpo-36719: regrtest closes explicitly WindowsLoadTracker (GH-12965)
Regrtest.finalize() now closes explicitly the WindowsLoadTracker
instance.
(cherry picked from commit 00db7c73af)
bpo-28552, bpo-7774: Fix distutils.sysconfig if sys.executable is
None or an empty string: use os.getcwd() to initialize project_base.
Fix also the distutils build command: don't use sys.executable if
it's evaluated as false (None or empty string).
(cherry picked from commit 0ef8c157e9)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
Change test_time.test_monotonic() to test only the lower bound of elapsed time
after a sleep command rather than the upper bound. This prevents unnecessary
test failures on slow buildbots. Patch by Victor Stinner.
(cherry picked from commit d246a6766b)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
* bpo-35755: shutil.which() uses os.confstr("CS_PATH") (GH-12858)
shutil.which() and distutils.spawn.find_executable() now use
os.confstr("CS_PATH") if available instead of os.defpath, if the PATH
environment variable is not set.
Don't use os.confstr("CS_PATH") nor os.defpath if the PATH
environment variable is set to an empty string.
Changes:
* find_executable() now starts by checking for the executable in the
current working directly case. Add an explicit
"if not path: return None".
* Add tests for PATH='' (empty string), PATH=':' and for PATHEXT.
(cherry picked from commit 228a3c99bd)
* bpo-35755: Remove current directory from posixpath.defpath (GH-11586)
Document the change in a NEWS entry of the Security category.
(cherry picked from commit 2c4c02f8a8)
Fix Python Initialization code on FreeBSD to detect properly when
stdin file descriptor (fd 0) is invalid.
On FreeBSD, fstat() must be used to check if stdin (fd 0) is valid.
dup(0) doesn't fail if stdin is invalid in some cases.
(cherry picked from commit 3092d6b263)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
python-config --ldflags no longer includes flags of the LINKFORSHARED
variable. The LINKFORSHARED variable must only be used to build
executables.
(cherry picked from commit e65f01f78d)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
Fix test_imap4_host_default_value() of test_imaplib: catch also
errno.ENETUNREACH error.
(cherry picked from commit 3c7931e514)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
In Python having a trace function in effect while mock is imported causes isinstance to be wrong for MagicMocks. This is due to the usage of super() in some class methods, as this sets the __class__ attribute. To avoid this, as a workaround, alias the usage of super .
(cherry picked from commit 830b43d03c)
Co-authored-by: Xtreak <tir.karthi@gmail.com>
Fix test_sys.test_getallocatedblocks() when tracemalloc is enabled.
If the name of Python memory allocators cannot get read, consider
that pymalloc is disabled.
Fix the following error:
./python -X tracemalloc -m test test_sys -v -m test_getallocatedblocks
ERROR: test_getallocatedblocks (test.test_sys.SysModuleTest)
------------------------------------------------------------
Traceback (most recent call last):
File "Lib/test/test_sys.py", line 770, in test_getallocatedblocks
alloc_name = _testcapi.pymem_getallocatorsname()
RuntimeError: cannot get allocators name
(cherry picked from commit 9b8314cfe2)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
* bpo-36389: _PyObject_IsFreed() now also detects uninitialized memory (GH-12770)
Replace _PyMem_IsFreed() function with _PyMem_IsPtrFreed() inline
function. The function is now way more efficient, it became a simple
comparison on integers, rather than a short loop. It detects also
uninitialized bytes and "forbidden bytes" filled by debug hooks
on memory allocators.
Add unit tests on _PyObject_IsFreed().
(cherry picked from commit 2b00db6855)
* bpo-36389: Change PyMem_SetupDebugHooks() constants (GH-12782)
Modify CLEANBYTE, DEADDYTE and FORBIDDENBYTE constants: use 0xCD,
0xDD and 0xFD, rather than 0xCB, 0xBB and 0xFB, to use the same byte
patterns than Windows CRT debug malloc() and free().
(cherry picked from commit 4c409beb4c)
* bpo-36560: Fix reference leak hunting in regrtest (GH-12744)
Fix reference leak hunting in regrtest: compute also deltas (of
reference count, allocated memory blocks, file descriptor count)
during warmup, to ensure that everything is initialized before
starting to hunt reference leaks.
Other changes:
* Replace gc.collect() with support.gc_collect()
* Move calls to read memory statistics from dash_R_cleanup() to
dash_R()
* Pass regrtest 'ns' to dash_R()
* dash_R() is now more quiet with --quiet option (don't display
progress).
* Precompute the full range for "for it in range(repcount):" to
ensure that the iteration doesn't allocate anything new.
* dash_R() now is responsible to call warm_caches().
(cherry picked from commit 5aaac94eeb)
* bpo-36560: regrtest: don't collect the GC twice (GH-12747)
dash_R() function of libregrtest doesn't call support.gc_collect()
directly anymore: it's already called by dash_R_cleanup().
Call dash_R_cleanup() before starting the loop.
(cherry picked from commit bb4447897a)
Fix reference hunting (``python3 -m test -R 3:3``) when Python has no
built-in abc module: fix _get_dump() reimplementation of libregrtest.
(cherry picked from commit 79b5d29041)
Co-authored-by: Victor Stinner <vstinner@redhat.com>
The error messages in the parser module are referring to numeric IDs for the nodes. To improve readability, use the node names when reporting errors..
(cherry picked from commit cb0748d393)
Co-authored-by: tyomitch <tyomitch@gmail.com>
I have added documentation for `PyInterpreterState_Main()`.
I chose to place it under Advanced Debugger Support together with similar functions like `PyInterpreterState_Head()`, `PyInterpreterState_Next(`), and `PyInterpreterState_ThreadHead()` .
https://bugs.python.org/issue36157
(cherry picked from commit 8c61739def)
Co-authored-by: Joannah Nanjekye <33177550+nanjekyejoannah@users.noreply.github.com>
The following arguments can be passed as keyword arguments for passing
to other function if the corresponding required argument is passed as
positional:
- "func" in functools.partialmethod(), weakref.finalize(),
profile.Profile.runcall(), cProfile.Profile.runcall(),
bdb.Bdb.runcall(), trace.Trace.runfunc() and
curses.wrapper().
- "function" in unittest.addModuleCleanup() and
unittest.TestCase.addCleanup().
- "fn" in the submit() method of concurrent.futures.ThreadPoolExecutor
and concurrent.futures.ProcessPoolExecutor.
- "callback" in contextlib.ExitStack.callback(),
contextlib.AsyncExitStack.callback() and
contextlib.AsyncExitStack.push_async_callback().
- "c" and "typeid" in multiprocessing.managers.Server.create().
- "obj" in weakref.finalize().
(cherry picked from commit 42a139ed88)
The bug occurred when the encoded surrogate character is passed
to the incremental decoder in two chunks.
(cherry picked from commit 7a465cb5ee)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Errors during writing no longer prevent to properly close
the ZIP file.
(cherry picked from commit 2524fdefc9)
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Add a function to collect certificates from several certificate stores into one certificate collection store that is then enumerated. This ensures we load as many certificates as we can access.
(cherry picked from commit d93fbbf88e)
Co-authored-by: kctherookie <48805853+kctherookie@users.noreply.github.com>
Remove the PyMem_FREE() call added in cb90c89. The buffer will be
freed when PyTokenizer_Free() is called on the tokenizer state.
(cherry picked from commit cda139d1de)
Co-authored-by: Zackery Spytz <zspytz@gmail.com>
Add idlelib.pyshell alias at top; remove pyshell alias at bottom.
Remove obsolete __name__=='__main__' command.
(cherry picked from commit 6a258c8890)
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Set type_attr to NULL after the assignment to stgdict->proto (like
what is done with stgdict after the Py_SETREF() call) so that it is
not decrefed twice on error.
(cherry picked from commit 5e333784f0)
Co-authored-by: Zackery Spytz <zspytz@gmail.com>
Check for sys.abiflags before using since not all platforms have it defined.
(cherry picked from commit a7987e7193)
Co-authored-by: AraHaan <15173749+AraHaan@users.noreply.github.com>
* Add tests for grep findfiles.
* Move findfiles to module function.
* Change findfiles to use os.walk.
Based on a patch by Al Sweigart.
(cherry picked from commit d60f658fc0)
Co-authored-by: Cheryl Sabella <cheryl.sabella@gmail.com>
bpo-36256: Fix bug in parsermodule when parsing if statements
In the parser module, when validating nodes before starting the parsing with to create a ST in "parser_newstobject" there is a problem that appears when two arcs in the same DFA state has transitions with labels with the same type. For example, the DFA for if_stmt has a state with
two labels with the same type: "elif" and "else" (type NAME). The algorithm tries one by one the arcs until the label that starts the arc transition has a label with the same type of the current child label we are trying to accept. In this case, the arc for "elif" comes before the arc for "else"and passes this test (because the current child label is "else" and has the same type as "elif"). This lead to expecting a namedexpr_test (305) instead of a colon (11). The solution is to compare also the string representation (in case there is one) of the labels to see if the transition that we have is the correct one.
(cherry picked from commit 9a0000d15d)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
Be explicit that timedelta division converts an overall duration to the interval
units given by the denominator.
(cherry picked from commit f40b4a0b62)
Co-authored-by: Yasser A <yalshalaan@gmail.com>
Fix CFLAGS in customize_compiler() of distutils.sysconfig: when the
CFLAGS environment variable is defined, don't override CFLAGS variable with
the OPT variable anymore.
Initial patch written by David Malcolm.
Co-Authored-By: David Malcolm <dmalcolm@redhat.com>
(cherry picked from commit 86082c22d2)
Fix an unlikely memory leak on conversion from string to float in the
function _Py_dg_strtod() used by float(str), complex(str),
pickle.load(), marshal.load(), etc.
Fix an unlikely memory leak in _Py_dg_strtod() on "undfl:" label:
rewrite memory management in this function to always release all
memory before exiting the function. Initialize variables to NULL, and
set them to NULL after calling Bfree() at the "cont:" label.
Note: Bfree(NULL) is well defined: it does nothing.
(cherry picked from commit 9776b0636a)
Miss Islington (bot)
Steve Dower
Victor Stinner
Pablo Galindo
Gregory P. Smith
Miro Hrončok
Xtreak
Victor Stinner
Marcin Niemira
Berker Peksag
Inada Naoki
Serhiy Storchaka
Ned Deily
Harmon
Julien Palard