The error messages in `object.__new__` and `object.__init__` now aim
to point the user more directly at the name of the class being instantiated
in cases where they *haven't* been overridden (on the assumption that
the actual problem is a missing `__new__` or `__init__` definition in the
class body).
When they *have* been overridden, the errors still report themselves as
coming from object, on the assumption that the problem is with the call
up to the base class in the method implementation, rather than with the
way the constructor is being called.
Python now supports checking bytecode cache up-to-dateness with a hash of the
source contents rather than volatile source metadata. See the PEP for details.
While a fairly straightforward idea, quite a lot of code had to be modified due
to the pervasiveness of pyc implementation details in the codebase. Changes in
this commit include:
- The core changes to importlib to understand how to read, validate, and
regenerate hash-based pycs.
- Support for generating hash-based pycs in py_compile and compileall.
- Modifications to our siphash implementation to support passing a custom
key. We then expose it to importlib through _imp.
- Updates to all places in the interpreter, standard library, and tests that
manually generate or parse pyc files to grok the new format.
- Support in the interpreter command line code for long options like
--check-hash-based-pycs.
- Tests and documentation for all of the above.
* Convert asyncio/tasks.py to async/await
* Convert asyncio/queues.py to async/await
* Convert asyncio/test_utils.py to async/await
* Convert asyncio/base_subprocess.py to async/await
* Convert asyncio/subprocess.py to async/await
* Convert asyncio/streams.py to async/await
* Fix comments
* Convert asyncio/locks.py to async/await
* Convert asyncio.sleep to async def
* Add a comment
* Add missing news
* Convert stubs from AbstrctEventLoop to async functions
* Convert subprocess_shell/subprocess_exec
* Convert connect_read_pipe/connect_write_pip to async/await syntax
* Convert create_datagram_endpoint
* Convert create_unix_server/create_unix_connection
* Get rid of old style coroutines in unix_events.py
* Convert selector_events.py to async/await
* Convert wait_closed and create_connection
* Drop redundant line
* Convert base_events.py
* Code cleanup
* Drop redundant comments
* Fix indentation
* Add explicit tests for compatibility between old and new coroutines
* Convert windows event loop to use async/await
* Fix double awaiting of async function
* Convert asyncio/locks.py
* Improve docstring
* Convert tests to async/await
* Convert more tests
* Convert more tests
* Convert more tests
* Convert tests
* Improve test
* fix issue32208: update threading.Semaphore docs and add unit test to validate correct behavior
* add test for blocking
* Update threading.rst
* semaphore: remove documentation validation tests and move 'return value' test to BaseSemaphore
The original algorithm tried to delegate the folding to the tokens so
that those tokens whose folding rules differed could specify the
differences. However, this resulted in a lot of duplicated code because
most of the rules were the same.
The new algorithm moves all folding logic into a set of functions
external to the token classes, but puts the information about which
tokens can be folded in which ways on the tokens...with the exception of
mime-parameters, which are a special case (which was not even
implemented in the old folder).
This algorithm can still probably be improved and hopefully simplified
somewhat.
Note that some of the test expectations are changed. I believe the
changes are toward more desirable and consistent behavior: in general
when (re) folding a line the canonical version of the tokens is
generated, rather than preserving errors or extra whitespace.
Previously, CO_NOFREE was set in the compiler, which meant
it could end up being set incorrectly when code objects
were created directly. Setting it in the constructor based
on freevars and cellvars ensures it is always accurate,
regardless of how the code object is defined.
If the marshal or bytecode formats get changed, frozen.c needs to
be updated as well. It can be easy to miss this step and not doing
so can cause test_importlib to crash in mysterious ways. Add an
explict unit test to make it easier to track down the problem.
The current behaviour of yield expressions inside comprehensions and
generator expressions is essentially an accident of implementation - it
arises implicitly from the way the compiler handles yield expressions inside
nested functions and generators.
Since the current behaviour wasn't deliberately designed, and is inherently
confusing, we're deprecating it, with no current plans to reintroduce it.
Instead, our advice will be to use a named nested generator definition
for cases where this behaviour is desired.
* Fixed saving bytearrays.
* Identical objects will be saved only once.
* Equal references will be load as identical objects.
* Added support for saving and loading recursive data structures.
When PyGILState_Ensure() is called in a non-Python thread before
PyEval_InitThreads(), only call PyEval_InitThreads() after calling
PyThreadState_New() to fix a crash.
Add an unit test in test_embed.
* bpo-32101: Add sys.flags.dev_mode flag
Rename also the "Developer mode" to the "Development mode".
* bpo-32101: Add PYTHONDEVMODE environment variable
Mention it in the development chapiter.
* Add most_recent_first parameter to tracemalloc.Traceback.format to allow
reversing the order of the frames in the output
* Reversed default sorting of tracemalloc.Traceback frames
* Allowed negative limit, truncating from the other side.
* Fix _PyMem_SetupAllocators("debug"): always restore allocators to
the defaults, rather than only caling _PyMem_SetupDebugHooks().
* Add _PyMem_SetDefaultAllocator() helper to set the "default"
allocator.
* Add _PyMem_GetAllocatorsName(): get the name of the allocators
* main() now uses debug hooks on memory allocators if Py_DEBUG is
defined, rather than calling directly malloc()
* Document default memory allocators in C API documentation
* _Py_InitializeCore() now fails with a fatal user error if
PYTHONMALLOC value is an unknown memory allocator, instead of
failing with a fatal internal error.
* Add new tests on the PYTHONMALLOC environment variable
* Add support.with_pymalloc()
* Add the _testcapi.WITH_PYMALLOC constant and expose it as
support.with_pymalloc().
* sysconfig.get_config_var('WITH_PYMALLOC') doesn't work on Windows, so
replace it with support.with_pymalloc().
* pythoninfo: add _testcapi collector for pymem
``uuid.getnode()`` now preferentially returns universally administered MAC addresses if available, over locally administered MAC addresses. This makes a better guarantee for global uniqueness of UUIDs returned from ``uuid.uuid1()``. If only locally administered MAC addresses are available, the first such one found is returned.
Also improve internal code style by being explicit about ``return None`` rather than falling off the end of the function.
Improve the test robustness.
CPython migrated from CVS to Subversion, to Mercurial, and then to
Git. CVS and Subversion are not more used to develop CPython.
* platform module: drop support for sys.subversion. The
sys.subversion attribute has been removed in Python 3.3.
* Remove Misc/svnmap.txt
* Remove Tools/scripts/svneol.py
* Remove Tools/scripts/treesync.py
* Remove asyncio.selectors and asyncio._overlapped symbols from the
namespace of the asyncio module
* Replace "from asyncio import selectors" with "import selectors"
* Replace "from asyncio import _overlapped" with "import _overlapped"
asyncio.selectors was added to support Python 3.3, which doesn't have
selectors in its standard library, and Python 3.4 in the same code
base. Same rationale for asyncio._overlapped. Python 3.3 reached its
end of life, and asyncio is no more maintained as a third party
module on PyPI.
Some parts of the C API are only relevant to larger
applications embedding CPython as a runtime engine.
The helpers to test those APIs are already separated
out into Programs/_testembed.c, this update moves
the associated test cases out into their own dedicated
test file.
Improve UUID1 MAC address calculation and related tests.
There are two bits in the MAC address that are relevant to UUID1. The first is the locally administered vs. universally administered bit (second least significant of the first octet). Physical network interfaces such as ethernet ports and wireless adapters will always be universally administered, but some interfaces --such as the interface that MacBook Pros communicate with their Touch Bars-- are locally administered. The former are guaranteed to be globally unique, while the latter are demonstrably *not* globally unique and are in fact the same on every MBP with a Touch Bar. With this bit is set, the MAC is locally administered; with it unset it is universally administered.
The other bit is the multicast bit (least significant bit of the first octet). When no other MAC address can be found, RFC 4122 mandates that a random 48-bit number be generated. This randomly generated number *must* have the multicast bit set.
The improvements in uuid.py include:
* Preferentially return a universally administered MAC address, falling back to a locally administered address if none of the former can be found.
* Improve several coding style issues, such as adding explicit returns of None, using a more readable bitmask pattern, and assuming that the ultimate fallback, random MAC generation will not fail (and propagating any exception there instead of swallowing them).
Improvements in test_uuid.py include:
* Always testing the calculated MAC for universal administration, unless explicitly disabled (i.e. for the random case), or implicitly disabled due to running in the Travis environment. Travis test machines have *no* universally administered MAC address at the time of this writing.
The warnings module doesn't leak memory anymore in the hidden
warnings registry for the "ignore" action of warnings filters.
The warn_explicit() function doesn't add the warning key to the
registry anymore for the "ignore" action.
The NNTP server currently has troubles with SSL, whereas we don't
have the control on this server. This test blocks all CIs, so disable
it until a fix can be found.
The test.support.skip_unless_bind_unix_socket() decorator is used to skip
asyncio tests that fail because the platform lacks a functional bind()
function for unix domain sockets (as it is the case for non root users on the
recent Android versions that run now SELinux in enforcing mode).
bpo-32096, bpo-30860: Partially revert the commit
2ebc5ce42a8a9e047e790aefbf9a94811569b2b6:
* Move structures back from Include/internal/mem.h to
Objects/obmalloc.c
* Remove _PyObject_Initialize() and _PyMem_Initialize()
* Remove Include/internal/pymalloc.h
* Add test_capi.test_pre_initialization_api():
Make sure that it's possible to call Py_DecodeLocale(), and then call
Py_SetProgramName() with the decoded string, before Py_Initialize().
PyMem_RawMalloc() and Py_DecodeLocale() can be called again before
_PyRuntimeState_Init().
Co-Authored-By: Eric Snow <ericsnowcurrently@gmail.com>
Previously, 'msilib.OpenDatabase()' function raised a
cryptical exception message when it couldn't open or
create an MSI file. For example:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
_msi.MSIError: unknown error 6e
Adds a simpler and faster alternative to ExitStack for handling
single optional context managers without having to change the
lexical structure of your code.
* Rename support._match_test() to support.match_test(): make it
public
* Remove support.match_tests global variable. It is replaced with a
new support.set_match_tests() function, so match_test() doesn't
have to check each time if patterns were modified.
* Rewrite match_test(): use different code paths depending on the
kind of patterns for best performances.
Co-Authored-By: Serhiy Storchaka <storchaka@gmail.com>
The developer mode (-X dev) now creates all default warnings filters
to order filters in the correct order to always show ResourceWarning
and make BytesWarning depend on the -b option.
Write a functional test to make sure that ResourceWarning is logged
twice at the same location in the developer mode.
Add a new 'dev_mode' field to _PyCoreConfig.
Modify subprocess._args_from_interpreter_flags() to handle -X dev
option.
Add also unit tests for test.support.args_from_interpreter_flags()
and test.support.optim_args_from_interpreter_flags().
When Python is build is debug mode (Py_DEBUG), DeprecationWarning,
PendingDeprecationWarning and ImportWarning warnings are now
displayed by default.
test_venv: run "-m pip" and "-m ensurepip._uninstall" with -W
ignore::DeprecationWarning since pip code is not part of Python.
Add a new "developer mode": new "-X dev" command line option to
enable debug checks at runtime.
Changes:
* Add unit tests for -X dev
* test_cmd_line: replace test.support with support.
* Fix _PyRuntimeState_Fini(): Use the same memory allocator
than _PyRuntimeState_Init().
* Fix _PyMem_GetDefaultRawAllocator()
Parse more env vars in Py_Main():
* Add more options to _PyCoreConfig:
* faulthandler
* tracemalloc
* importtime
* Move code to parse environment variables from _Py_InitializeCore()
to Py_Main(). This change fixes a regression from Python 3.6:
PYTHONUNBUFFERED is now read before calling pymain_init_stdio().
* _PyFaulthandler_Init() and _PyTraceMalloc_Init() now take an
argument to decide if the module has to be enabled at startup.
* tracemalloc_start() is now responsible to check the maximum number
of frames.
Other changes:
* Cleanup Py_Main():
* Rename some pymain_xxx() subfunctions
* Add pymain_run_python() subfunction
* Cleanup Py_NewInterpreter()
* _PyInterpreterState_Enable() now reports failure
* init_hash_secret() now considers pyurandom() failure as an "user
error": don't fail with abort().
* pymain_optlist_append() and pymain_strdup() now sets err on memory
allocation failure.
* Setting sys.tracebacklimit to 0 or less now suppresses printing tracebacks.
* Setting sys.tracebacklimit to None now causes using the default limit.
* Setting sys.tracebacklimit to an integer larger than LONG_MAX now means using
the limit LONG_MAX rather than the default limit.
* Fixed integer overflows in the case of more than 2**31 traceback items on
Windows.
* Fixed output errors handling.
The openfp functions of aifp, sunau, and wave had pointed to the open
function of each module since 1993 as a matter of backwards
compatibility. In the case of aifc.openfp, it was both undocumented
and untested. This change begins the formal deprecation of those
openfp functions, with their removal coming in 3.9.
This additionally adds a TODO in test_pyclbr around using aifc.openfp,
though it shouldn't be changed until removal in 3.9.
kB (*kilo* byte) unit means 1000 bytes, whereas KiB ("kibibyte")
means 1024 bytes. KB was misused: replace kB or KB with KiB when
appropriate.
Same change for MB and GB which become MiB and GiB.
Change the output of Tools/iobench/iobench.py.
Round also the size of the documentation from 5.5 MB to 5 MiB.
blocksize was hardcoded to 8192, preventing efficient upload when using
file-like body. Add blocksize argument to __init__, so users can
configure the blocksize to fit their needs.
I tested this uploading data from /dev/zero to a web server dropping the
received data, to test the overhead of the HTTPConnection.send() with a
file-like object.
Here is an example 10g upload with the default buffer size (8192):
$ time ~/src/cpython/release/python upload-httplib.py 10 https://localhost:8000/
Uploaded 10.00g in 17.53 seconds (584.00m/s)
real 0m17.574s
user 0m8.887s
sys 0m5.971s
Same with 512k blocksize:
$ time ~/src/cpython/release/python upload-httplib.py 10 https://localhost:8000/
Uploaded 10.00g in 6.60 seconds (1551.15m/s)
real 0m6.641s
user 0m3.426s
sys 0m2.162s
In real world usage the difference will be smaller, depending on the
local and remote storage and the network.
See https://github.com/nirs/http-bench for more info.
All Blake2 params have to be encoded in little-endian byte order. For
the two multi-byte integer params, leaf_length and node_offset, that
means that assigning a native-endian integer to them appears to work on
little-endian platforms, but gives the wrong result on big-endian. The
current libb2 API doesn't make that very clear, and @sneves is working
on new API functions in the GH issue above. In the meantime, we can work
around the problem by explicitly assigning little-endian values to the
parameter block.
See https://github.com/BLAKE2/libb2/issues/12.
* bpo-31310: multiprocessing's semaphore tracker should be launched again if crashed
* Avoid mucking with process state in test.
Add a warning if the semaphore process died, as semaphores may then be leaked.
* Add NEWS entry
* bpo-31308: If multiprocessing's forkserver dies, launch it again when necessary.
* Fix test on Windows
* Add NEWS entry
* Adopt a different approach: ignore SIGINT and SIGTERM, as in semaphore tracker.
* Fix comment
* Make sure the test doesn't muck with process state
* Also test previously-started processes
* Update 2017-08-30-17-59-36.bpo-31308.KbexyC.rst
* Avoid masking SIGTERM in forkserver. It's not necessary and causes a race condition in test_many_processes.
Add new time functions:
* time.clock_gettime_ns()
* time.clock_settime_ns()
* time.monotonic_ns()
* time.perf_counter_ns()
* time.process_time_ns()
* time.time_ns()
Add new _PyTime functions:
* _PyTime_FromTimespec()
* _PyTime_FromNanosecondsObject()
* _PyTime_FromTimeval()
Other changes:
* Add also os.times() tests to test_os.
* pytime_fromtimeval() and pytime_fromtimeval() now return
_PyTime_MAX or _PyTime_MIN on overflow, rather than undefined
behaviour
* _PyTime_FromNanoseconds() parameter type changes from long long to
_PyTime_t
test_curses now saves/restores signals. On FreeBSD, the curses module
sets handlers of some signals, but don't restore old handlers when
the module is deinitialized.
The startup refactoring means command line settings
are now applied after settings are read from the
environment.
This updates the way command line settings are applied
to account for that, ensures more settings are first read
from the environment in _PyInitializeCore, and adds a
simple test case covering the flags that are easy to check.
Calendar.itermonthdates() will now consistently raise an exception when a date falls outside of the 0001-01-01 through 9999-12-31 range. To support applications that cannot tolerate such exceptions, the new methods itermonthdays3() and itermonthdays4() are added. The new methods return tuples and are not restricted by the range supported by datetime.date.
Thanks @serhiy-storchaka for suggesting the itermonthdays4() method and for the review.
Improve human friendliness of the Popen API: Add text=False as a
keyword-only argument to subprocess.Popen along with a Popen
attribute .text_mode and set this based on the
encoding/errors/universal_newlines/text arguments.
The universal_newlines parameter and attribute are maintained for
backwards compatibility.
This used to be the case on Python 2. Commit
212b590e11 changed the implementation for Python
3, making the `log()` method of LogAdapter call `logger._log()` directly. This
makes nested log adapters not execute their ``process()`` method. This patch
fixes the issue.
Also, now proxying `name`, too, to make `repr()` work with nested log adapters.
New tests added.
bpo-31803: time.clock() and time.get_clock_info('clock') now emit a
DeprecationWarning warning.
Replace time.clock() with time.perf_counter() in tests and demos.
Remove also hasattr(time, 'monotonic') in test_time since time.monotonic()
is now always available since Python 3.5.
Always pass -1, or INFTIM where defined, to the poll() system call when
a negative timeout is passed to the poll.poll([timeout]) method in the
select module. Various OSes throw an error with arbitrary negative
values.
Freeze all the objects tracked by gc - move them to a permanent generation
and ignore all the future collections. This can be used before a POSIX
fork() call to make the gc copy-on-write friendly or to speed up collection.
When regrtest in run inside IDLE, sys.stdout and sys.stderr are not
TextIOWrapper objects and have no file descriptor associated:
sys.stderr.fileno() raises io.UnsupportedOperation.
Disable faulthandler and don't replace sys.stdout in that case.
Pattern `[a-z]` with `IGNORECASE` flag can match to some non-ASCII characters.
Straightforward solution for this is using `IGNORECASE | ASCII` flag.
But users may subclass `Template` and override only `idpattern`. So we want to
avoid changing `Template.flags`.
So this commit uses local flag `-i` for `idpattern` and change `[a-z]` to `[a-zA-Z]`.
With the upgrade of buildbot.python.org from Buildbot 0.8.x to 0.9.x,
the xmlrpc interface has been removed. This test is now skipped until
it can be rewritten to query a suitable substitute.
Fix test_exception_errpipe_bad_data() and
test_exception_errpipe_normal() of test_subprocess: mock os.waitpid()
to avoid calling the real os.waitpid(0, 0) which is an unexpected
side effect of the test.
Update the compile tests for optimization levels to also check that
__debug__ blocks are included or excluded based on the optimization
level.
Patch by Diana Clarke.
* bpo-31158: Fix nondeterministic read in test_pty
* Reuse existing readline implementation from io.
Thx to @pitrou
* Updated comment
Ideally, this commit is fixuped into the previous commit. Since there is
already a comment on github, I won't rebase.
This was possible before. GH-1560 introduced a regression after 3.6.2 got
released where only sequences were accepted now. This commit addresses this
problem.
While a rare potential failure (it requires swapping out zlib.decompress() itself and forcing it to return a non-bytes object), this change prevents a potential C-level assertion failure and instead substitutes it with an exception.
Thanks to Oren Milman for the patch.
Class execution requires that __prepare__() methods return
a proper execution namespace. Check for that immediately
after calling __prepare__(), rather than passing it through
to the code execution machinery and potentially triggering
SystemError (in debug builds) or a cryptic TypeError
(in release builds).
Patch by Oren Milman.
* bpo-31479: Always reset the signal alarm in tests
Use "try: ... finally: signal.signal(0)" pattern to make sure that
tests don't "leak" a pending fatal signal alarm.
* Move two more alarm() calls into the try block
Fix also typo: replace signal.signal(0) with signal.alarm(0)
* Move another signal.alarm() into the try block
* bpo-31499, xml.etree: Fix xmlparser_gc_clear() crash
xml.etree: xmlparser_gc_clear() now sets self.parser to NULL to prevent a
crash in xmlparser_dealloc() if xmlparser_gc_clear() was called previously
by the garbage collector, because the parser was part of a reference cycle.
Co-Authored-By: Serhiy Storchaka <storchaka@gmail.com>
SSLSocket.wrap_bio() and SSLSocket.wrap_socket() hard-code SSLObject and
SSLSocket as return types. In the light of future deprecation of
ssl.wrap_socket() module function and direct instantiation of SSLSocket,
it is desirable to make the return type of SSLSocket.wrap_bio() and
SSLSocket.wrap_socket() customizable.
Signed-off-by: Christian Heimes <christian@python.org>
* Avoid calling "PyObject_GetAttrString()" (and potentially executing user code) with a live exception set.
* Ignore only AttributeError on attribute lookups in ElementTree.XMLParser() and propagate all other exceptions.
This makes the default behavior (without specifying `globalns` manually) more
predictable for users, finds the right globalns automatically.
Implementation for classes assumes has a `__module__` attribute and that module
is present in `sys.modules`. It does this recursively for all bases in the
MRO. For modules, the implementation just uses their `__dict__` directly.
This is backwards compatible, will just raise fewer exceptions in naive user
code.
Originally implemented and reviewed at https://github.com/python/typing/pull/470.
* bpo-31234: Join threads in test_hashlib
Use thread.join() to wait until the parallel hash tasks complete
rather than using events. Calling thread.join() prevent "dangling
thread" warnings.
* test_hashlib: minor PEP 8 coding style fixes
Some of the proxied methods use internal Logger state which isn't proxied,
causing failures if an adapter is applied to another adapter.
This commit fixes the issue, adds a new test for the use case.
Try to fix the "Warning -- threading_cleanup() failed to cleanup 1
threads" warning in test.lock_tests: wait a little bit longer to give
time to the threads to complete.
Warning seen on test_thread and test_importlib.
support.threading_cleanup() waits for 1 second before emitting a
warning if there are threads running in the background. With this
change, it now emits the warning immediately, to be able to catch
bugs more easily.
Łukasz Langa
Yury Selivanov
Sanyam Khurana
Benjamin Peterson
Andrew Svetlov
Garrett Berg
Pablo Galindo
Eric V. Smith
Ned Deily
Ivan Levkivskyi
Serhiy Storchaka
R. David Murray
Nick Coghlan
Neil Schemenauer
xdegaye
Victor Stinner
Jesse-Bakker
Barry Warsaw
Mandeep Singh
Berker Peksag
Riccardo Magliocchetti
Antoine Pitrou
Dong-hee Na
Andrey Egorov
benfogle
Brian Curtin
Paul Ganssle
Suren Nihalani
Marat Sharafutdinov
Nir Soffer
luzpaz
Jack O'Connor
Vincent Michel
jimmylai
Quentin Dawans
Terry Jan Reedy
Alexander Belopolsky
Vitor Pereira
Mario Corchero
Thomas Kluyver
andyclegg
Mark Shannon
Cheryl Sabella
Oren Milman
jlacoline
Zane Bitter
Riccardo Coccioli
brainfvck
INADA Naoki
Éric Araujo
Zachary Ware
Jelle Zijlstra
Victor K
pdox
diana
Thomas Moreau
Cornelius Diekmann
scoder
James Sexton
Zhiming Wang
Igor Filatov
Anthony Sottile
Raymond Hettinger
Christian Heimes
Feanil Patel