Improve stability of the proactor event loop, especially operations on
overlapped objects:
* Tulip issue 195: Don't call UnregisterWait() twice if a _WaitHandleFuture is
cancelled twice to fix a crash.
* IocpProactor.close(): cancel futures to cancel overlapped operations, instead
of cancelling directly overlapped operations. Future objects may not call
ov.cancel() if the future was cancelled or if the overlapped was already
cancelled. The cancel() method of the future may also catch exceptions. Log
also errors on cancellation.
* tests: rename "f" to "fut"
* Add a __repr__() method to IocpProactor
* Add a destructor to IocpProactor which closes it
* _OverlappedFuture.cancel() doesn't cancel the overlapped anymore if it is
done: if it is already cancelled or completed. Log also an error if the
cancellation failed.
* Add the address of the overlapped object in repr(_OverlappedFuture)
* _OverlappedFuture truncates the source traceback to hide the call to the
parent constructor (useless in debug).
Improve stability of the proactor event loop, especially operations on
overlapped objects:
* Tulip issue 195: Don't call UnregisterWait() twice if a _WaitHandleFuture is
cancelled twice to fix a crash.
* IocpProactor.close(): cancel futures to cancel overlapped operations, instead
of cancelling directly overlapped operations. Future objects may not call
ov.cancel() if the future was cancelled or if the overlapped was already
cancelled. The cancel() method of the future may also catch exceptions. Log
also errors on cancellation.
* tests: rename "f" to "fut"
* Add a __repr__() method to IocpProactor
* Add a destructor to IocpProactor which closes it
* _OverlappedFuture.cancel() doesn't cancel the overlapped anymore if it is
done: if it is already cancelled or completed. Log also an error if the
cancellation failed.
* Add the address of the overlapped object in repr(_OverlappedFuture)
* _OverlappedFuture truncates the source traceback to hide the call to the
parent constructor (useless in debug).
ignores BlockingIOError exceptions. Initial patch written by Xavier de Gaye.
Document also in asyncore documentation that recv() may raise BlockingIOError.
not a terminal to not write the ANSI sequence "\033[1034h" into stdout. This
sequence is used on some terminal (ex: TERM=xterm-256color") to enable support
of 8 bit characters.
terminal to not write the ANSI sequence "\033[1034h" into stdout. This sequence
is used on some terminal (ex: TERM=xterm-256color") to enable support of 8 bit
characters.
lines or narrowed for small screens.
Issie #19132: Turtledemo buttons no longer disappear when window is shrun.
Patch mostly by Lita Cho (21597) using idea from patch by Jan Kanis (18132).
Adds `load_package_tests` function to test.support, uses it in test_asyncio,
test_email, test_json, test_tools, test_importlib and all test_importlib
sub-packages to implement test discovery.
* Issue #19811, #22022: test_pathlib uses support.rmtree() instead of
shutil.rmtree() to remove the temporary directory.
* Issue #19629: Fix support.rmtree(), use os.lstat() to check if the file is a
directory, not os.path.isdir()
- Use _testcapi.raise_signal() in test_signal
- close also os.pipe() file descriptors in some test_signal tests where they
were not closed properly
- Remove faulthandler._sigill() and faulthandler._sigbus(): reuse
_testcapi.raise_signal() in test_faulthandler
Since Python 3.3, the C signal handler writes the signal number into the wakeup
file descriptor and then schedules the Python call using Py_AddPendingCall().
asyncio uses the wakeup file descriptor to wake up the event loop, and relies
on Py_AddPendingCall() to schedule the final callback with call_soon().
If the C signal handler is called in a thread different than the thread of the
event loop, the loop is awaken but Py_AddPendingCall() was not called yet. In
this case, the event loop has nothing to do and go to sleep again.
Py_AddPendingCall() is called while the event loop is sleeping again and so the
final callback is not scheduled immediatly.
This patch changes how asyncio handles signals. Instead of relying on
Py_AddPendingCall() and the wakeup file descriptor, asyncio now only relies on
the wakeup file descriptor. asyncio reads signal numbers from the wakeup file
descriptor to call its signal handler.
Since Python 3.3, the C signal handler writes the signal number into the wakeup
file descriptor and then schedules the Python call using Py_AddPendingCall().
asyncio uses the wakeup file descriptor to wake up the event loop, and relies
on Py_AddPendingCall() to schedule the final callback with call_soon().
If the C signal handler is called in a thread different than the thread of the
event loop, the loop is awaken but Py_AddPendingCall() was not called yet. In
this case, the event loop has nothing to do and go to sleep again.
Py_AddPendingCall() is called while the event loop is sleeping again and so the
final callback is not scheduled immediatly.
This patch changes how asyncio handles signals. Instead of relying on
Py_AddPendingCall() and the wakeup file descriptor, asyncio now only relies on
the wakeup file descriptor. asyncio reads signal numbers from the wakeup file
descriptor to call its signal handler.
BrokenPipeError
If you want to handle the BrokenPipeError, you can easily reimplement
communicate().
Add also a unit test to ensure that stdin.write() + stdin.drain() raises
BrokenPipeError.
If you want to handle the BrokenPipeError, you can easily reimplement
communicate().
Add also a unit test to ensure that stdin.write() + stdin.drain() raises
BrokenPipeError.
* Tulip issue #184: Log subprocess events in debug mode
- Log stdin, stdout and stderr transports and protocols
- Log process identifier (pid)
- Log connection of pipes
- Log process exit
- Log Process.communicate() tasks: feed stdin, read stdout and stderr
- Add __repr__() method to many classes related to subprocesses
* Add BaseSubprocessTransport._pid attribute. Store the pid so it is still
accessible after the process exited. It's more convinient for debug.
* create_connection(): add the socket in the "connected to" debug log
* Clean up some docstrings and comments. Remove unused unimplemented
_read_from_self().
* Tulip issue #183: log socket events in debug mode
- Log most important socket events: socket connected, new client, connection
reset or closed by peer (EOF), etc.
- Log time elapsed in DNS resolution (getaddrinfo)
- Log pause/resume reading
- Log time of SSL handshake
- Log SSL handshake errors
- Add a __repr__() method to many classes
* Fix ProactorEventLoop() in debug mode. ProactorEventLoop._make_self_pipe()
doesn't call call_soon() directly because it checks for the current loop
which fails, because the method is called to build the event loop.
* Cleanup _ProactorReadPipeTransport constructor. Not need to set again
_read_fut attribute to None, it is already done in the base class.
- loop, waiters and active_count attributes are now private
- attach(), detach() and wakeup() methods are now private
The sockets attribute remains public.
:c:type:`int` for the size to support reading more than 2 GB at once. On
Windows, the size is truncted to INT_MAX. As any call to os.read(), the OS
may read less bytes than the number of requested bytes.
* Tulip issue #182: Improve logs of BaseEventLoop._run_once()
- Don't log non-blocking poll
- Only log polling with a timeout if it gets events or if it timed out after
more than 1 second.
* Fix some pyflakes warnings: remove unused imports
* Tulip issue #182: Improve logs of BaseEventLoop._run_once()
- Don't log non-blocking poll
- Only log polling with a timeout if it gets events or if it timed out after
more than 1 second.
* Fix some pyflakes warnings: remove unused imports
- CoroWrapper.__del__() now reuses repr(CoroWrapper) to log the "... was never
yielded from" warning
- Improve CoroWrapper: copy also the qualified name on Python 3.4, not only on
Python 3.5+
- CoroWrapper.__del__() now reuses repr(CoroWrapper) to log the "... was never
yielded from" warning
- Improve CoroWrapper: copy also the qualified name on Python 3.4, not only on
Python 3.5+
- repr(Task) and repr(CoroWrapper) now also includes where these objects were
created. If the coroutine is not a generator (don't use "yield from"), use
the location of the function, not the location of the coro() wrapper.
- Fix create_task(): truncate the traceback to hide the call to create_task().
- repr(Task) and repr(CoroWrapper) now also includes where these objects were
created. If the coroutine is not a generator (don't use "yield from"), use
the location of the function, not the location of the coro() wrapper.
- Fix create_task(): truncate the traceback to hide the call to create_task().
- Issues #21936, #21163: Fix sporadic failures of
test_future_exception_never_retrieved()
- Handle.cancel() now clears references to callback and args
- In debug mode, repr(Handle) now contains the location where the Handle was
created.
- Issues #21936, #21163: Fix sporadic failures of
test_future_exception_never_retrieved()
- Handle.cancel() now clears references to callback and args
- In debug mode, repr(Handle) now contains the location where the Handle was
created.
asyncio: sync with Tulip
- Tulip issue 185: Add a create_task() method to event loops. The create_task()
method can be overriden in custom event loop to implement their own task
class. For example, greenio and Pulsar projects use their own task class. The
create_task() method is now preferred over creating directly task using the
Task class.
- tests: fix a warning
- fix typo in the name of a test function
- Update AbstractEventLoop: add new event loop methods; update also the unit test
Update asyncio documentation
- Document the new create_task() method
- "Hide" the Task class: point to the create_task() method for interoperability
- Rewrite the documentation of the Task class
- Document the "Pending task destroyed"
- Update output in debug mode of examples in the dev section
- Replace Task() with create_task() in examples
- Tulip issue 185: Add a create_task() method to event loops. The create_task()
method can be overriden in custom event loop to implement their own task
class. For example, greenio and Pulsar projects use their own task class. The
create_task() method is now preferred over creating directly task using the
Task class.
- tests: fix a warning
- fix typo in the name of a test function
- Update AbstractEventLoop: add new event loop methods; update also the unit test
- Tulip issue #181: Faster create_connection(). Call directly
waiter.set_result() in the constructor of _ProactorBasePipeTransport and
_SelectorSocketTransport, instead of using of delaying the call with
call_soon().
- Cleanup iscoroutine()
- Tulip issue #181: Faster create_connection(). Call directly
waiter.set_result() in the constructor of _ProactorBasePipeTransport and
_SelectorSocketTransport, instead of using of delaying the call with
call_soon().
- Cleanup iscoroutine()
When ``None`` was passed as a ``REG_BINARY`` value to SetValueEx,
PyMem_DEL was called on an uninitialized buffer. Patch by John Ehresman.
(Also an incidental typo fix in a comment in test_winreg)
* _UnixSubprocessTransport: fix file mode of stdin. Open stdin in write mode,
not in read mode
* Examples: close the event loop at exit
* More reliable CoroWrapper.__del__. If the constructor is interrupted by
KeyboardInterrupt or the coroutine objet is destroyed lately, some the
_source_traceback attribute doesn't exist anymore.
* repr(Task): include also the future the task is waiting for
* _UnixSubprocessTransport: fix file mode of stdin. Open stdin in write mode,
not in read mode
* Examples: close the event loop at exit
* More reliable CoroWrapper.__del__. If the constructor is interrupted by
KeyboardInterrupt or the coroutine objet is destroyed lately, some the
_source_traceback attribute doesn't exist anymore.
* repr(Task): include also the future the task is waiting for
to remove duplication and return info for tests. Rewrite corresponding tests.
Test_create_option_buttons was not testing anything because of buggy
comparisons. Use Python subscripting to get widget options.
test_utils.run_briefly() don't log the "destroy pending task" message anymore.
The log is redundant for run_until_complete() and useless in run_briefly().
- Sort imports
- Simplify/optimize iscoroutine(). Inline inspect.isgenerator(obj): replace it
with isinstance(obj, types.GeneratorType)
- CoroWrapper: check at runtime if Python has the yield-from bug #21209. If
Python has the bug, check if CoroWrapper.send() was called by yield-from to
decide if parameters must be unpacked or not.
- Fix "Task was destroyed but it is pending!" warning in
test_task_source_traceback()
- Sort imports
- Simplify/optimize iscoroutine(). Inline inspect.isgenerator(obj): replace it
with isinstance(obj, types.GeneratorType)
- CoroWrapper: check at runtime if Python has the yield-from bug #21209. If
Python has the bug, check if CoroWrapper.send() was called by yield-from to
decide if parameters must be unpacked or not.
- Fix "Task was destroyed but it is pending!" warning in
test_task_source_traceback()
file_wrapper object is destroyed. The destructor now closes the file if needed.
The close() method can now be called twice: the second call does nothing.
Future, Task and Handle objects are created. Pass the traceback to
call_exception_handler() in the 'source_traceback' key.
The traceback is truncated to hide internal calls in asyncio, show only the
traceback from user code.
Add tests for the new source_traceback, and a test for the 'Future/Task
exception was never retrieved' log.
Handle objects are created. Pass the traceback to call_exception_handler() in
the 'source_traceback' key.
The traceback is truncated to hide internal calls in asyncio, show only the
traceback from user code.
Add tests for the new source_traceback, and a test for the 'Future/Task
exception was never retrieved' log.
- Python issue 21163: Fix more "Task was destroyed but it is pending!" logs in
tests
- Add test to check that run_until_complete() checks the loop of the future
- Python issue 21163: Fix more "Task was destroyed but it is pending!" logs in
tests
- Add test to check that run_until_complete() checks the loop of the future
TimerHandle
- Uniformize repr() output to format "<Class ...>"
- On Python 3.5+, repr(Task) uses the qualified name instead of the short name
of the coroutine
issue20753 - robotparser tests should not rely upon external resource when not required.
Specifically, it was relying a URL which gave HTTP 403 and used it to assert
it's methods, this changes undoes that and provides a local http server with
similar properties.
Patch contributed by Vajrasky Kok.
Specifically, it was relying a URL which gave HTTP 403 and used it to assert
it's methods, this changes undoes that and provides a local http server with
similar properties.
Patch contributed by Vajrasky Kok.
issue15588 - Correct the quopri module documentation. Mention the correct types
of the parameters on encodestring and decodestring.
Patch by Petri Lehtinen.
coroutine is done: use the first line number of the code object instead of the
current line number of the generator frame.
The name of the coroutine is not enough because many coroutines may have the
same name. It's a common case in asyncio tests for example.
done: use the first line number of the code object instead of the current line
number of the generator frame.
The name of the coroutine is not enough because many coroutines may have the
same name. It's a common case in asyncio tests for example.
raises an exception if the current loop is not None.
Guido van Rossum wrote:
"The behavior that you can set the loop to None (and keep track of it
explicitly) is part of the spec, and this should still be supported even in
debug mode. The behavior that we raise an error if you are caught having
multiple active loops per thread is just a debugging heuristic, and it
shouldn't break code that follows the spec."
exception if the current loop is not None.
Guido van Rossum wrote:
"The behavior that you can set the loop to None (and keep track of it
explicitly) is part of the spec, and this should still be supported even in
debug mode. The behavior that we raise an error if you are caught having
multiple active loops per thread is just a debugging heuristic, and it
shouldn't break code that follows the spec."
reads all available bytes from the "self pipe", not only a single byte. This
change reduces the risk of having the pipe full and so getting the innocuous
"BlockingIOError: [Errno 11] Resource temporarily unavailable" message.
available bytes from the "self pipe", not only a single byte. This change
reduces the risk of having the pipe full and so getting the innocuous
"BlockingIOError: [Errno 11] Resource temporarily unavailable" message.
decorator on Python 3.5.
- Drop __slots__ optimization of CoroWrapper to be able to set the __qualname__
attribute.
- Add tests on __name__, __qualname__ and __module__ of a coroutine function
and coroutine object.
- Fix test_tasks when run in debug mode (PYTHONASYNCIODEBUG env var set) on
Python 3.3 or 3.4
Python 3.5
- Drop __slots__ optimization of CoroWrapper to be able to set the __qualname__
attribute.
- Add tests on __name__, __qualname__ and __module__ of a coroutine function
and coroutine object.
- Fix test_tasks when run in debug mode (PYTHONASYNCIODEBUG env var set) on
Python 3.3 or 3.4
On Python 3.5, generator now gets their name from the function, no more from
the code. So we get the expected "notmuch" name instead of the generic "coro"
name.
On Python 3.5, generator now gets their name from the function, no more from
the code. So we get the expected "notmuch" name instead of the generic "coro"
name.
name, and use it in the representation of a generator (``repr(gen)``). The
default name of the generator (``__name__`` attribute) is now get from the
function instead of the code. Use ``gen.gi_code.co_name`` to get the name of
the code.
Clean-up names of static functions.
Use Py_RETURN_NONE macro.
Expose private functions needed to support merge().
Move C imports to the bottom of the Python file.
Skipped tests are now marked as skipped, formerly commented-out or
renamed-so-it-doesn't-look-like-a-test tests are uncommented, properly named,
and unconditionally skipped, some tests that simply didn't run before
are now able to run, and a few are split into multiple methods instead of
skipping via 'return' in the middle of a method. Also, a couple of unused
files are removed completely.
repr(Handle) is shorter for function: "foo" instead of "<function foo at
0x...>". It now also includes the source of the callback, filename and line
number where it was defined, if available.
repr(Task) now also includes the current position in the code, filename and
line number, if available. If the coroutine (generator) is done, the line
number is omitted and "done" is added.
repr(Handle) is shorter for function: "foo" instead of "<function foo at
0x...>". It now also includes the source of the callback, filename and line
number where it was defined, if available.
repr(Task) now also includes the current position in the code, filename and
line number, if available. If the coroutine (generator) is done, the line
number is omitted and "done" is added.
Otherwise smtpd is restricted to 7bit clean data, since even if the
incoming data is actually utf-8, it will often break things to decode
it before parsing the message.
Patch by Maciej Szulik, with some adjustments (mostly the warning
support).
Add BaseEventLoop._closed attribute and use it to check if the event loop was
closed or not, instead of checking different attributes in each subclass of
BaseEventLoop.
run_forever() and run_until_complete() methods now raise a RuntimeError('Event loop is
closed') exception if the event loop was closed.
BaseProactorEventLoop.close() now also cancels "accept futures".
Add BaseEventLoop._closed attribute and use it to check if the event loop was
closed or not, instead of checking different attributes in each subclass of
BaseEventLoop.
run_forever() and run_until_complete() now raises a RuntimeError('Event loop is
closed') exception if the event loop was closed.
BaseProactorEventLoop.close() now also cancels "accept futures".
* Make sure that socketpair() close sockets on error. Close the listening
socket if sock.bind() raises an exception.
* asyncio now closes sockets on errors. Fix ResourceWarning:
create_connection(), create_datagram_endpoint() and create_unix_server()
methods of event loop now close the newly created socket on error.
Fix ResourceWarning: create_connection(), create_datagram_endpoint() and
create_unix_server() methods of event loop now close the newly created socket
on error.
regression was introduced recently with the introduction of the new "calloc"
functions (PyMem_RawCalloc, PyMem_Calloc, PyObject_Calloc).
Add also a unit test to check for the non-regression.
when we have a way to get an actual list of all open fds from the OS.
Fixes issue #21618: The subprocess module would ignore fds that were
inherited by the calling process and already higher than POSIX resource
limits would otherwise allow. On systems with a functioning /proc/self/fd
or /dev/fd interface the max is now ignored and all fds are closed.
when we have a way to get an actual list of all open fds from the OS.
Fixes issue #21618: The subprocess module would ignore fds that were
inherited by the calling process and already higher than POSIX resource
limits would otherwise allow. On systems with a functioning /proc/self/fd
or /dev/fd interface the max is now ignored and all fds are closed.
Along the way, dismantle importlib._bootstrap._SpecMethods as it was
no longer relevant and constructing the new function required
partially dismantling the class anyway.
directory that are too short to be parsed instead of letting a struct.unpack
error bubble up as this "bad data" appears in many real world zip files in the
wild and is ignored by other zip tools.
directory that are too short to be parsed instead of letting a struct.unpack
error bubble up as this "bad data" appears in many real world zip files in the
wild and is ignored by other zip tools.
Terry Jan Reedy
Victor Stinner
Zachary Ware
Serhiy Storchaka
Charles-François Natali
Senthil Kumaran
Antoine Pitrou
Raymond Hettinger
Tal Einat
Guido van Rossum
Ned Deily
Alexander Belopolsky
Berker Peksag
Benjamin Peterson
R David Murray
Jason R. Coombs
Jesus Cea
Yury Selivanov
Giampaolo Rodola'
Nick Coghlan
Gregory P. Smith
Vinay Sajip
Kushal Das
Barry Warsaw
Brett Cannon