blocksize was hardcoded to 8192, preventing efficient upload when using
file-like body. Add blocksize argument to __init__, so users can
configure the blocksize to fit their needs.
I tested this uploading data from /dev/zero to a web server dropping the
received data, to test the overhead of the HTTPConnection.send() with a
file-like object.
Here is an example 10g upload with the default buffer size (8192):
$ time ~/src/cpython/release/python upload-httplib.py 10 https://localhost:8000/
Uploaded 10.00g in 17.53 seconds (584.00m/s)
real 0m17.574s
user 0m8.887s
sys 0m5.971s
Same with 512k blocksize:
$ time ~/src/cpython/release/python upload-httplib.py 10 https://localhost:8000/
Uploaded 10.00g in 6.60 seconds (1551.15m/s)
real 0m6.641s
user 0m3.426s
sys 0m2.162s
In real world usage the difference will be smaller, depending on the
local and remote storage and the network.
See https://github.com/nirs/http-bench for more info.
All Blake2 params have to be encoded in little-endian byte order. For
the two multi-byte integer params, leaf_length and node_offset, that
means that assigning a native-endian integer to them appears to work on
little-endian platforms, but gives the wrong result on big-endian. The
current libb2 API doesn't make that very clear, and @sneves is working
on new API functions in the GH issue above. In the meantime, we can work
around the problem by explicitly assigning little-endian values to the
parameter block.
See https://github.com/BLAKE2/libb2/issues/12.
* bpo-31310: multiprocessing's semaphore tracker should be launched again if crashed
* Avoid mucking with process state in test.
Add a warning if the semaphore process died, as semaphores may then be leaked.
* Add NEWS entry
* bpo-31308: If multiprocessing's forkserver dies, launch it again when necessary.
* Fix test on Windows
* Add NEWS entry
* Adopt a different approach: ignore SIGINT and SIGTERM, as in semaphore tracker.
* Fix comment
* Make sure the test doesn't muck with process state
* Also test previously-started processes
* Update 2017-08-30-17-59-36.bpo-31308.KbexyC.rst
* Avoid masking SIGTERM in forkserver. It's not necessary and causes a race condition in test_many_processes.
Add new time functions:
* time.clock_gettime_ns()
* time.clock_settime_ns()
* time.monotonic_ns()
* time.perf_counter_ns()
* time.process_time_ns()
* time.time_ns()
Add new _PyTime functions:
* _PyTime_FromTimespec()
* _PyTime_FromNanosecondsObject()
* _PyTime_FromTimeval()
Other changes:
* Add also os.times() tests to test_os.
* pytime_fromtimeval() and pytime_fromtimeval() now return
_PyTime_MAX or _PyTime_MIN on overflow, rather than undefined
behaviour
* _PyTime_FromNanoseconds() parameter type changes from long long to
_PyTime_t
test_curses now saves/restores signals. On FreeBSD, the curses module
sets handlers of some signals, but don't restore old handlers when
the module is deinitialized.
Editor and output windows only see an empty last prompt line.
This simplifies the code and fixes a minor bug when newline is inserted.
Sys.ps1, if present, is read on Shell start-up, but is not set or changed.
The startup refactoring means command line settings
are now applied after settings are read from the
environment.
This updates the way command line settings are applied
to account for that, ensures more settings are first read
from the environment in _PyInitializeCore, and adds a
simple test case covering the flags that are easy to check.
Calendar.itermonthdates() will now consistently raise an exception when a date falls outside of the 0001-01-01 through 9999-12-31 range. To support applications that cannot tolerate such exceptions, the new methods itermonthdays3() and itermonthdays4() are added. The new methods return tuples and are not restricted by the range supported by datetime.date.
Thanks @serhiy-storchaka for suggesting the itermonthdays4() method and for the review.
Improve human friendliness of the Popen API: Add text=False as a
keyword-only argument to subprocess.Popen along with a Popen
attribute .text_mode and set this based on the
encoding/errors/universal_newlines/text arguments.
The universal_newlines parameter and attribute are maintained for
backwards compatibility.
This used to be the case on Python 2. Commit
212b590e11 changed the implementation for Python
3, making the `log()` method of LogAdapter call `logger._log()` directly. This
makes nested log adapters not execute their ``process()`` method. This patch
fixes the issue.
Also, now proxying `name`, too, to make `repr()` work with nested log adapters.
New tests added.
Even if one selects a font that defines a limited subset of the unicode
Basic Multilingual Plane, tcl/tk will use other fonts that define a
character. The expanded example give users of non-Latin characters
a better idea of what they might see in the IDLE shell and editors.
To make room for the expanded sample, frames on the Font tab are
re-arranged. The Font/Tabs help explains a bit about the additions.
bpo-31803: time.clock() and time.get_clock_info('clock') now emit a
DeprecationWarning warning.
Replace time.clock() with time.perf_counter() in tests and demos.
Remove also hasattr(time, 'monotonic') in test_time since time.monotonic()
is now always available since Python 3.5.
Always pass -1, or INFTIM where defined, to the poll() system call when
a negative timeout is passed to the poll.poll([timeout]) method in the
select module. Various OSes throw an error with arbitrary negative
values.
The new method allows the developer to control when to stop the
feature of mocks that automagically creates new mocks when accessing
an attribute that was not declared before
Signed-off-by: Mario Corchero <mariocj89@gmail.com>
Freeze all the objects tracked by gc - move them to a permanent generation
and ignore all the future collections. This can be used before a POSIX
fork() call to make the gc copy-on-write friendly or to speed up collection.
When regrtest in run inside IDLE, sys.stdout and sys.stderr are not
TextIOWrapper objects and have no file descriptor associated:
sys.stderr.fileno() raises io.UnsupportedOperation.
Disable faulthandler and don't replace sys.stdout in that case.
Pattern `[a-z]` with `IGNORECASE` flag can match to some non-ASCII characters.
Straightforward solution for this is using `IGNORECASE | ASCII` flag.
But users may subclass `Template` and override only `idpattern`. So we want to
avoid changing `Template.flags`.
So this commit uses local flag `-i` for `idpattern` and change `[a-z]` to `[a-zA-Z]`.
With the upgrade of buildbot.python.org from Buildbot 0.8.x to 0.9.x,
the xmlrpc interface has been removed. This test is now skipped until
it can be rewritten to query a suitable substitute.
Fix test_exception_errpipe_bad_data() and
test_exception_errpipe_normal() of test_subprocess: mock os.waitpid()
to avoid calling the real os.waitpid(0, 0) which is an unexpected
side effect of the test.
sre_compile does bit test (e.g. `flags & SRE_FLAG_IGNORECASE`) in loop.
`IntFlag.__and__` and `IntFlag.__new__` made it slower.
So this commit convert it to normal int before passing flags to `sre_compile()`.
While there is not real bug in this case, using re.IGNORECASE without re.ASCII
leads unexpected behavior.
Instead of adding re.ASCII, this commit removes re.IGNORECASE flag because
it's easier and simpler.
This commit removes dead copy of the pattern in email.util module too.
While the pattern is same, it is compiled separately because it had different flags.
Update the compile tests for optimization levels to also check that
__debug__ blocks are included or excluded based on the optimization
level.
Patch by Diana Clarke.
* bpo-31158: Fix nondeterministic read in test_pty
* Reuse existing readline implementation from io.
Thx to @pitrou
* Updated comment
Ideally, this commit is fixuped into the previous commit. Since there is
already a comment on github, I won't rebase.
DER_cert_to_PEM_cert() used textwrap.fill() to format PEM.
But it's library to wrap lines on word boundary, while PEM is
base64 encoded string.
Additionally, importing textwrap is little slow.
Passing a widget instead of an flist with a root widget opens the option of
creating a browser frame that is only part of a window. Passing a full file
name instead of pieces assumed to come from a .py file opens the possibility
of browsing python files that do not end in .py.
This was possible before. GH-1560 introduced a regression after 3.6.2 got
released where only sequences were accepted now. This commit addresses this
problem.
While a rare potential failure (it requires swapping out zlib.decompress() itself and forcing it to return a non-bytes object), this change prevents a potential C-level assertion failure and instead substitutes it with an exception.
Thanks to Oren Milman for the patch.
Class execution requires that __prepare__() methods return
a proper execution namespace. Check for that immediately
after calling __prepare__(), rather than passing it through
to the code execution machinery and potentially triggering
SystemError (in debug builds) or a cryptic TypeError
(in release builds).
Patch by Oren Milman.
The original module-level class and method browser became a module
browser, with the addition of module-level functions, years ago.
Nested classes and functions were added yesterday. For back-
compatibility, the virtual event <<open-class-browser>>, which
appears on the Keys tab of the Settings dialog, is not changed.
Patch by Cheryl Sabella.
* Maintain a list of BufferedWriter objects. Flush them on exit.
In Python 3, the buffer and the underlying file object are separate
and so the order in which objects are finalized matters. This is
unlike Python 2 where the file and buffer were a single object and
finalization was done for both at the same time. In Python 3, if
the file is finalized and closed before the buffer then the data in
the buffer is lost.
This change adds a doubly linked list of open file buffers. An atexit
hook ensures they are flushed before proceeding with interpreter
shutdown. This is addition does not remove the need to properly close
files as there are other reasons why buffered data could get lost during
finalization.
Initial patch by Armin Rigo.
* Use weakref.WeakSet instead of WeakKeyDictionary.
* Simplify buffered double-linked list types.
* In _flush_all_writers(), suppress errors from flush().
* Remove NEWS entry, use blurb.
* Take more care when flushing file buffers from atexit.
The previous implementation was not careful enough to avoid
causing issues in multi-threaded cases. Check for buf->ok
and buf->finalizing before actually doing the flush. Also,
increase the refcnt to ensure the object does not disappear.
* bpo-31479: Always reset the signal alarm in tests
Use "try: ... finally: signal.signal(0)" pattern to make sure that
tests don't "leak" a pending fatal signal alarm.
* Move two more alarm() calls into the try block
Fix also typo: replace signal.signal(0) with signal.alarm(0)
* Move another signal.alarm() into the try block
* bpo-31499, xml.etree: Fix xmlparser_gc_clear() crash
xml.etree: xmlparser_gc_clear() now sets self.parser to NULL to prevent a
crash in xmlparser_dealloc() if xmlparser_gc_clear() was called previously
by the garbage collector, because the parser was part of a reference cycle.
Co-Authored-By: Serhiy Storchaka <storchaka@gmail.com>
SSLSocket.wrap_bio() and SSLSocket.wrap_socket() hard-code SSLObject and
SSLSocket as return types. In the light of future deprecation of
ssl.wrap_socket() module function and direct instantiation of SSLSocket,
it is desirable to make the return type of SSLSocket.wrap_bio() and
SSLSocket.wrap_socket() customizable.
Signed-off-by: Christian Heimes <christian@python.org>
'Strip trailing whitespace' is not limited to spaces. Wording caters to beginners who
do know know the meaning of 'whitespace'. Multiline string literals are not skipped.
* News blurb.
* Avoid calling "PyObject_GetAttrString()" (and potentially executing user code) with a live exception set.
* Ignore only AttributeError on attribute lookups in ElementTree.XMLParser() and propagate all other exceptions.
This makes the default behavior (without specifying `globalns` manually) more
predictable for users, finds the right globalns automatically.
Implementation for classes assumes has a `__module__` attribute and that module
is present in `sys.modules`. It does this recursively for all bases in the
MRO. For modules, the implementation just uses their `__dict__` directly.
This is backwards compatible, will just raise fewer exceptions in naive user
code.
Originally implemented and reviewed at https://github.com/python/typing/pull/470.
* bpo-31234: Join threads in test_hashlib
Use thread.join() to wait until the parallel hash tasks complete
rather than using events. Calling thread.join() prevent "dangling
thread" warnings.
* test_hashlib: minor PEP 8 coding style fixes
Some of the proxied methods use internal Logger state which isn't proxied,
causing failures if an adapter is applied to another adapter.
This commit fixes the issue, adds a new test for the use case.
Try to fix the "Warning -- threading_cleanup() failed to cleanup 1
threads" warning in test.lock_tests: wait a little bit longer to give
time to the threads to complete.
Warning seen on test_thread and test_importlib.
support.threading_cleanup() waits for 1 second before emitting a
warning if there are threads running in the background. With this
change, it now emits the warning immediately, to be able to catch
bugs more easily.
socketserver.ThreadingMixIn now keeps a list of non-daemonic threads
to wait until all these threads complete in server_close().
Reenable test_logging skipped tests.
Fix SocketHandlerTest.tearDown(): close the socket handler before
stopping the server, so the server can join threads.
The ConfigSocketReceiver.serve_until_stopped() method from
logging.config.listen() now calls server_close() (of
socketserver.ThreadingTCPServer) rather than closing manually the
socket.
While this change has no effect yet, it will help to prevent dangling
threads once ThreadingTCPServer.server_close() will join spawned
threads (bpo-31233).
* test_thread.test_forkinthread() now waits until the thread completes.
* Check the status in the test method, not in the thread function
* Don't ignore RuntimeError anymore: since the commit
346cbd351e (bpo-16500,
os.register_at_fork(), os.fork() cannot fail anymore with
RuntimeError.
* Replace 0.01 literal with a new POLL_SLEEP constant
* test_forkinthread(): test if os.fork() exists rather than testing
the platform.
This undoes a853a8ba78 except for the pytime.c
parts. We want to continue to allow IEEE 754 doubles larger than FLT_MAX to be
rounded into finite floats. Tests were added to very this behavior.
About 10 IDLE features were implemented as supposedly optional
extensions. Their different behavior could be confusing or worse for
users and not good for maintenance. Hence the conversion.
The main difference for users is that user configurable key bindings
for builtin features are now handled uniformly. Now, editing a binding
in a keyset only affects its value in the keyset. All bindings are
defined together in the system-specific default keysets in config-
extensions.def. All custom keysets are saved as a whole in config-
extension.cfg. All take effect as soon as one clicks Apply or Ok.
The affected events are '<<force-open-completions>>', '<<expand-word>>',
'<<force-open-calltip>>', '<<flash-paren>>', '<<format-paragraph>>',
'<<run-module>>', '<<check-module>>', and '<<zoom-height>>'. Any
(global) customizations made before 3.6.3 will not affect their keyset-
specific customization after 3.6.3. and vice versa.
Inital patch by Charles Wohlganger, revised by Terry Jan Reedy.
* Working draft without _source
* Re-use itemgetter() instances
* Speed-up calls to __new__() with a pre-bound tuple.__new__()
* Add note regarding string interning
* Remove unnecessary create function wrappers
* Minor sync-ups with PR-2736. Mostly formatting and f-strings
* Bring-in qualname/__module fix-ups from PR-2736
* Formally remove the verbose flag and _source attribute
* Restore a test of potentially problematic field names
* Restore kwonly_args test but without the verbose option
* Adopt Inada's idea to reuse the docstrings for the itemgetters
* Neaten-up a bit
* Add news blurb
* Serhiy pointed-out the need for interning
* Jelle noticed as missing f on an f-string
* Add whatsnew entry for feature removal
* Accede to request for dict literals instead keyword arguments
* Leave the method.__module__ attribute pointing the actual location of the code
* Improve variable names and add a micro-optimization for an non-public helper function
* Simplify by in-lining reuse_itemgetter()
* Arrange steps in more logical order
* Save docstring in local cache instead of interning
test.support.HOST should be "localhost" as it was in the past. See the bpo-29639.
Tests that need the IP address should use HOSTv4 (added) or the existing HOSTv6 constant.
This changes the definition and fixes tests that needed updating to deal with HOST being
the hostname rather than the hardcoded IP address.
This is only the first step in addressing https://bugs.python.org/issue29639.
The SSL module now raises SSLCertVerificationError when OpenSSL fails to
verify the peer's certificate. The exception contains more information about
the error.
Original patch by Chi Hsuan Yen
Signed-off-by: Christian Heimes <christian@python.org>
* bpo-29136: Add TLS 1.3 support
TLS 1.3 introduces a new, distinct set of cipher suites. The TLS 1.3
cipher suites don't overlap with cipher suites from TLS 1.2 and earlier.
Since Python sets its own set of permitted ciphers, TLS 1.3 handshake
will fail as soon as OpenSSL 1.1.1 is released. Let's enable the common
AES-GCM and ChaCha20 suites.
Additionally the flag OP_NO_TLSv1_3 is added. It defaults to 0 (no op) with
OpenSSL prior to 1.1.1. This allows applications to opt-out from TLS 1.3
now.
Signed-off-by: Christian Heimes <christian@python.org>
f_trace_lines: enable/disable line trace events
f_trace_opcodes: enable/disable opcode trace events
These are intended primarily for testing of the interpreter
itself, as they make it much easier to emulate signals
arriving at unfortunate times.
* bpo-27340: Use memoryview in SSLSocket.sendall()
SSLSocket.sendall() now uses memoryview to create slices of data. This fix
support for all bytes-like object. It is also more efficient and avoids
costly copies.
Signed-off-by: Christian Heimes <christian@python.org>
* Cast view to bytes, fix typo
Signed-off-by: Christian Heimes <christian@python.org>
This adds support for parsing a command line where options and positionals are intermixed as is common in many unix commands. This is paul.j3's patch with a few tweaks.
* bpo-27584: New addition of vSockets to the python socket module
Support for AF_VSOCK on Linux only
* bpo-27584: Fixes for V2
Fixed syntax and naming problems.
Fixed #ifdef AF_VSOCK checking
Restored original aclocal.m4
* bpo-27584: Fixes for V3
Added checking for fcntl and thread modules.
* bpo-27584: Fixes for V4
Fixed white space error
* bpo-27584: Fixes for V5
Added back comma in (CID, port).
* bpo-27584: Fixes for V6
Added news file.
socket.rst now reflects first Linux introduction of AF_VSOCK.
Fixed get_cid in test_socket.py.
Replaced PyLong_FromLong with PyLong_FromUnsignedLong in socketmodule.c
Got rid of extra AF_VSOCK #define.
Added sockaddr_vm to sock_addr.
* bpo-27584: Fixes for V7
Minor cleanup.
* bpo-27584: Fixes for V8
Put back #undef AF_VSOCK as it is necessary when vm_sockets.h is not installed.
Add basic fuzz tests for a few common builtin functions.
This is an easy place to start, and these functions are probably safe.
We'll want to add more fuzz tests later. Lets bootstrap using these.
While the fuzz tests are included in CPython and compiled / tested on a
very basic level inside CPython itself, the actual fuzzing happens as
part of oss-fuzz (https://github.com/google/oss-fuzz). The reason to
include the tests in CPython is to make sure that they're maintained
as part of the CPython project, especially when (as some eventually
will) they use internal implementation details in the test.
(This will be necessary sometimes because e.g. the fuzz test should
never enter Python's interpreter loop, whereas some APIs only expose
themselves publicly as Python functions.)
This particular set of changes is part of testing Python's builtins,
tracked internally at Google by b/37562550.
The _xxtestfuzz module that this change adds need not be shipped with binary distributions of Python.
SSLObject.version() now correctly returns None when handshake over BIO has
not been performed yet.
Signed-off-by: Christian Heimes <christian@python.org>
One test case of test_xmlrpc uses HTTPServer with a subclass of
BaseHTTPRequestHandler. The BaseRequestHandler class logs to
sys.stderr by default. Override log_message() to not clobber
test output.
Signed-off-by: Christian Heimes <christian@python.org>
In case PROTOCOL_TLS_SERVER is used for both client context and server
context, the test thread dies with OSError. Catch OSError to avoid
traceback on sys.stderr
Signed-off-by: Christian Heimes <christian@python.org>
* fixed OrderedDict.__init__ docstring re PEP 468
* tightened comment and mirrored to C impl
* added space after period per marco-buttu
* preserved substituted for stable
* drop references to Python 3.6 and PEP 468
* Maintain a list of BufferedWriter objects. Flush them on exit.
In Python 3, the buffer and the underlying file object are separate
and so the order in which objects are finalized matters. This is
unlike Python 2 where the file and buffer were a single object and
finalization was done for both at the same time. In Python 3, if
the file is finalized and closed before the buffer then the data in
the buffer is lost.
This change adds a doubly linked list of open file buffers. An atexit
hook ensures they are flushed before proceeding with interpreter
shutdown. This is addition does not remove the need to properly close
files as there are other reasons why buffered data could get lost during
finalization.
Initial patch by Armin Rigo.
* Use weakref.WeakSet instead of WeakKeyDictionary.
* Simplify buffered double-linked list types.
* In _flush_all_writers(), suppress errors from flush().
* Remove NEWS entry, use blurb.
In Python 3.6, sorted() was removed from _make_key() for the lru_cache and instead rely on guaranteed keyword argument order preservation. This makes keyword argument handling faster but it also causes multiple callers with a different keyword argument order to be cached as separate items. Depending on your point of view, this is either a performance regression (increased number of cache misses) or a performance enhancement (faster computation of keys).
* bpo-1198569: Allow the braced pattern to be different
``string.Template`` subclasses can optionally define ``braceidpattern`` if
they want to specify different placeholder patterns inside and outside the
braces. If None (the default) it falls back to ``idpattern``.
Call doCleanups() to close the loop after calling
executor.shutdown(wait=True): see TestCase.set_event_loop() of
asyncio.test_utils.
Replace also gc.collect() with support.gc_collect().
* bpo-31326: ProcessPoolExecutor waits for the call queue thread
concurrent.futures.ProcessPoolExecutor.shutdown() now explicitly
closes the call queue. Moreover, shutdown(wait=True) now also join
the call queue thread, to prevent leaking a dangling thread.
* Fix for shutdown() being called twice.
* bpo-27144: concurrent.futures as_complie and map iterators do not keep
reference to returned object
* Some nits. Improve wordings in docstrings and comments, and avoid relying on
sys.getrefcount() in tests.
* Explicitly call shutdown(wait=True) on executors to wait until all
threads complete to prevent side effects between tests.
* Fix test_loop_self_reading_exception(): don't mock loop.close().
Previously, the original close() method was called rather than the
mock, because how set_event_loop() registered loop.close().
Use a pool of integer objects toprevent false alarm when checking for
memory block leaks. Fill the pool with values in -1000..1000 which
are the most common (reference, memory block, file descriptor)
differences.
Co-Authored-By: Antoine Pitrou <pitrou@free.fr>
* Make error message more informative
Replace assertions in error-reporting code with more-informative version that doesn't cause confusion over where and what the error is.
* Additional clarification + get travis to check
* Change from SystemError to TypeError
As suggested in PR comment by @pitrou, changing from SystemError; TypeError appears appropriate.
* NEWS file installation; ACKS addition (will do my best to justify it by additional work)
* Making current AssertionErrors in multiprocessing more informative
* Blurb added re multiprocessing managers.py, queues.py cleanup
* Further multiprocessing cleanup - went through pool.py
* Fix two asserts in multiprocessing/util.py
* Most asserts in multiprocessing more informative
* Didn't save right version
* Further work on multiprocessing error messages
* Correct typo
* Correct typo v2
* Blasted colon... serves me right for trying to work on two things at once
* Simplify NEWS entry
* Update 2017-08-18-17-16-38.bpo-5001.gwnthq.rst
* Update 2017-08-18-17-16-38.bpo-5001.gwnthq.rst
OK, never mind.
* Corrected (thanks to pitrou) error messages for notify
* Remove extraneous backslash in docstring.
Oren Milman
Nir Soffer
luzpaz
Serhiy Storchaka
Antoine Pitrou
Jack O'Connor
Victor Stinner
Vincent Michel
jimmylai
Alex Gaynor
Quentin Dawans
Terry Jan Reedy
Lovesh Harchandani
Ivan Levkivskyi
Alexander Belopolsky
Vitor Pereira
xdegaye
Benjamin Peterson
Mario Corchero
Nick Coghlan
Thomas Kluyver
andyclegg
Mark Shannon
Cheryl Sabella
jlacoline
Łukasz Langa
Ron
Zane Bitter
Riccardo Coccioli
Pablo Galindo
Ned Deily
brainfvck
INADA Naoki
Éric Araujo
Zachary Ware
Yury Selivanov
Jelle Zijlstra
Barry Warsaw
Victor K
Jakub Stasiak
pdox
diana
Thomas Moreau
Cornelius Diekmann
scoder
James Sexton
Zhiming Wang
Sorin Sbarnea
Stefan Grönke
Neil Schemenauer
Igor Filatov
Anthony Sottile
Rohit Balasubramanian
Raymond Hettinger
Christian Heimes
Feanil Patel
Joel Schaerer
wohlganger
Gregory P. Smith
Han Lee
Steve Dower
R. David Murray
caavery
Devin Jeanpierre
Ammar Askar
Jonathan Eunice
Grzegorz Grzywacz
Allen W. Smith, Ph.D