The sqlite3 module now fully implements the GC protocol, so there's no
need for this workaround anymore.
- Add and use managed resource helper for connections using TESTFN
- Sort test imports
- Split open-tests into their own test case
Automerge-Triggered-By: GH:vstinner
A number of places in the code base (notably ceval.c and frameobject.c) rely on mapping variable names to indices in the frame "locals plus" array (AKA fast locals), and thus opargs. Currently the compiler indirectly encodes that information on the code object as the tuples co_varnames, co_cellvars, and co_freevars. At runtime the dependent code must calculate the proper mapping from those, which isn't ideal and impacts performance-sensitive sections. This is something we can easily address in the compiler instead.
This change addresses the situation by replacing internal use of co_varnames, etc. with a single combined tuple of names in locals-plus order, along with a minimal array mapping each to its kind (local vs. cell vs. free). These two new PyCodeObject fields, co_fastlocalnames and co_fastllocalkinds, are not exposed to Python code for now, but co_varnames, etc. are still available with the same values as before (though computed lazily).
Aside from the (mild) performance impact, there are a number of other benefits:
* there's now a clear, direct relationship between locals-plus and variables
* code that relies on the locals-plus-to-name mapping is simpler
* marshaled code objects are smaller and serialize/de-serialize faster
Also note that we can take this approach further by expanding the possible values in co_fastlocalkinds to include specific argument types (e.g. positional-only, kwargs). Doing so would allow further speed-ups in _PyEval_MakeFrameVector(), which is where args get unpacked into the locals-plus array. It would also allow us to shrink marshaled code objects even further.
https://bugs.python.org/issue43693
Obviously, the former was a little misleading: not only rationals
may be considered as "infinite-precision, real numbers" - but, for
example, any real finite extension of the rationals.
On macOS, tk defers to system setting for button background when in normal state.
Give turtledemo button text a color that works on either light or dark background.
* Move up the comment about fields using in hashing/comparision.
* Group the fields more clearly.
* Add co_ncellvars and co_nfreevars.
* Raise ValueError if nlocals != len(varnames), rather than aborting.
The list lookups become a big burden for very long lists.
This patch changes the "happy flow" path of 2 lookups into 1 lookup.
Automerge-Triggered-By: GH:vsajip
Fix a regression in type() when a metaclass raises an exception. The
C function type_new() must properly report the exception when a
metaclass constructor raises an exception and the winner class is not
the metaclass.
A previous commit broke a check in sysconfig when building cpython itself.
This caused builds of the standard library modules to search a wrong
location (the installed location rather than the source directory) for
header files with the net effect that a ``make install``
incorrectly caused all extension modules to be rebuilt again and
with incorrect include file paths.
When building Python, we need two distinct "include" directories:
- source .h files
- install target for .h files
Note that this doesn't matter except when building Python from source.
Historically:
- source .h files were in the distutils scheme under 'include'
- the install directory was in the distutils.command.install scheme
under 'headers'
GH-24549 merged these; sysconfig is now the single source of truth and
distutils is derived from it.
This commit introduces a "secret" scheme path, 'headers', which contains
the install target. It is only present when building Python.
The distutils code uses it if present, and falls back to 'include'.
Co-authored-by: Ned Deily <nad@python.org>
The invalid assignment rules are very delicate since the parser can
easily raise an invalid assignment when a keyword argument is provided.
As they are very deep into the grammar tree, is very difficult to
specify in which contexts these rules can be used and in which don't.
For that, we need to use a different version of the rule that doesn't do
error checking in those situations where we don't want the rule to raise
(keyword arguments and generator expressions).
We also need to check if we are in left-recursive rule, as those can try
to eagerly advance the parser even if the parse will fail at the end of
the expression. Failing to do this allows the parser to start parsing a
call as a tuple and incorrectly identify a keyword argument as an
invalid assignment, before it realizes that it was not a tuple after all.
Fix a crash at Python exit when a deallocator function removes the
last strong reference to a heap type.
Don't read type memory after calling basedealloc() since
basedealloc() can deallocate the type and free its memory.
_PyMem_IsPtrFreed() argument is now constant.
* Remove 'zombie' frames. We won't need them once we are allocating fixed-size frames.
* Add co_nlocalplus field to code object to avoid recomputing size of locals + frees + cells.
* Move locals, cells and freevars out of frame object into separate memory buffer.
* Use per-threadstate allocated memory chunks for local variables.
* Move globals and builtins from frame object to per-thread stack.
* Move (slow) locals frame object to per-thread stack.
* Move internal frame functions to internal header.
Moreover, Py_FrozenMain() relies on Py_InitializeFromConfig() to
handle the PYTHONUNBUFFERED environment variable and configure C
stdio streams like stdout (make the stream unbuffered).
`shutil.unpack_archive()` tries to read the whole file into memory, making no use of any kind of smaller buffer. Process crashes for really large files: I.e. archive: ~1.7G, unpacked: ~10G. Before the crash it can easily take away all available RAM on smaller systems. Had to pull the code form `zipfile.Zipfile.extractall()` to fix this
Automerge-Triggered-By: GH:gpshead
For IPv4 mapped IPv6 addresses, defer privacy check to the mapped IPv4 address. Solves bug where public mapped IPv4 addresses are considered private by the IPv6 check.
Automerge-Triggered-By: GH:gpshead
When a Thread is not joined after it has stopped, its lock may remain in the _shutdown_locks set until interpreter shutdown. If many threads are created this way, the _shutdown_locks set could therefore grow endlessly. To avoid such a situation, purge expired locks each time a new one is added or removed.
Added two new attributes to ``_GenericAlias``:
* ``_typevar_types``, a single type or tuple of types indicating what types are treated as a ``TypeVar``. Used for ``isinstance`` checks.
* ``_paramspec_tvars ``, a boolean flag which guards special behavior for dealing with ``ParamSpec``. Setting it to ``True`` means this class deals with ``ParamSpec``.
Automerge-Triggered-By: GH:gvanrossum
Switch to lru_cache in urllib.parse.
urllib.parse now uses functool.lru_cache for its internal URL splitting and
quoting caches instead of rolling its own like its the 90s.
The undocumented internal Quoted class API is now deprecated
as it had no reason to be public and no existing OSS users were found.
The clear_cache() API remains undocumented but gets an explicit test as it
is used in a few projects' (twisted, gevent) tests as well as our own regrtest.
Added in bpo-41730 (GH-22186), the test apparently causes refleaks. The
test isn't worth hunting them down, so it's simply reverted.
This partially reverts commit 4a2d98a1e9.
A C function accessible by the default exception handler, but not by python code,
finds the existing name closest to the name causing a name or attribute error. For
such errors, call the default handler after capturing stderr and retrieve its message line.
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
"Zero cost" exception handling.
* Uses a lookup table to determine how to handle exceptions.
* Removes SETUP_FINALLY and POP_TOP block instructions, eliminating (most of) the runtime overhead of try statements.
* Reduces the size of the frame object by about 60%.
* Set content-length for simple http server 301s
When http.server.SimpleHTTPRequestHandler sends a 301 (Moved
Permanently) due to a missing file, it does not set a Content-Length
of 0. Unfortunately, certain clients can be left waiting for the
connection to be closed in this circumstance, even though no body
will be sent. At time of writing, both curl and Firefox demonstrate
this behavior.
* Test Content-Length on simple http server redirect
When serving a redirect, the SimpleHTTPRequestHandler will now send
`Content-Length: 0`. Several tests for http.server already cover
various behaviors and checks including redirection. This change only
adds one check for the expected Content-Length on the simplest case
for a redirect.
* Add news entry for SimpleHTTPRequestHandler fix
* Clarify the specific kind of 301
Co-authored-by: Senthil Kumaran <skumaran@gatech.edu>
Fixes http.client potential denial of service where it could get stuck reading lines from a malicious server after a 100 Continue response.
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Ignore objects that inspect.unwrap throws due to
too many wrappers. This is a very rare case, however
it can easily be surfaced when a module under doctest
imports unitest.mock.call into its namespace.
We simply skip any object that throws this exception.
This should handle the majority of cases.
The address tuple for CAN_RAW no longer returns the address family
after the introduction of CAN ISO-TP support in a30f6d45ac. However,
updating test_socket.CANTest.testSendFrame was missed as part of the
change, so the test incorrectly attempts to index past the last tuple
item to retrieve the address family.
This removes the now-redundant check for equality against socket.AF_CAN,
as the tuple will not contain the address family.
This adds IO, TextIO, BinaryIO, Match, and Pattern.
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com>
Since WPAR and LPAR both have a builddate for teh fileset bos.rte
The name of the fileset checked is modified.
To prevent a similiar situation (no builddate in ODM) a value
sufficient for pep425 activity if retrieved buildate is zero or NULL
Patch by M Felt.
Left click and drag to select lines. With selection, right click for context menu with copy and copy-with-prompts.
Also add copy-with-prompts to the text-box context menu.
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Only complain if the config target is >= 10.3 and the current target is
< 10.3. The check was originally added to ensure that incompatible
LDSHARED flags are not used, because -undefined dynamic_lookup is
used when building for 10.3 and later, and is not supported on older OS
versions. Apart from that, there should be no problem in general
with using an older target.
Authored-by: Joshua Root <jmr@macports.org>
* bpo-43926: Cleaner metadata with PEP 566 JSON support.
* Add blurb
* Add versionchanged and versionadded declarations for changes to metadata.
* Use descriptor for PEP 566
Reverts commit e653d4d8e8 and makes
parsing even more strict. Like socket.inet_pton() any leading zero
is now treated as invalid input.
Signed-off-by: Christian Heimes <christian@python.org>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
check_set_special_type_attr() and type_set_annotations()
now check for immutable flag (Py_TPFLAGS_IMMUTABLETYPE).
Co-authored-by: Victor Stinner <vstinner@python.org>
The PyStdPrinter_Type type now uses the
Py_TPFLAGS_DISALLOW_INSTANTIATION flag to disallow instantiation,
rather than seting a tp_init method which always fail.
Write also unit tests for PyStdPrinter_Type.
Add a new Py_TPFLAGS_DISALLOW_INSTANTIATION type flag to disallow
creating type instances: set tp_new to NULL and don't create the
"__new__" key in the type dictionary.
The flag is set automatically on static types if tp_base is NULL or
&PyBaseObject_Type and tp_new is NULL.
Use the flag on the following types:
* _curses.ncurses_version type
* _curses_panel.panel
* _tkinter.Tcl_Obj
* _tkinter.tkapp
* _tkinter.tktimertoken
* _xxsubinterpretersmodule.ChannelID
* sys.flags type
* sys.getwindowsversion() type
* sys.version_info type
Update MyStr example in the C API documentation to use
Py_TPFLAGS_DISALLOW_INSTANTIATION.
Add _PyStructSequence_InitType() function to create a structseq type
with the Py_TPFLAGS_DISALLOW_INSTANTIATION flag set.
type_new() calls _PyType_CheckConsistency() at exit.
* Add Py_TPFLAGS_SEQUENCE and Py_TPFLAGS_MAPPING, add to all relevant standard builtin classes.
* Set relevant flags on collections.abc.Sequence and Mapping.
* Use flags in MATCH_SEQUENCE and MATCH_MAPPING opcodes.
* Inherit Py_TPFLAGS_SEQUENCE and Py_TPFLAGS_MAPPING.
* Add NEWS
* Remove interpreter-state map_abc and seq_abc fields.
Add inspect.get_annotations, which safely computes the annotations defined on an object. It works around the quirks of accessing the annotations from various types of objects, and makes very few assumptions about the object passed in. inspect.get_annotations can also correctly un-stringize stringized annotations.
inspect.signature, inspect.from_callable, and inspect.from_function now call inspect.get_annotations to retrieve annotations. This means inspect.signature and inspect.from_callable can now un-stringize stringized annotations, too.
Remove call to macosx.setupApp, which calls macosc.overrideRootMenu, which modifies
the menus, which results in two failures in the second round of the leak test.
While working on another issue, I noticed two minor nits in the C implementation of the module object. Both are related to getting a module's name.
First, the C function module_dir() (module.__dir__) starts by ensuring the module dict is valid. If the module dict is invalid, it wants to format an exception using the name of the module, which it gets from PyModule_GetName(). However, PyModule_GetName() gets the name of the module from the dict. So getting the name in this circumstance will never succeed.
When module_dir() wants to format the error but can't get the name, it knows that PyModule_GetName() must have already raised an exception. So it leaves that exception alone and returns an error. The end result is that the exception raised here is kind of useless and misleading: dir(module) on a module with no __dict__ raises SystemError("nameless module"). I changed the code to actually raise the exception it wanted to raise, just without a real module name: TypeError("<module>.__dict__ is not a dictionary"). This seems more useful, and would do a better job putting the programmer who encountered this on the right track of figuring out what was going on.
Second, the C API function PyModule_GetNameObject() checks to see if the module has a dict. If m->md_dict is not NULL, it calls _PyDict_GetItemIdWithError(). However, it's possible for m->md_dict to be None. And if you call _PyDict_GetItemIdWithError(Py_None, ...) it will *crash*.
Unfortunately, this crash was due to my own bug in the other branch. Fixing my code made the crash go away. I assert that this is still possible at the API level.
The fix is easy: add a PyDict_Check() to PyModule_GetNameObject().
Unfortunately, I don't know how to add a unit test for this. Having changed module_dir() above, I can't find any other interfaces callable from Python that eventually call PyModule_GetNameObject(). So I don't know how to trick the runtime into reproducing this error.
Since both these changes are minor--each entails only a small edit to only one line--I didn't bother with a news item.
Change class and module objects to lazy-create empty annotations dicts on demand. The annotations dicts are stored in the object's `__dict__` for backwards compatibility.
* Add length parameter to PyLineTable_InitAddressRange and doen't use sentinel values at end of table. Makes the line number table more robust.
* Update PyCodeAddressRange to match PEP 626.
Removes the `list` call in the Popen `repr`.
Current implementation:
For cmd = `python --version`, with `shell=True`.
```bash
<Popen: returncode: None args: ['p', 'y', 't', 'h', 'o', 'n', ' ', '-', '-',...>
```
For `shell=False` and args=`['python', '--version']`, the output is correct:
```bash
<Popen: returncode: None args: ['python', '--version']>
```
With the new changes the `repr` yields:
For cmd = `python --version`, with `shell=True`:
```bash
<Popen: returncode: None args: 'python --version'>
```
For `shell=False` and args=`['python', '--version']`, the output:
```bash
<Popen: returncode: None args: ['python', '--version']>
```
Automerge-Triggered-By: GH:gpshead
Faster bz2/lzma/zlib via new output buffering.
Also adds .readall() function to _compression.DecompressReader class
to take best advantage of this in the consume-all-output at once scenario.
Often a 5-20% speedup in common scenarios due to less data copying.
Contributed by Ma Lin.
In 3.12 ``True`` or ``False`` will be returned for all containment checks,
with ``True`` being returned if the value is either a member of that enum
or one of its members' value.
In 3.12 the enum member, not the member's value, will be used for
format() calls. Format specifiers can be used to retain the current
display of enum members:
Example enumeration:
class Color(IntEnum):
RED = 1
GREEN = 2
BLUE = 3
Current behavior:
f'{Color.RED}' --> '1'
Future behavior:
f'{Color.RED}' --> 'RED'
Using d specifier:
f'{Color.RED:d}' --> '1'
Using specifiers can be done now and is future-compatible.
The internal `_ssl._SSLSocket` object now provides methods to retrieve
the peer cert chain and verified cert chain as a list of Certificate
objects. Certificate objects have methods to convert the cert to a dict,
PEM, or DER (ASN.1).
These are private APIs for now. There is a slim chance to stabilize the
approach and provide a public API for 3.10. Otherwise I'll provide a
stable API in 3.11.
Signed-off-by: Christian Heimes <christian@python.org>
* Add space after period to warning in _tzpath.py
Currently:
InvalidTZPathWarning: Invalid paths specified in PYTHONTZPATH environment variable.Paths should be absolute but found the following relative paths: ...
* Update _tzpath.py
asyncio.get_event_loop() emits now a deprecation warning when it creates a new event loop.
In future releases it will became an alias of asyncio.get_running_loop().
Revert 73ea546, increase logging, and improve stability of test.
Handle all OSErrors in a single block. OSError also takes care of
SSLError and socket's connection errors.
Partly reverts commit fb7e750. The
threaded connection handler must not raise an unhandled exception.
Depending on usage, it's possible for Flag members to have the _inverted_ attribute when they are testing, while the Flag being testing against will not have that attribute on its members -- so skip that comparison.
The argument order of `link_to()` is reversed compared to what one may expect, so:
a.link_to(b)
Might be expected to create *a* as a link to *b*, in fact it creates *b* as a link to *a*, making it function more like a "link from". This doesn't match `symlink_to()` nor the documentation and doesn't seem to be the original author's intent.
This PR deprecates `link_to()` and introduces `hardlink_to()`, which has the same argument order as `symlink_to()`.
test_wrong_cert_tls13 sometimes fails on some Windows buildbots. Turn
failing test case into skipped test case until we have more time to
investigate.
Signed-off-by: Christian Heimes <christian@python.org>
To improve the user experience understanding what part of the error messages associated with SyntaxErrors is wrong, we can highlight the whole error range and not only place the caret at the first character. In this way:
>>> foo(x, z for z in range(10), t, w)
File "<stdin>", line 1
foo(x, z for z in range(10), t, w)
^
SyntaxError: Generator expression must be parenthesized
becomes
>>> foo(x, z for z in range(10), t, w)
File "<stdin>", line 1
foo(x, z for z in range(10), t, w)
^^^^^^^^^^^^^^^^^^^^
SyntaxError: Generator expression must be parenthesized
This change:
* merges `distutils.sysconfig` into `sysconfig` while keeping the original functionality and
* marks `distutils.sysconfig` as deprecated
https://bugs.python.org/issue41282
The sys module uses the kernel32.dll version number, which can vary from the "actual" Windows version.
Since the best option for getting the version is WMI (which is expensive), we switch back to launching cmd.exe (which is also expensive, but a lot less code on our part).
sys.getwindowsversion() is not updated to avoid launching executables from that module.
Previously TestIntEnumConvert and TestStrEnumConvert would end up
converting the module level variables from their regular int form
to a `test.test_enum.X` instance after _convert would run. This
meant that after a single test ran, the next set of _convert
functions would be operating on the enum instances rather than
ints. This would cause some tests such as the one involving format
to fail when running under a mode that repeatedly runs test such
as the refleak finder.
add:
* `_simple_enum` decorator to transform a normal class into an enum
* `_test_simple_enum` function to compare
* `_old_convert_` to enable checking `_convert_` generated enums
`_simple_enum` takes a normal class and converts it into an enum:
@simple_enum(Enum)
class Color:
RED = 1
GREEN = 2
BLUE = 3
`_old_convert_` works much like` _convert_` does, using the original logic:
# in a test file
import socket, enum
CheckedAddressFamily = enum._old_convert_(
enum.IntEnum, 'AddressFamily', 'socket',
lambda C: C.isupper() and C.startswith('AF_'),
source=_socket,
)
`_test_simple_enum` takes a traditional enum and a simple enum and
compares the two:
# in the REPL or the same module as Color
class CheckedColor(Enum):
RED = 1
GREEN = 2
BLUE = 3
_test_simple_enum(CheckedColor, Color)
_test_simple_enum(CheckedAddressFamily, socket.AddressFamily)
Any important differences will raise a TypeError
add:
_simple_enum decorator to transform a normal class into an enum
_test_simple_enum function to compare
_old_convert_ to enable checking _convert_ generated enums
_simple_enum takes a normal class and converts it into an enum:
@simple_enum(Enum)
class Color:
RED = 1
GREEN = 2
BLUE = 3
_old_convert_ works much like _convert_ does, using the original logic:
# in a test file
import socket, enum
CheckedAddressFamily = enum._old_convert_(
enum.IntEnum, 'AddressFamily', 'socket',
lambda C: C.isupper() and C.startswith('AF_'),
source=_socket,
)
test_simple_enum takes a traditional enum and a simple enum and
compares the two:
# in the REPL or the same module as Color
class CheckedColor(Enum):
RED = 1
GREEN = 2
BLUE = 3
_test_simple_enum(CheckedColor, Color)
_test_simple_enum(CheckedAddressFamily, socket.AddressFamily)
Any important differences will raise a TypeError
Erlend Egeberg Aasland
Eric Snow
Victor Stinner
Mark Shannon
andrei kulakov
Gregory P. Smith
Tal Einat
Mariusz Felisiak
Serhiy Storchaka
Jason R. Coombs
Dong-hee Na
Sergey B Kirpichev
Terry Jan Reedy
Andreas Jansson
Sean Grady
Jürgen Gmach
Yonatan Goldschmidt
Raymond Hettinger
Zack Kneupper
Petr Viktorin
Filipe Laíns
Irit Katriel
Anthony Sottile
E-Paine
Zackery Spytz
Pablo Galindo
Barney Gale
native-api
Catherine Devlin
Ian Norton
uniocto
Igor Bolshakov
Pete Wicken
Sergey B Kirpichev
flizzywine
Ashwin Ramaswami
Batuhan Taskaya
Ken Jin
Rory Yorke
Antoine Pitrou
Miguel Brito
Yurii Karabas
Karl Dubost
Sebastian Rittau
Giampaolo Rodola
Zachary Ware
pxinwr
Inada Naoki
wyz23x2
Stephen Rosen
Linus Groh
Senthil Kumaran
Gen Xu
Alfred Perlstein
Kevin Follstad
karl ding
Ikko Ashimine
Jelle Zijlstra
Michael Felt
Ethan Furman
Shreyan Avigyan
Dennis Sweeney
Christian Heimes
Andrew Svetlov
Eric V. Smith
Ned Deily
Brandt Bucher
Roberto Hueso
Stéphane Bidoul
Tarjei Bærland
larryhastings
Nick Coghlan
Hristo Venev
Steve Dower
M. Kocher
Ma Lin
Tzu-ping Chung
uy-rrodriguez
Allen
Tymoteusz Wołodźko
Stéphane Bidoul
Brett Cannon
Carl Friedrich Bolz-Tereick
Lumír 'Frenzy' Balhar
Saiyang Gou
Ammar Askar