Commit Graph

750 Commits

Author SHA1 Message Date
Mark Shannon bbcf42449e
GH-90230: Add stats to breakdown the origin of calls to `PyEval_EvalFrame` (GH-93284) 2022-05-27 16:31:41 +01:00
larryhastings 5021064390
gh-92203: Add closure support to exec(). (#92204)
Add a closure keyword-only parameter to exec(). It can only be specified when exec-ing a code object that uses free variables. When specified, it must be a tuple, with exactly the number of cell variables referenced by the code object. closure has a default value of None, and it must be None if the code object doesn't refer to any free variables.
2022-05-06 10:09:35 -07:00
Dennis Sweeney da6c78584b
gh-90667: Add specializations of Py_DECREF when types are known (GH-30872) 2022-04-19 19:02:19 +01:00
Victor Stinner f0bc694856
bpo-47164: Add _PyCFunction_CAST() macro (GH-32192)
Use the macro in C files of the Python/ directory.
2022-03-31 10:02:34 +02:00
Kumar Aditya 6f3b9e2243
Use FASTCALL for __import__ (GH-31752) 2022-03-11 18:46:55 +02:00
Eric Snow 81c72044a1
bpo-46541: Replace core use of _Py_IDENTIFIER() with statically initialized global objects. (gh-30928)
We're no longer using _Py_IDENTIFIER() (or _Py_static_string()) in any core CPython code.  It is still used in a number of non-builtin stdlib modules.

The replacement is: PyUnicodeObject (not pointer) fields under _PyRuntimeState, statically initialized as part of _PyRuntime.  A new _Py_GET_GLOBAL_IDENTIFIER() macro facilitates lookup of the fields (along with _Py_GET_GLOBAL_STRING() for non-identifier strings).

https://bugs.python.org/issue46541#msg411799 explains the rationale for this change.

The core of the change is in:

* (new) Include/internal/pycore_global_strings.h - the declarations for the global strings, along with the macros
* Include/internal/pycore_runtime_init.h - added the static initializers for the global strings
* Include/internal/pycore_global_objects.h - where the struct in pycore_global_strings.h is hooked into _PyRuntimeState
* Tools/scripts/generate_global_objects.py - added generation of the global string declarations and static initializers

I've also added a --check flag to generate_global_objects.py (along with make check-global-objects) to check for unused global strings.  That check is added to the PR CI config.

The remainder of this change updates the core code to use _Py_GET_GLOBAL_IDENTIFIER() instead of _Py_IDENTIFIER() and the related _Py*Id functions (likewise for _Py_GET_GLOBAL_STRING() instead of _Py_static_string()).  This includes adding a few functions where there wasn't already an alternative to _Py*Id(), replacing the _Py_Identifier * parameter with PyObject *.

The following are not changed (yet):

* stop using _Py_IDENTIFIER() in the stdlib modules
* (maybe) get rid of _Py_IDENTIFIER(), etc. entirely -- this may not be doable as at least one package on PyPI using this (private) API
* (maybe) intern the strings during runtime init

https://bugs.python.org/issue46541
2022-02-08 13:39:07 -07:00
Victor Stinner 500c146387
bpo-46417: Clear more static types (GH-30796)
* Move PyContext static types into object.c static_types list.
* Rename PyContextTokenMissing_Type to _PyContextTokenMissing_Type
  and declare it in pycore_context.h.
* _PyHamtItems types are no long exported: replace PyAPI_DATA() with
  extern.
2022-01-22 18:55:48 +01:00
Victor Stinner 7835cbf949
bpo-46417: Use _PyType_CAST() in Python directory (GH-30769) 2022-01-21 23:30:17 +01:00
Géry Ogam 16bf9bd157
bpo-44024: Improve the TypeError message in getattr and hasattr (GH-25863)
Use common error message for non-string attribute name in the builtin
functions getattr and hasattr.
The special check no longer needed since Python 3.0.
2022-01-18 22:46:26 +02:00
Mark Shannon 135cabd328
bpo-44525: Copy free variables in bytecode to allow calls to inner functions to be specialized (GH-29595)
* Make internal APIs that take PyFrameConstructor take a PyFunctionObject instead.

* Add reference to function to frame, borrow references to builtins and globals.

* Add COPY_FREE_VARS instruction to allow specialization of calls to inner functions.
2021-11-23 09:53:24 +00:00
Victor Stinner 7cdc2a0f4b
pycore_pystate.h no longer redefines PyThreadState_GET() (GH-28921)
Redefining the PyThreadState_GET() macro in pycore_pystate.h is
useless since it doesn't affect files not including it. Either use
_PyThreadState_GET() directly, or don't use pycore_pystate.h internal
C API. For example, the _testcapi extension don't use the internal C
API, but use the public PyThreadState_Get() function instead.

Replace PyThreadState_Get() with _PyThreadState_GET(). The
_PyThreadState_GET() macro is more efficient than PyThreadState_Get()
and PyThreadState_GET() function calls which call fail with a fatal
Python error.

posixmodule.c and _ctypes extension now include <windows.h> before
pycore header files (like pycore_call.h).

_PyTraceback_Add() now uses _PyErr_Fetch()/_PyErr_Restore() instead
of PyErr_Fetch()/PyErr_Restore().

The _decimal and _xxsubinterpreters extensions are now built with the
Py_BUILD_CORE_MODULE macro defined to get access to the internal C
API.
2021-10-13 14:09:13 +02:00
Victor Stinner d943d19172
bpo-45439: Move _PyObject_CallNoArgs() to pycore_call.h (GH-28895)
* Move _PyObject_CallNoArgs() to pycore_call.h (internal C API).
* _ssl, _sqlite and _testcapi extensions now call the public
  PyObject_CallNoArgs() function, rather than _PyObject_CallNoArgs().
* _lsprof extension is now built with Py_BUILD_CORE_MODULE macro
  defined to get access to internal _PyObject_CallNoArgs().
2021-10-12 08:38:19 +02:00
Victor Stinner ce3489cfdb
bpo-45439: Rename _PyObject_CallNoArg() to _PyObject_CallNoArgs() (GH-28891)
Fix typo in the private _PyObject_CallNoArg() function name: rename
it to _PyObject_CallNoArgs() to be consistent with the public
function PyObject_CallNoArgs().
2021-10-12 00:42:23 +02:00
Serhiy Storchaka a5a56154f1
Remove trailing spaces. (GH-28706) 2021-10-03 16:58:14 +03:00
Pablo Galindo Salgado 1c7e98dc25
bpo-24076: Fix reference in sum() introduced by GH-28469 (GH-28493) 2021-09-21 18:38:57 +01:00
scoder debd804037
bpo-24076: Inline single digit unpacking in the integer fastpath of sum() (GH-28469) 2021-09-21 11:01:18 +02:00
Serhiy Storchaka 92bf8691fb
bpo-43413: Fix handling keyword arguments in subclasses of some buitin classes (GH-26456)
* Constructors of subclasses of some buitin classes (e.g. tuple, list,
  frozenset) no longer accept arbitrary keyword arguments.
* Subclass of set can now define a __new__() method with additional
  keyword parameters without overriding also __init__().
2021-09-12 13:27:50 +03:00
Yury Selivanov 2c3474a637
bpo-45123: PyAiter_Check and PyObject_GetAiter fix & rename. (GH-28194)
Fix PyAiter_Check to only check for the `__anext__` presense (not for
`__aiter__`). Rename `PyAiter_Check()` to `PyAIter_Check()`,
`PyObject_GetAiter()` -> `PyObject_GetAIter()`.

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
2021-09-07 11:52:30 +01:00
Pablo Galindo Salgado a40675c659
bpo-44856: Possible reference leak in error paths of update_bases() and __build_class__ (GH-27647) 2021-08-07 12:10:14 +02:00
Batuhan Taskaya 9af34c9351
bpo-20201: variadic arguments support for AC (GH-18609)
Implement support for `*args` in AC, and port `print()` to use it.
2021-07-16 18:43:02 +03:00
Erik Welch 6af4e6b266
bpo-43918: document signature and default argument of `anext` builtin (#25551)
Co-authored-by: Dennis Sweeney <36520290+sweeneyde@users.noreply.github.com>
2021-06-22 14:00:51 -07:00
Nick Coghlan 1e7b858575
bpo-43892: Make match patterns explicit in the AST (GH-25585)
Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
2021-04-28 22:58:44 -07:00
Victor Stinner d36d6a9c18
bpo-43244: Remove Yield macro from pycore_ast.h (GH-25243)
* pycore_ast.h no longer defines the Yield macro.
* Fix a compiler warning on Windows: "warning C4005: 'Yield': macro
  redefinition".
* Python-ast.c now defines directly functions with their real
  _Py_xxx() name, rather than xxx().
* Remove "#undef Yield" in C files including pycore_ast.h.
2021-04-07 13:01:09 +02:00
Pablo Galindo a02683ac38
bpo-31861: Fix reference leak in builtin_anext_impl() (GH-25008) 2021-03-24 01:42:13 +00:00
Victor Stinner 8370e07e1e
bpo-43244: Remove the pyarena.h header (GH-25007)
Remove the pyarena.h header file with functions:

* PyArena_New()
* PyArena_Free()
* PyArena_Malloc()
* PyArena_AddPyObject()

These functions were undocumented, excluded from the limited C API,
and were only used internally by the compiler.

Add pycore_pyarena.h header. Rename functions:

* PyArena_New() => _PyArena_New()
* PyArena_Free() => _PyArena_Free()
* PyArena_Malloc() => _PyArena_Malloc()
* PyArena_AddPyObject() => _PyArena_AddPyObject()
2021-03-24 02:23:01 +01:00
Victor Stinner a81fca6ec8
bpo-43244: Add pycore_compile.h header file (GH-25000)
Remove the compiler functions using "struct _mod" type, because the
public AST C API was removed:

* PyAST_Compile()
* PyAST_CompileEx()
* PyAST_CompileObject()
* PyFuture_FromAST()
* PyFuture_FromASTObject()

These functions were undocumented and excluded from the limited C API.

Rename functions:

* PyAST_CompileObject() => _PyAST_Compile()
* PyFuture_FromASTObject() => _PyFuture_FromAST()

Moreover, _PyFuture_FromAST() is no longer exported (replace
PyAPI_FUNC() with extern). _PyAST_Compile() remains exported for
test_peg_generator.

Remove also compatibility functions:

* PyAST_Compile()
* PyAST_CompileEx()
* PyFuture_FromAST()
2021-03-24 00:51:50 +01:00
Joshua Bronson f0a6fde882
bpo-31861: Add aiter and anext to builtins (#23847)
Co-authored-by: jab <jab@users.noreply.github.com>
Co-authored-by: Daniel Pope <mauve@mauveweb.co.uk>
Co-authored-by: Justin Wang <justin39@gmail.com>
2021-03-23 15:47:21 -07:00
Victor Stinner 94faa0724f
bpo-43244: Remove ast.h, asdl.h, Python-ast.h headers (GH-24933)
These functions were undocumented and excluded from the limited C
API.

Most names defined by these header files were not prefixed by "Py"
and so could create names conflicts. For example, Python-ast.h
defined a "Yield" macro which was conflict with the "Yield" name used
by the Windows <winbase.h> header.

Use the Python ast module instead.

* Move Include/asdl.h to Include/internal/pycore_asdl.h.
* Move Include/Python-ast.h to Include/internal/pycore_ast.h.
* Remove ast.h header file.
* pycore_symtable.h no longer includes Python-ast.h.
2021-03-23 20:47:40 +01:00
Dong-hee Na 86883d40e9
bpo-43575: Use PEP 590 vectorcall to speed up map() (GH-24955) 2021-03-22 19:01:14 +09:00
Victor Stinner eec8e61992
bpo-43244: Remove the PyAST_Validate() function (GH-24911)
Remove the PyAST_Validate() function. It is no longer possible to
build a AST object (mod_ty type) with the public C API. The function
was already excluded from the limited C API (PEP 384).

Rename PyAST_Validate() function to _PyAST_Validate(), move it to the
internal C API, and don't export it anymore (replace PyAPI_FUNC with
extern).

The function was added in bpo-12575 by
the commit 832bfe2ebd.
2021-03-18 14:57:49 +01:00
Dong-hee Na 9a9c11ad41
bpo-43287: Use PEP 590 vectorcall to speed up filter() (GH-24611) 2021-03-11 01:39:52 +09:00
Victor Stinner bcb094b41f
bpo-43268: Pass interp rather than tstate to internal functions (GH-24580)
Pass the current interpreter (interp) rather than the current Python
thread state (tstate) to internal functions which only use the
interpreter.

Modified functions:

* _PyXXX_Fini() and _PyXXX_ClearFreeList() functions
* _PyEval_SignalAsyncExc(), make_pending_calls()
* _PySys_GetObject(), sys_set_object(), sys_set_object_id(), sys_set_object_str()
* should_audit(), set_flags_from_config(), make_flags()
* _PyAtExit_Call()
* init_stdio_encoding()
* etc.
2021-02-19 15:10:45 +01:00
Mark Shannon 0332e569c1
bpo-42990: Further refactoring of PyEval_ functions. (GH-24368)
* Further refactoring of PyEval_EvalCode and friends. Break into make-frame, and eval-frame parts.

* Simplify function vector call using new _PyEval_Vector.

* Remove unused internal functions: _PyEval_EvalCodeWithName and _PyEval_EvalCode.

* Don't use legacy function PyEval_EvalCodeEx.
2021-02-01 10:42:03 +00:00
Brandt Bucher 226a012d1c
bpo-42536: GC track recycled tuples (GH-23623)
Several built-in and standard library types now ensure that their internal result tuples are always tracked by the garbage collector:

- collections.OrderedDict.items
- dict.items
- enumerate
- functools.reduce
- itertools.combinations
- itertools.combinations_with_replacement
- itertools.permutations
- itertools.product
- itertools.zip_longest
- zip

Previously, they could have become untracked by a prior garbage collection.
2020-12-04 19:45:57 -08:00
Victor Stinner 00d7abd7ef
bpo-42519: Replace PyMem_MALLOC() with PyMem_Malloc() (GH-23586)
No longer use deprecated aliases to functions:

* Replace PyMem_MALLOC() with PyMem_Malloc()
* Replace PyMem_REALLOC() with PyMem_Realloc()
* Replace PyMem_FREE() with PyMem_Free()
* Replace PyMem_Del() with PyMem_Free()
* Replace PyMem_DEL() with PyMem_Free()

Modify also the PyMem_DEL() macro to use directly PyMem_Free().
2020-12-01 09:56:42 +01:00
Serhiy Storchaka b510e101f8
bpo-42152: Use PyDict_Contains and PyDict_SetDefault if appropriate. (GH-22986)
If PyDict_GetItemWithError is only used to check whether the key is in dict,
it is better to use PyDict_Contains instead.

And if it is used in combination with PyDict_SetItem, PyDict_SetDefault can
replace the combination.
2020-10-26 12:47:57 +02:00
Victor Stinner c96d00e88e
bpo-41078: Fix bltinmodule.c with Py_TRACE_REFS (GH-21058)
Add pycore_object.h include to fix bltinmodule.c when Py_TRACE_REFS
macro is defined.
2020-06-22 18:02:49 +02:00
Victor Stinner 384621c42f
bpo-41078: Rename pycore_tupleobject.h to pycore_tuple.h (GH-21056) 2020-06-22 17:27:35 +02:00
Guido van Rossum 310f6aa7db
bpo-40636: PEP 618: add strict parameter to zip() (GH-20921)
zip() now supports PEP 618's strict parameter, which raises a
ValueError if the arguments are exhausted at different lengths.
Patch by Brandt Bucher.

Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
Co-authored-by: Ram Rachum <ram@rachum.com>
2020-06-19 12:16:57 +02:00
Alex Povel df773f8c54
bpo-40471: Fix grammar typo in 'issubclass' docstring (GH-19847)
Just a brief grammar fix.

See also <>.
2020-06-03 06:19:45 -07:00
Gregory P. Smith 6a5d3ff676
bpo-40636: Clarify the zip built-in docstring. (GH-20118)
Clarify the zip built-in docstring.

This puts much simpler text up front along with an example.

As it was, the zip built-in docstring was technically correct.  But too
technical for the reader who shouldn't _need_ to know about `__next__` and
`StopIteration` as most people do not need to understand the internal
implementation details of the iterator protocol in their daily life.

This is a documentation only change, intended to be backported to 3.8; it is
only tangentially related to PEP-618 which might offer new behavior options
in the future.

Wording based a bit more on enumerate per Brandt's suggestion.

This gets rid of the legacy wording paragraph which seems too tied to
implementation details of the iterator protocol which isn't relevant here.

Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
2020-05-15 14:26:00 -07:00
Guido van Rossum c001c09e90
bpo-40334: Support type comments (GH-19780)
This implements full support for # type: <type> comments, # type: ignore <stuff> comments, and the func_type parsing mode for ast.parse() and compile().

Closes https://github.com/we-like-parsers/cpython/issues/95.

(For now, you need to use the master branch of mypy, since another issue unique to 3.9 had to be fixed there, and there's no mypy release yet.)

The only thing missing is `feature_version=N`, which is being tracked in https://github.com/we-like-parsers/cpython/issues/124.
2020-04-30 12:12:19 -07:00
Guido van Rossum bc28805570
bpo-40334: Use old compiler when compile mode is func_type (GH-19692)
This is invoked by mypy, using ast.parse(source, "<func_type>", "func_type"). Since the new grammar doesn't yet support the func_type_input start symbol we must use the old compiler in this case to prevent a crash.

https://bugs.python.org/issue40334
2020-04-23 15:42:56 -07:00
Victor Stinner 1def7754b7
bpo-40334: Rename PyConfig.use_peg to _use_peg_parser (GH-19670)
* Rename PyConfig.use_peg to _use_peg_parser
* Document PyConfig._use_peg_parser and mark it a deprecated
* Mark -X oldparser option and PYTHONOLDPARSER env var as deprecated
  in the documentation.
* Add use_old_parser() and skip_if_new_parser() to test.support
* Remove sys.flags.use_peg: use_old_parser() uses
  _testinternalcapi.get_configs() instead.
* Enhance test_embed tests
* subprocess._args_from_interpreter_flags() copies -X oldparser
2020-04-23 03:03:24 +02:00
Pablo Galindo c5fc156852
bpo-40334: PEP 617 implementation: New PEG parser for CPython (GH-19503)
Co-authored-by: Guido van Rossum <guido@python.org>
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
2020-04-22 23:29:27 +01:00
Batuhan Taşkaya 4454057269
bpo-39562: Prevent collision of future and compiler flags (GH-19230)
The constant values of future flags in the __future__ module
is updated in order to prevent collision with compiler flags.
Previously PyCF_ALLOW_TOP_LEVEL_AWAIT was clashing
with CO_FUTURE_DIVISION.
2020-04-22 18:09:03 +02:00
Victor Stinner e5014be049
bpo-40268: Remove a few pycore_pystate.h includes (GH-19510) 2020-04-14 17:52:15 +02:00
Victor Stinner da7933ecc3
bpo-40268: Add _PyInterpreterState_GetConfig() (GH-19492)
Don't access PyInterpreterState.config member directly anymore, but
use new functions:

* _PyInterpreterState_GetConfig()
* _PyInterpreterState_SetConfig()
* _Py_GetConfig()
2020-04-13 03:04:28 +02:00
Petr Viktorin ffd9753a94
bpo-39245: Switch to public API for Vectorcall (GH-18460)
The bulk of this patch was generated automatically with:

    for name in \
        PyObject_Vectorcall \
        Py_TPFLAGS_HAVE_VECTORCALL \
        PyObject_VectorcallMethod \
        PyVectorcall_Function \
        PyObject_CallOneArg \
        PyObject_CallMethodNoArgs \
        PyObject_CallMethodOneArg \
    ;
    do
        echo $name
        git grep -lwz _$name | xargs -0 sed -i "s/\b_$name\b/$name/g"
    done

    old=_PyObject_FastCallDict
    new=PyObject_VectorcallDict
    git grep -lwz $old | xargs -0 sed -i "s/\b$old\b/$new/g"

and then cleaned up:

- Revert changes to in docs & news
- Revert changes to backcompat defines in headers
- Nudge misaligned comments
2020-02-11 17:46:57 +01:00
Victor Stinner a102ed7d2f
bpo-39573: Use Py_TYPE() macro in Python and Include directories (GH-18391)
Replace direct access to PyObject.ob_type with Py_TYPE().
2020-02-07 02:24:48 +01:00