Commit Graph

331 Commits

Author SHA1 Message Date
Kumar Aditya 01ef1f95da
GH-89988: Fix memory leak in pickle.Pickler dispatch_table lookup (GH-94298) 2022-06-28 10:01:43 +03:00
Serhiy Storchaka 6fd4c8ec77
gh-93741: Add private C API _PyImport_GetModuleAttrString() (GH-93742)
It combines PyImport_ImportModule() and PyObject_GetAttrString()
and saves 4-6 lines of code on every use.

Add also _PyImport_GetModuleAttr() which takes Python strings as arguments.
2022-06-14 07:15:26 +03:00
Dennis Sweeney 4c496f1f11
gh-92930: _pickle.c: Acquire strong references before calling save() (GH-92931) 2022-06-10 21:50:11 -04:00
Victor Stinner f62ad4f2c4
gh-89653: Use int type for Unicode kind (#92704)
Use the same type that PyUnicode_FromKindAndData() kind parameter
type (public C API): int.
2022-05-13 12:41:05 +02:00
Victor Stinner d716a0dfe2
Use static inline function Py_EnterRecursiveCall() (#91988)
Currently, calling Py_EnterRecursiveCall() and
Py_LeaveRecursiveCall() may use a function call or a static inline
function call, depending if the internal pycore_ceval.h header file
is included or not. Use a different name for the static inline
function to ensure that the static inline function is always used in
Python internals for best performance. Similar approach than
PyThreadState_GET() (function call) and _PyThreadState_GET() (static
inline function).

* Rename _Py_EnterRecursiveCall() to _Py_EnterRecursiveCallTstate()
* Rename _Py_LeaveRecursiveCall() to _Py_LeaveRecursiveCallTstate()
* pycore_ceval.h: Rename Py_EnterRecursiveCall() to
  _Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() and
  _Py_LeaveRecursiveCall()
2022-05-04 13:30:23 +02:00
Victor Stinner 7cdaf87ec5
gh-91731: Replace Py_BUILD_ASSERT() with static_assert() (#91730)
Python 3.11 now uses C11 standard which adds static_assert()
to <assert.h>.

* In pytime.c, replace Py_BUILD_ASSERT() with preprocessor checks on
  SIZEOF_TIME_T with #error.
* On macOS, py_mach_timebase_info() now accepts timebase members with
  the same size than _PyTime_t.
* py_get_monotonic_clock() now saturates GetTickCount64() to
  _PyTime_MAX: GetTickCount64() is unsigned, whereas _PyTime_t is
  signed.
2022-04-20 19:26:40 +02:00
Kumar Aditya ab0d35d70d
bpo-46712: share more global strings in deepfreeze (gh-32152)
(for gh-90868)
2022-04-19 11:41:36 -06:00
Victor Stinner 882d8096c2
bpo-46906: Add PyFloat_Pack8() to the C API (GH-31657)
Add new functions to pack and unpack C double (serialize and
deserialize):

* PyFloat_Pack2(), PyFloat_Pack4(), PyFloat_Pack8()
* PyFloat_Unpack2(), PyFloat_Unpack4(), PyFloat_Unpack8()

Document these functions and add unit tests.

Rename private functions and move them from the internal C API
to the public C API:

* _PyFloat_Pack2() => PyFloat_Pack2()
* _PyFloat_Pack4() => PyFloat_Pack4()
* _PyFloat_Pack8() => PyFloat_Pack8()
* _PyFloat_Unpack2() => PyFloat_Unpack2()
* _PyFloat_Unpack4() => PyFloat_Unpack4()
* _PyFloat_Unpack8() => PyFloat_Unpack8()

Replace the "unsigned char*" type with "char*" which is more common
and easy to use.
2022-03-12 00:10:02 +01:00
Eric Snow 81c72044a1
bpo-46541: Replace core use of _Py_IDENTIFIER() with statically initialized global objects. (gh-30928)
We're no longer using _Py_IDENTIFIER() (or _Py_static_string()) in any core CPython code.  It is still used in a number of non-builtin stdlib modules.

The replacement is: PyUnicodeObject (not pointer) fields under _PyRuntimeState, statically initialized as part of _PyRuntime.  A new _Py_GET_GLOBAL_IDENTIFIER() macro facilitates lookup of the fields (along with _Py_GET_GLOBAL_STRING() for non-identifier strings).

https://bugs.python.org/issue46541#msg411799 explains the rationale for this change.

The core of the change is in:

* (new) Include/internal/pycore_global_strings.h - the declarations for the global strings, along with the macros
* Include/internal/pycore_runtime_init.h - added the static initializers for the global strings
* Include/internal/pycore_global_objects.h - where the struct in pycore_global_strings.h is hooked into _PyRuntimeState
* Tools/scripts/generate_global_objects.py - added generation of the global string declarations and static initializers

I've also added a --check flag to generate_global_objects.py (along with make check-global-objects) to check for unused global strings.  That check is added to the PR CI config.

The remainder of this change updates the core code to use _Py_GET_GLOBAL_IDENTIFIER() instead of _Py_IDENTIFIER() and the related _Py*Id functions (likewise for _Py_GET_GLOBAL_STRING() instead of _Py_static_string()).  This includes adding a few functions where there wasn't already an alternative to _Py*Id(), replacing the _Py_Identifier * parameter with PyObject *.

The following are not changed (yet):

* stop using _Py_IDENTIFIER() in the stdlib modules
* (maybe) get rid of _Py_IDENTIFIER(), etc. entirely -- this may not be doable as at least one package on PyPI using this (private) API
* (maybe) intern the strings during runtime init

https://bugs.python.org/issue46541
2022-02-08 13:39:07 -07:00
Christian Heimes 03e9f5dc75
bpo-43974: Move Py_BUILD_CORE_MODULE into module code (GH-29157)
setup.py no longer defines Py_BUILD_CORE_MODULE. Instead every
module defines the macro before #include "Python.h" unless
Py_BUILD_CORE_BUILTIN is already defined.

Py_BUILD_CORE_BUILTIN is defined for every module that is built by
Modules/Setup.

The PR also simplifies Modules/Setup. Makefile and makesetup
already define Py_BUILD_CORE_BUILTIN and include Modules/internal
for us.

Signed-off-by: Christian Heimes <christian@python.org>
2021-10-22 15:36:28 +02:00
Victor Stinner 0a883a76cd
bpo-35134: Add Include/cpython/floatobject.h (GH-28957)
Split Include/floatobject.h into sub-files: add
Include/cpython/floatobject.h and
Include/internal/pycore_floatobject.h.
2021-10-14 23:41:06 +02:00
Victor Stinner aac29af678
bpo-45434: pyport.h no longer includes <stdlib.h> (GH-28914)
Include <stdlib.h> explicitly in C files.

Python.h includes <wchar.h>.
2021-10-13 19:25:53 +02:00
Christian Clauss dd02a696e5
Fix typos in the Modules directory (GH-28761) 2021-10-07 01:34:42 -07:00
Inada Naoki 9ad8f109ac
bpo-44029: Remove Py_UNICODE APIs (GH-25881)
Remove deprecated `Py_UNICODE` APIs: `PyUnicode_Encode`,
`PyUnicode_EncodeUTF7`, `PyUnicode_EncodeUTF8`,
`PyUnicode_EncodeUTF16`, `PyUnicode_EncodeUTF32`,
`PyUnicode_EncodeLatin1`, `PyUnicode_EncodeMBCS`,
`PyUnicode_EncodeDecimal`, `PyUnicode_EncodeRawUnicodeEscape`,
`PyUnicode_EncodeCharmap`, `PyUnicode_EncodeUnicodeEscape`,
`PyUnicode_TransformDecimalToASCII`, `PyUnicode_TranslateCharmap`,
`PyUnicodeEncodeError_Create`, `PyUnicodeTranslateError_Create`.

See :pep:`393` and :pep:`624` for reference.
2021-05-07 15:58:29 +09:00
Victor Stinner cdad2724e6
bpo-40137: Add pycore_moduleobject.h internal header (GH-25507)
Add pycore_moduleobject.h internal header file with static inline
functions to access module members:

* _PyModule_GetDict()
* _PyModule_GetDef()
* _PyModule_GetState()

These functions don't check at runtime if their argument has a valid
type and can be inlined even if Python is not built with LTO.

_PyType_GetModuleByDef() uses _PyModule_GetDef().

Replace PyModule_GetState() with _PyModule_GetState() in the
extension modules, considered as performance sensitive:

* _abc
* _functools
* _operator
* _pickle
* _queue
* _random
* _sre
* _struct
* _thread
* _winapi
* array
* posix

The following extensions are now built with the Py_BUILD_CORE_MODULE
macro defined, to be able to use the internal pycore_moduleobject.h
header: _abc, array, _operator, _queue, _sre, _struct.
2021-04-22 00:52:52 +02:00
Victor Stinner 32bd68c839
bpo-42519: Replace PyObject_MALLOC() with PyObject_Malloc() (GH-23587)
No longer use deprecated aliases to functions:

* Replace PyObject_MALLOC() with PyObject_Malloc()
* Replace PyObject_REALLOC() with PyObject_Realloc()
* Replace PyObject_FREE() with PyObject_Free()
* Replace PyObject_Del() with PyObject_Free()
* Replace PyObject_DEL() with PyObject_Free()
2020-12-01 10:37:39 +01:00
Victor Stinner 00d7abd7ef
bpo-42519: Replace PyMem_MALLOC() with PyMem_Malloc() (GH-23586)
No longer use deprecated aliases to functions:

* Replace PyMem_MALLOC() with PyMem_Malloc()
* Replace PyMem_REALLOC() with PyMem_Realloc()
* Replace PyMem_FREE() with PyMem_Free()
* Replace PyMem_Del() with PyMem_Free()
* Replace PyMem_DEL() with PyMem_Free()

Modify also the PyMem_DEL() macro to use directly PyMem_Free().
2020-12-01 09:56:42 +01:00
Serhiy Storchaka b510e101f8
bpo-42152: Use PyDict_Contains and PyDict_SetDefault if appropriate. (GH-22986)
If PyDict_GetItemWithError is only used to check whether the key is in dict,
it is better to use PyDict_Contains instead.

And if it is used in combination with PyDict_SetItem, PyDict_SetDefault can
replace the combination.
2020-10-26 12:47:57 +02:00
Serhiy Storchaka b4c98ed41e
bpo-41288: Refactor of unpickling NEWOBJ and NEWOBJ_EX opcodes. (GH-21472)
* Share code for NEWOBJ and NEWOBJ_EX.
* More detailed error messages.
2020-07-18 11:11:21 +03:00
Serhiy Storchaka 4f309abf55
bpo-41288: Fix a crash in unpickling invalid NEWOBJ_EX. (GH-21458)
Automerge-Triggered-By: @tiran
2020-07-13 05:49:26 -07:00
Victor Stinner d36cf5f1d2
bpo-40943: Replace PY_FORMAT_SIZE_T with "z" (GH-20781)
The PEP 353, written in 2005, introduced PY_FORMAT_SIZE_T. Python no
longer supports macOS 10.4 and Visual Studio 2010, but requires more
recent macOS and Visual Studio versions. In 2020 with Python 3.10, it
is now safe to use directly "%zu" to format size_t and "%zi" to
format Py_ssize_t.
2020-06-10 18:38:05 +02:00
Serhiy Storchaka 531d1e5412
bpo-39435: Make the first argument of pickle.loads() positional-only. (GH-19846)
It was positional-only de facto: documentation and two implementations
used three different name.
2020-05-02 09:38:01 +03:00
Victor Stinner 4a21e57fe5
bpo-40268: Remove unused structmember.h includes (GH-19530)
If only offsetof() is needed: include stddef.h instead.

When structmember.h is used, add a comment explaining that
PyMemberDef is used.
2020-04-15 02:35:41 +02:00
Serhiy Storchaka cd8295ff75
bpo-39943: Add the const qualifier to pointers on non-mutable PyUnicode data. (GH-19345) 2020-04-11 10:48:40 +03:00
Dong-hee Na 37fcbb65d4
bpo-40024: Update C extension modules to use PyModule_AddType() (GH-19119)
Update _asyncio, _bz2, _csv, _curses, _datetime,
_io, _operator, _pickle, _queue, blake2,
multibytecodec and overlapped C extension modules
to use PyModule_AddType().
2020-03-24 23:08:51 +01:00
Andy Lester dffe4c0709
bpo-39573: Finish converting to new Py_IS_TYPE() macro (GH-18601) 2020-03-04 14:15:20 +01:00
Hai Shi 1f577ce363
bpo-39378: partial of PickleState struct should be traversed. (GH-18046) 2020-03-02 08:28:44 +02:00
Antoine Pitrou 9f37872e30
bpo-39681: Fix C pickle regression with minimal file-like objects (#18592)
Fix a regression where the C pickle module wouldn't allow unpickling from a
file-like object that doesn't expose a readinto() method.
2020-02-23 23:33:53 +01:00
Petr Viktorin ffd9753a94
bpo-39245: Switch to public API for Vectorcall (GH-18460)
The bulk of this patch was generated automatically with:

    for name in \
        PyObject_Vectorcall \
        Py_TPFLAGS_HAVE_VECTORCALL \
        PyObject_VectorcallMethod \
        PyVectorcall_Function \
        PyObject_CallOneArg \
        PyObject_CallMethodNoArgs \
        PyObject_CallMethodOneArg \
    ;
    do
        echo $name
        git grep -lwz _$name | xargs -0 sed -i "s/\b_$name\b/$name/g"
    done

    old=_PyObject_FastCallDict
    new=PyObject_VectorcallDict
    git grep -lwz $old | xargs -0 sed -i "s/\b$old\b/$new/g"

and then cleaned up:

- Revert changes to in docs & news
- Revert changes to backcompat defines in headers
- Nudge misaligned comments
2020-02-11 17:46:57 +01:00
Victor Stinner 60ac6ed557
bpo-39573: Use Py_SET_SIZE() function (GH-18402)
Replace direct acccess to PyVarObject.ob_size with usage of
the Py_SET_SIZE() function.
2020-02-07 23:18:08 +01:00
Victor Stinner daa9756cb6
bpo-39573: Use Py_TYPE() macro in Modules directory (GH-18393)
Replace direct access to PyObject.ob_type with Py_TYPE().
2020-02-07 03:37:06 +01:00
Pierre Glaser 0f2f35e15f
bpo-39492: Fix a reference cycle between reducer_override and a Pickler instance (GH-18266)
This also needs a backport to 3.8


https://bugs.python.org/issue39492



Automerge-Triggered-By: @pitrou
2020-02-02 10:55:21 -08:00
Mark Dickinson e9652e8d58 bpo-39426: Fix outdated default and highest protocols in docs (GH-18154)
Some portions of the pickle documentation hadn't been updated for the pickle protocol changes in Python 3.8 (new protocol 5, default protocol 4). This PR fixes those docs.


https://bugs.python.org/issue39426
2020-01-24 02:03:22 -08:00
Claudiu Popa 6f03b236c1 bpo-38876: Raise pickle.UnpicklingError when loading an item from memo for invalid input (GH-17335)
The previous code was raising a `KeyError` for both the Python and C implementation.
This was caused by the specified index of an invalid input which did not exist
in the memo structure, where the pickle stores what objects it has seen.
The malformed input would have caused either a `BINGET` or `LONG_BINGET` load
from the memo, leading to a `KeyError` as the determined index was bogus.

https://bugs.python.org/issue38876



https://bugs.python.org/issue38876
2019-11-24 11:15:08 -08:00
Serhiy Storchaka 279f44678c
bpo-37206: Unrepresentable default values no longer represented as None. (GH-13933)
In ArgumentClinic, value "NULL" should now be used only for unrepresentable default values
(like in the optional third parameter of getattr). "None" should be used if None is accepted
as argument and passing None has the same effect as not passing the argument at all.
2019-09-14 12:24:05 +03:00
Serhiy Storchaka 41c57b3353
bpo-37994: Fix silencing all errors if an attribute lookup fails. (GH-15630)
Only AttributeError should be silenced.
2019-09-01 12:03:39 +03:00
Markus Mohrhard 898318b53d bpo-37502: handle default parameter for buffers argument of pickle.loads correctly (GH-14593) 2019-07-25 18:00:34 +02:00
Min ho Kim 96e12d5f4f Fix typos in docs, comments and test assert messages (#14872) 2019-07-21 16:12:33 -04:00
Jeroen Demeyer 59ad110d7a bpo-37547: add _PyObject_CallMethodOneArg (GH-14685) 2019-07-11 17:59:05 +09:00
Jeroen Demeyer 762f93ff2e bpo-37337: Add _PyObject_CallMethodNoArgs() (GH-14267) 2019-07-08 17:19:25 +09:00
Jeroen Demeyer 196a530e00 bpo-37483: add _PyObject_CallOneArg() function (#14558) 2019-07-04 19:31:34 +09:00
Victor Stinner 2ff58a24e8
bpo-37194: Add a new public PyObject_CallNoArgs() function (GH-13890)
Add a new public PyObject_CallNoArgs() function to the C API: call a
callable Python object without any arguments.

It is the most efficient way to call a callback without any argument.
On x86-64, for example, PyObject_CallFunctionObjArgs(func, NULL)
allocates 960 bytes on the stack per call, whereas
PyObject_CallNoArgs(func) only allocates 624 bytes per call.

It is excluded from stable ABI 3.8.

Replace private _PyObject_CallNoArg() with public
PyObject_CallNoArgs() in C extensions: _asyncio, _datetime,
_elementtree, _pickle, _tkinter and readline.
2019-06-17 14:27:23 +02:00
Serhiy Storchaka 38ab7d4721
bpo-31829: Make protocol 0 pickles be loadable in text mode in Python 2. (GH-11859)
Escape ``\r``, ``\0`` and ``\x1a`` (end-of-file on Windows) in Unicode strings.
2019-05-31 11:29:39 +03:00
Jeroen Demeyer 530f506ac9 bpo-36974: tp_print -> tp_vectorcall_offset and tp_reserved -> tp_as_async (GH-13464)
Automatically replace
tp_print -> tp_vectorcall_offset
tp_compare -> tp_as_async
tp_reserved -> tp_as_async
2019-05-30 19:13:39 -07:00
Antoine Pitrou 91f4380ced
bpo-36785: PEP 574 implementation (GH-7076) 2019-05-26 17:10:09 +02:00
Steve Dower b82e17e626
bpo-36842: Implement PEP 578 (GH-12613)
Adds sys.audit, sys.addaudithook, io.open_code, and associated C APIs.
2019-05-23 08:45:22 -07:00
Pierre Glaser 289f1f80ee bpo-35900: Enable custom reduction callback registration in _pickle (GH-12499)
Enable custom reduction callback registration for functions and classes in
_pickle.c, using the new Pickler's attribute ``reducer_override``.
2019-05-08 23:08:25 +02:00
Pierre Glaser 65d98d0f53 bpo-35900: Add a state_setter arg to save_reduce (GH-12588)
Allow reduction methods to return a 6-item tuple where the 6th item specifies a
custom state-setting method that's called instead of the regular
``__setstate__`` method.
2019-05-08 21:40:25 +02:00
Zackery Spytz 359bd4f61b bpo-18372: Add missing PyObject_GC_Track() calls in the pickle module (GH-8505) 2019-04-23 20:56:08 +09:00
Victor Stinner 5c75f37d47
bpo-36635: Change pyport.h for Py_BUILD_CORE_MODULE define (GH-12853)
Change PyAPI_FUNC(type), PyAPI_DATA(type) and PyMODINIT_FUNC macros
of pyport.h when Py_BUILD_CORE_MODULE is defined.

The Py_BUILD_CORE_MODULE define must be now be used to build a C
extension as a dynamic library accessing Python internals: export the
PyInit_xxx() function in DLL exports on Windows.

Changes:

* Py_BUILD_CORE_BUILTIN and Py_BUILD_CORE_MODULE now imply
  Py_BUILD_CORE directy in pyport.h.
* ceval.c compilation now fails with an error if Py_BUILD_CORE is not
  defined, just to ensure that Python is build with the correct
  defines.
* setup.py now compiles _pickle.c with Py_BUILD_CORE_MODULE define.
* setup.py compiles _json.c with Py_BUILD_CORE_MODULE define, rather
  than Py_BUILD_CORE_BUILTIN define
* PCbuild/pythoncore.vcxproj: Add Py_BUILD_CORE_BUILTIN define.
2019-04-17 23:02:26 +02:00