Commit Graph

6508 Commits

Author SHA1 Message Date
Victor Stinner bcda8f1d42
bpo-35081: Add Include/internal/pycore_object.h (GH-10640)
Move _PyObject_GC_TRACK() and _PyObject_GC_UNTRACK() from
Include/objimpl.h to Include/internal/pycore_object.h.
2018-11-21 22:27:47 +01:00
Zackery Spytz 062a57bf4b bpo-35269: Fix a possible segfault involving a newly-created coroutine (GH-10585)
coro->cr_origin wasn't initialized if compute_cr_origin() failed in
PyCoro_New(), which would cause a crash during the coroutine's
deallocation.



https://bugs.python.org/issue35269
2018-11-18 08:45:57 -08:00
Pablo Galindo bfb855bef6
bpo-34784: Implement correct cleanup in PyStructSequence new implementation (GH-10536)
PyTuple_Pack can fail and return NULL. If this happens, then PyType_FromSpecWithBases will incorrectly create a new type without bases. Also, it will crash on the Py_DECREF that follows. Also free members and type in error conditions.
2018-11-15 02:03:57 +00:00
INADA Naoki a757649fd6
bpo-35230: dict: Remove some macros (GH-10513)
Remove _Py_REF_DEBUG_COMMA, DK_DEBUG_INCREF, and DK_DEBUG_DECREF.
Convert DK_INCREF and DK_DECREF to static inline functions.
2018-11-14 18:39:27 +09:00
Gregory P. Smith 746b2d35ea
bpo-35214: Fix OOB memory access in unicode escape parser (GH-10506)
Discovered using clang's MemorySanitizer when it ran python3's
test_fstring test_misformed_unicode_character_name.

An msan build will fail by simply executing: ./python -c 'u"\N"'
2018-11-13 13:16:54 -08:00
Eddie Elizondo 474eedfb3d bpo-34784: Fix PyStructSequence_NewType with heap-allocated StructSequence (GH-9665) 2018-11-13 13:09:31 +01:00
Gregory P. Smith 3015fb8ce4
bpo-35214: Add _Py_ prefix to MEMORY_SANITIZER def. (GH-10503)
Rename our new MEMORY_SANITIZER define to _Py_MEMORY_SANITIZER.
Project based C Preprocessor namespacing at its finest. :P
2018-11-12 22:01:22 -08:00
Gregory P. Smith e6c77d8301
Disable getc_unlocked() with MemorySanitizer. (GH-10499)
clang's MemorySanitizer understand getc() but does not understand
getc_unlocked().  Workaround: Don't use it on msan builds.
2018-11-12 19:47:13 -08:00
Victor Stinner 621cebe81b
bpo-35081: Rename internal headers (GH-10275)
Rename Include/internal/ headers:

* pycore_hash.h -> pycore_pyhash.h
* pycore_lifecycle.h -> pycore_pylifecycle.h
* pycore_mem.h -> pycore_pymem.h
* pycore_state.h -> pycore_pystate.h

Add missing headers to Makefile.pre.in and PCbuild:

* pycore_condvar.h.
* pycore_hamt.h
* pycore_pyhash.h
2018-11-12 16:53:38 +01:00
Alexey Izbyshev fd3a91cbf9 closes bpo-35204: Disable thread and memory sanitizers for address_in_range(). (GH-10442)
This function may access memory which is mapped but is considered
free by libc allocator. It behaves so by design, therefore we
need to suppress sanitizer reports.

GCC doesn't support MSan, so disable only TSan for it.
2018-11-11 15:14:51 -08:00
Raymond Hettinger f9ec1b9f52
Neaten the code without any algorithmic change. (GH-10466)
Remove unneeded assertion (we already know so is a PySetObject *).
2018-11-11 14:35:47 -08:00
Victor Stinner d17a693fa0
bpo-35199: Add an internal _PyTuple_ITEMS() macro (GH-10434)
* _PyTuple_ITEMS() gives access to the tuple->ob_item field and cast the
  first argument to PyTupleObject*. This internal macro is only usable if
  Py_BUILD_CORE is defined.
* Replace &PyTuple_GET_ITEM(ob, 0) with _PyTuple_ITEMS(ob).
* Replace PyTuple_GET_ITEM(op, 1) with &_PyTuple_ITEMS(ob)[1].
2018-11-09 16:56:48 +01:00
Raymond Hettinger cf5863faab Optimize set.pop() to advance a pointer instead of indexing. (GH-10429)
Gives approx 20% speed-up using clang depending on the number of elements in the set (the less dense the set, the more the speed-up).

Uses the same entry++ logic used elsewhere in the setobject.c code.
2018-11-09 02:31:56 -08:00
Victor Stinner 9fc57a3848
bpo-35081: Add pycore_fileutils.h (GH-10371)
Move Py_BUILD_CORE code from Include/fileutils.h to a new
Include/internal/pycore_fileutils.h file.
2018-11-07 00:44:03 +01:00
Rémi Lapeyre 6531bf6309 bpo-33462: Add __reversed__ to dict and dict views (GH-6827) 2018-11-06 09:38:54 +09:00
Victor Stinner e281f7d80c
bpo-35081: Move accu.h to Include/internal/pycore_accu.h (GH-10271)
The accu.h header is no longer part of the Python C API: it has been
moved to the "internal" headers which are restricted to Python
itself.

Replace #include "accu.h" with #include "pycore_accu.h".
2018-11-01 02:30:36 +01:00
Victor Stinner 50b48572d9
bpo-35081: Add _PyThreadState_GET() internal macro (GH-10266)
If Py_BUILD_CORE is defined, the PyThreadState_GET() macro access
_PyRuntime which comes from the internal pycore_state.h header.
Public headers must not require internal headers.

Move PyThreadState_GET() and _PyInterpreterState_GET_UNSAFE() from
Include/pystate.h to Include/internal/pycore_state.h, and rename
PyThreadState_GET() to _PyThreadState_GET() there.

The PyThreadState_GET() macro of pystate.h is now redefined when
pycore_state.h is included, to use the fast _PyThreadState_GET().

Changes:

* Add _PyThreadState_GET() macro
* Replace "PyThreadState_GET()->interp" with
  _PyInterpreterState_GET_UNSAFE()
* Replace PyThreadState_GET() with _PyThreadState_GET() in internal C
  files (compiled with Py_BUILD_CORE defined), but keep
  PyThreadState_GET() in the public header files.
* _testcapimodule.c: replace PyThreadState_GET() with
  PyThreadState_Get(); the module is not compiled with Py_BUILD_CORE
  defined.
* pycore_state.h now requires Py_BUILD_CORE to be defined.
2018-11-01 01:51:40 +01:00
Victor Stinner 27e2d1f219
bpo-35081: Add pycore_ prefix to internal header files (GH-10263)
* Rename Include/internal/ header files:

  * pyatomic.h -> pycore_atomic.h
  * ceval.h -> pycore_ceval.h
  * condvar.h -> pycore_condvar.h
  * context.h -> pycore_context.h
  * pygetopt.h -> pycore_getopt.h
  * gil.h -> pycore_gil.h
  * hamt.h -> pycore_hamt.h
  * hash.h -> pycore_hash.h
  * mem.h -> pycore_mem.h
  * pystate.h -> pycore_state.h
  * warnings.h -> pycore_warnings.h

* PCbuild project, Makefile.pre.in, Modules/Setup: add the
  Include/internal/ directory to the search paths of header files.
* Update includes. For example, replace #include "internal/mem.h"
  with #include "pycore_mem.h".
2018-11-01 00:52:28 +01:00
Victor Stinner 2be00d987d
bpo-35081: Move Py_BUILD_CORE code to internal/mem.h (GH-10249)
* Add #include "internal/mem.h" to C files using
  _PyMem_SetDefaultAllocator().
* Include/internal/mem.h now requires Py_BUILD_CORE to be defined.
2018-10-31 20:19:24 +01:00
Zackery Spytz c9a6168924 Fix a possible crash in range.__reversed__(). (GH-10252) 2018-10-31 11:13:16 +02:00
Serhiy Storchaka 0353b4eaaf
bpo-33138: Change standard error message for non-pickleable and non-copyable types. (GH-6239) 2018-10-31 02:28:07 +02:00
Victor Stinner 9204fb8623
bpo-35081: Cleanup pystate.c and pystate.h (GH-10240)
* Remove _PyThreadState_Current
* Replace GET_TSTATE() with PyThreadState_GET()
* Replace GET_INTERP_STATE() with _PyInterpreterState_GET_UNSAFE()
* Replace direct access to _PyThreadState_Current with
  PyThreadState_GET()
* Replace _PyThreadState_Current with
  _PyRuntime.gilstate.tstate_current
* Rename SET_TSTATE() to _PyThreadState_SET(), name more
  consistent with _PyThreadState_GET()
* Update outdated comments
2018-10-30 15:13:17 +01:00
Victor Stinner 3c09dca4b5
bpo-35059: Convert _Py_Dealloc() to static inline function (GH-10223)
Convert _Py_Dealloc() macro into a static inline function. Moreover,
it is now also defined as a static inline function if Py_TRACE_REFS
is defined.
2018-10-30 14:48:26 +01:00
Serhiy Storchaka 3e429dcc24
bpo-33237: Improve AttributeError message for partially initialized module. (GH-6398) 2018-10-30 13:19:51 +02:00
Pablo Galindo 372d705d95
bpo-33234 Improve list() pre-sizing for inputs with known lengths (GH-9846)
The list() constructor isn't taking full advantage of known input
lengths or length hints. This commit makes the constructor
pre-size and not over-allocate when the input size is known (the
input collection implements __len__). One on the main advantages is
that this provides 12% difference in memory savings due to the difference
between overallocating and allocating exactly the input size.

For efficiency purposes and to avoid a performance regression for small
generators and collections, the size of the input object is calculated using
__len__ and not __length_hint__, as the later is considerably slower.
2018-10-28 20:16:26 +00:00
Pablo Galindo 49c75a8086
bpo-35064 prefix smelly symbols that appear with COUNT_ALLOCS with _Py_ (GH-10152)
Configuring python with ./configure --with-pydebug CFLAGS="-D COUNT_ALLOCS -O0"
makes "make smelly" fail as some symbols were being exported without the "Py_" or
"_Py" prefixes.
2018-10-28 15:02:17 +00:00
jdemeyer aeb1be5868 bpo-34751: improved hash function for tuples (GH-9471) 2018-10-27 20:06:38 -04:00
Victor Stinner 50fe3f8913
bpo-9263: _PyXXX_CheckConsistency() use _PyObject_ASSERT() (GH-10108)
Use _PyObject_ASSERT() in:

* _PyDict_CheckConsistency()
* _PyType_CheckConsistency()
* _PyUnicode_CheckConsistency()

_PyObject_ASSERT() dumps the faulty object if the assertion fails
to help debugging.
2018-10-26 18:47:15 +02:00
Victor Stinner 0862505a03
bpo-9263: Use _PyObject_ASSERT() in typeobject.c (GH-10111)
Replace assert() with _PyObject_ASSERT() in Objects/typeobject.c
to dump the faulty object on assertion failure to ease debugging.
2018-10-26 18:39:11 +02:00
Victor Stinner 24702044af
bpo-9263: Use _PyObject_ASSERT() in object.c (GH-10110)
Replace assert() with _PyObject_ASSERT() in Objects/object.c to dump
the faulty object on assertion failure to ease debugging.
2018-10-26 17:16:37 +02:00
Victor Stinner b4435e20a9
bpo-35059: Convert PyObject_INIT() to function (GH-10077)
* Convert PyObject_INIT() and PyObject_INIT_VAR() macros to static
  inline functions.
* Fix usage of these functions: cast to PyObject* or PyVarObject*.
2018-10-26 14:35:00 +02:00
Victor Stinner 3ec9af75f6
bpo-9263: _Py_NegativeRefcount() use _PyObject_AssertFailed() (GH-10109)
_Py_NegativeRefcount() now uses _PyObject_AssertFailed() to dump the
object to help debugging.
2018-10-26 02:12:34 +02:00
Victor Stinner 626bff8568
bpo-9263: Dump Python object on GC assertion failure (GH-10062)
Changes:

* Add _PyObject_AssertFailed() function.
* Add _PyObject_ASSERT() and _PyObject_ASSERT_WITH_MSG() macros.
* gc_decref(): replace assert() with _PyObject_ASSERT_WITH_MSG() to
  dump the faulty object if the assertion fails.

_PyObject_AssertFailed() calls:

* _PyMem_DumpTraceback(): try to log the traceback where the object
  memory has been allocated if tracemalloc is enabled.
* _PyObject_Dump(): log repr(obj).
* Py_FatalError(): log the current Python traceback.

_PyObject_AssertFailed() uses _PyObject_IsFreed() heuristic to check
if the object memory has been freed by a debug hook on Python memory
allocators.

Initial patch written by David Malcolm.

Co-Authored-By: David Malcolm <dmalcolm@redhat.com>
2018-10-25 17:31:10 +02:00
Victor Stinner 18618e652c
bpo-35059: Add Py_STATIC_INLINE() macro (GH-10093)
* Add Py_STATIC_INLINE() macro to declare a "static inline" function.
  If the compiler supports it, try to always inline the function even if no
  optimization level was specified.
* Modify pydtrace.h to use Py_STATIC_INLINE() when WITH_DTRACE is
  not defined.
* Add an unit test on Py_DECREF() to make sure that
  _Py_NegativeRefcount() reports the correct filename.
2018-10-25 17:28:11 +02:00
Victor Stinner 9e00e80e21
bpo-35053: Enhance tracemalloc to trace free lists (GH-10063)
tracemalloc now tries to update the traceback when an object is
reused from a "free list" (optimization for faster object creation,
used by the builtin list type for example).

Changes:

* Add _PyTraceMalloc_NewReference() function which tries to update
  the Python traceback of a Python object.
* _Py_NewReference() now calls _PyTraceMalloc_NewReference().
* Add an unit test.
2018-10-25 13:31:16 +02:00
Serhiy Storchaka c46db9232f
bpo-30863: Rewrite PyUnicode_AsWideChar() and PyUnicode_AsWideCharString(). (GH-2599)
They no longer cache the wchar_t* representation of string objects.
2018-10-23 22:58:24 +03:00
Victor Stinner 82af0b63b0
bpo-9263: _PyObject_Dump() detects freed memory (GH-10061)
_PyObject_Dump() now uses an heuristic to check if the object memory
has been freed: log "<freed object>" in that case.

The heuristic rely on the debug hooks on Python memory allocators
which fills the memory with DEADBYTE (0xDB) when memory is
deallocated. Use PYTHONMALLOC=debug to always enable these debug
hooks.
2018-10-23 17:39:40 +02:00
Eric Lippert 5a95ba29da Fix issue 34551 - remove redundant store (#9009)
The assignment of i/2 to nk is redundant because on this code path, nk is already the size of the dictionary, and i is already twice the size of the dictionary. I've replaced the store with an assertion that i/2 is nk.
2018-10-22 16:52:46 +01:00
Serhiy Storchaka 2c2044e789
bpo-34984: Improve error messages for bytes and bytearray constructors. (GH-9874) 2018-10-21 15:29:12 +03:00
Serhiy Storchaka 914f9a078f
bpo-34973: Fix crash in bytes constructor. (GH-9841)
Constructing bytes from mutating list could cause a crash.
2018-10-21 15:25:53 +03:00
Sergey Fedoseev a5259fb05d bpo-34574: Prevent OrderedDict iterators from exhaustion during pickling. (GH-9051) 2018-10-20 08:20:39 +03:00
Sergey Fedoseev 6395844e6a bpo-34573: Simplify __reduce__() of set and dict iterators. (GH-9050)
Simplify the pickling of set and dictionary objects iterators by consuming
the iterator into a list with PySequence_List.
2018-10-20 01:43:33 +01:00
Serhiy Storchaka 6f17e51345 bpo-33712: OrderedDict only creates od_fast_nodes cache if needed (GH-7349) 2018-10-20 01:27:45 +02:00
Serhiy Storchaka b2e2025941 bpo-33073: Rework int.as_integer_ratio() implementation (GH-9303)
* Simplify the C code.
* Simplify tests and make them more strict and robust.
* Add references in the documentation.
2018-10-19 23:46:31 +02:00
Serhiy Storchaka e890421e33
bpo-34974: Do not replace unexpected errors in bytes() and bytearray(). (GH-9852)
bytes and bytearray constructors converted unexpected exceptions
(e.g. MemoryError and KeyboardInterrupt) to TypeError.
2018-10-15 00:02:57 +03:00
Zackery Spytz a4b48f194a bpo-34940: Fix the error handling in _check_for_legacy_statements(). (GH-9764) 2018-10-12 11:20:59 +03:00
Raymond Hettinger f1aa8aed4a
Micro-optimize list index range checks (GH-9784) 2018-10-10 20:37:28 -07:00
Emanuele Gaifas fc8205cb4b Add missing closing quote and trailing period in str.isidentifier() docstring (GH-9756)
This rectifies commit ffc5a14d00.
2018-10-08 16:14:47 +05:30
Sanyam Khurana ffc5a14d00 bpo-33014: Clarify str.isidentifier docstring (GH-6088)
* bpo-33014: Clarify str.isidentifier docstring

* bpo-33014: Add code example in isidentifier documentation
2018-10-08 12:23:32 +05:30
Zackery Spytz ae62f01524 bpo-34910: Ensure that PyObject_Print() always returns -1 on error. (GH-9733) 2018-10-06 09:44:25 +03:00
Zackery Spytz 7bb9cd0a67 bpo-34899: Fix a possible assertion failure due to int_from_bytes_impl() (GH-9705)
The _PyLong_FromByteArray() call in int_from_bytes_impl() was
unchecked.
2018-10-06 00:02:23 +03:00
Zackery Spytz 96c5932794 bpo-34879: Fix a possible null pointer dereference in bytesobject.c (GH-9683)
formatfloat() was not checking if PyBytes_FromStringAndSize()
failed, which could lead to a null pointer dereference in
_PyBytes_FormatEx().
2018-10-03 09:01:30 +03:00
Victor Stinner e972c13624
bpo-30156: Remove property_descr_get() optimization (GH-9541)
property_descr_get() uses a "cached" tuple to optimize function
calls. But this tuple can be discovered in debug mode with
sys.getobjects(). Remove the optimization, it's not really worth it
and it causes 3 different crashes last years.

Microbenchmark:

./python -m perf timeit -v \
    -s "from collections import namedtuple; P = namedtuple('P', 'x y'); p = P(1, 2)" \
    --duplicate 1024 "p.x"

Result:

Mean +- std dev: [ref] 32.8 ns +- 0.8 ns -> [patch] 40.4 ns +- 1.3 ns: 1.23x slower (+23%)
2018-10-01 03:03:22 -07:00
INADA Naoki 2aaf98c16a bpo-34320: Fix dict(o) didn't copy order of dict subclass (GH-8624)
When dict subclass overrides order (`__iter__()`, `keys()`, and `items()`), `dict(o)`
should use it instead of dict ordering.


https://bugs.python.org/issue34320
2018-09-25 20:59:00 -07:00
Raymond Hettinger 73820a60cc
Fix compiler warning with a type cast (GH-9300) 2018-09-14 01:35:59 -07:00
Raymond Hettinger 00bc08ec11
Fix-up parenthesis, organization, and NULL check (GH-9297) 2018-09-14 01:00:11 -07:00
Lisa Roach 5ac704306f bpo-33073: Adding as_integer_ratio to ints. (GH-8750) 2018-09-13 23:56:23 -07:00
Benjamin Peterson e502451781
closes bpo-34646: Remove PyAPI_* macros from declarations. (GH-9218) 2018-09-12 12:06:42 -07:00
Sergey Fedoseev 6c7d67ce83 bpo-1621: Avoid signed integer overflow in set_table_resize(). (GH-9059)
Address a C undefined behavior signed integer overflow issue in set object table resizing.  Our -fwrapv compiler flag and practical reasons why sets are unlikely to get this large should mean this was never an issue but it was incorrect code that generates code analysis warnings.

<!-- issue-number: [bpo-1621](https://www.bugs.python.org/issue1621) -->
https://bugs.python.org/issue1621
<!-- /issue-number -->
2018-09-11 16:18:01 -07:00
Victor Stinner 998b806366
Revert "bpo-34595: Add %T format to PyUnicode_FromFormatV() (GH-9080)" (GH-9187)
This reverts commit 886483e2b9.
2018-09-12 00:23:25 +02:00
Peter Eisentraut 0e0bc4e221 Fix misleading mentions of tp_size in comments (GH-9093)
Many type object initializations labeled a field "tp_size" in the
comment, but the name of that field is tp_basicsize.
2018-09-10 09:46:08 -07:00
Victor Stinner 886483e2b9
bpo-34595: Add %T format to PyUnicode_FromFormatV() (GH-9080)
* Add %T format to PyUnicode_FromFormatV(), and so to
  PyUnicode_FromFormat() and PyErr_Format(), to format an object type
  name: equivalent to "%s" with Py_TYPE(obj)->tp_name.
* Replace Py_TYPE(obj)->tp_name with %T format in unicodeobject.c.
* Add unit test on %T format.
* Rename unicode_fromformat_write_cstr() to
  unicode_fromformat_write_utf8(), to make the intent more explicit.
2018-09-07 18:00:58 +02:00
jdemeyer 8f735485ac bpo-25750: fix refcounts in type_getattro() (GH-6118)
When calling tp_descr_get(self, obj, type), make sure that
we own a strong reference to "self".
2018-09-07 09:37:00 +02:00
Sergey Fedoseev 593bb30e82 closes bpo-34599: Improve performance of _Py_bytes_capitalize(). (GH-9083) 2018-09-06 21:54:49 -07:00
Victor Stinner 3d4226a832
bpo-34523: Support surrogatepass in locale codecs (GH-8995)
Add support for the "surrogatepass" error handler in
PyUnicode_DecodeFSDefault() and PyUnicode_EncodeFSDefault()
for the UTF-8 encoding.

Changes:

* _Py_DecodeUTF8Ex() and _Py_EncodeUTF8Ex() now support the
  surrogatepass error handler (_Py_ERROR_SURROGATEPASS).
* _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx() now use
  the _Py_error_handler enum instead of "int surrogateescape" to pass
  the error handler. These functions now return -3 if the error
  handler is unknown.
* Add unit tests on _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx()
  in test_codecs.
* Rename get_error_handler() to _Py_GetErrorHandler() and expose it
  as a private function.
* _freeze_importlib doesn't need config.filesystem_errors="strict"
  workaround anymore.
2018-08-29 22:21:32 +02:00
Victor Stinner b2457efc78
bpo-34523: Add _PyCoreConfig.filesystem_encoding (GH-8963)
_PyCoreConfig_Read() is now responsible to choose the filesystem
encoding and error handler. Using Py_Main(), the encoding is now
chosen even before calling Py_Initialize().

_PyCoreConfig.filesystem_encoding is now the reference, instead of
Py_FileSystemDefaultEncoding, for the Python filesystem encoding.

Changes:

* Add filesystem_encoding and filesystem_errors to _PyCoreConfig
* _PyCoreConfig_Read() now reads the locale encoding for the file
  system encoding.
* PyUnicode_EncodeFSDefault() and PyUnicode_DecodeFSDefaultAndSize()
  now use the interpreter configuration rather than
  Py_FileSystemDefaultEncoding and Py_FileSystemDefaultEncodeErrors
  global configuration variables.
* Add _Py_SetFileSystemEncoding() and _Py_ClearFileSystemEncoding()
  private functions to only modify Py_FileSystemDefaultEncoding and
  Py_FileSystemDefaultEncodeErrors in coreconfig.c.
* _Py_CoerceLegacyLocale() now takes an int rather than
  _PyCoreConfig for the warning.
2018-08-29 13:25:36 +02:00
Alexey Izbyshev b57b4ac042 closes bpo-34504: Remove the useless NULL check in PySequence_Check(). (GH-8935)
Reported by Svace static analyzer.
2018-08-25 16:52:27 -07:00
Alexey Izbyshev 5f79b50763 closes bpo-34501: PyType_FromSpecWithBases: Check spec->name before dereferencing it. (GH-8930)
Reported by Svace static analyzer.
2018-08-25 11:53:47 -07:00
Alexey Izbyshev 8fdd331bbf closes bpo-34493: Objects/genobject.c: Add missing NULL check to compute_cr_origin() (GH-8911) 2018-08-25 00:15:23 -07:00
Alexey Izbyshev 7ecae3ca0b closes bpo-34468: Objects/rangeobject.c: Fix an always-false condition in range_repr() (GH-8880)
Also, propagate the error from PyNumber_AsSsize_t() because we don't care
only about OverflowError which is not reported if the second argument is NULL.

Reported by Svace static analyzer.
2018-08-23 21:39:45 -07:00
Alexey Izbyshev f6247aac08 closes bpo-34477: Objects/typeobject.c: Add missing NULL check to type_init() (GH-8876)
Reported by Svace static analyzer.
2018-08-23 21:22:16 -07:00
Alexey Izbyshev ccd9975267 bpo-34436: Fix check that disables overallocation for the last fmt specifier (GH-8826)
Reported by Svace static analyzer.
2018-08-23 09:50:52 +02:00
Alexey Izbyshev 74a307d48e bpo-34435: Add missing NULL check to unicode_encode_ucs1(). (GH-8823)
Reported by Svace static analyzer.
2018-08-19 21:52:04 +03:00
Zackery Spytz e349bf2358 bpo-22602: Raise an exception in the UTF-7 decoder for ill-formed sequences starting with "+". (GH-8741)
The UTF-7 decoder now raises UnicodeDecodeError for ill-formed
sequences starting with "+" (as specified in RFC 2152).
2018-08-19 07:43:38 +03:00
Sergey Fedoseev 2fc46979b8 bpo-34151: Improve performance of some list operations (GH-8332) 2018-08-11 21:12:07 +08:00
Victor Stinner caba55b3b7
bpo-34301: Add _PyInterpreterState_Get() helper function (GH-8592)
sys_setcheckinterval() now uses a local variable to parse arguments,
before writing into interp->check_interval.
2018-08-03 15:33:52 +02:00
Serhiy Storchaka ac20e0f98d
bpo-1617161: Make the hash and equality of methods not depending on the value of self. (GH-7848)
* The hash of BuiltinMethodType instances no longer depends on the hash
  of __self__. It depends now on the hash of id(__self__).
* The hash and equality of ModuleType and MethodWrapperType instances no
  longer depend on the hash and equality of __self__. They depend now on
  the hash and equality of id(__self__).
* MethodWrapperType instances no longer support ordering.
2018-07-31 09:18:24 +03:00
Serhiy Storchaka 7cb7bcff20
bpo-20260: Implement non-bitwise unsigned int converters for Argument Clinic. (GH-8434) 2018-07-26 13:22:16 +03:00
Victor Stinner 53b7d4e402
bpo-34170: Add _PyCoreConfig.bytes_warning (GH-8447)
Add more fields to _PyCoreConfig:

* _check_hash_pycs_mode
* bytes_warning
* debug
* inspect
* interactive
* legacy_windows_fs_encoding
* legacy_windows_stdio
* optimization_level
* quiet
* unbuffered_stdio
* user_site_directory
* verbose
* write_bytecode

Changes:

* Remove pymain_get_global_config() and pymain_set_global_config()
  which became useless. These functions have been replaced by
  _PyCoreConfig_GetGlobalConfig() and
  _PyCoreConfig_SetGlobalConfig().
* sys.flags.dont_write_bytecode value is now restricted to 1 even if
  -B option is specified multiple times on the command line.
* PyThreadState_Clear() now uses the config from the current
  interpreter rather than using global Py_VerboseFlag
2018-07-25 01:37:05 +02:00
Serhiy Storchaka c5734998d9
bpo-33720: Refactor marshalling/unmarshalling floats. (GH-8071) 2018-07-24 10:55:47 +03:00
Serhiy Storchaka a6fdddb7df
bpo-32500: Fix error messages for sequence and mapping C API. (GH-7846)
Fix error messages for PySequence_Size(), PySequence_GetItem(),
PySequence_SetItem() and PySequence_DelItem() called with a mapping
and PyMapping_Size() called with a sequence.
2018-07-23 23:43:42 +03:00
Serhiy Storchaka bd47384e07
bpo-24618: Add a check in the code constructor. (GH-8283)
Check that the size of the varnames tuple is enough at least for all arguments.
2018-07-16 09:10:19 +03:00
INADA Naoki 16dfca4d82
bpo-34087: Fix buffer overflow in int(s) and similar functions (GH-8274)
`_PyUnicode_TransformDecimalAndSpaceToASCII()` missed trailing NUL char.
It caused buffer overflow in `_Py_string_to_number_with_underscores()`.

This bug is introduced in 9b6c60cb.
2018-07-14 12:06:43 +09:00
INADA Naoki 5ac9e6eee5
bpo-33597: Reduce PyGC_Head size (GH-7043) 2018-07-10 17:19:53 +09:00
Yury Selivanov 0b75228700
bpo-34042: Fix dict.copy() to maintain correct total refcount (GH-8119) 2018-07-06 12:20:07 -04:00
Robert Krzyzanowski 6f19fc6d56 fix two typos in Objects/odictobject.c comments (GH-8040) 2018-07-06 13:54:26 +03:00
Tal Einat c929df3b96
bpo-20180: complete AC conversion of Objects/stringlib/transmogrify.h (GH-8039)
* converted bytes methods: expandtabs, ljust, rjust, center, zfill
* updated char_convertor to properly set the C default value
2018-07-06 13:17:38 +03:00
INADA Naoki 3c452404ae
bpo-33418: Add tp_clear for function object (GH-8058)
Without tp_clear, GC can't break cyclic reference.
It will cause memory leak when cyclic reference is
created intentionally.
2018-07-04 11:15:50 +09:00
Bup fc93bd467e Change tp_size to tp_basicsize in comment and realign the comments (GH-6775) 2018-06-19 16:59:55 +08:00
Serhiy Storchaka ceeef10cdb
bpo-33818: PyExceptionClass_Name() will now return "const char *". (GH-7581) 2018-06-15 11:09:43 +03:00
Serhiy Storchaka 5cbefa9919
Clean up after bpo-33738. (GH-7627)
* Add declarations even if they are overridden by macros.
* Make the declaration and the definition of PyExceptionClass_Name
  consistent.
2018-06-11 15:01:47 +03:00
Christian Tismer 8398713cea bpo-33738: Address review comments in GH #7477 (GH-7585) 2018-06-10 18:48:28 -04:00
Christian Tismer ea62ce7f4f bpo-33738: Fix macros which contradict PEP 384 (GH-7477)
During development of the limited API support for PySide,
we saw an error in a macro that accessed a type field.

This patch fixes the 7 errors in the Python headers.
Macros which were not written as capitals were implemented
as function.

To do the necessary analysis again, a script was included that
parses all headers and looks for "->tp_" in serctions which can
be reached with active limited API.

It is easily possible to call this script as a test.

Error listing:

../../Include/objimpl.h:243
#define PyObject_IS_GC(o) (PyType_IS_GC(Py_TYPE(o)) && \
    (Py_TYPE(o)->tp_is_gc == NULL || Py_TYPE(o)->tp_is_gc(o)))
Action: commented only

../../Include/objimpl.h:362
#define PyType_SUPPORTS_WEAKREFS(t) ((t)->tp_weaklistoffset > 0)
Action: commented only

../../Include/objimpl.h:364
#define PyObject_GET_WEAKREFS_LISTPTR(o) \
    ((PyObject **) (((char *) (o)) + Py_TYPE(o)->tp_weaklistoffset))
Action: commented only

../../Include/pyerrors.h:143
#define PyExceptionClass_Name(x) \
     ((char *)(((PyTypeObject*)(x))->tp_name))
Action: implemented function

../../Include/abstract.h:593
#define PyIter_Check(obj) \
    ((obj)->ob_type->tp_iternext != NULL && \
     (obj)->ob_type->tp_iternext != &_PyObject_NextNotImplemented)
Action: implemented function

../../Include/abstract.h:713
#define PyIndex_Check(obj)                              \
    ((obj)->ob_type->tp_as_number != NULL &&            \
     (obj)->ob_type->tp_as_number->nb_index != NULL)
Action: implemented function

../../Include/abstract.h:924
#define PySequence_ITEM(o, i)\
    ( Py_TYPE(o)->tp_as_sequence->sq_item(o, i) )
Action: commented only
2018-06-09 14:32:25 -04:00
Yury Selivanov 52698c7ad9
bpo-33786: Fix asynchronous generators to handle GeneratorExit in athrow() (GH-7467) 2018-06-07 20:31:26 -04:00
Benjamin Peterson 7c69c1c0fb
update to Unicode 11.0.0 (closes bpo-33778) (GH-7439)
Also, standardize indentation of generated tables.
2018-06-06 20:14:28 -07:00
INADA Naoki d852142cd7
Replace _PyGC_REFS macros with higher level macros (GH-6852)
Only gcmodule.c uses _PyGC_REFS* macros now.
This makes easy to read GC code.
2018-05-17 11:07:21 +09:00
Grant Jenks 0904f766e1 Fix typo in __mul__ and __rmul__ docstring (GH-6674) 2018-05-09 01:00:19 +03:00
lekma 491bbedc20 bpo-33391: Fix refleak in set_symmetric_difference (GH-6670) 2018-05-02 18:29:10 +09:00
Serhiy Storchaka 6405feecda
bpo-33012: Fix invalid function casts for long_long. (GH-6652)
long_long() was used with three function types:
PyCFunction, getter and unaryfunction.
2018-04-30 15:35:08 +03:00
Serhiy Storchaka c1a6832f50
bpo-33330: Write exceptions occurred in PyImport_Cleanup() to stderr. (GH-6606)
They where silenced before.
2018-04-29 22:16:30 +03:00
Siddhesh Poyarekar 55edd0c185 bpo-33012: Fix invalid function cast warnings with gcc 8 for METH_NOARGS. (GH-6030)
METH_NOARGS functions need only a single argument but they are cast
into a PyCFunction, which takes two arguments.  This triggers an
invalid function cast warning in gcc8 due to the argument mismatch.
Fix this by adding a dummy unused argument.
2018-04-29 21:59:33 +03:00
Victor Stinner 078c4e3519
bpo-33041: Fix downcast warning on Windows (#6595)
Cast pointer difference from ssize_t to int: a frame is very unlikely
larger than 2 GB.
2018-04-27 14:30:01 +02:00
Ville Skyttä 61f82e0e33 Spelling fixes to docs, docstrings, and comments (GH-6374) 2018-04-20 16:08:45 -04:00
Gregory P. Smith 397f1b28c4
bpo-33312: Fix clang ubsan out of bounds warnings in dict. (GH-6537)
Fix clang ubsan (undefined behavior sanitizer) warnings in dictobject.c by
adjusting how the internal struct _dictkeysobject shared keys structure is
declared.

This remains ABI compatible.  We get rid of the union at the end of the
struct being used for conveinence to avoid typecasting in favor of char[]
variable length array at the end of a struct. This is known to clang to be
used for variable sized objects and will not cause an undefined behavior
problem.  Similarly, char arrays do not have strict aliasing undefined
behavior when cast.

PEP-007 does not currently list variable length arrays (VLAs) as allowed
in our subset of C99.  If this turns out to be a problem, the fix to this is
to change the char `dk_indices[]` into `dk_indices[1]` and restore the
three size computation subtractions this change removes:
  `- Py_MEMBER_SIZE(PyDictKeysObject, dk_indices)`

If this works as is I'll make a separate PR to update PEP-007.
2018-04-19 22:41:19 -07:00
Serhiy Storchaka b7e1eff843
bpo-33299: Return an object itself for some types in _PyCode_ConstantKey(). (GH-6513) 2018-04-19 08:28:04 +03:00
INADA Naoki 5fbc511f56
bpo-33205: dict: Change GROWTH_RATE to `used*3` (GH-6350) 2018-04-17 15:53:34 +09:00
Antoine Pitrou 480ab05d5f
bpo-33176: Add a toreadonly() method to memoryviews. (GH-6466) 2018-04-14 19:49:21 +02:00
Serhiy Storchaka 827d49f3cf
bpo-33031: Remove dead code in C implementation of OrderedDict. (GH-6120)
This code doesn't have effect on the final result, but causes
GCC 8 warnings and can have an undefined behavior.
2018-04-09 19:14:26 +03:00
INADA Naoki d1c82c5cc7
bpo-33199: Initialize ma_version_tag in PyDict_Copy (GH-6341) 2018-04-03 11:43:53 +09:00
Stefano Taschini 0301c9bdd1 Fix typo and edit for clarity in the docstrings of sys.float_info. (GH-2251) 2018-03-26 12:41:30 +03:00
Serhiy Storchaka 397466dfd9
bpo-30953: Improve error messages and add tests for jumping (GH-6196)
into/out of an except block.
2018-03-23 14:46:45 +02:00
Serhiy Storchaka 702f8f3611
bpo-33041: Rework compiling an "async for" loop. (#6142)
* Added new opcode END_ASYNC_FOR.
* Setting global StopAsyncIteration no longer breaks "async for" loops.
* Jumping into an "async for" loop is now disabled.
* Jumping out of an "async for" loop no longer corrupts the stack.
* Simplify the compiler.
2018-03-23 14:34:35 +02:00
Marcel Plch c2b0b12d1a bpo-32374: m_traverse may be called with m_state=NULL (GH-5140)
Multi-phase initialized modules allow m_traverse to be called while the
module is still being initialized, so module authors may need to account
for that.
2018-03-17 15:41:20 +10:00
xdegaye b8e9d6c5cd bpo-17288: Prevent jumps from 'return' and 'exception' trace events. (GH-6107)
(cherry picked from commit e32bbaf376)
2018-03-13 19:31:31 +02:00
Serhiy Storchaka 26c9f565d0
bpo-33026: Fix jumping out of "with" block by setting f_lineno. (#6026) 2018-03-11 08:32:47 +02:00
Thomas Nyberg 7023644e0c closes bpo-32980 Remove _PyFrame_Init (GH-5965) 2018-03-03 22:06:01 -08:00
Benjamin Peterson b18f8bc1a7
remove vestigal locking from obmalloc (GH-5805)
obmalloc has (empty) macros for locking in the allocator. These aren't needed in CPython; we rely on the GIL.
2018-02-24 11:59:10 -08:00
Serhiy Storchaka 520b7ae27e
bpo-17611. Move unwinding of stack for "pseudo exceptions" from interpreter to compiler. (GH-5006)
Co-authored-by: Mark Shannon <mark@hotpy.org>
Co-authored-by: Antoine Pitrou <antoine@python.org>
2018-02-22 23:33:30 +02:00
Eddie Elizondo 745dc65b17 closes bpo-32898: Fix debug build crash with COUNT_ALLOCS (GH-5800) 2018-02-21 20:55:18 -08:00
Xiang Zhang 2b77a921e6
bpo-29803: remove a redandunt op and fix a comment in unicodeobject.c (#660) 2018-02-13 18:33:32 +08:00
Oren Milman d019bc8319 bpo-31787: Prevent refleaks when calling __init__() more than once (GH-3995) 2018-02-13 19:28:33 +09:00
Serhiy Storchaka b7e2d67f7c
bpo-32827: Fix usage of _PyUnicodeWriter_Prepare() in decoding errors handler. (GH-5636) 2018-02-13 08:27:33 +02:00
Leo Arias c3d9508ff2 bpo-32746: Fix multiple typos (GH-5144)
Fix typos found by codespell in docs, docstrings, and comments.
2018-02-03 19:36:10 -05:00
oldk aa0735f597 bpo-32747: Remove trailing spaces in docstrings. (GH-5491) 2018-02-02 10:52:55 +02:00
Xiang Zhang 2c7fd46e11
bpo-32583: Fix possible crashing in builtin Unicode decoders (#5325)
When using customized decode error handlers, it is possible for builtin decoders
to write out-of-bounds and then crash.
2018-01-31 20:48:05 +08:00
Yury Selivanov 2a2270db9b
bpo-32703: Fix coroutine resource warning in case where there's an error (GH-5410)
The commit removes one unnecessary "if" clause in genobject.c.  That "if" clause was masking un-awaited coroutines warnings just to make writing unittests more convenient.
2018-01-29 14:31:47 -05:00
Victor Stinner 8017b804a0
bpo-28685: Fix compiler warning (GH-5423) 2018-01-29 13:47:06 +01:00
embg 1e34da49ef bpo-28685: Optimize sorted() list.sort() with type-specialized comparisons (#582) 2018-01-28 19:03:23 -08:00
Raymond Hettinger a4d0001256
bpo-32690: Preserve order of locals() (#5379) 2018-01-28 09:40:24 -08:00
Nitish Chandra 43c0f1ac5e bpo-32685: Improve suggestion for print statement (GH-5375)
Better account for single-line compound statements and
semi-colon separated statements when suggesting
Py3 replacements for Py2 print statements.

Initial patch by Nitish Chandra.
2018-01-28 20:56:02 +10:00
INADA Naoki bea57060c8
bpo-32677: Optimize str.isascii() (GH-5356) 2018-01-28 09:59:12 +09:00
INADA Naoki 7cc95f5069
Fix wrong assert in unicodeobject (GH-5340) 2018-01-28 02:07:09 +09:00
INADA Naoki a49ac99029
bpo-32677: Add .isascii() to str, bytes and bytearray (GH-5342) 2018-01-27 14:06:21 +09:00
Yury Selivanov 43c47fe096
bpo-32670: Enforce PEP 479. (#5327) 2018-01-26 15:24:24 -05:00
bennorth d7773d92bd bpo-18533: Avoid RecursionError from repr() of recursive dictview (#4823)
dictview_repr(): Use a Py_ReprEnter() / Py_ReprLeave() pair to check
for recursion, and produce "..." if so.

test_recursive_repr(): Check for the string rather than a
RecursionError.  (Test cannot be any tighter as contents are
implementation-dependent.)

test_deeply_nested_repr(): Add new test, replacing the original
test_recursive_repr().  It checks that a RecursionError is raised in
the case of a non-recursive but deeply nested structure.  (Very
similar to what test_repr_deep() in test/test_dict.py does for a
normal dict.)

OrderedDictTests: Add new test case, to test behavior on OrderedDict
instances containing their own values() or items().
2018-01-26 07:46:01 -08:00
INADA Naoki e76daebc0c
bpo-32571: Fix reading uninitialized memory (GH-5332)
Reported by Coverity Scan.
2018-01-26 16:22:51 +09:00
Serhiy Storchaka f320be77ff bpo-32571: Avoid raising unneeded AttributeError and silencing it in C code (GH-5222)
Add two new private APIs: _PyObject_LookupAttr() and _PyObject_LookupAttrId()
2018-01-25 17:49:40 +09:00
Yury Selivanov f23746a934
bpo-32436: Implement PEP 567 (#5027) 2018-01-22 19:11:18 -05:00
Yury Selivanov b0a7a037b8
bpo-31179: Make dict.copy() up to 5.5 times faster. (#3067) 2018-01-22 11:54:41 -05:00
Nathaniel J. Smith fc2f407829 bpo-32591: Add native coroutine origin tracking (#5250)
* Add coro.cr_origin and sys.set_coroutine_origin_tracking_depth
* Use coroutine origin information in the unawaited coroutine warning
* Stop using set_coroutine_wrapper in asyncio debug mode
* In BaseEventLoop.set_debug, enable debugging in the correct thread
2018-01-21 09:44:07 -05:00
Sanyam Khurana d57f26c753 bpo-32028: Fix suggestions for indented print statements (GH-4688)
The suggested replacement for print statements previously failed to account
for leading whitespace and hence could end up including unwanted text in
the proposed call to the print builtin.

Patch by Sanyam Khurana.
2018-01-20 13:12:22 +10:00
Raymond Hettinger fa78806041
Removed unnecesssary bit inversion which doesn't improve dispersion statistics (#5235) 2018-01-18 13:23:27 -08:00
INADA Naoki 378edee0a3
bpo-32544: Speed up hasattr() and getattr() (GH-5173)
AttributeError was raised always when attribute is not found.
This commit skip raising AttributeError when `tp_getattro` is `PyObject_GenericGetAttr`.
It makes hasattr() and getattr() about 4x faster when attribute is not found.
2018-01-16 20:52:41 +09:00
Raymond Hettinger b44c5169f6
bpo-26163: Frozenset hash improvement (#5194) 2018-01-16 01:30:26 -08:00
Victor Stinner 7ed7aead95
bpo-29240: Fix locale encodings in UTF-8 Mode (#5170)
Modify locale.localeconv(), time.tzname, os.strerror() and other
functions to ignore the UTF-8 Mode: always use the current locale
encoding.

Changes:

* Add _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx(). On decoding or
  encoding error, they return the position of the error and an error
  message which are used to raise Unicode errors in
  PyUnicode_DecodeLocale() and PyUnicode_EncodeLocale().
* Replace _Py_DecodeCurrentLocale() with _Py_DecodeLocaleEx().
* PyUnicode_DecodeLocale() now uses _Py_DecodeLocaleEx() for all
  cases, especially for the strict error handler.
* Add _Py_DecodeUTF8Ex(): return more information on decoding error
  and supports the strict error handler.
* Rename _Py_EncodeUTF8_surrogateescape() to _Py_EncodeUTF8Ex().
* Replace _Py_EncodeCurrentLocale() with _Py_EncodeLocaleEx().
* Ignore the UTF-8 mode to encode/decode localeconv(), strerror()
  and time zone name.
* Remove PyUnicode_DecodeLocale(), PyUnicode_DecodeLocaleAndSize()
  and PyUnicode_EncodeLocale() now ignore the UTF-8 mode: always use
  the "current" locale.
* Remove _PyUnicode_DecodeCurrentLocale(),
  _PyUnicode_DecodeCurrentLocaleAndSize() and
  _PyUnicode_EncodeCurrentLocale().
2018-01-15 10:45:49 +01:00
Raymond Hettinger 3329992e31
bpo-29476: Simplify set_add_entry() (#5175) 2018-01-14 10:20:13 -08:00
Victor Stinner cb3ae5588b
bpo-29240: Ignore UTF-8 Mode in time module (#5148)
time.strftime() must use the current LC_CTYPE encoding, not UTF-8
if the UTF-8 mode is enabled.

Add _PyUnicode_DecodeCurrentLocale() function.
2018-01-11 10:37:59 +01:00
Victor Stinner 2cba6b8579
bpo-29240: readline now ignores the UTF-8 Mode (#5145)
Add new fuctions ignoring the UTF-8 mode:

* _Py_DecodeCurrentLocale()
* _Py_EncodeCurrentLocale()
* _PyUnicode_DecodeCurrentLocaleAndSize()
* _PyUnicode_EncodeCurrentLocale()

Modify the readline module to use these functions.

Re-enable test_readline.test_nonascii().
2018-01-10 22:46:15 +01:00
Serhiy Storchaka ce5b0e9db1
bpo-32226: Make __class_getitem__ an automatic class method. (#5098) 2018-01-05 00:21:41 +02:00
Berker Peksag 78e24d4415
Fix outdated comment in typeobject.c (GH-5090)
The comment was added in 3abca127fe.
In d78448e912, it was changed to
use PyArg_ParseTuple instead.
2018-01-04 13:24:45 +03:00
Antoine Pitrou 14709144b5
bpo-32468: Better frame repr() (#5067)
bpo-32468: Better frame repr()
2017-12-31 22:35:22 +01:00
Victor Stinner 9dd762013f
bpo-32030: Add _Py_EncodeLocaleRaw() (#4961)
Replace Py_EncodeLocale() with _Py_EncodeLocaleRaw() in:

* _Py_wfopen()
* _Py_wreadlink()
* _Py_wrealpath()
* _Py_wstat()
* pymain_open_filename()

These functions are called early during Python intialization, only
the RAW memory allocator must be used.
2017-12-21 16:20:32 +01:00
Victor Stinner e47e698da6
bpo-32030: Add _Py_EncodeUTF8_surrogateescape() (#4960)
Py_EncodeLocale() now uses _Py_EncodeUTF8_surrogateescape(), instead
of using temporary unicode and bytes objects. So Py_EncodeLocale()
doesn't use the Python C API anymore.
2017-12-21 15:45:16 +01:00
Serhiy Storchaka 6b91a59721
bpo-32385: Clean up the C3 MRO algorithm implementation. (#4942)
Use tuples and raw arrays instead of lists.
2017-12-20 19:21:02 +02:00
Antoine Pitrou 1f1a34c314
bpo-32379: Faster MRO computation for single inheritance (#4932)
* bpo-32379: Faster MRO computation for single inheritance
2017-12-20 15:58:21 +01:00
Serhiy Storchaka a5552f023e
bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
Ivan Levkivskyi 2b5fd1e9ca
bpo-32226: Implementation of PEP 560 (core components) (#4732)
This part of the PEP implementation adds support for
__mro_entries__ and __class_getitem__ by updating
__build_class__ and PyObject_GetItem.
2017-12-14 23:32:56 +01:00
Ivan Levkivskyi 5364b5cd75
bpo-32225: Implementation of PEP 562 (#4731)
Implement PEP 562: module __getattr__ and __dir__.
The implementation simply updates module_getattro and
module_dir.
2017-12-14 11:59:44 +01:00
Victor Stinner 91106cd9ff
bpo-29240: PEP 540: Add a new UTF-8 Mode (#855)
* Add -X utf8 command line option, PYTHONUTF8 environment variable
  and a new sys.flags.utf8_mode flag.
* If the LC_CTYPE locale is "C" at startup: enable automatically the
  UTF-8 mode.
* Add _winapi.GetACP(). encodings._alias_mbcs() now calls
  _winapi.GetACP() to get the ANSI code page
* locale.getpreferredencoding() now returns 'UTF-8' in the UTF-8
  mode. As a side effect, open() now uses the UTF-8 encoding by
  default in this mode.
* Py_DecodeLocale() and Py_EncodeLocale() now use the UTF-8 encoding
  in the UTF-8 Mode.
* Update subprocess._args_from_interpreter_flags() to handle -X utf8
* Skip some tests relying on the current locale if the UTF-8 mode is
  enabled.
* Add test_utf8mode.py.
* _Py_DecodeUTF8_surrogateescape() gets a new optional parameter to
  return also the length (number of wide characters).
* pymain_get_global_config() and pymain_set_global_config() now
  always copy flag values, rather than only copying if the new value
  is greater than the old value.
2017-12-13 12:29:09 +01:00
Serhiy Storchaka 4ae06c5337
bpo-32241: Add the const qualifire to declarations of umodifiable strings. (#4748) 2017-12-12 13:55:04 +02:00
Sanyam Khurana 780acc89bc bpo-31506: Improve the error message logic for class instantiation (GH-4740)
The error messages in `object.__new__` and `object.__init__` now aim
to point the user more directly at the name of the class being instantiated
in cases where they *haven't* been overridden (on the assumption that
the actual problem is a missing `__new__` or `__init__` definition in the
class body).

When they *have* been overridden, the errors still report themselves as
coming from object, on the assumption that the problem is with the call
up to the base class in the method implementation, rather than with the
way the constructor is being called.
2017-12-10 10:14:22 +10:00
Victor Stinner dd431b32f4
PyLong_FromString(): fix Coverity CID 1424951 (#4738)
Explicitly cast digits (Py_ssize_t) to double to fix the following
false-alarm warning from Coverity:

"fsize_z = digits * log_base_BASE[base] + 1;"

CID 1424951: Incorrect expression (UNINTENDED_INTEGER_DIVISION)
Dividing integer expressions "9223372036854775783UL" and "4UL", and
then converting the integer quotient to type "double". Any remainder,
or fractional part of the quotient, is ignored.
2017-12-08 00:06:55 +01:00
Victor Stinner 6bf992a1ac
bpo-32030: Add pymain_get_global_config() (#4735)
* Py_Main() now starts by reading Py_xxx configuration variables to
  only work on its own private structure, and then later writes back
  the configuration into these variables.
* Replace Py_GETENV() with pymain_get_env_var() which ignores empty
  variables.
* Add _PyCoreConfig.dump_refs
* Add _PyCoreConfig.malloc_stats
* _PyObject_DebugMallocStats() is now responsible to check if debug
  hooks are installed. The function returns 1 if stats were written,
  or 0 if the hooks are disabled. Mark _PyMem_PymallocEnabled() as
  static.
2017-12-06 17:26:10 +01:00
Serhiy Storchaka 29ba688034
bpo-31619: Fixed integer overflow in converting huge strings to int. (#3884) 2017-12-03 22:16:21 +02:00
Serhiy Storchaka 1fb72d2ad2
bpo-32137: The repr of deeply nested dict now raises a RecursionError (#4570)
instead of crashing due to a stack overflow.

This perhaps will fix similar problems in other extension types.
2017-12-03 22:12:11 +02:00
Nick Coghlan 078f1814f1
bpo-32176: Set CO_NOFREE in the code object constructor (GH-4675)
Previously, CO_NOFREE was set in the compiler, which meant
it could end up being set incorrectly when code objects
were created directly. Setting it in the constructor based
on freevars and cellvars ensures it is always accurate,
regardless of how the code object is defined.
2017-12-03 11:12:20 +10:00
Victor Stinner b64de46aae
bpo-32030: Cleanup "path config" code (#4663)
* Rename PyPathConfig structure to _PyPathConfig and move it to
  Include/internal/pystate.h
* Rename path_config to _Py_path_config
* _PyPathConfig: Rename program_name field to program_full_path
* Add assert(str != NULL); to _PyMem_RawWcsdup(), _PyMem_RawStrdup()
  and _PyMem_Strdup().
* Rename calculate_path() to pathconfig_global_init(). The function
  now does nothing if it's already initiallized.
2017-12-01 18:27:09 +01:00
Victor Stinner 5d39e04290
bpo-32030: Rework memory allocators (#4625)
* Fix _PyMem_SetupAllocators("debug"): always restore allocators to
  the defaults, rather than only caling _PyMem_SetupDebugHooks().
* Add _PyMem_SetDefaultAllocator() helper to set the "default"
  allocator.
* Add _PyMem_GetAllocatorsName(): get the name of the allocators
* main() now uses debug hooks on memory allocators if Py_DEBUG is
  defined, rather than calling directly malloc()
* Document default memory allocators in C API documentation
* _Py_InitializeCore() now fails with a fatal user error if
  PYTHONMALLOC value is an unknown memory allocator, instead of
  failing with a fatal internal error.
* Add new tests on the PYTHONMALLOC environment variable
* Add support.with_pymalloc()
* Add the _testcapi.WITH_PYMALLOC constant and expose it as
   support.with_pymalloc().
* sysconfig.get_config_var('WITH_PYMALLOC') doesn't work on Windows, so
   replace it with support.with_pymalloc().
* pythoninfo: add _testcapi collector for pymem
2017-11-29 17:20:38 +01:00
Victor Stinner 46972b7bc3
bpo-32030: Add _PyMainInterpreterConfig_ReadEnv() (#4542)
Py_GetPath() and Py_Main() now call
_PyMainInterpreterConfig_ReadEnv() to share the same code to get
environment variables.

Changes:

* Add _PyMainInterpreterConfig_ReadEnv()
* Add _PyMainInterpreterConfig_Clear()
* Add _PyMem_RawWcsdup()
* _PyMainInterpreterConfig: rename pythonhome to home
* Rename _Py_ReadMainInterpreterConfig() to
  _PyMainInterpreterConfig_Read()
* Use _Py_INIT_USER_ERR(), instead of _Py_INIT_ERR(), for decoding
  errors: the user is able to fix the issue, it's not a bug in
  Python. Same change was made in _Py_INIT_NO_MEMORY().
* Remove _Py_GetPythonHomeWithConfig()
2017-11-24 22:55:40 +01:00
Victor Stinner 9e87e7776f
bpo-32096: Remove obj and mem from _PyRuntime (#4532)
bpo-32096, bpo-30860:  Partially revert the commit
2ebc5ce42a8a9e047e790aefbf9a94811569b2b6:

* Move structures back from Include/internal/mem.h to
  Objects/obmalloc.c
* Remove _PyObject_Initialize() and _PyMem_Initialize()
* Remove Include/internal/pymalloc.h
* Add test_capi.test_pre_initialization_api():
   Make sure that it's possible to call Py_DecodeLocale(), and then call
   Py_SetProgramName() with the decoded string, before Py_Initialize().

PyMem_RawMalloc() and Py_DecodeLocale() can be called again before
_PyRuntimeState_Init().

Co-Authored-By: Eric Snow <ericsnowcurrently@gmail.com>
2017-11-24 12:09:24 +01:00
Victor Stinner 6a54c676e6
bpo-31979: Remove unused align_maxchar() function (#4527) 2017-11-23 19:02:23 +01:00
Victor Stinner 25420fe290
bpo-32030: Add more options to _PyCoreConfig (#4485)
Py_Main() now handles two more -X options:

* -X showrefcount: new _PyCoreConfig.show_ref_count field
* -X showalloccount: new _PyCoreConfig.show_alloc_count field
2017-11-20 18:12:22 -08:00
Victor Stinner ccb0442a33
bpo-32043: New "developer mode": "-X dev" option (#4413)
Add a new "developer mode": new "-X dev" command line option to
enable debug checks at runtime.

Changes:

* Add unit tests for -X dev
* test_cmd_line: replace test.support with support.
* Fix _PyRuntimeState_Fini(): Use the same memory allocator
   than _PyRuntimeState_Init().
* Fix _PyMem_GetDefaultRawAllocator()
2017-11-16 03:20:31 -08:00
Victor Stinner f7e5b56c37
bpo-32030: Split Py_Main() into subfunctions (#4399)
* Don't use "Python runtime" anymore to parse command line options or
  to get environment variables: pymain_init() is now a strict
  separation.
* Use an error message rather than "crashing" directly with
  Py_FatalError(). Limit the number of calls to Py_FatalError(). It
  prepares the code to handle errors more nicely later.
* Warnings options (-W, PYTHONWARNINGS) and "XOptions" (-X) are now
  only added to the sys module once Python core is properly
  initialized.
* _PyMain is now the well identified owner of some important strings
  like: warnings options, XOptions, and the "program name". The
  program name string is now properly freed at exit.
  pymain_free() is now responsible to free the "command" string.
* Rename most methods in Modules/main.c to use a "pymain_" prefix to
  avoid conflits and ease debug.
* Replace _Py_CommandLineDetails_INIT with memset(0)
* Reorder a lot of code to fix the initialization ordering. For
  example, initializing standard streams now comes before parsing
  PYTHONWARNINGS.
* Py_Main() now handles errors when adding warnings options and
  XOptions.
* Add _PyMem_GetDefaultRawAllocator() private function.
* Cleanup _PyMem_Initialize(): remove useless global constants: move
  them into _PyMem_Initialize().
* Call _PyRuntime_Initialize() as soon as possible:
  _PyRuntime_Initialize() now returns an error message on failure.
* Add _PyInitError structure and following macros:

  * _Py_INIT_OK()
  * _Py_INIT_ERR(msg)
  * _Py_INIT_USER_ERR(msg): "user" error, don't abort() in that case
  * _Py_INIT_FAILED(err)
2017-11-15 15:48:08 -08:00
Serhiy Storchaka f8a4c03ede
bpo-30399: Get rid of trailing comma in the repr of BaseException. (#1650) 2017-11-15 17:53:28 +02:00
Sanyam Khurana 28b624825e bpo-16055: Fixes incorrect error text for int('1', base=1000) (#4376)
* bpo-16055: Fixes incorrect error text for int('1', base=1000)

* bpo-16055: Address review comments
2017-11-13 13:49:26 -08:00
Serhiy Storchaka 9b6c60cbce
bpo-31979: Simplify transforming decimals to ASCII (#4336)
in int(), float() and complex() parsers.

This also speeds up parsing non-ASCII numbers by around 20%.
2017-11-13 21:23:48 +02:00
Serhiy Storchaka 60c3d3551a
bpo-31572: Get rid of _PyObject_HasAttrId() in dict and OrderedDict. (#3728)
Silence only AttributeError when get "key" and "items" attributes in
the constructor and the update() method of dict and OrderedDict .
2017-11-11 16:19:56 +02:00
Serhiy Storchaka e2f92de6a9
Add the const qualifier to "char *" variables that refer to literal strings. (#4370) 2017-11-11 13:06:26 +02:00
Victor Stinner 8c663fd60e
Replace KB unit with KiB (#4293)
kB (*kilo* byte) unit means 1000 bytes, whereas KiB ("kibibyte")
means 1024 bytes. KB was misused: replace kB or KB with KiB when
appropriate.

Same change for MB and GB which become MiB and GiB.

Change the output of Tools/iobench/iobench.py.

Round also the size of the documentation from 5.5 MB to 5 MiB.
2017-11-08 14:44:44 -08:00
Serhiy Storchaka 3cc4c53a64
bpo-31626: Mark ends of the reallocated block in debug build. (#4210)
Few bytes at the begin and at the end of the reallocated blocks, as well
as the header and the trailer, now are erased before calling realloc()
in debug build.  This will help to detect using or double freeing the
reallocated block.
2017-11-07 12:46:42 +02:00
luzpaz a5293b4ff2 Fix miscellaneous typos (#4275) 2017-11-05 15:37:50 +02:00
stratakis e8b1965639 bpo-23699: Use a macro to reduce boilerplate code in rich comparison functions (GH-793) 2017-11-02 20:32:54 +10:00
Victor Stinner 9ed83c4085
bpo-18835: Cleanup pymalloc (#4200)
Cleanup pymalloc:

* Rename _PyObject_Alloc() to pymalloc_alloc()
* Rename _PyObject_FreeImpl() to pymalloc_free()
* Rename _PyObject_Realloc() to pymalloc_realloc()
* pymalloc_alloc() and pymalloc_realloc() don't fallback on the raw
  allocator anymore, it now must be done by the caller
* Add "success" and "failed" labels to pymalloc_alloc() and
  pymalloc_free()
* pymalloc_alloc() and pymalloc_free() don't update
  num_allocated_blocks anymore: it should be done in the caller
* _PyObject_Calloc() is now responsible to fill the memory block
  allocated by pymalloc with zeros
* Simplify pymalloc_alloc() prototype
* _PyObject_Realloc() now calls _PyObject_Malloc() rather than
  calling directly pymalloc_alloc()

_PyMem_DebugRawAlloc() and _PyMem_DebugRawRealloc():

* document the layout of a memory block
* don't increase the serial number if the allocation failed
* check for integer overflow before computing the total size
* add a 'data' variable to make the code easiler to follow

test_setallocators() of _testcapimodule.c now test also the context.
2017-10-31 12:18:10 -07:00
Serhiy Storchaka b484d5606c
bpo-31626: Fixed a bug in debug memory allocator. (#3844)
Removed a code that incorrectly detected in-place resizing in realloc()
 and wrote to freed memory.
2017-10-31 14:05:03 +02:00
Serhiy Storchaka a2314283ff
bpo-20047: Make bytearray methods partition() and rpartition() rejecting (#4158)
separators that are not bytes-like objects.
2017-10-29 02:11:54 +03:00
xdegaye 56d1f5ca32 bpo-30697: Fix PyErr_NormalizeException() when no memory (GH-2327) 2017-10-26 15:09:06 +02:00
Victor Stinner 086c3ae5f0 bpo-31835: Optimize also FASTCALL using __future__ (#4087)
_PyFunction_FastCallDict() and _PyFunction_FastCallKeywords() now
also takes the fast path if the code object uses __future__
(CO_FUTURE_xxx flags).
2017-10-25 05:26:17 -07:00
Mark Shannon ae3087c638 Move exc state to generator. Fixes bpo-25612 (#1773)
Move exception state information from frame objects to coroutine (generator/thread) object where it belongs.
2017-10-22 23:41:51 +02:00
Serhiy Storchaka 56cb465cc9 bpo-31825: Fixed OverflowError in the 'unicode-escape' codec (#4058)
and in codecs.escape_decode() when decode an escaped non-ascii byte.
2017-10-20 17:08:15 +03:00
Oren Milman 0ccc0f6c74 bpo-28280: Make PyMapping_Keys(), PyMapping_Values() and PyMapping_Items() always return a list (#3840) 2017-10-08 11:17:46 +03:00
Serhiy Storchaka 79ba471488 bpo-31655: Validate keyword names in SimpleNamespace constructor. (#3909) 2017-10-07 22:59:35 +03:00
Yury Selivanov faa135acbf bpo-31709: Drop support for asynchronous __aiter__. (#3903) 2017-10-06 02:08:57 -04:00
Serhiy Storchaka 85c0b8941f bpo-31619: Fixed a ValueError when convert a string with large number of underscores (#3827)
to integer with binary base.
2017-10-03 14:13:44 +03:00
scoder 2102c78903 bpo-31336: Speed up type creation. (#3279)
Speed up class creation by 10-20% by reducing the overhead in the
necessary special method lookups.
2017-10-01 11:37:47 +03:00
Serhiy Storchaka 0e950dd22b bpo-31579: Fixed a possible leak in enumerate() with large indices. (#3753) 2017-09-26 08:14:58 +03:00
Raymond Hettinger 8110dbd470 bpo-26491 Defer DECREFs until enumobject is in a consistent state (#3747) 2017-09-25 02:15:53 -07:00
Serhiy Storchaka 5e02c7826f bpo-31410: Optimized calling wrapper and classmethod descriptors. (#3481) 2017-09-21 14:25:36 +03:00
Serhiy Storchaka b3a77964ea bpo-27541: Reprs of subclasses of some classes now contain actual type name. (#3631)
Affected classes are bytearray, array, deque, defaultdict, count and repeat.
2017-09-21 14:24:13 +03:00
Victor Stinner 0ad05c32cc bpo-30486: Make cell_set_contents() symbol private (#3668)
Don't export the cell_set_contents() symbol in the C API.
2017-09-20 06:54:13 -07:00