Commit Graph

104 Commits

Author SHA1 Message Date
Yunlongs 66c28f50c7
bpo-41995: Fix null ptr deref in tracemalloc_copy_trace() (GH-22660)
Fix a null pointer dereference in tracemalloc_copy_trace()
of _tracemalloc.
2020-10-13 08:46:31 +02:00
Christian Heimes 9cfcdb7d6e
Remove dead code from tracemalloc (GH-21029)
tracemalloc_get_frame() checked filename == NULL two times in a row.

Signed-off-by: Christian Heimes <christian@python.org>
2020-06-27 14:13:47 +02:00
Huon Wilson 8b62644831
bpo-40630: Add tracemalloc.reset_peak (GH-20102)
The reset_peak function sets the peak memory size to the current size,
representing a resetting of that metric. This allows for recording the
peak of specific sections of code, ignoring other code that may have
had a higher peak (since the most recent `tracemalloc.start()` or
tracemalloc.clear_traces()` call).
2020-05-22 16:18:51 +02:00
Victor Stinner 5b0a30354d
bpo-40609: _Py_hashtable_t values become void* (GH-20065)
_Py_hashtable_t values become regular "void *" pointers.

* Add _Py_hashtable_entry_t.data member
* Remove _Py_hashtable_t.data_size member
* Remove _Py_hashtable_t.get_func member. It is no longer needed
  to specialize _Py_hashtable_get() for a specific value size, since
  all entries now have the same size (void*).
* Remove the following macros:

  * _Py_HASHTABLE_GET()
  * _Py_HASHTABLE_SET()
  * _Py_HASHTABLE_SET_NODATA()
  * _Py_HASHTABLE_POP()

* Rename _Py_hashtable_pop() to _Py_hashtable_steal()
* _Py_hashtable_foreach() callback now gets key and value rather than
  entry.
* Remove _Py_hashtable_value_destroy_func type. value_destroy_func
  callback now only has a single parameter: data (void*).
2020-05-13 04:40:30 +02:00
Victor Stinner d95bd4214c
bpo-40609: _tracemalloc allocates traces (GH-20064)
Rewrite _tracemalloc to store "trace_t*" rather than directly
"trace_t" in traces hash tables. Traces are now allocated on the heap
memory, outside the hash table.

Add tracemalloc_copy_traces() and tracemalloc_copy_domains() helper
functions.

Remove _Py_hashtable_copy() function since there is no API to copy a
key or a value.

Remove also _Py_hashtable_delete() function which was commented.
2020-05-13 03:52:11 +02:00
Victor Stinner 2d0a3d682f
bpo-40609: Add destroy functions to _Py_hashtable (GH-20062)
Add key_destroy_func and value_destroy_func parameters to
_Py_hashtable_new_full().

marshal.c and _tracemalloc.c use these destroy functions.
2020-05-13 02:50:18 +02:00
Victor Stinner f9b3b582b8
bpo-40609: Remove _Py_hashtable_t.key_size (GH-20060)
Rewrite _Py_hashtable_t type to always store the key as
a "const void *" pointer. Add an explicit "key" member to
_Py_hashtable_entry_t.

Remove _Py_hashtable_t.key_size member.

hash and compare functions drop their hash table parameter, and their
'key' parameter type becomes "const void *".
2020-05-13 02:26:02 +02:00
Victor Stinner 9e2ca17420
bpo-40609: Rewrite how _tracemalloc handles domains (GH-20059)
Rewrite how the _tracemalloc module stores traces of other domains.
Rather than storing the domain inside the key, it now uses a new hash
table with the domain as the key, and the data is a per-domain traces
hash table.

* Add tracemalloc_domain hash table.
* Remove _Py_tracemalloc_config.use_domain.
* Remove pointer_t and related functions.
2020-05-13 01:36:47 +02:00
Victor Stinner b617993b7c
bpo-40602: Rename hashtable.h to pycore_hashtable.h (GH-20044)
* Move Modules/hashtable.h to Include/internal/pycore_hashtable.h
* Move Modules/hashtable.c to Python/hashtable.c
* Python is now linked to hashtable.c. _tracemalloc is no longer
  linked to hashtable.c. Previously, marshal.c got hashtable.c via
  _tracemalloc.c which is built as a builtin module.
2020-05-12 02:42:19 +02:00
Victor Stinner 7036477323
bpo-40421: Add PyFrame_GetBack() function (GH-19765)
New PyFrame_GetBack() function: get the frame next outer frame.

Replace frame->f_back with PyFrame_GetBack(frame) in most code but
frameobject.c, ceval.c and genobject.c.
2020-04-29 03:28:46 +02:00
Victor Stinner 4386b9045e
bpo-40429: PyThreadState_GetFrame() returns a strong ref (GH-19781)
The PyThreadState_GetFrame() function now returns a strong reference
to the frame.
2020-04-29 03:01:43 +02:00
Victor Stinner 8852ad4208
bpo-40429: PyFrame_GetCode() now returns a strong reference (GH-19773) 2020-04-29 01:28:13 +02:00
Victor Stinner 6d86a2331e
bpo-40429: PyFrame_GetCode() result cannot be NULL (GH-19772)
Add frame_nslots() to factorize duplicate code.
2020-04-29 00:56:58 +02:00
Victor Stinner a42ca74fa3
bpo-40421: Add PyFrame_GetCode() function (GH-19757)
PyFrame_GetCode(frame): return a borrowed reference to the frame
code.

Replace frame->f_code with PyFrame_GetCode(frame) in most code,
except in frameobject.c, genobject.c and ceval.c.

Also add PyFrame_GetLineNumber() to the limited C API.
2020-04-28 19:01:31 +02:00
Victor Stinner 361dcdcefc
bpo-40268: Remove unused osdefs.h includes (GH-19532)
When the include is needed, add required symbol in a comment.
2020-04-15 03:24:57 +02:00
Victor Stinner 62183b8d6d
bpo-40268: Remove explicit pythread.h includes (#19529)
Remove explicit pythread.h includes: it is always included
by Python.h.
2020-04-15 02:04:42 +02:00
Victor Stinner e560f90602
bpo-40268: Move struct _gc_runtime_state to pycore_gc.h (GH-19515) 2020-04-14 18:30:41 +02:00
Victor Stinner 3072338642
bpo-39947: Use PyThreadState_GetFrame() (GH-19159)
_tracemalloc.c and _xxsubinterpretersmodule.c use
PyThreadState_GetFrame() and PyThreadState_GetInterpreter() to no
longer depend on the PyThreadState structure.
2020-03-25 19:52:02 +01:00
Serhiy Storchaka eebaa9bfc5
bpo-38249: Expand Py_UNREACHABLE() to __builtin_unreachable() in the release mode. (GH-16329)
Co-authored-by: Victor Stinner <vstinner@python.org>
2020-03-09 20:49:52 +02:00
Victor Stinner 40e547dfbb
bpo-39542: Make _Py_NewReference() opaque in C API (GH-18346)
_Py_NewReference() becomes a regular opaque function, rather than a
static inline function in the C API (object.h), to better hide
implementation details.

Move _Py_tracemalloc_config from public pymem.h to internal
pycore_pymem.h header.

Make _Py_AddToAllObjects() private.
2020-02-05 01:11:10 +01:00
Brandt Bucher d51a363a43 bpo-38823: Fix refleak in _tracemalloc init error handling (GH-17235) 2019-11-20 11:00:31 +01:00
Julien Danjou 8d59eb1b66 bpo-37961, tracemalloc: add Traceback.total_nframe (GH-15545)
Add a total_nframe field to the traces collected by the tracemalloc module.
This field indicates the original number of frames before it was truncated.
2019-10-15 14:00:16 +02:00
Joannah Nanjekye 2bc43cdc01 bpo-37878: Remove PyThreadState_DeleteCurrent() function (GH-15315)
* Rename PyThreadState_DeleteCurrent()
  to _PyThreadState_DeleteCurrent()
* Move it to the internal C API

Co-Authored-By: Carol Willing <carolcode@willingconsulting.com>
2019-09-05 18:06:49 +02:00
Victor Stinner ed48866c55
bpo-35134: Split traceback.h header (GH-13430)
Add new Include/cpython/traceback.h and Include/internal/traceback.h
header files.
2019-05-20 00:14:57 +02:00
Victor Stinner f966e5397e
bpo-29564:_PyMem_DumpTraceback() suggests enabling tracemalloc (GH-10510)
If tracemalloc is not tracing Python memory allocations,
_PyMem_DumpTraceback() now suggests to enable tracemalloc
to get the traceback where the memory block has been allocated.
2018-11-13 15:14:58 +01:00
Victor Stinner 626bff8568
bpo-9263: Dump Python object on GC assertion failure (GH-10062)
Changes:

* Add _PyObject_AssertFailed() function.
* Add _PyObject_ASSERT() and _PyObject_ASSERT_WITH_MSG() macros.
* gc_decref(): replace assert() with _PyObject_ASSERT_WITH_MSG() to
  dump the faulty object if the assertion fails.

_PyObject_AssertFailed() calls:

* _PyMem_DumpTraceback(): try to log the traceback where the object
  memory has been allocated if tracemalloc is enabled.
* _PyObject_Dump(): log repr(obj).
* Py_FatalError(): log the current Python traceback.

_PyObject_AssertFailed() uses _PyObject_IsFreed() heuristic to check
if the object memory has been freed by a debug hook on Python memory
allocators.

Initial patch written by David Malcolm.

Co-Authored-By: David Malcolm <dmalcolm@redhat.com>
2018-10-25 17:31:10 +02:00
Victor Stinner 9e00e80e21
bpo-35053: Enhance tracemalloc to trace free lists (GH-10063)
tracemalloc now tries to update the traceback when an object is
reused from a "free list" (optimization for faster object creation,
used by the builtin list type for example).

Changes:

* Add _PyTraceMalloc_NewReference() function which tries to update
  the Python traceback of a Python object.
* _Py_NewReference() now calls _PyTraceMalloc_NewReference().
* Add an unit test.
2018-10-25 13:31:16 +02:00
Victor Stinner a7368ac636
bpo-32030: Enhance Py_Main() (#4412)
Parse more env vars in Py_Main():

* Add more options to _PyCoreConfig:

  * faulthandler
  * tracemalloc
  * importtime

* Move code to parse environment variables from _Py_InitializeCore()
  to Py_Main(). This change fixes a regression from Python 3.6:
  PYTHONUNBUFFERED is now read before calling pymain_init_stdio().
* _PyFaulthandler_Init() and _PyTraceMalloc_Init() now take an
  argument to decide if the module has to be enabled at startup.
* tracemalloc_start() is now responsible to check the maximum number
  of frames.

Other changes:

* Cleanup Py_Main():

  * Rename some pymain_xxx() subfunctions
  * Add pymain_run_python() subfunction

* Cleanup Py_NewInterpreter()
* _PyInterpreterState_Enable() now reports failure
* init_hash_secret() now considers pyurandom() failure as an "user
  error": don't fail with abort().
* pymain_optlist_append() and pymain_strdup() now sets err on memory
  allocation failure.
2017-11-15 18:11:45 -08:00
Masayuki Yamamoto 731e189014 bpo-25658: Implement PEP 539 for Thread Specific Storage (TSS) API (GH-1362)
See PEP 539 for details.

Highlights of changes:

- Add Thread Specific Storage (TSS) API
- Document the Thread Local Storage (TLS) API as deprecated
- Update code that used TLS API to use TSS API
2017-10-06 20:41:34 +10:00
Barry Warsaw b2e5794870 bpo-31338 (#3374)
* Add Py_UNREACHABLE() as an alias to abort().
* Use Py_UNREACHABLE() instead of assert(0)
* Convert more unreachable code to use Py_UNREACHABLE()
* Document Py_UNREACHABLE() and a few other macros.
2017-09-14 18:13:16 -07:00
Antoine Pitrou a6a4dc816d bpo-31370: Remove support for threads-less builds (#3385)
* Remove Setup.config
* Always define WITH_THREAD for compatibility.
2017-09-07 18:56:24 +02:00
Segev Finer 39243779f4 bpo-31018: Switch to #pragma pack from __declspec(align) (#2848) 2017-07-25 10:47:43 +02:00
Xiang Zhang 4ed5ad79ec bpo-30961: Fix decrementing a borrowed reference in tracemalloc. (#2747) 2017-07-19 00:33:29 +08:00
Victor Stinner 5ea4c06773 bpo-30054: Expose tracemalloc C API (#1236)
* Make PyTraceMalloc_Track() and PyTraceMalloc_Untrack() functions
  public (remove the "_" prefix)
* Remove the _PyTraceMalloc_domain_t type: use directly unsigned
  int.
* Document methods

Note: methods are already tested in test_tracemalloc.
2017-06-20 17:46:36 +02:00
Serhiy Storchaka 97353845f8 Issue #20186: Regenerated Argument Clinic. 2017-02-05 22:58:46 +02:00
Serhiy Storchaka b451f91786 Issue #20186: Converted the tracemalloc module to Argument Clinic.
Based on patch by Georg Brandl.
2017-02-04 12:18:38 +02:00
Benjamin Peterson 2f8bfef158 replace PY_SIZE_MAX with SIZE_MAX 2016-09-07 09:26:18 -07:00
Benjamin Peterson ca47063998 replace Py_(u)intptr_t with the c99 standard types 2016-09-06 13:47:26 -07:00
Raymond Hettinger 15f44ab043 Issue #27895: Spelling fixes (Contributed by Ville Skyttä). 2016-08-30 10:47:49 -07:00
doko@ubuntu.com bc7315068f - make some internal symbols static 2016-05-18 01:06:01 +02:00
Serhiy Storchaka b6a9c9761c Issue #26778: Fixed "a/an/and" typos in code comment, documentation and error
messages.
2016-04-17 09:39:28 +03:00
Serhiy Storchaka 6a7b3a77b4 Issue #26778: Fixed "a/an/and" typos in code comment and documentation. 2016-04-17 08:32:47 +03:00
Martin Panter cda80940ed Issue #15984: Merge PyUnicode doc from 3.5 2016-04-15 02:27:11 +00:00
Martin Panter 6245cb3c01 Correct “an” → “a” with “Unicode”, “user”, “UTF”, etc
This affects documentation, code comments, and a debugging messages.
2016-04-15 02:14:19 +00:00
Victor Stinner 5e14a38e8e _tracemalloc: use compact key for traces
Issue #26588: Optimize memory footprint of _tracemalloc before non-zero domain
is used. Start with compact key (Py_uintptr_t) and also switch to pointer_t key
when the first memory block with a non-zero domain is tracked.
2016-03-23 22:03:55 +01:00
Victor Stinner 5dacbd4c42 Cleanup hashtable.h
Issue #26588:

* Pass the hash table rather than the key size to hash and compare functions
* _Py_HASHTABLE_READ_KEY() and _Py_HASHTABLE_ENTRY_READ_KEY() macros now expect
  the hash table as the first parameter, rather than the key size
* tracemalloc_get_traces_fill(): use _Py_HASHTABLE_ENTRY_READ_DATA() rather
  than pointer dereference
* Remove the _Py_HASHTABLE_ENTRY_WRITE_PKEY() macro
* Move "PKEY" and "PDATA" macros inside hashtable.c
2016-03-23 09:52:13 +01:00
Victor Stinner ca79ccd9e6 Issue #26588:
* Optimize tracemalloc_add_trace(): modify hashtable entry data (trace) if the
  memory block is already tracked, rather than trying to remove the old trace
  and then add a new trace.
* Add _Py_HASHTABLE_ENTRY_WRITE_DATA() macro
2016-03-23 09:38:54 +01:00
Victor Stinner e8c6b2fd1b Issue #26588:
* _Py_HASHTABLE_ENTRY_DATA: change type from "char *" to "const void *"
* Add _Py_HASHTABLE_ENTRY_WRITE_PKEY() macro
* Rename _Py_HASHTABLE_ENTRY_WRITE_DATA() macro to
  _Py_HASHTABLE_ENTRY_WRITE_PDATA()
* Add _Py_HASHTABLE_ENTRY_WRITE_DATA() macro
2016-03-23 09:25:01 +01:00
Victor Stinner 42bcf37fcf Issue #26588: Optimize tracemalloc_realloc()
No need to remove the old trace if the memory block didn't move.
2016-03-23 09:08:08 +01:00
Victor Stinner 35b0fd74dd Merge 3.5 2016-03-23 00:17:54 +01:00