Commit Graph

3691 Commits

Author SHA1 Message Date
Jeroen Demeyer 762f93ff2e bpo-37337: Add _PyObject_CallMethodNoArgs() (GH-14267) 2019-07-08 17:19:25 +09:00
Hansraj Das 10e5c66789 Typo fix in file Include/object.h (GH-14556)
* accross -> across
* Extra space removed in comment
2019-07-05 14:37:15 -07:00
Jeroen Demeyer 7b57c03a7c bpo-37151: remove _PyMethodDef_RawFastCall* functions (GH-14603) 2019-07-05 17:43:22 +02:00
Jeroen Demeyer 0d722f3cd6 bpo-36974: separate vectorcall functions for each calling convention (GH-13781) 2019-07-05 14:48:24 +02:00
Jeroen Demeyer 196a530e00 bpo-37483: add _PyObject_CallOneArg() function (#14558) 2019-07-04 19:31:34 +09:00
Jeroen Demeyer d4efd917ac bpo-36904: Optimize _PyStack_UnpackDict (GH-14517) 2019-07-02 18:49:40 +09:00
Victor Stinner 36242fd871
bpo-36763: Add PyConfig_SetWideStringList() (GH-14444) 2019-07-01 19:13:50 +02:00
Pablo Galindo 4a2edc34a4 bpo-37221: Add PyCode_NewWithPosOnlyArgs to be used internally and set PyCode_New as a compatibility wrapper (GH-13959)
Add PyCode_NewEx to be used internally and set PyCode_New as a compatibility wrapper
2019-07-01 12:35:05 +02:00
Steve Dower 9048c49322
bpo-37369: Fix initialization of sys members when launched via an app container (GH-14428)
sys._base_executable is now always defined on all platforms, and can be overridden through configuration.
Also adds test.support.PythonSymlink to encapsulate platform-specific logic for symlinking sys.executable
2019-06-29 10:34:11 -07:00
Jeroen Demeyer b1263d5a60 bpo-37337: Add _PyObject_VectorcallMethod() (GH-14228) 2019-06-28 18:49:00 +09:00
Victor Stinner 3939c321c9
bpo-20443: _PyConfig_Read() gets the absolute path of run_filename (GH-14053)
Python now gets the absolute path of the script filename specified on
the command line (ex: "python3 script.py"): the __file__ attribute of
the __main__ module, sys.argv[0] and sys.path[0] become an absolute
path, rather than a relative path.

* Add _Py_isabs() and _Py_abspath() functions.
* _PyConfig_Read() now tries to get the absolute path of
  run_filename, but keeps the relative path if _Py_abspath() fails.
* Reimplement os._getfullpathname() using _Py_abspath().
* Use _Py_isabs() in getpath.c.
2019-06-25 15:02:43 +02:00
Victor Stinner 36456df138
bpo-37392: Remove sys.setcheckinterval() (GH-14355)
Remove sys.getcheckinterval() and sys.setcheckinterval() functions.
They were deprecated since Python 3.2. Use sys.getswitchinterval()
and sys.setswitchinterval() instead.

Remove also check_interval field of the PyInterpreterState structure.
2019-06-25 03:01:08 +02:00
Jeroen Demeyer 7e1a9aacff bpo-37151: remove _PyCFunction_FastCallDict (GH-14269) 2019-06-21 00:38:45 +09:00
Victor Stinner b45d259bdd
bpo-36710: Use tstate in pylifecycle.c (GH-14249)
In pylifecycle.c: pass tstate argument, rather than interp argument,
to functions.
2019-06-20 00:05:23 +02:00
Victor Stinner 987a0dcfa1
bpo-36710: Remove PyImport_Cleanup() function (GH-14221)
* Rename PyImport_Cleanup() to _PyImport_Cleanup() and move it to the
  internal C API. Add 'tstate' parameters.
* Remove documentation of _PyImport_Init(), PyImport_Cleanup(),
  _PyImport_Fini(). All three were documented as "For internal use
  only.".
2019-06-19 10:36:10 +02:00
Victor Stinner 0a28f8d379
bpo-36710: Add tstate parameter in import.c (GH-14218)
* Add 'tstate' parameter to many internal import.c functions.
* _PyImportZip_Init() now gets 'tstate' parameter rather than
  'interp'.
* Add 'interp' parameter to _PyState_ClearModules() and rename it
  to _PyInterpreterState_ClearModules().
* Move private _PyImport_FindBuiltin() to the internal C API; add
  'tstate' parameter to it.
* Remove private _PyImport_AddModuleObject() from the C API:
  use public PyImport_AddModuleObject() instead.
* Remove private _PyImport_FindExtensionObjectEx() from the C API:
  use private _PyImport_FindExtensionObject() instead.
2019-06-19 02:54:39 +02:00
Victor Stinner 01b63ecac6
bpo-35134: Add Include/cpython/import.h header file (GH-14213)
* Add Include/cpython/import.h and Include/internal/pycore_import.h
  header files.
* Move _PyImport_ReInitLock() to the internal C API. Don't export the
  symbol anymore.
2019-06-19 00:48:09 +02:00
Jeroen Demeyer 59543347d1 bpo-37151: remove _PyFunction_FastCallDict (GH-13864) 2019-06-18 20:05:41 +09:00
Victor Stinner 343ed0ffe0
bpo-37189: Export old PyRun_XXX() functions (#14142)
Many PyRun_XXX() functions like PyRun_String() were no longer
exported in libpython38.dll by mistake. Export them again to fix the
ABI compatibiliy.
2019-06-18 00:15:13 +02:00
Victor Stinner 2ff58a24e8
bpo-37194: Add a new public PyObject_CallNoArgs() function (GH-13890)
Add a new public PyObject_CallNoArgs() function to the C API: call a
callable Python object without any arguments.

It is the most efficient way to call a callback without any argument.
On x86-64, for example, PyObject_CallFunctionObjArgs(func, NULL)
allocates 960 bytes on the stack per call, whereas
PyObject_CallNoArgs(func) only allocates 624 bytes per call.

It is excluded from stable ABI 3.8.

Replace private _PyObject_CallNoArg() with public
PyObject_CallNoArgs() in C extensions: _asyncio, _datetime,
_elementtree, _pickle, _tkinter and readline.
2019-06-17 14:27:23 +02:00
Jeroen Demeyer 0456df4a55 bpo-37231: remove _PyObject_FastCall_Prepend (GH-14153) 2019-06-17 20:41:32 +09:00
Jeroen Demeyer b2f94730d9 bpo-37249: add declaration of _PyObject_GetMethod (GH-14015) 2019-06-14 19:37:15 +09:00
Michael Felt d0eeb936d8 bpo-37077: Add native thread ID (TID) for AIX (GH-13624)
This is the followup  for issue36084



https://bugs.python.org/issue37077
2019-06-13 15:34:46 -07:00
Victor Stinner 838f26402d
bpo-36710: Pass explicitly tstate in sysmodule.c (GH-14060)
* Replace global var Py_VerboseFlag with interp->config.verbose.
* Add _PyErr_NoMemory(tstate) function.
* Add tstate parameter to _PyEval_SetCoroutineOriginTrackingDepth()
  and move the function to the internal API.
* Replace _PySys_InitMain(runtime, interp)
  with _PySys_InitMain(runtime, tstate).
2019-06-13 22:41:23 +02:00
Victor Stinner 022ac0a497
bpo-37253: Remove PyAST_obj2mod_ex() function (GH-14020)
PyAST_obj2mod_ex() is similar to PyAST_obj2mod() with an additional
'feature_version' parameter which is unused.
2019-06-13 09:18:45 +02:00
Victor Stinner 37d66d7d4b
bpo-37253: Add _PyCompilerFlags_INIT macro (GH-14018)
Add a new _PyCompilerFlags_INIT macro to initialize PyCompilerFlags
variables, rather than initializing cf_flags and cf_feature_version
explicitly in each variable.
2019-06-13 02:16:41 +02:00
David Carlier 5287022eee bpo-37160: Thread native ID NetBSD support (GH-13835) 2019-06-12 17:37:56 +02:00
Łukasz Langa 9ab2fb1c68
Bump to 3.9.0a0 2019-06-04 22:12:32 +02:00
Łukasz Langa 3b5deb0116
Python 3.8.0b1 2019-06-04 19:44:34 +02:00
Victor Stinner 0fd2c300c2
Revert "bpo-36818: Add PyInterpreterState.runtime field. (gh-13129)" (GH-13795)
This reverts commit 396e0a8d9d.
2019-06-04 03:15:09 +02:00
Victor Stinner e225bebc14
Revert "bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (gh-13714)" (GH-13780)
This reverts commit 6a150bcaeb.
2019-06-03 18:14:24 +02:00
David Carlier 0b9956e916 bpo-37087: Adding native ID support for OpenBSD (GH-13654) 2019-06-03 17:43:33 +02:00
Inada Naoki 91234a1636
bpo-26219: per opcode cache for LOAD_GLOBAL (GH-12884)
This patch implements per opcode cache mechanism, and use it in
only LOAD_GLOBAL opcode.

Based on Yury's opcache3.patch in bpo-26219.
2019-06-03 21:30:58 +09:00
Eric Snow 6a150bcaeb
bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (gh-13714) 2019-06-01 15:39:46 -06:00
Eric Snow 396e0a8d9d
bpo-36818: Add PyInterpreterState.runtime field. (gh-13129)
https://bugs.python.org/issue36818
2019-05-31 21:16:47 -06:00
Jeroen Demeyer 37788bc23f bpo-36974: rename _FastCallKeywords -> _Vectorcall (GH-13653) 2019-05-30 15:11:22 +02:00
Jeroen Demeyer c145f3bfbe bpo-36974: remove _PyObject_HasFastCall (GH-13460) 2019-05-30 12:43:58 +02:00
Antoine Pitrou ada319bb6d
bpo-32388: Remove cross-version binary compatibility requirement in tp_flags (GH-4944)
It is now allowed to add new fields at the end of the PyTypeObject struct without having to allocate a dedicated compatibility flag in tp_flags.

This will reduce the risk of running out of bits in the 32-bit tp_flags value.
2019-05-29 22:12:38 +02:00
Gregory P. Smith 0c2f930564
bpo-22385: Support output separators in hex methods. (#13578)
* bpo-22385: Support output separators in hex methods.

Also in binascii.hexlify aka b2a_hex.

The underlying implementation behind all hex generation in CPython uses the
same pystrhex.c implementation.  This adds support to bytes, bytearray,
and memoryview objects.

The binascii module functions exist rather than being slated for deprecation
because they return bytes rather than requiring an intermediate step through a
str object.

This change was inspired by MicroPython which supports sep in its binascii
implementation (and does not yet support the .hex methods).

https://bugs.python.org/issue22385
2019-05-29 11:46:58 -07:00
Jeroen Demeyer aacc77fbd7 bpo-36974: implement PEP 590 (GH-13185)
Co-authored-by: Jeroen Demeyer <J.Demeyer@UGent.be>
Co-authored-by: Mark Shannon <mark@hotpy.org>
2019-05-29 20:31:52 +02:00
Dino Viehland 415406999d
bpo-37001: Makes symtable.symtable have parity with compile for input (#13483)
* Makes symtable.symtable have parity for accepted datatypes
for source code as compile()

* Add NEWS blurb
2019-05-28 16:21:17 -07:00
MandarJKulkarni 33ce3f012f Fix comments in initconfig.h (GH-13636) 2019-05-28 18:55:05 -04:00
Zackery Spytz 3c8724fc60 bpo-33407: Implement Py_DEPRECATED() on MSVC (GH-8980) 2019-05-28 17:16:33 +02:00
Jeroen Demeyer eb65e2443a bpo-36922: implement PEP-590 Py_TPFLAGS_METHOD_DESCRIPTOR (GH-13338)
Co-authored-by: Mark Shannon <mark@hotpy.org>
2019-05-28 14:42:53 +02:00
Matthias Bussonnier 3880f263d2 bpo-36933: Remove sys.set_coroutine_wrapper (marked for removal in 3.8) (GH-13577)
It has been documented as deprecated and to be removed in 3.8; 

From a comment on another thread – which I can't find ; leave get_coro_wrapper() for now, but always return `None`.


https://bugs.python.org/issue36933
2019-05-28 00:10:59 -07:00
Victor Stinner cd590a7ced
bpo-1230540: Add threading.excepthook() (GH-13515)
Add a new threading.excepthook() function which handles uncaught
Thread.run() exception. It can be overridden to control how uncaught
exceptions are handled.

threading.ExceptHookArgs is not documented on purpose: it should not
be used directly.

* threading.excepthook() and threading.ExceptHookArgs.
* Add _PyErr_Display(): similar to PyErr_Display(), but accept a
  'file' parameter.
* Add _thread._excepthook(): C implementation of the exception hook
  calling _PyErr_Display().
* Add _thread._ExceptHookArgs: structseq type.
* Add threading._invoke_excepthook_wrapper() which handles the gory
  details to ensure that everything remains alive during Python
  shutdown.
* Add unit tests.
2019-05-28 00:39:52 +02:00
Eric V. Smith 6f6ff8a565
bpo-37050: Remove expr_text from FormattedValue ast node, use Constant node instead (GH-13597)
When using the "=" debug functionality of f-strings, use another Constant node (or a merged constant node) instead of adding expr_text to the FormattedValue node.
2019-05-27 15:31:52 -04:00
Victor Stinner 331a6a56e9
bpo-36763: Implement the PEP 587 (GH-13592)
* Add a whole new documentation page:
  "Python Initialization Configuration"
* PyWideStringList_Append() return type is now PyStatus,
  instead of int
* PyInterpreterState_New() now calls PyConfig_Clear() if
  PyConfig_InitPythonConfig() fails.
* Rename files:

  * Python/coreconfig.c => Python/initconfig.c
  * Include/cpython/coreconfig.h => Include/cpython/initconfig.h
  * Include/internal/: pycore_coreconfig.h => pycore_initconfig.h

* Rename structures

  * _PyCoreConfig => PyConfig
  * _PyPreConfig => PyPreConfig
  * _PyInitError => PyStatus
  * _PyWstrList => PyWideStringList

* Rename PyConfig fields:

  * use_module_search_paths => module_search_paths_set
  * module_search_path_env => pythonpath_env

* Rename PyStatus field: _func => func
* PyInterpreterState: rename core_config field to config
* Rename macros and functions:

  * _PyCoreConfig_SetArgv() => PyConfig_SetBytesArgv()
  * _PyCoreConfig_SetWideArgv() => PyConfig_SetArgv()
  * _PyCoreConfig_DecodeLocale() => PyConfig_SetBytesString()
  * _PyInitError_Failed() => PyStatus_Exception()
  * _Py_INIT_ERROR_TYPE_xxx enums => _PyStatus_TYPE_xxx
  * _Py_UnixMain() => Py_BytesMain()
  * _Py_ExitInitError() => Py_ExitStatusException()
  * _Py_PreInitializeFromArgs() => Py_PreInitializeFromBytesArgs()
  * _Py_PreInitializeFromWideArgs() => Py_PreInitializeFromArgs()
  * _Py_PreInitialize() => Py_PreInitialize()
  * _Py_RunMain() => Py_RunMain()
  * _Py_InitializeFromConfig() => Py_InitializeFromConfig()
  * _Py_INIT_XXX() => _PyStatus_XXX()
  * _Py_INIT_FAILED() => _PyStatus_EXCEPTION()

* Rename 'err' PyStatus variables to 'status'
* Convert RUN_CODE() macro to config_run_code() static inline function
* Remove functions:

  * _Py_InitializeFromArgs()
  * _Py_InitializeFromWideArgs()
  * _PyInterpreterState_GetCoreConfig()
2019-05-27 16:39:22 +02:00
Victor Stinner 71c52e3048
bpo-36829: Add _PyErr_WriteUnraisableMsg() (GH-13488)
* sys.unraisablehook: add 'err_msg' field to UnraisableHookArgs.
* Use _PyErr_WriteUnraisableMsg() in _ctypes _DictRemover_call()
  and gc delete_garbage().
2019-05-27 08:57:14 +02:00
Antoine Pitrou 91f4380ced
bpo-36785: PEP 574 implementation (GH-7076) 2019-05-26 17:10:09 +02:00
Victor Stinner 438a12dd9d
bpo-36710: Add tstate parameter in ceval.c (GH-13547)
* Fix a possible reference leak in _PyErr_Print() if exception
  is NULL.
* PyErr_BadInternalCall(): replace PyErr_Format() with _PyErr_SetString().
* Add pycore_pyerrors.h header file.
* New functions:

  * _PyErr_Clear()
  * _PyErr_Fetch()
  * _PyErr_Print()
  * _PyErr_Restore()
  * _PyErr_SetObject()
  * _PyErr_SetString()

* Add 'tstate' parameter to _PyEval_AddPendingCall().
2019-05-24 17:01:38 +02:00
Victor Stinner b3a9843cd1
Support Py_UNUSED() on clang (GH-13544) 2019-05-24 15:16:08 +02:00
Steve Dower b82e17e626
bpo-36842: Implement PEP 578 (GH-12613)
Adds sys.audit, sys.addaudithook, io.open_code, and associated C APIs.
2019-05-23 08:45:22 -07:00
Victor Stinner df22c03b93
bpo-36829: PyErr_WriteUnraisable() normalizes exception (GH-13507)
PyErr_WriteUnraisable() now creates a traceback object if there is no
current traceback. Moreover, call PyErr_NormalizeException() and
PyException_SetTraceback() to normalize the exception value. Ignore
silently any error.
2019-05-23 01:00:58 +02:00
Victor Stinner 022be02dcf
bpo-36763: Add _PyPreConfig._config_init (GH-13481)
* _PyPreConfig_GetGlobalConfig() and  _PyCoreConfig_GetGlobalConfig()
  now do nothing if the configuration was not initialized with
  _PyPreConfig_InitCompatConfig() and _PyCoreConfig_InitCompatConfig()
* Remove utf8_mode=-2 special case: use utf8_mode=-1 instead.
* Fix _PyPreConfig_InitPythonConfig():

  * isolated = 0 instead of -1
  * use_environment = 1 instead of -1

* Rename _PyConfig_INIT to  _PyConfig_INIT_COMPAT
* Rename _PyPreConfig_Init() to _PyPreConfig_InitCompatConfig()
* Rename _PyCoreConfig_Init() to _PyCoreConfig_InitCompatConfig()
* PyInterpreterState_New() now uses _PyCoreConfig_InitPythonConfig()
  as default configuration, but it's very quickly overriden anyway.
* _freeze_importlib.c uses _PyCoreConfig_SetString() to set
  program_name.
* Cleanup preconfig_init_utf8_mode(): cmdline is always non-NULL.
2019-05-22 23:58:50 +02:00
Jake Tesler b121f63155 bpo-36084: Add native thread ID (TID) to threading.Thread (GH-13463)
Add native thread ID (TID) to threading.Thread objects
(supported platforms: Windows, FreeBSD, Linux, macOS).
2019-05-22 17:43:16 +02:00
Michael J. Sullivan 933e1509ec bpo-36878: Track extra text added to 'type: ignore' in the AST (GH-13479)
GH-13238 made extra text after a # type: ignore accepted by the parser.
This finishes the job and actually plumbs the extra text through the
parser and makes it available in the AST.
2019-05-22 15:54:20 +01:00
Victor Stinner ef9d9b6312
bpo-36829: Add sys.unraisablehook() (GH-13187)
Add new sys.unraisablehook() function which can be overridden to
control how "unraisable exceptions" are handled. It is called when an
exception has occurred but there is no way for Python to handle it.
For example, when a destructor raises an exception or during garbage
collection (gc.collect()).

Changes:

* Add an internal UnraisableHookArgs type used to pass arguments to
  sys.unraisablehook.
* Add _PyErr_WriteUnraisableDefaultHook().
* The default hook now ignores exception on writing the traceback.
* test_sys now uses unittest.main() to automatically discover tests:
  remove test_main().
* Add _PyErr_Init().
* Fix PyErr_WriteUnraisable(): hold a strong reference to sys.stderr
  while using it
2019-05-22 11:28:22 +02:00
Matthias Bussonnier 565b4f1ac7 bpo-34616: Add PyCF_ALLOW_TOP_LEVEL_AWAIT to allow top-level await (GH-13148)
Co-Authored-By: Yury Selivanov <yury@magic.io>
2019-05-21 16:12:02 -04:00
Victor Stinner d12e75734d
Revert "bpo-36084: Add native thread ID to threading.Thread objects (GH-11993)" (GH-13458)
This reverts commit 4959c33d25.
2019-05-21 12:44:57 +02:00
Victor Stinner 0f72147ce2
bpo-36763: Fix _PyRuntime.preconfig.coerce_c_locale (GH-13444)
_PyRuntime.preconfig.coerce_c_locale can now be used to
check if the C locale has been coerced.

* Fix _Py_LegacyLocaleDetected(): don't attempt to coerce the
  C locale if LC_ALL environment variable is set. Add 'warn'
  parameter: emit_stderr_warning_for_legacy_locale() must not
  the LC_ALL env var.
* _PyPreConfig_Write() sets coerce_c_locale to 0 if
  _Py_CoerceLegacyLocale() fails.
2019-05-20 17:16:38 +02:00
Victor Stinner 6d1c46746e
bpo-36763: Fix Python preinitialization (GH-13432)
* Add _PyPreConfig.parse_argv
* Add _PyCoreConfig._config_init field and _PyCoreConfigInitEnum enum
  type
* Initialization functions: reject preconfig=NULL and config=NULL
* Add config parameter to _PyCoreConfig_DecodeLocaleErr(): pass
  config->argv to _Py_PreInitializeFromPyArgv(), to parse config
  command line arguments in preinitialization.
* Add config parameter to _PyCoreConfig_SetString(). It now
  preinitializes Python.
* _PyCoreConfig_SetPyArgv() now also preinitializes Python for wide
  argv
* Fix _Py_PreInitializeFromCoreConfig(): don't pass args to
  _Py_PreInitializeFromPyArgv() if config.parse_argv=0.
* Use "char * const *" and "wchar_t * const *" types for 'argv'
  parameters and _PyArgv.argv.
* Add unit test on preinitialization from argv.
* _PyPreConfig.allocator type becomes int
* Add _PyPreConfig_InitFromPreConfig() and
  _PyPreConfig_InitFromCoreConfig() helper functions
2019-05-20 11:02:00 +02:00
Victor Stinner ed48866c55
bpo-35134: Split traceback.h header (GH-13430)
Add new Include/cpython/traceback.h and Include/internal/traceback.h
header files.
2019-05-20 00:14:57 +02:00
Serhiy Storchaka a5119e7d75
bpo-36957: Add _PyLong_Rshift() and _PyLong_Lshift(). (GH-13416) 2019-05-19 14:14:38 +03:00
Victor Stinner 410759fba8
bpo-36763: Remove _PyCoreConfig.dll_path (GH-13402) 2019-05-18 04:17:01 +02:00
Victor Stinner bab0db6076
bpo-36763: Use _PyCoreConfig_InitPythonConfig() (GH-13398)
_PyPreConfig_InitPythonConfig() and _PyCoreConfig_InitPythonConfig()
no longer inherit their values from global configuration variables.

Changes:

* _PyPreCmdline_Read() now ignores -X dev and PYTHONDEVMODE
  if dev_mode is already set.
* Inline _PyPreConfig_INIT macro into _PyPreConfig_Init() function.
* Inline _PyCoreConfig_INIT macro into _PyCoreConfig_Init() function.
* Replace _PyCoreConfig_Init() with _PyCoreConfig_InitPythonConfig()
  in most tests of _testembed.c.
* Replace _PyCoreConfig_Init() with _PyCoreConfig_InitIsolatedConfig()
  in _freeze_importlib.c.
* Move some initialization functions from the internal
  to the private API.
2019-05-18 03:21:27 +02:00
Victor Stinner b594784272
bpo-36763: _Py_InitializeFromArgs() argc becomes Py_ssize_t (GH-13396)
* The type of initlization function 'argc' parameters becomes
  Py_ssize_t, instead of int.
* Change _PyPreConfig_Copy() return type to void, instead of int.
  The function cannot fail anymore.
* Fix compilation warnings on Windows.
2019-05-18 00:38:16 +02:00
Victor Stinner 871ff77c1c
bpo-36763: Add _PyInitError functions (GH-13395)
* Add _PyInitError functions:

  * _PyInitError_Ok()
  * _PyInitError_Error()
  * _PyInitError_NoMemory()
  * _PyInitError_Exit()
  * _PyInitError_IsError()
  * _PyInitError_IsExit()
  * _PyInitError_Failed()

* frozenmain.c and _testembed.c now use functions rather than macros.
* Move _Py_INIT_xxx() macros to the internal API.
* Move _PyWstrList_INIT macro to the internal API.
2019-05-17 23:54:00 +02:00
Victor Stinner 12083284c5
bpo-36763: _Py_RunMain() doesn't call Py_Exit() anymore (GH-13390)
Py_Main() and _Py_RunMain() now return the exitcode rather than
calling Py_Exit(exitcode) when calling PyErr_Print() if the current
exception type is SystemExit.

* Add _Py_HandleSystemExit().
* Add pymain_exit_err_print().
* Add pymain_exit_print().
2019-05-17 23:05:29 +02:00
Victor Stinner bcfbbd7046
bpo-36945: Add _PyPreConfig.configure_locale (GH-13368)
_PyPreConfig_InitIsolatedConfig() sets configure_locale to 0 to
prevent Python to modify the LC_CTYPE locale. In that case,
coerce_c_locale an coerce_c_locale_warn are set to 0 as well.
2019-05-17 22:44:16 +02:00
Victor Stinner cab5d0741e
bpo-36763: Add _PyCoreConfig_InitPythonConfig() (GH-13388)
Add new functions to get the Python interpreter behavior:

* _PyPreConfig_InitPythonConfig()
* _PyCoreConfig_InitPythonConfig()

Add new functions to get an isolated configuration:

* _PyPreConfig_InitIsolatedConfig()
* _PyCoreConfig_InitIsolatedConfig()

Replace _PyPreConfig_INIT and _PyCoreConfig_INIT with new functions
_PyPreConfig_Init() and _PyCoreConfig_Init().

_PyCoreConfig: set configure_c_stdio and parse_argv to 0 by default
to behave as Python 3.6 in the default configuration.

_PyCoreConfig_Read() no longer sets coerce_c_locale_warn to 1 if it's
equal to 0. coerce_c_locale_warn must now be set to -1 (ex: using
_PyCoreConfig_InitPythonConfig()) to enable C locale coercion
warning.

Add unit tests for _PyCoreConfig_InitPythonConfig()
and _PyCoreConfig_InitIsolatedConfig().

Changes:

* Rename _PyCoreConfig_GetCoreConfig() to _PyPreConfig_GetCoreConfig()
* Fix core_read_precmdline(): handle parse_argv=0
* Fix _Py_PreInitializeFromCoreConfig(): pass coreconfig.argv
  to _Py_PreInitializeFromPyArgv(), except if parse_argv=0
2019-05-17 19:01:14 +02:00
Victor Stinner b16b4e4592
bpo-36763: Add PyMemAllocatorName (GH-13387)
* Add PyMemAllocatorName enum
* _PyPreConfig.allocator type becomes PyMemAllocatorName, instead of
  char*
* Remove _PyPreConfig_Clear()
* Add _PyMem_GetAllocatorName()
* Rename _PyMem_GetAllocatorsName() to
  _PyMem_GetCurrentAllocatorName()
* Remove _PyPreConfig_SetAllocator(): just call
  _PyMem_SetupAllocators() directly, we don't have do reallocate the
  configuration with the new allocator anymore!
* _PyPreConfig_Write() parameter becomes const, as it should be in
  the first place!
2019-05-17 15:20:52 +02:00
Victor Stinner fed02e15b3
bpo-36763: Remove _PyCoreConfig.program (GH-13373)
Use _PyCoreConfig.program_name instead.
2019-05-17 11:12:09 +02:00
Victor Stinner 870b035bc6
bpo-36763: Cleanup precmdline in _PyCoreConfig_Read() (GH-13371)
precmdline is only needed in _PyCoreConfig_Read(), not in
config_read_cmdline(). Reorganize _PyCoreConfig_Read().
2019-05-17 03:15:12 +02:00
Victor Stinner 54b43bb3bb
bpo-36763: Add _PyCoreConfig.configure_c_stdio (GH-13363)
Add tests for configure_c_stdio and pathconfig_warnings parameters.
2019-05-16 18:30:15 +02:00
Victor Stinner 9ef5dcaa0b
bpo-36763: Add _Py_InitializeMain() (GH-13362)
* Add a private _Py_InitializeMain() function.
* Add again _PyCoreConfig._init_main.
* _Py_InitializeFromConfig() now uses _init_main to decide
  if _Py_InitializeMainInterpreter() should be called.
* _PyCoreConfig: rename _frozen to pathconfig_warnings, its value is
  now the opposite of Py_FrozenFlag.
* Add an unit test for _init_main=0 and _Py_InitializeMain().
2019-05-16 17:38:16 +02:00
Victor Stinner ae239f6b06
bpo-36763: Add _PyCoreConfig.parse_argv (GH-13361)
* _PyCoreConfig_Read() doesn't parse nor update argv
  if parse_argv is 0.
* Move path configuration fields in _PyCoreConfig.
* Add an unit test for parse_argv=0.
* Remove unused "done": label in _Py_RunMain().
2019-05-16 17:02:56 +02:00
Victor Stinner dbacfc2273
bpo-36763: _PyInitError always use int for exitcode (GH-13360)
We cannot use "unsigned int" for exitcode on Windows, since
Py_Main() and _Py_RunMain() always return an "int".

Changes:

* _PyPathConfig_ComputeSysPath0() now returns -1 if an exception is
  raised.
* pymain_run_python() no longer uses _PyInitError but display the
  exception and set exitcode to 1 in case of error.
* Fix _Py_RunMain(): return an exitcode rather than calling
  exit() on pymain_run_python() failure.
* _Py_ExitInitError() no longer uses ExitProcess() on Windows, use
  exit() on all platforms.
* _Py_ExitInitError() now fails with a fatal error if 'err' is not an
  error not an exit.
2019-05-16 16:39:26 +02:00
Victor Stinner 410b85a7f7
bpo-36900: import.c uses PyInterpreterState.core_config (GH-13278)
Move _PyImportZip_Init() to the internal C API and add an 'interp'
parameter.
2019-05-13 17:12:45 +02:00
Victor Stinner d5d9e81ce9
bpo-36728: Remove PyEval_ReInitThreads() from C API (GH-13241)
Remove the PyEval_ReInitThreads() function from the Python C API.
It should not be called explicitly: use PyOS_AfterFork_Child()
instead.

Rename PyEval_ReInitThreads() to _PyEval_ReInitThreads() and add a
'runtime' parameter.
2019-05-13 12:35:37 +02:00
Jake Tesler 4959c33d25 bpo-36084: Add native thread ID to threading.Thread objects (GH-11993) 2019-05-12 19:08:24 +02:00
Victor Stinner 09532feeec
bpo-36710: Add 'ceval' local variable to ceval.c (GH-12934)
Add "struct _ceval_runtime_state *ceval = &_PyRuntime.ceval;" local
variables to function to better highlight the dependency on the
global variable _PyRuntime and to point directly to _PyRuntime.ceval
field rather than on the larger _PyRuntime.

Changes:

* Add _PyRuntimeState_GetThreadState(runtime) macro.
* Add _PyEval_AddPendingCall(ceval, ...) and
  _PyThreadState_Swap(gilstate, ...) functions.
* _PyThreadState_GET() macro now calls
  _PyRuntimeState_GetThreadState() using &_PyRuntime.
* Add 'ceval' parameter to COMPUTE_EVAL_BREAKER(),
  SIGNAL_PENDING_SIGNALS(), _PyEval_SignalAsyncExc(),
  _PyEval_SignalReceived() and _PyEval_FiniThreads() macros and
  functions.
* Add 'tstate' parameter to call_function(), do_call_core() and
  do_raise().
* Add 'runtime' parameter to _Py_CURRENTLY_FINALIZING(),
  _Py_FinishPendingCalls() and _PyThreadState_DeleteExcept()
  macros and functions.
* Declare 'runtime', 'tstate', 'ceval' and 'eval_breaker' variables
  as constant.
2019-05-10 23:39:09 +02:00
Eric Snow 86ea58149c
bpo-36737: Use the module state C-API for warnings. (gh-13159) 2019-05-10 13:29:55 -04:00
Jeroen Demeyer 351c67416b bpo-35983: skip trashcan for subclasses (GH-11841)
Add new trashcan macros to deal with a double deallocation that could occur when the `tp_dealloc` of a subclass calls the `tp_dealloc` of a base class and that base class uses the trashcan mechanism.

Patch by Jeroen Demeyer.
2019-05-10 19:21:10 +02:00
Eric V. Smith 9a4135e939
bpo-36817: Add f-string debugging using '='. (GH-13123)
If a "=" is specified a the end of an f-string expression, the f-string will evaluate to the text of the expression, followed by '=', followed by the repr of the value of the expression.
2019-05-08 16:28:48 -04:00
Łukasz Langa 14987d4dc8
Post v3.8.0a4 2019-05-07 16:56:31 +02:00
Łukasz Langa c1004b8546
Python 3.8.0a4 2019-05-06 20:30:25 +02:00
Victor Stinner c664b342a4
bpo-36475: Make PyThread_exit_thread with _Py_NO_RETURN (GH-13068) 2019-05-04 11:48:05 -04:00
Victor Stinner 4631da1242
bpo-36763: Remove _PyCoreConfig._init_main (GH-13066) 2019-05-02 15:30:21 -04:00
Victor Stinner 70005ac0fd
bpo-36763: _PyCoreConfig_SetPyArgv() preinitializes Python (GH-13037)
_PyCoreConfig_SetPyArgv() and _PyCoreConfig_SetWideString() now
pre-initialize Python if needed to ensure that the locale encoding is
properly configured.

* Add _Py_PreInitializeFromPyArgv() internal function.
* Add 'args' parameter to _Py_PreInitializeFromCoreConfig()
2019-05-02 15:25:34 -04:00
Victor Stinner 709d23dee6
bpo-36775: _PyCoreConfig only uses wchar_t* (GH-13062)
_PyCoreConfig: Change filesystem_encoding, filesystem_errors,
stdio_encoding and stdio_errors fields type from char* to wchar_t*.

Changes:

* PyInterpreterState: replace fscodec_initialized (int) with fs_codec
  structure.
* Add get_error_handler_wide() and unicode_encode_utf8() helper
  functions.
* Add error_handler parameter to unicode_encode_locale()
  and unicode_decode_locale().
* Remove _PyCoreConfig_SetString().
* Rename _PyCoreConfig_SetWideString() to _PyCoreConfig_SetString().
* Rename _PyCoreConfig_SetWideStringFromString()
  to _PyCoreConfig_DecodeLocale().
2019-05-02 14:56:30 -04:00
Victor Stinner 373893ce51
bpo-36763: Add _PyCoreConfig._config_version (GH-13065)
Add private _config_version field to _PyPreConfig and _PyCoreConfig
to prepare future ABI compatibility.
2019-05-02 14:46:29 -04:00
Victor Stinner 43fc3bb7cf
bpo-36775: Add _PyUnicode_InitEncodings() (GH-13057)
Move get_codec_name() and initfsencoding() from pylifecycle.c to
unicodeobject.c.

Rename also "init" functions in pylifecycle.c.
2019-05-02 11:54:20 -04:00
Victor Stinner e251095a3f
bpo-36775: Add _Py_FORCE_UTF8_FS_ENCODING macro (GH-13056)
Add _Py_FORCE_UTF8_LOCALE and _Py_FORCE_UTF8_FS_ENCODING macros to
avoid factorize "#if defined(__ANDROID__) || defined(__VXWORKS__)"
and "#if defined(__APPLE__)".

Cleanup also config_init_fs_encoding().
2019-05-02 11:28:57 -04:00
Victor Stinner cb9fbd3588
bpo-36763: Make _PyCoreConfig.check_hash_pycs_mode public (GH-13052)
_PyCoreConfig: Rename _check_hash_pycs_mode field to
check_hash_pycs_mode (make it public) and change its type from "const
char*" to "wchar_t*".
2019-05-01 23:51:56 -04:00
Victor Stinner 1a9f0d8efd
bpo-36763: Add _PyCoreConfig_SetString() (GH-13035)
Add 3 new config methods:

* _PyCoreConfig_SetString()
* _PyCoreConfig_SetWideString()
* _PyCoreConfig_SetWideStringFromString()

Changes:

* _PyCoreConfig_Copy() returns _PyInitError.
* Add CONFIG_GET_ENV_DUP().
2019-05-01 15:22:52 +02:00
Victor Stinner db71975431
bpo-36763: Rework _PyInitError API (GH-13031)
* Remove _PyInitError.user_err field and _Py_INIT_USER_ERR() macro:
  use _Py_INIT_ERR() instead. _Py_ExitInitError() now longer calls
  abort() on error: exit with exit code 1 instead.
* Add _PyInitError._type private field.
* exitcode field type is now unsigned int on Windows.
* Rename prefix field to _func.
* Rename msg field to err_msg.
2019-05-01 05:35:33 +02:00
Victor Stinner 5f38b8407b
bpo-36763: Add _PyCoreConfig_SetArgv() (GH-13030)
* Add 2 new config methods:

  * _PyCoreConfig_SetArgv()
  * _PyCoreConfig_SetWideArgv()

* Add also an internal _PyCoreConfig_SetPyArgv() method.
* Remove 'args' parameter from _PyCoreConfig_Read().
2019-05-01 02:30:12 +02:00
Pablo Galindo 8c77b8cb91
bpo-36540: PEP 570 -- Implementation (GH-12701)
This commit contains the implementation of PEP570: Python positional-only parameters.

* Update Grammar/Grammar with new typedarglist and varargslist

* Regenerate grammar files

* Update and regenerate AST related files

* Update code object

* Update marshal.c

* Update compiler and symtable

* Regenerate importlib files

* Update callable objects

* Implement positional-only args logic in ceval.c

* Regenerate frozen data

* Update standard library to account for positional-only args

* Add test file for positional-only args

* Update other test files to account for positional-only args

* Add News entry

* Update inspect module and related tests
2019-04-29 13:36:57 +01:00
Victor Stinner 99fcc616d4
Revert "bpo-36356: Destroy the GIL at exit (GH-12453)" (GH613006)
This reverts commit b36e5d627d.
2019-04-29 13:04:07 +02:00
Victor Stinner b36e5d627d
bpo-36356: Destroy the GIL at exit (GH-12453)
* Add _PyEval_FiniThreads2(). _PyEval_FiniThreads() now only clears
  the pending lock, whereas _PyEval_FiniThreads2() destroys the GIL.
* pymain_free() now calls _PyEval_FiniThreads2().
* Py_FinalizeEx() now calls _PyEval_FiniThreads().
2019-04-29 11:15:56 +02:00
Victor Stinner 6d13e5b35b
bpo-36722: Don't define ALT_SOABI for Py_TRACE_REFS build (GH-12973)
Py_TRACE_REFS ABI is incompatible with release and debug (Py_DEBUG)
ABI.
2019-04-26 18:56:19 +02:00
Victor Stinner 4cb525a1f0
bpo-36356: pymain_exit_error() only call pymain_free() for exit (GH-12968)
Add _Py_INIT_HAS_EXITCODE() macro.
2019-04-26 13:05:47 +02:00
Victor Stinner 87d23a041d
bpo-36724: Add _PyWarnings_Fini() (#12963)
Py_FinalizeEx() now clears _PyRuntime.warnings variables and
_PyRuntime.exitfuncs.

Changes:

* Add _PyWarnings_Fini(): called by Py_FinalizeEx()
* call_ll_exitfuncs() now clears _PyRuntime.exitfuncs while iterating
  on it (on backward order).
2019-04-26 05:49:26 +02:00
Victor Stinner 9db0324712
bpo-36710: Add runtime parameter in gcmodule.c (GH-12958)
Add 'state' or 'runtime' parameter to functions in gcmodule.c to
avoid to rely directly on the global variable _PyRuntime.
2019-04-26 02:32:01 +02:00
Victor Stinner f4e4703e74
bpo-36465: Make release and debug ABI compatible (GH-12615)
Release build and debug build are now ABI compatible: the Py_DEBUG
define no longer implies Py_TRACE_REFS define which introduces the
only ABI incompatibility.

A new "./configure --with-trace-refs" build option is now required to
get Py_TRACE_REFS define which adds sys.getobjects() function and
PYTHONDUMPREFS environment variable.

Changes:

* Add ./configure --with-trace-refs
* Py_DEBUG no longer implies Py_TRACE_REFS
2019-04-25 00:56:28 +02:00
Victor Stinner 43125224d6
bpo-36710: Add runtime variable to Py_InitializeEx() (GH-12939)
Py_InitializeEx() now uses a runtime variable passed to subfunctions,
rather than working directly on the global variable _PyRuntime.

Add 'runtime' parameter to _PyCoreConfig_Write(), _PySys_Create(),
_PySys_InitMain(), _PyGILState_Init(),
emit_stderr_warning_for_legacy_locale() and other subfunctions.
2019-04-24 18:23:53 +02:00
Victor Stinner 8e91c246e4
bpo-36710: Add runtime variable to Py_FinalizeEx() (GH-12937)
* Add a 'runtime' variable to Py_FinalizeEx() rather than working
  directly on the global variable _PyRuntime
* Add a 'runtime' parameter to _PyGC_Fini(), _PyGILState_Fini()
  and call_ll_exitfuncs()
2019-04-24 17:24:01 +02:00
Victor Stinner b930a2d2b1
bpo-36710: PyOS_AfterFork_Child() pass runtime parameter (GH-12936)
The PyOS_AfterFork_Child() function now pass a 'runtime' parameter to
subfunctions.

* Fix _PyRuntimeState_ReInitThreads(): use the correct memory allocator
* Add runtime parameter to _PyRuntimeState_ReInitThreads(),
  _PyGILState_Reinit() and _PyInterpreterState_DeleteExceptMain()
* Move _PyGILState_Reinit() to the internal C API.
2019-04-24 17:14:33 +02:00
Victor Stinner 8bb3230149
bpo-36710: Add runtime parameter to _PyThreadState_Init() (GH-12935)
* Add 'runtime' parameter to _PyThreadState_Init()
* Add 'gilstate' parameter to _PyGILState_NoteThreadState()
* Move _PyThreadState_Init() and _PyThreadState_DeleteExcept()
   to the internal C API.
2019-04-24 16:47:40 +02:00
Inada Naoki 09415ff0eb
fix warnings by adding more const (GH-12924) 2019-04-23 20:39:37 +09:00
tyomitch 84b4784f12 use `const` in graminit.c (GH-12713) 2019-04-23 18:29:57 +09:00
Steve Dower 264490797a
bpo-33608: Normalize atomic macros so that they all expect an atomic struct (GH-12877) 2019-04-22 11:13:11 -07:00
Victor Stinner 23bace26ec
bpo-36635: Add _testinternalcapi module (GH-12841)
Add a new _testinternalcapi module to test the internal C API.

Move _Py_GetConfigsAsDict() function to the internal C API:
_testembed now uses _testinternalcapi to access the function.
2019-04-18 11:37:26 +02:00
Paul Monson 11efd79076 bpo-36071 Add support for Windows ARM32 in ctypes/libffi (GH-12059) 2019-04-17 18:09:16 -07:00
Victor Stinner 5c75f37d47
bpo-36635: Change pyport.h for Py_BUILD_CORE_MODULE define (GH-12853)
Change PyAPI_FUNC(type), PyAPI_DATA(type) and PyMODINIT_FUNC macros
of pyport.h when Py_BUILD_CORE_MODULE is defined.

The Py_BUILD_CORE_MODULE define must be now be used to build a C
extension as a dynamic library accessing Python internals: export the
PyInit_xxx() function in DLL exports on Windows.

Changes:

* Py_BUILD_CORE_BUILTIN and Py_BUILD_CORE_MODULE now imply
  Py_BUILD_CORE directy in pyport.h.
* ceval.c compilation now fails with an error if Py_BUILD_CORE is not
  defined, just to ensure that Python is build with the correct
  defines.
* setup.py now compiles _pickle.c with Py_BUILD_CORE_MODULE define.
* setup.py compiles _json.c with Py_BUILD_CORE_MODULE define, rather
  than Py_BUILD_CORE_BUILTIN define
* PCbuild/pythoncore.vcxproj: Add Py_BUILD_CORE_BUILTIN define.
2019-04-17 23:02:26 +02:00
Inada Naoki 926b0cb5f6
bpo-36641: Add "const" to PyDoc_VAR macro (GH-12854)
It reduces "data" segment in python about 200KB.
2019-04-17 08:39:46 +09:00
Victor Stinner 9820c07e41
bpo-35134: Add Include/cpython/pymem.h (GH-12840)
Move unstable CPython API from Include/pymem.h into a new
Include/cpython/pymem.h header file.
2019-04-15 17:00:19 +02:00
Pablo Galindo f2cf1e3e28
bpo-36623: Clean parser headers and include files (GH-12253)
After the removal of pgen, multiple header and function prototypes that lack implementation or are unused are still lying around.
2019-04-13 17:05:14 +01:00
Victor Stinner 0fc91eef34
bpo-36389: Add _PyObject_CheckConsistency() function (GH-12803)
Add a new _PyObject_CheckConsistency() function which can be used to
help debugging. The function is available in release mode.

Add a 'check_content' parameter to _PyDict_CheckConsistency().
2019-04-12 21:51:34 +02:00
Eric Snow b75b1a3504
bpo-33608: Revert "Factor out a private, per-interpreter _Py_AddPendingCall()." (gh-12806)
This reverts commit f13c5c8b94 (gh-12360).
2019-04-12 10:20:10 -06:00
Eric Snow f13c5c8b94
bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (gh-12360)
This is effectively an un-revert of #11617 and #12024 (reverted in #12159). Portions of those were merged in other PRs (with lower risk) and this represents the remainder. Note that I found 3 different bugs in the original PRs and have fixed them here.
2019-04-12 09:18:16 -06:00
Victor Stinner 4c409beb4c
bpo-36389: Change PyMem_SetupDebugHooks() constants (GH-12782)
Modify CLEANBYTE, DEADDYTE and FORBIDDENBYTE constants: use 0xCD,
0xDD and 0xFD, rather than 0xCB, 0xBB and 0xFB, to use the same byte
patterns than Windows CRT debug malloc() and free().
2019-04-11 13:01:15 +02:00
Victor Stinner 2b00db6855
bpo-36389: _PyObject_IsFreed() now also detects uninitialized memory (GH-12770)
Replace _PyMem_IsFreed() function with _PyMem_IsPtrFreed() inline
function. The function is now way more efficient, it became a simple
comparison on integers, rather than a short loop. It detects also
uninitialized bytes and "forbidden bytes" filled by debug hooks
on memory allocators.

Add unit tests on _PyObject_IsFreed().
2019-04-11 11:33:27 +02:00
Dmitry Marakasov a0da131a74 bpo-36527: silence -Wunused-parameter warnings in object.h (GH-12688) 2019-04-06 18:04:47 +09:00
Victor Stinner 2f54908afc
bpo-36471: Add _Py_RunMain() (GH-12618)
* Add config_read_cmdline() subfunction. Remove _PyCmdline structure.
* _PyCoreConfig_Read() now also parses config->argv command line
  arguments
2019-03-29 15:13:46 +01:00
Victor Stinner d929f1838a
bpo-36443: Disable C locale coercion and UTF-8 Mode by default (GH-12589)
bpo-36443, bpo-36202: Since Python 3.7.0, calling Py_DecodeLocale()
before Py_Initialize() produces mojibake if the LC_CTYPE locale is
coerced and/or if the UTF-8 Mode is enabled by the user
configuration. This change fix the issue by disabling LC_CTYPE
coercion and UTF-8 Mode by default. They must now be enabled
explicitly (opt-in) using the new _Py_PreInitialize() API with
_PyPreConfig.

When embedding Python, set coerce_c_locale and utf8_mode attributes
of _PyPreConfig to -1 to enable automatically these parameters
depending on the LC_CTYPE locale, environment variables and command
line arguments

Alternative: Setting Py_UTF8Mode to 1 always explicitly enables the
UTF-8 Mode.

Changes:

* _PyPreConfig_INIT now sets coerce_c_locale and utf8_mode to 0 by
  default.
* _Py_InitializeFromArgs() and _Py_InitializeFromWideArgs() can now
  be called with config=NULL.
2019-03-27 18:28:46 +01:00
Victor Stinner 5ac27a50ff
bpo-36444: Rework _Py_InitializeFromConfig() API (GH-12576) 2019-03-27 13:40:14 +01:00
Eddie Elizondo 364f0b0f19 bpo-35810: Incref heap-allocated types in PyObject_Init (GH-11661)
* Incref heap-allocated types in PyObject_Init
* Add documentation and porting notes to What's New
2019-03-27 12:52:18 +01:00
Victor Stinner 484f20d2ff
bpo-36444: Add _PyCoreConfig._init_main (GH-12572)
* Add _PyCoreConfig._init_main: if equals to zero,
  _Py_InitializeFromConfig() doesn't call
  _Py_InitializeMainInterpreter().
* Add interp_p parameter to _Py_InitializeFromConfig().
* pymain_init() now calls _Py_InitializeFromConfig().
* Make _Py_InitializeCore() private.
2019-03-27 02:04:16 +01:00
Victor Stinner 8b9dbc017a
bpo-36444: Remove _PyMainInterpreterConfig (GH-12571) 2019-03-27 01:36:16 +01:00
Victor Stinner f8ba6f5afc
bpo-36301: Cleanup preconfig.c and coreconfig.c (GH-12563)
* _PyCoreConfig_Write() now updates _PyRuntime.preconfig
* Remove _PyPreCmdline_Copy()
* _PyPreCmdline_Read() now accepts _PyPreConfig and _PyCoreConfig
  optional configurations.
* Rename _PyPreConfig_ReadFromArgv() to _PyPreConfig_Read(). Simplify
  the code.
* Calling _PyCoreConfig_Read() no longer adds the warning options
  twice: don't add a warning option if it's already in the list.
* Rename _PyCoreConfig_ReadFromArgv() to _PyCoreConfig_Read().
* Rename config_from_cmdline() to _PyCoreConfig_ReadFromArgv().
* Add more assertions on _PyCoreConfig in _PyCoreConfig_Read().
* Move some functions.
* Make some config functions private.
2019-03-26 16:58:50 +01:00
Łukasz Langa b4d8f28a8a
Post v3.8.0a3 2019-03-26 10:11:11 +01:00
Łukasz Langa 9f966dfd90 Python 3.8.0a3
-----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE4/8oOcBIslwITevpsmmV4xAlBWgFAlyZLdQACgkQsmmV4xAl
 BWi6Gg/+LvFG+HLD5t80y4mIJlPAxxhyFyPQNw4AlQ9Tt6perkkLVMa0ugvBPUE4
 fzwaofesW8CZEGR/wV/DGgZFACLbSNxBrrwMYD4QJ0+T/LoNeY9zGlnKOrlGbzvJ
 4tQtQF5hjaIxfTMq2pH+NP4ODuUm3zuVgtVdDmK02kRqvhiCiw1RFyDGaVgrm/uM
 i78NUsih5oq7uv+pfWIfrGx7Wn4hsqW/Xq+N5S9YsNAgCzqGhfVPdU2gtw8nhq/F
 SBBYkSsg4aOzlrZTot4SBffMBQ66nvvoAQfiAh2Etf7/PpzpdjG5BLTVx0UHIzCS
 PIhkeOf27HAi6ul9xzgnJMKQXjdpU2JinzURWqAIz/gd3Kyd7iM5TPRMoWNx0iaN
 MYdJaPjmBHTiMDYS9MoiSc8XWVqzmratMHMKsIq2oQbCHzU4OV5cLFE46f69BrVD
 Ml3JrxpvzT8GZhI9zTK1Su14gGdtsndv1P+nW5/eusCno4IWlDLpc9GAFBGklsl9
 5T5nvckfZc/RsyeQFuoayXYJi9WfX+ib2PUVk9aFx3j0zFN3yVXevKqOBWtn6dl1
 flUvu7bOCCHseFCGyraC8+Kr15i/Bf7LTZZZ9eaQVBpTSd5GwlRw68dAmcXZ8PIz
 8tYaZtF6WwAilYGBhxs7spN6ridGjJ3vMi7jGVZzwPVAy3Ky05k=
 =JzCc
 -----END PGP SIGNATURE-----
gpgsig -----BEGIN PGP SIGNATURE-----
 
 iQJEBAABCAAuFiEE4/8oOcBIslwITevpsmmV4xAlBWgFAlyZ7CQQHGx1a2FzekBs
 YW5nYS5wbAAKCRCyaZXjECUFaIFCD/4xM7QqftFYOhNdIibuoUeZZMpSNx85/fZL
 r8RLPSFQoUMS3fR21QmlXy2VHJzQfeyNBN8w2firnWBWL2vGmXL+Abfg2jG3MvD/
 cxQskRuarp+VAPiLYvN2dPavnq0p1ZNdqU74np+ZgH942+1TbBhQH+A8CpBKnNPq
 5K8hMiAx3alRq+Pkj0ro325xZeAvjc/mo866KB9kxNEl5S+uXJEEw3Z3FcrHaj98
 nxXdfTWFVhur2KeG44HxR+GTw35Sbbi6kbzmnUkiN+itSCZVPqyMQD514z6QYsnD
 JLjnaDt+kKKBTArMqLffmS057UcHJnhmfa9VkwkTIhtqwA388iL4A1lWOoGYe6Xq
 8S/BJZ7QCH3bawoIq1t8w51kWnWnsCITMN6oYhNQpGUKDtDAcgqyGqHaMLgIBn2L
 51ZVCAIc8QawWx0Jf1eEivTc66SDTR6LQhIuDbS5ElKAkwPCxW+iSWqyv3qsQq2q
 mwSGOMpB/xDvhUEDkgUrFflEKz92mESAcwO+Sz3nViKM46wj84PLyCAHDdoC3oJn
 5m+JwaKAhGob0aQNQd/FU61gP5qChKXVVB6vBo8b6iWZ8eOMs92HQk5zbKeC5L2A
 eLx5EioVp2ZFYCKusFaad2mu/wFYkRabtc5ZpA0UgNGHrHnwAmQUknmRajbn7h2U
 +58Rf0ozrw==
 =nJsD
 -----END PGP SIGNATURE-----

Merge tag 'v3.8.0a3'

Python 3.8.0a3
2019-03-26 10:08:49 +01:00
Victor Stinner 20004959d2
bpo-36301: Remove _PyCoreConfig.preconfig (GH-12546)
* Replace _PyCoreConfig.preconfig with 3 new fields in _PyCoreConfig:
  isolated, use_environment, dev_mode.
* Add _PyPreCmdline.dev_mode.
* Add _Py_PreInitializeFromPreConfigInPlace().
2019-03-26 02:31:11 +01:00
Victor Stinner f78a5e9ce8
bpo-36301: Add _Py_GetEnv() function (GH-12542)
* Make _PyPreConfig_GetEnv(), _PyCoreConfig_GetEnv() and
  _PyCoreConfig_GetEnvDup() private
* _Py_get_env_flag() first parameter becomes "int use_environment"
2019-03-26 00:03:15 +01:00
Victor Stinner 1075d1684a
bpo-36301: Add _Py_GetConfigsAsDict() function (GH-12540)
* Add _Py_GetConfigsAsDict() function to get all configurations as a
  dict.
* dump_config() of _testembed.c now dumps preconfig as a separated
  key: call _Py_GetConfigsAsDict().
* Make _PyMainInterpreterConfig_AsDict() private.
2019-03-25 23:19:57 +01:00
Łukasz Langa 9a448855b5
v3.8.0a3 2019-03-25 20:36:40 +01:00
Victor Stinner a6fbc4e25e
bpo-36301: Add _Py_PreInitializeFromConfig() (GH-12536)
* Initialize _PyPreConfig.dev_mode to -1.
* _PyPreConfig_Read(): coreconfig has the priority over preconfig.
* _PyCoreConfig_Read() now calls _PyPreCmdline_Read() internally.
* config_from_cmdline() now pass _PyPreCmdline to config_read().
* Add _PyPreCmdline_Copy().
2019-03-25 18:37:10 +01:00
Victor Stinner f72346c475
bpo-36301: Cleanup preconfig code (GH-12535)
Prepare code to move some _PyPreConfig parameters into _PyPreCmdline.
Changes:

* _PyCoreConfig_ReadFromArgv(): remove preconfig parameter,
  use _PyRuntime.preconfig.
* Add _PyPreCmdline_GetPreConfig() (called by _PyPreConfig_Read()).
* Rename _PyPreCmdline_Init() to _PyPreCmdline_SetArgv()
* Factorize _Py_PreInitializeFromPreConfig() code: add
  pyinit_preinit().
* _PyPreConfig_Read() now sets coerce_c_locale to 2 if it must be
  coerced.
* Remove _PyCoreConfig_ReadPreConfig().
* _PyCoreConfig_Write() now copies updated preconfig into _PyRuntime.
2019-03-25 17:54:58 +01:00
Victor Stinner 6d5ee973f0
bpo-36301: Add _PyRuntimeState.preconfig (GH-12506)
_PyPreConfig_Write() now writes the applied pre-configuration into
_PyRuntimeState.preconfig.
2019-03-23 12:05:43 +01:00
Victor Stinner fa15376848
bpo-36301: Add _PyPreCmdline internal API (GH-12458)
_PyCoreConfig_ReadFromArgv() now reuses the code parsing command line
options from preconfig.c.
2019-03-20 04:25:38 +01:00
Victor Stinner f29084d611
bpo-36301: Add _PyRuntime.pre_initialized (GH-12457)
* Add _PyRuntime.pre_initialized: set to 1 when Python
  is pre-initialized
* Add _Py_PreInitialize() and _Py_PreInitializeFromPreConfig().
* _PyCoreConfig_Read() now calls  _Py_PreInitialize().
* Move _PyPreConfig_GetGlobalConfig() and
  _PyCoreConfig_GetGlobalConfig() calls from main.c to preconfig.c
  and coreconfig.c.
2019-03-20 02:20:13 +01:00
Victor Stinner dcf617152e
bpo-36236: Handle removed cwd at Python init (GH-12424)
At Python initialization, the current directory is no longer
prepended to sys.path if it has been removed.

Rename _PyPathConfig_ComputeArgv0() to
_PyPathConfig_ComputeSysPath0() to avoid confusion between argv[0]
and sys.path[0].
2019-03-19 16:09:27 +01:00
Victor Stinner f5f336a819
bpo-36356: pymain_free() calls _PyRuntime_Finalize() (GH-12435)
Ensure that _PyRuntime_Finalize() is always call. This change fix a
few memory leaks when running "python3 -V".
2019-03-19 14:53:58 +01:00
Victor Stinner fecc4f2b47
bpo-36356: Release Unicode interned strings on Valgrind (#12431)
When Python is compiled with Valgrind support, release Unicode
interned strings at exit in _PyUnicode_Fini().

* Rename _Py_ReleaseInternedUnicodeStrings() to
  unicode_release_interned() and make it private.
* unicode_release_interned() is now called from _PyUnicode_Fini():
  it must be called with a running Python thread state for TRASHCAN,
  it cannot be called from pymain_free().
* Don't display statistics on interned strings at exit anymore
2019-03-19 14:20:29 +01:00
Victor Stinner 5f9cf23502
bpo-36301: Error if decoding pybuilddir.txt fails (GH-12422)
Python initialization now fails if decoding pybuilddir.txt
configuration file fails at startup.

_PyPathConfig_Calculate() now reports memory allocation failure and
decoding error on decoding pybuilddir.txt content from
UTF-8/surrogateescape.
2019-03-19 01:46:25 +01:00
Victor Stinner 1be0d1135f
bpo-36352: Clarify fileutils.h documentation (GH-12406)
The last parameter of _Py_wreadlink(), _Py_wrealpath() and
_Py_wgetcwd() is a length, not a size: number of characters including
the trailing NUL character.

Enhance also documentation of error conditions.
2019-03-18 17:47:26 +01:00
Rémi Lapeyre b4b97af8be Fix typo in _PyObject_FastCallDict documentation (GH-12383) 2019-03-18 11:07:53 +01:00
Inada Naoki 6a16b18224
bpo-36297: remove "unicode_internal" codec (GH-12342) 2019-03-18 15:44:11 +09:00
Eric Snow d2fdd1fedf
bpo-36124: Add PyInterpreterState.dict. (gh-12132) 2019-03-15 17:47:43 -06:00
Eric Snow c11183cdcf
bpo-36097: Use only public C-API in the_xxsubinterpreters module (adding as necessary). (gh-12359) 2019-03-15 16:35:46 -06:00
Eric Snow 842a2f07f2
bpo-33608: Deal with pending calls relative to runtime shutdown. (gh-12246) 2019-03-15 15:47:51 -06:00
Victor Stinner 74f6568bbd
bpo-36301: Add _PyWstrList structure (GH-12343)
Replace messy _Py_wstrlist_xxx() functions with a new clean
_PyWstrList structure and new _PyWstrList_xxx() functions.

Changes:

* Add _PyCoreConfig.use_module_search_paths to decide if
  _PyCoreConfig.module_search_paths should be computed or not, to
  support empty search path list.
* _PyWstrList_Clear() sets length to 0 and items to NULL, whereas
  _Py_wstrlist_clear() only freed memory.
* _PyWstrList_Append() returns an int, whereas _Py_wstrlist_append()
  returned _PyInitError.
* _PyWstrList uses Py_ssize_t for the length, instead of int.
* Replace (int, wchar_t**) with _PyWstrList in:

  * _PyPreConfig
  * _PyCoreConfig
  * _PyPreCmdline
  * _PyCmdline

* Replace "int orig_argv; wchar_t **orig_argv;"
  with "_PyWstrList orig_argv".
* _PyCmdline and _PyPreCmdline now also copy wchar_argv.
* Rename _PyArgv_Decode() to _PyArgv_AsWstrList().
* PySys_SetArgvEx() now pass the fixed (argc, argv) to
  _PyPathConfig_ComputeArgv0() (don't pass negative argc or NULL
  argv).
* _PyOS_GetOpt() uses Py_ssize_t
2019-03-15 15:08:05 +01:00
Serhiy Storchaka 3191391515
bpo-36127: Argument Clinic: inline parsing code for keyword parameters. (GH-12058) 2019-03-14 10:32:22 +02:00
Guido van Rossum 10f8ce6688 bpo-36280: Add Constant.kind field (GH-12295)
The value is a string for string and byte literals, None otherwise.
It is 'u' for u"..." literals, 'b' for b"..." literals, '' for "..." literals.
The 'r' (raw) prefix is ignored.
Does not apply to f-strings.

This appears sufficient to make mypy capable of using the stdlib ast module instead of typed_ast (assuming a mypy patch I'm working on).

WIP: I need to make the tests pass. @ilevkivskyi @serhiy-storchaka 



https://bugs.python.org/issue36280
2019-03-13 13:00:46 -07:00
tyomitch 1b304f992d Remove d_initial from the parser as it is unused (GH-12212)
d_initial, the first state of a particular DFA in the parser has always been initialized to 0 in the old pgen as well as the new pgen. As this value is not used and the first state of each DFA is assumed to be the first element in the array representing it, remove d_initial from the parser to reduce complexity.
2019-03-09 15:35:50 +00:00
Eric Snow 8479a3426e
bpo-33608: Make sure locks in the runtime are properly re-created. (gh-12245) 2019-03-08 23:44:33 -07:00
Eric Snow 5be45a6105
bpo-33608: Minor cleanup related to pending calls. (gh-12247) 2019-03-08 22:47:07 -07:00
Eric Snow 2aab5d310c
Fix the Py_atomic_* macros. (#12240)
The macros were working only because our usage happened to parse correctly.  Changing that usage (e.g. with pointers) would break the macros.  This fixes that.
2019-03-08 12:06:56 -07:00
Guido van Rossum 495da29225 bpo-35975: Support parsing earlier minor versions of Python 3 (GH-12086)
This adds a `feature_version` flag to `ast.parse()` (documented) and `compile()` (hidden) that allow tweaking the parser to support older versions of the grammar. In particular if `feature_version` is 5 or 6, the hacks for the `async` and `await` keyword from PEP 492 are reinstated. (For 7 or higher, these are unconditionally treated as keywords, but they are still special tokens rather than `NAME` tokens that the parser driver recognizes.)



https://bugs.python.org/issue35975
2019-03-07 12:38:08 -08:00
Jeremy Kloth edad38e3e0 bpo-9566: Fix compiler warnings in gcmodule.c (GH-11010)
Change PyDTrace_GC_DONE() argument type from int to Py_ssize_t.
2019-03-06 17:54:12 +01:00
Victor Stinner 4fffd380a4
bpo-36142: _PyPreConfig_Read() sets LC_CTYPE (GH-12188)
* _PyPreConfig_Read() now sets temporarily LC_CTYPE to the user
  preferred locale, as _PyPreConfig_Write() will do permanentely.
* Fix _PyCoreConfig_Clear(): clear run_xxx attributes
* _Py_SetArgcArgv() doesn't have to be exported
* _PyCoreConfig_SetGlobalConfig() no longer applies preconfig
2019-03-06 01:44:31 +01:00
Victor Stinner c656e25667
bpo-36142: Add _PyPreConfig_SetAllocator() (GH-12187)
* _PyPreConfig_Write() now reallocates the pre-configuration with the
  new memory allocator.
* It is no longer needed to force the "default raw memory allocator"
  to clear pre-configuration and core configuration. Simplify the
  code.
* _PyPreConfig_Write() now does nothing if called after
  Py_Initialize(): no longer check if the allocator is the same.
* Remove _PyMem_GetDebugAllocatorsName(): dev mode sets again
  allocator to "debug".
2019-03-06 01:13:43 +01:00
Victor Stinner 7d2ef3ef50
bpo-36142: _PyPreConfig_Write() sets the allocator (GH-12186)
* _PyPreConfig_Write() now sets the memory allocator.
* _PyPreConfig_Write() gets a return type: _PyInitError.
* _Py_InitializeCore() now reads and writes the pre-configuration
  (set the memory allocator, configure the locale) before reading and
  writing the core configuration.
2019-03-06 00:36:56 +01:00
Victor Stinner a9df651eb4
bpo-36142: Add _PyMem_GetDebugAllocatorsName() (GH-12185)
The development mode now uses the effective name of the debug memory
allocator ("pymalloc_debug" or "malloc_debug"). So the name doesn't
change after setting the memory allocator.
2019-03-05 23:31:54 +01:00
Serhiy Storchaka d8b3a98c90
bpo-36187: Remove NamedStore. (GH-12167)
NamedStore has been replaced with Store. The difference between
NamedStore and Store is handled when precess the NamedExpr node
one level upper.
2019-03-05 20:42:06 +02:00
Victor Stinner b35be4b333
bpo-36142: Add _PyPreConfig.allocator (GH-12181)
* Move 'allocator' and 'dev_mode' fields from _PyCoreConfig
  to _PyPreConfig.
* Fix InitConfigTests of test_embed: dev_mode sets allocator to
  "debug", add a new tests for env vars with dev mode enabled.
2019-03-05 17:37:44 +01:00
Victor Stinner 5a02e0d1c8
bpo-36142: Add _PyPreConfig.utf8_mode (GH-12174)
* Move following fields from _PyCoreConfig to _PyPreConfig:

  * coerce_c_locale
  * coerce_c_locale_warn
  * legacy_windows_stdio
  * utf8_mode

* _PyPreConfig_ReadFromArgv() is now responsible to choose the
  filesystem encoding
* _PyPreConfig_Write() now sets the LC_CTYPE locale
2019-03-05 12:32:09 +01:00
Victor Stinner 6dcb54228e
bpo-36142: Add _PyPreConfig_ReadFromArgv() (GH-12173)
The new function is now responsible to parse -E and -I command line
arguments.
2019-03-05 02:44:12 +01:00
Victor Stinner cad1f747da
bpo-36142: Add _PyPreConfig structure (GH-12172)
* Add _PyPreConfig structure
* Move 'ignored' and 'use_environment' fields from _PyCoreConfig
  to _PyPreConfig
* Add a new "_PyPreConfig preconfig;" field to _PyCoreConfig
2019-03-05 02:01:27 +01:00
Victor Stinner 4d61e6e3b8
Revert: bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (GH-11617) (GH-12159)
* Revert "bpo-36097: Use only public C-API in the_xxsubinterpreters module (adding as necessary). (#12003)"

This reverts commit bcfa450f21.

* Revert "bpo-33608: Simplify ceval's DISPATCH by hoisting eval_breaker ahead of time. (gh-12062)"

This reverts commit bda918bf65.

* Revert "bpo-33608: Use _Py_AddPendingCall() in _PyCrossInterpreterData_Release(). (gh-12024)"

This reverts commit b05b711a2c.

* Revert "bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (GH-11617)"

This reverts commit ef4ac967e2.
2019-03-04 14:21:28 +01:00
pxinwr f4b0a1c0da bpo-31904: Add encoding support for VxWorks RTOS (GH-12051)
Use UTF-8 as the system encoding on VxWorks.

The main reason are:

1. The locale is frequently misconfigured.
2. Missing some functions to deal with locale in VxWorks C library.
2019-03-04 10:02:06 +01:00
Eric Snow bcfa450f21
bpo-36097: Use only public C-API in the_xxsubinterpreters module (adding as necessary). (#12003) 2019-03-01 16:50:31 -07:00
Pablo Galindo 1f24a719e7
bpo-35808: Retire pgen and use pgen2 to generate the parser (GH-11814)
Pgen is the oldest piece of technology in the CPython repository, building it requires various #if[n]def PGEN hacks in other parts of the code and it also depends more and more on CPython internals. This commit removes the old pgen C code and replaces it for a new version implemented in pure Python. This is a modified and adapted version of lib2to3/pgen2 that can generate grammar files compatibles with the current parser.

This commit also eliminates all the #ifdef and code branches related to pgen, simplifying the code and making it more maintainable. The regen-grammar step now uses $(PYTHON_FOR_REGEN) that can be any version of the interpreter, so the new pgen code maintains compatibility with older versions of the interpreter (this also allows regenerating the grammar with the current CI solution that uses Python3.5). The new pgen Python module also makes use of the Grammar/Tokens file that holds the token specification, so is always kept in sync and avoids having to maintain duplicate token definitions.
2019-03-01 15:34:44 -08:00
Eric Snow bda918bf65
bpo-33608: Simplify ceval's DISPATCH by hoisting eval_breaker ahead of time. (gh-12062)
This includes fixes to various _Py_atomic_* macros.
2019-03-01 13:15:45 -07:00
Victor Stinner 91b9ecf82c
bpo-36142: Add preconfig.c (GH-12128)
* Add _PyArgv_Decode() function
* Move _Py_ClearFileSystemEncoding() and _Py_SetFileSystemEncoding()
  to preconfig.c.
2019-03-01 17:52:56 +01:00
Victor Stinner 95e2cbf32f
bpo-36142: Move command line parsing to coreconfig.c (GH-12123)
* Add _PyCoreConfig_ReadFromArgv() function which parses command line
  options: move code from main.c to coreconfig.c.
* Add _PyCoreConfig_Write() to write the new configuration: coerce
  the LC_CTYPE locale, set Py_xxx global configuration variables,
  etc.
* _PyCoreConfig_ReadFromArgv() now only changes the LC_CTYPE locale
  temporarily. _PyCoreConfig_Write() becomes responsible to set the
  LC_CTYPE locale.
* Add _Py_SetArgcArgv() and _Py_ClearArgcArgv() functions
* Rename many "pymain_xxx()" functions
* Add "const" to some function parameters
* Reorganize main.c to declare functions in the order in which they
  are called.
2019-03-01 16:25:19 +01:00
Victor Stinner 62be763348
bpo-36142: Remove _PyMain structure (GH-12120)
* Move fields from _PyMain to _PyCoreConfig:

  * skip_first_line
  * run_command
  * run_module
  * run_filename

* Replace _PyMain.stdin_is_interactive with a new
  stdin_is_interactive(config) function
* Rename _PyMain to _PyArgv. Add "const _PyArgv *args" field
  to _PyCmdline.
2019-03-01 13:10:14 +01:00
Victor Stinner dfe884759d
bpo-36142: Rework error reporting in pymain_main() (GH-12113)
Add a new _Py_INIT_EXIT() macro to be able to exit Python with an
exitcode using _PyInitError API. Rewrite function calls by
pymain_main() to use _PyInitError.

Changes:

* Remove _PyMain.err and _PyMain.status field
* Add _Py_INIT_EXIT() macro and _PyInitError.exitcode field.
* Rename _Py_FatalInitError() to _Py_ExitInitError().
2019-03-01 12:14:41 +01:00
Victor Stinner f684d83d86
bpo-36142: Exclude coreconfig.h from Py_LIMITED_API (GH-12111)
The whole coreconfig.h header is now excluded from Py_LIMITED_API.
Move functions definitions into a new internal pycore_coreconfig.h
header.

* Move Include/coreconfig.h to Include/cpython/coreconfig.h
* coreconfig.h header is now excluded from Py_LIMITED_API
* Move functions to pycore_coreconfig.h
2019-03-01 03:44:13 +01:00
pxinwr 32f5fdd7f4 bpo-31904: Add cross-build support for VxWorks RTOS (GH-11968) 2019-02-27 12:09:28 +01:00
Sergey Fedoseev f1b9abe35f bpo-36030: Remove _PyStack_AsTuple() and _PyStack_AsTupleSlice() (GH-12032) 2019-02-25 22:37:26 +01:00
Łukasz Langa b5853e2650
Post v3.8.0a2 2019-02-25 21:08:21 +01:00
Łukasz Langa 3025a7eebc Python 3.8.0a2
-----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE4/8oOcBIslwITevpsmmV4xAlBWgFAlxz2v0ACgkQsmmV4xAl
 BWjC4g/+OlySJwyU24YgF8XHihQiDSBwFGXgjdQ+KewOLbgCOofftXvEjhlozHVJ
 YxCiAGBPeFOUpsBZL1OHtihHSEYMS/XsGGWe/MKU+LaVtNnrLZAIFdX9uMkpfebg
 5Ndjs9PY5afxV0+QRfm0gz65V2JdHk4XuTbt5TUcji2i1rB021LKRXwnqex2R7Ly
 Mnwhmo/BDyT2t1+LYNEjBQ3QIQWZ9g/WbXdhRvxfccydVzwk9YX40g3yx0LkokDY
 Ej8AMk5A/A7a7gHOJebVqyLPbsg1rIfFe8r0JvCftH3YEQIshpbUgZsZCExgOsLm
 mGehPnOfc1D1mAKiWoDgm52+u9ae+IcxfktYM+spiUY69mQhwOhigPpfdUfZngW9
 H5sRu9dq9+KmDGrOb3jDY4coUiXnaRuGMcfosGqm+Q7gJXoRxE4zljVJwRdQpPea
 xpVeb4W3q58bfv+R60C88wuwv6OjdE0rL9VKLXndCoHV3YZTexS162S6d00rOOhB
 wYBViTxSh42+lUqn8CWJFDUQFnbTj2zArCMpDjIIDTWvWJEEczhEwKgDW7l5/sZE
 UOEm1xwn+3258gsxzPDRn1mZM99X1OyLx/hnSfAU2mv/2NFKRXYWgAmIhXtQqLHi
 t3BxwVgruP/yhswrPTf0Wz3RfZh4KH/5SH75O0YTykdzKMyPFIQ=
 =21b3
 -----END PGP SIGNATURE-----
gpgsig -----BEGIN PGP SIGNATURE-----
 
 iQJEBAABCAAuFiEE4/8oOcBIslwITevpsmmV4xAlBWgFAlx0SwkQHGx1a2FzekBs
 YW5nYS5wbAAKCRCyaZXjECUFaDW3D/9OM6xt4MNsIuRLfqI7NEqt6HRFrk96CCyW
 ++QPk+hKXKmRDNFRiriJoAZjCJRjvY7wqrt3dbJfVkKYnBvyxPCO+9rLgcI/J71G
 Wyi+mcm4TeVl5YfTFN9+/Y7rA3upcrGQsr/3eYDtT2y1x3XmxmpniwKNWyTDSMDg
 7lEwANLVj5u2FTojQjZHNVfc8D+V0w676s2IfGcIm8COckY+w1ti9QLF8qgLshAg
 ocEW8CKYF4S69QyYwngNiheGdO423kmAxOyWMCeWBdS5jq6wMmMIG9H31P35RdEi
 QtSkhq6sSabt+IwTYTe/SjxFXmSh8NgbnKJkgq3w33iEuNr1SVc3vCak+sDYtV6N
 dtyYUtYU+fqyrbhMP2bC0TR9T0aV63EXXXRAnhgIR0ZF3AtsnuLoqsWuhtobwypb
 DWem1jhhDywqZib1SCdgDBbW943yqcd86slC2cVlZzZi8odUKP6Lz6kSIadtWItb
 IKrSc8XCJ9J/1Kl1LxBtrm2IWBlH1VB+VHyT8lxuYR6NRrcFVzTIZBAjWtdUsE16
 hZocozrbkQkDPUs2qMkt6Op17dAfhb5geb7QdEsmTiMaiMYtESx7tmA5kihs7gkC
 QMPtPMQMFmPCnd8iE4W8kEKvHr+g7xbGU9JWUpAZomWZrzj6OnsCfVQRi8kmMZ+V
 RsFLW7aLLA==
 =MEsg
 -----END PGP SIGNATURE-----

Merge tag 'v3.8.0a2'

Python 3.8.0a2
2019-02-25 21:07:31 +01:00
Sergey Fedoseev 234531b446 bpo-36030: Add _PyTuple_FromArray() function (GH-11954) 2019-02-25 17:59:12 +01:00
Serhiy Storchaka a24107b04c
bpo-35459: Use PyDict_GetItemWithError() instead of PyDict_GetItem(). (GH-11112) 2019-02-25 17:59:46 +02:00
Serhiy Storchaka 6a44f6eef3
bpo-36048: Use __index__() instead of __int__() for implicit conversion if available. (GH-11952)
Deprecate using the __int__() method in implicit conversions of Python
numbers to C integers.
2019-02-25 17:57:58 +02:00
Łukasz Langa 23f4589b4b
v3.8.0a2 2019-02-25 13:08:32 +01:00
Eric Snow ef4ac967e2
bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (GH-11617)
This involves moving the global "pending calls" state to PyInterpreterState.

https://bugs.python.org/issue33608
2019-02-24 15:40:47 -08:00
Eric Snow 64d6cc826d
bpo-35724: Explicitly require the main interpreter for signal-handling. (GH-11530)
Ensure that the main interpreter is active (in the main thread) for signal-handling operations. This is increasingly relevant as people use subinterpreters more.

https://bugs.python.org/issue35724
2019-02-23 15:40:43 -07:00
Eric Snow be3b295838
bpo-35886: Make PyInterpreterState an opaque type in the public API. (GH-11731)
Move PyInterpreterState into the "internal" header files.
2019-02-23 11:35:52 -07:00
Gregory P. Smith 38f11cc3f6
bpo-1054041: Exit properly after an uncaught ^C. (#11862)
* bpo-1054041: Exit properly by a signal after a ^C.

An uncaught KeyboardInterrupt exception means the user pressed ^C and
our code did not handle it.  Programs that install SIGINT handlers are
supposed to reraise the SIGINT signal to the SIG_DFL handler in order
to exit in a manner that their calling process can detect that they
died due to a Ctrl-C.  https://www.cons.org/cracauer/sigint.html

After this change on POSIX systems

 while true; do python -c 'import time; time.sleep(23)'; done

can be stopped via a simple Ctrl-C instead of the shell infinitely
restarting a new python process.

What to do on Windows, or if anything needs to be done there has not
yet been determined.  That belongs in its own PR.

TODO(gpshead): A unittest for this behavior is still needed.

* Do the unhandled ^C check after pymain_free.

* Return STATUS_CONTROL_C_EXIT on Windows.

* Fix ifdef around unistd.h include.

* 📜🤖 Added by blurb_it.

* Add STATUS_CTRL_C_EXIT to the os module on Windows

* Add unittests.

* Don't send CTRL_C_EVENT in the Windows test.

It was causing CI systems to bail out of the entire test suite.

See https://dev.azure.com/Python/cpython/_build/results?buildId=37980
for example.

* Correct posix test (fail on macOS?) check.

* STATUS_CONTROL_C_EXIT must be unsigned.

* Improve the error message.

* test typo :)

* Skip if the bash version is too old.

...and rename the windows test to reflect what it does.

* min bash version is 4.4, detect no bash.

* restore a blank line i didn't mean to delete.

* PyErr_Occurred() before the Py_DECREF(co);

* Don't add os.STATUS_CONTROL_C_EXIT as a constant.

* Update the Windows test comment.

* Refactor common logic into a run_eval_code_obj fn.
2019-02-16 12:57:40 -08:00
Łukasz Langa ca7d2933a3
Post 3.8.0a1 2019-02-04 09:39:24 +01:00
Łukasz Langa e75eeb00b5
v3.8.0a1 2019-02-03 14:17:38 +01:00
Guido van Rossum dcfcd146f8 bpo-35766: Merge typed_ast back into CPython (GH-11645) 2019-01-31 12:40:27 +01:00
Emily Morehouse 8f59ee01be
bpo-35224: PEP 572 Implementation (#10497)
* Add tokenization of :=
- Add token to Include/token.h. Add token to documentation in Doc/library/token.rst.
- Run `./python Lib/token.py` to regenerate Lib/token.py.
- Update Parser/tokenizer.c: add case to handle `:=`.

* Add initial usage of := in grammar.

* Update Python.asdl to match the grammar updates. Regenerated Include/Python-ast.h and Python/Python-ast.c

* Update AST and compiler files in Python/ast.c and Python/compile.c. Basic functionality, this isn't scoped properly

* Regenerate Lib/symbol.py using `./python Lib/symbol.py`

* Tests - Fix failing tests in test_parser.py due to changes in token numbers for internal representation

* Tests - Add simple test for := token

* Tests - Add simple tests for named expressions using expr and suite

* Tests - Update number of levels for nested expressions to prevent stack overflow

* Update symbol table to handle NamedExpr

* Update Grammar to allow assignment expressions in if statements.
Regenerate Python/graminit.c accordingly using `make regen-grammar`

* Tests - Add additional tests for named expressions in RoundtripLegalSyntaxTestCase, based on examples and information directly from PEP 572

Note: failing tests are currently commented out (4 out of 24 tests currently fail)

* Tests - Add temporary syntax test failure tests in test_parser.py

Note: There is an outstanding TODO for this -- syntax tests need to be
moved to a different file (presumably test_syntax.py), but this is
covering what needs to be tested at the moment, and it's more convenient
to run a single test for the time being

* Add support for allowing assignment expressions as function argument annotations. Uncomment tests for these cases because they all pass now!

* Tests - Move existing syntax tests out of test_parser.py and into test_named_expressions.py. Refactor syntax tests to use unittest

* Add TargetScopeError exception to extend SyntaxError

Note: This simply creates the TargetScopeError exception, it is not yet
used anywhere

* Tests - Update tests per PEP 572

Continue refactoring test suite:
The named expression test suite now checks for any invalid cases that
throw exceptions (no longer limited to SyntaxErrors), assignment tests
to ensure that variables are properly assigned, and scope tests to
ensure that variable availability and values are correct

Note:
- There are still tests that are marked to skip, as they are not yet
implemented
- There are approximately 300 lines of the PEP that have not yet been
addressed, though these may be deferred

* Documentation - Small updates to XXX/todo comments

- Remove XXX from child description in ast.c
- Add comment with number of previously supported nested expressions for
3.7.X in test_parser.py

* Fix assert in seq_for_testlist()

* Cleanup - Denote "Not implemented -- No keyword args" on failing test case. Fix PEP8 error for blank lines at beginning of test classes in test_parser.py

* Tests - Wrap all file opens in `with...as` to ensure files are closed

* WIP: handle f(a := 1)

* Tests and Cleanup - No longer skips keyword arg test. Keyword arg test now uses a simpler test case and does not rely on an external file. Remove print statements from ast.c

* Tests - Refactor last remaining test case that relied on on external file to use a simpler test case without the dependency

* Tests - Add better description of remaning skipped tests. Add test checking scope when using assignment expression in a function argument

* Tests - Add test for nested comprehension, testing value and scope. Fix variable name in skipped comprehension scope test

* Handle restriction of LHS for named expressions - can only assign to LHS of type NAME. Specifically, restrict assignment to tuples

This adds an alternative set_context specifically for named expressions,
set_namedexpr_context. Thus, context is now set differently for standard
assignment versus assignment for named expressions in order to handle
restrictions.

* Tests - Update negative test case for assigning to lambda to match new error message. Add negative test case for assigning to tuple

* Tests - Reorder test cases to group invalid syntax cases and named assignment target errors

* Tests - Update test case for named expression in function argument - check that result and variable are set correctly

* Todo - Add todo for TargetScopeError based on Guido's comment (2b3acd37bd (r30472562))

* Tests - Add named expression tests for assignment operator in function arguments

Note: One of two tests are skipped, as function arguments are currently treating
an assignment expression inside of parenthesis as one child, which does
not properly catch the named expression, nor does it count arguments
properly

* Add NamedStore to expr_context. Regenerate related code with `make regen-ast`

* Add usage of NamedStore to ast_for_named_expr in ast.c. Update occurances of checking for Store to also handle NamedStore where appropriate

* Add ste_comprehension to _symtable_entry to track if the namespace is a comprehension. Initialize ste_comprehension to 0. Set set_comprehension to 1 in symtable_handle_comprehension

* s/symtable_add_def/symtable_add_def_helper. Add symtable_add_def to handle grabbing st->st_cur and passing it to symtable_add_def_helper. This now allows us to call the original code from symtable_add_def by instead calling symtable_add_def_helper with a different ste.

* Refactor symtable_record_directive to take lineno and col_offset as arguments instead of stmt_ty. This allows symtable_record_directive to be used for stmt_ty and expr_ty

* Handle elevating scope for named expressions in comprehensions.

* Handle error for usage of named expression inside a class block

* Tests - No longer skip scope tests. Add additional scope tests

* Cleanup - Update error message for named expression within a comprehension within a class. Update comments. Add assert for symtable_extend_namedexpr_scope to validate that we always find at least a ModuleScope if we don't find a Class or FunctionScope

* Cleanup - Add missing case for NamedStore in expr_context_name. Remove unused var in set_namedexpr_content

* Refactor - Consolidate set_context and set_namedexpr_context to reduce duplicated code. Special cases for named expressions are handled by checking if ctx is NamedStore

* Cleanup - Add additional use cases for ast_for_namedexpr in usage comment. Fix multiple blank lines in test_named_expressions

* Tests - Remove unnecessary test case. Renumber test case function names

* Remove TargetScopeError for now. Will add back if needed

* Cleanup - Small comment nit for consistency

* Handle positional argument check with named expression

* Add TargetScopeError exception definition. Add documentation for TargetScopeError in c-api docs. Throw TargetScopeError instead of SyntaxError when using a named expression in a comprehension within a class scope

* Increase stack size for parser by 200. This is a minimal change (approx. 5kb) and should not have an impact on any systems. Update parser test to allow 99 nested levels again

* Add TargetScopeError to exception_hierarchy.txt for test_baseexception.py_

* Tests - Major update for named expression tests, both in test_named_expressions and test_parser

- Add test for TargetScopeError
- Add tests for named expressions in comprehension scope and edge cases
- Add tests for named expressions in function arguments (declarations
and call sites)
- Reorganize tests to group them more logically

* Cleanup - Remove unnecessary comment

* Cleanup - Comment nitpicks

* Explicitly disallow assignment expressions to a name inside parentheses, e.g.: ((x) := 0)

- Add check for LHS types to detect a parenthesis then a name (see note)
- Add test for this scenario
- Update tests for changed error message for named assignment to a tuple
(also, see note)

Note: This caused issues with the previous error handling for named assignment
to a LHS that contained an expression, such as a tuple. Thus, the check
for the LHS of a named expression must be changed to be more specific if
we wish to maintain the previous error messages

* Cleanup - Wrap lines more strictly in test file

* Revert "Explicitly disallow assignment expressions to a name inside parentheses, e.g.: ((x) := 0)"

This reverts commit f1531400ca7d7a2d148830c8ac703f041740896d.

* Add NEWS.d entry

* Tests - Fix error in test_pickle.test_exceptions by adding TargetScopeError to list of exceptions

* Tests - Update error message tests to reflect improved messaging convention (s/can't/cannot)

* Remove cases that cannot be reached in compile.c. Small linting update.

* Update Grammar/Tokens to add COLONEQUAL. Regenerate all files

* Update TargetScopeError PRE_INIT and POST_INIT, as this was purposefully left out when fixing rebase conflicts

* Add NamedStore back and regenerate files

* Pass along line number and end col info for named expression

* Simplify News entry

* Fix compiler warning and explicity mark fallthrough
2019-01-24 16:49:56 -07:00
Victor Stinner ab67281e95
bpo-35713: Reorganize sys module initialization (GH-11658)
* Rename _PySys_BeginInit() to _PySys_InitCore().
* Rename _PySys_EndInit() to _PySys_InitMain().
* Add _PySys_Create(). It calls _PySys_InitCore() which becomes
  private.
* Add _PySys_SetPreliminaryStderr().
* Rename _Py_ReadyTypes() to _PyTypes_Init().
* Misc code cleanup.
2019-01-23 15:04:40 +01:00
Victor Stinner 6d43f6f081
bpo-35713: Split _Py_InitializeCore into subfunctions (GH-11650)
* Split _Py_InitializeCore_impl() into subfunctions: add multiple pycore_init_xxx() functions
* Preliminary sys.stderr is now set earlier to get an usable
  sys.stderr ealier.
* Move code into _Py_Initialize_ReconfigureCore() to be able to call
  it from _Py_InitializeCore().
* Split _PyExc_Init(): create a new _PyBuiltins_AddExceptions()
  function.
* Call _PyExc_Init() earlier in _Py_InitializeCore_impl()
  and new_interpreter() to get working exceptions earlier.
* _Py_ReadyTypes() now returns _PyInitError rather than calling
  Py_FatalError().
* Misc code cleanup
2019-01-22 21:18:05 +01:00
Victor Stinner bf4ac2d2fd
bpo-35713: Rework Python initialization (GH-11647)
* The PyByteArray_Init() and PyByteArray_Fini() functions have been
  removed. They did nothing since Python 2.7.4 and Python 3.2.0, were
  excluded from the limited API (stable ABI), and were not
  documented.
* Move "_PyXXX_Init()" and "_PyXXX_Fini()" declarations from
  Include/cpython/pylifecycle.h to
  Include/internal/pycore_pylifecycle.h. Replace
  "PyAPI_FUNC(TYPE)" with "extern TYPE".
* _PyExc_Init() now returns an error on failure rather than calling
  Py_FatalError(). Move macros inside _PyExc_Init() and undefine them
  when done. Rewrite macros to make them look more like statement:
  add ";" when using them, add "do { ... } while (0)".
* _PyUnicode_Init() now returns a _PyInitError error rather than call
  Py_FatalError().
* Move stdin check from _PySys_BeginInit() to init_sys_streams().
* _Py_ReadyTypes() now returns a _PyInitError error rather than
  calling Py_FatalError().
2019-01-22 17:39:03 +01:00
Ivan Levkivskyi 9932a22897
bpo-33416: Add end positions to Python AST (GH-11605)
The majority of this PR is tediously passing `end_lineno` and `end_col_offset` everywhere. Here are non-trivial points:
* It is not possible to reconstruct end positions in AST "on the fly", some information is lost after an AST node is constructed, so we need two more attributes for every AST node `end_lineno` and `end_col_offset`.
* I add end position information to both CST and AST.  Although it may be technically possible to avoid adding end positions to CST, the code becomes more cumbersome and less efficient.
* Since the end position is not known for non-leaf CST nodes while the next token is added, this requires a bit of extra care (see `_PyNode_FinalizeEndPos`). Unless I made some mistake, the algorithm should be linear.
* For statements, I "trim" the end position of suites to not include the terminal newlines and dedent (this seems to be what people would expect), for example in
  ```python
  class C:
      pass

  pass
  ```
  the end line and end column for the class definition is (2, 8).
* For `end_col_offset` I use the common Python convention for indexing, for example for `pass` the `end_col_offset` is 4 (not 3), so that `[0:4]` gives one the source code that corresponds to the node.
* I added a helper function `ast.get_source_segment()`, to get source text segment corresponding to a given AST node. It is also useful for testing.

An (inevitable) downside of this PR is that AST now takes almost 25% more memory. I think however it is probably justified by the benefits.
2019-01-22 11:18:22 +00:00
Minmin Gong 7a2368063f bpo-35758: Fix building on ARM + MSVC (gh-11531)
* Disable x87 control word for non-x86 targets

On msvc, x87 control word is only available on x86 target. Need to disable it for other targets to prevent compiling problems.

* Include immintrin.h on x86 and x64 only

Immintrin.h is only available on x86 and x64. Need to disable it for other targets to prevent compiling problems.
2019-01-21 21:49:40 +01:00
Eric Snow fdf282d609
bpo-35423: Stop using the "pending calls" machinery for signals. (gh-10972)
This change separates the signal handling trigger in the eval loop from the "pending calls" machinery. There is no semantic change and the difference in performance is insignificant.

The change makes both components less confusing. It also eliminates the risk of changes to the pending calls affecting signal handling. This is particularly relevant for some upcoming pending calls changes I have in the works.
2019-01-11 14:26:55 -07:00
Serhiy Storchaka 4fa9591025
bpo-35582: Argument Clinic: inline parsing code for positional parameters. (GH-11313) 2019-01-11 16:01:14 +02:00
Gregory P. Smith 387512c7ec
bpo-28503: Use crypt_r() when available instead of crypt() (GH-11373)
Use crypt_r() when available instead of crypt() in the crypt module.

As a nice side effect: This also avoids a memory sanitizer flake as clang msan doesn't know about crypt's internal libc allocated buffer.
2018-12-30 15:42:32 -08:00
Serhiy Storchaka 32d96a2b5b
bpo-23867: Argument Clinic: inline parsing code for a single positional parameter. (GH-9689) 2018-12-25 13:23:47 +02:00
Serhiy Storchaka 8ac658114d
bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370)
"Include/token.h", "Lib/token.py" (containing now some data moved from
"Lib/tokenize.py") and new files "Parser/token.c" (containing the code
moved from "Parser/tokenizer.c") and "Doc/library/token-list.inc" (included
in "Doc/library/token.rst") are now generated from "Grammar/Tokens" by
"Tools/scripts/generate_token.py". The script overwrites files only if
needed and can be used on the read-only sources tree.

"Lib/symbol.py" is now generated by "Tools/scripts/generate_symbol_py.py"
instead of been executable itself.

Added new make targets "regen-token" and "regen-symbol" which are now
dependencies of "regen-all".

The documentation contains now strings for operators and punctuation tokens.
2018-12-22 11:18:40 +02:00
Arthur Neufeld 3e8f962e63 bpo-35259: Limit `Py_FinalizeEx()` to `Py_LIMITED_API >= 0x03060000`. (GH-10620) 2018-12-20 23:11:03 +02:00
Serhiy Storchaka bb86bf4c4e
bpo-35444: Unify and optimize the helper for getting a builtin object. (GH-11047)
This speeds up pickling of some iterators.

This fixes also error handling in pickling methods when fail to
look up builtin "getattr".
2018-12-11 08:28:18 +02:00
Victor Stinner 54ba556c6c
bpo-35134: Create Include/cpython/tupleobject.h (GH-10764)
Move tupleobject.h code surrounded by "#ifndef Py_LIMITED_API"
to a new Include/cpython/tupleobject.h header file.

Add cpython/ header files to Makefile.pre.in and pythoncore project
of PCbuild.
2018-11-28 13:01:32 +01:00
Victor Stinner 9bdd2de84c
bpo-35134: Don't define types twice in header files (GH-10754)
Fix the following clang warning:

    Include/cpython/pystate.h:217:3: warning: redefinition of
    typedef 'PyThreadState' is a C11 feature [-Wtypedef-redefinition]
2018-11-27 23:54:59 +01:00
Victor Stinner f2a9d5c837
bpo-35134: Create Include/cpython/pystate.h (GH-10733)
Move pystate.h code surrounded by "#ifndef Py_LIMITED_API"
to a new Include/cpython/pystate.h header file.
2018-11-27 00:20:00 +01:00
Victor Stinner ffedd9ad2a
bpo-35134: Create Include/cpython/dictobject.h (GH-10732)
* Move dictobject.h code surrounded by "#ifndef Py_LIMITED_API"
  to a new Include/cpython/dictobject.h header file.
* Add PyAPI_FUNC() to _PyDictView_New().
* Reorganize dictobject.h: move views and iterators at the end.
2018-11-27 00:12:26 +01:00
Victor Stinner dd12aa0aea
bpo-35134: Create Include/cpython/pylifecycle.h (GH-10731)
Move pylifecycle.h code surrounded by "#ifndef Py_LIMITED_API"
to a new Include/cpython/pylifecycle.h header file.
2018-11-27 00:12:05 +01:00
Victor Stinner 4060283fce
bpo-35134: Create Include/cpython/abstract.h (GH-10728)
Move abstract.h code surrounded by "#ifndef Py_LIMITED_API"
to a new Include/cpython/abstract.h header file.
2018-11-26 22:42:04 +01:00
Victor Stinner 5a8c240b1d
bpo-35134: Add Include/cpython/pyerrors.h (GH-10727)
Move pyerrors.h code surrounded by "#ifndef Py_LIMITED_API"
to a new Include/cpython/pyerrors.h header file.
2018-11-26 22:11:25 +01:00
Victor Stinner 75e4699b31
bpo-35134: Create Include/cpython/unicodeobject.h (GH-10680)
Move unicodeobject.h code surrounded by "#ifndef Py_LIMITED_API"
to a new Include/cpython/unicodeobject.h header file.
2018-11-26 17:29:38 +01:00
Victor Stinner 6eb996685e
bpo-35134: Create Include/cpython/object.h (GH-10679)
* Move object.h code surrounded by "#ifndef Py_LIMITED_API"
  to a new Include/cpython/object.h header file.
* "typedef struct _typeobject PyTypeObject;" is now always defined
  in object.h, but if Py_LIMITED_API is not defined,
  Include/cpython/object.h also defines the structure.
* Complete the printfunc comment to mention Py_LIMITED_API define.
2018-11-26 17:09:16 +01:00
Victor Stinner 59423e3ddd
bpo-33954: Fix _PyUnicode_InsertThousandsGrouping() (GH-10623)
Fix str.format(), float.__format__() and complex.__format__() methods
for non-ASCII decimal point when using the "n" formatter.

Changes:

* Rewrite _PyUnicode_InsertThousandsGrouping(): it now requires
  a _PyUnicodeWriter object for the buffer and a Python str object
  for digits.
* Rename FILL() macro to unicode_fill(), convert it to static inline function,
  add "assert(0 <= start);" and rework its code.
2018-11-26 13:40:01 +01:00
Victor Stinner df108dc661
Add assertion to _PyTuple_CAST(op) (GH-10712)
Add "assert(PyTuple_Check(op));" to _PyTuple_CAST() to check that the
argument is a tuple object in debug mode.

PyTuple_GET_SIZE() now uses _PyTuple_CAST() to get its assertion.
2018-11-26 13:37:13 +01:00
Victor Stinner ec13b9322d
bpo-35081: Add Include/internal/pycore_tupleobject.h (GH-10705)
Move _PyTuple_ITEMS() to a new header file:
Include/internal/pycore_tupleobject.h
2018-11-25 23:56:17 +01:00
Victor Stinner 8ac6539d85
bpo-35081: Add _PyTuple_CAST() (GH-10704) 2018-11-25 23:30:32 +01:00
Victor Stinner e421106b9e
bpo-35134: Create Include/cpython/ subdirectory (GH-10624)
Include/*.h should be the "portable Python API", whereas
Include/cpython/*.h should be the "CPython API": CPython
implementation details.

Changes:

* Create Include/cpython/ subdirectory
* "make install" now creates $prefix/include/cpython and copy
  Include/cpython/* to $prefix/include/cpython
* Create Include/cpython/objimpl.h: move objimpl.h code
  surrounded by "#ifndef Py_LIMITED_API" to cpython/objimpl.h.
* objimpl.h now includes cpython/objimpl.h
* Windows installer (MSI) now also install Include/ subdirectories:
  Include/cpython/ and Include/internal/.
2018-11-23 17:00:00 +01:00
Victor Stinner b509d52083
bpo-35059: PyObject_INIT() casts to PyObject* (GH-10674)
PyObject_INIT() and PyObject_INIT_VAR() now cast their first argument
to PyObject*, as done in Python 3.7.

Revert partially commit b4435e20a9.
2018-11-23 14:27:38 +01:00
Victor Stinner 353933e712
bpo-34523: Fix C locale coercion on FreeBSD CURRENT (GH-10672)
bpo-34523, bpo-35290: C locale coercion now resets the Python
internal "force ASCII" mode. This change fix the filesystem encoding
on FreeBSD CURRENT, which has a new "C.UTF-8" locale, when
the UTF-8 mode is disabled.

Add _Py_ResetForceASCII(): _Py_SetLocaleFromEnv() now calls it.
2018-11-23 13:08:26 +01:00
Victor Stinner 3bb183d7fb
bpo-35177, Python-ast.h: Fix "Yield" compiler warning (GH-10664)
Partially revert commit 5f2df88b63e50d23914e97ec778861a52abdeaad:
add "#undef Yield" to .c files after including Python-ast.h.

Fix the warning:

    winbase.h(102): warning C4005: 'Yield': macro redefinition
2018-11-22 18:38:38 +01:00
Victor Stinner 4d73ae7761
bpo-18407: ast.c uses Py_ssize_t for asdl_seq_LEN() iterator (GH-10655)
When iterating using asdl_seq_LEN(), use 'Py_ssize_t' type instead of
'int' for the iterator variable, to avoid downcast on 64-bit platforms.

_Py_asdl_int_seq_new() now also ensures that the index is greater than
or equal to 0.
2018-11-22 14:45:16 +01:00
Victor Stinner 2ff8fb7639
bpo-35059: Add _PyObject_CAST() macro (GH-10645)
Add _PyObject_CAST() and _PyVarObject_CAST() macros to cast argument
to PyObject* and PyVarObject* properly.
2018-11-22 02:57:29 +01:00
Victor Stinner 271753a27a
bpo-35059: Convert _PyObject_GC_TRACK() to inline function (GH-10643)
* Add _PyObject_ASSERT_FROM() and _PyObject_ASSERT_FAILED_MSG()
  macros.
* PyObject_GC_Track() now calls _PyObject_ASSERT_FAILED_MSG(),
  instead of Py_FatalError(), if the object is already tracked, to
  dump more information on error.
* _PyObject_GC_TRACK() no longer checks if the object is already
  tracked at runtime, use an assertion instead for best performances;
  PyObject_GC_Track() still checks at runtime.
* pycore_object.h now includes pycore_pystate.h.
* Convert _PyObject_GC_TRACK() and _PyObject_GC_UNTRACK() macros to
  inline functions.
2018-11-22 01:02:54 +01:00
Victor Stinner f1d002c1e0
bpo-35059: Enhance _PyObject_AssertFailed() (GH-10642)
Enhance _PyObject_AssertFailed()

* Exchange 'expr' and 'msg' parameters
* 'expr' and 'func' arguments can now be NULL
2018-11-21 23:53:44 +01:00
Victor Stinner bcda8f1d42
bpo-35081: Add Include/internal/pycore_object.h (GH-10640)
Move _PyObject_GC_TRACK() and _PyObject_GC_UNTRACK() from
Include/objimpl.h to Include/internal/pycore_object.h.
2018-11-21 22:27:47 +01:00
Victor Stinner 3e21ad1a25
bpo-35081: Move _PyGC_FINALIZED() back to C API (GH-10626)
Partially revert commit 1a6be91e6f,
move back PyGC API from the internal API to the C API:

* _PyGCHead_NEXT(g), _PyGCHead_SET_NEXT(g, p)
* _PyGCHead_PREV(g), _PyGCHead_SET_PREV(g, p)
* _PyGCHead_FINALIZED(g), _PyGCHead_SET_FINALIZED(g)
* _PyGC_FINALIZED(o), _PyGC_SET_FINALIZED(o)
* _PyGC_PREV_MASK_FINALIZED
* _PyGC_PREV_MASK_COLLECTING
* _PyGC_PREV_SHIFT
* _PyGC_PREV_MASK

_PyObject_GC_TRACK(o) and _PyObject_GC_UNTRACK(o) remain in the
internal API.
2018-11-21 00:43:09 +01:00
Victor Stinner 02e6bf7f20
bpo-28604: Fix localeconv() for different LC_MONETARY (GH-10606)
locale.localeconv() now sets temporarily the LC_CTYPE locale to the
LC_MONETARY locale if the two locales are different and monetary
strings are non-ASCII. This temporary change affects other threads.

Changes:

* locale.localeconv() can now set LC_CTYPE to LC_MONETARY to decode
  monetary fields.
* Add LocaleInfo.grouping_buffer: copy localeconv() grouping string
  since it can be replaced anytime if a different thread calls
  localeconv().
* _Py_GetLocaleconvNumeric() now requires a "struct lconv *"
  structure, so locale.localeconv() now longer calls localeconv()
  twice. Moreover, the function now requires all arguments to be
  non-NULL.
* Rename STATIC_LOCALE_INFO_INIT to LocaleInfo_STATIC_INIT.
* Move _Py_GetLocaleconvNumeric() definition from fileutils.h
  to pycore_fileutils.h. pycore_fileutils.h now includes locale.h.
* The _locale module is now built with Py_BUILD_CORE defined.
2018-11-20 16:20:16 +01:00
Steve Dower 177a41a07b
bpo-34725: Adds _Py_SetProgramFullPath so embedders may override sys.executable (GH-9860) 2018-11-17 20:41:48 -08:00
Victor Stinner 01de89cb59
bpo-35233: InitConfigTests tests more config vars (GH-10541)
test_embed.InitConfigTests tests more configuration variables.

Changes:

* InitConfigTests tests more core configuration variables:

  * base_exec_prefix
  * base_prefix
  * exec_prefix
  * home
  * legacy_windows_fs_encoding
  * legacy_windows_stdio
  * module_search_path_env
  * prefix

* "_testembed init_from_config" tests more variables:

  * argv
  * warnoptions
  * xoptions

* InitConfigTests: add check_global_config(), check_core_config() and
  check_main_config() subfunctions to cleanup the code. Move also
  constants at the class level (ex: COPY_MAIN_CONFIG).
* Fix _PyCoreConfig_AsDict(): don't set stdio_encoding twice
* Use more macros in _PyCoreConfig_AsDict() and
  _PyMainInterpreterConfig_AsDict() to reduce code duplication.
* Other minor cleanups.
2018-11-14 17:39:45 +01:00
INADA Naoki a757649fd6
bpo-35230: dict: Remove some macros (GH-10513)
Remove _Py_REF_DEBUG_COMMA, DK_DEBUG_INCREF, and DK_DEBUG_DECREF.
Convert DK_INCREF and DK_DECREF to static inline functions.
2018-11-14 18:39:27 +09:00
Victor Stinner 7ddd56f4d8
bpo-35233: Rewrite test_embed.InitConfigTests (GH-10524)
* Fix _PyCoreConfig_SetGlobalConfig(): set also Py_FrozenFlag
* Fix _PyCoreConfig_AsDict(): export also xoptions
* Add _Py_GetGlobalVariablesAsDict() and _testcapi.get_global_config()
* test.pythoninfo: dump also global configuration variables
* _testembed now serializes global, core and main configurations
  using JSON to reuse _Py_GetGlobalVariablesAsDict(),
  _PyCoreConfig_AsDict() and _PyMainInterpreterConfig_AsDict(),
  rather than duplicating code.
* test_embed.InitConfigTests now test much more configuration
  variables
2018-11-14 00:24:28 +01:00
Victor Stinner 00b137c72f
bpo-35233: Fix _PyMainInterpreterConfig_Copy() (GH-10519)
* Fix _PyMainInterpreterConfig_Copy():
   copy 'install_signal_handlers' attribute
* Add _PyMainInterpreterConfig_AsDict()
* Add unit tests on the main interpreter configuration
  to test_embed.InitConfigTests
* test.pythoninfo: log also main_config
2018-11-13 19:59:26 +01:00
Victor Stinner 1a6be91e6f
bpo-35081: Make some _PyGC macros internal (GH-10507)
* Move "GC" macros together:

  * PyObject_IS_GC()
  * _Py_AS_GC()
  * _PyObject_GC_IS_TRACKED()
  * _PyObject_GC_MAY_BE_TRACKED()

* Mark other GC macros as internal (#ifdef Py_BUILD_CORE):

  * _PyGCHead_NEXT(g), _PyGCHead_SET_NEXT(g, p)
  * _PyGCHead_PREV(g), _PyGCHead_SET_PREV(g, p)
  * _PyGCHead_FINALIZED(g), _PyGCHead_SET_FINALIZED(g)
  * _PyGC_FINALIZED(o), _PyGC_SET_FINALIZED(o)
  * _PyObject_GC_TRACK(o), _PyObject_GC_UNTRACK(o)
  * _PyGC_PREV_MASK_FINALIZED
  * _PyGC_PREV_MASK_COLLECTING
  * _PyGC_PREV_SHIFT
  * _PyGC_PREV_MASK

* Replace _PyGC_generation0 with _PyRuntime.gc.generation0
* _queuemodule.c: replace _PyObject_GC_UNTRACK()
  with with PyObject_GC_UnTrack()
* Document that  _PyObject_GC_TRACK() _PyObject_GC_UNTRACK() macros
  have been removed from the public C API.
2018-11-13 12:52:18 +01:00
Paul Ganssle 0d12672b30 bpo-35081: Remove Py_BUILD_CORE from datetime.h (GH-10416)
Datetime macros like PyDate_Check() have two implementations, one using
the C API capsule and one using direct access to the datetime type
symbols defined in _datetimemodule.c. Since the direct access versions
of the macros are only used in _datetimemodule.c, they have been moved
out of "datetime.h" and into _datetimemodule.c.

The _PY_DATETIME_IMPL macro is currently necessary in order to avoid
both duplicate definitions of these macros in _datetimemodule.c and
unnecessary declarations of C API capsule-related macros and varibles in
datetime.h.

Co-Authored-By: Victor Stinner <vstinner@redhat.com>
2018-11-13 09:02:25 +01:00
Gregory P. Smith 3015fb8ce4
bpo-35214: Add _Py_ prefix to MEMORY_SANITIZER def. (GH-10503)
Rename our new MEMORY_SANITIZER define to _Py_MEMORY_SANITIZER.
Project based C Preprocessor namespacing at its finest. :P
2018-11-12 22:01:22 -08:00
Gregory P. Smith 1584a00815
bpo-35214: Initial clang MemorySanitizer support (GH-10479)
Adds configure flags for msan and ubsan builds to make it easier to enable.
These also encode the detail that address sanitizer and memory sanitizer
should disable pymalloc.

Define MEMORY_SANITIZER when appropriate at build time and adds workarounds
to existing code to mark things as initialized where the sanitizer is otherwise unable to
determine that.  This lets our build succeed under the memory sanitizer.  not all tests
pass without sanitizer failures yet but we're in pretty good shape after this.
2018-11-12 12:07:14 -08:00
Victor Stinner 621cebe81b
bpo-35081: Rename internal headers (GH-10275)
Rename Include/internal/ headers:

* pycore_hash.h -> pycore_pyhash.h
* pycore_lifecycle.h -> pycore_pylifecycle.h
* pycore_mem.h -> pycore_pymem.h
* pycore_state.h -> pycore_pystate.h

Add missing headers to Makefile.pre.in and PCbuild:

* pycore_condvar.h.
* pycore_hamt.h
* pycore_pyhash.h
2018-11-12 16:53:38 +01:00
Victor Stinner 5f2df88b63
bpo-35177: Add dependencies between header files (GH-10361)
* ast.h now includes Python-ast.h and node.h
* parsetok.h now includes node.h and grammar.h
* symtable.h now includes Python-ast.h
* Modify asdl_c.py to enhance Python-ast.h:

  * Add #ifndef/#define Py_PYTHON_AST_H to be able to include the header
    twice
  * Add "extern { ... }" for C++
  * Undefine "Yield" macro conflicting with winbase.h

* Remove "#undef Yield" from C files, it's now done in Python-ast.h
* Remove now useless includes in C files
2018-11-12 00:56:19 +01:00
Victor Stinner d17a693fa0
bpo-35199: Add an internal _PyTuple_ITEMS() macro (GH-10434)
* _PyTuple_ITEMS() gives access to the tuple->ob_item field and cast the
  first argument to PyTupleObject*. This internal macro is only usable if
  Py_BUILD_CORE is defined.
* Replace &PyTuple_GET_ITEM(ob, 0) with _PyTuple_ITEMS(ob).
* Replace PyTuple_GET_ITEM(op, 1) with &_PyTuple_ITEMS(ob)[1].
2018-11-09 16:56:48 +01:00
Victor Stinner 130893debf
bpo-35081: Internal headers require Py_BUILD_CORE (GH-10363)
* All internal header files now require Py_BUILD_CORE or
  Py_BUILD_CORE_BUILTIN to be defined.
* _json.c is now compiled with Py_BUILD_CORE_BUILTIN to access
  pycore_accu.h header.
* Add an example to Modules/Setup to show how to build _json
  as a built-in module; it requires non trivial compiler options.
2018-11-09 13:03:37 +01:00
Victor Stinner 9fc57a3848
bpo-35081: Add pycore_fileutils.h (GH-10371)
Move Py_BUILD_CORE code from Include/fileutils.h to a new
Include/internal/pycore_fileutils.h file.
2018-11-07 00:44:03 +01:00
Victor Stinner 5ed6995675
bpo-35081: Add _PyCoreConfig_AsDict() (GH-10362)
_testcapimodule.c must not include pycore_pathconfig.h, since it's an
internal header files.

Changes:

* Add _PyCoreConfig_AsDict() function to coreconfig.c.
* Remove pycore_pathconfig.h include from _testcapimodule.h.
* pycore_pathconfig.h now requires Py_BUILD_CORE to be defined.
* _testcapimodule.c compilation now fails if it's built with
  Py_BUILD_CORE defined.
2018-11-06 15:59:52 +01:00
Rémi Lapeyre 6531bf6309 bpo-33462: Add __reversed__ to dict and dict views (GH-6827) 2018-11-06 09:38:54 +09:00
Alexey Izbyshev e2ed5adcb5 bpo-35147: Fix _Py_NO_RETURN for GCC (GH-10300)
Use `__GNUC__` instead of non-existing `__GNUC_MAJOR__`.





https://bugs.python.org/issue35147
2018-11-02 09:32:26 -07:00
Victor Stinner a1c249c405
bpo-35081: And pycore_lifecycle.h and pycore_pathconfig.h (GH-10273)
* And pycore_lifecycle.h and pycore_pathconfig.h headers to
  Include/internal/
* Move Py_BUILD_CORE specific code from coreconfig.h and
  pylifecycle.h to pycore_pathconfig.h and pycore_lifecycle.h
* Move _Py_wstrlist_XXX() definitions and _PyPathConfig code
  from pycore_state.h to pycore_pathconfig.h
* Move "Init" and "Fini" function definitions from pylifecycle.c to
  pycore_lifecycle.h.
2018-11-01 03:15:58 +01:00
Victor Stinner e281f7d80c
bpo-35081: Move accu.h to Include/internal/pycore_accu.h (GH-10271)
The accu.h header is no longer part of the Python C API: it has been
moved to the "internal" headers which are restricted to Python
itself.

Replace #include "accu.h" with #include "pycore_accu.h".
2018-11-01 02:30:36 +01:00
Victor Stinner 50b48572d9
bpo-35081: Add _PyThreadState_GET() internal macro (GH-10266)
If Py_BUILD_CORE is defined, the PyThreadState_GET() macro access
_PyRuntime which comes from the internal pycore_state.h header.
Public headers must not require internal headers.

Move PyThreadState_GET() and _PyInterpreterState_GET_UNSAFE() from
Include/pystate.h to Include/internal/pycore_state.h, and rename
PyThreadState_GET() to _PyThreadState_GET() there.

The PyThreadState_GET() macro of pystate.h is now redefined when
pycore_state.h is included, to use the fast _PyThreadState_GET().

Changes:

* Add _PyThreadState_GET() macro
* Replace "PyThreadState_GET()->interp" with
  _PyInterpreterState_GET_UNSAFE()
* Replace PyThreadState_GET() with _PyThreadState_GET() in internal C
  files (compiled with Py_BUILD_CORE defined), but keep
  PyThreadState_GET() in the public header files.
* _testcapimodule.c: replace PyThreadState_GET() with
  PyThreadState_Get(); the module is not compiled with Py_BUILD_CORE
  defined.
* pycore_state.h now requires Py_BUILD_CORE to be defined.
2018-11-01 01:51:40 +01:00
Victor Stinner 27e2d1f219
bpo-35081: Add pycore_ prefix to internal header files (GH-10263)
* Rename Include/internal/ header files:

  * pyatomic.h -> pycore_atomic.h
  * ceval.h -> pycore_ceval.h
  * condvar.h -> pycore_condvar.h
  * context.h -> pycore_context.h
  * pygetopt.h -> pycore_getopt.h
  * gil.h -> pycore_gil.h
  * hamt.h -> pycore_hamt.h
  * hash.h -> pycore_hash.h
  * mem.h -> pycore_mem.h
  * pystate.h -> pycore_state.h
  * warnings.h -> pycore_warnings.h

* PCbuild project, Makefile.pre.in, Modules/Setup: add the
  Include/internal/ directory to the search paths of header files.
* Update includes. For example, replace #include "internal/mem.h"
  with #include "pycore_mem.h".
2018-11-01 00:52:28 +01:00
Victor Stinner 2be00d987d
bpo-35081: Move Py_BUILD_CORE code to internal/mem.h (GH-10249)
* Add #include "internal/mem.h" to C files using
  _PyMem_SetDefaultAllocator().
* Include/internal/mem.h now requires Py_BUILD_CORE to be defined.
2018-10-31 20:19:24 +01:00
Victor Stinner 31368a4f0e
bpo-35081: Move Include/pyatomic.c to Include/internal/ (GH-10239)
Add pyatomic.h to the VS project (it wasn't referenced).
2018-10-30 15:14:25 +01:00
Victor Stinner 9204fb8623
bpo-35081: Cleanup pystate.c and pystate.h (GH-10240)
* Remove _PyThreadState_Current
* Replace GET_TSTATE() with PyThreadState_GET()
* Replace GET_INTERP_STATE() with _PyInterpreterState_GET_UNSAFE()
* Replace direct access to _PyThreadState_Current with
  PyThreadState_GET()
* Replace _PyThreadState_Current with
  _PyRuntime.gilstate.tstate_current
* Rename SET_TSTATE() to _PyThreadState_SET(), name more
  consistent with _PyThreadState_GET()
* Update outdated comments
2018-10-30 15:13:17 +01:00
Victor Stinner 3c09dca4b5
bpo-35059: Convert _Py_Dealloc() to static inline function (GH-10223)
Convert _Py_Dealloc() macro into a static inline function. Moreover,
it is now also defined as a static inline function if Py_TRACE_REFS
is defined.
2018-10-30 14:48:26 +01:00
Victor Stinner e1b29950bf
bpo-32030: Make _PySys_AddXOptionWithError() private (GH-10236)
Make _PySys_AddXOptionWithError() and _PySys_AddWarnOptionWithError()
functions private again. They are no longer needed to initialize Python:
_PySys_EndInit() is now responsible to add these options instead.

Moreover, PySys_AddWarnOptionUnicode() now clears the exception on
failure if possible.
2018-10-30 14:31:42 +01:00
Victor Stinner 905f1ace5f
bpo-34523: Fix config_init_fs_encoding() for ASCII (GH-10232)
* bpo-34523, bpo-34403: Fix config_init_fs_encoding(): it now uses
  ASCII if _Py_GetForceASCII() is true.
* Fix a regression of commit b2457efc78.
* Fix also a memory leak: get_locale_encoding() already allocates
  memory, no need to duplicate the string.
2018-10-30 12:58:10 +01:00
Serhiy Storchaka 3e429dcc24
bpo-33237: Improve AttributeError message for partially initialized module. (GH-6398) 2018-10-30 13:19:51 +02:00
Victor Stinner 541497e619
bpo-35059: Convert Py_XINCREF() to static inline function (GH-10224)
Convert Py_XINCREF() and Py_XDECREF() macros into static inline
functions.
2018-10-29 20:52:41 +01:00
Victor Stinner 542497aa9f
bpo-35059: Remove Py_STATIC_INLINE() macro (GH-10216)
"static inline" should be used directly. Forcing the compiler to
inline is not recommended.
2018-10-29 14:49:24 +01:00
Victor Stinner 2aaf0c1204
bpo-35059: Convert Py_INCREF() to static inline function (GH-10079)
* Convert Py_INCREF() and Py_DECREF() macros into static inline
  functions.
* Remove _Py_CHECK_REFCNT() macro: code moved into Py_DECREF().
2018-10-29 13:43:07 +01:00
Pablo Galindo 49c75a8086
bpo-35064 prefix smelly symbols that appear with COUNT_ALLOCS with _Py_ (GH-10152)
Configuring python with ./configure --with-pydebug CFLAGS="-D COUNT_ALLOCS -O0"
makes "make smelly" fail as some symbols were being exported without the "Py_" or
"_Py" prefixes.
2018-10-28 15:02:17 +00:00
Victor Stinner b4435e20a9
bpo-35059: Convert PyObject_INIT() to function (GH-10077)
* Convert PyObject_INIT() and PyObject_INIT_VAR() macros to static
  inline functions.
* Fix usage of these functions: cast to PyObject* or PyVarObject*.
2018-10-26 14:35:00 +02:00
Victor Stinner c89a932714
bpo-35053: Define _PyTraceMalloc_NewReference in object.h (GH-10107)
_PyTraceMalloc_NewReference() is now called by _Py_NewReference(), so
move its definition to object.h. Moreover, define it even if
Py_LIMITED_API is defined, since _Py_NewReference() is also exposed
even if Py_LIMITED_API is defined.
2018-10-26 00:01:56 +02:00
Victor Stinner 626bff8568
bpo-9263: Dump Python object on GC assertion failure (GH-10062)
Changes:

* Add _PyObject_AssertFailed() function.
* Add _PyObject_ASSERT() and _PyObject_ASSERT_WITH_MSG() macros.
* gc_decref(): replace assert() with _PyObject_ASSERT_WITH_MSG() to
  dump the faulty object if the assertion fails.

_PyObject_AssertFailed() calls:

* _PyMem_DumpTraceback(): try to log the traceback where the object
  memory has been allocated if tracemalloc is enabled.
* _PyObject_Dump(): log repr(obj).
* Py_FatalError(): log the current Python traceback.

_PyObject_AssertFailed() uses _PyObject_IsFreed() heuristic to check
if the object memory has been freed by a debug hook on Python memory
allocators.

Initial patch written by David Malcolm.

Co-Authored-By: David Malcolm <dmalcolm@redhat.com>
2018-10-25 17:31:10 +02:00
Victor Stinner 18618e652c
bpo-35059: Add Py_STATIC_INLINE() macro (GH-10093)
* Add Py_STATIC_INLINE() macro to declare a "static inline" function.
  If the compiler supports it, try to always inline the function even if no
  optimization level was specified.
* Modify pydtrace.h to use Py_STATIC_INLINE() when WITH_DTRACE is
  not defined.
* Add an unit test on Py_DECREF() to make sure that
  _Py_NegativeRefcount() reports the correct filename.
2018-10-25 17:28:11 +02:00
Victor Stinner 6279c1c500
bpo-35053: Add Include/tracemalloc.h (GH-10091)
* Modify object.h to ensure that pymem.h is included,
  to get _Py_tracemalloc_config variable.
* Move _PyTraceMalloc_XXX() functions to tracemalloc.h,
  they need PyObject type. Break circular dependency between pymem.h
  and object.h.
2018-10-25 15:54:13 +02:00
Victor Stinner 9e00e80e21
bpo-35053: Enhance tracemalloc to trace free lists (GH-10063)
tracemalloc now tries to update the traceback when an object is
reused from a "free list" (optimization for faster object creation,
used by the builtin list type for example).

Changes:

* Add _PyTraceMalloc_NewReference() function which tries to update
  the Python traceback of a Python object.
* _Py_NewReference() now calls _PyTraceMalloc_NewReference().
* Add an unit test.
2018-10-25 13:31:16 +02:00
Victor Stinner 82af0b63b0
bpo-9263: _PyObject_Dump() detects freed memory (GH-10061)
_PyObject_Dump() now uses an heuristic to check if the object memory
has been freed: log "<freed object>" in that case.

The heuristic rely on the debug hooks on Python memory allocators
which fills the memory with DEADBYTE (0xDB) when memory is
deallocated. Use PYTHONMALLOC=debug to always enable these debug
hooks.
2018-10-23 17:39:40 +02:00
Shivank98 e3d1455fe4 Update opcode.h header comment to mention the source data file (GH-9935)
This is intended to help code explorers find out more about what's defined there.
2018-10-18 14:53:18 -05:00
Stéphane Wirtel 74a8b6ea7e bpo-24658: Fix read/write greater than 2 GiB on macOS (GH-1705)
On macOS, fix reading from and writing into a file with a size larger than 2 GiB.
2018-10-18 01:05:04 +02:00
Serhiy Storchaka 3f22811fef
bpo-32892: Use ast.Constant instead of specific constant AST types. (GH-9445) 2018-09-27 17:42:37 +03:00
Yury Selivanov 2ec872b31e
bpo-34762: Fix contextvars C API to use PyObject* pointer types. (GH-9473) 2018-09-21 15:33:56 -04:00
Victor Stinner 06e7608207
Revert "bpo-34589: Add -X coerce_c_locale command line option (GH-9378)" (GH-9430)
* Revert "bpo-34589: Add -X coerce_c_locale command line option (GH-9378)"

This reverts commit dbdee0073c.

* Revert "bpo-34589: C locale coercion off by default (GH-9073)"

This reverts commit 7a0791b699.

* Revert "bpo-34589: Make _PyCoreConfig.coerce_c_locale private (GH-9371)"

This reverts commit 188ebfa475.
2018-09-19 14:56:36 -07:00
Christian Heimes cb5778f00c bpo-34623: Use XML_SetHashSalt in _elementtree (GH-9146)
The C accelerated _elementtree module now initializes hash randomization
salt from _Py_HashSecret instead of libexpat's default CPRNG.

Signed-off-by: Christian Heimes <christian@python.org>



https://bugs.python.org/issue34623
2018-09-18 05:38:58 -07:00
Serhiy Storchaka 7bdf28265a
bpo-32455: Add jump parameter to dis.stack_effect(). (GH-6610)
Add C API function PyCompile_OpcodeStackEffectWithJump().
2018-09-18 09:54:26 +03:00
Victor Stinner 7a0791b699
bpo-34589: C locale coercion off by default (GH-9073)
Py_Initialize() and Py_Main() cannot enable the C locale coercion
(PEP 538) anymore: it is always disabled. It can now only be enabled
by the Python program ("python3).

test_embed: get_filesystem_encoding() doesn't have to set PYTHONUTF8
nor PYTHONCOERCECLOCALE, these variables are already set in the
parent.
2018-09-17 16:22:29 -07:00
Victor Stinner 188ebfa475
bpo-34589: Make _PyCoreConfig.coerce_c_locale private (GH-9371)
_PyCoreConfig:

* Rename coerce_c_locale to _coerce_c_locale
* Rename coerce_c_locale_warn to _coerce_c_locale_warn

These fields are now private (name prefixed by "_").
2018-09-17 15:13:17 -07:00
Eric Snow 5903296045
bpo-34651: Only allow the main interpreter to fork. (gh-9279)
When os.fork() is called (on platforms that support it) all threads but the current one are destroyed in the child process. Consequently we must ensure that all but the associated interpreter are likewise destroyed. The main interpreter is critical for runtime operation, so we must ensure that fork only happens in the main interpreter.

https://bugs.python.org/issue34651
2018-09-14 14:17:20 -07:00
Benjamin Peterson ea13740a37
bpo-34674: Assume unistd.h exists on Unix. (GH-9290) 2018-09-13 21:57:31 -07:00
Victor Stinner 2094c2bea4
bpo-34567: pythoninfo gets coreconfig (GH-9043)
* Add _testcapi.get_coreconfig() to get the _PyCoreConfig of the
  interpreter
* test.pythoninfo now gets the core configuration using
  _testcapi.get_coreconfig()
2018-09-03 17:06:39 +02:00
Victor Stinner fbca90856d
bpo-34523: Use _PyCoreConfig instead of globals (GH-9005)
Use the core configuration of the interpreter, rather
than using global configuration variables. For example, replace
Py_QuietFlag with core_config->quiet.
2018-08-30 00:50:45 +02:00
Victor Stinner de42755674
bpo-34523: Py_FileSystemDefaultEncoding NULL by default (GH-9003)
* Py_FileSystemDefaultEncoding and Py_FileSystemDefaultEncodeErrors
  default value is now NULL: initfsencoding() set them
  during Python initialization.
* Document how Python chooses the filesystem encoding and error
  handler.
* Add an assertion to _PyCoreConfig_Read().
2018-08-29 23:26:55 +02:00
Victor Stinner 3d4226a832
bpo-34523: Support surrogatepass in locale codecs (GH-8995)
Add support for the "surrogatepass" error handler in
PyUnicode_DecodeFSDefault() and PyUnicode_EncodeFSDefault()
for the UTF-8 encoding.

Changes:

* _Py_DecodeUTF8Ex() and _Py_EncodeUTF8Ex() now support the
  surrogatepass error handler (_Py_ERROR_SURROGATEPASS).
* _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx() now use
  the _Py_error_handler enum instead of "int surrogateescape" to pass
  the error handler. These functions now return -3 if the error
  handler is unknown.
* Add unit tests on _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx()
  in test_codecs.
* Rename get_error_handler() to _Py_GetErrorHandler() and expose it
  as a private function.
* _freeze_importlib doesn't need config.filesystem_errors="strict"
  workaround anymore.
2018-08-29 22:21:32 +02:00
Victor Stinner b2457efc78
bpo-34523: Add _PyCoreConfig.filesystem_encoding (GH-8963)
_PyCoreConfig_Read() is now responsible to choose the filesystem
encoding and error handler. Using Py_Main(), the encoding is now
chosen even before calling Py_Initialize().

_PyCoreConfig.filesystem_encoding is now the reference, instead of
Py_FileSystemDefaultEncoding, for the Python filesystem encoding.

Changes:

* Add filesystem_encoding and filesystem_errors to _PyCoreConfig
* _PyCoreConfig_Read() now reads the locale encoding for the file
  system encoding.
* PyUnicode_EncodeFSDefault() and PyUnicode_DecodeFSDefaultAndSize()
  now use the interpreter configuration rather than
  Py_FileSystemDefaultEncoding and Py_FileSystemDefaultEncodeErrors
  global configuration variables.
* Add _Py_SetFileSystemEncoding() and _Py_ClearFileSystemEncoding()
  private functions to only modify Py_FileSystemDefaultEncoding and
  Py_FileSystemDefaultEncodeErrors in coreconfig.c.
* _Py_CoerceLegacyLocale() now takes an int rather than
  _PyCoreConfig for the warning.
2018-08-29 13:25:36 +02:00
Victor Stinner dfe0dc7453
bpo-34485: Add _PyCoreConfig.stdio_encoding (GH-8881)
* Add stdio_encoding and stdio_errors fields to _PyCoreConfig.
* Add unit tests on stdio_encoding and stdio_errors.
2018-08-29 11:47:29 +02:00
Victor Stinner 124b9eb4e4
bpo-34485: Add _Py_ClearStandardStreamEncoding() (GH-8982)
* Move Py_SetStandardStreamEncoding() from pylifecycle.c
  to coreconfig.c
* Add _Py_ClearStandardStreamEncoding() private function.
* pymain_free() now calls _Py_ClearStandardStreamEncoding().
* Add assertions add the end of _PyCoreConfig_Read()
* _PyCoreConfig_Copy(): rename COPY_STR_ATTR() macro
  to COPY_WSTR_ATTR().
* Fix get_stdio_errors() indentation.
2018-08-29 01:29:06 +02:00
Victor Stinner d500e5307a
bpo-34403: On HP-UX, force ASCII for C locale (GH-8969)
On HP-UX with C or POSIX locale, sys.getfilesystemencoding() now returns
"ascii" instead of "roman8" (when the UTF-8 Mode is disabled and the C locale
is not coerced).

nl_langinfo(CODESET) announces "roman8" whereas it uses the Latin1
encoding in practice.
2018-08-28 17:27:36 +02:00
Victor Stinner caba55b3b7
bpo-34301: Add _PyInterpreterState_Get() helper function (GH-8592)
sys_setcheckinterval() now uses a local variable to parse arguments,
before writing into interp->check_interval.
2018-08-03 15:33:52 +02:00
Victor Stinner 80b762f010
bpo-31650: Remove _Py_CheckHashBasedPycsMode global config var (GH-8608)
bpo-31650, bpo-34170: Replace _Py_CheckHashBasedPycsMode with
_PyCoreConfig._check_hash_pycs_mode. Modify PyInit__imp() and
zipimport to get the parameter from the current interpreter core
configuration.

Remove Include/internal/import.h file.
2018-08-01 18:18:07 +02:00
Victor Stinner 6c785c0ebd
bpo-34170: Add Python/coreconfig.c for _PyCoreConfig (GH-8607)
* Add Include/coreconfig.h
* Move config_*() and _PyCoreConfig_*() functions from Modules/main.c
  to a new Python/coreconfig.c file.
* Inline _Py_ReadHashSeed() into config_init_hash_seed()
* Move global configuration variables to coreconfig.c
2018-08-01 17:56:14 +02:00
Victor Stinner cfc8831f5e
Define _Py_NO_RETURN for Microsoft C compiler (GH-8606) 2018-08-01 16:41:25 +02:00
Victor Stinner 9851227382
bpo-34170: Rename _PyCoreConfig.unbuffered_stdip (GH-8594)
* Rename _PyCoreConfig.unbuffered_stdio to buffered_stdio
* Rename _PyCoreConfig.debug to parser_debug
2018-08-01 03:07:00 +02:00
Victor Stinner b75d7e2435
bpo-34170: Add _PyCoreConfig._frozen parameter (GH-8591)
Modify frozenmain.c to use _Py_InitializeFromConfig().
2018-08-01 02:13:04 +02:00
Victor Stinner 56b29b6d6f
bpo-34170, test_embed: write Py_Initialize() tests (GH-8484) 2018-07-26 18:57:56 +02:00
Victor Stinner d145775b45
bpo-34170: Cleanup pymain_read_conf() (GH-8476)
* Config: Rename ignore_environment field to use_environment.
* _PyCoreConfig_Read(): if isolated is set, use_environment and
  site_import are now always set to 0.
* Inline pymain_free_raw() into pymain_free()
* Move config_init_warnoptions() call into pymain_read_conf_impl()
* _PyCoreConfig_Read(): don't replace values if they are already set:
  faulthandler, pycache_prefix, home.
2018-07-26 16:04:56 +02:00
Serhiy Storchaka 7cb7bcff20
bpo-20260: Implement non-bitwise unsigned int converters for Argument Clinic. (GH-8434) 2018-07-26 13:22:16 +03:00
Victor Stinner ecf411c59e
bpo-34170: Enhance _PyCoreConfig_Read() (GH-8468)
* Inline cmdline_get_env_flags() into config_read_env_vars():
  _PyCoreConfig_Read() now reads much more environment variables like
  PYTHONVERBOSE.
* Allow to override faulthandler and allocator even if dev_mode=1.
  PYTHONMALLOC is now the priority over PYTHONDEVMODE.
* Fix _PyCoreConfig_Copy(): copy also install_signal_handlers,
  coerce_c_locale and coerce_c_locale_warn
* _PyCoreConfig.install_signal_handlers default is now 1: install
  signals by default
* Fix also a compiler warning: don't define _PyPathConfig type twice.
2018-07-26 02:37:22 +02:00
Victor Stinner 60b04c9f6f
bpo-34228: Allow PYTHONTRACEMALLOC=0 (GH-8467)
PYTHONTRACEMALLOC=0 environment variable and -X tracemalloc=0 command
line option are now allowed to disable explicitly tracemalloc at
startup.
2018-07-25 19:23:53 +02:00
Victor Stinner 1dc6e3906a
bpo-34170: Add _Py_InitializeFromConfig() (GH-8454)
* If _Py_InitializeCore() is called twice, the second call now copies
  and apply (partially) the new configuration.
* Rename _Py_CommandLineDetails to _PyCmdline
* Move more code into pymain_init(). The core configuration created
  by Py_Main() is new destroyed before running Python to reduce the
  memory footprint.
* _Py_InitializeCore() now returns the created interpreter.
  _Py_InitializeMainInterpreter() now expects an interpreter.
* Remove _Py_InitializeEx_Private(): _freeze_importlib now uses
  _Py_InitializeFromConfig()
* _PyCoreConfig_InitPathConfig() now only computes the path
  configuration if needed.
2018-07-25 02:49:17 +02:00
Victor Stinner 53b7d4e402
bpo-34170: Add _PyCoreConfig.bytes_warning (GH-8447)
Add more fields to _PyCoreConfig:

* _check_hash_pycs_mode
* bytes_warning
* debug
* inspect
* interactive
* legacy_windows_fs_encoding
* legacy_windows_stdio
* optimization_level
* quiet
* unbuffered_stdio
* user_site_directory
* verbose
* write_bytecode

Changes:

* Remove pymain_get_global_config() and pymain_set_global_config()
  which became useless. These functions have been replaced by
  _PyCoreConfig_GetGlobalConfig() and
  _PyCoreConfig_SetGlobalConfig().
* sys.flags.dont_write_bytecode value is now restricted to 1 even if
  -B option is specified multiple times on the command line.
* PyThreadState_Clear() now uses the config from the current
  interpreter rather than using global Py_VerboseFlag
2018-07-25 01:37:05 +02:00
Victor Stinner d19d8d5279
bpo-34170: Add _PyCoreConfig.isolated (GH-8417)
* _PyCoreConfig: add isolated and site_import attributes
* Replace Py_IgnoreEnvironment with config->ignore_environment when
  reading the current configuration
* _PyCoreConfig_Read() now sets ignore_environment, utf8_mode,
  isolated and site_import from Py_IgnoreEnvironment, Py_UTF8Mode,
  Py_IsolatedFlag and Py_NoSiteFlag
* _Py_InitializeCore() now sets Py_xxx flags from the configuration
* pymain_read_conf() now uses _PyCoreConfig_Copy() to save/restore
  the configuration.
* Rename _disable_importlib of _PyCoreConfig to _install_importlib
* _PyCoreConfig_SetGlobalConfig() now also set
  Py_HashRandomizationFlag
* Replace !Py_NoSiteFlag with core_config->site_import
2018-07-24 13:55:48 +02:00
Serhiy Storchaka c5734998d9
bpo-33720: Refactor marshalling/unmarshalling floats. (GH-8071) 2018-07-24 10:55:47 +03:00
Victor Stinner f2626ce6d4
bpo-34170: _PyCoreConfig_Read() leaves Py_IsolatedFlag unchanged (GH-8361)
* _PyCoreConfig_Read() no longer directly modifies Py_IsolatedFlag
  and Py_NoSiteFlag global configuration flags. The function now
  requires two pointers to integer, so these flags can be set later,
  to avoid side effets in _PyCoreConfig_Read().
* pathconfig_global_init() now leaves Py_IsolatedFlag and
  Py_NoSiteFlag unchanged.
* Fix pathconfig_global_init(): avoid computing the path
  configuration twice, use _PyCoreConfig_SetPathConfig().
2018-07-21 03:54:20 +02:00
Victor Stinner b1147e43da
bpo-34170: Rework _PyCoreConfig_Read() to avoid side effect (GH-8353)
Rework _PyCoreConfig_Read() function which *reads* core configuration
to not *modify* the path configuration.

A new _PyCoreConfig_SetPathConfig() function now recreates the path
configuration from the core configuration. This function is now
called very late in _Py_InitializeCore(), just before calling
initimport().

Changes:

* Add _PyCoreConfig.dll_path
* Py_SetPath() now fails with a fatal python error on memory
   allocation failure.
* Rename _PyPathConfig_Calculate() to _PyPathConfig_Calculate_impl()
* Replace _PyPathConfig_Init() with _PyPathConfig_Calculate(): the
  function now requires a _PyPathConfig
* Add _PyPathConfig_SetGlobal() to set the _Py_path_config global
  variable.
* Add _PyCoreConfig_InitPathConfig(): compute the path configuration
* Add _PyCoreConfig_SetPathConfig(): set path configuration from core
  configuration
* Rename wstrlist_append() to _Py_wstrlist_append()
* _Py_wstrlist_append() now handles integer overflow.
2018-07-21 02:06:16 +02:00
INADA Naoki 5ac9e6eee5
bpo-33597: Reduce PyGC_Head size (GH-7043) 2018-07-10 17:19:53 +09:00
Jeremy Cline f874bd1f06 Fix redundant declaration of _PyImport_AddModuleObject (GH-7992) 2018-06-29 16:47:18 -06:00
Carl Meyer b193fa996a bpo-33499: Add PYTHONPYCACHEPREFIX env var for alt bytecode cache location. (GH-6834)
In some development setups it is inconvenient or impossible to write bytecode
caches to the code tree, but the bytecode caches are still useful. The
PYTHONPYCACHEPREFIX environment variable allows specifying an alternate
location for cached bytecode files, within which a directory tree mirroring the code
tree will be created. This cache tree is then used (for both reading and writing)
instead of the local `__pycache__` subdirectory within each source directory.

Exposed at runtime as sys.pycache_prefix (defaulting to None), and can
be set from the CLI as "-X pycache_prefix=path".

Patch by Carl Meyer.
2018-06-16 14:40:56 +10:00
Serhiy Storchaka ceeef10cdb
bpo-33818: PyExceptionClass_Name() will now return "const char *". (GH-7581) 2018-06-15 11:09:43 +03:00
Serhiy Storchaka 5cbefa9919
Clean up after bpo-33738. (GH-7627)
* Add declarations even if they are overridden by macros.
* Make the declaration and the definition of PyExceptionClass_Name
  consistent.
2018-06-11 15:01:47 +03:00
Christian Tismer 8398713cea bpo-33738: Address review comments in GH #7477 (GH-7585) 2018-06-10 18:48:28 -04:00
Christian Tismer ea62ce7f4f bpo-33738: Fix macros which contradict PEP 384 (GH-7477)
During development of the limited API support for PySide,
we saw an error in a macro that accessed a type field.

This patch fixes the 7 errors in the Python headers.
Macros which were not written as capitals were implemented
as function.

To do the necessary analysis again, a script was included that
parses all headers and looks for "->tp_" in serctions which can
be reached with active limited API.

It is easily possible to call this script as a test.

Error listing:

../../Include/objimpl.h:243
#define PyObject_IS_GC(o) (PyType_IS_GC(Py_TYPE(o)) && \
    (Py_TYPE(o)->tp_is_gc == NULL || Py_TYPE(o)->tp_is_gc(o)))
Action: commented only

../../Include/objimpl.h:362
#define PyType_SUPPORTS_WEAKREFS(t) ((t)->tp_weaklistoffset > 0)
Action: commented only

../../Include/objimpl.h:364
#define PyObject_GET_WEAKREFS_LISTPTR(o) \
    ((PyObject **) (((char *) (o)) + Py_TYPE(o)->tp_weaklistoffset))
Action: commented only

../../Include/pyerrors.h:143
#define PyExceptionClass_Name(x) \
     ((char *)(((PyTypeObject*)(x))->tp_name))
Action: implemented function

../../Include/abstract.h:593
#define PyIter_Check(obj) \
    ((obj)->ob_type->tp_iternext != NULL && \
     (obj)->ob_type->tp_iternext != &_PyObject_NextNotImplemented)
Action: implemented function

../../Include/abstract.h:713
#define PyIndex_Check(obj)                              \
    ((obj)->ob_type->tp_as_number != NULL &&            \
     (obj)->ob_type->tp_as_number->nb_index != NULL)
Action: implemented function

../../Include/abstract.h:924
#define PySequence_ITEM(o, i)\
    ( Py_TYPE(o)->tp_as_sequence->sq_item(o, i) )
Action: commented only
2018-06-09 14:32:25 -04:00
Eric Snow 63799136e6
bpo-33615: Re-enable a subinterpreter test. (gh-7251)
For bpo-32604 I added extra subinterpreter-related tests (see #6914), which caused a few buildbots to crash. This patch fixes the crash by ensuring that refcounts in channels are handled properly.
2018-06-01 18:45:20 -06:00
Serhiy Storchaka 143ce5c6db
bpo-33691: Add _PyAST_GetDocString(). (GH-7236) 2018-05-30 10:56:16 +03:00
Serhiy Storchaka 73cbe7a01a
bpo-32911: Revert bpo-29463. (GH-7121) (GH-7197)
Remove the docstring attribute of AST types and restore docstring
expression as a first stmt in their body.

Co-authored-by: INADA Naoki <methane@users.noreply.github.com>
2018-05-29 12:04:55 +03:00
Serhiy Storchaka 4e29f566e8
Add missed details of the C API introduced in 3.7. (GH-7047)
* Set the limited API version for PyImport_GetModule and PyOS_*Fork
  functions.
* Add PyImport_GetModule and Py_UTF8Mode in PC/python3.def.
* Add several functions in Doc/data/refcounts.dat.
2018-05-22 20:59:42 +03:00
Serhiy Storchaka f5b1183610
bpo-5945: Improve mappings and sequences C API docs. (GH-7029) 2018-05-22 11:02:44 +03:00
Serhiy Storchaka 64fddc423f bpo-33475: Fix and improve converting annotations to strings. (GH-6774) 2018-05-16 23:17:48 -04:00
Serhiy Storchaka b00854caa0
bpo-20171: Convert the _curses and _curses_panel modules to Argument Clinic. (GH-4251) 2018-05-10 11:27:23 +03:00
Ville Skyttä 61f82e0e33 Spelling fixes to docs, docstrings, and comments (GH-6374) 2018-04-20 16:08:45 -04:00
Serhiy Storchaka 702f8f3611
bpo-33041: Rework compiling an "async for" loop. (#6142)
* Added new opcode END_ASYNC_FOR.
* Setting global StopAsyncIteration no longer breaks "async for" loops.
* Jumping into an "async for" loop is now disabled.
* Jumping out of an "async for" loop no longer corrupts the stack.
* Simplify the compiler.
2018-03-23 14:34:35 +02:00
Thomas Nyberg 7023644e0c closes bpo-32980 Remove _PyFrame_Init (GH-5965) 2018-03-03 22:06:01 -08:00
Nitish Chandra 3a087beddd bpo-32836: Remove obsolete code from symtable pass (GH-5680)
When comprehensions switched to using a nested scope, the old
code for generating a temporary name to hold the accumulation
target became redundant, but was never actually removed.

Patch by Nitish Chandra.
2018-02-27 07:31:20 +10:00
Serhiy Storchaka 520b7ae27e
bpo-17611. Move unwinding of stack for "pseudo exceptions" from interpreter to compiler. (GH-5006)
Co-authored-by: Mark Shannon <mark@hotpy.org>
Co-authored-by: Antoine Pitrou <antoine@python.org>
2018-02-22 23:33:30 +02:00
Eric Snow 4c6955e2b0
bpo-32604: Clean up created subinterpreters before runtime finalization. (gh-5709) 2018-02-16 18:53:40 -07:00
Alexey 517da1e58f Fix typo in Include/objimpl.h, the word "has" was missing (GH-5568)
It now reads: ...be aware that Python has no control over...
2018-02-06 16:07:30 -07:00
Alexey Izbyshev c1e46e94de bpo-32777: Fix _Py_set_inheritable async-safety in subprocess (GH-5560)
Fix a rare but potential pre-exec child process deadlock in subprocess on POSIX systems when marking file descriptors inheritable on exec in the child process.  This bug appears to have been introduced in 3.4 with the inheritable file descriptors support.

This also changes Python/fileutils.c `set_inheritable` to use the "slow" two `fcntl` syscall path instead of the "fast" single `ioctl` syscall path when asked to be async signal safe (by way of being asked not to raise exceptions).  `ioctl` is not a POSIX async-signal-safe approved function.

ref: http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html
2018-02-05 22:09:34 -08:00
Yury Selivanov 383b32fe10
Revert "bpo-31356: Add context manager to temporarily disable GC GH-5495
This reverts commit 72a0d218dc.

The reverted commit had a few issues so it was unanimously decided
to undo it. See the bpo issue for details.
2018-02-02 09:31:06 -05:00
Ned Deily 5489bdad51 Start of 3.8.0a0 2018-01-31 17:44:09 -05:00
Ned Deily 6e41cd95b5 Update NEWS, docs, and patchlevel for 3.7.0b1 2018-01-31 17:14:30 -05:00
Eric Snow 7f8bfc9b9a
bpo-32604: Expose the subinterpreters C-API in a "private" stdlib module. (gh-1748)
The module is primarily intended for internal use in the test suite.  Building the module under Windows will come in a follow-up PR.
2018-01-29 18:23:44 -07:00
Mark Shannon 332cd5ee4f bpo-32550. Remove the STORE_ANNOTATION bytecode. (GH-5181) 2018-01-29 16:41:04 -08:00
Pablo Galindo 72a0d218dc bpo-31356: Add context manager to temporarily disable GC (GH-4224) 2018-01-29 12:37:09 -08:00
Benjamin Peterson c65ef772c3
rename _imp initialization function to follow conventions (#5432)
When the C imp module became _imp in 6f44d66bc4, the initialization function should have been renamed from PyInit_imp to PyInit__imp.
2018-01-29 11:33:57 -08:00
INADA Naoki a49ac99029
bpo-32677: Add .isascii() to str, bytes and bytearray (GH-5342) 2018-01-27 14:06:21 +09:00
Guido van Rossum 95e4d58913 String annotations [PEP 563] (#4390)
* Document `from __future__ import annotations`
* Provide plumbing and tests for `from __future__ import annotations`
* Implement unparsing the AST back to string form

This is required for PEP 563 and as such only implements a part of the
unparsing process that covers expressions.
2018-01-26 08:20:18 -08:00
Serhiy Storchaka f320be77ff bpo-32571: Avoid raising unneeded AttributeError and silencing it in C code (GH-5222)
Add two new private APIs: _PyObject_LookupAttr() and _PyObject_LookupAttrId()
2018-01-25 17:49:40 +09:00
Paul Ganssle 04af5b1ba9 bpo-10381: Add timezone to datetime C API (#5032)
* Add timezone to datetime C API

* Add documentation for timezone C API macros

* Add dedicated tests for datetime type check macros

* Remove superfluous C API test

* Drop support for TimeZoneType in datetime C API

* Expose UTC singleton to the datetime C API

* Update datetime C-API documentation to include links

* Add reference count information for timezone constructors
2018-01-24 17:29:30 -05:00
Victor Stinner 8ded5b8037
bpo-32030: Add _PyCoreConfig.module_search_paths (#4954)
_PyCoreConfig_Read() is now responsible to compute sys.path.
So sys.path is now computed before calling _Py_InitializeCore().

Changes:

* Add module_search_path, module_search_paths, executable, prefix,
  base_prefix, exec_prefix and base_exec_prefix to _PyCoreConfig.
* _PyMainInterpreterConfig_Read() now only converts wchar_t** lists
  into a Python list, it doesn't compute sys.path anymore.
2018-01-24 17:03:28 +01:00
stratakis e768c86ef4 bpo-32635: Fix a segfault when importing the crypt module with libxcrypt. (#5284)
glibc is deprecating libcrypt in favor of libxcrypt, however python assumes
that crypt.h will always be included. This change makes the header inclusion
explicit when libxcrypt is present on the system.
2018-01-23 16:11:24 +01:00
Yury Selivanov f23746a934
bpo-32436: Implement PEP 567 (#5027) 2018-01-22 19:11:18 -05:00
Victor Stinner 13ff24582c
bpo-32593: Drop FreeBSD 9 and older support (#5232)
Drop support of FreeBSD 9 and older.
2018-01-22 18:32:50 +01:00
Nathaniel J. Smith fc2f407829 bpo-32591: Add native coroutine origin tracking (#5250)
* Add coro.cr_origin and sys.set_coroutine_origin_tracking_depth
* Use coroutine origin information in the unawaited coroutine warning
* Stop using set_coroutine_wrapper in asyncio debug mode
* In BaseEventLoop.set_debug, enable debugging in the correct thread
2018-01-21 09:44:07 -05:00
INADA Naoki 378edee0a3
bpo-32544: Speed up hasattr() and getattr() (GH-5173)
AttributeError was raised always when attribute is not found.
This commit skip raising AttributeError when `tp_getattro` is `PyObject_GenericGetAttr`.
It makes hasattr() and getattr() about 4x faster when attribute is not found.
2018-01-16 20:52:41 +09:00
Victor Stinner cb064fc232
bpo-31900: Fix localeconv() encoding for LC_NUMERIC (#4174)
* Add _Py_GetLocaleconvNumeric() function: decode decimal_point and
  thousands_sep fields of localeconv() from the LC_NUMERIC encoding,
  rather than decoding from the LC_CTYPE encoding.
* Modify locale.localeconv() and "n" formatter of str.format() (for
  int, float and complex to use _Py_GetLocaleconvNumeric()
  internally.
2018-01-15 15:58:02 +01:00
Victor Stinner 7ed7aead95
bpo-29240: Fix locale encodings in UTF-8 Mode (#5170)
Modify locale.localeconv(), time.tzname, os.strerror() and other
functions to ignore the UTF-8 Mode: always use the current locale
encoding.

Changes:

* Add _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx(). On decoding or
  encoding error, they return the position of the error and an error
  message which are used to raise Unicode errors in
  PyUnicode_DecodeLocale() and PyUnicode_EncodeLocale().
* Replace _Py_DecodeCurrentLocale() with _Py_DecodeLocaleEx().
* PyUnicode_DecodeLocale() now uses _Py_DecodeLocaleEx() for all
  cases, especially for the strict error handler.
* Add _Py_DecodeUTF8Ex(): return more information on decoding error
  and supports the strict error handler.
* Rename _Py_EncodeUTF8_surrogateescape() to _Py_EncodeUTF8Ex().
* Replace _Py_EncodeCurrentLocale() with _Py_EncodeLocaleEx().
* Ignore the UTF-8 mode to encode/decode localeconv(), strerror()
  and time zone name.
* Remove PyUnicode_DecodeLocale(), PyUnicode_DecodeLocaleAndSize()
  and PyUnicode_EncodeLocale() now ignore the UTF-8 mode: always use
  the "current" locale.
* Remove _PyUnicode_DecodeCurrentLocale(),
  _PyUnicode_DecodeCurrentLocaleAndSize() and
  _PyUnicode_EncodeCurrentLocale().
2018-01-15 10:45:49 +01:00
Victor Stinner cb3ae5588b
bpo-29240: Ignore UTF-8 Mode in time module (#5148)
time.strftime() must use the current LC_CTYPE encoding, not UTF-8
if the UTF-8 mode is enabled.

Add _PyUnicode_DecodeCurrentLocale() function.
2018-01-11 10:37:59 +01:00
Victor Stinner 2cba6b8579
bpo-29240: readline now ignores the UTF-8 Mode (#5145)
Add new fuctions ignoring the UTF-8 mode:

* _Py_DecodeCurrentLocale()
* _Py_EncodeCurrentLocale()
* _PyUnicode_DecodeCurrentLocaleAndSize()
* _PyUnicode_EncodeCurrentLocale()

Modify the readline module to use these functions.

Re-enable test_readline.test_nonascii().
2018-01-10 22:46:15 +01:00
Ned Deily 2ab3a8fd26 Post-release version bump to a4+ 2018-01-09 10:27:57 -05:00
Ned Deily 07c9d8547c Bump to 3.7.0a4 2018-01-08 22:05:14 -05:00
Nathaniel J. Smith 735ae8d139 bpo-29137: Remove fpectl module (#4789)
This module has never been enabled by default, never worked correctly
on x86-64, and caused ABI problems that caused C extension
compatibility. See bpo-29137 for details/discussion.
2018-01-05 23:15:34 -08:00
Serhiy Storchaka 3dfbaf51f0
bpo-32372: Move __debug__ optimization to the AST level. (#4925) 2017-12-25 12:47:50 +02:00
Serhiy Storchaka 1b3029ac83
bpo-29084: Exclude C API for OrderedDict from the limited C API. (#4900) 2017-12-25 02:08:42 +02:00
Victor Stinner 9bee329130
bpo-32030: Add _Py_FindEnvConfigValue() (#4963)
Add a new _Py_FindEnvConfigValue() function: code shared between
Windows and Unix implementations of _PyPathConfig_Calculate() to read
the pyenv.cfg file.

_Py_FindEnvConfigValue() now uses _Py_DecodeUTF8_surrogateescape()
instead of using a Python Unicode string, the Python API must not be
used early during Python initialization. Same change in Unix
search_for_exec_prefix(): use _Py_DecodeUTF8_surrogateescape().

Cleanup also encode_current_locale(): PyMem_RawFree/PyMem_Free can be
called with NULL.

Fix also "NUL byte" => "NULL byte" typo.
2017-12-21 16:49:13 +01:00
Victor Stinner 9dd762013f
bpo-32030: Add _Py_EncodeLocaleRaw() (#4961)
Replace Py_EncodeLocale() with _Py_EncodeLocaleRaw() in:

* _Py_wfopen()
* _Py_wreadlink()
* _Py_wrealpath()
* _Py_wstat()
* pymain_open_filename()

These functions are called early during Python intialization, only
the RAW memory allocator must be used.
2017-12-21 16:20:32 +01:00
Victor Stinner 9cfc00262c
bpo-32030: Complete _PyCoreConfig_Read() (#4946)
* Add _PyCoreConfig.install_signal_handlers
* Remove _PyMain.config: _PyMainInterpreterConfig usage is now
  restricted to pymain_init_python_main().
* Rename _PyMain.core_config to _PyMain.config
* _PyMainInterpreterConfig_Read() now creates the xoptions dictionary
   from the core config
* Fix _PyMainInterpreterConfig_Read(): don't replace xoptions and
  argv if they are already set.
2017-12-20 19:36:46 +01:00
Victor Stinner ca719ac42b
bpo-32030: Add _PyCoreConfig.warnoptions (#4936)
Merge _PyCoreConfig_ReadEnv() into _PyCoreConfig_Read(), and
_Py_CommandLineDetails usage is now restricted to pymain_cmdline().

Changes:

* _PyCoreConfig: Add nxoption, xoptions, nwarnoption and warnoptions
* Add _PyCoreConfig.program: argv[0] or ""
* Move filename, command, module and xoptions from
  _Py_CommandLineDetails to _PyMain. xoptions _Py_OptList becomes
  (int, wchar_t**) list.
* Add pymain_cmdline() function
* Rename copy_argv() to copy_wstrlist(). Rename clear_argv() to
  clear_wstrlist(). Remove _Py_OptList structure: use (int,
  wchar_t**) list instead.
* Rename pymain_set_flag_from_env() to pymain_get_env_flag()
* Rename pymain_set_flags_from_env() to pymain_get_env_flags()
* _PyMainInterpreterConfig_Read() now creates the warnoptions from
  _PyCoreConfig.warnoptions
* Inline pymain_add_warning_dev_mode() and
  pymain_add_warning_bytes_flag() into config_init_warnoptions()
* Inline pymain_get_program_name() into _PyCoreConfig_Read()
* _Py_CommandLineDetails: Replace warning_options with nwarnoption
  and warnoptions. Replace env_warning_options with nenv_warnoption
  and env_warnoptions.
* pymain_warnings_envvar() now has a single implementation for
  Windows and Unix: use config_get_env_var_dup() to also get the
  variable as wchar_t* on Unix.
2017-12-20 18:00:19 +01:00
Marcel Plch 776407fe89 bpo-31901: atexit callbacks should be run at subinterpreter shutdown (#4611)
Change atexit behavior and PEP-489 multiphase init support.
2017-12-20 11:17:58 +01:00
Victor Stinner c4bca95106
bpo-32030: Add _PyCoreConfig.argv (#4934)
* Add argc and argv to _PyCoreConfig
* _PyMainInterpreterConfig_Read() now builds its argv from
  _PyCoreConfig.arg
* Move _PyMain.env_warning_options into _Py_CommandLineDetails
* Reorder pymain_free()
2017-12-19 23:48:17 +01:00
Victor Stinner 5d8624647d
bpo-32030: Fix compiler warnings (#4921)
Fix compiler warnings in Py_FinalizeEx(): only define variables if
they are needed, add #ifdef.

Other cleanup changes:

* _PyWarnings_InitWithConfig() is no more needed: call
  _PyWarnings_Init() instead.
* Inline pymain_init_main_interpreter() in its caller. This
  subfunction is no more justifed.
2017-12-19 11:35:58 +01:00
Benjamin Peterson 4c72bc4a38
add 'extern' to pygetopt.h symbols, so then don't end up in comdat (#4909)
bpo-32264
2017-12-17 10:34:00 -08:00
Andrew Svetlov 950840261c
Fix PyObject_Hash signature in comment (#4905) 2017-12-16 21:08:05 +02:00
Victor Stinner 9454060e84
bpo-29240, bpo-32030: Py_Main() re-reads config if encoding changes (#4899)
bpo-29240, bpo-32030: If the encoding change (C locale coerced or
UTF-8 Mode changed), Py_Main() now reads again the configuration with
the new encoding.

Changes:

* Add _Py_UnixMain() called by main().
* Rename pymain_free_pymain() to pymain_clear_pymain(), it can now be
  called multipled times.
* Rename pymain_parse_cmdline_envvars() to pymain_read_conf().
* Py_Main() now clears orig_argc and orig_argv at exit.
* Remove argv_copy2, Py_Main() doesn't modify argv anymore. There is
  no need anymore to get two copies of the wchar_t** argv.
* _PyCoreConfig: add coerce_c_locale and coerce_c_locale_warn.
* Py_UTF8Mode is now initialized to -1.
* Locale coercion (PEP 538) now respects -I and -E options.
2017-12-16 04:54:22 +01:00
Serhiy Storchaka a5552f023e
bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
Benjamin Peterson e425bd7517
move pygetopt.h to internal (closes bpo-32264) (#4830) 2017-12-14 23:48:12 -08:00
Victor Stinner 41264f1cd4
bpo-32030: Add _PyMainInterpreterConfig.executable (#4876)
* Add new fields to _PyMainInterpreterConfig:

  * executable
  * prefix
  * base_prefix
  * exec_prefix
  * base_exec_prefix

* _PySys_EndInit() now sets sys attributes from
  _PyMainInterpreterConfig
2017-12-15 02:05:29 +01:00
Victor Stinner da273412c4
bpo-32030: Add _PyCoreConfig_Copy() (#4874)
Each interpreter now has its core_config and main_config copy:

* Add _PyCoreConfig_Copy() and _PyMainInterpreterConfig_Copy()
* Move _PyCoreConfig_Read(), _PyCoreConfig_Clear() and
  _PyMainInterpreterConfig_Clear() from Python/pylifecycle.c to
  Modules/main.c
* Fix _Py_InitializeEx_Private(): call _PyCoreConfig_ReadEnv() before
  _Py_InitializeCore()
2017-12-15 01:46:02 +01:00
Victor Stinner 358e5e17a5
bpo-32329: Fix -R option for hash randomization (#4873)
bpo-32329, bpo-32030:

* The -R option now turns on hash randomization when the
  PYTHONHASHSEED environment variable is set to 0 Previously, the
  option was ignored.
* sys.flags.hash_randomization is now properly set to 0 when hash
  randomization is turned off by PYTHONHASHSEED=0.
* _PyCoreConfig_ReadEnv() now reads the PYTHONHASHSEED environment
  variable. _Py_HashRandomization_Init() now only apply the
  configuration, it doesn't read PYTHONHASHSEED anymore.
2017-12-15 00:51:22 +01:00
Victor Stinner 374c6e178a
bpo-32030: Add _PyMainInterpreterConfig.warnoptions (#4855)
Add warnoptions and xoptions fields to _PyMainInterpreterConfig.
2017-12-14 12:05:26 +01:00
INADA Naoki 7ea143ae79
bpo-29469: Move constant folding to AST optimizer (GH-2858) 2017-12-14 16:47:20 +09:00
Victor Stinner b5fd9ad05e
bpo-32030: Rewrite _PyMainInterpreterConfig (#4854)
_PyMainInterpreterConfig now contains Python objects, whereas
_PyCoreConfig contains wchar_t* strings.

Core config:

* Rename _PyMainInterpreterConfig_ReadEnv() to _PyCoreConfig_ReadEnv()
* Move 3 strings from _PyMainInterpreterConfig to _PyCoreConfig:
  module_search_path_env, home, program_name.
* Add _PyCoreConfig_Clear()
* _PyPathConfig_Calculate() now takes core config rather than main
  config
* _PyMainInterpreterConfig_Read() now requires also a core config

Main config:

* Add _PyMainInterpreterConfig.module_search_path: sys.path list
* Add _PyMainInterpreterConfig.argv: sys.argv list
* _PyMainInterpreterConfig_Read() now computes module_search_path
2017-12-14 02:20:52 +01:00
Victor Stinner 11a247df88
bpo-32030: Add _PyPathConfig_ComputeArgv0() (#4845)
Changes:

* Split _PySys_SetArgvWithError() into subfunctions for Py_Main():

  * Create the Python list object
  * Set sys.argv to the list
  * Compute argv0
  * Prepend argv0 to sys.path

* Add _PyPathConfig_ComputeArgv0()
* Remove _PySys_SetArgvWithError()
* Py_Main() now splits the code to compute sys.argv/path0 and the
  code to update the sys module: add pymain_compute_argv()
  subfunction.
2017-12-13 21:05:57 +01:00
Victor Stinner d5dda98fa8
pymain_set_sys_argv() now copies argv (#4838)
bpo-29240, bpo-32030:

* Rename pymain_set_argv() to pymain_set_sys_argv()
* pymain_set_sys_argv() now creates of copy of argv and modify the
  copy, rather than modifying pymain->argv
* Call pymain_set_sys_argv() earlier: before pymain_run_python(), but
  after pymain_get_importer().
* Add _PySys_SetArgvWithError() to handle errors
2017-12-13 17:31:16 +01:00
Victor Stinner 91106cd9ff
bpo-29240: PEP 540: Add a new UTF-8 Mode (#855)
* Add -X utf8 command line option, PYTHONUTF8 environment variable
  and a new sys.flags.utf8_mode flag.
* If the LC_CTYPE locale is "C" at startup: enable automatically the
  UTF-8 mode.
* Add _winapi.GetACP(). encodings._alias_mbcs() now calls
  _winapi.GetACP() to get the ANSI code page
* locale.getpreferredencoding() now returns 'UTF-8' in the UTF-8
  mode. As a side effect, open() now uses the UTF-8 encoding by
  default in this mode.
* Py_DecodeLocale() and Py_EncodeLocale() now use the UTF-8 encoding
  in the UTF-8 Mode.
* Update subprocess._args_from_interpreter_flags() to handle -X utf8
* Skip some tests relying on the current locale if the UTF-8 mode is
  enabled.
* Add test_utf8mode.py.
* _Py_DecodeUTF8_surrogateescape() gets a new optional parameter to
  return also the length (number of wide characters).
* pymain_get_global_config() and pymain_set_global_config() now
  always copy flag values, rather than only copying if the new value
  is greater than the old value.
2017-12-13 12:29:09 +01:00
Serhiy Storchaka 4ae06c5337
bpo-32241: Add the const qualifire to declarations of umodifiable strings. (#4748) 2017-12-12 13:55:04 +02:00
Chris e0720cd9e4 Fix small typo in tupleobject.h (#4801) 2017-12-12 09:59:30 +02:00
Benjamin Peterson 42aa93b8ff
closes bpo-31650: PEP 552 (Deterministic pycs) implementation (#4575)
Python now supports checking bytecode cache up-to-dateness with a hash of the
source contents rather than volatile source metadata. See the PEP for details.

While a fairly straightforward idea, quite a lot of code had to be modified due
to the pervasiveness of pyc implementation details in the codebase. Changes in
this commit include:

- The core changes to importlib to understand how to read, validate, and
  regenerate hash-based pycs.

- Support for generating hash-based pycs in py_compile and compileall.

- Modifications to our siphash implementation to support passing a custom
  key. We then expose it to importlib through _imp.

- Updates to all places in the interpreter, standard library, and tests that
  manually generate or parse pyc files to grok the new format.

- Support in the interpreter command line code for long options like
  --check-hash-based-pycs.

- Tests and documentation for all of the above.
2017-12-09 10:26:52 -08:00
Victor Stinner 92a3c6f493
bpo-32030: Add _PyImport_Fini2() (#4737)
PyImport_ExtendInittab() now uses PyMem_RawRealloc() rather than
PyMem_Realloc(). PyImport_ExtendInittab() can be called before
Py_Initialize() whereas only the PyMem_Raw allocator is supposed to
be used before Py_Initialize().

Add _PyImport_Fini2() to release the memory allocated by
PyImport_ExtendInittab() at exit. PyImport_ExtendInittab() now forces
the usage of the default raw allocator, to be able to release memory
in _PyImport_Fini2().

Don't export these functions anymore to be C API, only to
Py_BUILD_CORE:

* _PyExc_Fini()
* _PyImport_Fini()
* _PyGC_DumpShutdownStats()
* _PyGC_Fini()
* _PyType_Fini()
* _Py_HashRandomization_Fini()
2017-12-06 18:12:59 +01:00
Victor Stinner 6bf992a1ac
bpo-32030: Add pymain_get_global_config() (#4735)
* Py_Main() now starts by reading Py_xxx configuration variables to
  only work on its own private structure, and then later writes back
  the configuration into these variables.
* Replace Py_GETENV() with pymain_get_env_var() which ignores empty
  variables.
* Add _PyCoreConfig.dump_refs
* Add _PyCoreConfig.malloc_stats
* _PyObject_DebugMallocStats() is now responsible to check if debug
  hooks are installed. The function returns 1 if stats were written,
  or 0 if the hooks are disabled. Mark _PyMem_PymallocEnabled() as
  static.
2017-12-06 17:26:10 +01:00
Victor Stinner 672b6baa71
bpo-32030: pass interp to _PyImport_Init() (#4736)
Remove also the initstr variable, unused since the commit
e69f0df45b pushed in 2012: "bpo-13959:
Re-implement imp.find_module() in Lib/imp.py"
2017-12-06 17:25:50 +01:00
Ned Deily a2a25eb037 Bump to 3.7.0a3+ 2017-12-05 21:06:12 -05:00
Ned Deily d86f29e9c9 Python 3.7.0a3
-----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCgAdFiEEDZbfTUEQ5cQ/v7F/LTR+pqplQh0FAlomWNwACgkQLTR+pqpl
 Qh34Aw/+LDgBou5bRTgHXRAyx4vU8gDT97K0Fm604FXTcQLtTMo/3LDVa+7NbROg
 EWIaSxOjObRwZxjTVU1P063QKBDpjzeqK0PE+DYtEfpKVcqm8tADG30jHx9FYnXp
 yalV6/NzzHhcdxdhwrcbbAYnykk7xcUqEAimJ6YLgQYqe5Z1cemprjuPIcDWAagV
 Jw9ARZI9somzoqDRNAT1s72rJ7V/nKFGOcZ0PL+pv65o7essht+v7ReSxLsbyJYC
 ZAAxTiJT6fpyGc6/ZcKJ6xnbxINvlkdfqwELFMM7/+6l5Xwg07pZSEzOdh1Ng1sz
 woBIJkAoCTV4xAiKZ4DW6vxEeRFDnJgSO7Vd2zFKpSa9iLorZBEpNSObP/1gLTUC
 0fah7ddY2ZIHsi6vdAKnD877FoYFlNsSnlYxHxE1dOOB/5LbHd25/4gTjOHvav8n
 Xrd0qZ4SVVNSdqvDIhsf3GHnao9N5wKkcs9YBX/BK8ryRdIyUof3a2vXoOJtsNNt
 poyIhmrnqjRYttDVHefPk+vi/rS3l3kLfj71AKdmfO5Fv7xFXbzcJBU3ax5SBvbp
 w1XoMoTOSJDZt/GdMvC2htsG5PZWw8Bg8RGqBrLyUFEN0rnoIh5ykEH6YuPSusDQ
 fFUuaE1pGETNhrE50YWYYb4MGOE4xakdRV6cT402DhrzexFwuCA=
 =CREY
 -----END PGP SIGNATURE-----

Merge tag 'v3.7.0a3'
2017-12-05 20:57:44 -05:00
Victor Stinner 33c377ed9b
bpo-32030: Simplify _PyCoreConfig_INIT macro (#4728)
* Simplify _PyCoreConfig_INIT, _PyMainInterpreterConfig_INIT,
  _PyPathConfig_INIT macros: no need to set fields to 0/NULL, it's
  redundant (the C language sets them to 0/NULL for us).
* Fix typo: pymain_run_statup() => pymain_run_startup()
* Remove a few XXX/TODO
2017-12-05 15:12:41 +01:00
Ned Deily 90a6785f63 Bump to 3.7.0a3 2017-12-05 03:27:19 -05:00
Victor Stinner 31a8393cf6
Revert "bpo-32197: Try to fix a compiler error on OS X introduced in bpo-32030. (#4681)" (#4694)
* Revert "bpo-32197: Try to fix a compiler error on OS X introduced in bpo-32030. (#4681)"

This reverts commit 13badcbc60.

Re-apply commits:

* "bpo-32030: _PyPathConfig_Init() sets home and program_name (#4673)"
  commit af5a895073.
* "bpo-32030: Fix config_get_program_name() on macOS (#4669)"
  commit e23c06e2b0.
* "bpo-32030: Add Python/pathconfig.c (#4668)"
  commit 0ea395ae96.
* "bpo-32030: Don't call _PyPathConfig_Fini() in Py_FinalizeEx() (#4667)"
  commit ebac19dad6.
* "bpo-32030: Fix Py_GetPath(): init program_name (#4665)"
  commit 9ac3d88827.

* Fix compilation error on macOS
2017-12-04 13:39:15 +01:00
Serhiy Storchaka 13badcbc60
bpo-32197: Try to fix a compiler error on OS X introduced in bpo-32030. (#4681)
* Revert "bpo-32030: _PyPathConfig_Init() sets home and program_name (#4673)"

This reverts commit af5a895073.

* Revert "bpo-32030: Fix config_get_program_name() on macOS (#4669)"

This reverts commit e23c06e2b0.

* Revert "bpo-32030: Add Python/pathconfig.c (#4668)"

This reverts commit 0ea395ae96.

* Revert "bpo-32030: Don't call _PyPathConfig_Fini() in Py_FinalizeEx() (#4667)"

This reverts commit ebac19dad6.

* Revert "bpo-32030: Fix Py_GetPath(): init program_name (#4665)"

This reverts commit 9ac3d88827.
2017-12-02 21:36:00 +02:00
Victor Stinner af5a895073
bpo-32030: _PyPathConfig_Init() sets home and program_name (#4673)
_PyPathConfig_Init() now also initialize home and program_name:

* Rename existing _PyPathConfig_Init() to _PyPathConfig_Calculate().
  Add a new _PyPathConfig_Init() function in pathconfig.c which
  handles the _Py_path_config variable and call
  _PyPathConfig_Calculate().
* Add home and program_name fields to _PyPathConfig.home
* _PyPathConfig_Init() now initialize home and program_name
  from main_config
* Py_SetProgramName(), Py_SetPythonHome() and Py_GetPythonHome() now
  calls Py_FatalError() on failure, instead of silently ignoring
  failures.
* config_init_home() now gets directly _Py_path_config.home to only
  get the value set by Py_SetPythonHome(), or NULL if
  Py_SetPythonHome() was not called.
* config_get_program_name() now gets directly
  _Py_path_config.program_name to only get the value set by
  Py_SetProgramName(), or NULL if Py_SetProgramName() was not called.
* pymain_init_python() doesn't call Py_SetProgramName() anymore,
  _PyPathConfig_Init() now always sets the program name
* Call _PyMainInterpreterConfig_Read() in
  pymain_parse_cmdline_envvars_impl() to control the memory allocator
* C API documentation: it's no more safe to call Py_GetProgramName()
  before Py_Initialize().
2017-12-02 10:11:32 +01:00
Victor Stinner 0ea395ae96
bpo-32030: Add Python/pathconfig.c (#4668)
* Factorize code from PC/getpathp.c and Modules/getpath.c to remove
  duplicated code
* rename pathconfig_clear() to _PyPathConfig_Clear()
* Inline _PyPathConfig_Fini() in pymain_impl() and then remove it,
  since it's a oneliner
2017-12-01 20:50:58 +01:00
Victor Stinner ebac19dad6
bpo-32030: Don't call _PyPathConfig_Fini() in Py_FinalizeEx() (#4667)
Changes:

* _PyPathConfig_Fini() cannot be called in Py_FinalizeEx().
  Py_Initialize() and Py_Finalize() can be called multiple times, but
  it must not "forget" parameters set by Py_SetProgramName(),
  Py_SetPath() or Py_SetPythonHome(), whereas _PyPathConfig_Fini()
  clear all these parameters.
* config_get_program_name() and calculate_program_full_path() now
  also decode paths using Py_DecodeLocale() to use the
  surrogateescape error handler, rather than decoding using
  mbstowcs() which is strict.
* Change _Py_CheckPython3() prototype: () => (void)
* Truncate a few lines which were too long
2017-12-01 20:09:52 +01:00
Victor Stinner b64de46aae
bpo-32030: Cleanup "path config" code (#4663)
* Rename PyPathConfig structure to _PyPathConfig and move it to
  Include/internal/pystate.h
* Rename path_config to _Py_path_config
* _PyPathConfig: Rename program_name field to program_full_path
* Add assert(str != NULL); to _PyMem_RawWcsdup(), _PyMem_RawStrdup()
  and _PyMem_Strdup().
* Rename calculate_path() to pathconfig_global_init(). The function
  now does nothing if it's already initiallized.
2017-12-01 18:27:09 +01:00
Victor Stinner 5d39e04290
bpo-32030: Rework memory allocators (#4625)
* Fix _PyMem_SetupAllocators("debug"): always restore allocators to
  the defaults, rather than only caling _PyMem_SetupDebugHooks().
* Add _PyMem_SetDefaultAllocator() helper to set the "default"
  allocator.
* Add _PyMem_GetAllocatorsName(): get the name of the allocators
* main() now uses debug hooks on memory allocators if Py_DEBUG is
  defined, rather than calling directly malloc()
* Document default memory allocators in C API documentation
* _Py_InitializeCore() now fails with a fatal user error if
  PYTHONMALLOC value is an unknown memory allocator, instead of
  failing with a fatal internal error.
* Add new tests on the PYTHONMALLOC environment variable
* Add support.with_pymalloc()
* Add the _testcapi.WITH_PYMALLOC constant and expose it as
   support.with_pymalloc().
* sysconfig.get_config_var('WITH_PYMALLOC') doesn't work on Windows, so
   replace it with support.with_pymalloc().
* pythoninfo: add _testcapi collector for pymem
2017-11-29 17:20:38 +01:00
Serhiy Storchaka 598ceae876
bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
Victor Stinner 9316ee4da2
bpo-32030: Add _PyPathConfig_Init() (#4551)
* Add _PyPathConfig_Init() and _PyPathConfig_Fini()
* Remove _Py_GetPathWithConfig()
* _PyPathConfig_Init() returns _PyInitError to allow to handle errors
  properly
* Add pathconfig_clear()
* Windows calculate_path_impl(): replace Py_FatalError() with
  _PyInitError
* Py_FinalizeEx() now calls _PyPathConfig_Fini() to release memory
* Fix _Py_InitializeMainInterpreter() regression: don't initialize
  path config if _disable_importlib is false
* PyPathConfig now uses dynamically allocated memory
2017-11-25 03:17:57 +01:00
Victor Stinner f04ebe2a4d
bpo-32030: Add _PyMainInterpreterConfig.program_name (#4548)
* Py_Main() now calls Py_SetProgramName() earlier to be able to get
  the program name in _PyMainInterpreterConfig_ReadEnv().
* Rename prog to program_name
* Rename progpath to program_name
2017-11-25 00:01:23 +01:00
Victor Stinner 46972b7bc3
bpo-32030: Add _PyMainInterpreterConfig_ReadEnv() (#4542)
Py_GetPath() and Py_Main() now call
_PyMainInterpreterConfig_ReadEnv() to share the same code to get
environment variables.

Changes:

* Add _PyMainInterpreterConfig_ReadEnv()
* Add _PyMainInterpreterConfig_Clear()
* Add _PyMem_RawWcsdup()
* _PyMainInterpreterConfig: rename pythonhome to home
* Rename _Py_ReadMainInterpreterConfig() to
  _PyMainInterpreterConfig_Read()
* Use _Py_INIT_USER_ERR(), instead of _Py_INIT_ERR(), for decoding
  errors: the user is able to fix the issue, it's not a bug in
  Python. Same change was made in _Py_INIT_NO_MEMORY().
* Remove _Py_GetPythonHomeWithConfig()
2017-11-24 22:55:40 +01:00
Victor Stinner da9c8c36ae
bpo-32125: Remove Py_UseClassExceptionsFlag flag (#4544)
This flag was deprecated and wasn't used anymore since Python 2.0.
2017-11-24 22:06:38 +01:00
Victor Stinner 9e87e7776f
bpo-32096: Remove obj and mem from _PyRuntime (#4532)
bpo-32096, bpo-30860:  Partially revert the commit
2ebc5ce42a8a9e047e790aefbf9a94811569b2b6:

* Move structures back from Include/internal/mem.h to
  Objects/obmalloc.c
* Remove _PyObject_Initialize() and _PyMem_Initialize()
* Remove Include/internal/pymalloc.h
* Add test_capi.test_pre_initialization_api():
   Make sure that it's possible to call Py_DecodeLocale(), and then call
   Py_SetProgramName() with the decoded string, before Py_Initialize().

PyMem_RawMalloc() and Py_DecodeLocale() can be called again before
_PyRuntimeState_Init().

Co-Authored-By: Eric Snow <ericsnowcurrently@gmail.com>
2017-11-24 12:09:24 +01:00
Victor Stinner 0327bde9da
bpo-32030: Rewrite calculate_path() (#4521)
* calculate_path() rewritten in Modules/getpath.c and PC/getpathp.c
* Move global variables into a new PyPathConfig structure.
* calculate_path():

  * Split the huge calculate_path() function into subfunctions.
  * Add PyCalculatePath structure to pass data between subfunctions.
  * Document PyCalculatePath fields.
  * Move cleanup code into a new calculate_free() subfunction
  * calculate_init() now handles Py_DecodeLocale() failures properly
  * calculate_path() is now atomic: only replace PyPathConfig
    (path_config) at once on success.

* _Py_GetPythonHomeWithConfig() now returns an error on failure
* Add _Py_INIT_NO_MEMORY() helper: report a memory allocation failure
* Coding style fixes (PEP 7)
2017-11-23 17:03:20 +01:00
xdegaye c06c22e9a9
bpo-29040: Support building Android with Unified Headers (GH-4492) 2017-11-23 11:44:38 +01:00
Victor Stinner 1f15111a6e
bpo-32030: Add _PyMainInterpreterConfig.pythonhome (#4513)
* Py_Main() now reads the PYTHONHOME environment variable
* Add _Py_GetPythonHomeWithConfig() private function
* Add _PyWarnings_InitWithConfig()
* init_filters() doesn't get the current core configuration from the
  current interpreter or Python thread anymore. Pass explicitly the
  configuration to _PyWarnings_InitWithConfig().
* _Py_InitializeCore() now fails on _PyWarnings_InitWithConfig()
  failure.
* Pass configuration as constant
2017-11-23 10:43:14 +01:00
Victor Stinner e32e79f7d8
bpo-32030: Move PYTHONPATH to _PyMainInterpreterConfig (#4511)
Move _PyCoreConfig.module_search_path_env to _PyMainInterpreterConfig
structure.
2017-11-23 01:49:45 +01:00
Victor Stinner d434110974
bpo-32030: Add _PyCoreConfig.module_search_path_env (#4504)
Changes:

* Py_Main() initializes _PyCoreConfig.module_search_path_env from
  the PYTHONPATH environment variable.
* PyInterpreterState_New() now initializes core_config and config
  fields
* Compute sys.path a little bit ealier in
  _Py_InitializeMainInterpreter() and new_interpreter()
* Add _Py_GetPathWithConfig() private function.
2017-11-23 00:12:09 +01:00
Victor Stinner 25420fe290
bpo-32030: Add more options to _PyCoreConfig (#4485)
Py_Main() now handles two more -X options:

* -X showrefcount: new _PyCoreConfig.show_ref_count field
* -X showalloccount: new _PyCoreConfig.show_alloc_count field
2017-11-20 18:12:22 -08:00
Victor Stinner 09f3a8a124
bpo-32089: Fix warnings filters in dev mode (#4482)
The developer mode (-X dev) now creates all default warnings filters
to order filters in the correct order to always show ResourceWarning
and make BytesWarning depend on the -b option.

Write a functional test to make sure that ResourceWarning is logged
twice at the same location in the developer mode.

Add a new 'dev_mode' field to _PyCoreConfig.
2017-11-20 17:32:40 -08:00
Victor Stinner a7368ac636
bpo-32030: Enhance Py_Main() (#4412)
Parse more env vars in Py_Main():

* Add more options to _PyCoreConfig:

  * faulthandler
  * tracemalloc
  * importtime

* Move code to parse environment variables from _Py_InitializeCore()
  to Py_Main(). This change fixes a regression from Python 3.6:
  PYTHONUNBUFFERED is now read before calling pymain_init_stdio().
* _PyFaulthandler_Init() and _PyTraceMalloc_Init() now take an
  argument to decide if the module has to be enabled at startup.
* tracemalloc_start() is now responsible to check the maximum number
  of frames.

Other changes:

* Cleanup Py_Main():

  * Rename some pymain_xxx() subfunctions
  * Add pymain_run_python() subfunction

* Cleanup Py_NewInterpreter()
* _PyInterpreterState_Enable() now reports failure
* init_hash_secret() now considers pyurandom() failure as an "user
  error": don't fail with abort().
* pymain_optlist_append() and pymain_strdup() now sets err on memory
  allocation failure.
2017-11-15 18:11:45 -08:00
Victor Stinner f7e5b56c37
bpo-32030: Split Py_Main() into subfunctions (#4399)
* Don't use "Python runtime" anymore to parse command line options or
  to get environment variables: pymain_init() is now a strict
  separation.
* Use an error message rather than "crashing" directly with
  Py_FatalError(). Limit the number of calls to Py_FatalError(). It
  prepares the code to handle errors more nicely later.
* Warnings options (-W, PYTHONWARNINGS) and "XOptions" (-X) are now
  only added to the sys module once Python core is properly
  initialized.
* _PyMain is now the well identified owner of some important strings
  like: warnings options, XOptions, and the "program name". The
  program name string is now properly freed at exit.
  pymain_free() is now responsible to free the "command" string.
* Rename most methods in Modules/main.c to use a "pymain_" prefix to
  avoid conflits and ease debug.
* Replace _Py_CommandLineDetails_INIT with memset(0)
* Reorder a lot of code to fix the initialization ordering. For
  example, initializing standard streams now comes before parsing
  PYTHONWARNINGS.
* Py_Main() now handles errors when adding warnings options and
  XOptions.
* Add _PyMem_GetDefaultRawAllocator() private function.
* Cleanup _PyMem_Initialize(): remove useless global constants: move
  them into _PyMem_Initialize().
* Call _PyRuntime_Initialize() as soon as possible:
  _PyRuntime_Initialize() now returns an error message on failure.
* Add _PyInitError structure and following macros:

  * _Py_INIT_OK()
  * _Py_INIT_ERR(msg)
  * _Py_INIT_USER_ERR(msg): "user" error, don't abort() in that case
  * _Py_INIT_FAILED(err)
2017-11-15 15:48:08 -08:00
Serhiy Storchaka 9b6c60cbce
bpo-31979: Simplify transforming decimals to ASCII (#4336)
in int(), float() and complex() parsers.

This also speeds up parsing non-ASCII numbers by around 20%.
2017-11-13 21:23:48 +02:00
xdegaye 1588be66d7
bpo-28180: Fix the implementation of PEP 538 on Android (GH-4334) 2017-11-12 12:45:59 +01:00
Ned Batchelder 01ae58d446 Correct the location of a function mentioned in a comment (GH-4327) 2017-11-09 19:55:34 +03:00
Victor Stinner 8c663fd60e
Replace KB unit with KiB (#4293)
kB (*kilo* byte) unit means 1000 bytes, whereas KiB ("kibibyte")
means 1024 bytes. KB was misused: replace kB or KB with KiB when
appropriate.

Same change for MB and GB which become MiB and GiB.

Change the output of Tools/iobench/iobench.py.

Round also the size of the documentation from 5.5 MB to 5 MiB.
2017-11-08 14:44:44 -08:00
luzpaz a5293b4ff2 Fix miscellaneous typos (#4275) 2017-11-05 15:37:50 +02:00
Anselm Kruis 9e33973332 bpo-31890: define METH_STACKLESS (#4159)
Add METH_STACKLESS to prevent future collisions.
2017-11-02 15:54:57 -07:00
Victor Stinner c29b585fd4
bpo-31784: Implement PEP 564: add time.time_ns() (#3989)
Add new time functions:

* time.clock_gettime_ns()
* time.clock_settime_ns()
* time.monotonic_ns()
* time.perf_counter_ns()
* time.process_time_ns()
* time.time_ns()

Add new _PyTime functions:

* _PyTime_FromTimespec()
* _PyTime_FromNanosecondsObject()
* _PyTime_FromTimeval()

Other changes:

* Add also os.times() tests to test_os.
* pytime_fromtimeval() and pytime_fromtimeval() now return
  _PyTime_MAX or _PyTime_MIN on overflow, rather than undefined
  behaviour
* _PyTime_FromNanoseconds() parameter type changes from long long to
  _PyTime_t
2017-11-02 07:28:27 -07:00
stratakis e8b1965639 bpo-23699: Use a macro to reduce boilerplate code in rich comparison functions (GH-793) 2017-11-02 20:32:54 +10:00
Masayuki Yamamoto 8bc7d63560 bpo-25720: Fix the method for checking pad state of curses WINDOW (#4164)
Modify the code to use ncurses is_pad() instead of checking WINDOW
_flags field.  If your platform does not provide the is_pad(), the
existing way that checks the field will be enabled.

Note: This change does not drop support for platforms where do not
have both WINDOW _flags field and is_pad().
2017-11-01 14:05:26 +02:00
Serhiy Storchaka baac01e629
bpo-31891: Fix building the curses module on NetBSD. (#4165) 2017-10-31 13:56:44 +02:00
Erik Bray 031c4bfadb bpo-31877: Add _Py_LegacyLocaleDetected and _PyCoerceLegacyLocale to pylifecycle.h (GH-4134)
Only declaring these as interns inside the CLI's main C module
caused build problems on some platforms (notably Cygwin), so
this switches them to a regular underscore prefixed "private" C
API declaration.
2017-10-27 19:46:03 +10:00
Serhiy Storchaka bdf4298ae2 Fix trailing whitespaces in C files. (#4130) 2017-10-26 16:59:40 +03:00
xdegaye 56d1f5ca32 bpo-30697: Fix PyErr_NormalizeException() when no memory (GH-2327) 2017-10-26 15:09:06 +02:00
pdox 1896793520 bpo-31857: Make the behavior of USE_STACKCHECK deterministic (#4098) 2017-10-25 23:03:01 -07:00
Victor Stinner 850a18e03e bpo-30768: Recompute timeout on interrupted lock (GH-4103)
Fix the pthread+semaphore implementation of
PyThread_acquire_lock_timed() when called with timeout > 0 and
intr_flag=0: recompute the timeout if sem_timedwait() is interrupted
by a signal (EINTR).

See also the PEP 475.

The pthread implementation of PyThread_acquire_lock() now fails with
a fatal error if the timeout is larger than PY_TIMEOUT_MAX, as done
in the Windows implementation.

The check prevents any risk of overflow in PyThread_acquire_lock().

Add also PY_DWORD_MAX constant.
2017-10-24 16:53:32 -07:00
Mark Shannon ae3087c638 Move exc state to generator. Fixes bpo-25612 (#1773)
Move exception state information from frame objects to coroutine (generator/thread) object where it belongs.
2017-10-22 23:41:51 +02:00
Pablo Galindo 2c15b29aea bpo-31786: Make functions in the select module blocking when timeout is a small negative value. (#4003) 2017-10-17 17:14:41 +03:00
Ned Deily 2bd37c227e post 3.7.0a2 development 2017-10-17 02:10:44 -04:00
Ned Deily f7ac4fe52a Bump to 3.7.0a2 2017-10-16 23:50:50 -04:00
brainfvck c75edabbb6 bpo-31558: Add gc.freeze() (#3705)
Freeze all the objects tracked by gc - move them to a permanent generation
and ignore all the future collections. This can be used before a POSIX
fork() call to make the gc copy-on-write friendly or to speed up collection.
2017-10-16 12:49:41 -07:00
Victor Stinner bdaeb7d237 bpo-31773: _PyTime_GetPerfCounter() uses _PyTime_t (GH-3983)
* Rewrite win_perf_counter() to only use integers internally.
* Add _PyTime_MulDiv() which compute "ticks * mul / div"
  in two parts (int part and remaining) to prevent integer overflow.
* Clock frequency is checked at initialization for integer overflow.
* Enhance also pymonotonic() to reduce the precision loss on macOS
  (mach_absolute_time() clock).
2017-10-16 08:44:31 -07:00
Victor Stinner cba9a0c6de bpo-31773: time.perf_counter() uses again double (GH-3964)
time.clock() and time.perf_counter() now use again C double
internally.

Remove also _PyTime_GetWinPerfCounterWithInfo(): use
_PyTime_GetPerfCounterDoubleWithInfo() instead on Windows.
2017-10-12 08:51:56 -07:00
Victor Stinner a997c7b434 bpo-31415: Add _PyTime_GetPerfCounter() and use it for -X importtime (#3936)
* Add _PyTime_GetPerfCounter()
* Use _PyTime_GetPerfCounter() for -X importtime
2017-10-10 02:51:50 -07:00
Masayuki Yamamoto 731e189014 bpo-25658: Implement PEP 539 for Thread Specific Storage (TSS) API (GH-1362)
See PEP 539 for details.

Highlights of changes:

- Add Thread Specific Storage (TSS) API
- Document the Thread Local Storage (TLS) API as deprecated
- Update code that used TLS API to use TSS API
2017-10-06 20:41:34 +10:00
Yury Selivanov faa135acbf bpo-31709: Drop support for asynchronous __aiter__. (#3903) 2017-10-06 02:08:57 -04:00
Jelle Zijlstra ac317700ce bpo-30406: Make async and await proper keywords (#1669)
Per PEP 492, 'async' and 'await' should become proper keywords in 3.7.
2017-10-05 23:24:46 -04:00
Christian Heimes 3d2b407da0 bpo-31574: importlib dtrace (#3749)
Importlib was instrumented with two dtrace probes to profile import timing.

Signed-off-by: Christian Heimes <christian@python.org>
2017-09-29 15:53:19 -07:00
Benjamin Peterson 288d1daada remove support for BSD/OS (closes bpo-31624) (#3812) 2017-09-28 22:44:27 -07:00
Ned Deily 453408a505 post 3.7.0a1 development 2017-09-19 03:12:46 -04:00
Ned Deily 8f51bb436f Bump to 3.6.0a1 2017-09-19 01:09:03 -04:00
Serhiy Storchaka 4ab46d7949 bpo-31497: Add private helper _PyType_Name(). (#3630)
This function returns the last component of tp_name after a dot.
Returns tp_name itself if it doesn't contain a dot.
2017-09-17 21:11:04 +03:00
Eric Snow 3f9eee6eb4 bpo-28411: Support other mappings in PyInterpreterState.modules. (#3593)
The concrete PyDict_* API is used to interact with PyInterpreterState.modules in a number of places. This isn't compatible with all dict subclasses, nor with other Mapping implementations. This patch switches the concrete API usage to the corresponding abstract API calls.

We also add a PyImport_GetModule() function (and some other helpers) to reduce a bunch of code duplication.
2017-09-15 16:35:20 -06:00
Serhiy Storchaka 5d84cb368c bpo-31464: asdl_c.py no longer emits trailing spaces in Python-ast.h. (#3568) 2017-09-14 20:28:22 -07:00
Barry Warsaw b2e5794870 bpo-31338 (#3374)
* Add Py_UNREACHABLE() as an alias to abort().
* Use Py_UNREACHABLE() instead of assert(0)
* Convert more unreachable code to use Py_UNREACHABLE()
* Document Py_UNREACHABLE() and a few other macros.
2017-09-14 18:13:16 -07:00
Eric Snow d393c1b227 bpo-28411: Isolate PyInterpreterState.modules (#3575)
A bunch of code currently uses PyInterpreterState.modules directly instead of PyImport_GetModuleDict(). This complicates efforts to make changes relative to sys.modules. This patch switches to using PyImport_GetModuleDict() uniformly. Also, a number of related uses of sys.modules are updated for uniformity for the same reason.

Note that this code was already reviewed and merged as part of #1638. I reverted that and am now splitting it up into more focused parts.
2017-09-14 12:18:12 -06:00
Eric Snow dae0276bb6 bpo-30860: Fix a refleak. (#3567)
Resolves bpo-31420.

(This was accidentally reverted when in #3565.)
2017-09-14 00:35:58 -07:00
Eric Snow 93c92f7d1d bpo-31404: Revert "remove modules from Py_InterpreterState (#1638)" (#3565)
PR #1638, for bpo-28411, causes problems in some (very) edge cases. Until that gets sorted out, we're reverting the merge. PR #3506, a fix on top of #1638, is also getting reverted.
2017-09-13 23:46:04 -07:00
Serhiy Storchaka 13ad3b7a82 bpo-31462: Remove trailing whitespaces. (#3564) 2017-09-14 09:38:36 +03:00
Eric Snow fc1bf872e9 bpo-30860: Move windows.h include out of internal/*.h. (#3458)
PR #3397 introduced a large number of warnings to the Windows build. This patch fixes them.
2017-09-11 18:30:43 -07:00
Eric Snow 8728018624 bpo-30860: Fix a refleak. (#3506)
* Drop warnoptions from PyInterpreterState.

* Drop xoptions from PyInterpreterState.

* Don't set warnoptions and _xoptions again.

* Decref after adding to sys.__dict__.

* Drop an unused macro.

* Check sys.xoptions *before* we delete it.
2017-09-11 17:59:22 -07:00
Eric Snow ba6d5d1def bpo-30860: Always provide serialno. (#3507)
This fixes bpo-31407.
2017-09-11 17:02:24 -07:00