Commit Graph

2996 Commits

Author SHA1 Message Date
Nathaniel J. Smith fc2f407829 bpo-32591: Add native coroutine origin tracking (#5250)
* Add coro.cr_origin and sys.set_coroutine_origin_tracking_depth
* Use coroutine origin information in the unawaited coroutine warning
* Stop using set_coroutine_wrapper in asyncio debug mode
* In BaseEventLoop.set_debug, enable debugging in the correct thread
2018-01-21 09:44:07 -05:00
INADA Naoki 378edee0a3
bpo-32544: Speed up hasattr() and getattr() (GH-5173)
AttributeError was raised always when attribute is not found.
This commit skip raising AttributeError when `tp_getattro` is `PyObject_GenericGetAttr`.
It makes hasattr() and getattr() about 4x faster when attribute is not found.
2018-01-16 20:52:41 +09:00
Victor Stinner cb064fc232
bpo-31900: Fix localeconv() encoding for LC_NUMERIC (#4174)
* Add _Py_GetLocaleconvNumeric() function: decode decimal_point and
  thousands_sep fields of localeconv() from the LC_NUMERIC encoding,
  rather than decoding from the LC_CTYPE encoding.
* Modify locale.localeconv() and "n" formatter of str.format() (for
  int, float and complex to use _Py_GetLocaleconvNumeric()
  internally.
2018-01-15 15:58:02 +01:00
Victor Stinner 7ed7aead95
bpo-29240: Fix locale encodings in UTF-8 Mode (#5170)
Modify locale.localeconv(), time.tzname, os.strerror() and other
functions to ignore the UTF-8 Mode: always use the current locale
encoding.

Changes:

* Add _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx(). On decoding or
  encoding error, they return the position of the error and an error
  message which are used to raise Unicode errors in
  PyUnicode_DecodeLocale() and PyUnicode_EncodeLocale().
* Replace _Py_DecodeCurrentLocale() with _Py_DecodeLocaleEx().
* PyUnicode_DecodeLocale() now uses _Py_DecodeLocaleEx() for all
  cases, especially for the strict error handler.
* Add _Py_DecodeUTF8Ex(): return more information on decoding error
  and supports the strict error handler.
* Rename _Py_EncodeUTF8_surrogateescape() to _Py_EncodeUTF8Ex().
* Replace _Py_EncodeCurrentLocale() with _Py_EncodeLocaleEx().
* Ignore the UTF-8 mode to encode/decode localeconv(), strerror()
  and time zone name.
* Remove PyUnicode_DecodeLocale(), PyUnicode_DecodeLocaleAndSize()
  and PyUnicode_EncodeLocale() now ignore the UTF-8 mode: always use
  the "current" locale.
* Remove _PyUnicode_DecodeCurrentLocale(),
  _PyUnicode_DecodeCurrentLocaleAndSize() and
  _PyUnicode_EncodeCurrentLocale().
2018-01-15 10:45:49 +01:00
Victor Stinner cb3ae5588b
bpo-29240: Ignore UTF-8 Mode in time module (#5148)
time.strftime() must use the current LC_CTYPE encoding, not UTF-8
if the UTF-8 mode is enabled.

Add _PyUnicode_DecodeCurrentLocale() function.
2018-01-11 10:37:59 +01:00
Victor Stinner 2cba6b8579
bpo-29240: readline now ignores the UTF-8 Mode (#5145)
Add new fuctions ignoring the UTF-8 mode:

* _Py_DecodeCurrentLocale()
* _Py_EncodeCurrentLocale()
* _PyUnicode_DecodeCurrentLocaleAndSize()
* _PyUnicode_EncodeCurrentLocale()

Modify the readline module to use these functions.

Re-enable test_readline.test_nonascii().
2018-01-10 22:46:15 +01:00
Ned Deily 2ab3a8fd26 Post-release version bump to a4+ 2018-01-09 10:27:57 -05:00
Ned Deily 07c9d8547c Bump to 3.7.0a4 2018-01-08 22:05:14 -05:00
Nathaniel J. Smith 735ae8d139 bpo-29137: Remove fpectl module (#4789)
This module has never been enabled by default, never worked correctly
on x86-64, and caused ABI problems that caused C extension
compatibility. See bpo-29137 for details/discussion.
2018-01-05 23:15:34 -08:00
Serhiy Storchaka 3dfbaf51f0
bpo-32372: Move __debug__ optimization to the AST level. (#4925) 2017-12-25 12:47:50 +02:00
Serhiy Storchaka 1b3029ac83
bpo-29084: Exclude C API for OrderedDict from the limited C API. (#4900) 2017-12-25 02:08:42 +02:00
Victor Stinner 9bee329130
bpo-32030: Add _Py_FindEnvConfigValue() (#4963)
Add a new _Py_FindEnvConfigValue() function: code shared between
Windows and Unix implementations of _PyPathConfig_Calculate() to read
the pyenv.cfg file.

_Py_FindEnvConfigValue() now uses _Py_DecodeUTF8_surrogateescape()
instead of using a Python Unicode string, the Python API must not be
used early during Python initialization. Same change in Unix
search_for_exec_prefix(): use _Py_DecodeUTF8_surrogateescape().

Cleanup also encode_current_locale(): PyMem_RawFree/PyMem_Free can be
called with NULL.

Fix also "NUL byte" => "NULL byte" typo.
2017-12-21 16:49:13 +01:00
Victor Stinner 9dd762013f
bpo-32030: Add _Py_EncodeLocaleRaw() (#4961)
Replace Py_EncodeLocale() with _Py_EncodeLocaleRaw() in:

* _Py_wfopen()
* _Py_wreadlink()
* _Py_wrealpath()
* _Py_wstat()
* pymain_open_filename()

These functions are called early during Python intialization, only
the RAW memory allocator must be used.
2017-12-21 16:20:32 +01:00
Victor Stinner 9cfc00262c
bpo-32030: Complete _PyCoreConfig_Read() (#4946)
* Add _PyCoreConfig.install_signal_handlers
* Remove _PyMain.config: _PyMainInterpreterConfig usage is now
  restricted to pymain_init_python_main().
* Rename _PyMain.core_config to _PyMain.config
* _PyMainInterpreterConfig_Read() now creates the xoptions dictionary
   from the core config
* Fix _PyMainInterpreterConfig_Read(): don't replace xoptions and
  argv if they are already set.
2017-12-20 19:36:46 +01:00
Victor Stinner ca719ac42b
bpo-32030: Add _PyCoreConfig.warnoptions (#4936)
Merge _PyCoreConfig_ReadEnv() into _PyCoreConfig_Read(), and
_Py_CommandLineDetails usage is now restricted to pymain_cmdline().

Changes:

* _PyCoreConfig: Add nxoption, xoptions, nwarnoption and warnoptions
* Add _PyCoreConfig.program: argv[0] or ""
* Move filename, command, module and xoptions from
  _Py_CommandLineDetails to _PyMain. xoptions _Py_OptList becomes
  (int, wchar_t**) list.
* Add pymain_cmdline() function
* Rename copy_argv() to copy_wstrlist(). Rename clear_argv() to
  clear_wstrlist(). Remove _Py_OptList structure: use (int,
  wchar_t**) list instead.
* Rename pymain_set_flag_from_env() to pymain_get_env_flag()
* Rename pymain_set_flags_from_env() to pymain_get_env_flags()
* _PyMainInterpreterConfig_Read() now creates the warnoptions from
  _PyCoreConfig.warnoptions
* Inline pymain_add_warning_dev_mode() and
  pymain_add_warning_bytes_flag() into config_init_warnoptions()
* Inline pymain_get_program_name() into _PyCoreConfig_Read()
* _Py_CommandLineDetails: Replace warning_options with nwarnoption
  and warnoptions. Replace env_warning_options with nenv_warnoption
  and env_warnoptions.
* pymain_warnings_envvar() now has a single implementation for
  Windows and Unix: use config_get_env_var_dup() to also get the
  variable as wchar_t* on Unix.
2017-12-20 18:00:19 +01:00
Marcel Plch 776407fe89 bpo-31901: atexit callbacks should be run at subinterpreter shutdown (#4611)
Change atexit behavior and PEP-489 multiphase init support.
2017-12-20 11:17:58 +01:00
Victor Stinner c4bca95106
bpo-32030: Add _PyCoreConfig.argv (#4934)
* Add argc and argv to _PyCoreConfig
* _PyMainInterpreterConfig_Read() now builds its argv from
  _PyCoreConfig.arg
* Move _PyMain.env_warning_options into _Py_CommandLineDetails
* Reorder pymain_free()
2017-12-19 23:48:17 +01:00
Victor Stinner 5d8624647d
bpo-32030: Fix compiler warnings (#4921)
Fix compiler warnings in Py_FinalizeEx(): only define variables if
they are needed, add #ifdef.

Other cleanup changes:

* _PyWarnings_InitWithConfig() is no more needed: call
  _PyWarnings_Init() instead.
* Inline pymain_init_main_interpreter() in its caller. This
  subfunction is no more justifed.
2017-12-19 11:35:58 +01:00
Benjamin Peterson 4c72bc4a38
add 'extern' to pygetopt.h symbols, so then don't end up in comdat (#4909)
bpo-32264
2017-12-17 10:34:00 -08:00
Andrew Svetlov 950840261c
Fix PyObject_Hash signature in comment (#4905) 2017-12-16 21:08:05 +02:00
Victor Stinner 9454060e84
bpo-29240, bpo-32030: Py_Main() re-reads config if encoding changes (#4899)
bpo-29240, bpo-32030: If the encoding change (C locale coerced or
UTF-8 Mode changed), Py_Main() now reads again the configuration with
the new encoding.

Changes:

* Add _Py_UnixMain() called by main().
* Rename pymain_free_pymain() to pymain_clear_pymain(), it can now be
  called multipled times.
* Rename pymain_parse_cmdline_envvars() to pymain_read_conf().
* Py_Main() now clears orig_argc and orig_argv at exit.
* Remove argv_copy2, Py_Main() doesn't modify argv anymore. There is
  no need anymore to get two copies of the wchar_t** argv.
* _PyCoreConfig: add coerce_c_locale and coerce_c_locale_warn.
* Py_UTF8Mode is now initialized to -1.
* Locale coercion (PEP 538) now respects -I and -E options.
2017-12-16 04:54:22 +01:00
Serhiy Storchaka a5552f023e
bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
Benjamin Peterson e425bd7517
move pygetopt.h to internal (closes bpo-32264) (#4830) 2017-12-14 23:48:12 -08:00
Victor Stinner 41264f1cd4
bpo-32030: Add _PyMainInterpreterConfig.executable (#4876)
* Add new fields to _PyMainInterpreterConfig:

  * executable
  * prefix
  * base_prefix
  * exec_prefix
  * base_exec_prefix

* _PySys_EndInit() now sets sys attributes from
  _PyMainInterpreterConfig
2017-12-15 02:05:29 +01:00
Victor Stinner da273412c4
bpo-32030: Add _PyCoreConfig_Copy() (#4874)
Each interpreter now has its core_config and main_config copy:

* Add _PyCoreConfig_Copy() and _PyMainInterpreterConfig_Copy()
* Move _PyCoreConfig_Read(), _PyCoreConfig_Clear() and
  _PyMainInterpreterConfig_Clear() from Python/pylifecycle.c to
  Modules/main.c
* Fix _Py_InitializeEx_Private(): call _PyCoreConfig_ReadEnv() before
  _Py_InitializeCore()
2017-12-15 01:46:02 +01:00
Victor Stinner 358e5e17a5
bpo-32329: Fix -R option for hash randomization (#4873)
bpo-32329, bpo-32030:

* The -R option now turns on hash randomization when the
  PYTHONHASHSEED environment variable is set to 0 Previously, the
  option was ignored.
* sys.flags.hash_randomization is now properly set to 0 when hash
  randomization is turned off by PYTHONHASHSEED=0.
* _PyCoreConfig_ReadEnv() now reads the PYTHONHASHSEED environment
  variable. _Py_HashRandomization_Init() now only apply the
  configuration, it doesn't read PYTHONHASHSEED anymore.
2017-12-15 00:51:22 +01:00
Victor Stinner 374c6e178a
bpo-32030: Add _PyMainInterpreterConfig.warnoptions (#4855)
Add warnoptions and xoptions fields to _PyMainInterpreterConfig.
2017-12-14 12:05:26 +01:00
INADA Naoki 7ea143ae79
bpo-29469: Move constant folding to AST optimizer (GH-2858) 2017-12-14 16:47:20 +09:00
Victor Stinner b5fd9ad05e
bpo-32030: Rewrite _PyMainInterpreterConfig (#4854)
_PyMainInterpreterConfig now contains Python objects, whereas
_PyCoreConfig contains wchar_t* strings.

Core config:

* Rename _PyMainInterpreterConfig_ReadEnv() to _PyCoreConfig_ReadEnv()
* Move 3 strings from _PyMainInterpreterConfig to _PyCoreConfig:
  module_search_path_env, home, program_name.
* Add _PyCoreConfig_Clear()
* _PyPathConfig_Calculate() now takes core config rather than main
  config
* _PyMainInterpreterConfig_Read() now requires also a core config

Main config:

* Add _PyMainInterpreterConfig.module_search_path: sys.path list
* Add _PyMainInterpreterConfig.argv: sys.argv list
* _PyMainInterpreterConfig_Read() now computes module_search_path
2017-12-14 02:20:52 +01:00
Victor Stinner 11a247df88
bpo-32030: Add _PyPathConfig_ComputeArgv0() (#4845)
Changes:

* Split _PySys_SetArgvWithError() into subfunctions for Py_Main():

  * Create the Python list object
  * Set sys.argv to the list
  * Compute argv0
  * Prepend argv0 to sys.path

* Add _PyPathConfig_ComputeArgv0()
* Remove _PySys_SetArgvWithError()
* Py_Main() now splits the code to compute sys.argv/path0 and the
  code to update the sys module: add pymain_compute_argv()
  subfunction.
2017-12-13 21:05:57 +01:00
Victor Stinner d5dda98fa8
pymain_set_sys_argv() now copies argv (#4838)
bpo-29240, bpo-32030:

* Rename pymain_set_argv() to pymain_set_sys_argv()
* pymain_set_sys_argv() now creates of copy of argv and modify the
  copy, rather than modifying pymain->argv
* Call pymain_set_sys_argv() earlier: before pymain_run_python(), but
  after pymain_get_importer().
* Add _PySys_SetArgvWithError() to handle errors
2017-12-13 17:31:16 +01:00
Victor Stinner 91106cd9ff
bpo-29240: PEP 540: Add a new UTF-8 Mode (#855)
* Add -X utf8 command line option, PYTHONUTF8 environment variable
  and a new sys.flags.utf8_mode flag.
* If the LC_CTYPE locale is "C" at startup: enable automatically the
  UTF-8 mode.
* Add _winapi.GetACP(). encodings._alias_mbcs() now calls
  _winapi.GetACP() to get the ANSI code page
* locale.getpreferredencoding() now returns 'UTF-8' in the UTF-8
  mode. As a side effect, open() now uses the UTF-8 encoding by
  default in this mode.
* Py_DecodeLocale() and Py_EncodeLocale() now use the UTF-8 encoding
  in the UTF-8 Mode.
* Update subprocess._args_from_interpreter_flags() to handle -X utf8
* Skip some tests relying on the current locale if the UTF-8 mode is
  enabled.
* Add test_utf8mode.py.
* _Py_DecodeUTF8_surrogateescape() gets a new optional parameter to
  return also the length (number of wide characters).
* pymain_get_global_config() and pymain_set_global_config() now
  always copy flag values, rather than only copying if the new value
  is greater than the old value.
2017-12-13 12:29:09 +01:00
Serhiy Storchaka 4ae06c5337
bpo-32241: Add the const qualifire to declarations of umodifiable strings. (#4748) 2017-12-12 13:55:04 +02:00
Chris e0720cd9e4 Fix small typo in tupleobject.h (#4801) 2017-12-12 09:59:30 +02:00
Benjamin Peterson 42aa93b8ff
closes bpo-31650: PEP 552 (Deterministic pycs) implementation (#4575)
Python now supports checking bytecode cache up-to-dateness with a hash of the
source contents rather than volatile source metadata. See the PEP for details.

While a fairly straightforward idea, quite a lot of code had to be modified due
to the pervasiveness of pyc implementation details in the codebase. Changes in
this commit include:

- The core changes to importlib to understand how to read, validate, and
  regenerate hash-based pycs.

- Support for generating hash-based pycs in py_compile and compileall.

- Modifications to our siphash implementation to support passing a custom
  key. We then expose it to importlib through _imp.

- Updates to all places in the interpreter, standard library, and tests that
  manually generate or parse pyc files to grok the new format.

- Support in the interpreter command line code for long options like
  --check-hash-based-pycs.

- Tests and documentation for all of the above.
2017-12-09 10:26:52 -08:00
Victor Stinner 92a3c6f493
bpo-32030: Add _PyImport_Fini2() (#4737)
PyImport_ExtendInittab() now uses PyMem_RawRealloc() rather than
PyMem_Realloc(). PyImport_ExtendInittab() can be called before
Py_Initialize() whereas only the PyMem_Raw allocator is supposed to
be used before Py_Initialize().

Add _PyImport_Fini2() to release the memory allocated by
PyImport_ExtendInittab() at exit. PyImport_ExtendInittab() now forces
the usage of the default raw allocator, to be able to release memory
in _PyImport_Fini2().

Don't export these functions anymore to be C API, only to
Py_BUILD_CORE:

* _PyExc_Fini()
* _PyImport_Fini()
* _PyGC_DumpShutdownStats()
* _PyGC_Fini()
* _PyType_Fini()
* _Py_HashRandomization_Fini()
2017-12-06 18:12:59 +01:00
Victor Stinner 6bf992a1ac
bpo-32030: Add pymain_get_global_config() (#4735)
* Py_Main() now starts by reading Py_xxx configuration variables to
  only work on its own private structure, and then later writes back
  the configuration into these variables.
* Replace Py_GETENV() with pymain_get_env_var() which ignores empty
  variables.
* Add _PyCoreConfig.dump_refs
* Add _PyCoreConfig.malloc_stats
* _PyObject_DebugMallocStats() is now responsible to check if debug
  hooks are installed. The function returns 1 if stats were written,
  or 0 if the hooks are disabled. Mark _PyMem_PymallocEnabled() as
  static.
2017-12-06 17:26:10 +01:00
Victor Stinner 672b6baa71
bpo-32030: pass interp to _PyImport_Init() (#4736)
Remove also the initstr variable, unused since the commit
e69f0df45b pushed in 2012: "bpo-13959:
Re-implement imp.find_module() in Lib/imp.py"
2017-12-06 17:25:50 +01:00
Ned Deily a2a25eb037 Bump to 3.7.0a3+ 2017-12-05 21:06:12 -05:00
Ned Deily d86f29e9c9 Python 3.7.0a3
-----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCgAdFiEEDZbfTUEQ5cQ/v7F/LTR+pqplQh0FAlomWNwACgkQLTR+pqpl
 Qh34Aw/+LDgBou5bRTgHXRAyx4vU8gDT97K0Fm604FXTcQLtTMo/3LDVa+7NbROg
 EWIaSxOjObRwZxjTVU1P063QKBDpjzeqK0PE+DYtEfpKVcqm8tADG30jHx9FYnXp
 yalV6/NzzHhcdxdhwrcbbAYnykk7xcUqEAimJ6YLgQYqe5Z1cemprjuPIcDWAagV
 Jw9ARZI9somzoqDRNAT1s72rJ7V/nKFGOcZ0PL+pv65o7essht+v7ReSxLsbyJYC
 ZAAxTiJT6fpyGc6/ZcKJ6xnbxINvlkdfqwELFMM7/+6l5Xwg07pZSEzOdh1Ng1sz
 woBIJkAoCTV4xAiKZ4DW6vxEeRFDnJgSO7Vd2zFKpSa9iLorZBEpNSObP/1gLTUC
 0fah7ddY2ZIHsi6vdAKnD877FoYFlNsSnlYxHxE1dOOB/5LbHd25/4gTjOHvav8n
 Xrd0qZ4SVVNSdqvDIhsf3GHnao9N5wKkcs9YBX/BK8ryRdIyUof3a2vXoOJtsNNt
 poyIhmrnqjRYttDVHefPk+vi/rS3l3kLfj71AKdmfO5Fv7xFXbzcJBU3ax5SBvbp
 w1XoMoTOSJDZt/GdMvC2htsG5PZWw8Bg8RGqBrLyUFEN0rnoIh5ykEH6YuPSusDQ
 fFUuaE1pGETNhrE50YWYYb4MGOE4xakdRV6cT402DhrzexFwuCA=
 =CREY
 -----END PGP SIGNATURE-----

Merge tag 'v3.7.0a3'
2017-12-05 20:57:44 -05:00
Victor Stinner 33c377ed9b
bpo-32030: Simplify _PyCoreConfig_INIT macro (#4728)
* Simplify _PyCoreConfig_INIT, _PyMainInterpreterConfig_INIT,
  _PyPathConfig_INIT macros: no need to set fields to 0/NULL, it's
  redundant (the C language sets them to 0/NULL for us).
* Fix typo: pymain_run_statup() => pymain_run_startup()
* Remove a few XXX/TODO
2017-12-05 15:12:41 +01:00
Ned Deily 90a6785f63 Bump to 3.7.0a3 2017-12-05 03:27:19 -05:00
Victor Stinner 31a8393cf6
Revert "bpo-32197: Try to fix a compiler error on OS X introduced in bpo-32030. (#4681)" (#4694)
* Revert "bpo-32197: Try to fix a compiler error on OS X introduced in bpo-32030. (#4681)"

This reverts commit 13badcbc60.

Re-apply commits:

* "bpo-32030: _PyPathConfig_Init() sets home and program_name (#4673)"
  commit af5a895073.
* "bpo-32030: Fix config_get_program_name() on macOS (#4669)"
  commit e23c06e2b0.
* "bpo-32030: Add Python/pathconfig.c (#4668)"
  commit 0ea395ae96.
* "bpo-32030: Don't call _PyPathConfig_Fini() in Py_FinalizeEx() (#4667)"
  commit ebac19dad6.
* "bpo-32030: Fix Py_GetPath(): init program_name (#4665)"
  commit 9ac3d88827.

* Fix compilation error on macOS
2017-12-04 13:39:15 +01:00
Serhiy Storchaka 13badcbc60
bpo-32197: Try to fix a compiler error on OS X introduced in bpo-32030. (#4681)
* Revert "bpo-32030: _PyPathConfig_Init() sets home and program_name (#4673)"

This reverts commit af5a895073.

* Revert "bpo-32030: Fix config_get_program_name() on macOS (#4669)"

This reverts commit e23c06e2b0.

* Revert "bpo-32030: Add Python/pathconfig.c (#4668)"

This reverts commit 0ea395ae96.

* Revert "bpo-32030: Don't call _PyPathConfig_Fini() in Py_FinalizeEx() (#4667)"

This reverts commit ebac19dad6.

* Revert "bpo-32030: Fix Py_GetPath(): init program_name (#4665)"

This reverts commit 9ac3d88827.
2017-12-02 21:36:00 +02:00
Victor Stinner af5a895073
bpo-32030: _PyPathConfig_Init() sets home and program_name (#4673)
_PyPathConfig_Init() now also initialize home and program_name:

* Rename existing _PyPathConfig_Init() to _PyPathConfig_Calculate().
  Add a new _PyPathConfig_Init() function in pathconfig.c which
  handles the _Py_path_config variable and call
  _PyPathConfig_Calculate().
* Add home and program_name fields to _PyPathConfig.home
* _PyPathConfig_Init() now initialize home and program_name
  from main_config
* Py_SetProgramName(), Py_SetPythonHome() and Py_GetPythonHome() now
  calls Py_FatalError() on failure, instead of silently ignoring
  failures.
* config_init_home() now gets directly _Py_path_config.home to only
  get the value set by Py_SetPythonHome(), or NULL if
  Py_SetPythonHome() was not called.
* config_get_program_name() now gets directly
  _Py_path_config.program_name to only get the value set by
  Py_SetProgramName(), or NULL if Py_SetProgramName() was not called.
* pymain_init_python() doesn't call Py_SetProgramName() anymore,
  _PyPathConfig_Init() now always sets the program name
* Call _PyMainInterpreterConfig_Read() in
  pymain_parse_cmdline_envvars_impl() to control the memory allocator
* C API documentation: it's no more safe to call Py_GetProgramName()
  before Py_Initialize().
2017-12-02 10:11:32 +01:00
Victor Stinner 0ea395ae96
bpo-32030: Add Python/pathconfig.c (#4668)
* Factorize code from PC/getpathp.c and Modules/getpath.c to remove
  duplicated code
* rename pathconfig_clear() to _PyPathConfig_Clear()
* Inline _PyPathConfig_Fini() in pymain_impl() and then remove it,
  since it's a oneliner
2017-12-01 20:50:58 +01:00
Victor Stinner ebac19dad6
bpo-32030: Don't call _PyPathConfig_Fini() in Py_FinalizeEx() (#4667)
Changes:

* _PyPathConfig_Fini() cannot be called in Py_FinalizeEx().
  Py_Initialize() and Py_Finalize() can be called multiple times, but
  it must not "forget" parameters set by Py_SetProgramName(),
  Py_SetPath() or Py_SetPythonHome(), whereas _PyPathConfig_Fini()
  clear all these parameters.
* config_get_program_name() and calculate_program_full_path() now
  also decode paths using Py_DecodeLocale() to use the
  surrogateescape error handler, rather than decoding using
  mbstowcs() which is strict.
* Change _Py_CheckPython3() prototype: () => (void)
* Truncate a few lines which were too long
2017-12-01 20:09:52 +01:00
Victor Stinner b64de46aae
bpo-32030: Cleanup "path config" code (#4663)
* Rename PyPathConfig structure to _PyPathConfig and move it to
  Include/internal/pystate.h
* Rename path_config to _Py_path_config
* _PyPathConfig: Rename program_name field to program_full_path
* Add assert(str != NULL); to _PyMem_RawWcsdup(), _PyMem_RawStrdup()
  and _PyMem_Strdup().
* Rename calculate_path() to pathconfig_global_init(). The function
  now does nothing if it's already initiallized.
2017-12-01 18:27:09 +01:00
Victor Stinner 5d39e04290
bpo-32030: Rework memory allocators (#4625)
* Fix _PyMem_SetupAllocators("debug"): always restore allocators to
  the defaults, rather than only caling _PyMem_SetupDebugHooks().
* Add _PyMem_SetDefaultAllocator() helper to set the "default"
  allocator.
* Add _PyMem_GetAllocatorsName(): get the name of the allocators
* main() now uses debug hooks on memory allocators if Py_DEBUG is
  defined, rather than calling directly malloc()
* Document default memory allocators in C API documentation
* _Py_InitializeCore() now fails with a fatal user error if
  PYTHONMALLOC value is an unknown memory allocator, instead of
  failing with a fatal internal error.
* Add new tests on the PYTHONMALLOC environment variable
* Add support.with_pymalloc()
* Add the _testcapi.WITH_PYMALLOC constant and expose it as
   support.with_pymalloc().
* sysconfig.get_config_var('WITH_PYMALLOC') doesn't work on Windows, so
   replace it with support.with_pymalloc().
* pythoninfo: add _testcapi collector for pymem
2017-11-29 17:20:38 +01:00
Serhiy Storchaka 598ceae876
bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
Victor Stinner 9316ee4da2
bpo-32030: Add _PyPathConfig_Init() (#4551)
* Add _PyPathConfig_Init() and _PyPathConfig_Fini()
* Remove _Py_GetPathWithConfig()
* _PyPathConfig_Init() returns _PyInitError to allow to handle errors
  properly
* Add pathconfig_clear()
* Windows calculate_path_impl(): replace Py_FatalError() with
  _PyInitError
* Py_FinalizeEx() now calls _PyPathConfig_Fini() to release memory
* Fix _Py_InitializeMainInterpreter() regression: don't initialize
  path config if _disable_importlib is false
* PyPathConfig now uses dynamically allocated memory
2017-11-25 03:17:57 +01:00
Victor Stinner f04ebe2a4d
bpo-32030: Add _PyMainInterpreterConfig.program_name (#4548)
* Py_Main() now calls Py_SetProgramName() earlier to be able to get
  the program name in _PyMainInterpreterConfig_ReadEnv().
* Rename prog to program_name
* Rename progpath to program_name
2017-11-25 00:01:23 +01:00
Victor Stinner 46972b7bc3
bpo-32030: Add _PyMainInterpreterConfig_ReadEnv() (#4542)
Py_GetPath() and Py_Main() now call
_PyMainInterpreterConfig_ReadEnv() to share the same code to get
environment variables.

Changes:

* Add _PyMainInterpreterConfig_ReadEnv()
* Add _PyMainInterpreterConfig_Clear()
* Add _PyMem_RawWcsdup()
* _PyMainInterpreterConfig: rename pythonhome to home
* Rename _Py_ReadMainInterpreterConfig() to
  _PyMainInterpreterConfig_Read()
* Use _Py_INIT_USER_ERR(), instead of _Py_INIT_ERR(), for decoding
  errors: the user is able to fix the issue, it's not a bug in
  Python. Same change was made in _Py_INIT_NO_MEMORY().
* Remove _Py_GetPythonHomeWithConfig()
2017-11-24 22:55:40 +01:00
Victor Stinner da9c8c36ae
bpo-32125: Remove Py_UseClassExceptionsFlag flag (#4544)
This flag was deprecated and wasn't used anymore since Python 2.0.
2017-11-24 22:06:38 +01:00
Victor Stinner 9e87e7776f
bpo-32096: Remove obj and mem from _PyRuntime (#4532)
bpo-32096, bpo-30860:  Partially revert the commit
2ebc5ce42a8a9e047e790aefbf9a94811569b2b6:

* Move structures back from Include/internal/mem.h to
  Objects/obmalloc.c
* Remove _PyObject_Initialize() and _PyMem_Initialize()
* Remove Include/internal/pymalloc.h
* Add test_capi.test_pre_initialization_api():
   Make sure that it's possible to call Py_DecodeLocale(), and then call
   Py_SetProgramName() with the decoded string, before Py_Initialize().

PyMem_RawMalloc() and Py_DecodeLocale() can be called again before
_PyRuntimeState_Init().

Co-Authored-By: Eric Snow <ericsnowcurrently@gmail.com>
2017-11-24 12:09:24 +01:00
Victor Stinner 0327bde9da
bpo-32030: Rewrite calculate_path() (#4521)
* calculate_path() rewritten in Modules/getpath.c and PC/getpathp.c
* Move global variables into a new PyPathConfig structure.
* calculate_path():

  * Split the huge calculate_path() function into subfunctions.
  * Add PyCalculatePath structure to pass data between subfunctions.
  * Document PyCalculatePath fields.
  * Move cleanup code into a new calculate_free() subfunction
  * calculate_init() now handles Py_DecodeLocale() failures properly
  * calculate_path() is now atomic: only replace PyPathConfig
    (path_config) at once on success.

* _Py_GetPythonHomeWithConfig() now returns an error on failure
* Add _Py_INIT_NO_MEMORY() helper: report a memory allocation failure
* Coding style fixes (PEP 7)
2017-11-23 17:03:20 +01:00
xdegaye c06c22e9a9
bpo-29040: Support building Android with Unified Headers (GH-4492) 2017-11-23 11:44:38 +01:00
Victor Stinner 1f15111a6e
bpo-32030: Add _PyMainInterpreterConfig.pythonhome (#4513)
* Py_Main() now reads the PYTHONHOME environment variable
* Add _Py_GetPythonHomeWithConfig() private function
* Add _PyWarnings_InitWithConfig()
* init_filters() doesn't get the current core configuration from the
  current interpreter or Python thread anymore. Pass explicitly the
  configuration to _PyWarnings_InitWithConfig().
* _Py_InitializeCore() now fails on _PyWarnings_InitWithConfig()
  failure.
* Pass configuration as constant
2017-11-23 10:43:14 +01:00
Victor Stinner e32e79f7d8
bpo-32030: Move PYTHONPATH to _PyMainInterpreterConfig (#4511)
Move _PyCoreConfig.module_search_path_env to _PyMainInterpreterConfig
structure.
2017-11-23 01:49:45 +01:00
Victor Stinner d434110974
bpo-32030: Add _PyCoreConfig.module_search_path_env (#4504)
Changes:

* Py_Main() initializes _PyCoreConfig.module_search_path_env from
  the PYTHONPATH environment variable.
* PyInterpreterState_New() now initializes core_config and config
  fields
* Compute sys.path a little bit ealier in
  _Py_InitializeMainInterpreter() and new_interpreter()
* Add _Py_GetPathWithConfig() private function.
2017-11-23 00:12:09 +01:00
Victor Stinner 25420fe290
bpo-32030: Add more options to _PyCoreConfig (#4485)
Py_Main() now handles two more -X options:

* -X showrefcount: new _PyCoreConfig.show_ref_count field
* -X showalloccount: new _PyCoreConfig.show_alloc_count field
2017-11-20 18:12:22 -08:00
Victor Stinner 09f3a8a124
bpo-32089: Fix warnings filters in dev mode (#4482)
The developer mode (-X dev) now creates all default warnings filters
to order filters in the correct order to always show ResourceWarning
and make BytesWarning depend on the -b option.

Write a functional test to make sure that ResourceWarning is logged
twice at the same location in the developer mode.

Add a new 'dev_mode' field to _PyCoreConfig.
2017-11-20 17:32:40 -08:00
Victor Stinner a7368ac636
bpo-32030: Enhance Py_Main() (#4412)
Parse more env vars in Py_Main():

* Add more options to _PyCoreConfig:

  * faulthandler
  * tracemalloc
  * importtime

* Move code to parse environment variables from _Py_InitializeCore()
  to Py_Main(). This change fixes a regression from Python 3.6:
  PYTHONUNBUFFERED is now read before calling pymain_init_stdio().
* _PyFaulthandler_Init() and _PyTraceMalloc_Init() now take an
  argument to decide if the module has to be enabled at startup.
* tracemalloc_start() is now responsible to check the maximum number
  of frames.

Other changes:

* Cleanup Py_Main():

  * Rename some pymain_xxx() subfunctions
  * Add pymain_run_python() subfunction

* Cleanup Py_NewInterpreter()
* _PyInterpreterState_Enable() now reports failure
* init_hash_secret() now considers pyurandom() failure as an "user
  error": don't fail with abort().
* pymain_optlist_append() and pymain_strdup() now sets err on memory
  allocation failure.
2017-11-15 18:11:45 -08:00
Victor Stinner f7e5b56c37
bpo-32030: Split Py_Main() into subfunctions (#4399)
* Don't use "Python runtime" anymore to parse command line options or
  to get environment variables: pymain_init() is now a strict
  separation.
* Use an error message rather than "crashing" directly with
  Py_FatalError(). Limit the number of calls to Py_FatalError(). It
  prepares the code to handle errors more nicely later.
* Warnings options (-W, PYTHONWARNINGS) and "XOptions" (-X) are now
  only added to the sys module once Python core is properly
  initialized.
* _PyMain is now the well identified owner of some important strings
  like: warnings options, XOptions, and the "program name". The
  program name string is now properly freed at exit.
  pymain_free() is now responsible to free the "command" string.
* Rename most methods in Modules/main.c to use a "pymain_" prefix to
  avoid conflits and ease debug.
* Replace _Py_CommandLineDetails_INIT with memset(0)
* Reorder a lot of code to fix the initialization ordering. For
  example, initializing standard streams now comes before parsing
  PYTHONWARNINGS.
* Py_Main() now handles errors when adding warnings options and
  XOptions.
* Add _PyMem_GetDefaultRawAllocator() private function.
* Cleanup _PyMem_Initialize(): remove useless global constants: move
  them into _PyMem_Initialize().
* Call _PyRuntime_Initialize() as soon as possible:
  _PyRuntime_Initialize() now returns an error message on failure.
* Add _PyInitError structure and following macros:

  * _Py_INIT_OK()
  * _Py_INIT_ERR(msg)
  * _Py_INIT_USER_ERR(msg): "user" error, don't abort() in that case
  * _Py_INIT_FAILED(err)
2017-11-15 15:48:08 -08:00
Serhiy Storchaka 9b6c60cbce
bpo-31979: Simplify transforming decimals to ASCII (#4336)
in int(), float() and complex() parsers.

This also speeds up parsing non-ASCII numbers by around 20%.
2017-11-13 21:23:48 +02:00
xdegaye 1588be66d7
bpo-28180: Fix the implementation of PEP 538 on Android (GH-4334) 2017-11-12 12:45:59 +01:00
Ned Batchelder 01ae58d446 Correct the location of a function mentioned in a comment (GH-4327) 2017-11-09 19:55:34 +03:00
Victor Stinner 8c663fd60e
Replace KB unit with KiB (#4293)
kB (*kilo* byte) unit means 1000 bytes, whereas KiB ("kibibyte")
means 1024 bytes. KB was misused: replace kB or KB with KiB when
appropriate.

Same change for MB and GB which become MiB and GiB.

Change the output of Tools/iobench/iobench.py.

Round also the size of the documentation from 5.5 MB to 5 MiB.
2017-11-08 14:44:44 -08:00
luzpaz a5293b4ff2 Fix miscellaneous typos (#4275) 2017-11-05 15:37:50 +02:00
Anselm Kruis 9e33973332 bpo-31890: define METH_STACKLESS (#4159)
Add METH_STACKLESS to prevent future collisions.
2017-11-02 15:54:57 -07:00
Victor Stinner c29b585fd4
bpo-31784: Implement PEP 564: add time.time_ns() (#3989)
Add new time functions:

* time.clock_gettime_ns()
* time.clock_settime_ns()
* time.monotonic_ns()
* time.perf_counter_ns()
* time.process_time_ns()
* time.time_ns()

Add new _PyTime functions:

* _PyTime_FromTimespec()
* _PyTime_FromNanosecondsObject()
* _PyTime_FromTimeval()

Other changes:

* Add also os.times() tests to test_os.
* pytime_fromtimeval() and pytime_fromtimeval() now return
  _PyTime_MAX or _PyTime_MIN on overflow, rather than undefined
  behaviour
* _PyTime_FromNanoseconds() parameter type changes from long long to
  _PyTime_t
2017-11-02 07:28:27 -07:00
stratakis e8b1965639 bpo-23699: Use a macro to reduce boilerplate code in rich comparison functions (GH-793) 2017-11-02 20:32:54 +10:00
Masayuki Yamamoto 8bc7d63560 bpo-25720: Fix the method for checking pad state of curses WINDOW (#4164)
Modify the code to use ncurses is_pad() instead of checking WINDOW
_flags field.  If your platform does not provide the is_pad(), the
existing way that checks the field will be enabled.

Note: This change does not drop support for platforms where do not
have both WINDOW _flags field and is_pad().
2017-11-01 14:05:26 +02:00
Serhiy Storchaka baac01e629
bpo-31891: Fix building the curses module on NetBSD. (#4165) 2017-10-31 13:56:44 +02:00
Erik Bray 031c4bfadb bpo-31877: Add _Py_LegacyLocaleDetected and _PyCoerceLegacyLocale to pylifecycle.h (GH-4134)
Only declaring these as interns inside the CLI's main C module
caused build problems on some platforms (notably Cygwin), so
this switches them to a regular underscore prefixed "private" C
API declaration.
2017-10-27 19:46:03 +10:00
Serhiy Storchaka bdf4298ae2 Fix trailing whitespaces in C files. (#4130) 2017-10-26 16:59:40 +03:00
xdegaye 56d1f5ca32 bpo-30697: Fix PyErr_NormalizeException() when no memory (GH-2327) 2017-10-26 15:09:06 +02:00
pdox 1896793520 bpo-31857: Make the behavior of USE_STACKCHECK deterministic (#4098) 2017-10-25 23:03:01 -07:00
Victor Stinner 850a18e03e bpo-30768: Recompute timeout on interrupted lock (GH-4103)
Fix the pthread+semaphore implementation of
PyThread_acquire_lock_timed() when called with timeout > 0 and
intr_flag=0: recompute the timeout if sem_timedwait() is interrupted
by a signal (EINTR).

See also the PEP 475.

The pthread implementation of PyThread_acquire_lock() now fails with
a fatal error if the timeout is larger than PY_TIMEOUT_MAX, as done
in the Windows implementation.

The check prevents any risk of overflow in PyThread_acquire_lock().

Add also PY_DWORD_MAX constant.
2017-10-24 16:53:32 -07:00
Mark Shannon ae3087c638 Move exc state to generator. Fixes bpo-25612 (#1773)
Move exception state information from frame objects to coroutine (generator/thread) object where it belongs.
2017-10-22 23:41:51 +02:00
Pablo Galindo 2c15b29aea bpo-31786: Make functions in the select module blocking when timeout is a small negative value. (#4003) 2017-10-17 17:14:41 +03:00
Ned Deily 2bd37c227e post 3.7.0a2 development 2017-10-17 02:10:44 -04:00
Ned Deily f7ac4fe52a Bump to 3.7.0a2 2017-10-16 23:50:50 -04:00
brainfvck c75edabbb6 bpo-31558: Add gc.freeze() (#3705)
Freeze all the objects tracked by gc - move them to a permanent generation
and ignore all the future collections. This can be used before a POSIX
fork() call to make the gc copy-on-write friendly or to speed up collection.
2017-10-16 12:49:41 -07:00
Victor Stinner bdaeb7d237 bpo-31773: _PyTime_GetPerfCounter() uses _PyTime_t (GH-3983)
* Rewrite win_perf_counter() to only use integers internally.
* Add _PyTime_MulDiv() which compute "ticks * mul / div"
  in two parts (int part and remaining) to prevent integer overflow.
* Clock frequency is checked at initialization for integer overflow.
* Enhance also pymonotonic() to reduce the precision loss on macOS
  (mach_absolute_time() clock).
2017-10-16 08:44:31 -07:00
Victor Stinner cba9a0c6de bpo-31773: time.perf_counter() uses again double (GH-3964)
time.clock() and time.perf_counter() now use again C double
internally.

Remove also _PyTime_GetWinPerfCounterWithInfo(): use
_PyTime_GetPerfCounterDoubleWithInfo() instead on Windows.
2017-10-12 08:51:56 -07:00
Victor Stinner a997c7b434 bpo-31415: Add _PyTime_GetPerfCounter() and use it for -X importtime (#3936)
* Add _PyTime_GetPerfCounter()
* Use _PyTime_GetPerfCounter() for -X importtime
2017-10-10 02:51:50 -07:00
Masayuki Yamamoto 731e189014 bpo-25658: Implement PEP 539 for Thread Specific Storage (TSS) API (GH-1362)
See PEP 539 for details.

Highlights of changes:

- Add Thread Specific Storage (TSS) API
- Document the Thread Local Storage (TLS) API as deprecated
- Update code that used TLS API to use TSS API
2017-10-06 20:41:34 +10:00
Yury Selivanov faa135acbf bpo-31709: Drop support for asynchronous __aiter__. (#3903) 2017-10-06 02:08:57 -04:00
Jelle Zijlstra ac317700ce bpo-30406: Make async and await proper keywords (#1669)
Per PEP 492, 'async' and 'await' should become proper keywords in 3.7.
2017-10-05 23:24:46 -04:00
Christian Heimes 3d2b407da0 bpo-31574: importlib dtrace (#3749)
Importlib was instrumented with two dtrace probes to profile import timing.

Signed-off-by: Christian Heimes <christian@python.org>
2017-09-29 15:53:19 -07:00
Benjamin Peterson 288d1daada remove support for BSD/OS (closes bpo-31624) (#3812) 2017-09-28 22:44:27 -07:00
Ned Deily 453408a505 post 3.7.0a1 development 2017-09-19 03:12:46 -04:00
Ned Deily 8f51bb436f Bump to 3.6.0a1 2017-09-19 01:09:03 -04:00
Serhiy Storchaka 4ab46d7949 bpo-31497: Add private helper _PyType_Name(). (#3630)
This function returns the last component of tp_name after a dot.
Returns tp_name itself if it doesn't contain a dot.
2017-09-17 21:11:04 +03:00
Eric Snow 3f9eee6eb4 bpo-28411: Support other mappings in PyInterpreterState.modules. (#3593)
The concrete PyDict_* API is used to interact with PyInterpreterState.modules in a number of places. This isn't compatible with all dict subclasses, nor with other Mapping implementations. This patch switches the concrete API usage to the corresponding abstract API calls.

We also add a PyImport_GetModule() function (and some other helpers) to reduce a bunch of code duplication.
2017-09-15 16:35:20 -06:00
Serhiy Storchaka 5d84cb368c bpo-31464: asdl_c.py no longer emits trailing spaces in Python-ast.h. (#3568) 2017-09-14 20:28:22 -07:00
Barry Warsaw b2e5794870 bpo-31338 (#3374)
* Add Py_UNREACHABLE() as an alias to abort().
* Use Py_UNREACHABLE() instead of assert(0)
* Convert more unreachable code to use Py_UNREACHABLE()
* Document Py_UNREACHABLE() and a few other macros.
2017-09-14 18:13:16 -07:00
Eric Snow d393c1b227 bpo-28411: Isolate PyInterpreterState.modules (#3575)
A bunch of code currently uses PyInterpreterState.modules directly instead of PyImport_GetModuleDict(). This complicates efforts to make changes relative to sys.modules. This patch switches to using PyImport_GetModuleDict() uniformly. Also, a number of related uses of sys.modules are updated for uniformity for the same reason.

Note that this code was already reviewed and merged as part of #1638. I reverted that and am now splitting it up into more focused parts.
2017-09-14 12:18:12 -06:00
Eric Snow dae0276bb6 bpo-30860: Fix a refleak. (#3567)
Resolves bpo-31420.

(This was accidentally reverted when in #3565.)
2017-09-14 00:35:58 -07:00