Changes:
* Split _PySys_SetArgvWithError() into subfunctions for Py_Main():
* Create the Python list object
* Set sys.argv to the list
* Compute argv0
* Prepend argv0 to sys.path
* Add _PyPathConfig_ComputeArgv0()
* Remove _PySys_SetArgvWithError()
* Py_Main() now splits the code to compute sys.argv/path0 and the
code to update the sys module: add pymain_compute_argv()
subfunction.
bpo-29240, bpo-32030:
* Rename pymain_set_argv() to pymain_set_sys_argv()
* pymain_set_sys_argv() now creates of copy of argv and modify the
copy, rather than modifying pymain->argv
* Call pymain_set_sys_argv() earlier: before pymain_run_python(), but
after pymain_get_importer().
* Add _PySys_SetArgvWithError() to handle errors
* Add -X utf8 command line option, PYTHONUTF8 environment variable
and a new sys.flags.utf8_mode flag.
* If the LC_CTYPE locale is "C" at startup: enable automatically the
UTF-8 mode.
* Add _winapi.GetACP(). encodings._alias_mbcs() now calls
_winapi.GetACP() to get the ANSI code page
* locale.getpreferredencoding() now returns 'UTF-8' in the UTF-8
mode. As a side effect, open() now uses the UTF-8 encoding by
default in this mode.
* Py_DecodeLocale() and Py_EncodeLocale() now use the UTF-8 encoding
in the UTF-8 Mode.
* Update subprocess._args_from_interpreter_flags() to handle -X utf8
* Skip some tests relying on the current locale if the UTF-8 mode is
enabled.
* Add test_utf8mode.py.
* _Py_DecodeUTF8_surrogateescape() gets a new optional parameter to
return also the length (number of wide characters).
* pymain_get_global_config() and pymain_set_global_config() now
always copy flag values, rather than only copying if the new value
is greater than the old value.
* Py_Main() now starts by reading Py_xxx configuration variables to
only work on its own private structure, and then later writes back
the configuration into these variables.
* Replace Py_GETENV() with pymain_get_env_var() which ignores empty
variables.
* Add _PyCoreConfig.dump_refs
* Add _PyCoreConfig.malloc_stats
* _PyObject_DebugMallocStats() is now responsible to check if debug
hooks are installed. The function returns 1 if stats were written,
or 0 if the hooks are disabled. Mark _PyMem_PymallocEnabled() as
static.
* bpo-32101: Add sys.flags.dev_mode flag
Rename also the "Developer mode" to the "Development mode".
* bpo-32101: Add PYTHONDEVMODE environment variable
Mention it in the development chapiter.
* Don't use "Python runtime" anymore to parse command line options or
to get environment variables: pymain_init() is now a strict
separation.
* Use an error message rather than "crashing" directly with
Py_FatalError(). Limit the number of calls to Py_FatalError(). It
prepares the code to handle errors more nicely later.
* Warnings options (-W, PYTHONWARNINGS) and "XOptions" (-X) are now
only added to the sys module once Python core is properly
initialized.
* _PyMain is now the well identified owner of some important strings
like: warnings options, XOptions, and the "program name". The
program name string is now properly freed at exit.
pymain_free() is now responsible to free the "command" string.
* Rename most methods in Modules/main.c to use a "pymain_" prefix to
avoid conflits and ease debug.
* Replace _Py_CommandLineDetails_INIT with memset(0)
* Reorder a lot of code to fix the initialization ordering. For
example, initializing standard streams now comes before parsing
PYTHONWARNINGS.
* Py_Main() now handles errors when adding warnings options and
XOptions.
* Add _PyMem_GetDefaultRawAllocator() private function.
* Cleanup _PyMem_Initialize(): remove useless global constants: move
them into _PyMem_Initialize().
* Call _PyRuntime_Initialize() as soon as possible:
_PyRuntime_Initialize() now returns an error message on failure.
* Add _PyInitError structure and following macros:
* _Py_INIT_OK()
* _Py_INIT_ERR(msg)
* _Py_INIT_USER_ERR(msg): "user" error, don't abort() in that case
* _Py_INIT_FAILED(err)
The concrete PyDict_* API is used to interact with PyInterpreterState.modules in a number of places. This isn't compatible with all dict subclasses, nor with other Mapping implementations. This patch switches the concrete API usage to the corresponding abstract API calls.
We also add a PyImport_GetModule() function (and some other helpers) to reduce a bunch of code duplication.
A bunch of code currently uses PyInterpreterState.modules directly instead of PyImport_GetModuleDict(). This complicates efforts to make changes relative to sys.modules. This patch switches to using PyImport_GetModuleDict() uniformly. Also, a number of related uses of sys.modules are updated for uniformity for the same reason.
Note that this code was already reviewed and merged as part of #1638. I reverted that and am now splitting it up into more focused parts.
PR #1638, for bpo-28411, causes problems in some (very) edge cases. Until that gets sorted out, we're reverting the merge. PR #3506, a fix on top of #1638, is also getting reverted.
* Drop warnoptions from PyInterpreterState.
* Drop xoptions from PyInterpreterState.
* Don't set warnoptions and _xoptions again.
* Decref after adding to sys.__dict__.
* Drop an unused macro.
* Check sys.xoptions *before* we delete it.
* group the (stateful) runtime globals into various topical structs
* consolidate the topical structs under a single top-level _PyRuntimeState struct
* add a check-c-globals.py script that helps identify runtime globals
Other globals are excluded (see globals.txt and check-c-globals.py).
f_trace_lines: enable/disable line trace events
f_trace_opcodes: enable/disable opcode trace events
These are intended primarily for testing of the interpreter
itself, as they make it much easier to emulate signals
arriving at unfortunate times.
* group the (stateful) runtime globals into various topical structs
* consolidate the topical structs under a single top-level _PyRuntimeState struct
* add a check-c-globals.py script that helps identify runtime globals
Other globals are excluded (see globals.txt and check-c-globals.py).
`PYTHONFRAMEWORK` is defined in `Makefile` and it shoulnd't be used
in `pyconfig.h`.
`sysconfig.py --generate-posix-vars` reads config vars from Makefile
and `pyconfig.h`. Conflicting variables should be avoided.
Especially, string config variables in Makefile are unquoted, but
in `pyconfig.h` are keep quoted. So it should be private (starts with
underscore).
Fix a reference in subinterpreters, like test_callbacks_leak() of
test_atexit.
warnoptions is a list used to pass options from the command line to
the sys module constructor. Before this change, the list was shared
by multiple interpreter which is not the expected behaviour. Each
interpreter should have their own independent mutable world.
This change duplicates the list in each interpreter. So each
interpreter owns its own list, so each interpreter can clear its own
list.
PEP 432 specifies a number of large changes to interpreter startup code, including exposing a cleaner C-API. The major changes depend on a number of smaller changes. This patch includes all those smaller changes.
sys.version and the platform module python_build(),
python_branch(), and python_revision() functions now use
git information rather than hg when building from a repo.
Based on original patches by Brett Cannon and Steve Dower.
bltinmodule.c: Added in b744ba1 and no longer necessary since d64e8a7
posixmodule.c: Added in d1cd4d4 and no longer necessary since efb00c0
pythonrun.c: Added in 73d538b and no longer necessary since d600951
sysmodule.c: Added in 5467d4c and no longer necessary since a2c17c5
Issue #28915: Replace _PyObject_CallMethodId() with
_PyObject_CallMethodIdObjArgs() when the format string only use the format 'O'
for objects, like "(O)".
_PyObject_CallMethodIdObjArgs() avoids the code to parse a format string and
avoids the creation of a temporary tuple.
Replace
_PyObject_CallArg1(func, arg)
with
PyObject_CallFunctionObjArgs(func, arg, NULL)
Using the _PyObject_CallArg1() macro increases the usage of the C stack, which
was unexpected and unwanted. PyObject_CallFunctionObjArgs() doesn't have this
issue.
Issue #28858: The change b9c9691c72c5 introduced a regression. It seems like
_PyObject_CallArg1() uses more stack memory than
PyObject_CallFunctionObjArgs().
* PyObject_CallFunctionObjArgs(func, NULL) => _PyObject_CallNoArg(func)
* PyObject_CallFunctionObjArgs(func, arg, NULL) => _PyObject_CallArg1(func, arg)
PyObject_CallFunctionObjArgs() allocates 40 bytes on the C stack and requires
extra work to "parse" C arguments to build a C array of PyObject*.
_PyObject_CallNoArg() and _PyObject_CallArg1() are simpler and don't allocate
memory on the C stack.
This change is part of the fastcall project. The change on listsort() is
related to the issue #23507.
Issue #28799:
* Remove the PyEval_GetCallStats() function.
* Deprecate the untested and undocumented sys.callstats() function.
* Remove the CALL_PROFILE special build
Use the sys.setprofile() function, cProfile or profile module to profile
function calls.
plat-$(PLATFORM_TRIPLET).
Rename the config directory (LIBPL) from config-$(LDVERSION) to
config-$(LDVERSION)-$(PLATFORM_TRIPLET).
Install the platform specifc _sysconfigdata module into the platform
directory and rename it to include the ABIFLAGS.
Issue #26516:
* Add PYTHONMALLOC environment variable to set the Python memory
allocators and/or install debug hooks.
* PyMem_SetupDebugHooks() can now also be used on Python compiled in release
mode.
* The PYTHONMALLOCSTATS environment variable can now also be used on Python
compiled in release mode. It now has no effect if set to an empty string.
* In debug mode, debug hooks are now also installed on Python memory allocators
when Python is configured without pymalloc.
Issue #26154: Add a new private _PyThreadState_UncheckedGet() function which
gets the current thread state, but don't call Py_FatalError() if it is NULL.
Python 3.5.1 removed the _PyThreadState_Current symbol from the Python C API to
no more expose complex and private atomic types. Atomic types depends on the
compiler or can even depend on compiler options. The new function
_PyThreadState_UncheckedGet() allows to get the variable value without having
to care of the exact implementation of atomic types.
Changes:
* Replace direct usage of the _PyThreadState_Current variable with a call to
_PyThreadState_UncheckedGet().
* In pystate.c, replace direct usage of the _PyThreadState_Current variable
with the PyThreadState_GET() macro for readability.
* Document also PyThreadState_Get() in pystate.h
Issue #25274: sys.setrecursionlimit() now raises a RecursionError if the new
recursion limit is too low depending at the current recursion depth. Modify
also the "lower-water mark" formula to make it monotonic. This mark is used to
decide when the overflowed flag of the thread state is reset.
crash when a generator is created in a C thread that is destroyed while the
generator is still used. The issue was that a generator contains a frame, and
the frame kept a reference to the Python state of the destroyed C thread. The
crash occurs when a trace function is setup.
the size of the fullpath buffer, not PATH_MAX. fullpath is declared using
MAXPATHLEN or MAX_PATH depending on the OS, and PATH_MAX is not declared on
IRIX.
instead of creating temporary Unicode string objects
Add also more identifiers in pythonrun.c to avoid temporary Unicode string
objets for the interactive interpreter.
-I
Run Python in isolated mode. This also implies -E and -s. In isolated mode
sys.path contains neither the script’s directory nor the user’s
site-packages directory. All PYTHON* environment variables are ignored,
too. Further restrictions may be imposed to prevent the user from
injecting malicious code.
PyStructSequence_InitType() except that it has a return value (0 on success,
-1 on error).
* PyStructSequence_InitType2() now raises MemoryError on memory allocation failure
* Fix also some calls to PyDict_SetItemString(): handle error
defined in sysmodule.c instead of straight out of a Unicode object.
Thanks to Amaury Forgeot d'Arc for noticing the bug and Eric Snow for
writing the patch.
in order to make algorithmic complexity attacks on (e.g.) web apps much more complicated.
The environment variable PYTHONHASHSEED and the new command line flag -R control this
behavior.
in order to make algorithmic complexity attacks on (e.g.) web apps much more complicated.
The environment variable PYTHONHASHSEED and the new command line flag -R control this
behavior.
* Expose the build flags to Python as sys.abiflags
* Shared library libpythonX.Y<abiflags>.so
* python-config --abiflags
* Make two distutils tests that failed with --enable-shared (even before this
patch) succeed.
* Fix a few small style issues.
* _Py_fopen() and _Py_stat() come from Python/import.c
* (_Py)_wrealpath() comes from Python/sysmodule.c
* _Py_char2wchar(), _Py_wchar2char() and _Py_wfopen() come from Modules/main.c
* (_Py)_wstat(), (_Py)_wgetcwd(), _Py_wreadlink() come from Modules/getpath.c
Victor Stinner
Serhiy Storchaka
Mark Shannon
Barry Warsaw
Eric Snow
Nick Coghlan
Antoine Pitrou
INADA Naoki
Segev Finer
Ned Deily
Yen Chi Hsuan
Steve Dower
Benjamin Peterson
Yury Selivanov
Larry Hastings
doko@ubuntu.com
Stefan Krah
Yury Selivanov
Antoine Pitrou
Ned Deily
Christian Heimes
Ezio Melotti
Andrew Kuchling
Brett Cannon
David Malcolm
Vinay Sajip
Georg Brandl
Petri Lehtinen
Victor Stinner
Martin v. Löwis
Jesus Cea
Éric Araujo
Brett Cannon
Raymond Hettinger