* Add _testcapi.get_coreconfig() to get the _PyCoreConfig of the
interpreter
* test.pythoninfo now gets the core configuration using
_testcapi.get_coreconfig()
bpo-34485, bpo-34544: On some FreeBSD, nl_langinfo(CODESET) fails if
LC_ALL or LC_CTYPE is set to an invalid locale name. Replace
_Py_SetLocaleFromEnv(LC_CTYPE) with _Py_SetLocaleFromEnv(LC_ALL) to
initialize properly locales.
Partially revert commit 177d921c8c.
The *max* value is no longer treated as a special case in the main loop. Besides making the main loop simpler and branchless, this also lets us relax the input restriction of *vec* to contain only non-negative values.
Use the core configuration of the interpreter, rather
than using global configuration variables. For example, replace
Py_QuietFlag with core_config->quiet.
Add support for the "surrogatepass" error handler in
PyUnicode_DecodeFSDefault() and PyUnicode_EncodeFSDefault()
for the UTF-8 encoding.
Changes:
* _Py_DecodeUTF8Ex() and _Py_EncodeUTF8Ex() now support the
surrogatepass error handler (_Py_ERROR_SURROGATEPASS).
* _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx() now use
the _Py_error_handler enum instead of "int surrogateescape" to pass
the error handler. These functions now return -3 if the error
handler is unknown.
* Add unit tests on _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx()
in test_codecs.
* Rename get_error_handler() to _Py_GetErrorHandler() and expose it
as a private function.
* _freeze_importlib doesn't need config.filesystem_errors="strict"
workaround anymore.
Py_DecodeLocale() and Py_EncodeLocale() now use the UTF-8 encoding on
Windows if Py_LegacyWindowsFSEncodingFlag is zero.
pymain_read_conf() now sets Py_LegacyWindowsFSEncodingFlag in its
loop, but restore its value at exit.
_PyCoreConfig_Read() is now responsible to choose the filesystem
encoding and error handler. Using Py_Main(), the encoding is now
chosen even before calling Py_Initialize().
_PyCoreConfig.filesystem_encoding is now the reference, instead of
Py_FileSystemDefaultEncoding, for the Python filesystem encoding.
Changes:
* Add filesystem_encoding and filesystem_errors to _PyCoreConfig
* _PyCoreConfig_Read() now reads the locale encoding for the file
system encoding.
* PyUnicode_EncodeFSDefault() and PyUnicode_DecodeFSDefaultAndSize()
now use the interpreter configuration rather than
Py_FileSystemDefaultEncoding and Py_FileSystemDefaultEncodeErrors
global configuration variables.
* Add _Py_SetFileSystemEncoding() and _Py_ClearFileSystemEncoding()
private functions to only modify Py_FileSystemDefaultEncoding and
Py_FileSystemDefaultEncodeErrors in coreconfig.c.
* _Py_CoerceLegacyLocale() now takes an int rather than
_PyCoreConfig for the warning.
On Windows, the LC_CTYPE is now set to the user preferred locale at
startup: _Py_SetLocaleFromEnv(LC_CTYPE) is now called during the
Python initialization. Previously, the LC_CTYPE locale was "C" at
startup, but changed when calling setlocale(LC_CTYPE, "") or
setlocale(LC_ALL, "").
pymain_read_conf() now also calls _Py_SetLocaleFromEnv(LC_CTYPE) to
behave as _Py_InitializeCore(). Moreover, it doesn't save/restore the
LC_ALL anymore.
On Windows, standard streams like sys.stdout now always use
surrogateescape error handler by default (ignore the locale).
* _Py_InitializeCore() now sets the LC_CTYPE locale to the user
preferred locale before checking if the C locale should be coerced
or not in _PyCoreConfig_Read().
* Fix pymain_read_conf(): remember if the C locale has been coerced
when the configuration should be read again if the encoding has
changed.
Python now gets the locale encoding with C code to initialize the encoding
of standard streams like sys.stdout. Moreover, the encoding is now
initialized to the Python codec name to get a normalized encoding name and
to ensure that the codec is loaded. The change avoids importing
_bootlocale and _locale modules at startup by default.
When the PYTHONIOENCODING environment variable only contains an encoding,
the error handler is now is now set explicitly to "strict".
Rename also get_default_standard_stream_error_handler() to
get_stdio_errors().
Reduce the buffer to format the "cpXXX" string (Windows locale encoding).
The current C implementations **crash** if the input includes a surrogate
Unicode code point, which is not possible to encode in UTF-8.
Important notes:
1. It is possible to pass a non-UTF-8 string as a separator to the
`.isoformat()` methods.
2. The pure-Python `datetime.fromisoformat()` implementation accepts
strings with a surrogate as the separator.
In `datetime.fromisoformat()`, in the special case of non-UTF-8 separators,
this implementation will take a performance hit by making a copy of the
input string and replacing the separator with 'T'.
Co-authored-by: Alexey Izbyshev <izbyshev@ispras.ru>
Co-authored-by: Paul Ganssle <paul@ganssle.io>
Introduce a configure check for strsignal(3) which defines HAVE_STRSIGNAL for
signalmodule.c. Add some common signals on HP-UX. This change applies for
Windows and HP-UX.
bpo-34170, bpo-34207: pymain_read_conf() now sets Py_UTF8Mode to
config->utf8_mode. pymain_read_conf() calls indirectly
Py_DecodeLocale() and Py_EncodeLocale() which depend on Py_UTF8Mode.
os.readlink() now accepts path-like and bytes objects on Windows.
Previously, support for path-like and bytes objects was only
implemented on Unix.
This commit also merges Unix and Windows implementations of
os.readlink() in one function and adds basic unit tests to increase
test coverage of the function.
bpo-34170, bpo-34326: Fix pymain_run_file(): use
PyRun_AnyFileExFlags(closeit=1) instead of calling fclose(fp)
explicitly to close the input file before running the code.
* Inline pymain_run_file() and pymain_open_filename() into
pymain_run_filename()
* Created pymain_run_stdin() which is pymain_run_filename() with
filename=NULL
* Rename pymain_run_filename() to pymain_run_file()
bpo-31650, bpo-34170: Replace _Py_CheckHashBasedPycsMode with
_PyCoreConfig._check_hash_pycs_mode. Modify PyInit__imp() and
zipimport to get the parameter from the current interpreter core
configuration.
Remove Include/internal/import.h file.
* Add Include/coreconfig.h
* Move config_*() and _PyCoreConfig_*() functions from Modules/main.c
to a new Python/coreconfig.c file.
* Inline _Py_ReadHashSeed() into config_init_hash_seed()
* Move global configuration variables to coreconfig.c
* Fix integer overflow in os.readv(), os.writev(), os.preadv()
and os.pwritev() and in os.sendfile() with headers or trailers
arguments (on BSD-based OSes and MacOS).
* Fix sending the part of the file in os.sendfile() on MacOS.
Using the trailers argument could cause sending more bytes from
the input file than was specified.
Thanks Ned Deily for testing on 32-bit MacOS.
* help(hashlib) didn't work because of incorrect module name in blake2b and
blake2s classes.
* Constructors blake2*(), sha3_*(), shake_*() and keccak_*() incorrectly
accepted keyword argument "string" for binary data, but documented as
accepting the "data" keyword argument. Now this parameter is positional-only.
* Keyword-only parameters in blake2b() and blake2s() were not documented as
keyword-only.
* Default value for some parameters of blake2b() and blake2s() was None,
which is not acceptable value.
* The length argument for shake_*.digest() was wrapped out to 32 bits.
* The argument for shake_128.digest() and shake_128.hexdigest() was not
positional-only as intended.
* TypeError messages for incorrect arguments in all constructors sha3_*(),
shake_*() and keccak_*() incorrectly referred to sha3_224.
Also made the following enhancements:
* More accurately specified input and result types for strings, bytes and
bytes-like objects.
* Unified positional parameter names for update() and constructors.
* Improved formatting.
This list is never used: if detect_types is on, this list will be
replaced with another one before row_cast_map is used, if
detect_types is off, row_cast_map is not used at all.
* Config: Rename ignore_environment field to use_environment.
* _PyCoreConfig_Read(): if isolated is set, use_environment and
site_import are now always set to 0.
* Inline pymain_free_raw() into pymain_free()
* Move config_init_warnoptions() call into pymain_read_conf_impl()
* _PyCoreConfig_Read(): don't replace values if they are already set:
faulthandler, pycache_prefix, home.
* Inline cmdline_get_env_flags() into config_read_env_vars():
_PyCoreConfig_Read() now reads much more environment variables like
PYTHONVERBOSE.
* Allow to override faulthandler and allocator even if dev_mode=1.
PYTHONMALLOC is now the priority over PYTHONDEVMODE.
* Fix _PyCoreConfig_Copy(): copy also install_signal_handlers,
coerce_c_locale and coerce_c_locale_warn
* _PyCoreConfig.install_signal_handlers default is now 1: install
signals by default
* Fix also a compiler warning: don't define _PyPathConfig type twice.
On Windows, passing a negative value to local results in an OSError because localtime_s on Windows does not support negative timestamps. Unfortunately this means that fold detection for timestamps between 0 and max_fold_seconds will result in this OSError since we subtract max_fold_seconds from the timestamp to detect a fold. However, since we know there haven't been any folds in the interval [0, max_fold_seconds) in any timezone, we can hackily just forego fold detection for this time range on Windows.
* If _Py_InitializeCore() is called twice, the second call now copies
and apply (partially) the new configuration.
* Rename _Py_CommandLineDetails to _PyCmdline
* Move more code into pymain_init(). The core configuration created
by Py_Main() is new destroyed before running Python to reduce the
memory footprint.
* _Py_InitializeCore() now returns the created interpreter.
_Py_InitializeMainInterpreter() now expects an interpreter.
* Remove _Py_InitializeEx_Private(): _freeze_importlib now uses
_Py_InitializeFromConfig()
* _PyCoreConfig_InitPathConfig() now only computes the path
configuration if needed.
Add more fields to _PyCoreConfig:
* _check_hash_pycs_mode
* bytes_warning
* debug
* inspect
* interactive
* legacy_windows_fs_encoding
* legacy_windows_stdio
* optimization_level
* quiet
* unbuffered_stdio
* user_site_directory
* verbose
* write_bytecode
Changes:
* Remove pymain_get_global_config() and pymain_set_global_config()
which became useless. These functions have been replaced by
_PyCoreConfig_GetGlobalConfig() and
_PyCoreConfig_SetGlobalConfig().
* sys.flags.dont_write_bytecode value is now restricted to 1 even if
-B option is specified multiple times on the command line.
* PyThreadState_Clear() now uses the config from the current
interpreter rather than using global Py_VerboseFlag
* _PyCoreConfig: add isolated and site_import attributes
* Replace Py_IgnoreEnvironment with config->ignore_environment when
reading the current configuration
* _PyCoreConfig_Read() now sets ignore_environment, utf8_mode,
isolated and site_import from Py_IgnoreEnvironment, Py_UTF8Mode,
Py_IsolatedFlag and Py_NoSiteFlag
* _Py_InitializeCore() now sets Py_xxx flags from the configuration
* pymain_read_conf() now uses _PyCoreConfig_Copy() to save/restore
the configuration.
* Rename _disable_importlib of _PyCoreConfig to _install_importlib
* _PyCoreConfig_SetGlobalConfig() now also set
Py_HashRandomizationFlag
* Replace !Py_NoSiteFlag with core_config->site_import
* _PyCoreConfig_Read() no longer directly modifies Py_IsolatedFlag
and Py_NoSiteFlag global configuration flags. The function now
requires two pointers to integer, so these flags can be set later,
to avoid side effets in _PyCoreConfig_Read().
* pathconfig_global_init() now leaves Py_IsolatedFlag and
Py_NoSiteFlag unchanged.
* Fix pathconfig_global_init(): avoid computing the path
configuration twice, use _PyCoreConfig_SetPathConfig().
Rework _PyCoreConfig_Read() function which *reads* core configuration
to not *modify* the path configuration.
A new _PyCoreConfig_SetPathConfig() function now recreates the path
configuration from the core configuration. This function is now
called very late in _Py_InitializeCore(), just before calling
initimport().
Changes:
* Add _PyCoreConfig.dll_path
* Py_SetPath() now fails with a fatal python error on memory
allocation failure.
* Rename _PyPathConfig_Calculate() to _PyPathConfig_Calculate_impl()
* Replace _PyPathConfig_Init() with _PyPathConfig_Calculate(): the
function now requires a _PyPathConfig
* Add _PyPathConfig_SetGlobal() to set the _Py_path_config global
variable.
* Add _PyCoreConfig_InitPathConfig(): compute the path configuration
* Add _PyCoreConfig_SetPathConfig(): set path configuration from core
configuration
* Rename wstrlist_append() to _Py_wstrlist_append()
* _Py_wstrlist_append() now handles integer overflow.
Py_Main() can again be called after Py_Initialize(), as in Python
3.6. The new configuration is ignored, except of
_PyMainInterpreterConfig.argv which is used to update sys.argv.
bpo-32430: Rename Modules/Setup.dist to Modules/Setup
Remove the necessity to copy the former manually to the latter when updating the local source tree.
They can be exposed when some C API calls fail due to lack of
memory.
* Failed Py_BuildValue() could cause an assertion error in the
following TextIOWrapper.tell().
* input_chunk could be decrefed twice in TextIOWrapper.seek()
after failed Py_BuildValue().
* initvalue could leak in StringIO.__getstate__() after failed
PyDict_Copy().
* `flags` is indeed deprecated, but there is a validation on its value for
backwards compatibility reasons. This adds mention of this in the docs.
* The docs say that `sizehint` is deprecated and ignored, but it is still
used when `epoll_create1()` is unavailable. This adds mention of this in
the docs.
* `sizehint=-1` is acceptable again, and is replaced with `FD_SETSIZE-1`.
This is needed to have a default value available at the Python level,
since `FD_SETSIZE` is not exposed to Python. (see: bpo-31938)
* Reject `sizehint=0` since it is invalid to pass on to `epoll_create()`.
The relevant tests have also been updated.
* Fix also PyInit__gdbm() to catch errors.
* test.pythoninfo: add gdbm.version
* test_dbm_gnu now logs GDBM_VERSION when run in verbose mode.
* pythoninfo: rename function to collect_gdbm()
bpo-33671
* use memoryview() with size == file size on Windows, see https://github.com/python/cpython/pull/7160#discussion_r195405230
* release intermediate (sliced) memoryview immediately
* replace "OSX" occurrences with "macOS"
* add some unittests for copyfileobj()
In some development setups it is inconvenient or impossible to write bytecode
caches to the code tree, but the bytecode caches are still useful. The
PYTHONPYCACHEPREFIX environment variable allows specifying an alternate
location for cached bytecode files, within which a directory tree mirroring the code
tree will be created. This cache tree is then used (for both reading and writing)
instead of the local `__pycache__` subdirectory within each source directory.
Exposed at runtime as sys.pycache_prefix (defaulting to None), and can
be set from the CLI as "-X pycache_prefix=path".
Patch by Carl Meyer.
Fix "LC_ALL=C python3.7 -V": reset properly the command line parser
when the encoding changes after reading the Python configuration.
Fix pymain_read_conf(): use memset(0) to reset properly cmdline.
Hangul composition check boundaries are wrong for the second character
([0x1161, 0x1176) instead of [0x1161, 0x1176]) and third character ((0x11A7, 0x11C3)
instead of [0x11A7, 0x11C3]).
All the subinterpreter tests were disabled in gh-7513. This commit re-enables them, but leaves one bad test disabled. The test is partly causing problems because it makes assumptions about the availability of a high-level interpreters module (see PEP 554). So I'm disabling the test until such a high-level module is available.
* have shutil.copyfileobj use sendfile() if possible
* refactoring: use ctx manager
* add test with non-regular file obj
* emulate case where file size can't be determined
* reference _copyfileobj_sendfile directly
* add test for offset() at certain position
* add test for empty file
* add test for non regular file dst
* small refactoring
* leave copyfileobj() alone in order to not introduce any incompatibility
* minor refactoring
* remove old test
* update docstring
* update docstring; rename exception class
* detect platforms which only support file to socket zero copy
* don't run test on platforms where file-to-file zero copy is not supported
* use tempfiles
* reset verbosity
* add test for smaller chunks
* add big file size test
* add comment
* update doc
* update whatsnew doc
* update doc
* catch Exception
* remove unused import
* add test case for error on second sendfile() call
* turn docstring into comment
* add one more test
* update comment
* add Misc/NEWS entry
* get rid of COPY_BUFSIZE; it belongs to another PR
* update doc
* expose posix._fcopyfile() for OSX
* merge from linux branch
* merge from linux branch
* expose fcopyfile
* arg clinic for the win implementation
* convert path type to path_t
* expose CopyFileW
* fix windows tests
* release GIL
* minor refactoring
* update doc
* update comment
* update docstrings
* rename functions
* rename test classes
* update doc
* update doc
* update docstrings and comments
* avoid do import nt|posix modules if unnecessary
* set nt|posix modules to None if not available
* micro speedup
* update description
* add doc note
* use better wording in doc
* rename function using 'fastcopy' prefix instead of 'zerocopy'
* use :ref: in rst doc
* change wording in doc
* add test to make sure sendfile() doesn't get called aymore in case it doesn't support file to file copies
* move CopyFileW in _winapi and actually expose CopyFileExW instead
* fix line endings
* add tests for mode bits
* add docstring
* remove test file mode class; let's keep it for later when Istart addressing OSX fcopyfile() specific copies
* update doc to reflect new changes
* update doc
* adjust tests on win
* fix argument clinic error
* update doc
* OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags arg to python
* osx / copyfile: use path_t instead of char
* do not set dst name in the OSError exception in order to remain consistent with platforms which cannot do that (e.g. linux)
* add same file test
* add test for same file
* have osx copyfile() pre-emptively check if src and dst are the same, otherwise it will return immedialtey and src file content gets deleted
* turn PermissionError into appropriate SameFileError
* expose ERROR_SHARING_VIOLATION in order to raise more appropriate SameFileError
* honour follow_symlinks arg when using CopyFileEx
* update Misc/NEWS
* expose CreateDirectoryEx mock
* change C type
* CreateDirectoryExW actual implementation
* provide specific makedirs() implementation for win
* fix typo
* skeleton for SetNamedSecurityInfo
* get security info for src path
* finally set security attrs
* add unit tests
* mimick os.makedirs() behavior and raise if dst dir exists
* set 2 paths for OSError object
* set 2 paths for OSError object
* expand windows test
* in case of exception on os.sendfile() set filename and filename2 exception attributes
* set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX
* update doc
* do not use CreateDirectoryEx() in copytree() if source dir is a symlink (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation
* use bytearray() and readinto()
* use memoryview() with bytearray()
* refactoring + introduce a new _fastcopy_binfileobj() fun
* remove CopyFileEx and other C wrappers
* remove code related to CopyFileEx
* Recognize binary files in copyfileobj()
...and use fastest _fastcopy_binfileobj() when possible
* set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable
* use ctx manager for memoryview()
* update doc
* remove outdated doc
* remove last CopyFileEx remnants
* OSX - use fcopyfile(3) instead of copyfile(3)
...as an extra safety measure: in case src/dst are "exotic" files (non
regular or living on a network fs etc.) we better fail on open() instead
of copyfile(3) as we're not quite sure what's gonna happen in that
case.
* update doc
The documentation for CERT_NONE, CERT_OPTIONAL, and CERT_REQUIRED were
misleading and partly wrong. It fails to explain that OpenSSL behaves
differently in client and server mode. Also OpenSSL does validate the
cert chain everytime. With SSL_VERIFY_NONE a validation error is not
fatal in client mode and does not request a client cert in server mode.
Also discourage people from using CERT_OPTIONAL in client mode.
A datetime object d is aware if d.tzinfo is not None and
d.tzinfo.utcoffset(d) does not return None. If d.tzinfo is None,
or if d.tzinfo is not None but d.tzinfo.utcoffset(d) returns None,
d is naive.
This commit ensures that instances with non-None d.tzinfo, but
d.tzinfo.utcoffset(d) returning None are treated as naive.
In addition, C acceleration code will raise TypeError if
d.tzinfo.utcoffset(d) returns an object with the type other than
timedelta.
* Updated the documentation.
Assume that the term "naive" is defined elsewhere and remove the
not entirely correct clarification. Thanks, Tim.
This call became unneeded after the posix module was converted to the
Argument Clinic in 2f93635d34 and should
have been removed as part of that change.
Move the floor() call into fbound() to call floor() on a double
rather than an int. The change should enhance the rounding.
Document also (int)double rounding mode.
For bpo-32604 I added extra subinterpreter-related tests (see #6914), which caused a few buildbots to crash. This patch fixes the crash by ensuring that refcounts in channels are handled properly.
Fix a crash in Python initialization when parsing the command line
options.
Fix memcpy() size parameter: previously, we read one wchar_t after
the end of _PyOS_optarg. Moreover, don't copy the trailingg NUL
character: we write it manually anyway.
Thanks Christoph Gohlke for the bug report and the fix!
* Failure in adding to gc.garbage is no longer fatal.
* An exception in tp_clear() no longer lead to crash (though tp_clear() should not leave exceptions).
* Fixed a leak when the GC fails to add an object with __del__ into
the gc.garbage list.
* PyGC_Collect() can now be called when an exception is set and
preserves it.
* Fixed an undefined behavior with comparing a dead pointer with NULL.
Fix the following bugs in the C implementation:
* get_future_loop() silenced all exceptions raised when look up the get_loop
attribute, not just an AttributeError.
* enter_task() silenced all exceptions raised when look up the current task,
not just a KeyError.
* repr() was called for a borrowed link in enter_task() and task_step_impl().
* str() was used instead of repr() in formatting one error message (in
Python implementation too).
* There where few reference leaks in error cases.
The ssl module now contains OP_NO_RENEGOTIATION constant, available with
OpenSSL 1.1.0h or 1.1.1.
Note, OpenSSL 1.1.0h hasn't been released yet.
Signed-off-by: Christian Heimes <christian@python.org>
* Expose the sigset_t converter via private API _Py_Sigset_Converter().
* Use Argument Clinic for parsing sigset_t in signalmodule.c.
* Raise ValueError instead OverflowError for integers out of
the C long range.
Based on patch by Pablo Galindo Salgado.
Erik Janssens
Alexander Buchkovsky
Pablo Galindo
Victor Stinner
Raymond Hettinger
Thomas Herzog
Zackery Spytz
Sergey Fedoseev
Gregory P. Smith
Alexey Izbyshev
Paul Ganssle
Michael Osipov
Berker Peksag
Alex Grönholm
INADA Naoki
jdemeyer
Serhiy Storchaka
Stéphane Wirtel
Ammar Askar
ValeriyaSinevich
Antoine Pitrou
Benjamin Peterson
Tal Einat
Xiang Zhang
Giampaolo Rodola
Carl Meyer
Wonsup Yoon
Eric Snow
Ned Deily
Christian Heimes
Alexander Belopolsky
pkerling
Yury Selivanov
Zvezdan Petkovic
Alex Gaynor
Segev Finer
Oren Milman