Commit Graph

104195 Commits

Author SHA1 Message Date
Victor Stinner 0a8e57248b
bpo-36721: Add --embed option to python-config (GH-13500)
To embed Python into an application, a new --embed option must be
passed to "python3-config --libs --embed" to get "-lpython3.8" (link
the application to libpython). To support both 3.8 and older, try
"python3-config --libs --embed" first and fallback to "python3-config
--libs" (without --embed) if the previous command fails.

Add a pkg-config "python-3.8-embed" module to embed Python into an
application: "pkg-config python-3.8-embed --libs" includes
"-lpython3.8".  To support both 3.8 and older, try "pkg-config
python-X.Y-embed --libs" first and fallback to "pkg-config python-X.Y
--libs" (without --embed) if the previous command fails (replace
"X.Y" with the Python version).

On the other hand, "pkg-config python3.8 --libs" no longer contains
"-lpython3.8". C extensions must not be linked to libpython (except
on Android, case handled by the script); this change is backward
incompatible on purpose.

"make install" now also installs "python-3.8-embed.pc".
2019-05-23 03:30:23 +02:00
Batuhan Taşkaya ef5bb25e2d bpo-27737: Allow whitespace only headers encoding (#13478) 2019-05-22 21:13:16 -04:00
Alexey Izbyshev 6bc5917903 bpo-35091: Objects/listobject.c: Replace overflow checks in gallop fu… (GH-10202)
…nctions with asserts

The actual overflow can never happen because of the following:
* The size of a list can't be greater than PY_SSIZE_T_MAX / sizeof(PyObject*).
* The size of a pointer on all supported plaftorms is at least 4 bytes.
* ofs is positive and less than the list size at the beginning of each iteration.





https://bugs.python.org/issue35091
2019-05-22 17:01:08 -07:00
Victor Stinner df22c03b93
bpo-36829: PyErr_WriteUnraisable() normalizes exception (GH-13507)
PyErr_WriteUnraisable() now creates a traceback object if there is no
current traceback. Moreover, call PyErr_NormalizeException() and
PyException_SetTraceback() to normalize the exception value. Ignore
silently any error.
2019-05-23 01:00:58 +02:00
Victor Stinner 5edcf26358
bpo-36763: Rename private Python initialization functions (GH-13511)
* Rename private C functions:

  * _Py_Initialize_ReconfigureCore => pyinit_core_reconfigure
  * _Py_InitializeCore_impl => pyinit_core_config
  * _Py_InitializeCore = > pyinit_core
  * _Py_InitializeMainInterpreter => pyinit_main
  * init_python => pyinit_python

* Rename _testembed.c commands: add "test_" prefix.
2019-05-23 00:57:57 +02:00
Paul Monson cfb241bd29 bpo-36941: Project file fixups for Windows ARM64 (GH-13477) 2019-05-22 15:16:21 -07:00
Victor Stinner 022be02dcf
bpo-36763: Add _PyPreConfig._config_init (GH-13481)
* _PyPreConfig_GetGlobalConfig() and  _PyCoreConfig_GetGlobalConfig()
  now do nothing if the configuration was not initialized with
  _PyPreConfig_InitCompatConfig() and _PyCoreConfig_InitCompatConfig()
* Remove utf8_mode=-2 special case: use utf8_mode=-1 instead.
* Fix _PyPreConfig_InitPythonConfig():

  * isolated = 0 instead of -1
  * use_environment = 1 instead of -1

* Rename _PyConfig_INIT to  _PyConfig_INIT_COMPAT
* Rename _PyPreConfig_Init() to _PyPreConfig_InitCompatConfig()
* Rename _PyCoreConfig_Init() to _PyCoreConfig_InitCompatConfig()
* PyInterpreterState_New() now uses _PyCoreConfig_InitPythonConfig()
  as default configuration, but it's very quickly overriden anyway.
* _freeze_importlib.c uses _PyCoreConfig_SetString() to set
  program_name.
* Cleanup preconfig_init_utf8_mode(): cmdline is always non-NULL.
2019-05-22 23:58:50 +02:00
Victor Stinner e4d300e07c
bpo-36829: Add test.support.catch_unraisable_exception() (GH-13490)
* Copy test_exceptions.test_unraisable() to
  test_sys.UnraisableHookTest().
* Use catch_unraisable_exception() in test_coroutines,
  test_exceptions, test_generators.
2019-05-22 23:44:02 +02:00
Sean 904e34d4e6 bpo-24882: Let ThreadPoolExecutor reuse idle threads before creating new thread (#6375)
* Fixes issue 24882

* Add news file entry for change.

* Change test_concurrent_futures.ThreadPoolShutdownTest

Adjust the shutdown test so that, after submitting three jobs
to the executor, the test checks for less than three threads,
instead of looking for exactly three threads.

If idle threads are being recycled properly, then we should have
less than three threads.

* Switched idle count to semaphor, Updated tests

As suggested by reviewer tomMoral, swapped lock-protected counter
with a semaphore to track the number of unused threads.

Adjusted test_threads_terminate to wait for completiton of the
previous future before submitting a new one (and checking the
number of threads used).

Also added a new test to confirm the thread pool can be saturated.

* Updates tests as requested by pitrou.

* Correct minor whitespace error.

* Make test_saturation faster
2019-05-22 23:29:58 +02:00
Sam Martin 2a3a2ece50 bpo-33110: Catch errors raised when running add_done_callback on already completed futures (GH-13141)
Wrap the callback call within the `add_done_callback` function within concurrent.futures, in order to behave in an identical manner to callbacks added to a running future are triggered once it has completed.
2019-05-22 23:29:02 +02:00
Michael J. Sullivan d8a82e2897 bpo-36878: Only allow text after `# type: ignore` if first character ASCII (GH-13504)
This disallows things like `# type: ignoreé`, which seems wrong.

Also switch to using Py_ISALNUM for the alnum check, for consistency
with other code (and maybe correctness re: locale issues?).


https://bugs.python.org/issue36878
2019-05-22 13:43:36 -07:00
Victor Stinner 0c2b6a3943
bpo-35907, CVE-2019-9948: urllib rejects local_file:// scheme (GH-13474)
CVE-2019-9948: Avoid file reading as disallowing the unnecessary URL
scheme in URLopener().open() and URLopener().retrieve()
of urllib.request.

Co-Authored-By: SH <push0ebp@gmail.com>
2019-05-22 22:15:01 +02:00
Matthias Bussonnier 2ddbd21aec bpo-34616: Document top level async in whatsnew/3.8. (GH-13484)
https://bugs.python.org/issue34616
2019-05-22 12:07:45 -07:00
Jake Tesler b121f63155 bpo-36084: Add native thread ID (TID) to threading.Thread (GH-13463)
Add native thread ID (TID) to threading.Thread objects
(supported platforms: Windows, FreeBSD, Linux, macOS).
2019-05-22 17:43:16 +02:00
Jelle Zijlstra b3be407288 bpo-33482: fix codecs.StreamRecoder.writelines (GH-6779)
A very simple fix. I found this while writing typeshed stubs for StreamRecoder.


https://bugs.python.org/issue33482
2019-05-22 08:18:26 -07:00
Michael J. Sullivan 933e1509ec bpo-36878: Track extra text added to 'type: ignore' in the AST (GH-13479)
GH-13238 made extra text after a # type: ignore accepted by the parser.
This finishes the job and actually plumbs the extra text through the
parser and makes it available in the AST.
2019-05-22 15:54:20 +01:00
Paul Dagnelie 4c7a46eb3c bpo-36972: Add SupportsIndex (GH-13448)
In order to support typing checks calling hex(), oct() and bin() on user-defined classes, a SupportIndex protocol is required. The ability to check these at runtime would be good to add for completeness sake. This is pretty much just a copy of SupportsInt with the names tweaked.
2019-05-22 15:23:01 +01:00
Marcel Plch 33e71e01e9 bpo-31862: Port binascii to PEP 489 multiphase initialization (GH-4108) 2019-05-22 13:51:25 +02:00
Jeroen Demeyer 77aa396bb9 bpo-36907: fix refcount bug in _PyStack_UnpackDict() (GH-13381) 2019-05-22 13:09:35 +02:00
Jeroen Demeyer b892d3ea46 bpo-36994: add test for profiling method_descriptor with **kwargs (GH-13461)
It adds a missing testcase for bpo-34125. This is testing code which is
affected by PEP 590, so missing this test might accidentally break
CPython if we screw up with implementing PEP 590.
2019-05-22 22:05:02 +12:00
Victor Stinner ef9d9b6312
bpo-36829: Add sys.unraisablehook() (GH-13187)
Add new sys.unraisablehook() function which can be overridden to
control how "unraisable exceptions" are handled. It is called when an
exception has occurred but there is no way for Python to handle it.
For example, when a destructor raises an exception or during garbage
collection (gc.collect()).

Changes:

* Add an internal UnraisableHookArgs type used to pass arguments to
  sys.unraisablehook.
* Add _PyErr_WriteUnraisableDefaultHook().
* The default hook now ignores exception on writing the traceback.
* test_sys now uses unittest.main() to automatically discover tests:
  remove test_main().
* Add _PyErr_Init().
* Fix PyErr_WriteUnraisable(): hold a strong reference to sys.stderr
  while using it
2019-05-22 11:28:22 +02:00
Berker Peksag 2725cb01d7
bpo-36948: Fix test_urlopener_retrieve_file on Windows (GH-13476) 2019-05-22 02:00:35 +03:00
Yury Selivanov eca18aac7b bpo-34616: Fix code style and unbreak buildbots (GH-13473)
See also PR GH-13148.
2019-05-21 17:20:21 -04:00
Batuhan Taşkaya 7abf8c6081 bpo-25652: Fix __rmod__ of UserString (GH-13326)
The ``__rmod__`` method of ``collections.UserString`` class had a bug that made it unusable. 


https://bugs.python.org/issue25652
2019-05-21 13:27:36 -07:00
Matthias Bussonnier 565b4f1ac7 bpo-34616: Add PyCF_ALLOW_TOP_LEVEL_AWAIT to allow top-level await (GH-13148)
Co-Authored-By: Yury Selivanov <yury@magic.io>
2019-05-21 16:12:02 -04:00
Batuhan Taşkaya aa32a7e111 bpo-23378: Add an extend action to argparse (GH-13305)
Add an extend action to argparse


https://bugs.python.org/issue23378
2019-05-21 10:47:42 -07:00
Jörg Stucke d5c120f7eb bpo-36035: fix Path.rglob for broken links (GH-11988)
Links creating an infinite symlink loop would raise an exception.
2019-05-21 19:44:40 +02:00
Max Bernstein ccb7ca728e bpo-36929: Modify io/re tests to allow for missing mod name (#13392)
* bpo-36929: Modify io/re tests to allow for missing mod name

For a vanishingly small number of internal types, CPython sets the
tp_name slot to mod_name.type_name, either in the PyTypeObject or the
PyType_Spec. There are a few minor places where this surfaces:

* Custom repr functions for those types (some of which ignore the
  tp_name in favor of using a string literal, such as _io.TextIOWrapper)
* Pickling error messages

The test suite only tests the former. This commit modifies the test
suite to allow Python implementations to omit the module prefix.

https://bugs.python.org/issue36929
2019-05-21 10:09:21 -07:00
Chris Angelico ad098b6750 Annotate the unexplained assignment in exception unbinding (GH-11448) 2019-05-21 09:34:19 -04:00
Lihua Zhao 4fb1502189 bpo-36648: fix mmap issue for VxWorks (GH-12394)
The mmap module set MAP_SHARED flag when map anonymous memory, however VxWorks
only support MAP_PRIVATE when map anonymous memory, this commit clear MAP_SHARED
and set MAP_PRIVATE.
2019-05-21 12:50:14 +02:00
pxinwr f2d7ac7e5b bpo-31904: Add posix module support for VxWorks (GH-12118) 2019-05-21 12:46:37 +02:00
Victor Stinner d12e75734d
Revert "bpo-36084: Add native thread ID to threading.Thread objects (GH-11993)" (GH-13458)
This reverts commit 4959c33d25.
2019-05-21 12:44:57 +02:00
Erik Janssens 925af1d99b bpo-36965: Fix includes in main.c on Windows with non-MSC compilers (GH-13421)
Include windows.h rather than crtdbg.h to get STATUS_CONTROL_C_EXIT constant.
Moreover, include windows.h on Windows, not only when MSC is used.
2019-05-21 12:11:11 +02:00
Xtreak e7cb23bf20 Fix RuntimeWarning in unittest.mock asyncio example (GH-13449)
* This PR fixes the `RuntimeWarning` in `inspect.isawaitable(mock())` where `mock()` was not awaited.
* Fix typo in asynctest project.
2019-05-21 01:47:17 -07:00
Matthias Bussonnier d0ebf13e50 bpo-36932: use proper deprecation-removed directive (GH-13349)
.. And update some deprecation warnings with version numbers.




https://bugs.python.org/issue36932
2019-05-20 23:20:10 -07:00
Andre Delfino 3099ae4075 Remove workaround for defaults in namedtuple now that we have the defaults parameter (GH-13263) 2019-05-20 20:52:17 -04:00
Cheryl Sabella 6220c02e09
bpo-35563: Add reference links to warnings.rst (GH-11289) 2019-05-20 18:45:05 -04:00
Rémi Lapeyre bf457c7d82 bpo-36969: Make PDB args command display keyword only arguments (GH-13452) 2019-05-20 23:17:30 +01:00
Matthias Bussonnier 1a3faf9d97 bpo-36952: Remove the bufsize parameter in fileinput.input(). (GH-13400)
This parameter is marked as deprecated since 3.6 and for removal in 3.8.
It already had no effects.
2019-05-20 23:44:11 +03:00
Batuhan Taşkaya 4011d865d0 bpo-23896: Add a grammar where exec isn't a stmt (#13272)
https://bugs.python.org/issue23896
2019-05-20 13:27:10 -07:00
Thomas Moreau c09a9f56c0 bpo-36888: Add multiprocessing.parent_process() (GH-13247) 2019-05-20 21:37:05 +02:00
Batuhan Taşkaya 5ae1c84bcd bpo-36949: Implement __repr__ on WeakSet (GH-13415) 2019-05-20 10:01:07 -07:00
Lisa Roach 77b3b7701a
bpo-26467: Adds AsyncMock for asyncio Mock library support (GH-9296) 2019-05-20 09:19:53 -07:00
Victor Stinner 0f72147ce2
bpo-36763: Fix _PyRuntime.preconfig.coerce_c_locale (GH-13444)
_PyRuntime.preconfig.coerce_c_locale can now be used to
check if the C locale has been coerced.

* Fix _Py_LegacyLocaleDetected(): don't attempt to coerce the
  C locale if LC_ALL environment variable is set. Add 'warn'
  parameter: emit_stderr_warning_for_legacy_locale() must not
  the LC_ALL env var.
* _PyPreConfig_Write() sets coerce_c_locale to 0 if
  _Py_CoerceLegacyLocale() fails.
2019-05-20 17:16:38 +02:00
Geoff Shannon 522ccef869 bpo-22865: Expand on documentation for the pty.spawn function (GH-11980) 2019-05-20 17:06:16 +02:00
Andrew Svetlov 45a24b85f3
Pass _asyncio_internal=True into stream tests on windows (#13442) 2019-05-20 17:38:57 +03:00
Victor Stinner 425717fee1
bpo-36763: Fix encoding/locale tests in test_embed (GH-13443)
* Fix encoding and locale tests in test_embed.InitConfigTests.
* InitConfigTests now only computes EXPECTED_CONFIG once.
* Add tests for PYTHONWARNINGS and PYTHONPATH env vars
2019-05-20 16:38:48 +02:00
Niklas Fiekas 9932fd91e8 bpo-35721: Close socket pair if Popen in _UnixSubprocessTransport fails (GH-11553)
This slightly expands an existing test case `test_popen_error` to trigger a `ResourceWarning` and fixes it.


https://bugs.python.org/issue35721
2019-05-20 05:02:16 -07:00
Victor Stinner 6d1c46746e
bpo-36763: Fix Python preinitialization (GH-13432)
* Add _PyPreConfig.parse_argv
* Add _PyCoreConfig._config_init field and _PyCoreConfigInitEnum enum
  type
* Initialization functions: reject preconfig=NULL and config=NULL
* Add config parameter to _PyCoreConfig_DecodeLocaleErr(): pass
  config->argv to _Py_PreInitializeFromPyArgv(), to parse config
  command line arguments in preinitialization.
* Add config parameter to _PyCoreConfig_SetString(). It now
  preinitializes Python.
* _PyCoreConfig_SetPyArgv() now also preinitializes Python for wide
  argv
* Fix _Py_PreInitializeFromCoreConfig(): don't pass args to
  _Py_PreInitializeFromPyArgv() if config.parse_argv=0.
* Use "char * const *" and "wchar_t * const *" types for 'argv'
  parameters and _PyArgv.argv.
* Add unit test on preinitialization from argv.
* _PyPreConfig.allocator type becomes int
* Add _PyPreConfig_InitFromPreConfig() and
  _PyPreConfig_InitFromCoreConfig() helper functions
2019-05-20 11:02:00 +02:00
Terry Jan Reedy 6d965b39b7
bpo-36958: In IDLE, print exit message (GH-13435)
Print any argument other than None or int passed to SystemExit
or sys.exit().
2019-05-19 22:52:22 -04:00