Commit Graph

46018 Commits

Author SHA1 Message Date
Batuhan Taskaya 2135e10dc7
bpo-40663: Correctly handle annotations with subscripts in ast_unparse.c (GH-20156) 2020-05-18 19:23:48 +01:00
Irit Katriel e6578a226d
bpo-40662: Fixed ast.get_source_segment for ast nodes that have incomplete location information (GH-20157)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-05-18 19:14:12 +01:00
Lysandros Nikolaou 7b7a21bc4f
bpo-40661: Fix segfault when parsing invalid input (GH-20165)
Fix segfaults when parsing very complex invalid input, like `import äˆ ð£„¯ð¢·žð±‹á”€ð””ð‘©±å®ä±¬ð©¾\n𗶽`.

Co-authored-by: Guido van Rossum <guido@python.org>
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2020-05-18 18:32:03 +01:00
Serhiy Storchaka 08b47c367a
bpo-40257: Revert changes to inspect.getdoc() (GH-20073) 2020-05-18 20:25:07 +03:00
Nathaniel J. Smith 58205a0217
bpo-39148: fixup to account for IPV6_ENABLED being moved (GH-20170) 2020-05-18 00:56:47 -07:00
Kjell Braden 442634c42f
bpo-39148: enable ipv6 for datagrams in Proactor (GH-19121)
Ifdef is not necessary, as AF_INET6 is supported from Windows Vista, and other code in overlapped.c uses AF_INET6 and is not ifdef'd.
Change the raised exception so users are not fooled to think it comes from Windows API.

Automerge-Triggered-By: @njsmith
2020-05-17 23:21:30 -07:00
Chris Jerdonek da742ba826
bpo-31033: Improve the traceback for cancelled asyncio tasks (GH-19951)
When an asyncio.Task is cancelled, the exception traceback now
starts with where the task was first interrupted.  Previously,
the traceback only had "depth one."
2020-05-17 22:47:31 -07:00
Paul Ganssle e527ec8abe
bpo-40536: Add zoneinfo.available_timezones (GH-20158)
This was not specified in the PEP, but it will likely be a frequently requested feature if it's not included.

This includes only the "canonical" zones, not a simple listing of every valid value of `key` that can be passed to `Zoneinfo`, because it seems likely that that's what people will want.
2020-05-17 21:55:11 -04:00
Raymond Hettinger 9681953c99
bpo-39058: Preserve attribute order in argparse Namespace reprs. (GH-17621) 2020-05-17 18:53:01 -07:00
Christian Heimes 54f2898fe7
bpo-40645: Implement HMAC in C (GH-20129)
The internal module ``_hashlib`` wraps and exposes OpenSSL's HMAC API. The
new code will be used in Python 3.10 after the internal implementation
details of the pure Python HMAC module are no longer part of the public API.

The code is based on a patch by Petr Viktorin for RHEL and Python 3.6.

Co-Authored-By: Petr Viktorin <encukou@gmail.com>
2020-05-17 13:49:10 +02:00
Erlend Egeberg Aasland 9a45bfe6f4
bpo-35569: Expose RFC 3542 IPv6 socket options on macOS (GH-19526) 2020-05-17 02:32:46 -04:00
Chris Jerdonek d7184d3dbd
bpo-29587: Add another test for the gen.throw() fix. (GH-19859) 2020-05-16 21:14:48 -07:00
Pablo Galindo 6341fc7257
bpo-38870: Use subTest in test_unparse for better error reporting (GH-20141) 2020-05-17 03:53:57 +01:00
Pablo Galindo af8e5f84d9
Use subTest in test_exceptions for better error reporting (GH-20140) 2020-05-17 01:22:00 +01:00
Christian Heimes 837f9e42e3
bpo-40645: Deprecated internal details of hmac.HMAC (GH-20132) 2020-05-17 01:05:40 +02:00
Batuhan Taskaya dff92bb31f
bpo-38870: Implement round tripping support for typed AST in ast.unparse (GH-17797) 2020-05-17 00:04:12 +01:00
Batuhan Taskaya e966af7cff
bpo-38870: Correctly handle empty docstrings in ast.unparse (GH-18768)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-05-16 23:49:07 +01:00
Batuhan Taskaya d5a980a607
bpo-40165: Suppress stderr when checking if test_stty_match should be skipped (GH-19325) 2020-05-16 23:38:02 +01:00
Batuhan Taskaya 25160cdc47
bpo-38870: Don't put unnecessary parentheses on class declarations in ast.parse (GH-20134) 2020-05-16 22:53:25 +01:00
Batuhan Taskaya ce4a753dcb
bpo-38870: Do not separate factor prefixes in ast.unparse (GH-20133) 2020-05-16 22:46:11 +01:00
Christian Heimes d5b3f6b7f9
bpo-37630: Use SHA3 and SHAKE XOF from OpenSSL (GH-16049)
OpenSSL 1.1.1 comes with SHA3 and SHAKE builtin.

Signed-off-by: Christian Heimes <christian@python.org>

Automerge-Triggered-By: @tiran
2020-05-16 13:27:06 -07:00
Paul Ganssle 1b97b9b0ad
bpo-24416: Return named tuple from date.isocalendar() (GH-20113)
{date, datetime}.isocalendar() now return a private custom named tuple object
IsoCalendarDate rather than a simple tuple.

In order to leave IsocalendarDate as a private class and to improve what
backwards compatibility is offered for pickling the result of a
datetime.isocalendar() call, add a __reduce__ method to the named tuples that
reduces them to plain tuples. (This is the part of this PR most likely to cause
problems — if it causes major issues, switching to a strucseq or equivalent
would be prudent).

The pure python implementation of IsoCalendarDate uses positional-only
arguments, since it is private and only constructed by position anyway; the
equivalent change in the argument clinic on the C side would require us to move
the forward declaration of the type above the clinic import for whatever
reason, so it seems preferable to hold off on that for now.

bpo-24416: https://bugs.python.org/issue24416

Original PR by Dong-hee Na with only minor alterations by Paul Ganssle.

Co-authored-by: Dong-hee Na <donghee.na92@gmail.com>
2020-05-16 10:02:59 -04:00
Dong-hee Na aa92a7cf21
bpo-39305: Update nntplib to merge nntplib.NNTP and nntplib._NNTPBase (GH-19817) 2020-05-16 19:31:54 +09:00
Hai Shi 372fa3ead5
bpo-40275: lazy import modules in test.support (GH-20128)
Automerge-Triggered-By: @vstinner
2020-05-16 03:01:39 -07:00
Paul Ganssle 62972d9d73
bpo-40503: PEP 615: Tests and implementation for zoneinfo (GH-19909)
This is the initial implementation of PEP 615, the zoneinfo module,
ported from the standalone reference implementation (see
https://www.python.org/dev/peps/pep-0615/#reference-implementation for a
link, which has a more detailed commit history).

This includes (hopefully) all functional elements described in the PEP,
but documentation is found in a separate PR. This includes:

1. A pure python implementation of the ZoneInfo class
2. A C accelerated implementation of the ZoneInfo class
3. Tests with 100% branch coverage for the Python code (though C code
   coverage is less than 100%).
4. A compile-time configuration option on Linux (though not on Windows)

Differences from the reference implementation:

- The module is arranged slightly differently: the accelerated module is
  `_zoneinfo` rather than `zoneinfo._czoneinfo`, which also necessitates
  some changes in the test support function. (Suggested by Victor
  Stinner and Steve Dower.)
- The tests are arranged slightly differently and do not include the
  property tests. The tests live at test/test_zoneinfo/test_zoneinfo.py
  rather than test/test_zoneinfo.py or test/test_zoneinfo/__init__.py
  because we may do some refactoring in the future that would likely
  require this separation anyway; we may:
        - include the property tests
        - automatically run all the tests against both pure Python and C,
          rather than manually constructing C and Python test classes (similar
          to the way this works with test_datetime.py, which generates C
          and Python test cases from datetimetester.py).
- This includes a compile-time configuration option on Linux (though not
  on Windows); added with much help from Thomas Wouters.
- Integration into the CPython build system is obviously different from
  building a standalone zoneinfo module wheel.
- This includes configuration to install the tzdata package as part of
  CI, though only on the coverage jobs. Introducing a PyPI dependency as
  part of the CI build was controversial, and this is seen as less of a
  major change, since the coverage jobs already depend on pip and PyPI.

Additional changes that were introduced as part of this PR, most / all of
which were backported to the reference implementation:

- Fixed reference and memory leaks

    With much debugging help from Pablo Galindo

- Added smoke tests ensuring that the C and Python modules are built

    The import machinery can be somewhat fragile, and the "seamlessly falls
    back to pure Python" nature of this module makes it so that a problem
    building the C extension or a failure to import the pure Python version
    might easily go unnoticed.

- Adjustments to zoneinfo.__dir__

    Suggested by Petr Viktorin.

- Slight refactorings as suggested by Steve Dower.

- Removed unnecessary if check on std_abbr

    Discovered this because of a missing line in branch coverage.
2020-05-16 10:20:06 +02:00
Zackery Spytz 6b6092f533
bpo-39075: types.SimpleNamespace no longer sorts attributes in its repr (GH-19430) 2020-05-15 18:27:54 -07:00
Chris Jerdonek 1ce5841eca
bpo-31033: Add a msg argument to Future.cancel() and Task.cancel() (GH-19979) 2020-05-15 16:55:50 -07:00
Gregory P. Smith 6a5d3ff676
bpo-40636: Clarify the zip built-in docstring. (GH-20118)
Clarify the zip built-in docstring.

This puts much simpler text up front along with an example.

As it was, the zip built-in docstring was technically correct.  But too
technical for the reader who shouldn't _need_ to know about `__next__` and
`StopIteration` as most people do not need to understand the internal
implementation details of the iterator protocol in their daily life.

This is a documentation only change, intended to be backported to 3.8; it is
only tangentially related to PEP-618 which might offer new behavior options
in the future.

Wording based a bit more on enumerate per Brandt's suggestion.

This gets rid of the legacy wording paragraph which seems too tied to
implementation details of the iterator protocol which isn't relevant here.

Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>
2020-05-15 14:26:00 -07:00
romasku 382a5635bd
bpo-40607: Reraise exception during task cancelation in asyncio.wait_for() (GH-20054)
Currently, if asyncio.wait_for() timeout expires, it cancels
inner future and then always raises TimeoutError. In case
those future is task, it can handle cancelation mannually,
and those process can lead to some other exception. Current
implementation silently loses thoses exception.

To resolve this, wait_for will check was the cancelation
successfull or not. In case there was exception, wait_for
will reraise it.

Co-authored-by: Roman Skurikhin <roman.skurikhin@cruxlab.com>
2020-05-15 13:12:05 -07:00
Victor Stinner 6e57237faf
bpo-40055: test_distutils leaves warnings filters unchanged (GH-20095)
distutils.tests now saves/restores warnings filters to leave them
unchanged. Importing tests imports docutils which imports
pkg_resources which adds a warnings filter.
2020-05-15 18:06:23 +02:00
Guido van Rossum 15bc9ab301
bpo-40612: Fix SyntaxError edge cases in traceback formatting (GH-20072)
This fixes both the traceback.py module and the C code for formatting syntax errors (in Python/pythonrun.c). They now both consistently do the following:

- Suppress caret if it points left of text
- Allow caret pointing just past end of line
- If caret points past end of line, clip to *just* past end of line

The syntax error formatting code in traceback.py was mostly rewritten; small, subtle changes were applied to the C code in pythonrun.c.

There's still a difference when the text contains embedded newlines. Neither handles these very well, and I don't think the case occurs in practice.

Automerge-Triggered-By: @gvanrossum
2020-05-14 19:22:48 -07:00
Pablo Galindo 16ab07063c
bpo-40334: Correctly identify invalid target in assignment errors (GH-20076)
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
2020-05-15 02:04:52 +01:00
Joannah Nanjekye 7ba1f75f3f
bpo-38872: Document exec symbol for codeop.compile_command (GH-20047)
* Document exec symbol for codeop.compile_command

* Remove extra statements

Co-authored-by: nanjekyejoannah <joannah.nanjekye@ibm.com>
2020-05-14 21:59:46 -03:00
Victor Stinner 4b972faf60
bpo-40462: Fix typo in test_json (GH-20094) 2020-05-15 01:02:10 +02:00
Victor Stinner edf2643bbb
bpo-40460: Fix typo in idlelib/zzdummy.py (GH-20093)
Replace ztest with ztext.
2020-05-15 00:51:51 +02:00
Lysandros Nikolaou ce21cfca7b
bpo-40618: Disallow invalid targets in augassign and except clauses (GH-20083)
This commit fixes the new parser to disallow invalid targets in the
following scenarios:
- Augmented assignments must only accept a single target (Name,
  Attribute or Subscript), but no tuples or lists.
- `except` clauses should only accept a single `Name` as a target.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-05-14 21:13:50 +01:00
Pablo Galindo bcc3036095
bpo-40619: Correctly handle error lines in programs without file mode (GH-20090) 2020-05-14 21:11:48 +01:00
Victor Stinner f2c3b6823b
Revert "bpo-32604: [_xxsubinterpreters] Propagate exceptions. (GH-19768)" (GH-20089)
* Revert "bpo-40613: Remove compiler warning from _xxsubinterpretersmodule (GH-20069)"

This reverts commit fa0a66e62d.

* Revert "bpo-32604: [_xxsubinterpreters] Propagate exceptions. (GH-19768)"

This reverts commit a1d9e0accd.
2020-05-14 18:46:24 +02:00
Lumír 'Frenzy' Balhar e77d428856
bpo-40495: compileall option to hardlink duplicate pyc files (GH-19901)
compileall is now able to use hardlinks to prevent duplicates in a
case when .pyc files for different optimization levels have the same content.

Co-authored-by: Miro Hrončok <miro@hroncok.cz>
Co-authored-by: Victor Stinner <vstinner@python.org>
2020-05-14 16:17:22 +02:00
Hai Shi 7443d42021
bpo-40275: Import locale module lazily in gettext (GH-19905) 2020-05-14 03:22:30 +02:00
Arkadiusz Hiler 6f2f475d5a
bpo-40597: email: Use CTE if lines are longer than max_line_length consistently (gh-20038)
raw_data_manager (default for EmailPolicy, EmailMessage)
does correct wrapping of 'text' parts as long as the message contains
characters outside of 7bit US-ASCII set: base64 or qp
Content-Transfer-Encoding is applied if the lines would be too long
without it.  It did not, however, do this for ascii-only text,
which could result in lines that were longer than
policy.max_line_length or even the rfc 998  maximum.

This changeset fixes the heuristic so that if lines are longer than
policy.max_line_length, it will always apply a
content-transfer-encoding so that the lines are wrapped correctly.
2020-05-13 20:53:26 -04:00
Chris Jerdonek 75cd8e48c6
bpo-29587: Make gen.throw() chain exceptions with yield from (GH-19858)
The previous commits on bpo-29587 got exception chaining working
with gen.throw() in the `yield` case. This patch also gets the
`yield from` case working.

As a consequence, implicit exception chaining now also works in
the asyncio scenario of awaiting on a task when an exception is
already active.

Tests are included for both the asyncio case and the pure
generator-only case.
2020-05-13 16:18:27 -07:00
Michael Graczyk d72ea60521
issue-25872: Fix KeyError using linecache from multiple threads (GH-18007)
The crash that this fixes occurs when using traceback and other modules from multiple threads; 
del cache[filename] can raise a KeyError.
2020-05-13 18:41:57 -04:00
Lysandros Nikolaou a15c9b3a05
bpo-40334: Always show the caret on SyntaxErrors (GH-20050)
This commit fixes SyntaxError locations when the caret is not displayed,
by doing the following:

- `col_number` always gets set to the location of the offending
  node/expr. When no caret is to be displayed, this gets achieved
  by setting the object holding the error line to None.

- Introduce a new function `_PyPegen_raise_error_known_location`,
  which can be called, when an arbitrary `lineno`/`col_offset`
  needs to be passed. This function then gets used in the grammar
  (through some new macros and inline functions) so that SyntaxError
  locations of the new parser match that of the old.
2020-05-13 20:36:27 +01:00
Tzanetos Balitsaris b809717c1e
bpo-40331: Increase test coverage for the statistics module (GH-19608) 2020-05-13 13:29:31 +03:00
Steve Dower d6b727e2c9
bpo-40501: Replace ctypes code in uuid with native module (GH-19948) 2020-05-12 23:32:32 +01:00
Serhiy Storchaka 5650e76f63
bpo-40596: Fix str.isidentifier() for non-canonicalized strings containing non-BMP characters on Windows. (GH-20053) 2020-05-12 16:18:00 +03:00
Serhiy Storchaka 74ea6b5a75
bpo-40593: Improve syntax errors for invalid characters in source code. (GH-20033) 2020-05-12 12:42:04 +03:00
Batuhan Taskaya f3a5b7ada0
bpo-39481: remove generic classes from ipaddress/mmap (GH-20045)
These were added by mistake (see https://bugs.python.org/issue39481#msg366288).
2020-05-11 19:32:40 -07:00
Tim Peters b1b4c790e7
bpo-40480: restore ability to join fnmatch.translate() results (GH-20049)
In translate(), generate unique group names across calls.

The restores the undocumented ability to get a valid regexp
by joining multiple translate() results via `|`.
2020-05-11 21:19:20 -05:00
Raymond Hettinger 21cdb711e3
bpo-40571: Make lru_cache(maxsize=None) more discoverable (GH-20019) 2020-05-11 17:00:53 -07:00
Shantanu 27c0d9b54a
bpo-40334: produce specialized errors for invalid del targets (GH-19911) 2020-05-11 14:53:58 -07:00
Brad Solomon ef7973a981
bpo-40561: Add docstrings for webbrowser open functions (GH-19999)
Co-authored-by: Brad Solomon <brsolomon@deloitte.com>
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
2020-05-11 14:50:11 -04:00
Pablo Galindo 5b956ca42d
bpo-40585: Normalize errors messages in codeop when comparing them (GH-20030)
With the new parser, the error message contains always the trailing
newlines, causing the comparison of the repr of the error messages
in codeop to fail. This commit makes the new parser mirror the old parser's
behaviour regarding trailing newlines.
2020-05-11 01:41:26 +01:00
Serhiy Storchaka 2fbc57af85
bpo-40257: Tweak docstrings for special generic aliases. (GH-20022)
* Add the terminating period.
* Omit module name for builtin types.
2020-05-10 15:14:27 +03:00
Serhiy Storchaka 0122d48681
bpo-40397: Fix subscription of nested generic alias without parameters. (GH-20021) 2020-05-10 13:39:40 +03:00
Serhiy Storchaka fcb285609a
bpo-40397: Remove __args__ and __parameters__ from _SpecialGenericAlias (GH-19984) 2020-05-10 11:53:16 +03:00
Pablo Galindo ac7a92cc0a
bpo-40334: Avoid collisions between parser variables and grammar variables (GH-19987)
This is for the C generator:
- Disallow rule and variable names starting with `_`
- Rename most local variable names generated by the parser to start with `_`

Exceptions:
- Renaming `p` to `_p` will be a separate PR
- There are still some names that might clash, e.g.
  - anything starting with `Py`
  - C reserved words (`if` etc.)
  - Macros like `EXTRA` and `CHECK`
2020-05-09 21:34:50 -07:00
Jason R. Coombs 2c3d508c5f
bpo-40570: Improve compatibility of uname_result with late-bound .platform (#20015)
* bpo-40570: Improve compatibility of uname_result with late-bound .platform.

* Add test capturing ability to cast uname to a tuple.
2020-05-09 10:12:41 -04:00
Jason R. Coombs 7f7e706d78
bpo-39791: Add files() to importlib.resources (GH-19722)
* bpo-39791: Update importlib.resources to support files() API (importlib_resources 1.5).

* 📜🤖 Added by blurb_it.

* Add some documentation about the new objects added.

Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
2020-05-08 19:20:26 -04:00
Raymond Hettinger 81a5fc38e8
bpo-40541: Add optional *counts* parameter to random.sample() (GH-19970) 2020-05-08 07:53:15 -07:00
Zackery Spytz 02fa0ea9c1
bpo-40273: Reversible mappingproxy (FH-19513) 2020-05-07 22:25:50 -07:00
Pablo Galindo db9163ceef
bpo-40555: Check for p->error_indicator in loop rules after the main loop is done (GH-19986) 2020-05-08 03:38:44 +01:00
Eric Snow a1d9e0accd
bpo-32604: [_xxsubinterpreters] Propagate exceptions. (GH-19768)
(Note: PEP 554 is not accepted and the implementation in the code base is a private one for use in the test suite.)

If code running in a subinterpreter raises an uncaught exception then the "run" call in the calling interpreter fails. A RunFailedError is raised there that summarizes the original exception as a string. The actual exception type, __cause__, __context__, state, etc. are all discarded. This turned out to be functionally insufficient in practice. There is a more helpful solution (and PEP 554 has been updated appropriately).

This change adds the exception propagation behavior described in PEP 554 to the _xxsubinterpreters module. With this change a copy of the original exception is set to __cause__ on the RunFailedError. For now we are using "pickle", which preserves the exception's state. We also preserve the original __cause__, __context__, and __traceback__ (since "pickle" does not preserve those).

https://bugs.python.org/issue32604
2020-05-07 08:56:01 -06:00
Petr Viktorin e1becf46b4
bpo-38787: C API for module state access from extension methods (PEP 573) (GH-19936)
Module C state is now accessible from C-defined heap type methods (PEP 573).
Patch by Marcel Plch and Petr Viktorin.

Co-authored-by: Marcel Plch <mplch@redhat.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
2020-05-07 15:39:59 +02:00
Lysandros Nikolaou 4638c64295
bpo-40334: Error message for invalid default args in function call (GH-19973)
When parsing something like `f(g()=2)`, where the name of a default arg
is not a NAME, but an arbitrary expression, a specialised error message
is emitted.
2020-05-07 11:44:06 +01:00
Lysandros Nikolaou 2f37c355ab
bpo-40334: Fix error location upon parsing an invalid string literal (GH-19962)
When parsing a string with an invalid escape, the old parser used to
point to the beginning of the invalid string. This commit changes the new
parser to match that behaviour, since it's currently pointing to the
end of the string (or to be more precise, to the beginning of the next
token).
2020-05-07 11:37:51 +01:00
Serhiy Storchaka c1c7d8ead9
bpo-40397: Refactor typing._GenericAlias (GH-19719)
Make the design more object-oriented.
Split _GenericAlias on two almost independent classes: for special
generic aliases like List and for parametrized generic aliases like List[int].
Add specialized subclasses for Callable, Callable[...], Tuple and Union[...].
2020-05-07 04:09:33 +03:00
Pablo Galindo 99db2a1db7
bpo-40334: Allow trailing comma in parenthesised context managers (GH-19964) 2020-05-06 22:54:34 +01:00
Batuhan Taskaya 091951a67c
bpo-40528: Improve and clear several aspects of the ASDL definition code for the AST (GH-19952) 2020-05-06 15:29:32 +01:00
Victor Stinner 2668a9a5aa
bpo-40527: Fix command line argument parsing (GH-19955) 2020-05-06 22:22:17 +09:00
Tim Peters b9c46a2c2d
bpo-40480 "fnmatch" exponential execution time (GH-19908)
bpo-40480:  create different regexps in the presence of multiple `*`
patterns to prevent fnmatch() from taking exponential time.
2020-05-05 21:28:24 -05:00
Pablo Galindo 96074de573
bpo-40523: Add pass-throughs for hash() and reversed() to weakref.proxy objects (GH-19946) 2020-05-05 22:58:19 +01:00
Dennis Sweeney 1253c3ef70
bpo-40504: Allow weakrefs to lru_cache objects (GH-19938) 2020-05-05 14:14:32 -07:00
Curtis Bucher c21c51235a
bpo-40355: Improve error messages in ast.literal_eval with malformed Dict nodes (GH-19868)
Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-05-05 20:40:56 +01:00
Victor Stinner b0be6b3b94
bpo-29587: _PyErr_ChainExceptions() checks exception (GH-19902)
_PyErr_ChainExceptions() now ensures that the first parameter is an
exception type, as done by _PyErr_SetObject().

* The following function now check PyExceptionInstance_Check() in an
  assertion using a new _PyBaseExceptionObject_cast() helper
  function:

  * PyException_GetTraceback(), PyException_SetTraceback()
  * PyException_GetCause(), PyException_SetCause()
  * PyException_GetContext(), PyException_SetContext()

* PyExceptionClass_Name() now checks PyExceptionClass_Check() with an
  assertion.

* Remove XXX comment and add gi_exc_state variable to _gen_throw().

* Remove comment from test_generators
2020-05-05 17:07:41 +02:00
Dennis Sweeney 1e7e4519a8
bpo-40459: Fix NameError in platform.py (GH-19855) 2020-05-05 11:33:17 +09:00
Hai Shi 975408c065
bpo-40275: test.support imports lazily locale import (GH-19761) 2020-05-04 20:05:02 +02:00
Dong-hee Na 785f5e6d67
bpo-40489: Add test case for dict contain use after free (GH-19906) 2020-05-05 02:30:42 +09:00
Hai Shi caa3ef284a
bpo-40135: Fix multiprocessing test_shared_memory_across_processes() (GH-19892)
Don't define shared memory block's name in test_shared_memory_across_processes():
use SharedMemory(create=True) instead.
2020-05-04 17:05:54 +02:00
Lysandros Nikolaou 846d8b28ab
bpo-40246: Revert reporting of invalid string prefixes (GH-19888)
Due to backwards compatibility concerns regarding keywords immediately followed by a string without whitespace between them (like in `bg="#d00" if clear else"#fca"`) will fail to parse,
commit 41d5b94af4 has to be reverted.
2020-05-04 12:32:18 +01:00
Lysandros Nikolaou e10e7c771b
bpo-40334: Spacialized error message for invalid args after bare '*' (GH-19865)
When parsing things like `def f(*): pass` the old parser used to output `SyntaxError: named arguments must follow bare *`, which the new parser wasn't able to do.
2020-05-04 11:58:31 +01:00
Serhiy Storchaka 41a64587a0
bpo-40408: Fix support of nested type variables in GenericAlias. (GH-19836) 2020-05-04 10:56:05 +03:00
Shantanu 603d354626
bpo-40493: fix function type comment parsing (GH-19894)
The grammar for func_type_input rejected things like `(*t1) ->t2`. This fixes that.

Automerge-Triggered-By: @gvanrossum
2020-05-03 22:08:14 -07:00
Batuhan Taskaya 493bf1cc31
bpo-38870: Don't start generated output with newlines in ast.unparse (GH-19636) 2020-05-03 18:11:51 +01:00
Chris Jerdonek 21893fbb74
bpo-29587: allow chaining NULL exceptions in _gen_throw() (GH-19877)
This is a follow-up to GH-19823 that removes the check that the
exception value isn't NULL, prior to calling _PyErr_ChainExceptions().
This enables implicit exception chaining for gen.throw() in more
circumstances.

The commit also adds a test that a particular code snippet involving
gen.throw() doesn't crash.  The test shows why the new
`gi_exc_state.exc_type != Py_None` check that was added is necessary.
Without the new check, the code snippet (as well as a number of other
tests) crashes on certain platforms (e.g. Fedora but not Mac).
2020-05-03 00:07:57 -07:00
Raymond Hettinger 0400a7f2f8
Minor code cleanups for statistics (GH-19873)
* Minor cleanups:  Removed unused code.  Move C import near its Python version.

* Clean-up whitespace
2020-05-02 19:30:24 -07:00
Raymond Hettinger 190fac99c5
bpo-40465: Deprecate the optional argument to random.shuffle(). (#19867) 2020-05-02 16:45:32 -07:00
Sander 766352320f
bpo-40419: timeit CLI docs now mention 1,2,5,10,... trials instead of powers of 10 (GH-19752) 2020-05-02 19:12:05 +03:00
Serhiy Storchaka 531d1e5412
bpo-39435: Make the first argument of pickle.loads() positional-only. (GH-19846)
It was positional-only de facto: documentation and two implementations
used three different name.
2020-05-02 09:38:01 +03:00
Pablo Galindo 9dbaa8d9f0
Fix some scripts in the peg generator folder (GH-19853) 2020-05-02 05:23:06 +01:00
Chris Jerdonek 02047265eb
bpo-29587: Update gen.throw() to chain exceptions (#19823)
Before this commit, if an exception was active inside a generator
when calling gen.throw(), that exception was lost (i.e. there was
no implicit exception chaining).  This commit fixes that by
setting exc.__context__ when calling gen.throw(exc).
2020-05-01 18:14:19 -07:00
Raymond Hettinger 4168f1e460
Simplify choice()'s interaction with the private _randbelow() method (GH-19831) 2020-05-01 10:34:19 -07:00
Pablo Galindo ea7297cf8f
bpo-40334: unskip test_function_type in test_unparse with the new parser (GH-19837) 2020-05-01 08:02:06 -07:00
Batuhan Taskaya 76c1b4d5c5
bpo-40334: Improve column offsets for thrown syntax errors by Pegen (GH-19782) 2020-05-01 14:13:43 +01:00
Furkan Önder 719e14d283
bpo-40462: fix variable and function names (GH-19832)
Automerge-Triggered-By: @vstinner
2020-05-01 05:49:35 -07:00
Victor Stinner 252346acd9
bpo-40453: Add PyConfig._isolated_subinterpreter (GH-19820)
An isolated subinterpreter cannot spawn threads, spawn a child
process or call os.fork().

* Add private _Py_NewInterpreter(isolated_subinterpreter) function.
* Add isolated=True keyword-only parameter to
  _xxsubinterpreters.create().
* Allow again os.fork() in "non-isolated" subinterpreters.
2020-05-01 11:33:44 +02:00
Noah Doersing 8bcfd31cc0
Change 'exception happened' to 'exception occurred' in two places (#19767) 2020-04-30 21:30:10 -07:00
Lysandros Nikolaou 3e0a6f37df
bpo-40334: Add support for feature_version in new PEG parser (GH-19827)
`ast.parse` and `compile` support a `feature_version` parameter that
tells the parser to parse the input string, as if it were written in
an older Python version.
The `feature_version` is propagated to the tokenizer, which uses it
to handle the three different stages of support for `async` and
`await`. Additionally, it disallows the following at parser level:
- The '@' operator in < 3.5
- Async functions in < 3.5
- Async comprehensions in < 3.6
- Underscores in numeric literals in < 3.6
- Await expression in < 3.5
- Variable annotations in < 3.6
- Async for-loops in < 3.5
- Async with-statements in < 3.5
- F-strings in < 3.6

Closes we-like-parsers/cpython#124.
2020-04-30 20:27:52 -07:00
Victor Stinner eb0d359b4b
bpo-40443: Remove unused imports in stdlib (GH-19815) 2020-05-01 02:38:00 +02:00
Victor Stinner 2935e65c36
bpo-40275: Fix name error in support.socket_helper (GH-19825)
Replace TestFailed with support.TestFailed.

Bug spotted by pyflakes.
2020-05-01 02:35:24 +02:00
Victor Stinner 17014e4586
Remove dead code in test__xxsubinterpreters (GH-19826) 2020-05-01 02:34:22 +02:00
Victor Stinner 3c7f9db850
Revert "bpo-29587: Enable implicit exception chaining with gen.throw() (GH-19811)" (#19821)
This reverts commit 2514a632fb.
2020-04-30 22:44:24 +02:00
Chris Jerdonek 2514a632fb
bpo-29587: Enable implicit exception chaining with gen.throw() (GH-19811)
Before this commit, if an exception was active inside a generator
when calling gen.throw(), then that exception was lost (i.e. there
was no implicit exception chaining).  This commit fixes that.
2020-04-30 12:18:05 -07:00
Guido van Rossum c001c09e90
bpo-40334: Support type comments (GH-19780)
This implements full support for # type: <type> comments, # type: ignore <stuff> comments, and the func_type parsing mode for ast.parse() and compile().

Closes https://github.com/we-like-parsers/cpython/issues/95.

(For now, you need to use the master branch of mypy, since another issue unique to 3.9 had to be fixed there, and there's no mypy release yet.)

The only thing missing is `feature_version=N`, which is being tracked in https://github.com/we-like-parsers/cpython/issues/124.
2020-04-30 12:12:19 -07:00
Jelle Zijlstra efb8dd5b3e
compileall: Fix typos in docstring (GH-19810) 2020-04-30 16:09:11 +05:30
Victor Stinner e488e300f5
bpo-40443: Remove unused imports in distutils (GH-19802) 2020-04-30 11:28:09 +02:00
Victor Stinner 90549676e0
bpo-40443: Remove unused imports in the stdlib (GH-19803) 2020-04-30 11:26:33 +02:00
lrjball 3209cbd99b
bpo-40394 - difflib.SequenceMatched.find_longest_match default args (GH-19742)
* bpo-40394 - difflib.SequenceMatched.find_longest_match default args

Added default args to find_longest_match, as well as related tests.
2020-04-29 22:42:45 -05:00
Victor Stinner 6900f16d22
bpo-40443: Remove unused imports in idlelib (GH-19801) 2020-04-29 21:28:51 -04:00
Vlad Serebrennikov 138a9b9c2a
bpo-40389: Improve repr of typing.Optional (#19714) 2020-04-29 18:06:39 -07:00
Victor Stinner b1e11c31c5
bpo-40443: Remove unused imports in tests (GH-19804) 2020-04-30 02:21:30 +02:00
Victor Stinner 57572b103e
bpo-40443: Remove unused imports in tests (GH-19805) 2020-04-30 01:48:37 +02:00
Lysandros Nikolaou 69e802ed81
bpo-40334: Fix test_peg_parser to actually use the old parser (GH-19778)
Now that the default parser is the new PEG parser, ast.parse uses it, which means that we don't actually test something in test_peg_parser. This commit introduces a new keyword argument (`oldparser`) for `_peg_parser.parse_string` for specifying that a string needs to be parsed with the old parser. This keyword argument is used in the tests to actually compare the ASTs the new parser generates with those generated by the old parser.
2020-04-29 23:53:30 +01:00
karl ding 360371f79c
bpo-40291: Add support for CAN_J1939 sockets (GH-19538)
Add support for CAN_J1939 sockets that wrap SAE J1939 protocol
functionality provided by Linux 5.4+ kernels.
2020-04-29 15:31:19 -07:00
Victor Stinner 2d8757758d
bpo-40286: Remove C implementation of Random.randbytes() (GH-19797)
Remove _random.Random.randbytes(): the C implementation of
randbytes(). Implement the method in Python to ease subclassing:
randbytes() now directly reuses getrandbits().
2020-04-29 18:49:00 +02:00
Victor Stinner e3dfb9b967
bpo-9216: Expose OpenSSL FIPS_mode() as _hashlib.get_fips_mode() (GH-19703)
test.pythoninfo logs OpenSSL FIPS_mode() and Linux
/proc/sys/crypto/fips_enabled in a new "fips" section.

Co-Authored-By: Petr Viktorin <encukou@gmail.com>
2020-04-29 09:04:22 -07:00
Mark Shannon 57697245e1
bpo-40228: More robust frame.setlineno. (GH-19437)
More robust frame.setlineno. Makes no assumptions about source->bytecode translation.
2020-04-29 16:49:45 +01:00
Victor Stinner ec9bea4a37
bpo-40436: Fix code parsing gdb version (GH-19792)
test_gdb and test.pythoninfo now check gdb command exit code.
2020-04-29 17:11:48 +02:00
Pablo Galindo 5089bcd33f
Add missing sys import to socket_helper.py (GH-19791) 2020-04-29 12:32:31 +01:00
Serhiy Storchaka bfb1cf4465
bpo-40275: Move transient_internet from test.support to socket_helper (GH-19711) 2020-04-29 10:36:20 +03:00
Lysandros Nikolaou 6d65087655
bpo-40334: Disallow invalid single statements in the new parser (GH-19774)
After parsing is done in single statement mode, the tokenizer buffer has to be checked for additional lines and a `SyntaxError` must be raised, in case there are any.

Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
2020-04-29 02:42:27 +01:00
Victor Stinner a4dfe8ede5
bpo-39995: Fix concurrent.futures _ThreadWakeup (GH-19760)
Fix a race condition in concurrent.futures._ThreadWakeup: access to
_ThreadWakeup is now protected with the shutdown lock.
2020-04-29 03:32:06 +02:00
Hai Shi 66abe98a81
bpo-40275: Move requires_hashdigest() to test.support.hashlib_helper (GH-19716)
Add a new test.support.hashlib_helper submodule.
2020-04-29 03:11:29 +02:00
Lysandros Nikolaou 37af21b667
bpo-40334: Fix shifting of nested f-strings in the new parser (GH-19771)
`JoinedStr`s and `FormattedValue also needs to be shifted, in order to correctly compute the location information of nested f-strings.
2020-04-29 01:43:50 +01:00
Lewis Gaul f7bbf58aa9
bpo-38880: List interpreters associated with a channel end (GH-17323)
This PR adds the functionality requested by https://github.com/ericsnowcurrently/multi-core-python/issues/52.

Automerge-Triggered-By: @ericsnowcurrently
2020-04-28 17:18:42 -07:00
Miro Hrončok 49f70db83e
bpo-40431: Fix syntax typo in turtledemo (GH-19777)
***   File "/usr/lib64/python3.9/turtledemo/__main__.py", line 275
        bg="#d00" if clear == NORMAL else"#fca")
                                        ^
    SyntaxError: invalid string prefix
2020-04-29 02:00:07 +02:00
Eric Snow 5e8c691594
bpo-32604: Add support for a "default" arg in channel_recv(). (GH-19770)
This allows the caller to avoid creation of an exception when the channel is empty (just like `dict.get()` works).  `ChannelEmptyError` is still raised if no default is provided.

Automerge-Triggered-By: @ericsnowcurrently
2020-04-28 16:11:32 -07:00
Karthikeyan Singaravelan 521c8d6806
bpo-39966: Revert "bpo-25597: Ensure wraps' return value is used for magic methods in MagicMock" (GH-19734)
* Revert "bpo-25597: Ensure wraps' return value is used for magic methods in MagicMock (#16029)"

This reverts commit 72b1004657.
2020-04-28 20:22:31 +01:00
Ethan Onstott d9a43e20fa
bpo-40025: Require _generate_next_value_ to be defined before members (GH-19098)
require `_generate_next_value_` to be defined before members
2020-04-28 10:20:55 -07:00
Łukasz Langa d66685aad6
Merge tag 'v3.9.0a6' 2020-04-28 16:27:46 +02:00
Lysandros Nikolaou 3d53d8756f
bpo-40334: Don't skip test_parser:test_trigget_memory_error (GH-19744)
This test has been changed to always use the old parser, so no need for it to be skipped.
2020-04-28 01:24:50 +01:00
Lysandros Nikolaou d55133f49f
bpo-40334: Catch E_EOF error, when the tokenizer returns ERRORTOKEN (GH-19743)
An E_EOF error was only being caught after the parser exited before this commit. There are some cases though, where the tokenizer returns ERRORTOKEN *and* has set an E_EOF error (like when EOF directly follows a line continuation character) which weren't correctly handled before.
2020-04-28 01:23:35 +01:00
Łukasz Langa bc1c8af8ef
Python 3.9.0a6 2020-04-27 22:44:04 +02:00
Victor Stinner 5d1f32d33b
bpo-39995: Split test_concurrent_futures.test_crash() into sub-tests (GH-19739)
Now only test_error_during_result_unpickle_in_result_handler()
captures and ignores sys.stderr in the test process.

Tools like test.bisect_cmd don't support subTest() but only
work with the granularity of one method.

Remove unused ExecutorDeadlockTest._sleep_id() method.
2020-04-27 21:36:51 +02:00
Victor Stinner 1a275013d1
bpo-30966: concurrent.futures.Process.shutdown() closes queue (GH-19738)
Process.shutdown(wait=True) of concurrent.futures now closes
explicitly the result queue.
2020-04-27 20:53:37 +02:00
Pablo Galindo b94dbd7ac3
bpo-40334: Support PyPARSE_DONT_IMPLY_DEDENT in the new parser (GH-19736) 2020-04-27 18:35:58 +01:00
Pablo Galindo 2b74c835a7
bpo-40334: Support CO_FUTURE_BARRY_AS_BDFL in the new parser (GH-19721)
This commit also allows to pass flags to the new parser in all interfaces and fixes a bug in the parser generator that was causing to inline rules with actions, making them disappear.
2020-04-27 18:02:07 +01:00
Victor Stinner 9adccc1384
bpo-30966: Add multiprocessing.SimpleQueue.close() (GH-19735)
Add a new close() method to multiprocessing.SimpleQueue to explicitly
close the queue.

Automerge-Triggered-By: @pitrou
2020-04-27 09:11:10 -07:00
Dong-hee Na c5c42815ec
bpo-40375: Implement imaplib.IMAP4.unselect (GH-19712) 2020-04-27 23:52:55 +09:00
Serhiy Storchaka 6292be7adf
bpo-40398: Fix typing.get_args() for special generic aliases. (GH-19720) 2020-04-27 10:27:21 +03:00
Serhiy Storchaka 68b352a698
bpo-40396: Support GenericAlias in the typing functions. (GH-19718) 2020-04-26 21:21:08 +03:00
Nickolena Fisher cfaf4c09ab
Fix typo in Lib/typing.py (GH-19717) 2020-04-26 10:49:11 -07:00
Serhiy Storchaka 515fce4fc4
bpo-40275: Avoid importing logging in test.support (GH-19601)
Import logging lazily in assertLogs() in unittest.
Move TestHandler from test.support to logging_helper.
2020-04-25 11:35:18 +03:00
Serhiy Storchaka 16994912c9
bpo-40275: Avoid importing socket in test.support (GH-19603)
* Move socket related functions from test.support to socket_helper.
* Import socket, nntplib and urllib.error lazily in transient_internet().
* Remove importing multiprocess.
2020-04-25 10:06:29 +03:00
Serhiy Storchaka 3c8a5b459d
bpo-40275: Avoid importing asyncio in test.support (GH-19600)
* Import asyncio lazily in unittest (only when IsolatedAsyncioTestCase is used).
* Import asyncio.events lazily in test.support.
2020-04-25 10:04:10 +03:00
Carl Meyer 503de7149d
bpo-40360: Deprecate lib2to3 module in light of PEP 617 (GH-19663)
Deprecate lib2to3 module in light of PEP 617.

We anticipate removal in the 3.12 timeframe.
2020-04-24 11:19:46 -07:00
Lysandros Nikolaou 24ffe705c3
bpo-40334: Rewrite test_c_parser to avoid memory leaks (GH-19694)
Previously every test was building an extension module and
loading it into sys.modules. The tearDown function was thus
not able to clean up correctly, resulting in memory leaks.

With this commit, every test function now builds the extension
module and runs the actual test code in a new process
(using assert_python_ok), so that sys.modules stays intact
and no memory gets leaked.
2020-04-24 14:51:09 +01:00
Raymond Hettinger 4cc4d6048e
Expand the implementation comments (GH-19699) 2020-04-24 02:33:07 -07:00
Serhiy Storchaka 40ded947f8
bpo-40336: Refactor typing._SpecialForm (GH-19620) 2020-04-23 21:26:48 +03:00
Victor Stinner d663d34685
bpo-39983: Add test.support.print_warning() (GH-19683)
Log "Warning -- ..." test warnings into sys.__stderr__ rather than
sys.stderr, to ensure to display them even if sys.stderr is captured.

test.libregrtest.utils.print_warning() now calls
test.support.print_warning().
2020-04-23 19:03:52 +02:00
Pablo Galindo 9e6a1312c1
bpo-40370: Use the same compile and link args as the interpreter used in test_peg_generator (GH-19674) 2020-04-23 14:46:22 +01:00
Lysandros Nikolaou 8d1cbfffea
bpo-40334: Suppress all output in test_peg_generator (GH-19675) 2020-04-23 13:22:16 +01:00
Victor Stinner 1def7754b7
bpo-40334: Rename PyConfig.use_peg to _use_peg_parser (GH-19670)
* Rename PyConfig.use_peg to _use_peg_parser
* Document PyConfig._use_peg_parser and mark it a deprecated
* Mark -X oldparser option and PYTHONOLDPARSER env var as deprecated
  in the documentation.
* Add use_old_parser() and skip_if_new_parser() to test.support
* Remove sys.flags.use_peg: use_old_parser() uses
  _testinternalcapi.get_configs() instead.
* Enhance test_embed tests
* subprocess._args_from_interpreter_flags() copies -X oldparser
2020-04-23 03:03:24 +02:00
Pablo Galindo c5fc156852
bpo-40334: PEP 617 implementation: New PEG parser for CPython (GH-19503)
Co-authored-by: Guido van Rossum <guido@python.org>
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
2020-04-22 23:29:27 +01:00
sweeneyde a81849b031
bpo-39939: Add str.removeprefix and str.removesuffix (GH-18939)
Added str.removeprefix and str.removesuffix methods and corresponding
bytes, bytearray, and collections.UserString methods to remove affixes
from a string if present. See PEP 616 for a full description.
2020-04-22 23:05:48 +02:00
Anthony Sottile 39652cd8bd
bpo-40260: Remove unnecessary newline in compile() call (GH-19641)
Because some people subclass this class and call undocumented methods, and we don't want to break them.
2020-04-22 19:42:53 +01:00
Batuhan Taşkaya 4454057269
bpo-39562: Prevent collision of future and compiler flags (GH-19230)
The constant values of future flags in the __future__ module
is updated in order to prevent collision with compiler flags.
Previously PyCF_ALLOW_TOP_LEVEL_AWAIT was clashing
with CO_FUTURE_DIVISION.
2020-04-22 18:09:03 +02:00
Steve Dower 9b49893900
bpo-40214: Fix ctypes WinDLL test with insecure flags (GH-19652) 2020-04-22 17:04:46 +01:00
Victor Stinner 9bee32b34e
bpo-40138: Fix Windows os.waitpid() for large exit code (GH-19637)
Fix the Windows implementation of os.waitpid() for exit code
larger than "INT_MAX >> 8". The exit status is now interpreted as an
unsigned number.

os.waitstatus_to_exitcode() now accepts wait status larger than
INT_MAX.
2020-04-22 16:30:35 +02:00
Joshua Root b310700976
bpo-38360: macOS: support alternate form of -isysroot flag (GH-16480)
It is possible to use either '-isysroot /some/path' (with a space) or
'-isysroot/some/path' (no space in between). Support both forms in
places where special handling of -isysroot is done, rather than just
the first form.
Co-authored-by: Ned Deily <nad@python.org>
2020-04-22 03:44:10 -04:00
Miro Hrončok 3a69f3caee
bpo-38439: Add 256px IDLE icon (GH-17473)
Icon author: Andrew Clover, bpo-1490384
2020-04-22 03:21:44 -04:00
Raymond Hettinger 75bedbe2ed
bpo-40327: Improve atomicity, speed, and memory efficiency of the items() loop (GH-19628) 2020-04-21 16:20:52 -07:00
Pablo Galindo 11a7f158ef
bpo-40335: Correctly handle multi-line strings in tokenize error scenarios (GH-19619)
Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>
2020-04-21 01:53:04 +01:00
HongWeipeng a25a04fea5
bpo-39942:Fix failure in `TypeVar` when missing `__name__` (GH-19616)
https://bugs.python.org/issue39942
2020-04-20 21:01:53 +01:00
Antoine Pitrou eba9f6155d
bpo-40330: Fix utf-8 size check in ShareableList (GH-19606)
The item size must be checked after encoding to bytes, not before.

Automerge-Triggered-By: @pitrou
2020-04-20 11:54:55 -07:00
Barry 9b0b5d2bae
bpo-40260: Revert breaking changes made in modulefinder (GH-19595) 2020-04-20 15:58:42 +01:00
Galden df8913f7c4
Fix typo in Lib/tracepack.py (GH-19605)
Typo fix: "emites" -> "emit".
2020-04-19 19:17:37 -07:00
Tim Hoffmann 8aea4b3605
bpo-40148: Add PurePath.with_stem() (GH-19295)
Add PurePath.with_stem()
2020-04-19 17:29:49 +02:00
Thomas Krennwallner c8f1715283
bpo-38891: avoid quadratic item access performance of ShareableList (GH-18996)
Avoid linear runtime of ShareableList.__getitem__ and
ShareableList.__setitem__ by storing running allocated bytes in
ShareableList._allocated_bytes instead of the number of bytes for
a particular stored item.

Co-authored-by: Antoine Pitrou <antoine@python.org>
2020-04-19 17:19:24 +02:00
Kyle Stanley 1ac6e37929
bpo-39207: Spawn workers on demand in ProcessPoolExecutor (GH-19453)
Roughly based on 904e34d4e6, but with a few substantial differences.

/cc @pitrou @brianquinlan
2020-04-19 07:00:59 -07:00
Raymond Hettinger 4fe002045f
bpo-40325: Deprecate set object support in random.sample() (GH-19591) 2020-04-19 00:36:42 -07:00
Karthikeyan Singaravelan 696136b993
bpo-35113: Fix inspect.getsource to return correct source for inner classes (#10307)
* Use ast module to find class definition

* Add NEWS entry

* Fix class with multiple children and move decorator code to the method

* Fix PR comments

1. Use node.decorator_list to select decorators
2. Remove unwanted variables in ClassVisitor
3. Simplify stack management as per review

* Add test for nested functions and async calls

* Fix pydoc test since comments are returned now correctly

* Set event loop policy as None to fix environment related change

* Refactor visit_AsyncFunctionDef and tests

* Refactor to use local variables and fix tests

* Add patch attribution

* Use self.addCleanup for asyncio

* Rename ClassVisitor to ClassFinder and fix asyncio cleanup

* Return first class inside conditional in case of multiple definitions. Remove decorator for class source.

* Add docstring to make the test correct

* Modify NEWS entry regarding decorators

* Return decorators too for bpo-15856

* Move ast and the class source code to top. Use proper Exception.
2020-04-18 21:49:32 +05:30
Hakan Çelik ce578831a4
Add spaces around the ":=" operator in ast_unparse.c (GH-19568) 2020-04-18 17:17:19 +01:00
Serhiy Storchaka 12446e6a60
bpo-40179: Fix translation of #elif in Argument Clinic (GH-19364)
Co-authored-by: Ammar Askar <ammar@ammaraskar.com>
2020-04-18 17:52:48 +03:00
Chih-Hsuan Yen fb940408ce
bpo-35967: Skip test with `uname -p` on Android (GH-19577)
The uname binary on Android does not support -p [1]. Here is a sample
log:
```
0:06:03 load avg: 0.56 [254/421/8] test_platform failed -- running: test_asyncio (5 min 53 sec)
uname: Unknown option p (see "uname --help")
test test_platform failed -- Traceback (most recent call last):
  File "/data/local/tmp/lib/python3.9/test/test_platform.py", line 170, in test_uname_processor
    proc_res = subprocess.check_output(['uname', '-p'], text=True).strip()
  File "/data/local/tmp/lib/python3.9/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/data/local/tmp/lib/python3.9/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['uname', '-p']' returned non-zero exit status 1.
```
[1] https://android.googlesource.com/platform/external/toybox/+/refs/heads/master/toys/posix/uname.c

Automerge-Triggered-By: @jaraco
2020-04-18 07:20:54 -07:00
Serhiy Storchaka 7e64414f57
bpo-40257: Improve help for the typing module (GH-19546)
* Show docstring for special forms.
* Show docstring for special generic aliases.
* Show documentation for __origin__ for generic aliases.
2020-04-18 17:13:21 +03:00
Galden c606624af8
Fix two typos in multiprocessing (GH-19571) 2020-04-18 08:58:29 +02:00
Victor Stinner 87502ddd71
bpo-40286: Use random.randbytes() in tests (GH-19575) 2020-04-17 22:54:38 +02:00
Serhiy Storchaka 223221b290
bpo-40286: Makes simpler the relation between randbytes() and getrandbits() (GH-19574) 2020-04-17 23:51:28 +03:00
Barney Gale 5b1d9184bb
bpo-39894: Route calls from pathlib.Path.samefile() to os.stat() via the path accessor (GH-18836) 2020-04-17 19:47:27 +02:00
Barney Gale c746c4f353
bpo-39897: Remove needless `Path(self.parent)` call, which makes `is_mount()` misbehave in `Path` subclasses. (GH-18839) 2020-04-17 19:42:06 +02:00
Antoine Pitrou 75a3378810
bpo-40282: Allow random.getrandbits(0) (GH-19539) 2020-04-17 19:32:14 +02:00
Victor Stinner 9f5fe7910f
bpo-40286: Add randbytes() method to random.Random (GH-19527)
Add random.randbytes() function and random.Random.randbytes()
method to generate random bytes.

Modify secrets.token_bytes() to use SystemRandom.randbytes()
rather than calling directly os.urandom().

Rename also genrand_int32() to genrand_uint32(), since it returns an
unsigned 32-bit integer, not a signed integer.

The _random module is now built with Py_BUILD_CORE_MODULE defined.
2020-04-17 19:05:35 +02:00
Barney Gale 22386bb4ef
bpo-39901: Move `pathlib.Path.owner()` and `group()` implementations into the path accessor. (GH-18844) 2020-04-17 18:41:07 +02:00
Mariusz Felisiak 06a35542aa
bpo-40300: Allow empty logging.Formatter.default_msec_format. (GH-19551) 2020-04-17 17:02:47 +01:00
Victor Stinner 1ae035b7e8
bpo-40302: Add pycore_byteswap.h header file (GH-19552)
Add a new internal pycore_byteswap.h header file with the following
functions:

* _Py_bswap16()
* _Py_bswap32()
* _Py_bswap64()

Use these functions in _ctypes, sha256 and sha512 modules,
and also use in the UTF-32 encoder.

sha256, sha512 and _ctypes modules are now built with the internal
C API.
2020-04-17 17:47:20 +02:00
Inada Naoki 485e715cb1
bpo-40287: Fix SpooledTemporaryFile.seek() return value (GH-19540)
It has not returned the file position after the seek.
2020-04-17 15:56:35 +09:00
Batuhan Taşkaya 5565c30f0b
bpo-39793: use the same domain on make_msgid tests (#18698)
* bpo-39793: use same domain on make_msgid tests

* apply suggestions
2020-04-16 13:29:12 -04:00
Raymond Hettinger 70f027dd22
bpo-40290: Add zscore() to statistics.NormalDist. (GH-19547) 2020-04-16 10:25:14 -07:00
Jason R. Coombs 518835f335
bpo-35967 resolve platform.processor late (GH-12239)
* Replace flag-flip indirection with direct inspection

* Use any for simpler code

* Avoid flag flip and set results directly.

* Resolve processor in a single function.

* Extract processor handling into a namespace (class)

* Remove _syscmd_uname, unused

* Restore platform.processor behavior to match prior expectation (reliant on uname -p in a subprocess).

* Extract '_unknown_as_blank' function.

* Override uname_result to resolve the processor late.

* Add a test intended to capture the expected values from 'uname -p'

* Instead of trying to keep track of all of the possible outputs on different systems (probably a fool's errand), simply assert that except for the known platform variance, uname().processor matches the output of 'uname -p'

* Use a skipIf directive

* Use contextlib.suppress to suppress the error. Inline strip call.

* 📜🤖 Added by blurb_it.

* Remove use of contextlib.suppress (it would fail with NameError if it had any effect). Rely on _unknown_as_blank to replace unknown with blank.

Co-authored-by: blurb-it[bot] <blurb-it[bot]@users.noreply.github.com>
2020-04-16 08:28:09 -04:00
Hakan Çelik 6a5bf15c71
bpo-40209: Use tokenize.open in test_unparse (GH-19399) 2020-04-16 11:11:55 +01:00
Shantanu 01508dcde4
Fix typo in exception thrown by ast.unparse (GH-19534) 2020-04-16 11:10:12 +01:00
Jason R. Coombs e72cbcb346
bpo-35967: Make test_platform.test_uname_processor more lenient to satisfy build bots. (GH-19544)
* bpo-35967: Make test more lenient to satisfy build bots.

* Update Lib/test/test_platform.py

Co-Authored-By: Kyle Stanley <aeros167@gmail.com>

* Expect '' for 'unknown'

Co-authored-by: Kyle Stanley <aeros167@gmail.com>
2020-04-15 19:55:35 -04:00
Serhiy Storchaka fbf2786c4c
bpo-40257: Output object's own docstring in pydoc (GH-19479) 2020-04-15 23:00:20 +03:00
Russell Davis ba1bcffe5c
bpo-29255: Wait in KqueueSelector.select when no fds are registered (GH-19508)
Also partially fixes bpo-25680 (there's still a discrepancy in behavior
on Windows that needs to be fixed).
2020-04-15 11:57:06 -07:00
Jason R. Coombs 4b4e90a518
bpo-35967: Baseline values for uname -p (GH-12824)
* Add a test intended to capture the expected values from 'uname -p'

* Instead of trying to keep track of all of the possible outputs on different systems (probably a fool's errand), simply assert that except for the known platform variance, uname().processor matches the output of 'uname -p'

* Use a skipIf directive

* Use contextlib.suppress to suppress the error. Inline strip call.
2020-04-15 14:32:01 -04:00
Lysandros Nikolaou 9a4b38f66b
bpo-40267: Fix message when last input character produces a SyntaxError (GH-19521)
When there is a SyntaxError after reading the last input character from
the tokenizer and if no newline follows it, the error message used to be
`unexpected EOF while parsing`, which is wrong.
2020-04-15 11:22:10 -07:00
Jason R. Coombs 574547a75c
Clean up compatibility code in importlib fixtures (#19156) 2020-04-15 13:55:43 -04:00
Ammar Askar a86b522d8f
bpo-40277: Add a repr() to namedtuple's _tuplegetter to aid with introspection (GH-19537) 2020-04-14 23:36:08 -07:00
Ethan Smith d01628e411
bpo-39481: PEP 585 for dataclasses, mailbox, contextvars (GH-19425) 2020-04-14 16:14:15 -07:00
Miro Hrončok 4c0a31fb08
bpo-9216: Nobody expects the geohashing FIPS inquisition (GH-19520)
Automerge-Triggered-By: @tiran
2020-04-14 12:49:11 -07:00
Barry d42e582063
bpo-40260: Update modulefinder to use io.open_code() and respect coding comments (GH-19488) 2020-04-14 20:16:06 +01:00
Batuhan Taşkaya aade1cc453
bpo-395222: Correctly unparse unicode prefix in ast_unparse.c (GH-19512) 2020-04-14 19:55:01 +01:00
Zackery Spytz 96515e9f67
bpo-32033: Fix test_pwd failures on Android (GH-19502) 2020-04-14 20:11:46 +02:00
Dong-hee Na a5900ecf9f
bpo-40221: Update multiprocessing to use _at_fork_reinit (GH-19511) 2020-04-15 01:35:36 +09:00
Victor Stinner 4cf65a630a
regrtest: log timeout at startup (GH-19514)
Reduce also worker timeout.
2020-04-14 18:29:44 +02:00
Dong-hee Na e1945307d3
bpo-40221: Update multiprocessing to use _at_fork_reinit (GH-19477) 2020-04-14 22:15:52 +09:00
Ethan Smith 8ef875028a
bpo-39481: Make weakref and WeakSet generic (GH-19497) 2020-04-13 21:54:40 -07:00
Ethan Smith cecf049673
bpo-39481: Make functools.cached_property, partial, partialmethod generic (#19427) 2020-04-13 21:53:04 -07:00
Galden 02152b7332
Fix typo from Lib/asyncio/events.py (GH-19410) 2020-04-14 09:04:32 +09:00
Batuhan Taşkaya 990ea4200f
bpo-40208: Remove deprecated has_exec method of SymbolTable (GH-19396) 2020-04-14 08:51:32 +09:00
Sebastian Pedersen a1a0eb4a39
bpo-39380: Change ftplib encoding from latin-1 to utf-8 (GH-18048)
Add the encoding in ftplib.FTP and ftplib.FTP_TLS to the
constructor as keyword-only and change the default from "latin-1" to "utf-8"
to follow RFC 2640.
2020-04-14 01:07:56 +02:00
Batuhan Taşkaya 258f5179f9
bpo-32894: Support unparsing of infinity numbers in ast_unparser.c (GH-17426) 2020-04-14 00:51:31 +02:00
Victor Stinner 4c3da783cf
bpo-40091: Fix a hang at fork in the logging module (GH-19416)
Fix a hang at fork in the logging module: the new private
_at_fork_reinit() method is now used to reinitialize locks at fork in
the child process.

The createLock() method is no longer used at fork.
2020-04-14 00:25:34 +02:00
Chih-Hsuan Yen 25a6833f79
bpo-39481: fix test_genericalias on Android (GH-19469)
Android bionic does not implement shm_open/shm_unlink [1].
As a result _posixshmem extension does not exist and
multiprocessing.shared_memory cannot be imported.

[1] https://android.googlesource.com/platform/bionic/+/master/docs/status.md
2020-04-13 15:00:16 -07:00
Victor Stinner 0135598d72
bpo-40241: Add pycore_gc.h header file (GH-19494)
Move the PyGC_Head structure and the following private macros to the
internal C API:

* _PyGCHead_FINALIZED()
* _PyGCHead_NEXT()
* _PyGCHead_PREV()
* _PyGCHead_SET_FINALIZED()
* _PyGCHead_SET_NEXT()
* _PyGCHead_SET_PREV()
* _PyGC_FINALIZED()
* _PyGC_PREV_MASK
* _PyGC_PREV_MASK_COLLECTING
* _PyGC_PREV_MASK_FINALIZED
* _PyGC_PREV_SHIFT
* _PyGC_SET_FINALIZED()
* _PyObject_GC_IS_TRACKED()
* _PyObject_GC_MAY_BE_TRACKED()
* _Py_AS_GC(o)

Keep the private _PyGC_FINALIZED() macro in the public C API for
backward compatibility with Python 3.8: make it an alias to the new
PyObject_GC_IsFinalized() function.

Move the SIZEOF_PYGC_HEAD constant from _testcapi module to
_testinternalcapi module.
2020-04-13 11:38:42 +02:00
Pablo Galindo 70c188eee0
Add double quote cases to invalid prefix tests (GH-19489) 2020-04-13 02:47:35 +01:00
Victor Stinner 14d5331eb5
bpo-40234: Revert "bpo-37266: Daemon threads are now denied in subinterpreters (GH-14049)" (GH-19456)
This reverts commit 066e5b1a91.
2020-04-12 23:45:09 +02:00
Lysandros Nikolaou 41d5b94af4
bpo-40246: Report a better error message for invalid string prefixes (GH-19476) 2020-04-12 19:21:00 +01:00
Oren Milman 402e1cdb13
bpo-31758: Prevent crashes when using an uninitialized _elementtree.XMLParser object (GH-3997) 2020-04-12 16:36:41 +02:00
Alex Itkes 63e5b59c06
bpo-13743: Add some documentation strings to xml.dom.minidom (GH-16355) 2020-04-12 16:21:58 +02:00
mefistotelis 5fd8123dfd
bpo-39011: Preserve line endings within ElementTree attributes (GH-18468)
* bpo-39011: Preserve line endings within attributes

Line endings within attributes were previously normalized to "\n" in Py3.7/3.8.
This patch removes that normalization, as line endings which were
replaced by entity numbers should be preserved in original form.
2020-04-12 14:51:58 +02:00
Serhiy Storchaka 4b222c9491
bpo-40126: Fix reverting multiple patches in unittest.mock. (GH-19351)
Patcher's __exit__() is now never called if its __enter__() is failed.
Returning true from __exit__() silences now the exception.
2020-04-11 10:59:24 +03:00
Batuhan Taşkaya 0361556537
bpo-39481: PEP 585 for a variety of modules (GH-19423)
- concurrent.futures
- ctypes
- http.cookies
- multiprocessing
- queue
- tempfile
- unittest.case
- urllib.parse
2020-04-10 07:46:36 -07:00
Ethan Smith e3ec44d692
bpo-39481: PEP 585 for difflib, filecmp, fileinput (#19422) 2020-04-09 21:47:31 -07:00
Ethan Smith 7c4185d62d
bpo-39481: PEP 585 for enumerate, AsyncGeneratorType, mmap (GH-19421) 2020-04-09 21:25:53 -07:00
Batuhan Taşkaya 2fa67df605
bpo-39481: PEP 585 for ipaddress.py (GH-19418) 2020-04-09 21:04:54 -07:00
Ethan Smith a8403d057d
Generic itertools.chain (GH-19417) 2020-04-09 20:28:08 -07:00
pxinwr 5cd2803009
bpo-31904: Fix test_c_locale_coercion encodings for VxWorks RTOS (GH-19448) 2020-04-09 17:46:23 +02:00
Serhiy Storchaka f228bf2300
bpo-40187: Refactor typing.TypedDict. (GH-19372) 2020-04-08 11:03:27 +03:00
Serhiy Storchaka a2ec06938f
bpo-40185: Refactor typing.NamedTuple (GH-19371) 2020-04-08 10:59:04 +03:00
Batuhan Taşkaya f9dd51e7db
bpo-39481: Make os.DirEntry generic (GH-19415) 2020-04-07 14:37:19 -07:00
Victor Stinner 87255be696
bpo-40089: Add _at_fork_reinit() method to locks (GH-19195)
Add a private _at_fork_reinit() method to _thread.Lock,
_thread.RLock, threading.RLock and threading.Condition classes:
reinitialize the lock after fork in the child process; reset the lock
to the unlocked state.

Rename also the private _reset_internal_locks() method of
threading.Event to _at_fork_reinit().

* Add _PyThread_at_fork_reinit() private function. It is excluded
  from the limited C API.
* threading.Thread._reset_internal_locks() now calls
  _at_fork_reinit() on self._tstate_lock rather than creating a new
  Python lock object.
2020-04-07 23:11:49 +02:00
Guido van Rossum 48b069a003
bpo-39481: Implementation for PEP 585 (#18239)
This implements things like `list[int]`,
which returns an object of type `types.GenericAlias`.
This object mostly acts as a proxy for `list`,
but has attributes `__origin__` and `__args__`
that allow recovering the parts (with values `list` and `(int,)`.

There is also an approximate notion of type variables;
e.g. `list[T]` has a `__parameters__` attribute equal to `(T,)`.
Type variables are objects of type `typing.TypeVar`.
2020-04-07 09:50:06 -07:00
Zachary Ware f407e209c1
bpo-40214: Temporarily disable a ctypes test (GH-19404)
Only one particular sub-test of
ctypes.test.test_loading.test_load_dll_with_flags is disabled, which
caused failures on Azure Pipelines CI.
2020-04-07 01:39:58 -05:00
Pablo Galindo 799d7d61a9
bpo-40196: Fix a bug in the symtable when reporting inspecting global variables (GH-19391) 2020-04-06 17:05:57 +01:00
Serhiy Storchaka 6fed3c8540
bpo-40182: Remove the _field_types attribute of the NamedTuple class (GH-19368) 2020-04-05 00:43:20 +03:00
Serhiy Storchaka a94e6272f1
bpo-36517: Raise error on multiple inheritance with NamedTuple (GH-19363) 2020-04-04 21:31:30 +03:00
Raymond Hettinger 1ae6445391
Convert tuples to sets for faster searches (GH-19365) 2020-04-04 11:03:04 -07:00
Tal Einat 52013e5b6d
bpo-38689: avoid IDLE hanging when calltip fails getting a signature (GH-17152)
Inspect.signature failed on the test case because its isinstance call raised.
2020-04-03 23:05:58 -04:00
Pablo Galindo 40cf35c5b0
bpo-40141: Include the value in the column position for keyword AST nodes (GH-19348) 2020-04-03 21:02:26 +01:00
Pablo Galindo 254ec78341
bpo-40147: Move the check for duplicate keywords to the compiler (GH-19289) 2020-04-03 20:37:13 +01:00
Michael Felt 76db37b1d3
bpo-40112: distutils test_search_cpp: Fix logic to determine if C compiler is xlc on AIX (GH-19225) 2020-04-03 16:38:28 +02:00
Victor Stinner 7a51a7e19f
bpo-40140: test_builtin.PtyTests registers SIGHUP handler (GH-19314)
test_builtin.PtyTests now registers an handler for SIGHUP signal.
Closing the PTY file descriptor can emit a SIGHUP signal: just ignore
it.

run_child() now also closes the PTY file descriptor before waiting
for the process completition, otherwise the test hangs on AIX.
2020-04-03 00:40:25 +02:00
Tim Hatch 3c3aa4516c
lib2to3: Support named assignment expressions (GH-12702)
There are two copies of the grammar -- the one used by Python itself as
Grammar/Grammar, and the one used by lib2to3 which has necessarily diverged at
Lib/lib2to3/Grammar.txt because it needs to support older syntax an we want it
to be reasonable stable to avoid requiring fixer rewrites.

This brings suport for syntax like `if x:= foo():` to match what the live
Python grammar does.

This should've been added at the time of the walrus operator itself, but lib2to3 being
independent is often overlooked.  So we do consider this a bugfix rather than enhancement.
2020-04-02 15:34:54 -07:00
Derek Keeler 45217af29c
bpo-38972: Link to instructions to change PowerShell execution policy (GH-19131) 2020-04-02 12:00:21 -07:00
Kyle Stanley 8ec7cb53f0
bpo-40115: Fix refleak in test_asyncio.test_run_in_executor_cancel() (GH-19282)
Call explicitly self.loop.shutdown_default_executor().
2020-04-02 04:46:44 +02:00
Victor Stinner 9a679a0e47
bpo-40094: CGIHTTPRequestHandler logs exit code (GH-19285)
CGIHTTPRequestHandler of http.server now logs the CGI script exit
code, rather than the CGI script exit status of os.waitpid().

For example, if the script is killed by signal 11, it now logs:
"CGI script exit code -11."
2020-04-02 03:42:05 +02:00
Victor Stinner 0b297d4ff1
bpo-39503: CVE-2020-8492: Fix AbstractBasicAuthHandler (GH-18284)
The AbstractBasicAuthHandler class of the urllib.request module uses
an inefficient regular expression which can be exploited by an
attacker to cause a denial of service. Fix the regex to prevent the
catastrophic backtracking. Vulnerability reported by Ben Caller
and Matt Schwager.

AbstractBasicAuthHandler of urllib.request now parses all
WWW-Authenticate HTTP headers and accepts multiple challenges per
header: use the realm of the first Basic challenge.

Co-Authored-By: Serhiy Storchaka <storchaka@gmail.com>
2020-04-02 02:52:20 +02:00