Commit Graph

815 Commits

Author SHA1 Message Date
Filipe Laíns 233841ab78
bpo-45379: add custom error string for FROZEN_DISABLED (GH-29190)
Signed-off-by: Filipe Laíns <lains@riseup.net>
Co-authored-by: Gareth Rees <gdr@garethrees.org>
2021-10-28 22:20:07 +02:00
Victor Stinner 354c35220d
bpo-45482: Rename namespaceobject.h to pycore_namespace.h (GH-28975)
Rename Include/namespaceobject.h to
Include/internal/pycore_namespace.h.

The _testmultiphase extension is now built with the
Py_BUILD_CORE_MODULE macro defined to access _PyNamespace_Type.

object.c: remove unused "pycore_context.h" include.
2021-10-15 15:21:21 +02:00
Eric Snow 79cf20e48d
bpo-21736: Set __file__ on frozen stdlib modules. (gh-28656)
Currently frozen modules do not have __file__ set.  In their spec, origin is set to "frozen" and they are marked as not having a location.  (Similarly, for frozen packages __path__ is set to an empty list.)  However, for frozen stdlib modules we are able to extrapolate __file__ as long as we can determine the stdlib directory at runtime.  (We now do so since gh-28586.)  Having __file__ set is helpful for a number of reasons.  Likewise, having a non-empty __path__ means we can import submodules of a frozen package from the filesystem (e.g. we could partially freeze the encodings module).

This change sets __file__ (and adds to __path__) for frozen stdlib modules.  It uses sys._stdlibdir (from gh-28586) and the frozen module alias information (from gh-28655).  All that work is done in FrozenImporter (in Lib/importlib/_bootstrap.py). 
 Also, if a frozen module is imported before importlib is bootstrapped (during interpreter initialization) then we fix up that module and its spec during the importlib bootstrapping step (i.e. imporlib._bootstrap._setup()) to match what gets set by FrozenImporter, including setting the file info (if the stdlib dir is known).  To facilitate this, modules imported using PyImport_ImportFrozenModule() have __origname__ set using the frozen module alias info.  __origname__ is popped off during importlib bootstrap.

(To be clear, even with this change the new code to set __file__ during fixups in imporlib._bootstrap._setup() doesn't actually get triggered yet.  This is because sys._stdlibdir hasn't been set yet in interpreter initialization at the point importlib is bootstrapped.  However, we do fix up such modules at that point to otherwise match the result of importing through FrozenImporter, just not the __file__ and __path__ parts.  Doing so will require changes in the order in which things happen during interpreter initialization.  That can be addressed separately.  Once it is, the file-related fixup code from this PR will kick in.)

Here are things this change does not do:

* set __file__ for non-stdlib modules (no way of knowing the parent dir)
* set __file__ if the stdlib dir is not known (nor assume the expense of finding it)
* relatedly, set __file__ if the stdlib is in a zip file
* verify that the filename set to __file__ actually exists (too expensive)
* update __path__ for frozen packages that alias a non-package (since there is no package dir)

Other things this change skips, but we may do later:

* set __file__ on modules imported using PyImport_ImportFrozenModule()
* set co_filename when we unmarshal the frozen code object while importing the module (e.g. in FrozenImporter.exec_module()) -- this would allow tracebacks to show source lines
* implement FrozenImporter.get_filename() and FrozenImporter.get_source()

https://bugs.python.org/issue21736
2021-10-14 15:32:18 -06:00
Victor Stinner 713bb19356
bpo-45434: Mark the PyTokenizer C API as private (GH-28924)
Rename PyTokenize functions to mark them as private:

* PyTokenizer_FindEncodingFilename() => _PyTokenizer_FindEncodingFilename()
* PyTokenizer_FromString() => _PyTokenizer_FromString()
* PyTokenizer_FromFile() => _PyTokenizer_FromFile()
* PyTokenizer_FromUTF8() => _PyTokenizer_FromUTF8()
* PyTokenizer_Free() => _PyTokenizer_Free()
* PyTokenizer_Get() => _PyTokenizer_Get()

Remove the unused PyTokenizer_FindEncoding() function.

import.c: remove unused #include "errcode.h".
2021-10-13 17:22:14 +02:00
Christian Clauss db693df3e1
Fix typos in the Python directory (GH-28767) 2021-10-06 15:55:27 -07:00
Eric Snow 08285d563e
bpo-45020: Identify which frozen modules are actually aliases. (gh-28655)
In the list of generated frozen modules at the top of Tools/scripts/freeze_modules.py, you will find that some of the modules have a different name than the module (or .py file) that is actually frozen. Let's call each case an "alias". Aliases do not come into play until we get to the (generated) list of modules in Python/frozen.c. (The tool for freezing modules, Programs/_freeze_module, is only concerned with the source file, not the module it will be used for.)

Knowledge of which frozen modules are aliases (and the identity of the original module) normally isn't important. However, this information is valuable when we go to set __file__ on frozen stdlib modules. This change updates Tools/scripts/freeze_modules.py to map aliases to the original module name (or None if not a stdlib module) in Python/frozen.c. We also add a helper function in Python/import.c to look up a frozen module's alias and add the result of that function to the frozen info returned from find_frozen().

https://bugs.python.org/issue45020
2021-10-05 11:26:37 -06:00
Eric Snow c3d9ac8b34
bpo-45324: Capture data in FrozenImporter.find_spec() to use in exec_module(). (gh-28633)
Before this change we end up duplicating effort and throwing away data in FrozenImporter.find_spec().  Now we do the work once in find_spec() and the only thing we do in FrozenImporter.exec_module() is turn the raw frozen data into a code object and then exec it.

We've added _imp.find_frozen(), add an arg to _imp.get_frozen_object(), and updated FrozenImporter.  We've also moved some code around to reduce duplication, get a little more consistency in outcomes, and be more efficient.

Note that this change is mostly necessary if we want to set __file__ on frozen stdlib modules. (See https://bugs.python.org/issue21736.)

https://bugs.python.org/issue45324
2021-10-05 10:01:27 -06:00
Hai Shi b9bb74871b
bpo-44050: Extension modules can share state when they don't support sub-interpreters. (GH-27794)
Automerge-Triggered-By: GH:encukou
2021-10-05 06:19:32 -07:00
Eric Snow a65c86889e
bpo-45020: Add -X frozen_modules=[on|off] to explicitly control use of frozen modules. (gh-28320)
Currently we freeze several modules into the runtime. For each of these modules it is essential to bootstrapping the runtime that they be frozen. Any other stdlib module that we later freeze into the runtime is not essential. We can just as well import from the .py file.  This PR lets users explicitly choose which should be used, with the new "-X frozen_modules=[on|off]" CLI flag. The default is "off" for now.

https://bugs.python.org/issue45020
2021-09-14 17:31:45 -06:00
Eric Snow a2d8c4b81b
bpo-45019: Do some cleanup related to frozen modules. (gh-28319)
There are a few things I missed in gh-27980. This is a follow-up that will make subsequent PRs cleaner. It includes fixes to tests and tools that reference the frozen modules.

https://bugs.python.org/issue45019
2021-09-13 16:18:37 -06:00
Victor Stinner 489699ca05
bpo-44441: _PyImport_Fini2() resets PyImport_Inittab (GH-26874)
Py_RunMain() now resets PyImport_Inittab to its initial value at
exit. It must be possible to call PyImport_AppendInittab() or
PyImport_ExtendInittab() at each Python initialization.
2021-06-23 14:13:27 +02:00
Victor Stinner d36d6a9c18
bpo-43244: Remove Yield macro from pycore_ast.h (GH-25243)
* pycore_ast.h no longer defines the Yield macro.
* Fix a compiler warning on Windows: "warning C4005: 'Yield': macro
  redefinition".
* Python-ast.c now defines directly functions with their real
  _Py_xxx() name, rather than xxx().
* Remove "#undef Yield" in C files including pycore_ast.h.
2021-04-07 13:01:09 +02:00
Victor Stinner 94faa0724f
bpo-43244: Remove ast.h, asdl.h, Python-ast.h headers (GH-24933)
These functions were undocumented and excluded from the limited C
API.

Most names defined by these header files were not prefixed by "Py"
and so could create names conflicts. For example, Python-ast.h
defined a "Yield" macro which was conflict with the "Yield" name used
by the Windows <winbase.h> header.

Use the Python ast module instead.

* Move Include/asdl.h to Include/internal/pycore_asdl.h.
* Move Include/Python-ast.h to Include/internal/pycore_ast.h.
* Remove ast.h header file.
* pycore_symtable.h no longer includes Python-ast.h.
2021-03-23 20:47:40 +01:00
junyixie 88d9983b56
bpo-43551: Fix PyImport_Import() for subinterpreters (GH-24929)
Avoid static variables.
2021-03-22 10:47:10 +01:00
Antoine Pitrou 2fd16ef406
bpo-43517: Fix false positive in detection of circular imports (#24895) 2021-03-20 20:07:44 +01:00
Victor Stinner bcb094b41f
bpo-43268: Pass interp rather than tstate to internal functions (GH-24580)
Pass the current interpreter (interp) rather than the current Python
thread state (tstate) to internal functions which only use the
interpreter.

Modified functions:

* _PyXXX_Fini() and _PyXXX_ClearFreeList() functions
* _PyEval_SignalAsyncExc(), make_pending_calls()
* _PySys_GetObject(), sys_set_object(), sys_set_object_id(), sys_set_object_str()
* should_audit(), set_flags_from_config(), make_flags()
* _PyAtExit_Call()
* init_stdio_encoding()
* etc.
2021-02-19 15:10:45 +01:00
Victor Stinner 101bf69ff1
bpo-43268: _Py_IsMainInterpreter() now expects interp (GH-24577)
The _Py_IsMainInterpreter() function now expects interp rather than
tstate.
2021-02-19 13:33:31 +01:00
Serhiy Storchaka 4db8988420
bpo-41994: Fix refcount issues in Python/import.c (GH-22632)
https://bugs.python.org/issue41994
2021-01-12 15:43:32 +01:00
Victor Stinner 6223071421
bpo-1635741: Convert _imp to multi-phase init (GH-23378)
Convert the _imp extension module to the multi-phase initialization
API (PEP 489).

* Add _PyImport_BootstrapImp() which fix a bootstrap issue: import
  the _imp module before importlib is initialized.
* Add create_builtin() sub-function, used by _imp_create_builtin().
* Initialize PyInterpreterState.import_func earlier, in
  pycore_init_builtins().
* Remove references to _PyImport_Cleanup(). This function has been
  renamed to finalize_modules() and moved to pylifecycle.c.
2020-11-18 23:18:29 +01:00
Victor Stinner ef75a625cd
bpo-42260: Initialize time and warnings earlier at startup (GH-23249)
* Call _PyTime_Init() and _PyWarnings_InitState() earlier during the
  Python initialization.
* Inline _PyImportHooks_Init() into _PySys_InitCore().
* The _warnings initialization function no longer call
  _PyWarnings_InitState() to prevent resetting filters_version to 0.
* _PyWarnings_InitState() now returns an int and no longer clear the
  state in case of error (it's done anyway at Python exit).
* Rework init_importlib(), fix refleaks on errors.
2020-11-12 15:14:13 +01:00
Victor Stinner dff1ad5090
bpo-42208: Move _PyImport_Cleanup() to pylifecycle.c (GH-23040)
Move _PyImport_Cleanup() to pylifecycle.c, rename it to
finalize_modules(), split it (200 lines) into many smaller
sub-functions and cleanup the code.
2020-10-30 18:03:28 +01:00
Victor Stinner 8b3414818f
bpo-42208: Pass tstate to _PyGC_CollectNoFail() (GH-23038)
Move private _PyGC_CollectNoFail() to the internal C API.

Remove the private _PyGC_CollectIfEnabled() which was just an alias
to the public PyGC_Collect() function since Python 3.8.

Rename functions:

* collect() => gc_collect_main()
* collect_with_callback() => gc_collect_with_callback()
* collect_generations() => gc_collect_generations()
2020-10-30 17:00:00 +01:00
Serhiy Storchaka b510e101f8
bpo-42152: Use PyDict_Contains and PyDict_SetDefault if appropriate. (GH-22986)
If PyDict_GetItemWithError is only used to check whether the key is in dict,
it is better to use PyDict_Contains instead.

And if it is used in combination with PyDict_SetItem, PyDict_SetDefault can
replace the combination.
2020-10-26 12:47:57 +02:00
Serhiy Storchaka 8287aadb75
bpo-41993: Fix possible issues in remove_module() (GH-22631)
* PyMapping_HasKey() is not safe because it silences all exceptions and can return incorrect result.
* Informative exceptions from PyMapping_DelItem() are overridden with RuntimeError and
  the original exception raised before calling remove_module() is lost.
* There is a race condition between PyMapping_HasKey() and PyMapping_DelItem().
2020-10-11 16:51:07 +03:00
Victor Stinner 45b34a04a5
bpo-40232: _PyImport_ReInitLock() can now safely use its lock (GH-20597)
Since _PyImport_ReInitLock() now calls _PyThread_at_fork_reinit() on
the import lock, the lock is now in a known state: unlocked. It
became safe to acquire it after fork.
2020-06-02 17:13:49 +02:00
Victor Stinner 26881c8fae
PyOS_AfterFork_Child() uses PyStatus (GH-20596)
PyOS_AfterFork_Child() helper functions now return a PyStatus:
PyOS_AfterFork_Child() is now responsible to handle errors.

* Move _PySignal_AfterFork() to the internal C API
* Add #ifdef HAVE_FORK on _PyGILState_Reinit(), _PySignal_AfterFork()
  and _PyInterpreterState_DeleteExceptMain().
2020-06-02 15:51:37 +02:00
Robert Rouhani f40bd466bf
bpo-40417: Fix deprecation warning in PyImport_ReloadModule (GH-19750)
I can add another commit with the new test case I wrote to verify that the warning was being printed before my change, stopped printing after my change, and that the function does not return null after my change.

Automerge-Triggered-By: @brettcannon
2020-05-01 16:28:06 -07:00
Gregory Szorc 64224a4727
bpo-40412: Nullify inittab_copy during finalization (GH-19746)
Otherwise we leave a dangling pointer to free'd memory. If we
then initialize a new interpreter in the same process and call
PyImport_ExtendInittab, we will (likely) crash when calling
PyMem_RawRealloc(inittab_copy, ...) since the pointer address
is bogus.

Automerge-Triggered-By: @brettcannon
2020-05-01 11:07:54 -07:00
Victor Stinner 8852ad4208
bpo-40429: PyFrame_GetCode() now returns a strong reference (GH-19773) 2020-04-29 01:28:13 +02:00
Victor Stinner a42ca74fa3
bpo-40421: Add PyFrame_GetCode() function (GH-19757)
PyFrame_GetCode(frame): return a borrowed reference to the frame
code.

Replace frame->f_code with PyFrame_GetCode(frame) in most code,
except in frameobject.c, genobject.c and ceval.c.

Also add PyFrame_GetLineNumber() to the limited C API.
2020-04-28 19:01:31 +02:00
Victor Stinner 361dcdcefc
bpo-40268: Remove unused osdefs.h includes (GH-19532)
When the include is needed, add required symbol in a comment.
2020-04-15 03:24:57 +02:00
Victor Stinner d9ea5cae1d
bpo-40268: Remove unused pycore_pymem.h includes (GH-19531) 2020-04-15 02:57:50 +02:00
Victor Stinner 62183b8d6d
bpo-40268: Remove explicit pythread.h includes (#19529)
Remove explicit pythread.h includes: it is always included
by Python.h.
2020-04-15 02:04:42 +02:00
Dong-hee Na 62f75fe3dd
bpo-40232: Update PyOS_AfterFork_Child() to use _PyThread_at_fork_reinit() (GH-19450) 2020-04-15 01:16:24 +09:00
Victor Stinner e5014be049
bpo-40268: Remove a few pycore_pystate.h includes (GH-19510) 2020-04-14 17:52:15 +02:00
Victor Stinner 81a7be3fa2
bpo-40268: Rename _PyInterpreterState_GET_UNSAFE() (GH-19509)
Rename _PyInterpreterState_GET_UNSAFE() to _PyInterpreterState_GET()
for consistency with _PyThreadState_GET() and to have a shorter name
(help to fit into 80 columns).

Add also "assert(tstate != NULL);" to the function.
2020-04-14 15:14:01 +02:00
Victor Stinner 4a3fe08353
bpo-40268: Include explicitly pycore_interp.h (GH-19505)
pycore_pystate.h no longer includes pycore_interp.h:
it's now included explicitly in files accessing PyInterpreterState.
2020-04-14 14:26:24 +02:00
Victor Stinner da7933ecc3
bpo-40268: Add _PyInterpreterState_GetConfig() (GH-19492)
Don't access PyInterpreterState.config member directly anymore, but
use new functions:

* _PyInterpreterState_GetConfig()
* _PyInterpreterState_SetConfig()
* _Py_GetConfig()
2020-04-13 03:04:28 +02:00
Andy Lester fc2d8d62af
bpo-39943: Remove unnecessary casts in import.c that remove constness (GH-19209) 2020-03-30 13:19:14 -07:00
Victor Stinner 1c1e68cf3e
bpo-38644: Use _PySys_Audit(): pass tstate explicitly (GH-19183)
Add the dependency to tstate more explicit.
2020-03-27 15:11:45 +01:00
Victor Stinner 87d3b9db4a
bpo-39882: Add _Py_FatalErrorFormat() function (GH-19157) 2020-03-25 19:27:36 +01:00
Victor Stinner ff4584caca
bpo-39947: Use _PyInterpreterState_GET_UNSAFE() (GH-18978)
Replace _PyInterpreterState_Get() function call with
_PyInterpreterState_GET_UNSAFE() macro which is more efficient but
don't check if tstate or interp is NULL.

_Py_GetConfigsAsDict() now uses _PyThreadState_GET().
2020-03-13 18:03:56 +01:00
Victor Stinner 9e5d30cc99
bpo-39882: Py_FatalError() logs the function name (GH-18819)
The Py_FatalError() function is replaced with a macro which logs
automatically the name of the current function, unless the
Py_LIMITED_API macro is defined.

Changes:

* Add _Py_FatalErrorFunc() function.
* Remove the function name from the message of Py_FatalError() calls
  which included the function name.
* Update tests.
2020-03-07 00:54:20 +01:00
Petr Viktorin ffd9753a94
bpo-39245: Switch to public API for Vectorcall (GH-18460)
The bulk of this patch was generated automatically with:

    for name in \
        PyObject_Vectorcall \
        Py_TPFLAGS_HAVE_VECTORCALL \
        PyObject_VectorcallMethod \
        PyVectorcall_Function \
        PyObject_CallOneArg \
        PyObject_CallMethodNoArgs \
        PyObject_CallMethodOneArg \
    ;
    do
        echo $name
        git grep -lwz _$name | xargs -0 sed -i "s/\b_$name\b/$name/g"
    done

    old=_PyObject_FastCallDict
    new=PyObject_VectorcallDict
    git grep -lwz $old | xargs -0 sed -i "s/\b$old\b/$new/g"

and then cleaned up:

- Revert changes to in docs & news
- Revert changes to backcompat defines in headers
- Nudge misaligned comments
2020-02-11 17:46:57 +01:00
Eddie Elizondo 4590f72259
bpo-38076 Clear the interpreter state only after clearing module globals (GH-18039)
Currently, during runtime destruction, `_PyImport_Cleanup` is clearing the interpreter state before clearing out the modules themselves. This leads to a segfault on modules that rely on the module state to clear themselves up.

For example, let's take the small snippet added in the issue by @DinoV :
```
import _struct

class C:
    def __init__(self):
        self.pack = _struct.pack
    def __del__(self):
        self.pack('I', -42)

_struct.x = C()
```

The module `_struct` uses the module state to run `pack`. Therefore, the module state has to be alive until after the module has been cleared out to successfully run `C.__del__`. This happens at line 606, when `_PyImport_Cleanup` calls `_PyModule_Clear`. In fact, the loop that calls `_PyModule_Clear` has in its comments: 

> Now, if there are any modules left alive, clear their globals to minimize potential leaks.  All C extension modules actually end up here, since they are kept alive in the interpreter state.

That means that we can't clear the module state (which is used by C Extensions) before we run that loop.

Moving `_PyInterpreterState_ClearModules` until after it, fixes the segfault in the code snippet.

Finally, this updates a test in `io` to correctly assert the error that it now throws (since it now finds the io module state). The test that uses this is: `test_create_at_shutdown_without_encoding`. Given this test is now working is a proof that the module state now stays alive even when `__del__` is called at module destruction time. Thus, I didn't add a new tests for this.


https://bugs.python.org/issue38076
2020-02-04 02:29:25 -08:00
Hai Shi 46874c26ee
bpo-39487: Merge duplicated _Py_IDENTIFIER identifiers in C code (GH-18254)
Moving repetitive `_Py_IDENTIFIER` instances to a global location helps identify them more easily in regards to sub-interpreter support.
2020-01-30 15:20:25 -08:00
David Carlier aabdeb766b bpo-38960: DTrace build fix for FreeBSD. (GH-17451)
DTrace build fix for FreeBSD.

- allowing passing an extra flag as it need to define the arch size.
- casting some probe's arguments.
2020-01-28 13:53:32 +01:00
Victor Stinner 2582d46fbc
bpo-38858: new_interpreter() reuses pycore_init_builtins() (GH-17351)
new_interpreter() now calls _PyBuiltin_Init() to create the builtins
module and calls _PyImport_FixupBuiltin(), rather than using
_PyImport_FindBuiltin(tstate, "builtins").

pycore_init_builtins() is now responsible to initialize
intepr->builtins_copy: inline _PyImport_Init() and remove this
function.
2019-11-22 19:24:49 +01:00
Victor Stinner 82c83bd907
bpo-38858: _PyImport_FixupExtensionObject() handles subinterpreters (GH-17350)
If _PyImport_FixupExtensionObject() is called from a subinterpreter,
leave extensions unchanged and don't copy the module dictionary
into def->m_base.m_copy.
2019-11-22 18:52:27 +01:00
Victor Stinner 67e0de6f0b
bpo-36854: gcmodule.c gets its state from tstate (GH-17285)
* Add GCState type for readability
* gcmodule.c now gets its gcstate from tstate
* _PyGC_DumpShutdownStats() now expects tstate rather than runtime
* Rename "state" to "gcstate" for readability: to avoid confusion
  between "state" and "tstate" for example.
* collect() now only expects tstate: it gets gcstate from tstate.
* Pass tstate to _PyErr_xxx() functions
2019-11-20 11:48:18 +01:00
Victor Stinner 61691d8336
bpo-38353: Cleanup includes in the internal C API (GH-16548)
Use forward declaration of types to avoid includes in the internal C
API. Add also comment to justify other includes.
2019-10-02 23:51:20 +02:00
Joannah Nanjekye 37c2220698 bpo-35943: Prevent PyImport_GetModule() from returning a partially-initialized module (GH-15057) 2019-09-11 13:47:39 +01:00
Ben Lewis 92420b3e67 bpo-37409: fix relative import with no parent (#14956)
Relative imports use resolve_name to get the absolute target name,
which first seeks the current module's absolute package name from the globals:
If __package__ (and __spec__.parent) are missing then
import uses __name__, truncating the last segment if
the module is a submodule rather than a package __init__.py
(which it guesses from whether __path__ is defined).

The __name__ attempt should fail if there is no parent package (top level modules),
if __name__ is '__main__' (-m entry points), or both (scripts).
That is, if both __name__ has no subcomponents and the module does not seem
to be a package __init__ module then import should fail.
2019-09-11 11:09:47 +01:00
Ngalim Siregar c5fa44944e bpo-37444: Update differing exception between builtins and importlib (GH-14869)
Imports now raise `TypeError` instead of `ValueError` for relative import failures. This makes things consistent between `builtins.__import__` and `importlib.__import__` as well as using a more natural import for the failure.


https://bugs.python.org/issue37444



Automerge-Triggered-By: @brettcannon
2019-08-02 22:46:02 -07:00
Min ho Kim c4cacc8c5e Fix typos in comments, docs and test names (#15018)
* Fix typos in comments, docs and test names

* Update test_pyparse.py

account for change in string length

* Apply suggestion: splitable -> splittable

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Apply suggestion: splitable -> splittable

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Apply suggestion: Dealloccte -> Deallocate

Co-Authored-By: Terry Jan Reedy <tjreedy@udel.edu>

* Update posixmodule checksum.

* Reverse idlelib changes.
2019-07-30 18:16:13 -04:00
Jeroen Demeyer 59ad110d7a bpo-37547: add _PyObject_CallMethodOneArg (GH-14685) 2019-07-11 17:59:05 +09:00
Jeroen Demeyer 762f93ff2e bpo-37337: Add _PyObject_CallMethodNoArgs() (GH-14267) 2019-07-08 17:19:25 +09:00
Jeroen Demeyer 196a530e00 bpo-37483: add _PyObject_CallOneArg() function (#14558) 2019-07-04 19:31:34 +09:00
Victor Stinner b45d259bdd
bpo-36710: Use tstate in pylifecycle.c (GH-14249)
In pylifecycle.c: pass tstate argument, rather than interp argument,
to functions.
2019-06-20 00:05:23 +02:00
Victor Stinner 987a0dcfa1
bpo-36710: Remove PyImport_Cleanup() function (GH-14221)
* Rename PyImport_Cleanup() to _PyImport_Cleanup() and move it to the
  internal C API. Add 'tstate' parameters.
* Remove documentation of _PyImport_Init(), PyImport_Cleanup(),
  _PyImport_Fini(). All three were documented as "For internal use
  only.".
2019-06-19 10:36:10 +02:00
Victor Stinner 0a28f8d379
bpo-36710: Add tstate parameter in import.c (GH-14218)
* Add 'tstate' parameter to many internal import.c functions.
* _PyImportZip_Init() now gets 'tstate' parameter rather than
  'interp'.
* Add 'interp' parameter to _PyState_ClearModules() and rename it
  to _PyInterpreterState_ClearModules().
* Move private _PyImport_FindBuiltin() to the internal C API; add
  'tstate' parameter to it.
* Remove private _PyImport_AddModuleObject() from the C API:
  use public PyImport_AddModuleObject() instead.
* Remove private _PyImport_FindExtensionObjectEx() from the C API:
  use private _PyImport_FindExtensionObject() instead.
2019-06-19 02:54:39 +02:00
Victor Stinner 0fd2c300c2
Revert "bpo-36818: Add PyInterpreterState.runtime field. (gh-13129)" (GH-13795)
This reverts commit 396e0a8d9d.
2019-06-04 03:15:09 +02:00
Eric Snow 396e0a8d9d
bpo-36818: Add PyInterpreterState.runtime field. (gh-13129)
https://bugs.python.org/issue36818
2019-05-31 21:16:47 -06:00
Zackery Spytz 249b7d59d8 bpo-20602: Do not clear sys.flags and sys.float_info during shutdown (GH-8096)
There is no need to clear these immutable objects during shutdown.
2019-05-30 13:08:24 +02:00
Victor Stinner 331a6a56e9
bpo-36763: Implement the PEP 587 (GH-13592)
* Add a whole new documentation page:
  "Python Initialization Configuration"
* PyWideStringList_Append() return type is now PyStatus,
  instead of int
* PyInterpreterState_New() now calls PyConfig_Clear() if
  PyConfig_InitPythonConfig() fails.
* Rename files:

  * Python/coreconfig.c => Python/initconfig.c
  * Include/cpython/coreconfig.h => Include/cpython/initconfig.h
  * Include/internal/: pycore_coreconfig.h => pycore_initconfig.h

* Rename structures

  * _PyCoreConfig => PyConfig
  * _PyPreConfig => PyPreConfig
  * _PyInitError => PyStatus
  * _PyWstrList => PyWideStringList

* Rename PyConfig fields:

  * use_module_search_paths => module_search_paths_set
  * module_search_path_env => pythonpath_env

* Rename PyStatus field: _func => func
* PyInterpreterState: rename core_config field to config
* Rename macros and functions:

  * _PyCoreConfig_SetArgv() => PyConfig_SetBytesArgv()
  * _PyCoreConfig_SetWideArgv() => PyConfig_SetArgv()
  * _PyCoreConfig_DecodeLocale() => PyConfig_SetBytesString()
  * _PyInitError_Failed() => PyStatus_Exception()
  * _Py_INIT_ERROR_TYPE_xxx enums => _PyStatus_TYPE_xxx
  * _Py_UnixMain() => Py_BytesMain()
  * _Py_ExitInitError() => Py_ExitStatusException()
  * _Py_PreInitializeFromArgs() => Py_PreInitializeFromBytesArgs()
  * _Py_PreInitializeFromWideArgs() => Py_PreInitializeFromArgs()
  * _Py_PreInitialize() => Py_PreInitialize()
  * _Py_RunMain() => Py_RunMain()
  * _Py_InitializeFromConfig() => Py_InitializeFromConfig()
  * _Py_INIT_XXX() => _PyStatus_XXX()
  * _Py_INIT_FAILED() => _PyStatus_EXCEPTION()

* Rename 'err' PyStatus variables to 'status'
* Convert RUN_CODE() macro to config_run_code() static inline function
* Remove functions:

  * _Py_InitializeFromArgs()
  * _Py_InitializeFromWideArgs()
  * _PyInterpreterState_GetCoreConfig()
2019-05-27 16:39:22 +02:00
Steve Dower b82e17e626
bpo-36842: Implement PEP 578 (GH-12613)
Adds sys.audit, sys.addaudithook, io.open_code, and associated C APIs.
2019-05-23 08:45:22 -07:00
Victor Stinner 410b85a7f7
bpo-36900: import.c uses PyInterpreterState.core_config (GH-13278)
Move _PyImportZip_Init() to the internal C API and add an 'interp'
parameter.
2019-05-13 17:12:45 +02:00
Zackery Spytz 94a64e9cd4 bpo-24048: Save the live exception during import.c's remove_module() (GH-13005)
Save the live exception during the course of remove_module().
2019-05-08 12:31:23 -04:00
Victor Stinner cb9fbd3588
bpo-36763: Make _PyCoreConfig.check_hash_pycs_mode public (GH-13052)
_PyCoreConfig: Rename _check_hash_pycs_mode field to
check_hash_pycs_mode (make it public) and change its type from "const
char*" to "wchar_t*".
2019-05-01 23:51:56 -04:00
Victor Stinner 9db0324712
bpo-36710: Add runtime parameter in gcmodule.c (GH-12958)
Add 'state' or 'runtime' parameter to functions in gcmodule.c to
avoid to rely directly on the global variable _PyRuntime.
2019-04-26 02:32:01 +02:00
Stefan Krah 027b09c5a1
bpo-36370: Check for PyErr_Occurred() after PyImport_GetModule() (GH-12504) 2019-03-25 21:50:58 +01:00
Stéphane Wirtel 0d765e3849 bpo-36362: Avoid unused variables when HAVE_DYNAMIC_LOADING is not defined (GH-12430)
https://bugs.python.org/issue36362
2019-03-19 16:37:20 -07:00
Serhiy Storchaka a24107b04c
bpo-35459: Use PyDict_GetItemWithError() instead of PyDict_GetItem(). (GH-11112) 2019-02-25 17:59:46 +02:00
Zackery Spytz 89c4f90df9 bpo-35470: Fix a reference counting bug in _PyImport_FindExtensionObjectEx(). (GH-11128) 2019-01-10 18:12:31 +02:00
Serhiy Storchaka 8905fcc85a
bpo-35454: Fix miscellaneous minor issues in error handling. (#11077)
* bpo-35454: Fix miscellaneous minor issues in error handling.

* Fix a null pointer dereference.
2018-12-11 08:38:03 +02:00
Victor Stinner 3bb183d7fb
bpo-35177, Python-ast.h: Fix "Yield" compiler warning (GH-10664)
Partially revert commit 5f2df88b63e50d23914e97ec778861a52abdeaad:
add "#undef Yield" to .c files after including Python-ast.h.

Fix the warning:

    winbase.h(102): warning C4005: 'Yield': macro redefinition
2018-11-22 18:38:38 +01:00
Victor Stinner 621cebe81b
bpo-35081: Rename internal headers (GH-10275)
Rename Include/internal/ headers:

* pycore_hash.h -> pycore_pyhash.h
* pycore_lifecycle.h -> pycore_pylifecycle.h
* pycore_mem.h -> pycore_pymem.h
* pycore_state.h -> pycore_pystate.h

Add missing headers to Makefile.pre.in and PCbuild:

* pycore_condvar.h.
* pycore_hamt.h
* pycore_pyhash.h
2018-11-12 16:53:38 +01:00
Victor Stinner 5f2df88b63
bpo-35177: Add dependencies between header files (GH-10361)
* ast.h now includes Python-ast.h and node.h
* parsetok.h now includes node.h and grammar.h
* symtable.h now includes Python-ast.h
* Modify asdl_c.py to enhance Python-ast.h:

  * Add #ifndef/#define Py_PYTHON_AST_H to be able to include the header
    twice
  * Add "extern { ... }" for C++
  * Undefine "Yield" macro conflicting with winbase.h

* Remove "#undef Yield" from C files, it's now done in Python-ast.h
* Remove now useless includes in C files
2018-11-12 00:56:19 +01:00
Serhiy Storchaka 34fd4c2019
bpo-35133: Fix mistakes when concatenate string literals on different lines. (GH-10284)
Two kind of mistakes:

1. Missed space. After concatenating there is no space between words.

2. Missed comma. Causes unintentional concatenating in a list of strings.
2018-11-05 16:20:25 +02:00
Victor Stinner a1c249c405
bpo-35081: And pycore_lifecycle.h and pycore_pathconfig.h (GH-10273)
* And pycore_lifecycle.h and pycore_pathconfig.h headers to
  Include/internal/
* Move Py_BUILD_CORE specific code from coreconfig.h and
  pylifecycle.h to pycore_pathconfig.h and pycore_lifecycle.h
* Move _Py_wstrlist_XXX() definitions and _PyPathConfig code
  from pycore_state.h to pycore_pathconfig.h
* Move "Init" and "Fini" function definitions from pylifecycle.c to
  pycore_lifecycle.h.
2018-11-01 03:15:58 +01:00
Victor Stinner 27e2d1f219
bpo-35081: Add pycore_ prefix to internal header files (GH-10263)
* Rename Include/internal/ header files:

  * pyatomic.h -> pycore_atomic.h
  * ceval.h -> pycore_ceval.h
  * condvar.h -> pycore_condvar.h
  * context.h -> pycore_context.h
  * pygetopt.h -> pycore_getopt.h
  * gil.h -> pycore_gil.h
  * hamt.h -> pycore_hamt.h
  * hash.h -> pycore_hash.h
  * mem.h -> pycore_mem.h
  * pystate.h -> pycore_state.h
  * warnings.h -> pycore_warnings.h

* PCbuild project, Makefile.pre.in, Modules/Setup: add the
  Include/internal/ directory to the search paths of header files.
* Update includes. For example, replace #include "internal/mem.h"
  with #include "pycore_mem.h".
2018-11-01 00:52:28 +01:00
Victor Stinner 2be00d987d
bpo-35081: Move Py_BUILD_CORE code to internal/mem.h (GH-10249)
* Add #include "internal/mem.h" to C files using
  _PyMem_SetDefaultAllocator().
* Include/internal/mem.h now requires Py_BUILD_CORE to be defined.
2018-10-31 20:19:24 +01:00
Serhiy Storchaka 3e429dcc24
bpo-33237: Improve AttributeError message for partially initialized module. (GH-6398) 2018-10-30 13:19:51 +02:00
Serhiy Storchaka c93c58b5d5
bpo-33331: Clean modules in the reversed order in PyImport_Cleanup(). (GH-6565)
Modules imported last are now cleared first at interpreter shutdown.

A newly imported module is moved to the end of sys.modules, behind
modules on which it depends.
2018-10-29 19:30:16 +02:00
Benjamin Peterson e502451781
closes bpo-34646: Remove PyAPI_* macros from declarations. (GH-9218) 2018-09-12 12:06:42 -07:00
Victor Stinner caba55b3b7
bpo-34301: Add _PyInterpreterState_Get() helper function (GH-8592)
sys_setcheckinterval() now uses a local variable to parse arguments,
before writing into interp->check_interval.
2018-08-03 15:33:52 +02:00
Victor Stinner 80b762f010
bpo-31650: Remove _Py_CheckHashBasedPycsMode global config var (GH-8608)
bpo-31650, bpo-34170: Replace _Py_CheckHashBasedPycsMode with
_PyCoreConfig._check_hash_pycs_mode. Modify PyInit__imp() and
zipimport to get the parameter from the current interpreter core
configuration.

Remove Include/internal/import.h file.
2018-08-01 18:18:07 +02:00
Victor Stinner 6c785c0ebd
bpo-34170: Add Python/coreconfig.c for _PyCoreConfig (GH-8607)
* Add Include/coreconfig.h
* Move config_*() and _PyCoreConfig_*() functions from Modules/main.c
  to a new Python/coreconfig.c file.
* Inline _Py_ReadHashSeed() into config_init_hash_seed()
* Move global configuration variables to coreconfig.c
2018-08-01 17:56:14 +02:00
ukwksk 5e6312c39e bpo-33443 Fix typo in Python/import.c (GH-6722) 2018-05-14 12:10:52 -07:00
Serhiy Storchaka c1a6832f50
bpo-33330: Write exceptions occurred in PyImport_Cleanup() to stderr. (GH-6606)
They where silenced before.
2018-04-29 22:16:30 +03:00
Serhiy Storchaka e9d9494d6b
bpo-33330: Improve error handling in PyImport_Cleanup(). (GH-6564) 2018-04-25 20:58:40 +03:00
Serhiy Storchaka 4e2442505c
bpo-32946: Speed up "from ... import ..." from non-packages. (GH-5873) 2018-03-11 10:52:37 +02:00
Benjamin Peterson c65ef772c3
rename _imp initialization function to follow conventions (#5432)
When the C imp module became _imp in 6f44d66bc4, the initialization function should have been renamed from PyInit_imp to PyInit__imp.
2018-01-29 11:33:57 -08:00
Benjamin Peterson 0c644fcda0
fix up signedness in PyImport_ExtendInittab (#4831)
As a result of 92a3c6f493, the compiler complains:

Python/import.c:2311:21: warning: comparison of integers of different signs: 'long' and 'unsigned long' [-Wsign-compare]
    if ((i + n + 1) <= PY_SSIZE_T_MAX / sizeof(struct _inittab)) {
         ~~~~~~~~~  ^  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This overflow is extremely unlikely to happen, but let's avoid undefined
behavior anyway.
2017-12-15 23:42:33 -08:00
Victor Stinner d233796f7d
import.c: Fix a GCC warning (#4822)
Fix the warning:

Python/import.c: warning: comparison between signed and unsigned integer expressions
     if ((i + n + 1) <= PY_SSIZE_T_MAX / sizeof(struct _inittab)) {
2017-12-12 23:29:28 +01:00
Benjamin Peterson 83620773ee
fix my byte-swapping implementation (#4772) 2017-12-09 12:18:56 -08:00
Benjamin Peterson 42aa93b8ff
closes bpo-31650: PEP 552 (Deterministic pycs) implementation (#4575)
Python now supports checking bytecode cache up-to-dateness with a hash of the
source contents rather than volatile source metadata. See the PEP for details.

While a fairly straightforward idea, quite a lot of code had to be modified due
to the pervasiveness of pyc implementation details in the codebase. Changes in
this commit include:

- The core changes to importlib to understand how to read, validate, and
  regenerate hash-based pycs.

- Support for generating hash-based pycs in py_compile and compileall.

- Modifications to our siphash implementation to support passing a custom
  key. We then expose it to importlib through _imp.

- Updates to all places in the interpreter, standard library, and tests that
  manually generate or parse pyc files to grok the new format.

- Support in the interpreter command line code for long options like
  --check-hash-based-pycs.

- Tests and documentation for all of the above.
2017-12-09 10:26:52 -08:00
Neil Schemenauer 11cc289490
Fix missing DECREF of mod. (#4749) 2017-12-07 16:24:59 -08:00
Victor Stinner 92a3c6f493
bpo-32030: Add _PyImport_Fini2() (#4737)
PyImport_ExtendInittab() now uses PyMem_RawRealloc() rather than
PyMem_Realloc(). PyImport_ExtendInittab() can be called before
Py_Initialize() whereas only the PyMem_Raw allocator is supposed to
be used before Py_Initialize().

Add _PyImport_Fini2() to release the memory allocated by
PyImport_ExtendInittab() at exit. PyImport_ExtendInittab() now forces
the usage of the default raw allocator, to be able to release memory
in _PyImport_Fini2().

Don't export these functions anymore to be C API, only to
Py_BUILD_CORE:

* _PyExc_Fini()
* _PyImport_Fini()
* _PyGC_DumpShutdownStats()
* _PyGC_Fini()
* _PyType_Fini()
* _Py_HashRandomization_Fini()
2017-12-06 18:12:59 +01:00
Victor Stinner 672b6baa71
bpo-32030: pass interp to _PyImport_Init() (#4736)
Remove also the initstr variable, unused since the commit
e69f0df45b pushed in 2012: "bpo-13959:
Re-implement imp.find_module() in Lib/imp.py"
2017-12-06 17:25:50 +01:00