gh-117378: Fix multiprocessing forkserver preload sys.path inheritance. (GH-126538)
`sys.path` was not properly being sent from the parent process when launching
the multiprocessing forkserver process to preload imports. This bug has been
there since the forkserver start method was introduced in Python 3.4. It was
always _supposed_ to inherit `sys.path` the same way the spawn method does.
Observable behavior change: A `''` value in `sys.path` will now be replaced in
the forkserver's `sys.path` with an absolute pathname
`os.path.abspath(os.getcwd())` saved at the time that `multiprocessing` was
imported in the parent process as it already was when using the spawn start
method. **This will only be observable during forkserver preload imports**.
The code invoked before calling things in another process already correctly sets `sys.path`.
Which is likely why this went unnoticed for so long as a mere performance issue in
some configurations.
A workaround for the bug on impacted Pythons is to set PYTHONPATH in the
environment before multiprocessing's forkserver process was started. Not perfect
as that is then inherited by other children, etc, but likely good enough for many
people's purposes.
(cherry picked from commit 9d08423b6e)
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Add os.waitstatus_to_exitcode() function to convert a wait status to an
exitcode.
Suggest waitstatus_to_exitcode() usage in the documentation when
appropriate.
Use waitstatus_to_exitcode() in:
* multiprocessing, os, subprocess and _bootsubprocess modules;
* test.support.wait_process();
* setup.py: run_command();
* and many tests.
multiprocessing tests now stop the ForkServer instance if it's
running: close the "alive" file descriptor to ask the server to stop
and then remove its UNIX address.
The multiprocessing.resource_tracker replaces the multiprocessing.semaphore_tracker module. Other than semaphores, resource_tracker also tracks shared_memory segments. Patch by Pierre Glaser.
* bpo-31308: If multiprocessing's forkserver dies, launch it again when necessary.
* Fix test on Windows
* Add NEWS entry
* Adopt a different approach: ignore SIGINT and SIGTERM, as in semaphore tracker.
* Fix comment
* Make sure the test doesn't muck with process state
* Also test previously-started processes
* Update 2017-08-30-17-59-36.bpo-31308.KbexyC.rst
* Avoid masking SIGTERM in forkserver. It's not necessary and causes a race condition in test_many_processes.
* Make error message more informative
Replace assertions in error-reporting code with more-informative version that doesn't cause confusion over where and what the error is.
* Additional clarification + get travis to check
* Change from SystemError to TypeError
As suggested in PR comment by @pitrou, changing from SystemError; TypeError appears appropriate.
* NEWS file installation; ACKS addition (will do my best to justify it by additional work)
* Making current AssertionErrors in multiprocessing more informative
* Blurb added re multiprocessing managers.py, queues.py cleanup
* Further multiprocessing cleanup - went through pool.py
* Fix two asserts in multiprocessing/util.py
* Most asserts in multiprocessing more informative
* Didn't save right version
* Further work on multiprocessing error messages
* Correct typo
* Correct typo v2
* Blasted colon... serves me right for trying to work on two things at once
* Simplify NEWS entry
* Update 2017-08-18-17-16-38.bpo-5001.gwnthq.rst
* Update 2017-08-18-17-16-38.bpo-5001.gwnthq.rst
OK, never mind.
* Corrected (thanks to pitrou) error messages for notify
* Remove extraneous backslash in docstring.
* bpo-26732: fix too many fds in processes started with the "forkserver" method
A child process would inherit as many fds as the number of still-running children.
* Add blurb and test comment
* Fix bpo-30596: Add close() method to multiprocessing.Process
* Raise ValueError if close() is called before the Process is finished running
* Add docs
* Add NEWS blurb
* Fix race condition in signal wakeup in forkserver (followup to PR #1989)
There's an admittedly well-known race condition where ECHILD can arrive
just before the C function epoll_wait() and the latter wouldn't therefore
return EINTR. The solution is to use set_wakeup_fd(), which was designed
to avoid such race conditions.
* Reset wakeup fd in child
* bpo-16500: Allow registering at-fork handlers
* Address Serhiy's comments
* Add doc for new C API
* Add doc for new Python-facing function
* Add NEWS entry + doc nit
* multiprocessing: open file with closefd=False to avoid ResourceWarning
* _test_multiprocessing: open file with O_EXCL to detect bugs in tests (if a
previous test forgot to remove TESTFN)
* test_sys_exit(): remove TESTFN after each loop iteration
Initial patch written by Serhiy Storchaka.
Miss Islington (bot)
Tian Gao
Dong-hee Na
Victor Stinner
Pablo Galindo
Victor Stinner
Thomas Moreau
Pierre Glaser
Antoine Pitrou
Allen W. Smith, Ph.D
Jon Dufresne
Davin Potts
Victor Stinner
Charles-François Natali
Richard Oudkerk