Commit Graph

74 Commits

Author SHA1 Message Date
Antoine Pitrou bdb1cf1ca5 Issue #12328: Fix multiprocessing's use of overlapped I/O on Windows.
Also, add a multiprocessing.connection.wait(rlist, timeout=None) function
for polling multiple objects at once.  Patch by sbt.

Complete changelist from sbt's patch:

* Adds a wait(rlist, timeout=None) function for polling multiple
  objects at once.  On Unix this is just a wrapper for
  select(rlist, [], [], timeout=None).

* Removes use of the SentinelReady exception and the sentinels argument
  to certain methods.  concurrent.futures.process has been changed to
  use wait() instead of SentinelReady.

* Fixes bugs concerning PipeConnection.poll() and messages of zero
  length.

* Fixes PipeListener.accept() to call ConnectNamedPipe() with
  overlapped=True.

* Fixes Queue.empty() and SimpleQueue.empty() so that they are
  threadsafe on Windows.

* Now PipeConnection.poll() and wait() will not modify the pipe except
  possibly by consuming a zero length message.  (Previously poll()
  could consume a partial message.)

* All of multiprocesing's pipe related blocking functions/methods are
  now interruptible by SIGINT on Windows.
2012-03-05 19:28:37 +01:00
Charles-François Natali ed4a8fc095 Issue #8184: multiprocessing: On Windows, don't set SO_REUSEADDR on Connection
sockets, and set FILE_FLAG_FIRST_PIPE_INSTANCE on named pipes, to make sure two
listeners can't bind to the same socket/pipe (or any existing socket/pipe).
2012-02-08 21:15:58 +01:00
Charles-François Natali edc67feaa4 Issue #8184: Fix a potential file descriptor leak when a
multiprocessing.Connection socket can't be bound.
2012-02-04 15:12:08 +01:00
Charles-François Natali 992ca5278e Issue #8184: Fix a potential file descriptor leak when a
multiprocessing.Connection socket can't be bound.
2012-02-04 14:55:53 +01:00
Charles-François Natali e6eabd4345 Issue #13215: multiprocessing.Connection: don't hammer the remote end with
retries in case of ECONNREFUSED.
2011-11-19 09:59:43 +01:00
Antoine Pitrou 6b4883dec0 PEP 3151 / issue #12555: reworking the OS and IO exception hierarchy. 2011-10-12 02:54:14 +02:00
Charles-François Natali 225aa4f8ec Issue #12996: multiprocessing.connection: transmit the header in network byte
order (endpoints machines can have different endianness).
2011-09-20 19:27:39 +02:00
Antoine Pitrou 600012028e Add comment 2011-07-09 01:03:46 +02:00
Antoine Pitrou 71a28a9174 Rebind locally the globals which can be looked up at shutdown
(to avoid the warnings seen on a buildbot)
2011-07-09 01:03:00 +02:00
Antoine Pitrou dd69649660 Issue #9205: concurrent.futures.ProcessPoolExecutor now detects killed
children and raises BrokenProcessPool in such a situation.  Previously it
would reliably freeze/deadlock.
2011-06-08 17:21:55 +02:00
Antoine Pitrou 5aa878c61a os.dup()-then-close() can be replaced with detach() 2011-05-09 21:00:28 +02:00
Antoine Pitrou 87cf220972 Issue #11743: Rewrite multiprocessing connection classes in pure Python. 2011-05-09 17:04:27 +02:00
Marc-André Lemburg 8f36af7a4c Normalize the encoding names for Latin-1 and UTF-8 to
'latin-1' and 'utf-8'.

These are optimized in the Python Unicode implementation
to result in more direct processing, bypassing the codec
registry.

Also see issue11303.
2011-02-25 15:42:01 +00:00
Victor Stinner 2b69506c79 Issue #10816: multiprocessing.SocketClient() closes the socket on error
Use a context manager to close immediatly the socket on error.
2011-01-03 15:47:59 +00:00
R. David Murray 3fc969a4a2 9162: fix license in multiprocessing files 2010-12-14 01:38:16 +00:00
Brian Curtin a6a3274aad Fix #9513 to remove relative imports from multiprocessing.
The test suite currently skips test_multiprocessing on Windows
because it fails on finding _multiprocessing in several
win32-specific blocks. Removing the relative import lets this
through and allows the test to run (and pass).
2010-08-04 15:47:24 +00:00
Antoine Pitrou 45d61a341d Merged revisions 76245 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r76245 | antoine.pitrou | 2009-11-13 23:31:18 +0100 (ven., 13 nov. 2009) | 6 lines

  Issue #7318: multiprocessing now uses a timeout when it fails to establish
  a connection with another process, rather than looping endlessly. The
  default timeout is 20 seconds, which should be amply sufficient for
  local connections.
........
2009-11-13 22:35:18 +00:00
Jesse Noller c5d28a0b03 Merge 70717 to 30maint 2009-03-30 16:37:36 +00:00
Neal Norwitz 5d6415ee26 Merged revisions 66023 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r66023 | neal.norwitz | 2008-08-24 18:50:24 -0700 (Sun, 24 Aug 2008) | 6 lines

  Fix problem reported by pychecker where AuthenticationError wasn't imported.
  Add some test coverage to this code.  More tests should be added (TODO added).

  R=Brett
  TESTED=./python -E -tt ./Lib/test/regrtest.py test_multiprocessing
........
2008-08-25 01:53:32 +00:00
Georg Brandl 6aa2d1fec7 Merged revisions 65459,65472,65481,65518,65536,65581,65609,65637,65641,65644-65645 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r65459 | gregory.p.smith | 2008-08-04 00:13:29 +0000 (Mon, 04 Aug 2008) | 4 lines

  - Issue #1857: subprocess.Popen.poll gained an additional _deadstate keyword
    argument in python 2.5, this broke code that subclassed Popen to include its
    own poll method.  Fixed my moving _deadstate to an _internal_poll method.
........
  r65472 | andrew.kuchling | 2008-08-04 01:43:43 +0000 (Mon, 04 Aug 2008) | 3 lines

  Bug 3228: Explicitly supply the file mode to avoid creating executable files,
  and add corresponding tests.
  Possible 2.5 backport candidate
........
  r65481 | gregory.p.smith | 2008-08-04 07:33:37 +0000 (Mon, 04 Aug 2008) | 22 lines

  Adds a sanity check to avoid a *very rare* infinite loop due to a corrupt tls
  key list data structure in the thread startup path.

  This change is a companion to r60148 which already successfully dealt with a
  similar issue on thread shutdown.

  In particular this loop has been observed happening from this call path:
   #0  in find_key ()
   #1  in PyThread_set_key_value ()
   #2  in _PyGILState_NoteThreadState ()
   #3  in PyThreadState_New ()
   #4  in t_bootstrap ()
   #5  in pthread_start_thread ()

  I don't know how this happens but it does, *very* rarely.  On more than
  one hardware platform.  I have not been able to reproduce it manually.
  (A flaky mutex implementation on the system in question is one hypothesis).

  As with r60148, the spinning we managed to observe in the wild was due to a
  single list element pointing back upon itself.
........
  r65518 | mark.dickinson | 2008-08-04 21:30:09 +0000 (Mon, 04 Aug 2008) | 7 lines

  Issue #1481296: (again!) Make conversion of a float NaN to an int or
  long raise ValueError instead of returning 0.  Also, change the error
  message for conversion of an infinity to an integer, replacing 'long' by
  'integer', so that it's appropriate for both long(float('inf')) and
  int(float('inf')).
........
  r65536 | andrew.kuchling | 2008-08-05 01:00:57 +0000 (Tue, 05 Aug 2008) | 1 line

  Bug 3228: take a test from Niels Gustaebel's patch, and based on his patch, check for having os.stat available
........
  r65581 | guido.van.rossum | 2008-08-07 18:51:38 +0000 (Thu, 07 Aug 2008) | 3 lines

  Patch by Ian Charnas from issue 3517.
  Add F_FULLFSYNC if it exists (OS X only so far).
........
  r65609 | antoine.pitrou | 2008-08-09 17:22:25 +0000 (Sat, 09 Aug 2008) | 3 lines

  #3205: bz2 iterator fails silently on MemoryError
........
  r65637 | georg.brandl | 2008-08-11 09:07:59 +0000 (Mon, 11 Aug 2008) | 3 lines

  - Issue #3537: Fix an assertion failure when an empty but presized dict
    object was stored in the freelist.
........
  r65641 | jesse.noller | 2008-08-11 14:28:07 +0000 (Mon, 11 Aug 2008) | 2 lines

  Remove the fqdn call for issue 3270
........
  r65644 | antoine.pitrou | 2008-08-11 17:21:36 +0000 (Mon, 11 Aug 2008) | 3 lines

  #3134: shutil referenced undefined WindowsError symbol
........
  r65645 | jesse.noller | 2008-08-11 19:00:15 +0000 (Mon, 11 Aug 2008) | 2 lines

  Fix the connection refused error part of issue 3419, use errno module instead of a static list of possible connection refused messages.
........
2008-08-12 08:35:52 +00:00
Georg Brandl 734e268062 Merged revisions 65437,65469,65476,65480,65502,65528,65539,65543,65558,65561-65562,65565,65591,65601,65608,65610,65639 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r65437 | georg.brandl | 2008-08-03 22:28:55 +0000 (Sun, 03 Aug 2008) | 2 lines

  Note the removal of several committers.
........
  r65469 | gregory.p.smith | 2008-08-04 01:03:50 +0000 (Mon, 04 Aug 2008) | 3 lines

  issue1606: Add warnings to the subprocess documentation about common pitfalls
  of using pipes that cause deadlocks.
........
  r65476 | georg.brandl | 2008-08-04 06:29:36 +0000 (Mon, 04 Aug 2008) | 2 lines

  Fix markup.
........
  r65480 | georg.brandl | 2008-08-04 07:31:50 +0000 (Mon, 04 Aug 2008) | 3 lines

  Clarify the meaning of the select() parameters and sync
  names with docstring.
........
  r65502 | gregory.p.smith | 2008-08-04 18:34:07 +0000 (Mon, 04 Aug 2008) | 2 lines

  more cleanup ups of the recently added warnings in the subprocess docs.
........
  r65528 | brett.cannon | 2008-08-04 21:52:25 +0000 (Mon, 04 Aug 2008) | 4 lines

  Add a note about all the modules/packages changed to silence -3 warnings. More
  changes are needed once some decisions are made, but this is the work up to this
  point.
........
  r65539 | andrew.kuchling | 2008-08-05 01:38:08 +0000 (Tue, 05 Aug 2008) | 6 lines

  #3367 from Kristjan Valur Jonsson:
  If a PyTokenizer_FromString() is called with an empty string, the
  tokenizer's line_start member never gets initialized.  Later, it is
  compared with the token pointer 'a' in parsetok.c:193 and that behavior
  can result in undefined behavior.
........
  r65543 | andrew.kuchling | 2008-08-05 02:05:23 +0000 (Tue, 05 Aug 2008) | 1 line

  #3367: revert rev. 65539: this change causes test_parser to fail
........
  r65558 | georg.brandl | 2008-08-06 17:20:41 +0000 (Wed, 06 Aug 2008) | 2 lines

  Fix longstringitem definition. #3505.
........
  r65561 | mark.dickinson | 2008-08-06 20:12:30 +0000 (Wed, 06 Aug 2008) | 2 lines

  Docstring typo
........
  r65562 | mark.dickinson | 2008-08-06 21:36:57 +0000 (Wed, 06 Aug 2008) | 2 lines

  Remove duplicate import
........
  r65565 | andrew.kuchling | 2008-08-07 01:47:34 +0000 (Thu, 07 Aug 2008) | 1 line

  Add some items
........
  r65591 | georg.brandl | 2008-08-08 06:42:20 +0000 (Fri, 08 Aug 2008) | 2 lines

  #3519: callee is an expression too.
........
  r65601 | georg.brandl | 2008-08-08 15:34:34 +0000 (Fri, 08 Aug 2008) | 2 lines

  Remove mention of backquotes in the tutorial.
........
  r65608 | guido.van.rossum | 2008-08-09 14:55:34 +0000 (Sat, 09 Aug 2008) | 2 lines

  Add news item about _sre.compile() re-bytecode validator.
........
  r65610 | antoine.pitrou | 2008-08-09 17:27:23 +0000 (Sat, 09 Aug 2008) | 3 lines

  move NEWS entry to the appropriate section (oops!)
........
  r65639 | georg.brandl | 2008-08-11 10:27:31 +0000 (Mon, 11 Aug 2008) | 2 lines

  #3540: fix exception name.
........
2008-08-12 08:18:18 +00:00
Georg Brandl 2ee470f7f9 Merged revisions 64722,64729,64753,64845-64846,64849,64871,64880-64882,64885,64888,64897,64900-64901,64915,64926-64929,64938-64941,64944,64961,64966,64973 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r64722 | georg.brandl | 2008-07-05 12:13:36 +0200 (Sat, 05 Jul 2008) | 4 lines

  #2663: support an *ignore* argument to shutil.copytree(). Patch by Tarek Ziade.

  This is a new feature, but Barry authorized adding it in the beta period.
........
  r64729 | mark.dickinson | 2008-07-05 13:33:52 +0200 (Sat, 05 Jul 2008) | 5 lines

  Issue 3188: accept float('infinity') as well as float('inf').  This
  makes the float constructor behave in the same way as specified
  by various other language standards, including C99, IEEE 754r,
  and the IBM Decimal standard.
........
  r64753 | gregory.p.smith | 2008-07-06 05:35:58 +0200 (Sun, 06 Jul 2008) | 4 lines

  - Issue #2862: Make int and float freelist management consistent with other
    freelists.  Changes their CompactFreeList apis into ClearFreeList apis and
    calls them via gc.collect().
........
  r64845 | raymond.hettinger | 2008-07-10 16:03:19 +0200 (Thu, 10 Jul 2008) | 1 line

  Issue 3301:  Bisect functions behaved badly when lo was negative.
........
  r64846 | raymond.hettinger | 2008-07-10 16:34:57 +0200 (Thu, 10 Jul 2008) | 1 line

  Issue 3285: Fractions from_float() and from_decimal() accept Integral arguments.
........
  r64849 | andrew.kuchling | 2008-07-10 16:43:31 +0200 (Thu, 10 Jul 2008) | 1 line

  Wording changes
........
  r64871 | raymond.hettinger | 2008-07-11 14:00:21 +0200 (Fri, 11 Jul 2008) | 1 line

  Add cautionary note on the use of PySequence_Fast_ITEMS.
........
  r64880 | amaury.forgeotdarc | 2008-07-11 23:28:25 +0200 (Fri, 11 Jul 2008) | 5 lines

  #3317 in zipfile module, restore the previous names of global variables:
  some applications relied on them.

  Also remove duplicated lines.
........
  r64881 | amaury.forgeotdarc | 2008-07-11 23:45:06 +0200 (Fri, 11 Jul 2008) | 3 lines

  #3342: In tracebacks, printed source lines were not indented since r62555.
  #3343: Py_DisplaySourceLine should be a private function. Rename it to _Py_DisplaySourceLine.
........
  r64882 | josiah.carlson | 2008-07-12 00:17:14 +0200 (Sat, 12 Jul 2008) | 2 lines

  Fix for the AttributeError in test_asynchat.
........
  r64885 | josiah.carlson | 2008-07-12 01:26:59 +0200 (Sat, 12 Jul 2008) | 2 lines

  Fixed test for asyncore.
........
  r64888 | matthias.klose | 2008-07-12 09:51:48 +0200 (Sat, 12 Jul 2008) | 2 lines

  - Fix bashisms in Tools/faqwiz/move-faqwiz.sh
........
  r64897 | benjamin.peterson | 2008-07-12 22:16:19 +0200 (Sat, 12 Jul 2008) | 1 line

  fix various doc typos #3320
........
  r64900 | alexandre.vassalotti | 2008-07-13 00:06:53 +0200 (Sun, 13 Jul 2008) | 2 lines

  Fixed typo.
........
  r64901 | benjamin.peterson | 2008-07-13 01:41:19 +0200 (Sun, 13 Jul 2008) | 1 line

  #1778443 robotparser fixes from Aristotelis Mikropoulos
........
  r64915 | nick.coghlan | 2008-07-13 16:52:36 +0200 (Sun, 13 Jul 2008) | 1 line

  Fix issue 3221 by emitting a RuntimeWarning instead of raising SystemError when the parent module can't be found during an absolute import (likely due to non-PEP 361 aware code which sets a module level __package__ attribute)
........
  r64926 | martin.v.loewis | 2008-07-13 22:31:49 +0200 (Sun, 13 Jul 2008) | 2 lines

  Add turtle into the module index.
........
  r64927 | alexandre.vassalotti | 2008-07-13 22:42:44 +0200 (Sun, 13 Jul 2008) | 3 lines

  Issue #3274: Use a less common identifier for the temporary variable
  in Py_CLEAR().
........
  r64928 | andrew.kuchling | 2008-07-13 23:43:25 +0200 (Sun, 13 Jul 2008) | 1 line

  Re-word
........
  r64929 | andrew.kuchling | 2008-07-13 23:43:52 +0200 (Sun, 13 Jul 2008) | 1 line

  Add various items; move ctypes items into a subsection of their own
........
  r64938 | andrew.kuchling | 2008-07-14 02:35:32 +0200 (Mon, 14 Jul 2008) | 1 line

  Typo fixes
........
  r64939 | andrew.kuchling | 2008-07-14 02:40:55 +0200 (Mon, 14 Jul 2008) | 1 line

  Typo fix
........
  r64940 | andrew.kuchling | 2008-07-14 03:18:16 +0200 (Mon, 14 Jul 2008) | 1 line

  Typo fix
........
  r64941 | andrew.kuchling | 2008-07-14 03:18:31 +0200 (Mon, 14 Jul 2008) | 1 line

  Expand the multiprocessing section
........
  r64944 | gregory.p.smith | 2008-07-14 08:06:48 +0200 (Mon, 14 Jul 2008) | 7 lines

  Fix posix.fork1() / os.fork1() to only call PyOS_AfterFork() in the child
  process rather than both parent and child.

  Does anyone actually use fork1()?  It appears to be a Solaris thing
  but if Python is built with pthreads on Solaris, fork1() and fork()
  should be the same.
........
  r64961 | jesse.noller | 2008-07-15 15:47:33 +0200 (Tue, 15 Jul 2008) | 1 line

  multiprocessing/connection.py patch to remove fqdn oddness for issue 3270
........
  r64966 | nick.coghlan | 2008-07-15 17:40:22 +0200 (Tue, 15 Jul 2008) | 1 line

  Add missing NEWS entry for r64962
........
  r64973 | jesse.noller | 2008-07-15 20:29:18 +0200 (Tue, 15 Jul 2008) | 1 line

  Revert 3270 patch: self._address is in pretty widespread use, need to revisit
........
2008-07-16 12:55:28 +00:00
Benjamin Peterson 1fcfe211f4 Merged revisions 64517,64519 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r64517 | benjamin.peterson | 2008-06-24 22:09:05 -0500 (Tue, 24 Jun 2008) | 1 line

  remove bytes alias in multiprocessing
........
  r64519 | benjamin.peterson | 2008-06-25 07:39:05 -0500 (Wed, 25 Jun 2008) | 1 line

  use byte literals in multiprocessing
........
2008-06-25 12:54:22 +00:00
Benjamin Peterson e711cafab1 Merged revisions 64104,64117 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r64104 | benjamin.peterson | 2008-06-10 21:40:25 -0500 (Tue, 10 Jun 2008) | 2 lines

  add the multiprocessing package to fulfill PEP 371
........
  r64117 | benjamin.peterson | 2008-06-11 07:26:31 -0500 (Wed, 11 Jun 2008) | 2 lines

  fix import of multiprocessing by juggling imports
........
2008-06-11 16:44:04 +00:00