NSTask and NSPipe has buffering issues?

  • I am having weird results with NSTask. It sometimes retrieves the output, and other times receives nothing.

    I have example code for this issue at https://dl.dropbox.com/u/610721/NSTask%20Issue.zip

    The result I have is below:

    0 Starting
    0 Received /bin/ls
    0 Done
    1 Starting
    1 Received
    1 Done
    2 Starting
    2 Received
    2 Done
    3 Starting
    3 Received
    3 Done
    4 Starting
    4 Received
    4 Done
    5 Starting
    5 Received
    5 Done
    6 Starting
    6 Received /bin/ls
    6 Done
    7 Starting
    7 Received
    7 Done
    8 Starting
    8 Received (null)
    8 Done
    9 Starting
    9 Received (null)
    9 Done
    10 Starting
    10 Received /bin/ls
    10 Done
    11 Starting
    11 Received /bin/ls
    11 Done
    12 Starting
    12 Received
    12 Done
    13 Starting
    13 Received
    13 Done
    14 Starting
    14 Received /bin/ls
    14 Done
    15 Starting
    15 Received /bin/ls
    15 Done
    16 Starting
    16 Received /bin/ls
    16 Done
    17 Starting
    17 Received /bin/ls
    17 Done
    18 Starting
    18 Received /bin/ls
    18 Done
    19 Starting
    19 Received /bin/ls
    19 Done

    In the sample, I have sorted it out per the task which was started. Received is any stderr or stdin output from the task ran. In this example, I am running "/usr/bin/which ls" as it's simple and runs quickly.

    My thinking is there is some sort of buffer on the NSPipe and by the time the task quits, the buffer ether didn't receive the data or it hasn't prepared it for sending the notification.

    Can anyone help me out with this issue? It's been bugging me for hours.

    Thanks,
    Mr. Gecko
  • On 31 Jul 2012, at 3:33 PM, Mr. Gecko wrote:
    > I am having weird results with NSTask. It sometimes retrieves the output, and other times receives nothing.

    I haven't looked at the code you posted, but is it possible that you're reading from the NSPipe in such a way that you're mixing up the "end-of-file" indication with the "no more data is available right now, but may arrive later" indication? That's a common reason for this kind of problem.

    > My thinking is there is some sort of buffer on the NSPipe and by the time the task quits, the buffer ether didn't receive the data or it hasn't prepared it for sending the notification.

    IIRC, NSPipe simply wraps a Unix pipe, which will hold unread data even after the writer closes its end of the pipe (that is, the writer can write some data and exit before the reader reads anything, and the reader will still get all the data followed by EOF).
  • What do you recommend after the task finishes? I have tried [[outPipe fileHandleForReading] readDataToEndOfFile] after the task finishes and it doesn't seem to improve. Can you maybe give me an example of how to do this right?

    On Jul 31, 2012, at 6:01 PM, Wim Lewis <wiml...> wrote:

    > I haven't looked at the code you posted, but is it possible that you're reading from the NSPipe in such a way that you're mixing up the "end-of-file" indication with the "no more data is available right now, but may arrive later" indication? That's a common reason for this kind of problem.
    >
    > IIRC, NSPipe simply wraps a Unix pipe, which will hold unread data even after the writer closes its end of the pipe (that is, the writer can write some data and exit before the reader reads anything, and the reader will still get all the data followed by EOF).
  • On Tue, Jul 31, 2012, at 03:33 PM, Mr. Gecko wrote:
    > I am having weird results with NSTask. It sometimes retrieves the output,
    > and other times receives nothing.
    >
    > I have example code for this issue at
    > https://dl.dropbox.com/u/610721/NSTask%20Issue.zip

    NSTask requires a running runloop. Your code is a plain-ol' Foundation
    tool and doesn't have a runloop.

    I wouldn't be surprised if NSPipe has the same requirement.

    --Kyle Sluder
  • I have multiple run loops.

    On the main thread I have [[NSRunLoop currentRunLoop] run];

    On the sub threads for NSTask and NSPipe, I have CFRunLoopRunInMode(kCFRunLoopDefaultMode, 1, YES);

    If you are saying that you have to use NSRunLoop in order for it to work, get it working in my example and have the run loop stop running.

    I tried multiple things as well.

    [[NSRunLoop currentRunLoop] runUntilDate:[NSDate dateWithTimeIntervalSinceNow:1]];

    I tried using CFRunLoopRun();
    I tried using CFRunLoopStop(CFRunLoopGetCurrent()) to stop [[NSRunLoop currentRunLoop] run];

    I am really getting nowhere. If you really think you know how to fix it, do so in the code and report back to me with working code. Don't make guess work as this has been bugging me for hours and I think I tried pretty much everything possible. I have not attempted to use c libraries to interact with the NSPipe.

    I can confirm if you do not stop the run loop, it will get everything… However… I cannot do this as I do not know how much data I am receiving from the task.

    Is there a way to have NSPipe read until it actually reaches the end of file? What I have tried so far doesn't seem to work.

    Thanks for helping me with this crazy task.

    On Jul 31, 2012, at 7:14 PM, Kyle Sluder <kyle...> wrote:

    > NSTask requires a running runloop. Your code is a plain-ol' Foundation
    > tool and doesn't have a runloop.
    >
    > I wouldn't be surprised if NSPipe has the same requirement.
    >
    > --Kyle Sluder
  • On Jul 31, 2012, at 6:53 PM, "Mr. Gecko" <grmrgecko...> wrote:

    >
    > I can confirm if you do not stop the run loop, it will get everything… However… I cannot do this as I do not know how much data I am receiving from the task.

    1) Yes, you need to run the run loop forever (NSDistantFuture).

    2) You don't need to know in advance how much data you need to read. Just stop running the run loop when you're done.

    3) Why are you using threads at all? You should just be able to fire off an NSTask from the main thread and let its run loop take care of notifying you.

    --Kyle Sluder
  • I need threads because the main thread is a network loop waiting for a connection and when it gets a connection, it spawns a thread like in my example. I need to be able to respond to the client with all the data from the task and I need to close out the connection and release the spawned thread/object.

    I need to do this in a reasonable amount of time as well. I cannot just slow down the speed by 1 second as that makes the response to the client 1 second slower.

    Do you recommend that I work with C APIs to make my own NSTask which works?

    In response to number 2, how can I stop the run loop and how could I know when I got everything? I don't have any way to verify I got everything.

    On Jul 31, 2012, at 9:00 PM, Kyle Sluder <kyle...> wrote:

    > 1) Yes, you need to run the run loop forever (NSDistantFuture).
    >
    > 2) You don't need to know in advance how much data you need to read. Just stop running the run loop when you're done.
    >
    > 3) Why are you using threads at all? You should just be able to fire off an NSTask from the main thread and let its run loop take care of notifying you.
    >
    > --Kyle Sluder
  • On Jul 31, 2012, at 7:43 PM, Mr. Gecko <grmrgecko...> wrote:

    > I need threads because the main thread is a network loop waiting for a connection and when it gets a connection, it spawns a thread like in my example. I need to be able to respond to the client with all the data from the task and I need to close out the connection and release the spawned thread/object.

    You don't need threads for this, as long as you use asynchronous I/O.

    > In response to number 2, how can I stop the run loop and how could I know when I got everything? I don't have any way to verify I got everything.

    You know you got everything when the pipe connected to the task's stdout reaches EOF.

    You might find MYTask useful: it's a general-purpose wrapper around NSTask that I wrote a few years ago (while writing a Mercurial client that did its work by invoking the "hg" tool.) It was surprisingly difficult to get everything to work right.
    https://bitbucket.org/snej/myutilities/src/319441e240fa/MYTask.m
    You're welcome to copy it (it's under a BSD license) or just use it as sample code to figure out how to use NSTask asynchronously.

    —Jens
  • NSPipe itself doesn't require that a run loop be run, but the "InBackgroundAndNotify" methods of the associated NSFileHandle objects do.

    You receive a zero-length NSData when (and only when) a read encounters EOF.

    There is an inherent race between the receipt of the task termination notification and getting end-of-file on the output and error pipes.  This race is not in Cocoa, it's in the kernel and the interprocess communication mechanisms.  You can't rely on having received all of the data by the time you are notified of the task termination.

    You should separately track when NSTask has posted the termination notification and when each pipe has gotten EOF.  Keep looping around the run loop until *all three* things have happened.  (In other words, generalize from your "running" variable.  You had the right idea but hadn't covered all of the important parts.)

    Regards,
    Ken
  • Thank you!

    That makes lots of sense and I was able to fix my code. I knew that I sometimes got zero length data, but never knew why. Now that I know it's the end of file… I can make lots of use of this knowledge.

    Thanks for the help.

    On Jul 31, 2012, at 11:20 PM, Ken Thomases <ken...> wrote:

    > NSPipe itself doesn't require that a run loop be run, but the "InBackgroundAndNotify" methods of the associated NSFileHandle objects do.
    >
    > You receive a zero-length NSData when (and only when) a read encounters EOF.
    >
    > There is an inherent race between the receipt of the task termination notification and getting end-of-file on the output and error pipes.  This race is not in Cocoa, it's in the kernel and the interprocess communication mechanisms.  You can't rely on having received all of the data by the time you are notified of the task termination.
    >
    > You should separately track when NSTask has posted the termination notification and when each pipe has gotten EOF.  Keep looping around the run loop until *all three* things have happened.  (In other words, generalize from your "running" variable.  You had the right idea but hadn't covered all of the important parts.)
    >
    > Regards,
    > Ken
    >
previous month august 2012 next month
MTWTFSS
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31    
Go to today