NSStream, NSInputStream, NSOutputStream
-
I modified slightly the Echo Server code sample from Apple with the
following results. One, I couldn't stream a large file from a
polling routine. More than likely it would cancel because of a
Sigterm 15.
It appears from reading the docs that the user cannot detect the end
of a stream and that the NSStreamEventEndEncountered only detects a
close. Two, when unarchiving a file in a client input stream delegate
method, if the stream terminated from the server because it was too
large, the client terminates on an unarchiver error because it didn't
get the whole stream. Third, the output stream methods shown in Echo
Server are polling methods, not delegate methods.
So:
1. How do I stream a large file between connections or is NSStream
the wrong tool? Can the stream size be modified?
2. What is the largest stream size?
3. Is it possible to detect a valid archive before I unarchive it, or
do I simply have to intercept the exception?
4. How does one trigger and make available a file an output stream so
that the delegate methods can be used?
Thank,
John MacMullin -
On 18 May '08, at 8:50 AM, John MacMullin wrote:
> I modified slightly the Echo Server code sample from Apple with the
> following results. One, I couldn't stream a large file from a
> polling routine. More than likely it would cancel because of a
> Sigterm 15.
Whenever you report a crash, a backtrace is very helpful. Or at least
tell us what function/method the crash was in. This shouldn't crash,
so the problem is probably in something in your code.
> It appears from reading the docs that the user cannot detect the end
> of a stream and that the NSStreamEventEndEncountered only detects a
> close.
"End of stream" and "close" are the same thing: a TCP input stream
ends when the other side closes the connection (or crashes...)
> Two, when unarchiving a file in a client input stream delegate
> method, if the stream terminated from the server because it was too
> large, the client terminates on an unarchiver error because it
> didn't get the whole stream.
A stream won't ever terminate due to length. You can send gigabyte
after gigabyte over a TCP stream, as many happy BitTorrent users can
attest ;-)
But yes, if the incoming stream closed before sending all the data the
reader needed, then I would expect the reader to report an error. This
shouldn't cause a crash, though; it sounds like your code isn't
handling the error gracefully.
> Third, the output stream methods shown in Echo Server are polling
> methods, not delegate methods.
The CocoaEcho sample? I just looked at it, and you're right. The
writing code is badly designed. The client will block in a 'while'
loop until all the data is sent, and the server code will simply drop
response data on the floor if there isn't room to send it. I just
filed feedback on the web page for the sample.
> 1. How do I stream a large file between connections or is NSStream
> the wrong tool? Can the stream size be modified?
Most of the CocoaEcho code is reasonable as a base for doing this. Rip
out the server-side code that echoes the data back, and instead write
the data to a file.
On the client/sender side, it's best to use the 'spaceAvailable'
delegate call, as you said. In response to that call, read some data
from the file into a buffer, then write it to the stream. Something
like 4k will do. Pay attention to the return value of the write, which
tells you how many bytes actually got written, and advance a file-
position instance variable by that amount. That'll tell you the
position to read from in the input file next time.
> 2. What is the largest stream size?
There isn't one. TCP was designed to handle arbitrary length streams.
There are internal byte counters but they just wrap around harmlessly
after 4GB.
> 3. Is it possible to detect a valid archive before I unarchive it,
> or do I simply have to intercept the exception?
No. If you have data in archive format, you have to just hand it to
NSUnarchiver and wrap the call in @try/@catch.
But you said you were sending a file? In that case you should just
send it as a stream of bytes. If you're reading the entire file into
an NSData and then sending that with NSArchiver, that's a huge amount
of overhead for no gain.
> 4. How does one trigger and make available a file an output stream
> so that the delegate methods can be used?
I don't understand that question. Can you give more detail? (Or
perhaps I answered it above under #1.)
—Jens -
On May 18, 2008, at 10:50 AM, John MacMullin wrote:
> I modified slightly the Echo Server code sample from Apple with the
> following results.
To which sample are you referring? I assume you mean the sample
called CocoaEcho at <http://developer.apple.com/samplecode/CocoaEcho/>.
> One, I couldn't stream a large file from a polling routine. More
> than likely it would cancel because of a Sigterm 15.
SIGTERM, really? I can't think of why or how a SIGTERM would be
raised by the system or the frameworks. I can see SIGPIPE, SIGTRAP,
or SIGABRT. And, of course, programming error can result in SIGBUS or
SIGSEGV.
Hmm. Perhaps launchd might terminate a service with SIGTERM if it
determined that the service was misbehaving somehow.
> It appears from reading the docs that the user cannot detect the end
> of a stream and that the NSStreamEventEndEncountered only detects a
> close.
What is it that you would call the end of a stream other than it being
closed?
> Two, when unarchiving a file in a client input stream delegate
> method, if the stream terminated from the server because it was too
> large, the client terminates on an unarchiver error because it
> didn't get the whole stream.
That doesn't seem surprising.
> Third, the output stream methods shown in Echo Server are polling
> methods, not delegate methods.
Well, it does blocking output, true. And it doesn't account for the
possibility of partial writes. If it were to account for that, I
would expect that it would loop around the [ostream write:buffer
maxLength:actuallyRead], advancing the buffer and decrementing the
length each time. And it would handle errors.
For it to properly perform asynchronous output using delegate methods,
it would need to buffer the data it receives from clients. But it
would have to impose some limit on the amount of data it is willing to
buffer before it can be written out. When it hit that limit, it would
have to stop reading from that client until it could send some of its
output.
By writing synchronously, it's effectively doing that with some
automatic, smallish buffer supplied by the underlying APIs. However,
a block on one client causes a block of the entire server.
A better approach might be to suspend processing of that one client's
input stream by removing it from the runloop. Then, when that
client's buffer drained a bit, due to some
NSStreamEventHasSpaceAvailable events, the input stream could be re-
added to the runloop. If you just ignore
NSStreamEventHasSpaceAvailable events rather than unscheduling the
input stream, then I think you'll get them in a tight loop and your
server will eat CPU like mad.
> 1. How do I stream a large file between connections or is NSStream
> the wrong tool? Can the stream size be modified?
>
> 2. What is the largest stream size?
There's no such thing as "stream size". That's one of the defining
characteristics of a stream.
> 3. Is it possible to detect a valid archive before I unarchive it,
> or do I simply have to intercept the exception?
I think you just have to catch the exception.
> 4. How does one trigger and make available a file an output stream
> so that the delegate methods can be used?
I don't understand what you're asking here.
-Ken -
Jens and Ken:
Thank you for your responses. I believe I now understand in part as
to what is happening as to the streams. Let me take the output
"trigger" first. The Stream Programming Guide states: "if the
delegate receives an NSStreamEventHasSpaceAvailable event and does
not write anything to the stream, it does not receive further space-
available events from the run loop until the NSOutputStream object
receives more bytes." This may answer the question of the
"trigger", which is I now believe is simply the write statement.
That means that instead of doing a polling loop outside of the
delegate, (or doing writes inside the NSStreamEventHasBytesAvailable
section of the delegate during reads) I think that all one needs to
do is to write one byte in the "external" function, then handle the
rest of the writes through the delegate
NSStreamEventHasSpaceAvailable event. That should be a lot easier
than doing a custom polling loop and trying to resolve the space
available issues outside of the delegate. In other words, it seems
that the delegate should do it for you after doing a one byte write.
I'll let you know how it works out. And I will address the other
questions after I have tested this issue.
Best regards,
John
On May 18, 2008, at 11:10 AM, Jens Alfke wrote:
>
> On 18 May '08, at 8:50 AM, John MacMullin wrote:
>
>> I modified slightly the Echo Server code sample from Apple with
>> the following results. One, I couldn't stream a large file from a
>> polling routine. More than likely it would cancel because of a
>> Sigterm 15.
>
> Whenever you report a crash, a backtrace is very helpful. Or at
> least tell us what function/method the crash was in. This shouldn't
> crash, so the problem is probably in something in your code.
>
>> It appears from reading the docs that the user cannot detect the
>> end of a stream and that the NSStreamEventEndEncountered only
>> detects a close.
>
> "End of stream" and "close" are the same thing: a TCP input stream
> ends when the other side closes the connection (or crashes...)
>
>> Two, when unarchiving a file in a client input stream delegate
>> method, if the stream terminated from the server because it was
>> too large, the client terminates on an unarchiver error because it
>> didn't get the whole stream.
>
> A stream won't ever terminate due to length. You can send gigabyte
> after gigabyte over a TCP stream, as many happy BitTorrent users
> can attest ;-)
>
> But yes, if the incoming stream closed before sending all the data
> the reader needed, then I would expect the reader to report an
> error. This shouldn't cause a crash, though; it sounds like your
> code isn't handling the error gracefully.
>
>> Third, the output stream methods shown in Echo Server are polling
>> methods, not delegate methods.
>
> The CocoaEcho sample? I just looked at it, and you're right. The
> writing code is badly designed. The client will block in a 'while'
> loop until all the data is sent, and the server code will simply
> drop response data on the floor if there isn't room to send it. I
> just filed feedback on the web page for the sample.
>
>> 1. How do I stream a large file between connections or is NSStream
>> the wrong tool? Can the stream size be modified?
>
> Most of the CocoaEcho code is reasonable as a base for doing this.
> Rip out the server-side code that echoes the data back, and instead
> write the data to a file.
>
> On the client/sender side, it's best to use the 'spaceAvailable'
> delegate call, as you said. In response to that call, read some
> data from the file into a buffer, then write it to the stream.
> Something like 4k will do. Pay attention to the return value of the
> write, which tells you how many bytes actually got written, and
> advance a file-position instance variable by that amount. That'll
> tell you the position to read from in the input file next time.
>
>> 2. What is the largest stream size?
>
> There isn't one. TCP was designed to handle arbitrary length
> streams. There are internal byte counters but they just wrap around
> harmlessly after 4GB.
>
>> 3. Is it possible to detect a valid archive before I unarchive it,
>> or do I simply have to intercept the exception?
>
> No. If you have data in archive format, you have to just hand it to
> NSUnarchiver and wrap the call in @try/@catch.
>
> But you said you were sending a file? In that case you should just
> send it as a stream of bytes. If you're reading the entire file
> into an NSData and then sending that with NSArchiver, that's a huge
> amount of overhead for no gain.
>
>> 4. How does one trigger and make available a file an output stream
>> so that the delegate methods can be used?
>
> I don't understand that question. Can you give more detail? (Or
> perhaps I answered it above under #1.)
>
> —Jens
-
On 18 May '08, at 9:19 PM, John MacMullin wrote:
> I think that all one needs to do is to write one byte in the
> "external" function, then handle the rest of the writes through the
> delegate NSStreamEventHasSpaceAvailable event.
You don't even need to do the one byte externally. Your delegate will
be told there's space available as soon as the socket's opened.
—Jens -
Continuing on my efforts re: my modified Echo Server, the following
code is crashing.
- (void)startStreamWrite:(NSOutputStream *)ostream
{
NSAutoreleasePool * pool = [[NSAutoreleasePool alloc]init];
while ([ostream hasSpaceAvailable]) {
if (remainingToWrite > 0) {
actuallyWritten = 0;
actuallyWritten = [ostream write:marker maxLength:1];
if ((actuallyWritten == -1) || (actuallyWritten == -1)) {
NSLog(@"[ostream streamError]:\n%@\n", [ostream streamError]);
NSLog(@"remainingToWrite:\n%u\n", remainingToWrite);
NSLog(@"ostream:\n%@\n", ostream);
} else {
remainingToWrite -= actuallyWritten;
marker += actuallyWritten;
NSLog(@"remainingToWrite:\n%u\n", remainingToWrite);
}
}
}
[pool release];
}
The error returned is: [ostream streamError]:
NSError "POSIX error: Broken pipe" Domain=NSPOSIXErrorDomain Code=32
I have looked at the stream error docs in NSStream and this error is
not even reference.
Any ideas on what this means and how I fix it would be greatly
appreciated.
John -
How are you creating the stream?
On May 19, 2008, at 7:07 PM, John MacMullin wrote:
> Continuing on my efforts re: my modified Echo Server, the following
> code is crashing.
>
> - (void)startStreamWrite:(NSOutputStream *)ostream
> {
> NSAutoreleasePool * pool = [[NSAutoreleasePool alloc]init];
> while ([ostream hasSpaceAvailable]) {
> if (remainingToWrite > 0) {
> actuallyWritten = 0;
> actuallyWritten = [ostream write:marker maxLength:1];
> if ((actuallyWritten == -1) || (actuallyWritten == -1)) {
> NSLog(@"[ostream streamError]:\n%@\n", [ostream streamError]);
> NSLog(@"remainingToWrite:\n%u\n", remainingToWrite);
> NSLog(@"ostream:\n%@\n", ostream);
> } else {
> remainingToWrite -= actuallyWritten;
> marker += actuallyWritten;
> NSLog(@"remainingToWrite:\n%u\n", remainingToWrite);
> }
> }
> }
> [pool release];
> }
>
> The error returned is: [ostream streamError]:
> NSError "POSIX error: Broken pipe" Domain=NSPOSIXErrorDomain Code=32
>
> I have looked at the stream error docs in NSStream and this error is
> not even reference.
>
> Any ideas on what this means and how I fix it would be greatly
> appreciated.
-
On May 19, 2008, at 8:07 PM, John MacMullin wrote:
> The error returned is: [ostream streamError]:
> NSError "POSIX error: Broken pipe" Domain=NSPOSIXErrorDomain Code=32
>
> I have looked at the stream error docs in NSStream and this error
> is not even reference.
>
> Any ideas on what this means and how I fix it would be greatly
> appreciated.
One of the BSD/POSIX routines used by the NSOutputStream
implementation is returning EPIPE.
It's documented in the man pages for the BSD routine in question
(write, send, or similar), and also in one central list: http://
developer.apple.com/documentation/Darwin/Reference/ManPages/man2/
intro.2.html
It means that the would-be recipient of the data you're writing has
closed their end of the connection. So, you should look at the
implementation of the client to figure out why it's closing the
connection.
Cheers,
Ken -
Ah, thank you.
John
On May 19, 2008, at 6:32 PM, Ken Thomases wrote:
> On May 19, 2008, at 8:07 PM, John MacMullin wrote:
>> The error returned is: [ostream streamError]:
>> NSError "POSIX error: Broken pipe" Domain=NSPOSIXErrorDomain Code=32
>>
>> I have looked at the stream error docs in NSStream and this error
>> is not even reference.
>>
>> Any ideas on what this means and how I fix it would be greatly
>> appreciated.
>
> One of the BSD/POSIX routines used by the NSOutputStream
> implementation is returning EPIPE.
>
> It's documented in the man pages for the BSD routine in question
> (write, send, or similar), and also in one central list: http://
> developer.apple.com/documentation/Darwin/Reference/ManPages/man2/
> intro.2.html
>
> It means that the would-be recipient of the data you're writing has
> closed their end of the connection. So, you should look at the
> implementation of the client to figure out why it's closing the
> connection.
>
> Cheers,
> Ken
