Memory limits under Garbage Collection, NSMutableData vs. malloc()

FROM : Rick Hoge
DATE : Tue Nov 13 23:30:09 2007

I have been experimenting with garbage collection under 10.5, and have 
had a few difficulties I don't quite understand.

I have a method that needs to allocate a very large chunk of memory 
(about 700MB in my test case) to do some processing (I am on a 32 bit 
Intel Core Duo with 2GB RAM).  Under Tiger, this always worked 
reasonably well and the performance was good.  However under Leopard, 
the same code has started failing under GC.  What's strange is that 
allocating the same amount of memory behaves quite differently 
depending on how it's done.  Specifically:

- if I use malloc() to allocate a traditional C buffer, it is no 
problem.  The allocation is also very fast (no perceptible latency)

- if I use [NSMutableData dataWithLength:theLength]; to allocate the 
same amount of memory, *without* garbage collection, it is very slow 
(10 seconds) but it is successful.

- if I use [NSMutableData dataWithLength:theLength]; to allocate the 
same amount of memory, *with* garbage collection (i.e. required), the 
allocation fails with the following message:

2007-11-13 17:11:06.686 MyApp[10413:10b] Here1
MyApp(10413,0x37cf60) malloc: *** mmap(size=788488192) failed (error 
code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
2007-11-13 17:11:06.699 MyApp[10413:10b] Here2

(Here1 and Here2 are from NSLog's that bracket the constructor 
statement)

I'm not surprised there's some performance difference between malloc() 
and NSMutableData's convenience constructor, but I'm a little 
surprised at how much.  Also I'm trying to figure out why GC leads to 
a failure, when the same code run non-GC does not.  I guess it's quite 
possible that the collector may not know in advance that it's about to 
be hit with a huge memory demand, so perhaps it's let a lot of garbage 
accumulate at the time of my constructor...

Obviously I can break the memory chunk up into smaller bits, or just 
use malloc().  I'd just like to understand what's going on under the 
hood a bit better to make appropriate choices in future code.

Thanks,

Rick