Re: Working with large files and Memory

FROM : Jens Alfke
DATE : Tue Mar 11 21:49:32 2008

On 11 Mar '08, at 10:18 AM, Jean-Daniel Dupas wrote:

> The first advice I can give you is "do not load the whole file into 
> memory".


Absolutely.

> Use read stream to read chunk of data and process them. (see 
> NSInputStream or NSFileHandle).


Or if the file is simple ascii text with newlines, you can use basic C 
stdio calls (fopen, fgets, fclose) to read a line at a time. You can 
either convert the line into an NSString, or just use something like 
sscanf to parse it.

In rare situations where you absolutely do have to load a huge file 
into memory, i.e. for an algorithm that requires random access, your 
best bet is to memory-map it. -[NSData 
dataWithContentsOfFile:options:] has an option flag to map the file. 
This will avoid a lot of copying, but it's still subject to the same 
address-space limit if your process is 32-bit, so don't expect to be 
able to load anything much over a gigabyte.

—Jens