Re: Working with large files and Memory

FROM : Clark Cox
DATE : Tue Mar 11 18:10:58 2008

On Tue, Mar 11, 2008 at 9:54 AM, Carl E. McIntosh <<email_removed>> wrote:
> Can you please give advice about handling large data files with memory
>  management techniques? I am attempting to read three large files (1
>  GB, 208 MB, 725 MB) sequentially and place the data into arrays for
>  processing. Here is my psuedocode:
>
>  1) Import a file into NSString.
>        NSString *aFileString = [NSString stringWithContentsOfFile:
>  fileLocation];  // Convert file at path to myFileString text holder;


Depending on the encoding of the file itself, this could end up
allocating a block of memory twice the size of the file.

>  2) Use NSScanner pull out integers and floats
>        NSScanner *aFileScanner = [[NSScanner alloc] initWithString:
>  aFileString];


This has the potential of copying the passed in string (not likely to
be a problem, as you string is immutable, but is still something to
keep in mind).

>
>  3) Store values into arrays.
>        float myFloats [100000][2000]; or
>        float myInts [100000][2000];


Again, *another* huge block of memory.

>  4) repeat three times with 3 different files.


Depending on how this is done (how tight your autorelease pools are,
etc.), the old string may not have been deallocated yet.

>  This algorithm works for smaller files but chokes on the larger files
>  and I get malloc errors. I've attempted to use NSZone's to the same
>  failure.
>
>  Can you please give advice about handling large data files with memory
>  management techniques?


You'll have to use less memory. Remember, that on a 32-bit machine,
you will never be able to allocate more than 4GB of memory in your
address space. On top of that, much of that memory is already used by
the system's frameworks (usually between 1 and 2 GB).

>  I have 4 GB ram and can hog off 2 - 3 GBs for the process. I don't
>  know how to explicitly allocate real memory. I'd rather not use
>  virtual memory. Any references or examples would be appreciated.


Memory doesn't work like that (i.e you cannot allocate "real" memory).
The closest that you can get is to allocate memory and to "wire" it
down; however, this is a horrible idea for allocations this large
(you'll still run out of address space as before). Your best bet is to
change your algorithm to not have to load the entire file at once
(perhaps load it a line at a time, or in blocks of 4K or so).

--
Clark S. Cox III
<email_removed>