Re: sorting large tables

FROM : Jens Alfke
DATE : Thu Mar 13 23:24:59 2008

On 13 Mar '08, at 1:42 PM, Daniel Child wrote:

> That makes a lot of sense to me, and that's the situation I'm trying 
> to get to. But right now all I have is the original (unsorted) raw 
> data. So I need to load it into memory and sort it.


You don't necessarily need to sort all of it at once. You just need to 
find the first few items, to display in your table view. If the user 
scrolls past those, you need to find more. You can do this by 
streaming the data from a file, keeping only the number of items you 
need in memory at once. This sort of task is typically called 
"external sorting", and Knuth and most other algorithm texts talk 
about it.

But that said, the sort you're doing shouldn't freeze or crash, not 
unless you eat up your processes' entire address space or use up so 
much VM that the system starts to thrash. You'll have to provide more 
details of what's happening.

—Jens