Weird memory leak?

FROM : Steven Frank
DATE : Tue Nov 19 21:29:01 2002

I am noticing weird behavior in a project I'm working on.  It is an app
that consumes a vast amount of text data, and generates from it an
indexed database that's about 200 MB in size in my testing environment.

Monitoring the system with top, once the app starts doing its
processing, the "free" amount in top's PhysMem line slowly decreases
down to nothing, and the system begins paging madly.

No problem, I thought, I just have a memory leak somewhere.

But the weird thing is, the RPRVT, RSHRD, RSIZE, and VSIZE columns for
the app itself do not go up.  They fluctuate slightly as things are
allocated and released during the process, but overall, they stay about
the same.  In fact, I can't account for the lost memory in _any_
running process.  Meanwhile, available memory in the rest of the system
drops to nothing.

Killing my app regains most of the memory, but there seems to always be
some that doesn't come back _until I delete the database file_.

I can't figure out what is going on, where this memory is going, or
what to do about it.

It is definitely my app causing the memory loss, 100% reproducible on
two different Jaguar machines, but I don't understand why the "missing
memory" is not showing up in my own app's statistics...  any ideas? 
There isn't any way to allocate "system memory" that doesn't belong to
any user process... is there?

I'm using the SQLite database engine, if that makes any difference. 
Everything else is fairly standard Cocoa...  no real tricks here.  I'm
not forking any secondary tasks or processes.

Thanks for any hints,

Steven