Re: Garbage collection, core data, and tight loops

FROM : John R. Timmer
DATE : Fri Nov 02 20:01:00 2007

Okay, so for completeness, i thought i'd share the resolution of this 
with the list, in case someone finds this thread in the future.

I set the app up to create small batches of ManagedObjects (several 
hundred at a time) and added a small pause (0.01 sec.) between batches 
via a delayed selector.

Before creating any ManagedObjects, i cancelled undo registration.

Once a batch was created, i processed pending changes, saved, and 
reset the managedObjectContext.
   incidentally, resetting appears to create a brand new undo manager 
for the context, so i disabled each time; there was no need to re-
enable when done.

Using an autorelease pool for each batch worked well, keeping memory 
use extremely low.

Using garbage collection resulted in a significant memory gain, but 
nowhere near bad enough to crash the program.  Oddly, the memory use 
did not subside after the loop had finished.

So I'll probably stick to manually handling memory, at least for the 
import app.

JT



On Oct 31, 2007, at 10:41 PM, Ben Trumbull wrote:

> At 8:31 AM -0700 10/31/07, Bill Bumgarner wrote:

>> On Oct 31, 2007, at 5:37 AM, John R. Timmer wrote:

>>> I have been periodically saving changes in the loop.

>>>> Are you asking Core Data to purge objects?

>>> I neglected to manually purge the objects - i assume that would 
>>> involve using the ManagedObjectContext's -reset method?

>>>> Does your object graph allow partial subgraphs to exist in memory 
>>>> without eventually faulting everything into memory?

>>> At this point, I'm only importing a single type of managed object 
>>> to see how well things work with millions of records, and to make 
>>> sure the project's feasible as designed before going further, so 
>>> this is a non-issue.

>>>>     ... do you have any kind of caches or hashes that are 
>>>> effectively keeping the objects around?

>>> No, nothing should be retaining the objects - i'm setting values 
>>> and moving on.
>>> I'll give it another shot with resetting the ManagedObjectContext 
>>> and see if that does the trick.

>>
>>
>> Ben will hopefully pipe up w/some info on how to do that.

>
> You can find information about memory management with Core Data in 
> the Core Data Programming Guide.
>
> You will want to disable the MOC's undo manager for both background 
> and batch operations.
>
> Objects with relationships are often caught in retain cycles.  These 
> can be broken using -refreshObject:mergeChanges:NO or -reset (which 
> invalidates all managed objects that context is observing) or 
> releasing (deallocate) the MOC (which also invalidates all the 
> managed objects its observing).  This problem goes away under GC as 
> both the MO and its MOC have, effectively, weak references to each 
> other.
>
> However, there is a complication.  Although we discourage it, under 
> some circumstances it is permissible to pass a MOC to another 
> thread. So the deallocation (or finalization) of managed objects 
> must be thread safe.  Because these objects are wired into a graph 
> of objects, and have framework resources, efficiently managing this 
> thread safety is quite complicated.
>
> A side effect of thread safe deallocation of managed objects is that 
> deallocation is deferred until the MOC can safely clean up state. 
> The MOC maintains a queue of pending deallocations, which is handles 
> during various operations like fetching, saving, the end of the 
> event, etc.  Of course, resetting or releasing the MOC also purges 
> any pending deallocations.
>
> You can manually force the MOC to poll with -processPendingChanges.
>
> Under the retain model, objects fetched are always in the 
> autorelease pool, as well.
> --
>
> -Ben