Re: Core Data performance advice... creating relationships.

FROM : Phil
DATE : Tue Jan 15 21:57:08 2008

On Jan 15, 2008, at 11:22 AM, I. Savant wrote:

>> I appreciate your followup messages.  I had tried a schema of similar
>> complexity (granted, it was a larger data set) and had fairly
>> unfavorable results under Tiger (re: memory usage and performance.)
>> It sounds like it might be worth another look under Leopard.

>
>  With a statement like that, you've *got* to provide more detail. :-D
> What was your schema? What was the size of the data set (approximate
> number of each entity's instances, if you know it)? What did you find
> to be slow/memory-intensive, specifically? How did you implement that
> which you found to be slow/memory-intensive?
>
>  Inquiring minds want to know! I'm not saying "I don't believe it,
> prove it!" ... merely curious. I too have come across a number of
> inefficiencies but most of them had fairly simple work-arounds once
> the problem area was identified.


No problem providing more detail...

When I was first looking at Core Data a couple of years ago, I 
attempted to migrate a moderate sized data set (~300 Meg in a vanilla 
SQLite database over about two dozen tables) for an application that 
was very data processing oriented (i.e. very little GUI, mostly an ETL 
job) to Core Data and after getting everything loaded, setting up 
relationships and predicate templates that were as selective as 
possible, it still appeared to be performing full table scans most, if 
not all, of the time because Core Data would apparently not see the 
value of index creation of critically important columns from a 
performance standpoint (I confirmed via the command line SQLite 
interface but had read and was told that CD would take care of such 
things automatically.  I'm sure it did but have no idea what criteria 
it used to determine since it didn't seem to take any of the hints I 
was giving it... was it using: relationships in the model? predicate 
templates? database stats? phase of the moon?) and I couldn't see a 
(supported) way to force it to create them.

For example, one of the key tables involved was ~10 Million rows of 
daily stock quote data (i.e. ticker, date_id, open, high, low, close, 
etc.) which had relationships to the security (ticker, companyName, 
etc.) and date (date_id, date, yearMonth, etc) tables.  For the life 
of me, I could never figure out how to get Core Data to decide to 
create the appropriate indexes so that when I was after all daily 
quotes where companyName='Apple Computer' and 
yearMonth='2007-11' (i.e. highly selective criteria) it would always 
want to fetch back 10 million managed objects even though I really 
only cared, at most, about a few thousand.  It seemed like the more 
selective my query got, the worse Core Data would perform.  I assumed 
that this was because it was deciding that it needed to fetch 
everything back and apply the filtering criteria itself which makes 
sense if that is what it was actually doing.  This assumption 
appeared to be supported by watching memory usage for my app memory 
usage to shoot up to 500+ Meg at times even though I was never working 
with more than a fraction of the data set at a time and being quite 
liberal in my use of autorelease pools.  (This was mostly speculation 
on my part because Core Data was fairly opaque re: what was going on 
with the data store so I was never really certain what was going on.)

This was all quite some time ago and I didn't bother revisiting it for 
anything nearly that large since as there were other things about Core 
Data that turned me off to using it for anything that was primarily 
database-oriented in nature.  As some of these issues appear to be 
addressed in Leopard, it looks like it's worth taking at least another 
look and possibly expanding the cases where I should consider it as a 
solution.

Thanks,
Phil