CoreData - sqlite - statement is still active

  • The problems I'm having with Core Data, of late, all seem to be
    crashers inside sqlite. Not sure what to do about it/them. The most
    annoying occurs, occasionally!, after fetching new instances of
    managed objects, same entity, one after another, and then calling
    valueForKey: on a related object. I think the problem is really in
    the original fetching. It seems like the standard call to
    "fetchedObjects = [self executeFetchRequest:fetchRequest
    error:anError];" with fetchRequest along the lines of
    <NSFetchRequest: 0xc5c4d80> (entity: Media; predicate: (name LIKE
    "Aqua"); sortDescriptors: (null); limit: 0) is not returning before
    another fetch begins. Can this be right?? Core Data, or really
    sqlite, seems pretty unhappy about this.

    As I say, 90%+ of these calls work perfectly fine, but the one that
    doesn't stops me cold. If anyone has any ideas/suggestions to figure
    out what's going on and/or a workaround to avoid it, I would be *so*
    happy.

    Thanks,

    Terrence

    Basically the typical backtrace looks like this:

    sqlite3_data_count
    sqlite3_data_count
    sqlite3_column_int64
    [NSSQLiteConnection _newObjectIDForCurrentRowWithEntity:]
    [NSSQLiteConnection newFetchedRow]
    [NSSQLChannel fetchRow]
    [NSSQLCore _newRowsForFetchRequest:]
    [NSSQLCore fetchRowForObjectID:]
    [NSSQLCore retainedDataForObjectID:withContext:]
    [NSFaultHandler fulfillFault:withContext:]
    [NSManagedObject(_NSInternalMethods)
    _genericValueForKey:withIndex:flags:]
    _NSGetUsingKeyValueGetter

    With this on the console:

    [snip]  fetching media with name Branching_Out
    [snip]  Uncaught exception - statement is still active
    [snip]  *** Attempt to remove unrecognized exception handler 0xbfffd094
    [snip]  fetching media with name Trippy
    [snip]  Uncaught exception - _obtainOpenChannel -- NSSQLCore
    0x7956730: no database channel is available
    [snip]  *** Attempt to remove unrecognized exception handler 0xbfffd094
    [snip]  fetching media with name Vector
    [snip]  Uncaught exception - _obtainOpenChannel -- NSSQLCore
    0x7956730: no database channel is available
    [snip]  *** Attempt to remove unrecognized exception handler 0xbfffd094
    [snip]  fetching media with name Open_All
    [snip]  Uncaught exception - _obtainOpenChannel -- NSSQLCore
    0x7956730: no database channel is available
    [snip]  *** Attempt to remove unrecognized exception handler 0xbfffd094
    [Switching to process 16968 thread 0x740b]
    Program received signal:  "EXC_BAD_ACCESS".
    [Switching to process 16968 thread 0x740b]
    warning: Previous frame identical to this frame (corrupt stack?)

    The "fetching media" statements are, obviously, my log statements. By
    this point in the app, it's already fetched about 50 media objects by
    name just fine.
  • On Oct 6, 2005, at 10:17 AM, Terrence Talbot wrote:
    > As I say, 90%+ of these calls work perfectly fine, but the one that
    > doesn't stops me cold. If anyone has any ideas/suggestions to
    > figure out what's going on and/or a workaround to avoid it, I would
    > be *so* happy.

    Some questions....

    - are you using threads?  ... properly?

    - do you diddle the sqlite database from outside of Core Data?

    b.bum
  • On Oct 6, 2005, at 10:32 AM, Bill Bumgarner wrote:

    > Some questions....
    >
    > - are you using threads?  ... properly?
    >
    > - do you diddle the sqlite database from outside of Core Data?
    >

    No, that's the thing: this should be a very straightforward exercise.
    Single document, single context, single thread. The only diddling
    I've done is a .dump (and now some selects, see below) of the
    document on the command-line to make sure the rows for these objects
    are there. They are...

    ????

    FWIW, here's an additional path that sometimes results from
    essentially the same set of requests. (That's the other tough thing:
    it seems like the "statement is still active" issue can catch
    CoreData at a couple of different points.)

    Backtrace:

    sqlite3VdbeExec
    sqlite3VdbeExec
    [NSSQLiteConnection _execute]
    [NSSQLiteConnection execute]
    [NSSQLChannel selectRowsWithFetchRequest:]
    [NSSQLCore _newRowsForFetchRequest:]
    [NSSQLCore objectsForFetchRequest:inContext:]
    [NSSQLCore executeRequest:withContext:]
    [NSPersistentStoreCoordinator(_NSInternalMethods)
    executeRequest:withContext:]
    [NSManagedObjectContext executeFetchRequest:error:]

    Console:

    [snip] fetching media with name Aqua
    [snip] executing fetch: <NSFetchRequest: 0xd5ae550> (entity: Media;
    predicate: (name LIKE "Aqua"); sortDescriptors: (null); limit: 0)
    [snip] fetched a single object
    [snip] successfully fetched media with name Aqua
    [snip] asking for media data with specifier: 151
    [snip] executing fetch: <NSFetchRequest: 0xd5b44c0> (entity: Media;
    predicate: (uniqueID LIKE "151"); sortDescriptors: (null); limit: 0)
    [snip] fetching media with name Cathedral
    [snip] executing fetch: <NSFetchRequest: 0xd5bbea0> (entity: Media;
    predicate: (name LIKE "Cathedral"); sortDescriptors: (null); limit: 0)
    [snip] fetched a single object
    [snip] media Aqua /Volumes/dwood/dev/biophony/trunk/KareliaSite/
    Results of the Design Survey.rtfd/Aqua.png
    [snip] Uncaught exception - statement is still active
    [snip] *** Attempt to remove unrecognized exception handler 0xbfffd094
    [snip] fetching media with name Branching_Out
    [snip] executing fetch: <NSFetchRequest: 0xd5b6820> (entity: Media;
    predicate: (name LIKE "Branching_Out"); sortDescriptors: (null);
    limit: 0)
    [snip] fetched a single object
    [snip] successfully fetched media with name Branching_Out
    [snip] asking for media data with specifier: 155
    [snip] executing fetch: <NSFetchRequest: 0xd5bed80> (entity: Media;
    predicate: (uniqueID LIKE "155"); sortDescriptors: (null); limit: 0)
    [snip] fetching media with name Trippy
    [snip] executing fetch: <NSFetchRequest: 0xd5bd6e0> (entity: Media;
    predicate: (name LIKE "Trippy"); sortDescriptors: (null); limit: 0)
    [Switching to process 17364 thread 0x740b]
    Program received signal:  "EXC_BAD_ACCESS".
    [Switching to process 17364 thread 0x740b]

    Printing description of fetchRequest:
    <NSFetchRequest: 0xd5bed80> (entity: Media; predicate: (uniqueID LIKE
    "155"); sortDescriptors: (null); limit: 0)
    ---------------
    Note that it starts a fetch for (name LIKE "Trippy") and crashes but
    that a 'po fetchRequest' shows that it's still stuck on (uniqueID
    LIKE "155"). (uniqueID is stored as a String)

    Also note that name Branching_Out and uniqueID 155 are the same
    object. From the .dump:

    INSERT INTO "ZMEDIA" VALUES(9, 12, 1, NULL, 'public.png',
    '685a2a459396c553d3c40dad8f0df19848546f45', 11, 145227248, NULL,
    'Branching_Out', '155', '/Volumes/dwood/dev/biophony/trunk/
    KareliaSite/Results of the Design Survey.rtfd/Branching Out.png', 10,
    NULL, NULL, 47);

    sqlite> select * from ZMEDIA where ZUNIQUEID like '155';
    9|12|1||public.png|685a2a459396c553d3c40dad8f0df19848546f45|11|
    145227248||Branching_Out|155|/Volumes/dwood/dev/biophony/trunk/
    KareliaSite/Results of the Design Survey.rtfd/Branching Out.png|10|||47

    sqlite> select * from ZMEDIA where ZNAME like 'Branching_Out';
    9|12|1||public.png|685a2a459396c553d3c40dad8f0df19848546f45|11|
    145227248||Branching_Out|155|/Volumes/dwood/dev/biophony/trunk/
    KareliaSite/Results of the Design Survey.rtfd/Branching Out.png|10|||47

    Ah, but 'Trippy' actually doesn't exist! It's real name is
    'Trippy_Tint'. My code first tries Trippy and then will try
    Trippy_Tint. (Part of an internal URL protocol parser.) It's fine
    with getting nil back for Trippy. But is CoreData ok with it?

    sqlite> select * from ZMEDIA where ZNAME like 'Trippy';
    sqlite>
    sqlite> select * from ZMEDIA where ZNAME like 'Trippy_Tints';
    9|8|1||public.png|8c4dfa8dc19ae08543c7e588b551afb9d0eeb417|11|
    145227248||Trippy_Tints|157|/Volumes/dwood/dev/biophony/trunk/
    KareliaSite/Results of the Design Survey.rtfd/Trippy Tints.png|11|||41

    Although the "statement is still active" happens a bit before that,
    on either Aqua or Cathedral which do exist.

    sqlite> select * from ZMEDIA where ZNAME like 'Cathedral';
    9|11|1||public.png|920d40982ab19ff5219345c2ab209d1cd315b89a|11|
    145227248||Cathedral|153|/Volumes/dwood/dev/biophony/trunk/
    KareliaSite/Results of the Design Survey.rtfd/Cathedral.png|9|||42
    sqlite> select * from ZMEDIA where ZNAME like 'Aqua';
    9|10|1||public.png|3e152c30b6b5cea6ceda8a6b0fcd4b9dfc6d450d|11|
    145227248||Aqua|151|/Volumes/dwood/dev/biophony/trunk/KareliaSite/
    Results of the Design Survey.rtfd/Aqua.png|14|||49

    Any ideas?
  • On Oct 6, 2005, at 12:03 PM, Terrence Talbot wrote:
    > No, that's the thing: this should be a very straightforward
    > exercise. Single document, single context, single thread. The only
    > diddling I've done is a .dump (and now some selects, see below) of
    > the document on the command-line to make sure the rows for these
    > objects are there. They are...

    Hmmm... I'm still missing something because I don't understand why it
    wouldn't work.

    - Did you change the model and try to use an existing store afterwords?

    - is the database integrity still intact?  (You can use 'pragma
    integrity_check;' in the SQLite3 command line to check it).

    b.bum
  • On Oct 6, 2005, at 3:30 PM, Bill Bumgarner wrote:

    > On Oct 6, 2005, at 12:03 PM, Terrence Talbot wrote:
    >
    >> No, that's the thing: this should be a very straightforward
    >> exercise. Single document, single context, single thread. The only
    >> diddling I've done is a .dump (and now some selects, see below) of
    >> the document on the command-line to make sure the rows for these
    >> objects are there. They are...
    >>
    >
    > Hmmm... I'm still missing something because I don't understand why
    > it wouldn't work.
    >
    > - Did you change the model and try to use an existing store
    > afterwords?
    >

    No. The model hasn't changed in about 100 checkins and the document
    was created well past that. In fact, I can build a new document that
    will exhibit the same behavior.

    > - is the database integrity still intact?  (You can use 'pragma
    > integrity_check;' in the SQLite3 command line to check it).
    >

    Yep.

    [<ttalbot...> Sample Sites]$sqlite3 Karelia\ Sample.svxSite
    SQLite version 3.1.3
    Enter ".help" for instructions
    sqlite> pragma integrity_check;
    ok
  • On Oct 6, 2005, at 3:30 PM, Bill Bumgarner wrote:

    > On Oct 6, 2005, at 12:03 PM, Terrence Talbot wrote:
    >
    >> No, that's the thing: this should be a very straightforward
    >> exercise. Single document, single context, single thread. The only
    >> diddling I've done is a .dump (and now some selects, see below) of
    >> the document on the command-line to make sure the rows for these
    >> objects are there. They are...
    >>
    >
    > Hmmm... I'm still missing something because I don't understand why
    > it wouldn't work.
    >

    I don't either. Let me try asking this a different way:

    Is there some way, given a single-threaded app, to get an
    executeFetchRequest: going before another one returns? Is there a way
    I could block, in addition to whatever CoreData is doing, to test this?

    Here's a run from a newly created document this morning. Note the log
    statements. It starts a fetch for Cathedral *before* the fetch for
    151 returns! (It then asks to fetch in a relationship for an object,
    Aqua, that was previously fetched successfully and dies. The fetch
    for Cathedral does not appear to return.)

    I rejiggered my code so that all fetches go through the same category
    method on NSManagedObjectContext. I'll paste the method below just to
    show the sequence of the log statements. I would have expected each
    fetch/return log to be paired and, up until one before the crash,
    they are. And it's not always the same object being fetched that
    causes the problem.

    The only pattern I can see, though in previous fetches it's ok, is
    that the problem *seems* to occur after consecutively fetching the
    same object first by name, then by uniqueID (both String attributes).
    For example, Aqua and 151 are the same object, below.

    I'm stumped as to what to try next.

    - Terrence

    --
    [snip] fetch: (entity: Media; predicate: name LIKE "Aqua";)
    [snip] one object returned for (entity: Media; predicate: name LIKE
    "Aqua";)

    [snip] asking for KTMediaURLProtocol dataWithResourceSpecifier: 151
    [snip] fetch: (entity: Media; predicate: uniqueID LIKE "151";)
    [snip] asking for KTMedia objectWithName: Cathedral
    [snip] fetch: (entity: Media; predicate: name LIKE "Cathedral";)
    [snip] one object returned for (entity: Media; predicate: uniqueID
    LIKE "151";)

    [snip] asking for mediaData.contents for Aqua
    [snip] An uncaught exception was raised
    [snip] statement is still active
    [snip] *** Uncaught exception: <NSInternalInconsistencyException>
    statement is still active
    Program received signal:  "EXC_BAD_ACCESS".
    --
    - (NSArray *)objectsWithEntityName:(NSString *)anEntityName
                              predicate:(NSPredicate *)aPredicate
                                  error:(NSError **)anError
    {
        NSAssert((nil != anEntityName), @"anEntityName cannot be nil");

        NSArray *fetchedObjects = nil;
        NSFetchRequest *fetchRequest = [[NSFetchRequest alloc] init];
        NSError *localError = nil;

        @try
        {
            [fetchRequest setEntity:[NSEntityDescription
    entityForName:anEntityName

    inManagedObjectContext:self]];
            if ( nil != aPredicate )
            {
                [fetchRequest setPredicate:aPredicate];
            }

            // LOG fetchRequest
            LOG((@"fetch: %@", [fetchRequest shortDescription]));

            fetchedObjects = [self executeFetchRequest:fetchRequest
    error:&localError];

            // LOG result
            if ( (nil != fetchedObjects) && [fetchedObjects count] == 0 )
            {
                LOG((@"no objects returned for %@\n\n", [fetchRequest
    shortDescription]));
            }
            else if ( (nil != fetchedObjects) && [fetchedObjects count]
    == 1 )
            {
                LOG((@"one object returned for %@\n\n", [fetchRequest
    shortDescription]));
            }
            else if ( (nil != fetchedObjects) && [fetchedObjects count]
    > 1 )
            {
                LOG((@"multiple objects returned for %@\n\n",
    [fetchRequest shortDescription]));
            }
        }
        @finally
        {
            if ( nil != localError )
            {
                LOG((@"error: %@", localError));
                anError = &localError;
            }
            [fetchRequest release];
        }

        if ( nil == fetchedObjects )
        {
            LOG((@"nil returned for %@\n\n", [fetchRequest
    shortDescription]));
        }

        return fetchedObjects;
    }
  • On Oct 6, 2005, at 3:30 PM, Bill Bumgarner wrote:

    > On Oct 6, 2005, at 12:03 PM, Terrence Talbot wrote:
    >
    >> No, that's the thing: this should be a very straightforward
    >> exercise. Single document, single context, single thread. The only
    >> diddling I've done is a .dump (and now some selects, see below) of
    >> the document on the command-line to make sure the rows for these
    >> objects are there. They are...
    >>
    >

    Ok, following up from off-line conversations:

    This did turn out to be a threading problem. Although my code wasn't
    intentionally touching the context outside of the default thread, it
    turns out that WebKit was calling into my code, via WebFrame's -
    loadRequest: method, from a different thread. That sequence, in turn,
    caused the context to execute fetches in other than the main thread.
    Blammo!

    via Chris Hanson:

    "You cannot access the same managed object context from multiple
    threads at once without locking the context appropriately. What's
    more, you cannot lock the context appropriately if some of that
    access happens as a result of bindings, since bindings aren't thread-
    safe. ... I strongly recommend either using a separate context in the
    thread invoking your implementation of -loadRequest:, or using some
    cross-thread communication mechanism to ensure that all access to
    your primary managed object context happens on the main thread."

    In my case, I don't think bindings are the issue, but that's
    certainly worth pouring over in case anyone runs into similar. I also
    can't really alter -loadRequest: as the threading that's going on is
    deep inside WebKit, but I will think about that over the weekend.

    My naive approach, which is actually working, is to have the the
    context -lock and -unlock itself in the -
    objectsWithEntityName:predicate:error method I included in a previous
    post in this thread. (That method actually has a zombie problem in
    that fetchedObjects is released before it's potentially used in a
    log, so fix that if you use it.) I also have to override valueForKey:
    in my managed object subclass and lock/unlock the context there. That
    works. My app *doesn't* crash.

    It's not a performance win, but at least my beta testers can move on
    and I can spend the weekend figuring out a better solution.

    Whew!

    Thanks to b.bum and Chris.
  • On Oct 7, 2005, at 5:53 PM, Terrence Talbot wrote:

    > Ok, following up from off-line
    > conversations:<ttalbot_lists...>
    >
    > This did turn out to be a threading problem. Although my code
    > wasn't intentionally touching the context outside of the default
    > thread, it turns out that WebKit was calling into my code, via
    > WebFrame's -loadRequest: method, from a different thread. That
    > sequence, in turn, caused the context to execute fetches in other
    > than the main thread. Blammo!
    >
    > [snip]
    > My naive approach, which is actually working, is to have the the
    > context -lock and -unlock itself in the -
    > objectsWithEntityName:predicate:error method I included in a
    > previous post in this thread. (That method actually has a zombie
    > problem in that fetchedObjects is released before it's potentially
    > used in a log, so fix that if you use it.) I also have to override
    > valueForKey: in my managed object subclass and lock/unlock the
    > context there. That works. My app *doesn't* crash.
    >

    One last quick post on this:

    Since it was actually a single, isolated case where WebKit was
    calling back into my code and causing problems for the context, I was
    able to boil it down to a couple of performSelectorOnMainThread:
    calls and all is happy. I'm no longer overriding valueForKey: or
    setting locks all over the place. Everything is working just fine. Go
    CoreData!
previous month october 2005 next month
MTWTFSS
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31            
Go to today