Ordered Core Data

  • I'm thinking about using Core Data to develop a new application.
    Unfortunately, it involves elements that need to be saved in order --
    an order that users might rearrange by dragging. Core Data doesn't
    guarantee anything about ordering (other than sorting by value). In
    fact, the OutlineEdit example doesn't read files in the same order it
    saves them.

    I suspect using an internal index can solve this, but I'm a little
    concerned about performance: what if the user adds an element at the
    very beginning of the list? This means that each element has to have
    its index incremented. I know Core Data is relatively fast, but this
    still seems like it could be asking for trouble, since the data set
    could be arbitrarily large.

    Any thoughts on solutions (or actual performance)?

    David Dunham    A Sharp, LLC
    Voice/Fax: 206 783 7404    http://a-sharp.com
    Efficiency is intelligent laziness.
  • Am 11.06.2005 um 19:19 schrieb David Dunham:

    > I'm thinking about using Core Data to develop a new application.
    > Unfortunately, it involves elements that need to be saved in order
    > -- an order that users might rearrange by dragging.

    > I suspect using an internal index can solve this, but I'm a little
    > concerned about performance: what if the user adds an element at
    > the very beginning of the list?

    As the data representation in memory (obviously) doesn't change, at
    worst a couple of pointers get shuffled around. This is why you
    usually read -arrangedObjects and not -managedObjectContext (they
    differ, as I've learned a few weeks ago here).

    > This means that each element has to have its index incremented.

    Isn't there another field you can sort the data by without the risk
    of having conflicts?

    If there isn't, how about creating the index in steps of 100? This
    way you can put new items in between and have to rewrite the whole
    index rarely.

    Above you said you'd have to _save_ the data in order. Since you have
    few influence how Core Data stores away the batch of information, I'm
    not sure where you can take advantage of it's capabilities. But
    perhaps you just have to make sure the data is viewed correctly
    sorted always. This is easily done, see that recent thread "Sorting
    Core Data data".

    HTH,
    Markus

    - - - - - - - - - - - - - - - - - - -
    Dipl. Ing. Markus Hitter
    http://www.jump-ing.de/
  • On Jun 11, 2005, at 10:19 AM, David Dunham wrote:

    > I suspect using an internal index can solve this, but I'm a little
    > concerned about performance: what if the user adds an element at
    > the very beginning of the list? This means that each element has to
    > have its index incremented. I know Core Data is relatively fast,
    > but this still seems like it could be asking for trouble, since the
    > data set could be arbitrarily large.
    >

    It's not clear why performance using Core Data would be significantly
    different from performance if you were using custom subclasses of
    NSObject that used an index?  That is, if you need to support
    ordering, then you have to provide a mechanism to do so whether or
    not you use Core Data.  In what respect might Core Data impede that
    mechanism?

    mmalc
  • On 11 Jun 2005, at 19:03, mmalcolm crawford wrote:

    > It's not clear why performance using Core Data would be
    > significantly different from performance if you were using custom
    > subclasses of NSObject that used an index?  That is, if you need to
    > support ordering, then you have to provide a mechanism to do so
    > whether or not you use Core Data.  In what respect might Core Data
    > impede that mechanism?

    Without Core Data, I can do without a per-element index, and just
    keep everything in an NSArray. NSArrays don't forget the order of
    insertion, the way Core Data does.

    David Dunham    A Sharp, LLC
    Voice/Fax: 206 783 7404    http://a-sharp.com
    Efficiency is intelligent laziness.
  • On Jun 12, 2005, at 5:08 PM, David Dunham wrote:

    >> It's not clear why performance using Core Data would be
    >> significantly different from performance if you were using custom
    >> subclasses of NSObject that used an index?  That is, if you need
    >> to support ordering, then you have to provide a mechanism to do so
    >> whether or not you use Core Data.  In what respect might Core Data
    >> impede that mechanism?
    > Without Core Data, I can do without a per-element index, and just
    > keep everything in an NSArray.
    >
    Sorry I misunderstood your original.

    In that case, I still don't think there should be a significant
    (compared to the rest of the application) overhead.  Would you be
    doing anything more than getting and setting the index instance
    variable?

    One other issue would be, at what stage are you actually setting the
    indexes.  If you have just a single ordered table managed by an array
    controller, then you might be able to defer updating the indexes
    until just before save...

    mmalc
  • On 12 Jun 2005, at 17:47, mmalcolm crawford wrote:

    > In that case, I still don't think there should be a significant
    > (compared to the rest of the application) overhead.  Would you be
    > doing anything more than getting and setting the index instance
    > variable?

    I guess I would, come to think of it.

    > One other issue would be, at what stage are you actually setting
    > the indexes.  If you have just a single ordered table managed by an
    > array controller, then you might be able to defer updating the
    > indexes until just before save...

    I'd been hoping to get all that free undo that Core Data provides,
    which I don't think this approach provides (since I'd still need to
    undo any array changes).

    I think I may have been missing the likelihood that the managed
    object context could simply keep things around in memory, much like
    an NSArray would, and that the overhead is therefore fairly low.

    David Dunham    A Sharp, LLC
    Voice/Fax: 206 783 7404    http://a-sharp.com
    "People seem to misinterpret complexity as sophistication" -- Niklaus
    Wirth
  • On Jun 11, 2005, at 1:19 PM, David Dunham wrote:

    > I suspect using an internal index can solve this, but I'm a little
    > concerned about performance: what if the user adds an element at
    > the very beginning of the list? This means that each element has to
    > have its index incremented. I know Core Data is relatively fast,
    > but this still seems like it could be asking for trouble, since the
    > data set could be arbitrarily large.
    >
    > Any thoughts on solutions (or actual performance)?

    "Premature optimization..."

    Protect yourself by encapsulating your unknown from the rest of your
    code.  Then implement the unknown in the simplest way that could
    possibly work.  If it doesn't perform for you, optimize within the
    encapsulated space as necessary.  But more than likely, YAGNI.

    Erik
  • On 13 Jun 2005, at 18:41, Erik Price wrote:

    >> Any thoughts on solutions (or actual performance)?
    >
    > "Premature optimization..."

    I'm treating it as selection of algorithm, which is always the most
    important part of optimization. (This isn't something I can just whip
    together with bindings.) It's sorta like the difference between
    choosing a stable sort or a non-stable one. (BTW I'll bet Core Data
    sorting isn't stable, since they can't normally get you information
    in the order you arrange it.)

    David Dunham    A Sharp, LLC
    Voice/Fax: 206 783 7404    http://a-sharp.com
    "People seem to misinterpret complexity as sophistication" -- Niklaus
    Wirth
  • I am getting a bit tired of this apparently deliberate obfuscation.

    The problem here, and it is a huge one, is that there is not a
    managed-object friendly version of NSArray, and it is a gaping hole
    in the implementation of CoreData. ***There is simply no transparent
    way of representing an ordered collection of arbitrary managed
    objects.***

    It is *obviously* not as simple as getting/setting an index variable
    which is why I find this kind of response so annoying.

    Let us assume we have 10,000 objects, each with an index variable.

    We want to insert an object at the beginning.

    We must now riffle down through our ordered collection and increment
    the index variable for all 10,000 objects. Of course we can start
    with indices 100 apart or whatever but it's a miserable way to have
    to think.

    There are ways around this, obviously, but they *all* involve either
    (a) rolling your own datastructures or (b) creating NSMutableArrays
    on the fly as temporary ivars in a managed object, with code to
    unroll them into indexed sets on save.

    As for (a) it is ridiculous that in a IDE like Cocoa, a user should
    have to write their own black/red tree implementation simply to be
    able to preserve the order of a collection of arbitrary elements (I
    soon found that linked lists weren't nearly fast enough).

    As for (b) it is obviously a kludge, and every time I raise it as a
    possibility I am warned off.

    This single lacuna has caused me endless headaches and I still have
    not had a really good solution proposed.

    On 12-Jun-05, at 8:47 PM, mmalcolm crawford wrote:

    > In that case, I still don't think there should be a significant
    > (compared to the rest of the application) overhead.  Would you be
    > doing anything more than getting and setting the index instance
    > variable?
  • It all boils down to the fact that the relational database model
    doesn't lend itself nicely to ordered collections of instances of
    entities with insertion and removal.

    It's not a core data problem. It's a relational database problem.
    You're going to find the same issues on any other RDB. I know I have.
    Usually it means you should think if that is really the right design
    to go. And if you really, really, really have to do it, then your
    life sucks a little more and you have to do it manually.

    If the main part of your application's data model has those
    characteristics, maybe you should think about not using Core Data, or
    you can just go and implement the manual indexing which from what
    I've heard about the performance of SQLite shouldn't be that bad.

    Hope that helps.

    Oscar Morales Vivo

    On Jun 14, 2005, at 09:04 , Johnny Deadman wrote:

    > I am getting a bit tired of this apparently deliberate obfuscation.
    >
    > The problem here, and it is a huge one, is that there is not a
    > managed-object friendly version of NSArray, and it is a gaping hole
    > in the implementation of CoreData. ***There is simply no
    > transparent way of representing an ordered collection of arbitrary
    > managed objects.***
    >
    > It is *obviously* not as simple as getting/setting an index
    > variable which is why I find this kind of response so annoying.
    >
    > Let us assume we have 10,000 objects, each with an index variable.
    >
    > We want to insert an object at the beginning.
    >
    > We must now riffle down through our ordered collection and
    > increment the index variable for all 10,000 objects. Of course we
    > can start with indices 100 apart or whatever but it's a miserable
    > way to have to think.
    >
    > There are ways around this, obviously, but they *all* involve
    > either (a) rolling your own datastructures or (b) creating
    > NSMutableArrays on the fly as temporary ivars in a managed object,
    > with code to unroll them into indexed sets on save.
    >
    > As for (a) it is ridiculous that in a IDE like Cocoa, a user should
    > have to write their own black/red tree implementation simply to be
    > able to preserve the order of a collection of arbitrary elements (I
    > soon found that linked lists weren't nearly fast enough).
    >
    > As for (b) it is obviously a kludge, and every time I raise it as a
    > possibility I am warned off.
    >
    > This single lacuna has caused me endless headaches and I still have
    > not had a really good solution proposed.
    >
    >
    > On 12-Jun-05, at 8:47 PM, mmalcolm crawford wrote:
    >
    >
    >> In that case, I still don't think there should be a significant
    >> (compared to the rest of the application) overhead.  Would you be
    >> doing anything more than getting and setting the index instance
    >> variable?
    >>
    >
    > _______________________________________________
    > MacOSX-dev mailing list
    > <MacOSX-dev...>
    > http://www.omnigroup.com/mailman/listinfo/macosx-dev
    >
  • On Jun 14, 2005, at 6:04 AM, Johnny Deadman wrote:

    > I am getting a bit tired of this apparently deliberate obfuscation.
    >
    There is no intent at obfuscation.  There is simply no way to give a
    concrete answer without knowing more about a given application.  The
    overhead that using an index incurs is a relative quantity, and
    depends on what else the application is doing.

    > The problem here, and it is a huge one, is that there is not a
    > managed-object friendly version of NSArray, and it is a gaping hole
    > in the implementation of CoreData. ***There is simply no
    > transparent way of representing an ordered collection of arbitrary
    > managed objects.***
    >
    Please file an enhancement request.

    > It is *obviously* not as simple as getting/setting an index
    > variable which is why I find this kind of response so annoying.
    > Let us assume we have 10,000 objects, each with an index variable.
    > We want to insert an object at the beginning.
    > We must now riffle down through our ordered collection and
    > increment the index variable for all 10,000 objects.
    >
    I don't see anything complex about that.  It's tedious, but to me
    it's simple.

    > There are ways around this, obviously, but they *all* involve
    > either (a) rolling your own datastructures or (b) creating
    > NSMutableArrays on the fly as temporary ivars in a managed object,
    > with code to unroll them into indexed sets on save.
    > As for (a) it is ridiculous that in a IDE like Cocoa, a user should
    > have to write their own black/red tree implementation simply to be
    > able to preserve the order of a collection of arbitrary elements (I
    > soon found that linked lists weren't nearly fast enough).
    > As for (b) it is obviously a kludge, and every time I raise it as a
    > possibility I am warned off.
    >
    I don't recall having seen you raise (b) as a possibility before.

    mmalc
  • One solution I'm toying with on the insertion-at-beginning-
    of-10,000-items problem is, on awakeFromInsert, set the index to -1.
    On save, if there're any set to -1, I reorder all, starting the -1's
    at 0. It isn't pretty and it's still just an idea for now, but it's
    one I'm toying with. Unfortunately, one problem that jumps out at me
    is what to do about a reverse-order sort. It's even uglier (in my
    mind) to find the highest-number++. Again, just a first thought. I've
    written 0 lines of code to attempt this so far.

      Maybe if someone here beats me to trying it out, they could let us
    all know how it works. Admittedly, it might be a total waste of time,
    but then so are most things I try ... ;-)

      I agree with Johnny, however. An ordered collection mechanism that
    works seamlessly with Core Data and Cocoa Bindings would be a hunk of
    solid gold for many developers. It's also very likely a pipe dream
    (at least for 10.4), given the relational database issues Oscar
    mentions. I imagine it would take a pretty big change to Cocoa and
    Core Data which is too late to change for Tiger ...

      Again, I may be talking out of my butt, being completely wrong
    about everything I just said (I defer to the list on that issue), but
    this *is* a pretty big speed bump that seems like it wants to be
    simple. ;-)

    On Jun 14, 2005, at 9:19 AM, Óscar Morales Vivó wrote:

    > It all boils down to the fact that the relational database model
    > doesn't lend itself nicely to ordered collections of instances of
    > entities with insertion and removal.
    >
    > It's not a core data problem. It's a relational database problem.
    > You're going to find the same issues on any other RDB. I know I
    > have. Usually it means you should think if that is really the right
    > design to go. And if you really, really, really have to do it, then
    > your life sucks a little more and you have to do it manually.
    >
    > If the main part of your application's data model has those
    > characteristics, maybe you should think about not using Core Data,
    > or you can just go and implement the manual indexing which from
    > what I've heard about the performance of SQLite shouldn't be that bad.
    >
    > Hope that helps.
    >
    > Oscar Morales Vivo
    >
    > On Jun 14, 2005, at 09:04 , Johnny Deadman wrote:
    >
    >
    >> I am getting a bit tired of this apparently deliberate obfuscation.
    >>
    >> The problem here, and it is a huge one, is that there is not a
    >> managed-object friendly version of NSArray, and it is a gaping
    >> hole in the implementation of CoreData. ***There is simply no
    >> transparent way of representing an ordered collection of arbitrary
    >> managed objects.***
    >>
    >> It is *obviously* not as simple as getting/setting an index
    >> variable which is why I find this kind of response so annoying.
    >>
    >> Let us assume we have 10,000 objects, each with an index variable.
    >>
    >> We want to insert an object at the beginning.
    >>
    >> We must now riffle down through our ordered collection and
    >> increment the index variable for all 10,000 objects. Of course we
    >> can start with indices 100 apart or whatever but it's a miserable
    >> way to have to think.
    >>
    >> There are ways around this, obviously, but they *all* involve
    >> either (a) rolling your own datastructures or (b) creating
    >> NSMutableArrays on the fly as temporary ivars in a managed object,
    >> with code to unroll them into indexed sets on save.
    >>
    >> As for (a) it is ridiculous that in a IDE like Cocoa, a user
    >> should have to write their own black/red tree implementation
    >> simply to be able to preserve the order of a collection of
    >> arbitrary elements (I soon found that linked lists weren't nearly
    >> fast enough).
    >>
    >> As for (b) it is obviously a kludge, and every time I raise it as
    >> a possibility I am warned off.
    >>
    >> This single lacuna has caused me endless headaches and I still
    >> have not had a really good solution proposed.
    >>
    >>
    >> On 12-Jun-05, at 8:47 PM, mmalcolm crawford wrote:
    >>
    >>
    >>
    >>> In that case, I still don't think there should be a significant
    >>> (compared to the rest of the application) overhead.  Would you be
    >>> doing anything more than getting and setting the index instance
    >>> variable?
    >>>
    >>>
    >>
    >> _______________________________________________
    >> MacOSX-dev mailing list
    >> <MacOSX-dev...>
    >> http://www.omnigroup.com/mailman/listinfo/macosx-dev
    >>
    >>
    >
    > _______________________________________________
    > Do not post admin requests to the list. They will be ignored.
    > Cocoa-dev mailing list      (<Cocoa-dev...>)
    > Help/Unsubscribe/Update your Subscription:
    > http://lists.apple.com/mailman/options/cocoa-dev/dev%
    > 40silentalcove.net
    >
    > This email sent to <dev...>
    >
  • On Jun 14, 2005, at 9:49 AM, mmalcolm crawford wrote:

    >> ***There is simply no transparent way of representing an ordered
    >> collection of arbitrary managed objects.***

    > Please file an enhancement request.

    Filed on the day Tiger was released I think... marked as 'dupe' FWIW

    >> It is *obviously* not as simple as getting/setting an index
    >> variable which is why I find this kind of response so annoying.
    >> Let us assume we have 10,000 objects, each with an index variable.
    >> We want to insert an object at the beginning.
    >> We must now riffle down through our ordered collection and
    >> increment the index variable for all 10,000 objects.
    >>
    > I don't see anything complex about that.  It's tedious, but to me
    > it's simple.

    (1) it is more than 'simply getting/setting an index variable', which
    is the obfuscation I was talking about
    (2) it is too slow for anything but trivial applications. If the time
    to insert an arbitrary item in a list of N item scales with N, you
    are in trouble.

    >> There are ways around this, obviously, but they *all* involve
    >> either (a) rolling your own datastructures or (b) creating
    >> NSMutableArrays on the fly
    >> [snip]

    > I don't recall having seen you raise (b) as a possibility before.

    I certainly have off-list, maybe not with you though. It may be that
    this turns out to be the best interim solution, despite the
    kludginess, since it offloads the overhead of maintaining the ordered
    list onto tried-and-trusted structures like NSArray.
  • Am 14.06.2005 um 15:04 schrieb Johnny Deadman:

    > The problem here, and it is a huge one, is that there is not a
    > managed-object friendly version of NSArray, and it is a gaping hole
    > in the implementation of CoreData. ***There is simply no
    > transparent way of representing an ordered collection of arbitrary
    > managed objects.***

    Without re-reseaching the issue in depth, the default object store
    for an NSManagedObjectContext is an NSMutableArray and you are free
    to re-arrange this array to your hearts contents. This shouldn't
    affect functionality of the levels above and below the
    NSManagedObjectContext.

    Even data exchange with the external store should continue to work
    since each NSManagedObject maintains an unique identifier which is
    invisible to the upper layers. For comparison: you have to set the
    identifier explicitely in the similar EOF framework.

    However, you get more stuff "for free" if you can use orderedObjects
    instead of the base array. orderedObjects behave a lot like an
    NSArray and are in fact an array of proxy objects, as far as I found
    out so far.

    > We want to insert an object at the beginning.

    If you insist in using an NSArray, not even an NSMutableArray, much
    less an NSManagedObjectContext, you have to roll your own storage
    handling methods, obviously.

    > This single lacuna has caused me endless headaches and I still have
    > not had a really good solution proposed.

    "Never fight the frameworks"

    Markus

    - - - - - - - - - - - - - - - - - - -
    Dipl. Ing. Markus Hitter
    http://www.jump-ing.de/
  • On Jun 14, 2005, at 6:04 AM, Johnny Deadman wrote:
    > The problem here, and it is a huge one, is that there is not a
    > managed-object friendly version of NSArray, and it is a gaping hole
    > in the implementation of CoreData. ***There is simply no
    > transparent way of representing an ordered collection of arbitrary
    > managed objects.***

    That is correct and, yes, it is a limitation in the 1.0 release of
    Core Data.

    In evaluating the feature set that could be engineered for a 1.0,
    every feature was considered in light of both engineering effort and
    return on investment of said engineering effort.

    Unfortunately, direct modeling and maintenance of ordered
    relationships fell of the list of 1.0 features for a number of reasons.

    First, it is very important to note that the automatic maintenance of
    ordered relationships has nothing to do with the display of ordered
    sets of data within the user interface.  In most cases, if the
    developer is displaying a set of objects in the UI, it is via a table
    view, a browser, an outline view, or some other UI widget that will
    either have an implied order or will allow the user to re-order the
    contents on the fly by clicking on a column header.    Furthermore,
    for table view and table view like widgets, the contents are very
    often qualified by a predicate or the contents are the result of a
    fetch request, which can have a sort order that incurs relatively
    little overhead.

    Conclusion:  Maintaining order of an entity's instances within the
    object model contributes little to most user interfaces.

    Secondly, offering a generic solution for maintaining ordered
    relationships is rife with caveats that make it extremely difficult
    to do in a universally useful fashion.  Some various random issues:

        - can an object be a member of a relationship more than once?
            - No?  What happens when the object is inserted a second time?
        - should it support an order implied by an attribute's value?
            - what about duplicate values?
                - should there be a secondary value?
        - can a relationship have multiple sort orders?
        - if a relationship TO an object should be modified by a change
    to that object, is there an efficient way to handle the change if the
    relationship hasn't been faulted?
            - does this mean that every relationship must be aware of
    its inverse?

    Conclusion:  Modeling ordered relationships is considerably more
    complex than just adding support for relationship containment within
    an array.

    Given that maintaining order on a relationship is just not that hard
    of a problem to solve when viewed from a per-app perspective (many of
    the above requirements are mutually exclusive, therefore any single
    use of ordered relationships will require answers to a limited number
    of the above questions).  Re-applying indices across all objects in
    a relationship will incur effectively no overhead in the in-memory,
    XML, or Binary store cases.  For the SQL store, it will be
    significantly slower to modify the 10,000 objects, but any of a
    number of approaches to the implementation/modeling can yield
    efficient solutions without adding a significant performance penalty.

    Finally, thank you for filing the enhancement request.  Even though
    it was returned as a dupe, the specific use case information that you
    may have included in the bug -- any details about the problem you
    were hoping to solve -- is captured and evaluated as a part of
    determining features contained in the next release of the product.

    b.bum
  • On Jun 14, 2005, at 8:50 AM, Bill Bumgarner wrote:
    > On Jun 14, 2005, at 6:04 AM, Johnny Deadman wrote:
    >> The problem here, and it is a huge one, is that there is not a
    >> managed-object friendly version of NSArray, and it is a gaping
    >> hole in the implementation of CoreData. ***There is simply no
    >> transparent way of representing an ordered collection of arbitrary
    >> managed objects.***
    > That is correct and, yes, it is a limitation in the 1.0 release of
    > Core Data.

    Just to avoid the confusion -- it is a limitation in 1.0, but not a
    huge limitation....
  • On Jun 14, 2005, at 6:04 AM, Johnny Deadman wrote:
    > The problem here, and it is a huge one, is that there is not a
    > managed-object friendly version of NSArray, and it is a gaping hole
    > in the implementation of CoreData. ***There is simply no
    > transparent way of representing an ordered collection of arbitrary
    > managed objects.***

    Relationships are not collections.  Relationships are relations.
    Relations can be used to represent containment, or they can be used
    to represent association.

      -- Chris
  • On Jun 14, 2005, at 7:28 AM, Markus Hitter wrote:
    > Without re-reseaching the issue in depth, the default object store
    > for an NSManagedObjectContext is an NSMutableArray and you are free
    > to re-arrange this array to your hearts contents. This shouldn't
    > affect functionality of the levels above and below the
    > NSManagedObjectContext.

    This is incorrect in a number of ways.  There is no such thing as a
    "default object store for an NSManagedObjectContext."
    NSMutableArrays are not used to represent relationships in Core Data,
    and with only a couple of exceptions any NSMutableSet you get from
    Core Data should *not* be manipulated directly.

    The only time you should mutate a set representing a relationship
    directly in client code is when it was returned by -
    mutableSetValueForKey:.  The *only* exception is when implementing
    mutable-set mutation accessors in your managed objects, and then you
    should *only* do so in the same manner as the templates that the
    Xcode design tools will generate for you, within a -
    willChangeValueForKey:withSetMutation:usingObjects: and -
    didChangeValueForKey:withSetMutation:usingObjects: pair.

      -- Chris
  • On Jun 14, 2005, at 5:38 PM, Chris Hanson wrote:

    >> The problem here, and it is a huge one, is that there is not a
    >> managed-object friendly version of NSArray, and it is a gaping
    >> hole in the implementation of CoreData. ***There is simply no
    >> transparent way of representing an ordered collection of arbitrary
    >> managed objects.***
    >>
    >
    > Relationships are not collections.  Relationships are relations.
    > Relations can be used to represent containment, or they can be used
    > to represent association.

    I assume you are being pedantic about RDBMS in some way that escapes me.

    Take this (ridiculously oversimplified) example. Let us say I wish to
    use CoreData to model the scenes in a screenplay.

    I naturally create two entities called Screenplay and Scene.

    Screenplay naturally has a relationship 'scenes' to Scene entities.

    Each scene may appear in an arbitrary number of Screenplays in
    different positions.

    Each screenplay must know, manipulate and persist the order of its
    Scenes.

    The Scene wishes to know nothing about Screenplays, indices, linked
    lists, or anything else. It is just a Scene and cares not for whether
    it is in a screenplay or on the dark side of the moon.

    It is plain from this description that the ordering of the scenes is
    an attribute of the Screenplay, and a fundamental one.

    How should this be represented in CoreData? Pedantically speaking?

    You see immediately why using an index attribute for the Scene is
    totally wrong.

    Several other options present themselves, all bad. One could create a
    new entity 'IndexedItem' with an 'index' attribute and a to-one
    relationship to the Scene entity, and add a new relationship
    'indexedScenes' which points to IndexedItems which in turn point to
    the Scenes. Ugh. But we want to generalize this so that we can index
    anything so we have to create an entity 'IndexableItem' and make that
    the parent of any entity we wish to index. Now our entity diagram is
    being twisted to facilitate the implementation. Yuk.

    (This is how I persist my temporary NSMutableArrays, however, for
    want of a better way).

    Or you can start writing linked lists which do much the same thing
    except that they maintain pointers to 'nextItem' instead of keeping
    an index, but it's just as ugly.

    <RANT>

    Isn't it obvious that this should be easier? I am hearing a lot of
    flim flam but no solutions that are not horrible.

    I am talking about Screenplays, but the same problem exists in
    hundreds of situations. Queues, waypoints, fill in the blanks. So
    much of the conversation on this topic assumes that ordering a data
    set is just something for the user interface, but in many models the
    ordering of the items in a collection is a fundamental property. I
    simply do not believe that the RDBMS world does not have good
    solutions for these problems that could be implemented in CoreData.

    </RANT>
  • On Jun 14, 2005, at 4:27 PM, John Brownlow wrote:
    > Or you can start writing linked lists which do much the same thing
    > except that they maintain pointers to 'nextItem' instead of keeping
    > an index, but it's just as ugly.

    Actually, depending on context of implementation, using a linked list
    is both quite easy and very efficient.  Obviously, inserting/deleting
    objects into a linked list is trivial and fast if you have both a
    next and a previous pointer.  Given that Core Data will maintain the
    inverse relationship automatically, it will also maintain the linked
    list pointers automatically.

    Given that many UI type usage contexts will require all of the
    objects to be faulted into memory anyway, walking the "next"
    relations and filling an NSMutableArray for display purposes is about
    three lines of code.

    So, for the particular example of Screenplay -> Scenes, a linked list
    might very well be a totally acceptable way of modeling this.

    In this particular example, the amount of overhead in the form of
    additional lines of code incurred by the developer is pretty trivial.

    Yeah -- it is a bummer that Core Data doesn't have direct modeling of
    ordered relationships.  See my other email for why that decision was
    made.

    b.bum
  • On 14-Jun-05, at 7:44 PM, Bill Bumgarner wrote:

    > Actually, depending on context of implementation, using a linked
    > list is both quite easy and very efficient.  Obviously, inserting/
    > deleting objects into a linked list is trivial and fast if you have
    > both a next and a previous pointer.  Given that Core Data will
    > maintain the inverse relationship automatically, it will also
    > maintain the linked list pointers automatically.

    Yes, I know. I did it after (I think you or) John suggested it off
    list. It is quite a general solution but... The problem is that
    accessing list items is really slow. I run through the list
    frequently while the user is typing and it slowed everything down. I
    implemented some caching of the list as an NSArray and lazy updating
    and that has just about brought the access times under control but it
    is messier now, and more error-prone, and I feel a heck of a lot less
    abstracted from the implementation.

    Using a b-tree in my case didn't really help because the tree tends
    to decompose into a linked list anyway.
  • On 15/06/2005, at 03:22, Johnny Deadman wrote:

    >
    > On 14-Jun-05, at 7:44 PM, Bill Bumgarner wrote:
    >
    >
    >> Actually, depending on context of implementation, using a linked
    >> list is both quite easy and very efficient.  Obviously, inserting/
    >> deleting objects into a linked list is trivial and fast if you
    >> have both a next and a previous pointer.  Given that Core Data
    >> will maintain the inverse relationship automatically, it will also
    >> maintain the linked list pointers automatically.
    >>
    >
    > Yes, I know. I did it after (I think you or) John suggested it off
    > list. It is quite a general solution but... The problem is that
    > accessing list items is really slow.

    Quite useless for random access with large data sets. I have planned
    on using Core Data as a solution for an application in which strict
    ordering plays a large role in content semantics. Now I seriously
    doubt I'll use it, because of its deficiencies in this regard.

    Does anybody know if DataCrux (http://treehouseideas.com) has a
    better solution to these problems?

    Sharon
  • On Jun 14, 2005, at 8:30 PM, Sharon Rosner wrote:
    > On 15/06/2005, at 03:22, Johnny Deadman wrote:
    >> On 14-Jun-05, at 7:44 PM, Bill Bumgarner wrote:
    >>> Actually, depending on context of implementation, using a linked
    >>> list is both quite easy and very efficient.  Obviously, inserting/
    >>> deleting objects into a linked list is trivial and fast if you
    >>> have both a next and a previous pointer.  Given that Core Data
    >>> will maintain the inverse relationship automatically, it will
    >>> also maintain the linked list pointers automatically.
    >> Yes, I know. I did it after (I think you or) John suggested it off
    >> list. It is quite a general solution but... The problem is that
    >> accessing list items is really slow.
    > Quite useless for random access with large data sets. I have
    > planned on using Core Data as a solution for an application in
    > which strict ordering plays a large role in content semantics. Now
    > I seriously doubt I'll use it, because of its deficiencies in this
    > regard.
    >
    > Does anybody know if DataCrux (http://treehouseideas.com) has a
    > better solution to these problems?

    Given that DataCrux is a relatively thin layer on top of SQLite, you
    will run in to the same problems in terms of random data access.

    Of course, I don't understand what "quite useless for random access
    with large data sets" actually means.

    Core Data uses Predicates to define Fetch Requests that allow for the
    construction and evaluation of arbitrary queries against the store or
    stores, including merging in changes that may be present in the in-
    memory object graph state.  This has proven to be extremely useful
    for random access to large data sets.  While Core Data does not
    offer the ability to evaluate hand tuned SQL, doing so has rarely
    proven to fix a performance problem that hasn't better been solved
    through other, more straightforward, means.

    What are your requirements?  I.e. how many objects of the type
    requiring a strict ordering will typically be present?  How will they
    be ordered?  How often will they be reordered? etc..?

    b.bum
  • Am 14.06.2005 um 23:44 schrieb Chris Hanson:

    > On Jun 14, 2005, at 7:28 AM, Markus Hitter wrote:
    >
    >> Without re-reseaching the issue in depth, the default object store
    >> for an NSManagedObjectContext is an NSMutableArray and you are
    >> free to re-arrange this array to your hearts contents. This
    >> shouldn't affect functionality of the levels above and below the
    >> NSManagedObjectContext.
    >
    > This is incorrect in a number of ways.

    You are right, I mixed up "NSManagedObjectContext" with
    "NSArrayController". In all tutorials I read so far, they use an
    NSArrayController, bound to an NSManagedObjectContext, to talk to the
    persistent store. The NSArrayController is the class using the
    NSMutableArray (inherited from NSObjectController) and the array
    (arrangedObjects) of proxy objects.

    Markus

    - - - - - - - - - - - - - - - - - - -
    Dipl. Ing. Markus Hitter
    http://www.jump-ing.de/
  • On Jun 15, 2005, at 1:41 AM, Markus Hitter wrote:

    > In all tutorials I read so far, they use an NSArrayController,
    > bound to an NSManagedObjectContext, to talk to the persistent store.
    >
    It may be useful to work through this one:
        <http://developer.apple.com/documentation/Cocoa/Conceptual/
    CoreDataUtilityTutorial/index.html
    >

    mmalc
  • Am 15.06.2005 um 11:05 schrieb mmalcolm crawford:

    > On Jun 15, 2005, at 1:41 AM, Markus Hitter wrote:
    >
    >> In all tutorials I read so far, they use an NSArrayController,
    >> bound to an NSManagedObjectContext, to talk to the persistent store.
    >>
    > It may be useful to work through this one:
    > <http://developer.apple.com/documentation/Cocoa/Conceptual/
    > CoreDataUtilityTutorial/index.html>

    Thanks for the link, mmalcolm, but I don't feel the need to get rid
    of the NSArrayController, currently. I've solved problems which sound
    similar to what the others are struggling with and I neither need
    enourmous amounds of code (five lines for the sorting stuff) nor do I
    see performance issues.

    Perhaps I should send a copy of my project to those others so they
    can push a lot of data into and measure the performance the way they
    like it.

    Have fun,
    Markus

    - - - - - - - - - - - - - - - - - - -
    Dipl. Ing. Markus Hitter
    http://www.jump-ing.de/
  • ...snip...

    > How many attributes per object?  ...relationships?
    >
    > I'm trying to get a relative feel for the data.
    >
    > Is it a time series data kind of application?

    Not really. Although it does have an indirect relationship to time.
    It's really a content creation application, much like a
    word-processor, but the content is not text. Objectsts have a few
    different kinds but they are basically very similar - about 10
    attributes per object. Relationships are currently not needed.

    > OK -- will this be done in large batches or typically one at a time?

    Either way. The user will be able to perform operations on large
    selections or on individual objects.

    > This is starting to sound a lot like a time-series kind of a
    > problem.  This is something that neither relational stores nor Core
    > Data are terribly well equipped to handle.
    >
    > I'll ask the Core Data folks tomorrow if they have any ideas -- my
    > office is next to theirs.
    >
    > b.bum
    >

    Thanks for all your help.

    s.
previous month june 2005 next month
MTWTFSS
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30      
Go to today