Core Data troubles - not removing objects

  • Hi all,

    I am pretty lost in the development of my app which uses Core Data and
    an SQLiteStore. I hope someone can point me towards the obvious point
    I am missing.

    Given are two entities with bidirectional 1-to-1 relationships set to
    cascade upon deletion. The first entity contains strings and the other
    one contains binary data (like images and stuff). I set up two
    NSArrayControllers in IB and made the second one contain a set of the
    selected item from the first and also set the Deletes objects on
    remove checkbox in the Inspector.

    My problem is that if I send the remove: or removeObject: message to
    the first NSArrayController and then query said item if it is
    isDeleted, I get true for the item itself, but false for the item
    connected through the relationship. I then manually removed the second
    object by sending a removeObject: message to it. Calling isDeleted now
    returns true.

    All is nice except the SQLiteStore file does not get reduced in size.
    I guess that somehow I am doing something wrong and the objects do not
    get removed form the object graph at the next save: operation.

    Is there anything obviously wrong or missing from my way of handling
    this? I could post some code if required.

    Thanks,
    ben
  • On Nov 1, 2007, at 2:04 AM, Benjámin Salánki wrote:

    > Hi all,
    >
    > I am pretty lost in the development of my app which uses Core Data
    > and an SQLiteStore. I hope someone can point me towards the obvious
    > point I am missing.
    >
    > Given are two entities with bidirectional 1-to-1 relationships set
    > to cascade upon deletion. The first entity contains strings and the
    > other one contains binary data (like images and stuff). I set up two
    > NSArrayControllers in IB and made the second one contain a set of
    > the selected item from the first and also set the Deletes objects on
    > remove checkbox in the Inspector.
    >
    > My problem is that if I send the remove: or removeObject: message to
    > the first NSArrayController and then query said item if it is
    > isDeleted, I get true for the item itself, but false for the item
    > connected through the relationship. I then manually removed the
    > second object by sending a removeObject: message to it. Calling
    > isDeleted now returns true.
    >
    The cascade delete rule isn't processed until you save the managed
    object context in which you made the change, you can force the delete
    rule processing by calling processPendingChanges.  I believe this is
    mentioned in the release notes here:  http://developer.apple.com/releasenotes/Cocoa/CoreDataReleaseNotes/index.ht
    ml


    > All is nice except the SQLiteStore file does not get reduced in
    > size. I guess that somehow I am doing something wrong and the
    > objects do not get removed form the object graph at the next save:
    > operation.
    >
    The sqlite database file format won't grow and shrink in response to
    every change you make, for performance reasons data is mapped to pages
    which may contain many individual data records (or a single data
    record can span many pages).

    > Is there anything obviously wrong or missing from my way of handling
    > this? I could post some code if required.
    >
    > Thanks,
    > ben
  • On Nov 1, 2007, at 11:40 PM, Adam Swift wrote:

    >
    > On Nov 1, 2007, at 2:04 AM, Benjámin Salánki wrote:
    >
    >> Hi all,
    >>
    >> I am pretty lost in the development of my app which uses Core Data
    >> and an SQLiteStore. I hope someone can point me towards the obvious
    >> point I am missing.
    >>
    >> Given are two entities with bidirectional 1-to-1 relationships set
    >> to cascade upon deletion. The first entity contains strings and the
    >> other one contains binary data (like images and stuff). I set up
    >> two NSArrayControllers in IB and made the second one contain a set
    >> of the selected item from the first and also set the Deletes
    >> objects on remove checkbox in the Inspector.
    >>
    >> My problem is that if I send the remove: or removeObject: message
    >> to the first NSArrayController and then query said item if it is
    >> isDeleted, I get true for the item itself, but false for the item
    >> connected through the relationship. I then manually removed the
    >> second object by sending a removeObject: message to it. Calling
    >> isDeleted now returns true.
    >>
    > The cascade delete rule isn't processed until you save the managed
    > object context in which you made the change, you can force the
    > delete rule processing by calling processPendingChanges.  I believe
    > this is mentioned in the release notes here:  http://developer.apple.com/releasenotes/Cocoa/CoreDataReleaseNotes/index.ht
    ml


    I know, already tried that too.

    >
    >
    >> All is nice except the SQLiteStore file does not get reduced in
    >> size. I guess that somehow I am doing something wrong and the
    >> objects do not get removed form the object graph at the next save:
    >> operation.
    >>
    > The sqlite database file format won't grow and shrink in response to
    > every change you make, for performance reasons data is mapped to
    > pages which may contain many individual data records (or a single
    > data record can span many pages).

    So are you saying removing an item from the store won't reduce its
    size on the disk? How can I tell my users that even if they do remove
    all their data they won't regain any of their precious hard disk
    space? Surely there has to be a way to reduce the size of the file if
    I remove data from it. PLease tell me it's possible.

    >
    >
    >> Is there anything obviously wrong or missing from my way of
    >> handling this? I could post some code if required.
    >>
    >> Thanks,
    >> ben
    >
  • On Nov 1, 2007, at 3:53 PM, Benjámin Salánki wrote:

    >> The sqlite database file format won't grow and shrink in response
    >> to every change you make, for performance reasons data is mapped to
    >> pages which may contain many individual data records (or a single
    >> data record can span many pages).
    >
    > So are you saying removing an item from the store won't reduce its
    > size on the disk? How can I tell my users that even if they do
    > remove all their data they won't regain any of their precious hard
    > disk space? Surely there has to be a way to reduce the size of the
    > file if I remove data from it. PLease tell me it's possible.

    You asked about removing *an* item.  If enough items are removed to
    free up a page in the database file, SQLite's automatic database
    vacuuming will reduce the size of the file as it rearranges the data
    to remove that page.  Similarly, the file size may be reduced if you
    remove an item that itself occupies multiple pages (such as a
    thumbnail image).

    An SQLite file is arranged as a collection of pages and the data
    within those pages is managed via B-trees, not as simple fixed-length
    records (which is what it sounds like you're expecting).  This is much
    more efficient for searching and for overall storage, since it allows
    SQLite to optimize how it stores both data and indexes in a single
    file, and is also the foundation of its data integrity (transaction &
    journaling) mechanism.

    However, the cost of this is that some delete operations may leave
    holes in the file.  If you delete some data and add other data, the
    holes left by the deleted data may be filled by the added data, or the
    file may be vacuumed to compact its data, whichever SQLite considers
    most appropriate based on the operations you're performing.

    Does that make sense?

      -- Chris
  • On Nov 2, 2007, at 3:04 AM, Chris Hanson wrote:

    > On Nov 1, 2007, at 3:53 PM, Benjámin Salánki wrote:
    >
    >>> The sqlite database file format won't grow and shrink in response
    >>> to every change you make, for performance reasons data is mapped
    >>> to pages which may contain many individual data records (or a
    >>> single data record can span many pages).
    >>
    >> So are you saying removing an item from the store won't reduce its
    >> size on the disk? How can I tell my users that even if they do
    >> remove all their data they won't regain any of their precious hard
    >> disk space? Surely there has to be a way to reduce the size of the
    >> file if I remove data from it. PLease tell me it's possible.
    >
    > You asked about removing *an* item.  If enough items are removed to
    > free up a page in the database file, SQLite's automatic database
    > vacuuming will reduce the size of the file as it rearranges the data
    > to remove that page.  Similarly, the file size may be reduced if you
    > remove an item that itself occupies multiple pages (such as a
    > thumbnail image).

    My problem is, that if I include a huge amount of data like an
    uncompressed image that will swell the SQLite file to 72MB, removing
    the item does not reduce the file size. I found that if I add another
    item later that should be around the same size the file size does not
    change so what you wrote makes sense in that the "holes" inside the
    file are filled up with new data but I can't get the file size to
    reduce even by a few bytes.

    I was thinking of maybe implementing a "consolidate database" function
    that would load the data from the file to memory, remove the file and
    recreate the items which would reduce the file size if I am correct.
    Or does anyone have a better solution?

    Ben

    >
    >
    > An SQLite file is arranged as a collection of pages and the data
    > within those pages is managed via B-trees, not as simple fixed-
    > length records (which is what it sounds like you're expecting).
    > This is much more efficient for searching and for overall storage,
    > since it allows SQLite to optimize how it stores both data and
    > indexes in a single file, and is also the foundation of its data
    > integrity (transaction & journaling) mechanism.
    >
    > However, the cost of this is that some delete operations may leave
    > holes in the file.  If you delete some data and add other data, the
    > holes left by the deleted data may be filled by the added data, or
    > the file may be vacuumed to compact its data, whichever SQLite
    > considers most appropriate based on the operations you're performing.
    >
    > Does that make sense?
    >
    > -- Chris
    >
  • > My problem is, that if I include a huge amount of data like an
    > uncompressed image that will swell the SQLite file to 72MB,

    Generally, large media files should be stored externally to the
    database, and a URI and metadata for it stored in a database record.

    > removing the item does not reduce the file size. I found that if I
    > add another
    > item later that should be around the same size the file size does not
    > change so what you wrote makes sense in that the "holes" inside the
    > file are filled up with new data but I can't get the file size to
    > reduce even by a few bytes.

    What you're seeing is internal fragmentation.

    > I was thinking of maybe implementing a "consolidate database" function
    > that would load the data from the file to memory, remove the file and
    > recreate the items which would reduce the file size if I am correct.

    That's very expensive.

    > Or does anyone have a better solution?

    Yes.  You can manually vacuum an SQLite database with:

    sqlite3 filename "vacuum"

    Programmatically, you can use an NSTask.

    - Ben
  • On Nov 3, 2007, at 11:58 PM, Ben Trumbull wrote:

    >> My problem is, that if I include a huge amount of data like an
    >> uncompressed image that will swell the SQLite file to 72MB,
    >
    > Generally, large media files should be stored externally to the
    > database, and a URI and metadata for it stored in a database record.

    I know, I was just testing to see if larger amounts of data would make
    any difference.

    >
    >
    >> removing the item does not reduce the file size. I found that if I
    >> add another
    >> item later that should be around the same size the file size does not
    >> change so what you wrote makes sense in that the "holes" inside the
    >> file are filled up with new data but I can't get the file size to
    >> reduce even by a few bytes.
    >
    > What you're seeing is internal fragmentation.
    >
    >> I was thinking of maybe implementing a "consolidate database"
    >> function
    >> that would load the data from the file to memory, remove the file and
    >> recreate the items which would reduce the file size if I am correct.
    >
    > That's very expensive.
    >
    >> Or does anyone have a better solution?
    >
    > Yes.  You can manually vacuum an SQLite database with:
    >
    > sqlite3 filename "vacuum"
    >

    Very nice, this should do the trick. Thanks for the help!

    ben

    > Programmatically, you can use an NSTask.
    >
    > - Ben
    >
  • On Nov 4, 2007, at 8:37 AM, Benjámin Salánki wrote:
    > On Nov 3, 2007, at 11:58 PM, Ben Trumbull wrote:
    >>> My problem is, that if I include a huge amount of data like an
    >>> uncompressed image that will swell the SQLite file to 72MB,
    >>
    >> Generally, large media files should be stored externally to the
    >> database, and a URI and metadata for it stored in a database record.
    >
    > I know, I was just testing to see if larger amounts of data would
    > make any difference.

    Not really -- you were testing whether or not a SQLite based backing
    store is appropriate for storing large BLOBs of data;  it is not.

    Which does nothing to answer the question as to whether or not a
    SQLite based backing store is appropriate for storing large quantities
    of small data values (typical non-binary fields of data);  which it
    very much is....

    b.bum
previous month november 2007 next month
MTWTFSS
      1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30    
Go to today