Versioning and CoreData
-
Hi,
I just created a CoreData project and a test document, then added
some fields to the model, and then opened the document again. This
worked, which goes completely against what the CoreData docs on
versioning say. Is this a quirk of the XML store, or did I overlook
something in the docs and adding optional attributes to a store
doesn't cause versioning problems? This would be very handy because
it's the most common update to a file format.
If it isn't, does anyone have any suggestion on the most painless
way of migrating between model formats? Basically I'd only want to
add stuff and write code once that migrates the old format to the
newer, richer format, and then just add to my model as I please.
Cheers,
-- M. Uli Kusterer
http://www.zathras.de -
I'm assuming by "model formats" you mean model versions and not
persistent store formats?
I've done this a few times now, so the last time I did it I embedded
the code into a reusable framework that I expand on as needed. It's
still pretty early in its development, but so far it has accomplished
the job fairly well.
The basic approach used by the process is this:
Check the store version (stored in the metadata) prior to the
NSApplicationMain() function being called. If it's an older version,
then instantiate a data migrator object. This is initialized to the
project's needs, such as in the following snippet.
//Build the migrator
MSDataMigrator *migrator = [MSDataMigrator dataMigrator];
[migrator setTarget:nil];
[migrator setCopiesOldAttributes:YES];
[migrator setStoreType:NSSQLiteStoreType];
[migrator setStoreLocation:persistentStorePath()];
[migrator setCurrentVersion:STORE_VERSION];
NSString *resourcePath = [[NSBundle mainBundle] resourcePath];
[migrator addModelAtPath:[resourcePath
stringByAppendingPathComponent:@"Version0_Model.mom"] forVersion:
[NSNumber numberWithInt:0]];
[migrator addModelAtPath:[resourcePath
stringByAppendingPathComponent:@"Version3_Model.mom"] forVersion:
[NSNumber numberWithInt:3]];
[migrator addPreviousStoreAtPath:[applicationSupportFolder()
stringByAppendingPathComponent:@"APreviousStore.sqlite"] forVersion:
[NSNumber numberWithInt:0]];
if (![migrator migrate])
{
if ([migrator error] != MSDataMigratorNoError)
{
NSLog(@"Migrator encountered error: %d", [migrator error]);
}
}
When the migrator runs, it steps through each entity type. A new
object for each object of this entity type is created in the new
context. If it's set to copy old attributes, it does so. Every
attribute that exists in the same form in both the old and new models
is copied directly.
While creating the new objects, it also builds an (old object id) ->
(new object id) mapping. A second stage rebuilds relationships using
this mapping.
The final stage iterates over a series of callbacks, which are set
prior to migration using this method:
- (void)setCallback:(SEL)aSelector forEntityName:(NSString *)
anEntityName withVersionChange:(MSVersionChange *)aVersionChange
For each object of the specified entity, the selector will be called
if the version change matches. The selector takes an old and new
object and returns a BOOL indicating success or failure.
This currently has a few limitations. Namely, it does not support an
entity type disappearing. If that happens, it will ignore the old
objects of that entity type and will not attempt to migrate them. If
you're at all interested, I'm more than willing to share the code
though it is still a work in progress and not necessarily ready for
every scenario.
On May 1, 2006, at 12:24 PM, Uli Kusterer wrote:> Hi,
>
> I just created a CoreData project and a test document, then added
> some fields to the model, and then opened the document again. This
> worked, which goes completely against what the CoreData docs on
> versioning say. Is this a quirk of the XML store, or did I overlook
> something in the docs and adding optional attributes to a store
> doesn't cause versioning problems? This would be very handy because
> it's the most common update to a file format.
>
> If it isn't, does anyone have any suggestion on the most painless
> way of migrating between model formats? Basically I'd only want to
> add stuff and write code once that migrates the old format to the
> newer, richer format, and then just add to my model as I please.
>
> Cheers,
> -- M. Uli Kusterer
> http://www.zathras.de
>
>
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Cocoa-dev mailing list (<Cocoa-dev...>)
> Help/Unsubscribe/Update your Subscription:
> http://lists.apple.com/mailman/options/cocoa-dev/<enigma0...>
>
> This email sent to <enigma0...> -
Uli,
On 1.5.2006, at 21:24, Uli Kusterer wrote:> I just created a CoreData project and a test document, then added
> some fields to the model, and then opened the document again. This
> worked, which goes completely against what the CoreData docs on
> versioning say. Is this a quirk of the XML store, or did I overlook
> something in the docs and adding optional attributes to a store
> doesn't cause versioning problems? This would be very handy because
> it's the most common update to a file format.
Haven't tested myself (caveat!), but I am comparatively positive this
is something which just happens to work with XML (for that is
essentially a bunch of NSDictionaries and you can ask a dictionary
for a non-existing key, and get a valid nil), but does not with SQL
(for you can't ask for a non-existing column).
---
Ondra ÃÅada
OCSoftware: <ocs...> http://www.ocs.cz
private <ondra...> http://www.ocs.cz/oc -
On Monday, May 01, 2006, at 02:25PM, Uli Kusterer <kusterer...> wrote:> I just created a CoreData project and a test document, then added
> some fields to the model, and then opened the document again. This
> worked, which goes completely against what the CoreData docs on
> versioning say. Is this a quirk of the XML store, or did I overlook
> something in the docs and adding optional attributes to a store
> doesn't cause versioning problems? This would be very handy because
> it's the most common update to a file format.
>
> If it isn't, does anyone have any suggestion on the most painless
> way of migrating between model formats? Basically I'd only want to
> add stuff and write code once that migrates the old format to the
> newer, richer format, and then just add to my model as I please.
Very interesting. Currently I do not use CoreData because it wasn't clear how to handle changes.
I'll run some experiments myself over the weekend; I would like to take advantange of CD, but also need an easy way to change the model.
Now, did you try only with an XML store? Or did things also work with binary and SQL?
--
Rick Sharp
Instant Interactive(tm) -
Am 01.05.2006 um 21:24 Uhr schrieb Uli Kusterer:> I just created a CoreData project and a test document, then added
> some fields to the model, and then opened the document again. This
> worked, which goes completely against what the CoreData docs on
> versioning say. Is this a quirk of the XML store,
Yes. This will not work with an SQLite store.
Andreas -
On May 1, 2006, at 12:24 PM, Uli Kusterer wrote:> I just created a CoreData project and a test document, then added
> some fields to the model, and then opened the document again. This
> worked, which goes completely against what the CoreData docs on
> versioning say.
>
<http://developer.apple.com/documentation/Cocoa/Conceptual/CoreData/
Articles/cdVersioning.html#//apple_ref/doc/uid/TP40002989-
DontLinkElementID_104>
"In some situations, it may be advantageous to use an XML store for
development. If all you do to a model is to add new entities and
attributes, then you can generally use a new model to open an XML
store created using an old model. Given that changing stores is
typically the matter of swapping a constant (although be aware of
some differences between storesâsee âPersistent Storesâ), using the
XML store during development and a different store (binary or SQLite)
for final testing and deployment is quite viable."
mmalc -
Am 01.05.2006 um 21:49 schrieb Ondra Cada:> Haven't tested myself (caveat!), but I am comparatively positive
> this is something which just happens to work with XML (for that is
> essentially a bunch of NSDictionaries and you can ask a dictionary
> for a non-existing key, and get a valid nil), but does not with SQL
> (for you can't ask for a non-existing column).
Thanks, I was afraid so, but hadn't gotten around to trying this out
yet.
Ryan Britton wrote:> I'm assuming by "model formats" you mean model versions and not
> persistent store formats?
Yup, sorry for the fuzziness.> When the migrator runs, it steps through each entity type. A new
> object for each object of this entity type is created in the new
> context. If it's set to copy old attributes, it does so. Every
> attribute that exists in the same form in both the old and new
> models is copied directly.
>
> While creating the new objects, it also builds an (old object id) -
>> (new object id) mapping. A second stage rebuilds relationships
> using this mapping.
Great solution to a horrible problem. Is there no way to have
CoreData do the copying based on the model file and just add the
model's default values for any new items? I had hoped there was an
easier way for the migration.> The final stage iterates over a series of callbacks, which are set
> prior to migration using this method:
> - (void)setCallback:(SEL)aSelector forEntityName:(NSString *)
> anEntityName withVersionChange:(MSVersionChange *)aVersionChange
> For each object of the specified entity, the selector will be
> called if the version change matches. The selector takes an old
> and new object and returns a BOOL indicating success or failure.
>
> This currently has a few limitations. Namely, it does not support
> an entity type disappearing. If that happens, it will ignore the
> old objects of that entity type and will not attempt to migrate
> them. If you're at all interested, I'm more than willing to share
> the code though it is still a work in progress and not necessarily
> ready for every scenario.
I may get back to you on that. I just got swamped in more work, but
once that's done I'd love to give it a try. Maybe I can somehow get
the info from the model file and automate the migration step some
more with the defaults in there.
Cheers,
-- M. Uli Kusterer
http://www.zathras.de -
Am 01.05.2006 um 22:27 schrieb mmalcolm crawford:> On May 1, 2006, at 12:24 PM, Uli Kusterer wrote:> Articles/cdVersioning.html#//apple_ref/doc/uid/TP40002989-
>
>> I just created a CoreData project and a test document, then added
>> some fields to the model, and then opened the document again. This
>> worked, which goes completely against what the CoreData docs on
>> versioning say.
>>
>
> <http://developer.apple.com/documentation/Cocoa/Conceptual/CoreData/> DontLinkElementID_104>
>
> "In some situations, it may be advantageous to use an XML store for
> development. If all you do to a model is to add new entities and
> attributes, then you can generally use a new model to open an XML
> store created using an old model. Given that changing stores is
> typically the matter of swapping a constant (although be aware of
> some differences between storesâsee âPersistent Storesâ), using the
> XML store during development and a different store (binary or
> SQLite) for final testing and deployment is quite viable."
Thank you. It's reassuring to see this is documented somewhere,
even though this still doesn't cover the case of a release 2.0 unless
you rely on an XML store for deployment.
Cheers,
-- M. Uli Kusterer
http://www.zathras.de -
On May 2, 2006, at 3:11 AM, Uli Kusterer wrote:>
> Great solution to a horrible problem. Is there no way to have
> CoreData do the copying based on the model file and just add the
> model's default values for any new items? I had hoped there was an
> easier way for the migration.
Core Data can handle defaults, but it cannot handle any of the
copying. If you have an acceptable default value for an added
attributes, then you do not need to necessarily set these in the
migration because Core Data will handle that part. Existing
attributes must be copied directly though. Fortunately, you can get
a list of these and iterate over them rather than having to hard code
them in.>> This currently has a few limitations. Namely, it does not support
>> an entity type disappearing. If that happens, it will ignore the
>> old objects of that entity type and will not attempt to migrate
>> them. If you're at all interested, I'm more than willing to share
>> the code though it is still a work in progress and not necessarily
>> ready for every scenario.
>
> I may get back to you on that. I just got swamped in more work,
> but once that's done I'd love to give it a try. Maybe I can somehow
> get the info from the model file and automate the migration step
> some more with the defaults in there.
There is one other limitation that I didn't mention here. Store size
plays a factor. This currently does no batching so large store sizes
may not be practical without adding some kind of batching
functionality. My data sets have been small so I've not done
anything like this.
Ryan -
Am 02.05.2006 um 15:11 schrieb Ryan Britton:> Core Data can handle defaults, but it cannot handle any of the
> copying. If you have an acceptable default value for an added
> attributes, then you do not need to necessarily set these in the
> migration because Core Data will handle that part. Existing
> attributes must be copied directly though. Fortunately, you can
> get a list of these and iterate over them rather than having to
> hard code them in.
Great to hear that! Thanks for letting me know of that.> There is one other limitation that I didn't mention here. Store
> size plays a factor. This currently does no batching so large
> store sizes may not be practical without adding some kind of
> batching functionality. My data sets have been small so I've not
> done anything like this.
What do you mean by "batching functionality"? You mean that doing
the conversion to a new model would require loading and writing out
of the entire store, and one would ideally try to load only a few
objects at a time, or what?
Cheers,
-- M. Uli Kusterer
http://www.zathras.de -
On May 6, 2006, at 7:04 AM, Uli Kusterer wrote:>> There is one other limitation that I didn't mention here. Store
>> size plays a factor. This currently does no batching so large
>> store sizes may not be practical without adding some kind of
>> batching functionality. My data sets have been small so I've not
>> done anything like this.
>
> What do you mean by "batching functionality"? You mean that doing
> the conversion to a new model would require loading and writing out
> of the entire store, and one would ideally try to load only a few
> objects at a time, or what?
>
Something like that, yes. If you're working with a store that has
50000 objects or some huge number, it may not be practical to do the
entire conversion with the store in memory. It would need some kind
of support to be able to do all of those objects in small batches so
that the memory could be reclaimed at each iteration rather than
ballooning (one useful trick I've found for this kind of thing is to
have a field in the model called "conversion" that you write the old
object ID into on the new, migrated object -- this lets you keep
track of all of the pairings using the store itself instead of
keeping some huge list). As it stands now though, I have not had a
store that would grow this large so I haven't done anything about it.
Ryan
