Multi-User using Core Data?
-
I am looking for examples/ideas/tips for how to make a Core Data
application multi-user. The idea would be for a common SQLite
database to be accessed by the application from several different
computers. Is this even possible with Core Data today? Searches of
the documentation and Goggling have not produced anything helpful
yet. Thanks for any pointers.
******************
J. Scott Anderson -
On Jun 9, 2005, at 8:59 AM, J. Scott Anderson wrote:
> I am looking for examples/ideas/tips for how to make a Core DataIn what sense "multi-user"?
> application multi-user. The idea would be for a common SQLite
> database to be accessed by the application from several different
> computers. Is this even possible with Core Data today?
>
If you use the SQLite store, then you can benefit from non-atomic
writes -- that is, a save does not overwrite the whole file -- which
in principle allows multiple users to edit the same store
simultaneously. Core Data uses optimistic locking that can prevent
one user overwriting modifications made by another user, but there is
no "distributed notification" that a value has been changed "under" a
user (you find out at save: time -- see also NSManagedObjectContext's
detectConflictsForObject:). You (obviously) also do not get database
login, row- or table-level locking, or any other related features you
typically associate with a client-server database.
mmalc -
Apple obviously made a conscious decision not to support remote
databases with Core Data - they already had an architecture in place
to do it (EOF), so it clearly could have been done had they really
wanted to. I have no reason if their reason not to include that
support was technical, political, or what, but that being said, there
are a couple of options give use a single data store for applications
on separate machines.
1) You could make the SQLite data file location a preference value,
and set it to a network drive location.
2) You could write a small, headless server application that would
vend the data to you client applications using Distributed Objects.
3) Create a subclass of NSPersistentStoreCoordinator that uses a
remote database. This would be no mean feat, but you would likely
make many friends by doing it =)
On Jun 9, 2005, at 11:59 AM, J. Scott Anderson wrote:
> I am looking for examples/ideas/tips for how to make a Core Data
> application multi-user. The idea would be for a common SQLite
> database to be accessed by the application from several different
> computers. Is this even possible with Core Data today? Searches of
> the documentation and Goggling have not produced anything helpful
> yet. Thanks for any pointers.
-
> 2) You could write a small, headless server application that would
> vend the data to you client applications using Distributed Objects.
I just asked a similar question... and this is the approach I've been
thinking about. I wonder what object(s) should be vended. The
PersistentStoreCoordinator, the ManagedObjectContext, or the
ManagedObjects themselves.
If I send an object to a client when edits it, do those changes get sent
properly to the server? What about undo, is there only one undo manager
per ManagedObjectContext?
> Core Data uses optimistic locking that can prevent
> one user overwriting modifications made by another user, but there is
> no "distributed notification" that a value has been changed "under" a
> user (you find out at save: time -- see also NSManagedObjectContext's
> detectConflictsForObject:). You (obviously) also do not get database
> login, row- or table-level locking, or any other related features you
> typically associate with a client-server database.
Is there a way to implement this notification and locking?
-Justin -
On 9 Jun 2005, at 21:50, Jeff LaMarche wrote:
> Apple obviously made a conscious decision not to support remote
> databases with Core Data - they already had an architecture in
> place to do it (EOF), so it clearly could have been done had they
> really wanted to. I have no reason if their reason not to include
> that support was technical, political, or what, but that being
> said, there are a couple of options give use a single data store
> for applications on separate machines.
I think that the main reason for not doing this is that CoreData is
essentially free but you have to pay for the Enterprise Objects
framework, so they felt it was necessary to "cripple" the freeware to
a specification that is useful for desktop applications but left the
corporates with deep pockets needing to buy EOF. This seems a
perfectly reasonable business decision, even if it's a little
annoying for those of us who want to run enterprise grade
applications but don't have the cash!
> 1) You could make the SQLite data file location a preference value,
> and set it to a network drive location.
This is fine until you have two applications accessing the same data
at the same time at which point the memory cache and the file will
get woefully out of sync and "bad things" will happen.
> 2) You could write a small, headless server application that would
> vend the data to you client applications using Distributed Objects.
This has many merits, though I've no idea how well things like KVO
work on distributed objects.
> 3) Create a subclass of NSPersistentStoreCoordinator that uses a
> remote database. This would be no mean feat, but you would likely
> make many friends by doing it =)
Actually, as I noted before (http://www.cocoabuilder.com/archive/
message/2005/5/18/136233) what you need to do is build a few new
subclasses of the NSSQLAdapter, NSSQLStatement and related classes.
It's clear from examining the guts of CoreData that it is perfectly
capable of working on other types of database back end. You would
need to arrange a suitable set of triggers and monitors on any tables
that are having their content cached but this is standard practice in
multiple-access database programming. I now more or less understand
how to build this but I don't really have the time or the database
experience to do this properly. If someone out in CocoaLand with
some DB experience and lots of free time wants to drop me a line I'd
be happy to tell them what I know.
Nicko -
On Jun 10, 2005, at 1:55 AM, Nicko van Someren wrote:
>> Apple obviously made a conscious decision not to support remote
>> databases with Core Data - they already had an architecture in
>> place to do it (EOF), so it clearly could have been done had they
>> really wanted to. I have no reason if their reason not to include
>> that support was technical, political, or what, but that being
>> said, there are a couple of options give use a single data store
>> for applications on separate machines.
> I think that the main reason for not doing this is that CoreData is
> essentially free but you have to pay for the Enterprise Objects
> framework, so they felt it was necessary to "cripple" the freeware
> to a specification that is useful for desktop applications but left
> the corporates with deep pockets needing to buy EOF.
Core Data is not a "crippled" EOF. Its goal, and how it contrasts
with EOF, is made clear in the FAQ:
<http://developer.apple.com/documentation/Cocoa/Conceptual/
CoreData/Articles/cdFAQ.html#//apple_ref/doc/uid/TP40001802-244739>
Also note: WebObjects is bundled for free with the Developer Tools...
>> 1) You could make the SQLite data file location a preferenceThis ("bad things" will happen) is not the case. Just like EOF, Core
>> value, and set it to a network drive location.
> This is fine until you have two applications accessing the same
> data at the same time at which point the memory cache and the file
> will get woefully out of sync and "bad things" will happen.
>
Data is designed to properly detect and deal with situations in which
the persistent store is modified by another application, and there
are well-specified patterns to follow if this occurs.
mmalc -
On 10 Jun 2005, at 10:23, mmalcolm crawford wrote:
>> CoreData/Articles/cdFAQ.html#//apple_ref/doc/uid/TP40001802-244739>
> On Jun 10, 2005, at 1:55 AM, Nicko van Someren wrote:
>
>>> Apple obviously made a conscious decision not to support remote
>>> databases with Core Data - they already had an architecture in
>>> place to do it (EOF), so it clearly could have been done had they
>>> really wanted to. I have no reason if their reason not to
>>> include that support was technical, political, or what, but that
>>> being said, there are a couple of options give use a single data
>>> store for applications on separate machines.
>>>
>> I think that the main reason for not doing this is that CoreData
>> is essentially free but you have to pay for the Enterprise Objects
>> framework, so they felt it was necessary to "cripple" the freeware
>> to a specification that is useful for desktop applications but
>> left the corporates with deep pockets needing to buy EOF.
>
> Core Data is not a "crippled" EOF. Its goal, and how it contrasts
> with EOF, is made clear in the FAQ:
> <http://developer.apple.com/documentation/Cocoa/Conceptual/
I appreciate that one is not a crippled copy of the other. What I
was meaning is that the SQL database functionality of one is a subset
of the other, e.g. a crippled specification. CoreData adds a bunch
of extra features too; fetched properties, local stores and store
aggregation being key, but the remote, shared database server support
is missing.
> Also note: WebObjects is bundled for free with the Developer Tools...
Really? WO is available as a download for paid-up commercial
developers through ADC Select and Premier membership but I don't see
any sign of it in the free developer tools.
>>> 1) You could make the SQLite data file location a preference
>>> value, and set it to a network drive location.
>>>
>> This is fine until you have two applications accessing the same
>> data at the same time at which point the memory cache and the file
>> will get woefully out of sync and "bad things" will happen.
> This ("bad things" will happen) is not the case. Just like EOF,
> Core Data is designed to properly detect and deal with situations
> in which the persistent store is modified by another application,
> and there are well-specified patterns to follow if this occurs.
Really? On the page of the link you gave above, under the heading
Change Management it says:
"There is an important behavioral difference between EOF and
Core Data with respect to change propagation. In Core Data, peer
managed object contexts are not "kept in sync" in the same was as are
editing contexts in EOF. Given two managed object contexts connected
to the same persistent store coordinator, and with the "same" managed
object in both contexts, if you modify one of the managed objects
then save, the other is not re-faulted (changes are not propagated
from one context to another). If you modify then save the other
managed object, then (at least if you use the default merge policy)
you will get an optimistic locking failure."
Clearly then there is no change propagation even if we are talking
through the same Persistent Store Coordinator, let alone if we are
accessing the same file from two different applications on two
different machines over a network. If we were using two application
on the same machine perhaps we could watch the database files (using
something like Uli's UKKQueue http://www.zathras.de/angelweb/
sourcecode.htm#UKKQueue ) and call refreshObject:mergeChanges: on
anything we care about, but as far as I know file change notification
does not work reliable over network mounts.
The section in the Core Data Programming Guide on Change Management
<http://developer.apple.com/documentation/Cocoa/Conceptual/CoreData/
Articles/cdChangeManagement.html> makes no mention whatsoever
regarding multiple applications accessing the same source (on the
same machine or different ones), though it makes repeated remarks
about the same application accessing the same data through multiple
Managed Object Contexts (which all talk to the same Persistent Store
Coordinator).
It would be very useful if Apple could make a clear statement (a
Technical Note perhaps) about how to have multiple applications
accessing the same data store through Core Data, and in particular
how to do this with the applications on different machines. If it is
genuinely the case that Apple did not deliberately dumb down the
specification of Core Data then it would be useful to know how to
achieve the same effect as WO/EOF through Core Data.
Nicko -
On Jun 10, 2005, at 7:24 AM, Nicko van Someren wrote:
>> Also note: WebObjects is bundled for free with the Developer Tools...
>>
>
> Really? WO is available as a download for paid-up commercial
> developers through ADC Select and Premier membership but I don't
> see any sign of it in the free developer tools.
>
It's part of the XCode Tools download (I just noticed it as of
2.1, but it may have been earlier??). -
On Jun 10, 2005, at 4:24 AM, Nicko van Someren wrote:
> On 10 Jun 2005, at 10:23, mmalcolm crawford wrote:
>> On Jun 10, 2005, at 1:55 AM, Nicko van Someren wrote:
> I appreciate that one is not a crippled copy of the other.
It's not clear, then, why you wrote that one is a crippled version of
the other.
>> Also note: WebObjects is bundled for free with the Developer Tools...Umm, yes, really. WebObject 5.3 was announced at WWDC. It is
> Really? WO is available as a download for paid-up commercial
> developers through ADC Select and Premier membership but I don't
> see any sign of it in the free developer tools.
>
bundled free with the developer tools. The deployment license is
bundled with Mac OS X Server.
>>> This is fine until you have two applications accessing the same
>>> data at the same time at which point the memory cache and the
>>> file will get woefully out of sync and "bad things" will happen.
>> This ("bad things" will happen) is not the case. Just like EOF,
>> Core Data is designed to properly detect and deal with situations
>> in which the persistent store is modified by another application,
>> and there are well-specified patterns to follow if this occurs.
> Really?
Again, yes, really...
> On the page of the link you gave above, under the heading ChangeThis represents a subtle but important change in the specified
> Management it says:
> "There is an important behavioral difference between EOF and
> Core Data with respect to change propagation. In Core Data, peer
> managed object contexts are not "kept in sync" in the same was as
> are editing contexts in EOF. Given two managed object contexts
> connected to the same persistent store coordinator, and with the
> "same" managed object in both contexts, if you modify one of the
> managed objects then save, the other is not re-faulted (changes are
> not propagated from one context to another). If you modify then
> save the other managed object, then (at least if you use the
> default merge policy) you will get an optimistic locking failure."
>
pattern of behavior -- it does not mean that something "bad" is
happening. You're confusing change propagation *within the stack*
with change propagation between the database and your application...
> Clearly then there is no change propagation even if we are talkingThere is no change propagation in EOF either, *from the database to
> through the same Persistent Store Coordinator, let alone if we are
> accessing the same file from two different applications on two
> different machines over a network.
>
the access layer*.
It's not clear to me that you understand the way EOF deals with
optimistic locking failures. Perhaps you could explain what it is
you perceive to be the difference between the way EOF and Core Data
deal with an external change to the persistent store?
mmalc -
On 10 Jun 2005, at 14:58, mmalcolm crawford wrote:
> On Jun 10, 2005, at 4:24 AM, Nicko van Someren wrote:
>> On 10 Jun 2005, at 10:23, mmalcolm crawford wrote:
>>> On Jun 10, 2005, at 1:55 AM, Nicko van Someren wrote:
>> I appreciate that one is not a crippled copy of the other.
>
> It's not clear, then, why you wrote that one is a crippled version
> of the other.
I didn't. In the same email that points out that the internals of
CoreData are neatly structures to support other sorts of SQL database
I wrote that "... felt it was necessary to "cripple" the freeware
[CoreData] to a specification that is useful for desktop
applications ...". I concede that this might be misinterpreted and
perhaps I should have phrased it better but all I was saying was that
CoreData does not deliver as much functionality as it is clearly
designed to be able to deliver, e.g. it is crippled, which is at odds
with EOF. Even though the underlying structure supports addition of
other sorts of SQL database Apple have kept the API at that level
closed and indeed have taken the trouble of setting classes such as
NSSQLAdaptor as non-exported in the shared library so that it is
difficult to subclass them.
>>> Also note: WebObjects is bundled for free with the Developer
>>> Tools...
>>>
>> Really? WO is available as a download for paid-up commercial
>> developers through ADC Select and Premier membership but I don't
>> see any sign of it in the free developer tools.
> Umm, yes, really. WebObject 5.3 was announced at WWDC. It is
> bundled free with the developer tools. The deployment license is
> bundled with Mac OS X Server.
OK, I wasn't at WWDC. What confused me was the way that http://
developer.apple.com/webobjects/ (still) says:
"WebObjects and the Apple Developer Connection
Order WebObjects 5.2 through the ADC Member Site and save over 25%.
Premier members receive one (1) copy free, and may purchase
additional copies for $399 US. Select members may purchase WebObjects
5.2 for $399 US. Please allow 1 to 2 weeks for delivery, depending on
location."
It's nice to know that it's now freely available. Thanks Apple.
...
>> On the page of the link you gave above, under the heading Change
>> Management it says:
>> "There is an important behavioral difference between EOF and
>> Core Data with respect to change propagation. In Core Data, peer
>> managed object contexts are not "kept in sync" in the same was as
>> are editing contexts in EOF. Given two managed object contexts
>> connected to the same persistent store coordinator, and with the
>> "same" managed object in both contexts, if you modify one of the
>> managed objects then save, the other is not re-faulted (changes
>> are not propagated from one context to another). If you modify
>> then save the other managed object, then (at least if you use the
>> default merge policy) you will get an optimistic locking failure."
>>
> This represents a subtle but important change in the specified
> pattern of behavior -- it does not mean that something "bad" is
> happening. You're confusing change propagation *within the stack*
> with change propagation between the database and your application...
OK, perhaps I'm missing something but I was under the impression that
the change propagation in EOF worked between instances of
applications, via the database.
>> Clearly then there is no change propagation even if we are talking
>> through the same Persistent Store Coordinator, let alone if we are
>> accessing the same file from two different applications on two
>> different machines over a network.
>>
> There is no change propagation in EOF either, *from the database to
> the access layer*.
Aha. That was my misapprehension.
> It's not clear to me that you understand the way EOF deals with
> optimistic locking failures. Perhaps you could explain what it is
> you perceive to be the difference between the way EOF and Core Data
> deal with an external change to the persistent store?
Well, my perception was that EOF allowed for triggers on external
changes, which you tell me is not the case, so I guess the point is
moot.
Returning to the original topic of this thread, since you seem to
know something about how this works, can you explain what CoreData
does about implementing it's locking on the SQLite files with a view
to understanding network sharing? SQLite has Shared->Reserved-
> Pending-> Exclusive locks and on Unix like machines these aretypically implemented using Posix advisory locks. Advisory locks
have been buggy on NFS for years; if AFP reliably implements
networked locking then things _should_ work if you simply point
multiple programs on multiple machines at a shared copy of the file.
That said, the authors of SQLite explicitly advise against using it
in this way; see the "Situations where another RDBMS may work better"
on http://www.sqlite.org/cvstrac/wiki?p=WhenToUseSqlite where it says
"A good rule of thumb is that you should avoid using SQLite in
situations where the same database will be accessed simultaneously
from many computers over a network filesystem." So we are still left
without a good way to have multi-user access to Core Data files :-
( Perhaps one day Apple will open up the SQL abstraction layer that
lurks within Core Data and either give us the RDBMS adaptors we need
or allow us to write our own...
Nicko -
On Jun 10, 2005, at 11:17 AM, Nicko van Someren wrote:
> I didn't. In the same email that points out that the internals of"Cripple" implies that there is functionality that is present and
> CoreData are neatly structures to support other sorts of SQL
> database I wrote that "... felt it was necessary to "cripple" the
> freeware [CoreData] to a specification that is useful for desktop
> applications ...". I concede that this might be misinterpreted and
> perhaps I should have phrased it better but all I was saying was
> that CoreData does not deliver as much functionality as it is
> clearly designed to be able to deliver, e.g. it is crippled, which
> is at odds with EOF.
>
complete, but which is deliberately turned off. This is simply not
the case.
> Even though the underlying structure supports addition of otherThere are many reasons why this may be the case, not least because
> sorts of SQL database Apple have kept the API at that level closed
> and indeed have taken the trouble of setting classes such as
> NSSQLAdaptor as non-exported in the shared library so that it is
> difficult to subclass them.
>
the implementation may not be considered sufficiently well abstracted
to allow subclassing, or because the API is not finalised. There are
many examples of functionality being restricted to private frameworks
for precisely these reasons...
I suspect that you misunderstand or do not fully appreciate what is
involved here. In many respects, the current Java implementation of
EOF is made much easier because of JDBC. EOF is able to make use of
a single JDBC adaptor, and database developers provide custom drivers
below that to support communication with their product. It may be
instructive to look at the documentation for WebObjects 4.5: <http://
developer.apple.com/documentation/LegacyTechnologies/WebObjects/
WebObjects_4.5/webobjects.html>
In particular note:
InformixEOAdaptor
ODBCEOAdaptor
OracleEOAdaptor
SybaseEOAdaptor
In principle, each database requires its own adaptor, client
libraries etc. In practice, it may be sufficient to implement an
ODBC adaptor, but that still requires third party support...
In summary, however, it's untrue to claim that Core Data is
"crippled" -- there is a considerable amount of extra work that would
be required to support additional persistent store types...
> Returning to the original topic of this thread, since you seem toBill would be better able to comment on the file-level locking issues
> know something about how this works, can you explain what CoreData
> does about implementing it's locking on the SQLite files with a
> view to understanding network sharing? SQLite has Shared->Reserved-
>> Pending->Exclusive locks and on Unix like machines these are
> typically implemented using Posix advisory locks. Advisory locks
> have been buggy on NFS for years; if AFP reliably implements
> networked locking then things _should_ work if you simply point
> multiple programs on multiple machines at a shared copy of the file.
>
here. In general, however, Core Data (as EOF) uses optimistic
locking to ensure that two users do not overwrite the same data.
This does not rely on file (or row) locking...
> Perhaps one day Apple will open up the SQL abstraction layer thatPlease submit enhancement requests.
> lurks within Core Data and either give us the RDBMS adaptors we
> need or allow us to write our own...
>
When doing so, it would be very useful if you could indicate, for
example, how critical such functionality is to your product, how many
users you believe you could bring to the platform if such
functionality were provided, and the relative importance of such
functionality -- e.g. given a choice between support for ordered
relationships and database adaptors, which *one* would you choose?
Note: The preceding example is purely an *example*. This should not
in any way be taken to suggest that this choice would be one that
would be made in practice, nor should it be taken to suggest that
Apple might provide either or both features in the future. It is
intended to highlight the fact that there are clearly resource
constraints, and that in any given release cycle there is a limit to
the amount of new functionality that can be implemented. I'm sure
the team would love to implement all the features that everyone is
asking for, but this is simply not possible. Feedback from the
developer community is very useful in determining priorities...
mmalc
To emphasise: Speaking only for himself. -
On Jun 11, 2005, at 7:27 PM, mmalcolm crawford wrote:
>> Returning to the original topic of this thread, since you seem to
>> know something about how this works, can you explain what CoreData
>> does about implementing it's locking on the SQLite files with a
>> view to understanding network sharing? SQLite has Shared-
>>> Reserved->Pending->Exclusive locks and on Unix like machines
>> these are typically implemented using Posix advisory locks.
>> Advisory locks have been buggy on NFS for years; if AFP reliably
>> implements networked locking then things _should_ work if you
>> simply point multiple programs on multiple machines at a shared
>> copy of the file.
> Bill would be better able to comment on the file-level locking
> issues here. In general, however, Core Data (as EOF) uses
> optimistic locking to ensure that two users do not overwrite the
> same data. This does not rely on file (or row) locking...
Sure. Off the cuff answer below. First, from a pure SQLite
perspective:
Of note, all locking in SQLite is performed on the entire database.
On filesystems that support POSIX advisory locks, SQLite uses the
same exact locking mechanisms as it would if you were to build the
stock SQLite.
For AFP, SQLite uses byte range locks to simulate a locking behavior
that is as close to advisory locks as you can get without having true
advisory locks. The only difference is that you might see a
situation where what should be a shared lock appears as an exclusive
to another process or connection. Given that your code should be
dealing with this situation anyway -- what if some random process
should grab the database and do something to tables you have no
control over? -- this should not pose a huge problem.
For NFS, SQLite will test to make sure that advisory locks really
work. If they do, they are used. If not, .lock files are used.
For SMB (Samba), SQLite will use flock() style locks. Since there
is no way to upgrade a shared flock() lock to an exclusive flock()
lock without offering a window of time within which another app can
effectively 'steal' the lock, flock() based locks will only support
exclusive locks.
Note that NFS does not fall back to flock() if advisory locks do not
work. Testing revealed that this was the safe way to go.
Note that mixing file systems when accessing a single SQLite database
file is a really, really bad idea. The alternative was to basically
go with .lock files on all filesystems.
All SQLite unit tests passed except those that should fail when
shared locks are not available.
The API for the additional locking support is at the end of
sqlite3.h. The underlying changes will be posted back to the SQLite
development community as time permits. Now that WWDC is over, time
is much more permissive...
---
What does this mean for Core Data?
Core Data does not keep SQLite connections open for any longer than
is necessary to perform a particular operation. While any given
transaction may be relatively involved-- may use a number of SQL
statements in its internal implementation -- Core Data will have
closed all transactions before returning control to your code (or the
main-event-loop, in the case of pure bindings based apps).
In other words, if you are doing a complex, large scale, fetch in one
process using Core Data, it will block other processes and will
sometimes do so more than others, depending on filesystem. However,
this situation should occur fairly rarely.
b.bum
