Core Data Multiuser

  • Hello, people!

    I'll bring this subject to surface again. I know it pops here and there from time to time, but I've never seen anything conclusive, besides advice to steer off this idea.

    For a long time, I have wanted to develop a "pseudo-CRM" software for my small business. It will be used for simple things, like managing customer and product data, sending quotes and invoices and stuff like that. I need it to be multiuser and I am fed up with web-based software. We use only Macs. Period.

    From what I've read in this and other lists, and also lots of forums, Core Data wasn't meant for multiuser access and I understand it. But...

    I am reading Marcus Zarra's book on Core Data ("Core Data - Apple’s API for Persisting Data on Mac OS X") and I have to say the book is excellent. And it got me thinking about this project again.

    One of the problems in using Core Data with multiple users accessing the same persistent store is concurrency. But I've seen techniques for that, such as using multiple MOCs. Marcus Zarra's book talks about it in chapter 9, about multithreading. I've seen it in Apple's documentation also.

    On chapter 11, the book talks about "distributed Core Data", using Distributed Objects to exchange NSManagedObjects between a client app and a server app. The latter deals with the MOC and the persistent store. Zarra warns about scalability problems, since NSManagedObjectContext is not thread-safe and, the, all clients' data would be dealt with serially... But, what if I used one MOC per client?

    What do you guys think about it? Is it a bad idea? I've studied a lot of alternatives (BaseTen, ODBC, Web Services), but I can't wrap my head around them...

    Cheers,
    Flavio
  • Am 13.07.2012 um 21:38 schrieb Flavio Donadio:

    > What do you guys think about it? Is it a bad idea? I've studied a lot of alternatives (BaseTen, ODBC, Web Services), but I can't wrap my head around them...

    Use WebObjects and EOF (the big mature brother of CoreData) on the server!

    We feed multiple hundred thousand iOS devices daily with it - as does Apple with iTunes and the Store.

    atze
  • On Jul 14, 2012, at 5:18 AM, Alexander Spohr wrote:

    >
    > Am 13.07.2012 um 21:38 schrieb Flavio Donadio:
    >
    >> What do you guys think about it? Is it a bad idea? I've studied a lot of alternatives (BaseTen, ODBC, Web Services), but I can't wrap my head around them...
    >
    > Use WebObjects and EOF (the big mature brother of CoreData) on the server!
    >
    > We feed multiple hundred thousand iOS devices daily with it - as does Apple with iTunes and the Store.

    Um, what?  Hasn't WebObjects effectively been orphaned [as in, Apple may still be using it internally, but they are no longer publicly updating it or providing support for it]?

    And yes, I agree it was kickass technology that I wish Apple would at least sell to somebody else to make a product out of instead just letting it die.

    As for the OP's request, might I suggest using the BaseTen framework.  Its an open-source PostgreSQL-backed database framework that works like CoreData [it even uses the CoreData database layout file to configure PostgreSQL].

    Eli
  • Am 15.07.2012 um 08:12 schrieb Eli Bach2:

    > On Jul 14, 2012, at 5:18 AM, Alexander Spohr wrote:
    >
    >> Am 13.07.2012 um 21:38 schrieb Flavio Donadio:
    >>
    >> Use WebObjects and EOF (the big mature brother of CoreData) on the server!
    >>
    >> We feed multiple hundred thousand iOS devices daily with it - as does Apple with iTunes and the Store.
    >
    > Um, what?  Apple may still be using it internally, but they are no longer publicly updating it or providing support for it?

    Support is here:
    http://www.wocommunity.org/

    We started our current project a year ago. WebObjects still does fine with heavy workloads.

    atze
  • On Jul 13, 2012, at 12:38 PM, Flavio Donadio <flavio...> wrote:

    > On chapter 11, the book talks about "distributed Core Data", using Distributed Objects to exchange NSManagedObjects between a client app and a server app. The latter deals with the MOC and the persistent store. Zarra warns about scalability problems, since NSManagedObjectContext is not thread-safe and, the, all clients' data would be dealt with serially... But, what if I used one MOC per client?

    In my experience — and yes I have tried it — using DO between multiple computers is a nightmare. I know it sounds so simple and appealing, but that's because it tries to sweep all the hard problems of networking[1] under the rug. The problems remain, and will bite you, hard.

    The inconvenient truth is that sending a message over a network is not at all the same as sending an Objective-C message between two objects, no matter what kind of clever wrapping is used to make it look like it. Network messages are orders of magnitude slower, they are unreliable for many different reasons, and they're not trustable unless you are very, very careful about authentication and sandboxing.

    There are basically two realistic ways to do what you want to do:

    (1) Client-server. The database lives on one server machine, as does the "business logic" (I hate that term) that manages your app. This could definitely be implemented with Core Data if you like. The client app just focuses on the UI, and there is some protocol by which it talks to the server to fetch data and to tell it to do stuff. In other words, the client app will not use Core Data.

    (2) Synchronized. Every client has a copy of the database, and the app operates on the local database. But the clients have to stay in sync by sending messages to the server whenever they make changes, and receiving notifications from the server when some other client makes a change. If conflicting changes are made, there has to be a way to resolve them.

    The bad news: Approach #1 is straightforward but means you have to abandon Core Data on the client, and design and implement your own network protocol, which is time-consuming. Approach #2 is lovely when it works but I assure you from experience that synchronization is very, very difficult to implement from scratch.

    I hate to make this post sound like an ad, but I'm developing (for my employer, Couchbase) a framework that implements #2. It's based on CouchDB[2], a very popular nonrelational ("NoSQL") distributed database that is really, really good at synchronization. My framework, TouchDB,[3] lets Mac or iOS or Android apps store databases locally, operate on them locally, and then replicate in real time with a CouchDB server. If there are multiple clients syncing with the same server, it's exactly the solution #2 I outlined above.

    Now, TouchDB isn't compatible with Core Data. But it does have a pretty solid Cocoa API that has an object-model layer a bit like a simplified Core Data. The people who've been using it like it a lot.

    —Jens

    [1]: http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing
    [2]: http://couchdb.apache.org
    [3]: http://touchdb.org
  • On 16/07/2012, at 18:39, Jens Alfke wrote:

    > In my experience — and yes I have tried it — using DO between multiple computers is a nightmare. I know it sounds so simple and appealing, but that's because it tries to sweep all the hard problems of networking[1] under the rug. The problems remain, and will bite you, hard.
    >
    > [...]
    >
    > The bad news: Approach #1 is straightforward but means you have to abandon Core Data on the client, and design and implement your own network protocol, which is time-consuming. Approach #2 is lovely when it works but I assure you from experience that synchronization is very, very difficult to implement from scratch.

    Well... This is one of the more down-to-earth arguments about why I should steer away from this technique. Coming from you, Jens, I can't ignore it.

    > I hate to make this post sound like an ad, but I'm developing (for my employer, Couchbase) a framework that implements #2. It's based on CouchDB[2], a very popular nonrelational ("NoSQL") distributed database that is really, really good at synchronization. My framework, TouchDB,[3] lets Mac or iOS or Android apps store databases locally, operate on them locally, and then replicate in real time with a CouchDB server. If there are multiple clients syncing with the same server, it's exactly the solution #2 I outlined above.

    I've just took a look at the links you sent and I'm downloading TouchDB. I'll have a careful look at it soon.

    > Now, TouchDB isn't compatible with Core Data. But it does have a pretty solid Cocoa API that has an object-model layer a bit like a simplified Core Data. The people who've been using it like it a lot.

    To be honest, I know a little about Cocoa and can write almost any simple app you can imagine. Core Data is not, by any measure, a simple framework. I think I understand it, so it feels comfortable for me to try and stick with it. But it doesn't have to be Core Data. It must be Cocoa, though.

    Cheers,
    Flavio
  • On Jul 16, 2012, at 4:11 PM, Flavio Donadio <flavio...> wrote:

    > To be honest, I know a little about Cocoa and can write almost any simple app you can imagine. Core Data is not, by any measure, a simple framework. I think I understand it, so it feels comfortable for me to try and stick with it. But it doesn't have to be Core Data. It must be Cocoa, though.

    To be honest, I never really got comfortable with Core Data. It always seemed to be very complex to do anything real with it, and there was a lot of boilerplate to set it up. (That was 5+ years ago, though. Maybe it's gotten simpler ;-)

    One of the nice things about CouchDB-like databases is that everything is JSON, which is a kissing cousin of a PList, so you can treat your data like property-lists if you want. Or alternatively, the CouchModel class lets you treat data items as objects with custom properties, which can often make the code cleaner. But the choice is yours, and you can in fact mix the two approaches.

    (I did a presentation about this a few months ago: <http://www.youtube.com/watch?v=U4h90TgWan0>. Just wherever it says "Couchbase Mobile", think "TouchDB".)

    —Jens
  • On Jul 16, 2012, at 2:39 PM, Jens Alfke <jens...> wrote:

    > (1) Client-server. The database lives on one server machine, as does the "business logic" (I hate that term) that manages your app. This could definitely be implemented with Core Data if you like. The client app just focuses on the UI, and there is some protocol by which it talks to the server to fetch data and to tell it to do stuff. In other words, the client app will not use Core Data.

    Another option for client-server is to create an NSIncrementalStore subclass to use on the client, regardless of what you use on the server.

    The server could be an XML or JSON-based REST web service written using any of a variety of frameworks, the server could be a Mac running a Core Data-based server using a custom protocol you design, the server could be an old mainframe running COBOL... Your NSIncrementalStore subclass can act as an adaptor between it and the rest of your application, allowing your application to be written using Core Data and allowing it to take advantage of its features.

      -- Chris
  • On Mon, 16 Jul 2012 14:39:42 -0700, Jens Alfke <jens...> wrote:
    > (1) Client-server. The database lives on one server machine, as does the "business logic" (I hate that term) that manages your app. This could definitely be implemented with Core Data if you like. The client app just focuses on the UI, and there is some protocol by which it talks to the server to fetch data and to tell it to do stuff. In other words, the client app will not use Core Data.

    Using the CoreDataIncrementalStore-API you can write your own persistence layer under CoreData and use the technology you want to.

    I've written a CoreDataIncrementalStore to support the Valentina Database-Server. Now I can use all given Apple technology in the GUI but for a multi user environment. Of course you still need to think about how to keep the clients synchronized or not or concurrency, but THIS has nothing todo with CoreData but is a general question when designing a multi-user db-app.

    regards

    Thorsten Hohage
    --
    objectmanufactur.com - Hamburg,Germany
  • Chris,

    I already took a look at NSIncrementalStore and it seems to solve one of my problems. I am fairly good with PHP, but I could always create a web service with Rails or something else...

    One of my gripes with using HTTP as the client-server base protocol is the request-response loop. This breaks a lot of desirable features. In my particular case, I want my users to receive "real-time" updates when other users edit records on the database (not really "real-time", but at least after a COMMIT on the database). I know there are ways to do this with HTTP nowadays, but it feels so cumbersome (I mean, at least when compared to the way KVO implements similar behavior) that I don't even want to touch that.

    Maybe I am too focused on the beauties of Cocoa and can't see any further... But...

    So far, IMO, the best option in terms of functionality for my project is the BaseTen framework. It feels like Core Data, uses Core Data model documents and supports PostgreSQL's LISTEN and NOTIFY commands (and a lot more). I think I would have to buy a license because, although my app will not be sold, it will be used in my business and I don't think the GPL allows that. My only fear is the project's future and the fact that [currently] I am not so proficient in Cocoa to fiddle with BaseTen's internals, if I ever need to...

    I am still open to ideas...

    Cheers,
    Flavio

    On 16/07/2012, at 23:43, Chris Hanson wrote:

    > On Jul 16, 2012, at 2:39 PM, Jens Alfke <jens...> wrote:
    >
    >> (1) Client-server. The database lives on one server machine, as does the "business logic" (I hate that term) that manages your app. This could definitely be implemented with Core Data if you like. The client app just focuses on the UI, and there is some protocol by which it talks to the server to fetch data and to tell it to do stuff. In other words, the client app will not use Core Data.
    >
    > Another option for client-server is to create an NSIncrementalStore subclass to use on the client, regardless of what you use on the server.
    >
    > The server could be an XML or JSON-based REST web service written using any of a variety of frameworks, the server could be a Mac running a Core Data-based server using a custom protocol you design, the server could be an old mainframe running COBOL... Your NSIncrementalStore subclass can act as an adaptor between it and the rest of your application, allowing your application to be written using Core Data and allowing it to take advantage of its features.
    >
    > -- Chris
    >
  • On Jul 17, 2012, at 7:05 AM, Flavio Donadio <flavio...> wrote:

    > One of my gripes with using HTTP as the client-server base protocol is the request-response loop. This breaks a lot of desirable features. In my particular case, I want my users to receive "real-time" updates when other users edit records on the database (not really "real-time", but at least after a COMMIT on the database).

    There are alternative message-oriented protocols that allow either side of the connection to send messages (and are more efficient than HTTP, too.) Google's SPDY[1] is one, and my own BLIP[2] is another.

    —Jens

    [1]: http://www.chromium.org/spdy/
    [2]: https://bitbucket.org/snej/mynetwork/wiki/BLIP/Overview
  • On Jul 17, 2012, at 7:05 AM, Flavio Donadio <flavio...> wrote:

    > I already took a look at NSIncrementalStore and it seems to solve one of my problems. I am fairly good with PHP, but I could always create a web service with Rails or something else…

    On the client side, it doesn’t matter what you use on the server side to create a web service. And vice-versa.

    > One of my gripes with using HTTP as the client-server base protocol is the request-response loop. This breaks a lot of desirable features. In my particular case, I want my users to receive "real-time" updates when other users edit records on the database (not really "real-time", but at least after a COMMIT on the database).

    Very few systems will actually do this for you, as it’s almost always *not* a desirable feature. For example, even if you use NSIncrementalStore to implement Core Data access to a web service, you won’t get this kind of functionality.

    In general, users A and B may both be looking at their own internally-consistent data sets. If user A changes something that’s also part of what user B is looking at, you almost always want to keep user B’s view of the data the same until they either explicitly choose to refresh their view, or until they actually attempt to save their own changes. At that point you can allow user B to resolve the conflict between their own changes and user A’s changes to the same objects.

    This is part of why there’s no “easy multiuser” in data management frameworks: The choices that need to be made often boil down to user experience, and can vary dramatically from one application to the next.

      -- Chris
  • Hi Flavio,

    While Apple, SQLite, myself and any sensible software developer advise against it, Core Data can run multi-user by placing the database on a server which supports AFP (e.g. a fast Synology NAS over gigabit ethernet).

    SQLite has limited optimistic locking support, but the record locking will only work over AFP.

    As mentioned elsewhere, you still have the (big) problem of keeping the cached objects of each Mac up to date.  We do it by getting fresh data from the database, then saving, every minute (plus or minus random seconds) during idle time.  The "refresh" code is complicated, ugly and big -- with bad smells abounding.

    To make it work at all in Leopard and beyond (when Apple finally put their foot down on this abomination) you need to set a special Apple SQL pragma when creating your persistent store.  Our line of code looks like this:

    [pragmaOptions setObject: [NSNull null] forKey: @"lock_proxy_file"];

    A retailer has been doing this for their Point of Sale (and back-office) on seven Macs for six years.  We lost a little data a couple times in the early days and we occasionally get a glitch under heavy Saturday load, but we've tamed the beast and now it works surprisingly well -- and fast.

    Be warned that there are, as Ben Trumbull once colorfully observed, "sharp edges".

    Cheers,

    Steve
  • Steve,

    I agree with Apple, SQLite, you and every other sensible developer out there: I won't try this! :)

    Cheers,
    Flavio

    On 18/07/2012, at 04:49, Steve Steinitz wrote:

    > Hi Flavio,
    >
    > While Apple, SQLite, myself and any sensible software developer advise against it, Core Data can run multi-user by placing the database on a server which supports AFP (e.g. a fast Synology NAS over gigabit ethernet).
    >
    > SQLite has limited optimistic locking support, but the record locking will only work over AFP.
    >
    > As mentioned elsewhere, you still have the (big) problem of keeping the cached objects of each Mac up to date.  We do it by getting fresh data from the database, then saving, every minute (plus or minus random seconds) during idle time.  The "refresh" code is complicated, ugly and big -- with bad smells abounding.
    >
    > To make it work at all in Leopard and beyond (when Apple finally put their foot down on this abomination) you need to set a special Apple SQL pragma when creating your persistent store.  Our line of code looks like this:
    >
    > [pragmaOptions setObject: [NSNull null] forKey: @"lock_proxy_file"];
    >
    > A retailer has been doing this for their Point of Sale (and back-office) on seven Macs for six years.  We lost a little data a couple times in the early days and we occasionally get a glitch under heavy Saturday load, but we've tamed the beast and now it works surprisingly well -- and fast.
    >
    > Be warned that there are, as Ben Trumbull once colorfully observed, "sharp edges".
    >
    > Cheers,
    >
    > Steve
previous month july 2012 next month
MTWTFSS
            1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31          
Go to today