creating temp files or temp folders in standard temp file locations in mdimporter on Mac OS X 10.8.3

  • Hi all,

    I posted this originally on the spotlight-dev list but now that I have some sample code that is failing and it is starting to look like a sandboxing issue, I thought I'd try the cocoa-dev list where I have been seeing some spotlight questions too.

    Basically, the subject says it all.  We are having problems with our Spotlight importer when creating temp files or temp folders in the standard temp file locations under Mac OS X 10.8.3/Xcode 4.6.2.  Everything works fine under Mac OS X 10.7.5/Xcode 4.5.2  I've re-read the entire Spotlight Importer Programming Guide and didn't find any help there, no information mentioned about sandbox issues.  I still suspect an app sandbox-like issue but would like more information on this.  Here is some sample code I ran at the beginning of our importer right when we get the GetMetadataForFile call:

        {
            NSString* tempDir = NSTemporaryDirectory();
            NSString* tempDirFolder = [tempDir stringByAppendingPathComponent:@"FinaleTempFiles"];

            NSError* theError = nil;

            BOOL fileOK = [[NSFileManager defaultManager] createDirectoryAtPath:tempDirFolder withIntermediateDirectories:NO attributes:nil error:&theError ];
            if ( !fileOK )
                NSLog(@"createDirectoryAtPath %@", [theError localizedDescription]);
        }

        {
            NSURL* tempDir = [NSURL fileURLWithPath:NSTemporaryDirectory()];
            NSURL* tempDirFolder = [tempDir URLByAppendingPathComponent:@"FinaleTempFiles"];

            NSError* theError = nil;

            BOOL fileOK = [[NSFileManager defaultManager] createDirectoryAtURL:tempDirFolder withIntermediateDirectories:NO attributes:nil error:&theError ];
            if ( !fileOK )
                NSLog(@"createDirectoryAtURL %@", [theError localizedDescription]);
        }

        {
            NSString* tempDir = NSTemporaryDirectory();
            NSString* tempDirFile = [tempDir stringByAppendingPathComponent:@"EmptyFile"];

            NSData* emptyContents = [[NSData alloc] init];

            BOOL success = [[NSFileManager defaultManager] createFileAtPath:tempDirFile contents:emptyContents attributes:nil];

            if ( !success )
                NSLog(@"createFileAtPath: Error was code: %d - message: %s", errno, strerror(errno));
        }

        {
            NSURL* tempDir = [NSURL fileURLWithPath:NSTemporaryDirectory()];
            NSURL* tempDirFile = [tempDir URLByAppendingPathComponent:@"EmptyFile"];

            NSData* emptyContents = [[NSData alloc] init];

            NSError *theError = nil;
            BOOL success = [emptyContents writeToURL:tempDirFile options:NSDataWritingAtomic error:&theError];
            if ( !success )
                NSLog(@"writeToURL %@ ", [theError localizedDescription]);
        }

    and the corresponding NSLog output:

    2013-05-31 18:52:46.830 mdimport[43160:707] createDirectoryAtPath You don’t have permission to save the file “FinaleTempFiles” in the folder “T”.
    2013-05-31 18:52:50.446 mdimport[43160:707] createDirectoryAtURL You don’t have permission to save the file “FinaleTempFiles” in the folder “T”.
    2013-05-31 18:52:57.821 mdimport[43160:707] createFileAtPath: Error was code: 1 - message: Operation not permitted
    2013-05-31 18:53:07.692 mdimport[43160:707] writeToURL You don’t have permission to save the file “EmptyFile”.

    I get the same output for our 32-bit and 64-bit builds.  Surely we must still be able to create temp files?

    Thanks,
    Chris

    _________________________________

    Chris Cianflone
    Senior Software Engineer
    www.makemusic.com<http://www.makemusic.com>
  • On Sat, Jun 1, 2013, at 11:45 AM, Cianflone, Chris wrote:
    > I get the same output for our 32-bit and 64-bit builds.  Surely we must
    > still be able to create temp files?

    Spotlight importers run within a worker process; thus, they inherit the
    sandbox of the worker process, not the sandbox of your app (which might
    not even be running). Because these processes are run automatically by
    the system with the permissions of the user, it makes sense to severely
    restrict the worker's sandbox, or else the process can silently behave
    in ways the user does not expect.

    While it would probably be a good idea to file a request that Spotlight
    importers be run within their owning app's sandbox container (thus
    granting it access to the container's temp directory), I'm kind of
    curious why your Spotlight importer needs to create temporary files at
    all.

    --Kyle Sluder
  • << I'm kind of
    curious why your Spotlight importer needs to create temporary files at
    all.
    >>

    I wish it weren't this way and I had already started looking at this as a backup plan if we can get the temp files to work. But it runs too deep to rid ourselves of temp files. I was almost going to say it was a legacy issue (which part of it is), but we recently added a new file format which needs to be unzipped to a temp file to continue reading the file.  And old files need to go through a converter process which uses temp files.

    Just to be clear too, I was running into the problem first when debugging in Xcode.  Then I tried installing the plugin in /Library/Spotlight and started seeing all these problems in the console:

    6/2/13 4:11:15.325 PM sandboxd[47546]: ([47726]) mdworker32(47726) deny file-write-create /private/var/folders/xh/sjj1bz211r56_tfmf5tt2025p36645/T/TemporaryItems/(A Document Being Saved By mdworker32) (import fstype:hfs fsflag:480D000 flags:240000005E diag:0 uti:com.makemusic.mus plugin:/Library/Spotlight/Finaled.mdimporter - find suspect file using: sudo mdutil -t 3889893)

    Using that info, I did some more googling and it looks like we are not alone with problems like this.

    Fearing we won't be able to get rid of temp files in our mdimporter, does anyone have any other suggestions other than telling our users, sorry it doesn't work under 10.8.

    Thanks,
    Chris

    ________________________________________
    From: Kyle Sluder [<kyle...>]
    Sent: Saturday, June 01, 2013 4:04 PM
    To: Cianflone, Chris; <cocoa-dev...>
    Subject: Re: creating temp files or temp folders in standard temp file locations in mdimporter on Mac OS X 10.8.3

    On Sat, Jun 1, 2013, at 11:45 AM, Cianflone, Chris wrote:
    > I get the same output for our 32-bit and 64-bit builds.  Surely we must
    > still be able to create temp files?

    Spotlight importers run within a worker process; thus, they inherit the
    sandbox of the worker process, not the sandbox of your app (which might
    not even be running). Because these processes are run automatically by
    the system with the permissions of the user, it makes sense to severely
    restrict the worker's sandbox, or else the process can silently behave
    in ways the user does not expect.

    While it would probably be a good idea to file a request that Spotlight
    importers be run within their owning app's sandbox container (thus
    granting it access to the container's temp directory), I'm kind of
    curious why your Spotlight importer needs to create temporary files at
    all.

    --Kyle Sluder
  • <<if we can get the temp files to work>>

    Whoops, that should obviously say "if we can't get the temp files to work".

    Chris

    ________________________________________
    From: cocoa-dev-bounces+ccianflone=<makemusic.com...> [cocoa-dev-bounces+ccianflone=<makemusic.com...>] on behalf of Cianflone, Chris [<ccianflone...>]
    Sent: Sunday, June 02, 2013 4:16 PM
    To: Kyle Sluder; <cocoa-dev...>
    Subject: RE: creating temp files or temp folders in standard temp file  locations in mdimporter on Mac OS X 10.8.3

    << I'm kind of
    curious why your Spotlight importer needs to create temporary files at
    all.
    >>

    I wish it weren't this way and I had already started looking at this as a backup plan if we can get the temp files to work. But it runs too deep to rid ourselves of temp files. I was almost going to say it was a legacy issue (which part of it is), but we recently added a new file format which needs to be unzipped to a temp file to continue reading the file.  And old files need to go through a converter process which uses temp files.

    Just to be clear too, I was running into the problem first when debugging in Xcode.  Then I tried installing the plugin in /Library/Spotlight and started seeing all these problems in the console:

    6/2/13 4:11:15.325 PM sandboxd[47546]: ([47726]) mdworker32(47726) deny file-write-create /private/var/folders/xh/sjj1bz211r56_tfmf5tt2025p36645/T/TemporaryItems/(A Document Being Saved By mdworker32) (import fstype:hfs fsflag:480D000 flags:240000005E diag:0 uti:com.makemusic.mus plugin:/Library/Spotlight/Finaled.mdimporter - find suspect file using: sudo mdutil -t 3889893)

    Using that info, I did some more googling and it looks like we are not alone with problems like this.

    Fearing we won't be able to get rid of temp files in our mdimporter, does anyone have any other suggestions other than telling our users, sorry it doesn't work under 10.8.

    Thanks,
    Chris

    ________________________________________
    From: Kyle Sluder [<kyle...>]
    Sent: Saturday, June 01, 2013 4:04 PM
    To: Cianflone, Chris; <cocoa-dev...>
    Subject: Re: creating temp files or temp folders in standard temp file locations in mdimporter on Mac OS X 10.8.3

    On Sat, Jun 1, 2013, at 11:45 AM, Cianflone, Chris wrote:
    > I get the same output for our 32-bit and 64-bit builds.  Surely we must
    > still be able to create temp files?

    Spotlight importers run within a worker process; thus, they inherit the
    sandbox of the worker process, not the sandbox of your app (which might
    not even be running). Because these processes are run automatically by
    the system with the permissions of the user, it makes sense to severely
    restrict the worker's sandbox, or else the process can silently behave
    in ways the user does not expect.

    While it would probably be a good idea to file a request that Spotlight
    importers be run within their owning app's sandbox container (thus
    granting it access to the container's temp directory), I'm kind of
    curious why your Spotlight importer needs to create temporary files at
    all.

    --Kyle Sluder
  • On Jun 2, 2013, at 2:16 PM, "Cianflone, Chris" <ccianflone...> wrote:

    > Just to be clear too, I was running into the problem first when debugging in Xcode.  Then I tried installing the plugin in /Library/Spotlight and started seeing all these problems in the console:

    This _probably_ won’t work, but what if your importer is contained within your app wrapper, at Contents/Library/Spotlight? Does the sandbox machinery perhaps magically do the right thing? You might need to independently sign the importer, or perhaps refrain from doing so.

    If it fails to help, at least you have more info for the enhancement request.

    --Kyle Suder
  • On Jun 1, 2013, at 14:04 , Kyle Sluder <kyle...> wrote:

    > Spotlight importers run within a worker process; thus, they inherit the
    > sandbox of the worker process, not the sandbox of your app (which might
    > not even be running).

    The part of this line of thinking that I don't understand is why the worker process, whatever it is, shouldn't have access to a temporary directory of its own.

    The sample code tries to create a subdirectory inside the directory pointed to by NSTemporaryDirectory(). If this code is running in the context of an app sandbox, then there should be no problem creating a subdirectory in its temp dir. If the code is running in (say) a background worker process, then it should have its own temp dir, and there should still be no problem.

    All of which makes me wonder if NSTemporaryDirectory() just isn't available in the worker process, for some non-apparent reason. In that case, isn't there a Unix-ey temp dir to use instead (/var/something or /usr/something)?

    Or have I missed an essential step in your reasoning?
  • On Jun 3, 2013, at 1:17 AM, Quincey Morris <quinceymorris...> wrote:

    > On Jun 1, 2013, at 14:04 , Kyle Sluder <kyle...> wrote:
    >
    >> Spotlight importers run within a worker process; thus, they inherit the
    >> sandbox of the worker process, not the sandbox of your app (which might
    >> not even be running).
    >
    > The part of this line of thinking that I don't understand is why the worker process, whatever it is, shouldn't have access to a temporary directory of its own.

    Well, from a security standpoint, you might start from the opposite point: why should an mdimporter plugin be able to write to the filesystem at all?

    It doesn't help that Spotlight traditionally reuses worker processes for multiple mdimporters, and you can't re-sandbox a process at runtime. But the notion of reusing worker processes seems antithetical to sandboxing anyway, so hopefully that technique isn't a stumbling block anymore.

    --Kyle Sluder
  • On Mon, 3 Jun 2013 01:17:38 -0700, Quincey Morris said:

    > The sample code tries to create a subdirectory inside the directory
    > pointed to by NSTemporaryDirectory(). If this code is running in the
    > context of an app sandbox, then there should be no problem creating a
    > subdirectory in its temp dir. If the code is running in (say) a
    > background worker process, then it should have its own temp dir, and
    > there should still be no problem.
    >
    > All of which makes me wonder if NSTemporaryDirectory() just isn't
    > available in the worker process, for some non-apparent reason. In that
    > case, isn't there a Unix-ey temp dir to use instead (/var/something or /
    > usr/something)?

    NSTemporaryDirectory() is an old path-based API.  Perhaps the newer URL-based APIs (URLForDirectory:inDomain:appropriateForURL:create:error:) might return a more appropriate temp directory...

    Cheers,

    --
    ____________________________________________________________
    Sean McBride, B. Eng                <sean...>
    Rogue Research                        www.rogue-research.com
    Mac Software Developer              Montréal, Québec, Canada
  • On Jun 3, 2013, at 09:58 , "Sean McBride" <sean...> wrote:

    > NSTemporaryDirectory() is an old path-based API.  Perhaps the newer URL-based APIs (URLForDirectory:inDomain:appropriateForURL:create:error:) might return a more appropriate temp directory...

    In this document:

    https://developer.apple.com/library/mac/#documentation/Security/Conceptual/
    AppSandboxDesignGuide/AppSandboxInDepth/AppSandboxInDepth.html


    (which was updated this year, so it shouldn't out-of-date, I'd hope), under the heading "Container Directories and File System Access", there's list of places a sandboxed app can access:

    > When you adopt App Sandbox, your application has access to the following locations:
    >
    > • The app container directory. Upon first launch, the operating system creates a special directory for use by your app—and only by your app—called a container. Each user on a system gets an individual container for your app, within their home directory; your app has unfettered read/write access to the container for the user who ran it.
    > […]
    > • Temporary directories, command-line tool directories, and specific world-readable locations. A sandboxed app has varying degrees of access to files in certain other well-defined locations.

    and then, under the heading "Powerbox and File System Access Outside of Your Container":

    > In addition [to locations made available via Powerbox], the system automatically permits a sandboxed app to:
    >
    > […]
    > • Read and write files in directories created by calling NSTemporaryDirectory.
    >
    > Note: The /tmp directory is not accessible from sandboxed apps. You must use the NSTemporaryDirectory function to obtain a temporary location for your app’s temporary files.

    That seems to answer your comment (NSTemporaryDirectory() does seem to be the right API) and Kyle's last comment (sandboxing doesn't intrinsically disable file writing -- it merely restricts places where files can be written without entitlements to a few known locations).

    The deeper question is whether a spotlight worker process is running in an *app* sandbox at all, or whether it has some other kind of security context.

    Under the circumstances, I'd suggest that the OP should use a TSI to find out why NSTemporaryDirectory() doesn't work.
  • On Mon, 3 Jun 2013 10:44:24 -0700, Quincey Morris said:

    >> Note: The /tmp directory is not accessible from sandboxed apps. You
    > must use the NSTemporaryDirectory function to obtain a temporary
    > location for your app’s temporary files.
    >
    > That seems to answer your comment (NSTemporaryDirectory() does seem to
    > be the right API) and Kyle's last comment (sandboxing doesn't
    > intrinsically disable file writing -- it merely restricts places where
    > files can be written without entitlements to a few known locations).

    Ouch.  In fact, the word "must" is troubling.  I'm using URLForDirectory:inDomain:appropriateForURL:create:error: with NSItemReplacementDirectory successfully with App Sandbox.  Hopefully they mean "must" as opposed to hardcoding /tmp.

    > The deeper question is whether a spotlight worker process is running in
    > an *app* sandbox at all, or whether it has some other kind of security
    > context.

    Indeed.

    Cheers,

    --
    ____________________________________________________________
    Sean McBride, B. Eng                <sean...>
    Rogue Research                        www.rogue-research.com
    Mac Software Developer              Montréal, Québec, Canada
  • Just an FYI that we have opened a TSI on this.  I'll let you know the details once we sort this out with Apple.

    Thanks,
    Chris
    ________________________________________
    From: cocoa-dev-bounces+ccianflone=<makemusic.com...> [cocoa-dev-bounces+ccianflone=<makemusic.com...>] on behalf of Sean McBride [<sean...>]
    Sent: Monday, June 03, 2013 12:51 PM
    To: Quincey Morris
    Cc: <cocoa-dev...>
    Subject: Re: creating temp files or temp folders in standard temp file  locations in mdimporter on Mac OS X 10.8.3

    On Mon, 3 Jun 2013 10:44:24 -0700, Quincey Morris said:

    >> Note: The /tmp directory is not accessible from sandboxed apps. You
    > must use the NSTemporaryDirectory function to obtain a temporary
    > location for your app’s temporary files.
    >
    > That seems to answer your comment (NSTemporaryDirectory() does seem to
    > be the right API) and Kyle's last comment (sandboxing doesn't
    > intrinsically disable file writing -- it merely restricts places where
    > files can be written without entitlements to a few known locations).

    Ouch.  In fact, the word "must" is troubling.  I'm using URLForDirectory:inDomain:appropriateForURL:create:error: with NSItemReplacementDirectory successfully with App Sandbox.  Hopefully they mean "must" as opposed to hardcoding /tmp.

    > The deeper question is whether a spotlight worker process is running in
    > an *app* sandbox at all, or whether it has some other kind of security
    > context.

    Indeed.

    Cheers,

    --
previous month june 2013 next month
MTWTFSS
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
Go to today