[NSBitmapImageRep getBitmapDataPlanes] copying data?

  • I am looking at the performance of my code, and have found that rather a lot of time is spent in [NSBitmapImageRep getBitmapDataPlanes]. This is rather disappointing because I had assumed this was a 'trivial' way of getting a pointer to the actual data store itself in the case of raster data. Investigation using shark appears to confirm that the call is making a copy of the data(?). Can anyone shed any light on what is going on and whether I can do anything about it?

    The sequence I'm using (hope I haven't missed anything out) is:

    NSImage *theImage = [[NSImage alloc] initWithContentsOfFile:fileName];
    ...
    NSObject *imageRepresentation = [[image representations] objectAtIndex:imgRepresentationIndex];
    bmp = (NSBitmapImageRep*)imageRepresentation;
    ...
    const unsigned char *srcData = [bmp bitmapData];

    The file is in fact a TIFF file, so it does contain raster data. Am I going about this the wrong way as a means of getting at the raw pixels stored in a file? Thanks for any suggestions.

    Jonny
  • On Feb 21, 2011, at 03:56, Jonathan Taylor wrote:

    > I am looking at the performance of my code, and have found that rather a lot of time is spent in [NSBitmapImageRep getBitmapDataPlanes]. This is rather disappointing because I had assumed this was a 'trivial' way of getting a pointer to the actual data store itself in the case of raster data. Investigation using shark appears to confirm that the call is making a copy of the data(?). Can anyone shed any light on what is going on and whether I can do anything about it?

    Assuming you're running on Snow Leopard, the relevant information is in this document:

    http://developer.apple.com/library/mac/#releasenotes/Cocoa/AppKit.html

    under this heading (about 60% of the way through the document):

    NSBitmapImageRep: CoreGraphics impedance matching and performance notes

    I think you'll find the answer to the question of why the data is being copied in there.
  • >> I am looking at the performance of my code, and have found that rather a lot of time is spent in [NSBitmapImageRep getBitmapDataPlanes]. This is rather disappointing because I had assumed this was a 'trivial' way of getting a pointer to the actual data store itself in the case of raster data. Investigation using shark appears to confirm that the call is making a copy of the data(?). Can anyone shed any light on what is going on and whether I can do anything about it?
    >
    > Assuming you're running on Snow Leopard, the relevant information is in this document:
    >
    > http://developer.apple.com/library/mac/#releasenotes/Cocoa/AppKit.html

    Thanks very much for your reply, that's very helpful. So do I understand it correctly that there is no way at all of peeking (read only) at the pixels of a NSBitmapImageRep without triggering a copy? That's a bit of a shame if so - guess I'll have to look into CoreImage.

    Cheers
    Jonny
  • On Feb 21, 2011, at 13:19, Jonny Taylor wrote:

    > So do I understand it correctly that there is no way at all of peeking (read only) at the pixels of a NSBitmapImageRep without triggering a copy? That's a bit of a shame if so - guess I'll have to look into CoreImage.

    If I understand the release notes properly, there are 3 choices:

    1. Draw from the NSBitmapImageRep into some kind of drawing context. This may not be as expensive as it sounds. You could draw into a 1-pixel rect if you wanted.

    2. Get the underlying NSBitmapImageRep data, thus causing a copy. If you're going to be accessing multiple pixel values, this is also not be as expensive as it sounds, so long as you don't do anything that forces the frameworks to copy the data for every pixel.

    3. Use something else, such as CGImage, instead.
  • On 21 Feb 2011, at 21:29, Quincey Morris wrote:
    > On Feb 21, 2011, at 13:19, Jonny Taylor wrote:
    >
    >> So do I understand it correctly that there is no way at all of peeking (read only) at the pixels of a NSBitmapImageRep without triggering a copy? That's a bit of a shame if so - guess I'll have to look into CoreImage.
    >
    > If I understand the release notes properly, there are 3 choices:
    >
    > 1. Draw from the NSBitmapImageRep into some kind of drawing context. This may not be as expensive as it sounds. You could draw into a 1-pixel rect if you wanted.
    >
    > 2. Get the underlying NSBitmapImageRep data, thus causing a copy. If you're going to be accessing multiple pixel values, this is also not be as expensive as it sounds, so long as you don't do anything that forces the frameworks to copy the data for every pixel.
    >
    > 3. Use something else, such as CGImage, instead.

    Thanks. In this case I need to access every single pixel value unfortunately. I was using NSImages for convenience since I do display some images in the GUI, etc, but the main work of the program involves image processing on the pixel values. Sounds like CGImage is probably what I need if I really want to maximise performance in this case. I'm actually doing a fair amount of work with the pixel data, but nevertheless this one copy (specifically the zero fills as well as memcpy) at the start takes considerably longer than the analysis itself.

    Jonny
  • On Feb 21, 2011, at 3:53 PM, Jonathan Taylor wrote:

    > On 21 Feb 2011, at 21:29, Quincey Morris wrote:
    >> 3. Use something else, such as CGImage, instead.
    >
    > Thanks. In this case I need to access every single pixel value unfortunately. I was using NSImages for convenience since I do display some images in the GUI, etc, but the main work of the program involves image processing on the pixel values. Sounds like CGImage is probably what I need if I really want to maximise performance in this case. I'm actually doing a fair amount of work with the pixel data, but nevertheless this one copy (specifically the zero fills as well as memcpy) at the start takes considerably longer than the analysis itself.

    NSBitmapImageRep is basically a wrapper around CGImage now.  That's what those release notes were saying.  So, switching to CGImage probably won't get you anything.

    Perhaps Quincey meant to say CIImage, a CoreImage object.  That's what the release notes suggest for doing pixel work.

    Regards,
    Ken
  • On Feb 21, 2011, at 13:53, Jonathan Taylor wrote:

    > I'm actually doing a fair amount of work with the pixel data, but nevertheless this one copy (specifically the zero fills as well as memcpy) at the start takes considerably longer than the analysis itself.

    I recently ran into this problem with zero fills, too. When you let NSBitmapImageRep allocate its own memory, it [apparently] creates a NSData object, and that elaborately zero fills the allocated memory that you're about to overwrite with your own data.

    Weirdly, the zero filling took *much* longer than copying pixel data into the NSData buffer afterwards, according to Instruments -- by a factor of something like 35x, IIRC, with a buffer size of about 6MB. I'm guessing that the zeroing was being done as a side-effect of a VM page allocation and that's what was so slow, but it was rather mysterious. (Instruments reported that the time was spent in a function called '__bzero', but again weirdly that didn't seem to be related to 'bzero', which I was using elsewhere in my code. Perhaps.)

    My solution was simply to allocate the bitmap memory myself and pass the pointer into the NSBitmapImageRep init. NSAllocateCollectable works fine for this if you're using garbage collection (no further memory management to worry about), but it might be a bit more complicated if you're stuck on retain/release.

    On Feb 21, 2011, at 14:02, Ken Thomases wrote:

    > NSBitmapImageRep is basically a wrapper around CGImage now.  That's what those release notes were saying.  So, switching to CGImage probably won't get you anything.
    >
    > Perhaps Quincey meant to say CIImage, a CoreImage object.  That's what the release notes suggest for doing pixel work.

    You're giving me too much credit -- I'm not sure what I was trying to say.

    OK, what I was *trying* to say is that NSBitmapImageRep in SnowLeopard is basically an immutable object pretending to be mutable. My suggestion was to use a really mutable object instead. I actually don't know if CGImage counts as mutable (in the pixel data sense) or not, because every time I try to use one my application crashes and I go back to NSBitmapImageRep.

    I think you're right though -- CIImage is probably the right thing to use. Again, I don't know because every time I try to use one my application crashes and ... ;)
  • On Mon, 21 Feb 2011 14:27:39 -0800, Quincey Morris said:

    > OK, what I was *trying* to say is that NSBitmapImageRep in SnowLeopard
    > is basically an immutable object pretending to be mutable. My suggestion
    > was to use a really mutable object instead. I actually don't know if
    > CGImage counts as mutable (in the pixel data sense) or not, because
    > every time I try to use one my application crashes and I go back to
    > NSBitmapImageRep.
    >
    > I think you're right though -- CIImage is probably the right thing to
    > use. Again, I don't know because every time I try to use one my
    > application crashes and ... ;)

    I too have ran into this issue recently, with an old project that is
    half-resurrected.

    Performance was good in 10.5, but now horrible in 10.6.  I'm not sure
    what to do instead.

    The CIImage docs say "Core Image images are immutable", and indeed I
    don't see any method like bitmapData. :(

    --
    ____________________________________________________________
    Sean McBride, B. Eng                <sean...>
    Rogue Research                        www.rogue-research.com
    Mac Software Developer              Montréal, Québec, Canada
  • > I too have ran into this issue recently, with an old project that is
    > half-resurrected.
    >
    > Performance was good in 10.5, but now horrible in 10.6.  I'm not sure
    > what to do instead.
    >
    > The CIImage docs say "Core Image images are immutable", and indeed I
    > don't see any method like bitmapData. :(
    Oh dear, looks like you're right! Immutable would have been fine by me, all I want is const access to the pixels, but that doesn't seem to be possible either.

    So as far as my situation goes, what I'm after is an efficient way of starting from a path to some sort of raster image and obtaining a raw pixel buffer I can read. [Obviously if actually reading from disk then the disk will be the bottleneck, but it's often in the disk cache, in which case the extra data copy associated with [NSBitmapImageRep getBitmapDataPlanes] is a problem. I was rather liking the file-format-agnostic abilities of initWithContentsOfFile. Any thoughts?
  • On Feb 21, 2011, at 5:07 PM, Jonathan Taylor wrote:

    > So as far as my situation goes, what I'm after is an efficient way of starting from a path to some sort of raster image and obtaining a raw pixel buffer I can read. [Obviously if actually reading from disk then the disk will be the bottleneck, but it's often in the disk cache, in which case the extra data copy associated with [NSBitmapImageRep getBitmapDataPlanes] is a problem. I was rather liking the file-format-agnostic abilities of initWithContentsOfFile. Any thoughts?

    What makes you think the data copy is "extra"?

    If you read that AppKit release note, you'll see that NSImage may not be decoding the file contents right off the bat.  For example, it mentions that "[i]f you initialize a NSImage from a JPEG file, then draw it in a PDF, you should get a PDF of the same file size as the original JPEG".  In other words, NSImage is keeping the image data in its original JPEG-compressed format, not as a rasterized bitmap.

    So, you may be seeing the one necessary/required data copy during decoding of the image, not anything extra.

    Regards,
    Ken
  • On Feb 21, 2011, at 15:07, Jonathan Taylor wrote:

    > So as far as my situation goes, what I'm after is an efficient way of starting from a path to some sort of raster image and obtaining a raw pixel buffer I can read. [Obviously if actually reading from disk then the disk will be the bottleneck, but it's often in the disk cache, in which case the extra data copy associated with [NSBitmapImageRep getBitmapDataPlanes] is a problem. I was rather liking the file-format-agnostic abilities of initWithContentsOfFile. Any thoughts?

    Ken already said some of my intended response in his reply, but ...

    It's even possible that the underlying CGImage is memory mapped to the original file so that the compressed data may not even have been read into the disk cache yet (or, not all of it), at the time you ask for the pixel data. You have to assume that retrieving the pixel data via 'getBitmapDataPlanes' might well be more efficient than anything you can do at the application level.

    You know you're stuck with disk reads at some point in the process, which are going to be as slow as they're going to be. I'm going to go out on a limb and say that if your application can't swallow the additional cost of uncompressing and/or copying inside 'getBitmapDataPlanes' *once* per image**, your application design is broken*** -- or your Mac hardware is too slow for that application. :)

    ** Assuming you arrange to get rid of the unwanted memory zeroing, if Instruments shows it to be a factor.

    *** That is, you probably need to bypass NSBitmapImageRep and go read just the parts of the file you want, which is going to be a huge PITA.
  • On Mon, Feb 21, 2011 at 3:49 PM, Ken Thomases <ken...> wrote:

    > On Feb 21, 2011, at 5:07 PM, Jonathan Taylor wrote:
    >
    >> So as far as my situation goes, what I'm after is an efficient way of
    > starting from a path to some sort of raster image and obtaining a raw pixel
    > buffer I can read. [Obviously if actually reading from disk then the disk
    > will be the bottleneck, but it's often in the disk cache, in which case the
    > extra data copy associated with [NSBitmapImageRep getBitmapDataPlanes] is a
    > problem. I was rather liking the file-format-agnostic abilities of
    > initWithContentsOfFile. Any thoughts?
    >
    > What makes you think the data copy is "extra"?
    >
    > If you read that AppKit release note, you'll see that NSImage may not be
    > decoding the file contents right off the bat.  For example, it mentions that
    > "[i]f you initialize a NSImage from a JPEG file, then draw it in a PDF, you
    > should get a PDF of the same file size as the original JPEG".  In other
    > words, NSImage is keeping the image data in its original JPEG-compressed
    > format, not as a rasterized bitmap.
    >
    > So, you may be seeing the one necessary/required data copy during decoding
    > of the image, not anything extra.
    >

    The other issue, as mentioned in those release notes, is pixel format.  It's
    pretty nearly always not correct to call bitmapData on a rep you did not
    create yourself with the  method of NSBitmapImageRep that explicitly gives
    pixel format.

    It sounds like you want to do this:
    (1) Read image file into NSBitmapImageRep
    (2) Directly access data buffer of NSBitmapImageRep

    The problem with that is that you are almost certainly hardcoding the
    expected pixel format (e.g., 8 bit per component ARGB host endian) into your
    app.  Probably you think this is safe because you created the image file in
    the first place.

    However!  ImageIO, the underlying home framework of all image file format
    reading and writing on Mac OS X, makes no promises about the relationship of
    the file to the in-memory pixel format.  Certainly you know that many file
    formats are compressed, whereas the buffer you returned from the bitmapData
    method of NSBitmapImageRep is not.  Further, there are many on-disk formats
    that are not supported by Quartz.  Besides that, you might figure that the
    OS will never reduce the number of pixel formats, so if you're currently
    getting something tightly related to what you see in the file, you will
    expect to get that in the future.  That's not the case though… CoreGraphics
    has talked about standardizing everything with less data than ARGB 32 host
    endian to ARGB 32 host endian right when its read in, to simplify and
    speedup later steps of the graphics pipeline.  There are formats for which
    that happens today (I think maybe RGB JPEG data is coming in as as ARGB as
    of 10.6, perhaps?).  That change was made as part of work in 10.6 to support
    decoding only rectangular blocks of an image throughout the graphics
    pipeline.  Asking for the bitmapData also defeats that pipeline and forces
    everything into memory.

    Plus, it's pretty hard to write code that deals expressly with pixel data
    without putting things in a standard colorspace.  RGBA (.3, .5, .6, 1.0) is
    a different color in sRGB vs generic RGB.  For most pixel-processing
    algorithms, you'll get different visual results if you naively apply the
    same mathematics in different colorspaces.

    Anyway, point is, in all likelihood, you do need to draw your image to a
    bitmap in known format in order to work with the pixels.  You probably only
    need to do this once though…  if you're seeing repeated copying in
    bitmapData, that suggests that you're alternately drawing and asking for
    pixel data from the image.  If you're never modifying the data, you could
    instead store the data on the side in addition to the rep.  Storing can be
    done via subclass, wrapper object, global dictionary mapping rep object to
    data, objc associative storage, etc.  Or, look at just drawing the image
    into a bitmap when you need to see the data.  That's less work for you to
    implement, and may find that it performs well, depending on what was really
    costing you before.  You could consider reusing one bitmap to draw in from
    multiple images to avoid malloc'ing and free'ing memory.

    -Ken
    Cocoa Frameworks
  • On 21 Feb 2011, at 23:49, Ken Thomases wrote:
    > What makes you think the data copy is "extra"?
    >
    > If you read that AppKit release note, you'll see that NSImage may not be decoding the file contents right off the bat.  For example, it mentions that "[i]f you initialize a NSImage from a JPEG file, then draw it in a PDF, you should get a PDF of the same file size as the original JPEG".  In other words, NSImage is keeping the image data in its original JPEG-compressed format, not as a rasterized bitmap.

    That's a very good point. In this case, though, I think there probably is some excess stuff going on. Looking at what's going on with Shark, approx 1/5th of this thread's time is spent in 'read' (backed by disk cache), 1/4 in copyImageBlockSetTIFF, which seems to be done lazily but I presume is the decoding phase you allude to [although in this case it's pretty much just a memcpy], and 1/5 in imageProvider_getBytes, which copies the data *again*.

    On 22 Feb 2011, at 00:05, Quincey Morris wrote:
    >
    > You know you're stuck with disk reads at some point in the process, which are going to be as slow as they're going to be. I'm going to go out on a limb and say that if your application can't swallow the additional cost of uncompressing and/or copying inside 'getBitmapDataPlanes' *once* per image**, your application design is broken*** -- or your Mac hardware is too slow for that application. :)

    Very true, though as I mentioned the disk cache often works in my favour here. To be honest the specific issue of getBitmapDataPlanes is not really on the most critical realtime path in my code. As much as anything, it was a case of having spotted this issue I wanted to understand it thoroughly because there is GUI drawing (using NSImages/bitmaps etc) that is taking place in parallel with the main performance-critical stuff. Also, as Ken Ferry's reply revealed, there are evidently some "impedance matching" issues I should really understand better, so this is all extremely helpful.

    On 22 Feb 2011, at 02:02, Ken Ferry wrote:
    > However!  ImageIO, the underlying home framework of all image file format reading and writing on Mac OS X, makes no promises about the relationship of the file to the in-memory pixel format.  Certainly you know that many file formats are compressed, whereas the buffer you returned from the bitmapData method of NSBitmapImageRep is not.  Further, there are many on-disk formats that are not supported by Quartz.  Besides that, you might figure that the OS will never reduce the number of pixel formats, so if you're currently getting something tightly related to what you see in the file, you will expect to get that in the future.  That's not the case though… CoreGraphics has talked about standardizing everything with less data than ARGB 32 host endian to ARGB 32 host endian right when its read in, to simplify and speedup later steps of the graphics pipeline.  There are formats for which that happens today (I think maybe RGB JPEG data is coming in as as ARGB as of 10.6, perhaps?).  That change was made as part of work in 10.6 to support decoding only rectangular blocks of an image throughout the graphics pipeline.  Asking for the bitmapData also defeats that pipeline and forces everything into memory.
    >
    > Plus, it's pretty hard to write code that deals expressly with pixel data without putting things in a standard colorspace.  RGBA (.3, .5, .6, 1.0) is a different color in sRGB vs generic RGB.  For most pixel-processing algorithms, you'll get different visual results if you naively apply the same mathematics in different colorspaces.
    >
    > Anyway, point is, in all likelihood, you do need to draw your image to a bitmap in known format in order to work with the pixels.  You probably only need to do this once though…  if you're seeing repeated copying in bitmapData, that suggests that you're alternately drawing and asking for pixel data from the image.  If you're never modifying the data, you could instead store the data on the side in addition to the rep.  Storing can be done via subclass, wrapper object, global dictionary mapping rep object to data, objc associative storage, etc.  Or, look at just drawing the image into a bitmap when you need to see the data.  That's less work for you to implement, and may find that it performs well, depending on what was really costing you before.  You could consider reusing one bitmap to draw in from multiple images to avoid malloc'ing and free'ing memory.
    Thanks very much - lots of food for thought there. I think some of the issues you mention ~should~ be less severe because I am working with greyscale data, but maybe it's not guaranteed. The pattern I was following was at least previously a "recommended" approach I presume (see for example CocoaCreateMovie/CopyNSImageToGWorld). It sounds from what you're saying as if drawing the image into a bitmap may actually now be potentially more efficient and quite a nice solution.

    However, in my case I would have to be extremely careful about matching formats etc. My code is doing analysis of scientific video data, and it is absolutely crucial that I can swap data between TIFF files and buffers in memory and leave the exact original greyscale values unchanged. From what you are saying, I'm starting to wonder whether there is any way at all that I can guarantee that, when using NSImage/BitmapImageRep. I like that approach because of the ease of working with other file types if necessary, and doing GUI display. I had assumed extracting the NSBitmapImageRep was safe (I do in fact check there is only one...), but it sounds like you're saying there's no promise of that? It may be that Cocoa's priorities/promises will never really be perfectly aligned to my needs here, but if you have any suggestions for a more appropriate approach I'd love to hear them!

    Thanks again to everyone.
    Jonny
previous month february 2011 next month
MTWTFSS
  1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28            
Go to today