iPhoto-like scroll view

  • I am currently experimenting with a custom view that basically shows
    thumbnails like iPhoto does. Now while the basic drawing is still
    terrible inefficient (loads NSImages on every drawRect(), so it still
    needs caching ....and let's not talk about the memory leak (flood?)
    of the current code), what I am really wondering is something else:

    Let's assume performance is bad to draw the rects that need an
    update. This somehow results into a very slow feeling of the
    application when I use the window scroll bar. What I am wondering is
    how to keep that still snappy and maybe just skip a few updates.
    Because when I scroll the bar at a very high speed I really only car
    about the final position of the bar. So how would I get the drawing
    to be a bit more lazy. Only do the update when the speed of the bar
    becomes slower. Does that require drawing to happen in different thread?

    Something related would be to only draw empty rects when the speed of
    the slider becomes too high. And then switch back to a full display
    at lower speeds. How could this be done?

    Any wise words?

    cheers
    --
    Torsten
  • It seems that everybody and their brothers are
    implementing iPhoto like views.

    First, scrolling has nothing to do with zooming
    repositioning thumbnail image display.  NSScrollView
    transparently handles scrolling regardless of the
    content view within the scroll view.  YES, it even
    works when scrolling an NSOpenGLView.

    Second, if you start out using NSImage and or
    NSImageCell or NSImageView or NSMatrix, you are on the
    wrong path.  An iPhoto like view screams "I want to be
    implemented with Open GL".

    You might get away with using NSBitmapImageRep or Core
    Image which both have fast paths through OpenGL and
    don't do excessive in-memory caching.

    Third, no mater how many thousands of images you have,
    there will never be more than few hundred or so
    visible at once.  I recommend just using NSOpenGLView
    with few hundred Open GL textures that are loaded on
    demand from disk as needed.

    Fourth, it seems literally everyone is re-writing this
    code, so search for other's experiences.

    Thumbnail creation (particularly what doesn't work) is
    discussed here:
    http://www.wilshipley.com/blog/2005/09/jpeg2000-cool-but-slow.html

    Image flow ideas with examples are here:
    http://www.cocoadev.com/index.pl?IPhotoLikeView

    Image downloading, image loading, image format
    consideration, image storage on disk, etc. are all
    discussed in various google accessible places.
  • On 08.09.2007, at 04:09, Thomas Davie wrote:

    > On 8 Sep 2007, at 02:52, Torsten Curdt wrote:
    >
    >> I am currently experimenting with a custom view that basically
    >> shows thumbnails like iPhoto does. Now while the basic drawing is
    >> still terrible inefficient (loads NSImages on every drawRect(), so
    >> it still needs caching ....and let's not talk about the memory
    >> leak (flood?) of the current code), what I am really wondering is
    >> something else:
    >>
    >> Let's assume performance is bad to draw the rects that need an
    >> update. This somehow results into a very slow feeling of the
    >> application when I use the window scroll bar. What I am wondering
    >> is how to keep that still snappy and maybe just skip a few
    >> updates. Because when I scroll the bar at a very high speed I
    >> really only car about the final position of the bar. So how would
    >> I get the drawing to be a bit more lazy. Only do the update when
    >> the speed of the bar becomes slower. Does that require drawing to
    >> happen in different thread?
    > My suggestion is not to think about ways round it like this just
    > now.  Instead, look at your app in the various profiling tools,
    > figure out what's slow, and optimize it.  You'll certainly get an
    > enormous gain by not loading the images from disk every time.
    > You'll probably get a decent gain by storing mipmaps of the images
    > and drawing an appropriately sized one.  Finally, you'll probably
    > get a decent gain by thinking about how much you actually need to
    > draw when the view scrolls.

    Sure - totally agree ...but I was wondering anyway. This was more a
    hypothetical question how one would solve this if optimization was
    just not enough. The badly written first-try code just brought up the
    question.

    cheers
    --
    Torsten
  • On 08.09.2007, at 05:28, Erik Buck wrote:

    > It seems that everybody and their brothers are
    > implementing iPhoto like views.

    What a waste of time ...isn't it? :) Maybe an indicator that it would
    be worth having a good implementation freely available to just use?
    MUPhotoView seems to be a step into that direction. On the other hand
    it might also just be an nice example for learning.

    > First, scrolling has nothing to do with zooming
    > repositioning thumbnail image display.  NSScrollView
    > transparently handles scrolling regardless of the
    > content view within the scroll view.  YES, it even
    > works when scrolling an NSOpenGLView.

    Sorry, don't undersand what you mean. I haven't touched the subject
    of zooming (yet) and of course the view only gets the request of what
    needs to be updated. Is that what you were saying?

    > Second, if you start out using NSImage and or
    > NSImageCell or NSImageView or NSMatrix, you are on the
    > wrong path.  An iPhoto like view screams "I want to be
    > implemented with Open GL".

    Ah, thanks for that pointer. I was asking a more experienced Cocoa
    guy because I was wondering whether it make sense to use CoreImage
    and he pointed me to simple NSImage.

    > You might get away with using NSBitmapImageRep or Core
    > Image which both have fast paths through OpenGL and
    > don't do excessive in-memory caching.

    OK

    > Third, no mater how many thousands of images you have,
    > there will never be more than few hundred or so
    > visible at once.  I recommend just using NSOpenGLView
    > with few hundred Open GL textures that are loaded on
    > demand from disk as needed.

    Interesting approach indeed.

    > Fourth, it seems literally everyone is re-writing this
    > code, so search for other's experiences.

    Yes, already did before. But please see my other post. This was less
    meant as a "how do I write a iPhoto scroll view" but "how do I keep
    the UI snappy if drawing takes longer". Sorry if that did not come
    across. But your comments were great either way!

    > Thumbnail creation (particularly what doesn't work) is
    > discussed here:
    > http://www.wilshipley.com/blog/2005/09/jpeg2000-cool-but-slow.html:)

    Cool

    > Image flow ideas with examples are here:
    > http://www.cocoadev.com/index.pl?IPhotoLikeView
    >
    > Image downloading, image loading, image format
    > consideration, image storage on disk, etc. are all
    > discussed in various google accessible places.

    Sure. Thanks!

    cheers
    --
    Torsten
  • On Sep 7, 2007, at 10:28 PM, Erik Buck wrote:

    > Second, if you start out using NSImage and or
    > NSImageCell or NSImageView or NSMatrix, you are on the
    > wrong path.  An iPhoto like view screams "I want to be
    > implemented with Open GL".
    >
    > You might get away with using NSBitmapImageRep or Core
    > Image which both have fast paths through OpenGL and
    > don't do excessive in-memory caching.

    I could have sworn that there was a session at WWDC a couple years
    back (maybe 2003?  2004?) that showed how one would go about such a
    thing.  It was probably one of the Cocoa performance sessions, but
    they showed different techniques for doing this, and I don't remember
    the fastest version using OpenGL at all (I believe it was all just
    using a bunch of NSImageCells) - it was just a matter of proper
    refreshing using the getRectsBeingDrawn:count: method (or maybe
    needToDrawRect:).

    Glenn Andreas                      <gandreas...>
      <http://www.gandreas.com/> wicked fun!
    quadrium2 | build, mutate, evolve, animate  | images, textures,
    fractals, art
  • I am working on an app right now that manages folders of images, some with
    tens of thousands of images.  I am creating thumbnails and allowing the user
    to work with a folder at at time.  So I have an equivalent issue of managing
    and displaying thousands of thumbnail images at a time.  I have it working
    effectively today, performance is super, here's my approach:

    I am using a custom view in an NSScrollView.  I have an array for the
    current folder that holds all the pertinent info (filename, date, size,
    database record id, and an NSImage for the thumb).  I load it all except the
    thumb when I go into a folder.  In my drawRect, when I go to paint the image
    for a file, if the thumb NSImage is nil, I load the thumbnail image from the
    database, and then paint it.  If it's there, I just paint it.  It works and
    displays very quickly.

    The problem is that you can't have several thousand NSImages around.  You'll
    run out of memory.  So, I have a "cache" scheme.  I keep count of how many I
    have loaded, and if I hit my limit (I have a preference for this, but my
    starting number is 250), I toss some of them so that I never have more than
    so many at a time.  My cache scheme is pretty modest, but still figures out
    what direction you're scrolling and disposes of the ones at the other end of
    the "window" that you're looking at.  This works fine and you can scroll
    through the folder almost as fast as you can hold the scroll control.

    Important note, making a thumb is expensive, you have to make them at most
    ONCE, not every time you need them.  It will kill your performance.  I store
    the premade thumbs in the database, don't know what you can do in your
    application, but this is a must.  I used the logic someone else in this
    thread noted to make the thumbs, but reading images off a disk and scaling
    them is hard work.  It takes about 1-2 seconds per image.  If you have even
    jut 50 in your scrollview, obviously you can't do it on the fly.

    Hope this helps, if you want more info on how I'm handling it, let me know.
    Chris

    > From: Torsten Curdt <tcurdt...>
    > Date: Sat, 8 Sep 2007 03:52:02 +0200
    > To: Mailing Cocoa List <cocoa-dev...>
    > Subject: iPhoto-like scroll view
    >
    > I am currently experimenting with a custom view that basically shows
    > thumbnails like iPhoto does. Now while the basic drawing is still
    > terrible inefficient (loads NSImages on every drawRect(), so it still
    > needs caching ....and let's not talk about the memory leak (flood?)
    > of the current code), what I am really wondering is something else:
    >
    > Let's assume performance is bad to draw the rects that need an
    > update. This somehow results into a very slow feeling of the
    > application when I use the window scroll bar. What I am wondering is
    > how to keep that still snappy and maybe just skip a few updates.
    > Because when I scroll the bar at a very high speed I really only car
    > about the final position of the bar. So how would I get the drawing
    > to be a bit more lazy. Only do the update when the speed of the bar
    > becomes slower. Does that require drawing to happen in different thread?
    >
    > Something related would be to only draw empty rects when the speed of
    > the slider becomes too high. And then switch back to a full display
    > at lower speeds. How could this be done?
    >
    > Any wise words?
    >
    > cheers
    > --
    > Torsten
  • On Sep 8, 2007, at 10:05 AM, Chris Williams wrote:
    > Important note, making a thumb is expensive, you have to make them
    > at most
    > ONCE, not every time you need them.  It will kill your
    > performance.  I store
    > the premade thumbs in the database, don't know what you can do in your
    > application, but this is a must.  I used the logic someone else in
    > this
    > thread noted to make the thumbs, but reading images off a disk and
    > scaling
    > them is hard work.  It takes about 1-2 seconds per image.  If you
    > have even
    > jut 50 in your scrollview, obviously you can't do it on the fly.

    You can get much better thumbnail creation performance by using
    CoreGraphic's thumbnail creation routine
    CGImageSourceCreateThumbnailAtIndex, since it can be smart enough to
    do things like use built in thumbnail stored in the specific image
    format, and other funky trickys like rendering a JPG at a different
    resolution (which is far faster than rendering it at the full
    resolution and scaling it down, not to mention less memory intensive).

    Glenn Andreas                      <gandreas...>
      <http://www.gandreas.com/> wicked fun!
    quadrium2 | build, mutate, evolve, animate  | images, textures,
    fractals, art
  • Been there, done that, got the t-shirt :)

    Actually, the limiting factor (for me at least) was just schleping the image
    in from the disk, and writing it back to the database.  Once I had it in
    memory, I played with all kinds of techniques to make the thumb and nothing
    made much of a difference.  So I went with one that was clean and did what I
    wanted.

    If you think about it, it makes sense.  Just copying a folder with several
    thousand images, each several hundred K in size, is a slow operation.

    From: glenn andreas <gandreas...>
    Subject: Re: iPhoto-like scroll view

    You can get much better thumbnail creation performance by using
    CoreGraphic's thumbnail creation routine
    CGImageSourceCreateThumbnailAtIndex, since it can be smart enough to
    do things like use built in thumbnail stored in the specific image
    format, and other funky trickys like rendering a JPG at a different
    resolution (which is far faster than rendering it at the full
    resolution and scaling it down, not to mention less memory intensive).
  • On Sep 8, 2007, at 10:43 AM, Chris Williams wrote:

    > Been there, done that, got the t-shirt :)
    >
    > Actually, the limiting factor (for me at least) was just schleping
    > the image
    > in from the disk, and writing it back to the database.  Once I had
    > it in
    > memory, I played with all kinds of techniques to make the thumb and
    > nothing
    > made much of a difference.  So I went with one that was clean and
    > did what I
    > wanted.
    >
    > If you think about it, it makes sense.  Just copying a folder with
    > several
    > thousand images, each several hundred K in size, is a slow operation.
    >

    That's one of the advantages of CGImageSourceCreateThumbnailAtIndex,
    though, is that in many cases it doesn't have to read the entire file
    to generate the thumbnail.  For example, many digital cameras embed a
    thumbnail in the EXIF data of a JPG file, which
    CGImageSourceCreateThumbnailAtIndex can use, allowing for it to only
    have to scan in the first couple of K of a file instead of reading
    the entire several hundred K.  Similar tricks can be done with
    progressive jpg files (again just reading the first couple of low
    resolution passes, and not the full resolution) and JPEG2000 files.

    You might want to check out <http://www.wilshipley.com/blog/2005/09/
    jpeg2000-cool-but-slow.html
    > as well which includes the following quote:
    > Followup (9/19): The news gets even better, as it turns out 10.4's
    > CGImageSourceCreateThumbnailAtIndex() is 3-7x faster at creating
    > JPEG thumbnails than the method I had to use under 10.3: creating a
    > CGImageRef using CGImageCreateWithJPEGDataProvider() and then
    > drawing it (with antialiasing turned on) into a smaller window.

    Glenn Andreas                      <gandreas...>
      <http://www.gandreas.com/> wicked fun!
    quadrium | flame : flame fractals & strange attractors : build,
    mutate, evolve, animate
  • But those embedded thumbs are almost always terrible quality (using firmware
    algorithms), they are tiny, and only exist in some formats from some
    sources.  My experience is that I never once found one I wanted to use ‹ in
    my application at least.

    However, I'll take your advice and try this (again) and time it.

    From: glenn andreas <gandreas...>
    Subject: Re: iPhoto-like scroll view

    That's one of the advantages of CGImageSourceCreateThumbnailAtIndex,
    though, is that in many cases it doesn't have to read the entire file
    to generate the thumbnail.  For example, many digital cameras embed a
    thumbnail in the EXIF data of a JPG file, which
    CGImageSourceCreateThumbnailAtIndex can use, allowing for it to only
    have to scan in the first couple of K of a file instead of reading
    the entire several hundred K.  Similar tricks can be done with
    progressive jpg files (again just reading the first couple of low
    resolution passes, and not the full resolution) and JPEG2000 files.

    You might want to check out <http://www.wilshipley.com/blog/2005/09/
    jpeg2000-cool-but-slow.html
    > as well which includes the following quote:
    > Followup (9/19): The news gets even better, as it turns out 10.4's
    > CGImageSourceCreateThumbnailAtIndex() is 3-7x faster at creating
    > JPEG thumbnails than the method I had to use under 10.3: creating a
    > CGImageRef using CGImageCreateWithJPEGDataProvider() and then
    > drawing it (with antialiasing turned on) into a smaller window.
  • On Sep 8, 2007, at 7:16 AM, glenn andreas wrote:

    >
    > On Sep 7, 2007, at 10:28 PM, Erik Buck wrote:
    >
    >> Second, if you start out using NSImage and or
    >> NSImageCell or NSImageView or NSMatrix, you are on the
    >> wrong path.  An iPhoto like view screams "I want to be
    >> implemented with Open GL".
    >>
    >> You might get away with using NSBitmapImageRep or Core
    >> Image which both have fast paths through OpenGL and
    >> don't do excessive in-memory caching.
    >
    > I could have sworn that there was a session at WWDC a couple years
    > back (maybe 2003?  2004?) that showed how one would go about such a
    > thing.  It was probably one of the Cocoa performance sessions, but
    > they showed different techniques for doing this, and I don't
    > remember the fastest version using OpenGL at all (I believe it was
    > all just using a bunch of NSImageCells) - it was just a matter of
    > proper refreshing using the getRectsBeingDrawn:count: method (or
    > maybe needToDrawRect:).

    Frankly, if the OpenGL way wasn't fastest, this just means that the
    guy who wrote the GL version made a mistake somewhere :) OpenGL is
    going to run circles around any CPU-based technique unless you are on
    a GPU from the stone ages, or your OpenGL code is very naively written.
  • On 08.09.2007, at 18:59, John Stiles wrote:
    > Frankly, if the OpenGL way wasn't fastest, this just means that the
    > guy who wrote the GL version made a mistake somewhere :) OpenGL is
    > going to run circles around any CPU-based technique unless you are
    > on a GPU from the stone ages, or your OpenGL code is very naively
    > written.

      The problem is not the actual speed of blitting your images to the
    screen. The part where you lose most time when drawing thumbnails is
    actually generating them and loading them from disk before that.
    Others have already mentioned the APIs available to speed that up,
    but I've found that if you take care to only redraw the parts you
    actually need to draw, and avoid unnecessary scaling through
    cacheing, the actual screen display is by far fast enough even with
    Quartz.

    Cheers,
    -- M. Uli Kusterer
    http://www.zathras.de
previous month september 2007 next month
MTWTFSS
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
Go to today