Memory not being released in a timely manner

  • Hi all,

    Can anybody advise what tools I should be using to debug a particular out of memory condition - and maybe even how I can fix it? The error that I ultimately encounter (in a 32-bit application on Snow Leopard) is:
    2013-06-03 15:44:30.271 MyApplication[25115:a0f] NSImage: Insufficient memory to allocate pixel data buffer of 1228800 bytes
    However as I'll explain I don't feel that I am doing anything that should result in running out of memory.

    The program loads and processes a large number (~2000) image files, and I believe that I am disposing of memory and draining autorelease pools correctly. Running the program under ObjectAlloc, it never reports memory usage over 300MB. The problem seems to be that, even though ObjectAlloc thinks the memory has been released, it is not actually being properly freed up until some time later. The memory usage as reported in Activity Monitor climbs higher and higher until, if left unattended, there is apparently no more memory available. Clicking the mouse or switching applications causes an immediate and significant drop in memory usage.

    Thus the situation seems to be that I believe I am doing everything I can for good memory management, but the OS is not actually freeing things up for re-use. I encountered a similar problem that I asked about on here last year:
    http://lists.apple.com/archives/cocoa-dev/2012/Jul/msg00602.html
    Although in that case the eventual conclusion was that I should be using a different approach in my code, it involved the same issue of memory not being released when I would have expected it to be. In that case from last year, I was wrong to rely on it being released when I expected, but I feel that this time it should be reasonable to expect that I won't run out of memory entirely, given that I am releasing all the buffers that I am finished with and only keeping a small working set allocated!

    This is of course partly just speculation because ObjectAlloc isn't giving any info about this unavailable but apparently not-allocated memory. So...
    1. Can anybody recommend a way of debugging this problem to get more concrete evidence for what is actually happening?
    2. Assuming my interpretation is correct, can anybody suggest a solution?

    Many thanks
    Jonny
  • Hi Johnny.

    This is a long-standing problem with AppKit. According to the documentation, "The Application Kit creates an autorelease pool on the main thread at the beginning of every cycle of the event loop, and drains it at the end, thereby releasing any autoreleased objects generated while processing an event." However, this is somewhat misleading. The "end" of the event loop cycle is immediately before the beginning. Thus, for example, if your app is in the background and not receiving events, then the autorelease pool will not be drained. That's why your memory drops significantly when you click the mouse or switch applications.

    We've run into this issue in a number of apps. As a workaround, we add a timer to the main thread and have it fire periodically. The timer's action method does the following:

    NSEvent *event = [NSEvent otherEventWithType:NSApplicationDefined location:NSZeroPoint modifierFlags:0 timestamp:[NSDate timeIntervalSinceReferenceDate] windowNumber:0 context:nil subtype:0 data1:0 data2:0]
    [NSApp postEvent:event atStart:YES];

    This basically "tickles" the event loop and causes the autorelease pool to drain.

    -Jeff

    On Jun 3, 2013, at 10:59 AM, Jonathan Taylor <Jonathan.Taylor...> wrote:

    > Hi all,
    >
    > Can anybody advise what tools I should be using to debug a particular out of memory condition - and maybe even how I can fix it? The error that I ultimately encounter (in a 32-bit application on Snow Leopard) is:
    > 2013-06-03 15:44:30.271 MyApplication[25115:a0f] NSImage: Insufficient memory to allocate pixel data buffer of 1228800 bytes
    > However as I'll explain I don't feel that I am doing anything that should result in running out of memory.
    >
    > The program loads and processes a large number (~2000) image files, and I believe that I am disposing of memory and draining autorelease pools correctly. Running the program under ObjectAlloc, it never reports memory usage over 300MB. The problem seems to be that, even though ObjectAlloc thinks the memory has been released, it is not actually being properly freed up until some time later. The memory usage as reported in Activity Monitor climbs higher and higher until, if left unattended, there is apparently no more memory available. Clicking the mouse or switching applications causes an immediate and significant drop in memory usage.
    >
    > Thus the situation seems to be that I believe I am doing everything I can for good memory management, but the OS is not actually freeing things up for re-use. I encountered a similar problem that I asked about on here last year:
    > http://lists.apple.com/archives/cocoa-dev/2012/Jul/msg00602.html
    > Although in that case the eventual conclusion was that I should be using a different approach in my code, it involved the same issue of memory not being released when I would have expected it to be. In that case from last year, I was wrong to rely on it being released when I expected, but I feel that this time it should be reasonable to expect that I won't run out of memory entirely, given that I am releasing all the buffers that I am finished with and only keeping a small working set allocated!
    >
    > This is of course partly just speculation because ObjectAlloc isn't giving any info about this unavailable but apparently not-allocated memory. So...
    > 1. Can anybody recommend a way of debugging this problem to get more concrete evidence for what is actually happening?
    > 2. Assuming my interpretation is correct, can anybody suggest a solution?
    >
    > Many thanks
    > Jonny
  • On Jun 3, 2013, at 8:59 AM, Jonathan Taylor wrote:

    > ...The memory usage as reported in Activity Monitor climbs higher and higher until, if left unattended, there is apparently no more memory available. Clicking the mouse or switching applications causes an immediate and significant drop in memory usage….

    I've seen this problem with GCD's handling of autorelease pools. It sounds very similar to what Jeff said, but I've only experienced it on 10.6, with 10.7+ behaving as desired.

    --
    Seth Willits
  • Hi Jeff,
    Thanks very much for your reply. That's brilliant. I had tried playing around with a few "tricks" I thought might prompt a pool drain, but to no joy. It's great to have a bit of code that will do the job for that - I'll give it a go tomorrow.
    Cheers
    Jonny

    On 4 Jun 2013, at 16:30, Jeff Johnson <publicposting...> wrote:

    > Hi Johnny.
    >
    > This is a long-standing problem with AppKit. According to the documentation, "The Application Kit creates an autorelease pool on the main thread at the beginning of every cycle of the event loop, and drains it at the end, thereby releasing any autoreleased objects generated while processing an event." However, this is somewhat misleading. The "end" of the event loop cycle is immediately before the beginning. Thus, for example, if your app is in the background and not receiving events, then the autorelease pool will not be drained. That's why your memory drops significantly when you click the mouse or switch applications.
    >
    > We've run into this issue in a number of apps. As a workaround, we add a timer to the main thread and have it fire periodically. The timer's action method does the following:
    >
    > NSEvent *event = [NSEvent otherEventWithType:NSApplicationDefined location:NSZeroPoint modifierFlags:0 timestamp:[NSDate timeIntervalSinceReferenceDate] windowNumber:0 context:nil subtype:0 data1:0 data2:0]
    > [NSApp postEvent:event atStart:YES];
    >
    > This basically "tickles" the event loop and causes the autorelease pool to drain.
    >
    > -Jeff
    >
    >
    > On Jun 3, 2013, at 10:59 AM, Jonathan Taylor <Jonathan.Taylor...> wrote:
    >
    >> Hi all,
    >>
    >> Can anybody advise what tools I should be using to debug a particular out of memory condition - and maybe even how I can fix it? The error that I ultimately encounter (in a 32-bit application on Snow Leopard) is:
    >> 2013-06-03 15:44:30.271 MyApplication[25115:a0f] NSImage: Insufficient memory to allocate pixel data buffer of 1228800 bytes
    >> However as I'll explain I don't feel that I am doing anything that should result in running out of memory.
    >>
    >> The program loads and processes a large number (~2000) image files, and I believe that I am disposing of memory and draining autorelease pools correctly. Running the program under ObjectAlloc, it never reports memory usage over 300MB. The problem seems to be that, even though ObjectAlloc thinks the memory has been released, it is not actually being properly freed up until some time later. The memory usage as reported in Activity Monitor climbs higher and higher until, if left unattended, there is apparently no more memory available. Clicking the mouse or switching applications causes an immediate and significant drop in memory usage.
    >>
    >> Thus the situation seems to be that I believe I am doing everything I can for good memory management, but the OS is not actually freeing things up for re-use. I encountered a similar problem that I asked about on here last year:
    >> http://lists.apple.com/archives/cocoa-dev/2012/Jul/msg00602.html
    >> Although in that case the eventual conclusion was that I should be using a different approach in my code, it involved the same issue of memory not being released when I would have expected it to be. In that case from last year, I was wrong to rely on it being released when I expected, but I feel that this time it should be reasonable to expect that I won't run out of memory entirely, given that I am releasing all the buffers that I am finished with and only keeping a small working set allocated!
    >>
    >> This is of course partly just speculation because ObjectAlloc isn't giving any info about this unavailable but apparently not-allocated memory. So...
    >> 1. Can anybody recommend a way of debugging this problem to get more concrete evidence for what is actually happening?
    >> 2. Assuming my interpretation is correct, can anybody suggest a solution?
    >>
    >> Many thanks
    >> Jonny
    >
  • On Jun 3, 2013, at 8:59 AM, Jonathan Taylor <Jonathan.Taylor...> wrote:

    > The program loads and processes a large number (~2000) image files, and I believe that I am disposing of memory and draining autorelease pools correctly. Running the program under ObjectAlloc, it never reports memory usage over 300MB.

    I don’t know if ObjectAlloc is supported anymore; you should be using Instruments.

    It’s been a long time since I used that app, so I don’t know if it reports all heap allocations or just Obj-C objects. The problem is that a small object (like an NSImageRep) can be hanging onto huge memory buffers for pixels.

    > The problem seems to be that, even though ObjectAlloc thinks the memory has been released, it is not actually being properly freed up until some time later. The memory usage as reported in Activity Monitor climbs higher and higher until, if left unattended, there is apparently no more memory available.

    There are a lot of different numbers measuring different kinds of memory usage, and not all of them are relevant to this. What specific value in Activity Monitor are you talking about? The most relevant one is Private Mem (RPRVT).

    —Jens
  • On Jun 4, 2013, at 8:30 AM, Jeff Johnson <publicposting...> wrote:

    > The "end" of the event loop cycle is immediately before the beginning. Thus, for example, if your app is in the background and not receiving events, then the autorelease pool will not be drained.

    You can work around this by wrapping your own autorelease pools around code that allocates significant amounts of memory, rather than relying on the runloop’s outer autorelease pool to free it for you.

    —Jens
  • On Jun 4, 2013, at 12:35 PM, Jens Alfke <jens...> wrote:

    > On Jun 4, 2013, at 8:30 AM, Jeff Johnson <publicposting...> wrote:
    >
    >> The "end" of the event loop cycle is immediately before the beginning. Thus, for example, if your app is in the background and not receiving events, then the autorelease pool will not be drained.
    >
    > You can work around this by wrapping your own autorelease pools around code that allocates significant amounts of memory, rather than relying on the runloop’s outer autorelease pool to free it for you.
    >
    > —Jens

    In some cases, yes, although the architecture sometimes makes that difficult. And even code that allocates only a small amount of memory can become very significant if it is called repeatedly over the course of the app's running for days or weeks at a time.

    -Jeff
  • On 06/06/2013, at 11:03 AM, Jeff Johnson <publicposting...> wrote:

    > In some cases, yes, although the architecture sometimes makes that difficult.

    Can you give an example?

    > And even code that allocates only a small amount of memory can become very significant if it is called repeatedly over the course of the app's running for days or weeks at a time.

    That would surely be a leak? That's quite different from just being slow to autorelease something. An app that runs for weeks without once spinning its run loop? And if that's the case, it's your responsibility to use autorelease pools correctly.

    --Graham
  • Thankyou all for your replies. A few responses -

    On 4 Jun 2013, at 16:30, Jeff Johnson wrote:
    > We've run into this issue in a number of apps. As a workaround, we add a timer to the main thread and have it fire periodically. The timer's action method does the following:

    Thanks again! I've implemented your code and that has "fixed" my problem - great news.

    On 4 Jun 2013, at 18:33, Jens Alfke wrote:
    > I don’t know if ObjectAlloc is supported anymore; you should be using Instruments.
    >
    > It’s been a long time since I used that app, so I don’t know if it reports all heap allocations or just Obj-C objects. The problem is that a small object (like an NSImageRep) can be hanging onto huge memory buffers for pixels.

    Apologies, I meant the Allocations tool within Instruments. [Incidentally, in my install ObjectAlloc just launches Instruments/Allocations]

    As far as I can see, the Allocations tool does not report memory that is pending release from an autorelease pool. It would be interesting to know if there was a way of monitoring that. Re your suggestion about wrapping code with my own autorelease pools, I have done so but I am pretty sure the movie generation APIs that I am using launch their own threads, ones that I don't have control over. Without being able to see exactly what allocations are still pending release, I can't be sure, but I am as certain as I can be that that's where the memory-pending-autorelease must be accumulating.

    >> The problem seems to be that, even though ObjectAlloc thinks the memory has been released, it is not actually being properly freed up until some time later. The memory usage as reported in Activity Monitor climbs higher and higher until, if left unattended, there is apparently no more memory available.
    >
    > There are a lot of different numbers measuring different kinds of memory usage, and not all of them are relevant to this. What specific value in Activity Monitor are you talking about? The most relevant one is Private Mem (RPRVT).

    Yep, that's the one. (I was trying to keep my original email reasonably concise!)

    Cheers
    Jonny.
  • On Jun 6, 2013, at 4:41 AM, Jonathan Taylor <Jonathan.Taylor...> wrote:
    > As far as I can see, the Allocations tool does not report memory that is pending release from an autorelease pool. It would be interesting to know if there was a way of monitoring that. Re your suggestion about wrapping code with my own autorelease pools, I have done so but I am pretty sure the movie generation APIs that I am using launch their own threads, ones that I don't have control over. Without being able to see exactly what allocations are still pending release, I can't be sure, but I am as certain as I can be that that's where the memory-pending-autorelease must be accumulating.

    The Allocations instrument should report objects with pending autoreleases as ordinary live objects. (Note that many objects with retain count == pending autorelease count will be retained again before the autorelease pool pops.)

    In OS X 10.8 and iOS 6 simulator, you can set environment variable OBJC_PRINT_POOL_HIGHWATER=YES to get debugging logs of the autorelease pool high-water mark on each thread. This can detect code that accumulates lots of autorelease garbage without spinning any pools. (The high-water mark is not checked until the pool is popped, so you'd have to actually finish the work to see the result.)

    --
    Greg Parker    <gpakrer...>    Runtime Wrangler
  • > The Allocations instrument should report objects with pending autoreleases as ordinary live objects. (Note that many objects with retain count == pending autorelease count will be retained again before the autorelease pool pops.)
    >
    > In OS X 10.8 and iOS 6 simulator, you can set environment variable OBJC_PRINT_POOL_HIGHWATER=YES to get debugging logs of the autorelease pool high-water mark on each thread. This can detect code that accumulates lots of autorelease garbage without spinning any pools. (The high-water mark is not checked until the pool is popped, so you'd have to actually finish the work to see the result.)

    Thanks for your comments Greg, very interesting. Unfortunately (in a sense!) the problem doesn't seem to happen on 10.8, so I can't reproduce the problem and get those debug logs.

    > "The Allocations instrument should report objects with pending autoreleases as ordinary live objects"

    Do you happen to know if that's true of Allocations on 10.6? Is that definitely true for objects that have actually passed out of scope, and it would therefore be forbidden to re-retain them (if you know what I mean), but which haven't actually been freed "behind the scenes"?

    I'm not seeing Allocations report large amounts of memory. I assume that it should be reporting anything that is present within my 32-bit address space (i.e. I assume there isn't any way that external libraries could allocate memory that is in some way hidden from Allocations)? If that is the case, I suppose the only other possibility I can think of is that I am fragmenting the address space to such an extent that there is no way of allocating any further buffers (size of order 1MB).

    Currently my understanding of the problem is pretty much limited to:
    - Allocations claims only 300MB allocated memory
    - 1.2MB buffer cannot be allocated ("insufficient memory")
    - Occurs when program is in background, and "tickling" autorelease pools at 1 second intervals makes the problem go away.
    - Occurs on 10.6, not on 10.8

    I do have a solution in the form of the "tickling", but I'd be very interested if you had any suggestions on ways I could dig further into the underlying cause.

    Cheers
    Jonny
  • On 7 Jun 2013, at 6:05 AM, Jonathan Taylor <Jonathan.Taylor...> wrote:

    > [Greg Parker:]
    >> "The Allocations instrument should report objects with pending autoreleases as ordinary live objects"
    >
    > Do you happen to know if that's true of Allocations on 10.6? Is that definitely true for objects that have actually passed out of scope, and it would therefore be forbidden to re-retain them (if you know what I mean), but which haven't actually been freed "behind the scenes"?

    Greg Parker's lightest word would annihilate even my deepest thoughts, but this is my no-inside-information understanding of the semantics of the autorelease pool:

    It's just a broker with no insight into the state of the objects it holds. It does a retain when an object is presented, counts the retains it does, and does N releases when the pool is drained. Nothing special, and no way to decide whether all of the ownership count relates to what may be a stack of autorelease pools.

    There are optimizations — ARC includes a lot of them — but people have relied on those semantics for decades, and there's not much scope to change them. It has to act-as-if.

    If you already can pick out the object you're interested in in the Allocations history, and you turn retain tracking on, you can study the retain count and see how many times the object has been autoreleased. But Instruments has no way of flagging whether a retain pattern is other than what you expect/intend.

    The out-of-scope compile-time condition you suggest would be very hard to detect reliably. Clang may be insightful enough that it could instrument the object code to give the Allocations instrument clues, but that would be the next OS at the earliest, and doesn't help you on Snow Leopard.

    I have no clue about your root problem, which is bizarre according to what you've told us. My approach to such things is to blunder around until something suggests itself. It smells like you're doing a lot of processing with temporary objects, in a loop, without bracketing the loop body in @autoreleasepool{}, but I remember you saying you're not.

    [I remember vaguely that CG operates its own heap for image buffers, which is what your error message complains about. I dealt with it a long time ago, and I may be misremembering. What does the VM Tracker instrument say?]

    — F

    --
    Fritz Anderson
    Xcode 4 Unleashed: 4.5 supplement for free!
    http://www.informit.com/store/xcode-4-unleashed-9780672333279
  • > It smells like you're doing a lot of processing with temporary objects, in a loop, without bracketing the loop body in @autoreleasepool{}, but I remember you saying you're not.

    Oh dear. Oh dear.

    You are right. I know this is what everybody has been telling me all along. I have a tiny one-line GCD block that is re-posted back onto the main event loop (for thread safety reasons), which I had overlooked. It was setting off a chain reaction of binding notifications etc that resulted in the image as displayed in the GUI being updated. An autoreleased image buffer involved in this wasn't being caught by an explicit pool, and so was falling foul of the problem described by Jeff Johnson whereby autoreleased objects weren't being cleaned up in a timely manner.

    So, wrapping that single line of code in an autorelease pool has fixed it. I'm so sorry for taxing everyone's patience on this one!

    Your suggestion of the VM Tracker instrument (which I had not spotted before) did bring up some interesting results though. These autoreleased image buffers that were causing the problem are definitely NOT reported at all by Allocations ("live" bytes stays stable at 250MB), but they do show up under VM Tracker as "CG image" and "CG raster data" - although I don't think there's a way of getting any further details about the buffers they provide the backing for?

    I do think it's interesting though (and a bit worrying) that the only way I could pinpoint the actual problem was by reading through the relevant bits of my code over and over - I wasn't able to glean any info from Instruments that really narrowed things down, other than to confirm that there were definitely image buffers accumulating somewhere.

    Jonny.
  • On 7 Jun 2013, at 11:44 AM, Jonathan Taylor <Jonathan.Taylor...> wrote:

    > I do think it's interesting though (and a bit worrying) that the only way I could pinpoint the actual problem was by reading through the relevant bits of my code over and over - I wasn't able to glean any info from Instruments that really narrowed things down, other than to confirm that there were definitely image buffers accumulating somewhere.

    This is the tragedy of the developer's existence. New tools make the day-to-day stuff much easier, but sometimes you must back off to 1970-era techniques, and audit your code. Tragedy never leaves us.

    <http://www.joelonsoftware.com/articles/LeakyAbstractions.html>

    --
    Fritz Anderson
    Xcode 4 Unleashed: 4.5 supplement for free!
    http://www.informit.com/store/xcode-4-unleashed-9780672333279
  • On Jun 7, 2013, at 9:44 AM, Jonathan Taylor <Jonathan.Taylor...> wrote:

    > Your suggestion of the VM Tracker instrument (which I had not spotted before) did bring up some interesting results though. These autoreleased image buffers that were causing the problem are definitely NOT reported at all by Allocations ("live" bytes stays stable at 250MB), but they do show up under VM Tracker as "CG image" and "CG raster data" - although I don't think there's a way of getting any further details about the buffers they provide the backing for?

    Those sorts of buffers are probably magical at the kernel level — one of the jobs of CG/Quartz is to shuffle pixmaps between CPU and GPU memory, and doing that efficiently is important to overall graphics performance. I wouldn’t be surprised if there were special types of memory pages that were optimized for this, that don’t live in normal address space.

    I just took a look at the Quartz Debug app and didn’t find any window for viewing memory usage, but someone might know of other tools. Sounds like it would be a good idea to add that to the regular memory-tracking instruments.

    —Jens
  • On Jun 7, 2013, at 11:44 AM, Jonathan Taylor wrote:

    > So, wrapping that single line of code in an autorelease pool has fixed it.

    > I do think it's interesting though (and a bit worrying) that the only way I could pinpoint the actual problem was by reading through the relevant bits of my code over and over - I wasn't able to glean any info from Instruments that really narrowed things down, other than to confirm that there were definitely image buffers accumulating somewhere.

    These two statements contradict each other.  The large chunks of memory were not objects or even malloc blocks (probably obtained using vm_allocate() or the like), but they were owned by regular objects.  Those regular objects were still alive and keeping the large chunks around.  The fact that the use of an autorelease pool fixed the problem is proof of that.

    So, the Allocations instrument should have been showing you those living objects.  True, it couldn't show you the size of memory ultimately dependent on those objects, but that's often the case (i.e. a small object that owns a large NSData or something; Allocations would sort the NSData to the top but you'd have to hunt to find the small object that owns it).

    You might have had better luck if you had taken heapshots between two points in your app's lifetime and observed what objects had been created after the first and still living at the second.
    http://www.friday.com/bbum/2010/10/17/when-is-a-leak-not-a-leak-using-heaps
    hot-analysis-to-find-undesirable-memory-growth/


    Regards,
    Ken
  • Thanks for your comments Ken. It was really good to learn about how to use heapshots effectively, that's definitely something I'm going to use again in future. In this case it did provide ~some~ more information on what is going on.

    Just to be clear, the problem is now solved by wrapping the correct bit of code in an explicit autorelease pool, but I'm still very interested in understanding what's going on and why Allocations is not reporting large chunks of memory - I still maintain that this is not happening. If I understood your reply correctly, you're saying this isn't what you would expect. The following results are with the explicit autorelease pool (bug fix for my issue) disabled so that the problem manifests itself.

    With the use of heapshots, it does turn out there are in fact ~small~ objects accumulating that are reported by Allocations. An example screenshot can be viewed here:
    https://docs.google.com/file/d/0Bye8FKbpg3dYRWF0YWo1djNmZVU/edit?usp=sharin
    g

    However this is only reporting relatively small amounts of memory still outstanding from during the image processing run. The fact that there are Image/Bitmap objects in there should ring alarm bells, and watching one of those objects would have given me a very strong clue as to what was going wrong, had I done it earlier. For example:
    https://docs.google.com/file/d/0Bye8FKbpg3dYcnlQRWt0ZDlVdjA/edit?usp=sharin
    g

    is pretty strong evidence that it's a delay in autorelease pool drain that is causing all this memory to sit around.

    However, if I understand you correctly, you are convinced that the (large) backing buffer for the bitmap data should be being reported by Allocations. As you can hopefully see from these screenshots, while the NS objects themselves are being reported as live, the backing memory itself really doesn't seem to be showing up in Allocations (though the VM tracker knows about it. When you wrote:

    > So, the Allocations instrument should have been showing you those living objects.  True, it couldn't show you the size of memory ultimately dependent on those objects, but that's often the case (i.e. a small object that owns a large NSData or something; Allocations would sort the NSData to the top but you'd have to hunt to find the small object that owns it).

    ... it seems that my observations match that in terms of the top-level objects, but there really doesn't seem to be any sign at all of the backing memory. Weird.

    Cheers
    Jonny
  • On Jun 10, 2013, at 08:23 , Jonathan Taylor <Jonathan.Taylor...> wrote:

    > However, if I understand you correctly, you are convinced that the (large) backing buffer for the bitmap data should be being reported by Allocations. As you can hopefully see from these screenshots, while the NS objects themselves are being reported as live, the backing memory itself really doesn't seem to be showing up in Allocations (though the VM tracker knows about it.

    It's quite possible that the image files are being mapped into memory, rather than being read into allocated buffers. That would produce the results you've described (an increase in VM usage, but no big allocation footprint).
  • On Jun 10, 2013, at 10:23 AM, Jonathan Taylor wrote:

    > Thanks for your comments Ken.

    You're welcome.

    > It was really good to learn about how to use heapshots effectively, that's definitely something I'm going to use again in future. In this case it did provide ~some~ more information on what is going on.
    >
    > Just to be clear, the problem is now solved by wrapping the correct bit of code in an explicit autorelease pool, but I'm still very interested in understanding what's going on and why Allocations is not reporting large chunks of memory - I still maintain that this is not happening. If I understood your reply correctly, you're saying this isn't what you would expect.

    > However, if I understand you correctly, you are convinced that the (large) backing buffer for the bitmap data should be being reported by Allocations.

    No, you've misunderstood.  I was suggesting that there were large allocations that don't show up as objects.  My theory was that they were not allocated via malloc but by something like vm_allocate(), which the Allocations instrument doesn't track.

    Regards,
    Ken
  • On Jun 10, 2013, at 9:15 AM, Quincey Morris <quinceymorris...> wrote:

    > It's quite possible that the image files are being mapped into memory, rather than being read into allocated buffers. That would produce the results you've described (an increase in VM usage, but no big allocation footprint).

    I don’t think r/o memory-mapped address space shows up under RPRVT. (It’s not ‘private’ — any other process that mmap’ed the same file would be sharing that address space.)

    —Jens
  • On Jun 7, 2013, at 9:44 AM, Jonathan Taylor <Jonathan.Taylor...> wrote:
    > Your suggestion of the VM Tracker instrument (which I had not spotted before) did bring up some interesting results though. These autoreleased image buffers that were causing the problem are definitely NOT reported at all by Allocations ("live" bytes stays stable at 250MB), but they do show up under VM Tracker as "CG image" and "CG raster data" - although I don't think there's a way of getting any further details about the buffers they provide the backing for?

    You're probably seeing something like this:
    1. Some code runs a loop that generates autorelease garbage without spinning any autorelease pools.
    2. Some of the autoreleased but not deallocated objects are NSImage objects.
    3. The NSImage objects allocate their pixel data using something other than malloc(). For example, the memory is allocated and shared with the WindowServer process using the Mach memory machinery.
    4. The NSImage objects are visible in the Allocations instrument, but they are small and hard to see. The lost memory is mostly the NSImage pixel data which is visible only to the VM Tracker instrument.

    --
    Greg Parker    <gparker...>    Runtime Wrangler
previous month june 2013 next month
MTWTFSS
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
Go to today