Triggering an NSTextStorage Bug

  • This has taken me a *very long time* to narrow down to its simplest bits and it's very puzzling to me.

    The short story is I have an NSTextStorage subclass as I'm doing many different things, syntax coloring among them. I would swear that everything I'm doing is a legitimate implementation, but there is obviously something causing internal behavior in the text system to go haywire. The result is that the text system itself (not me!) is over-releasing the NSParagraphStyle. The memory management in my code is solid, so the problem has to be in the text system itself.

    The example project:
    http://www.sethwillits.com/temp/PGSOverRelease.zip

    There are a number of factors involved in reliably reproducing this behavior and this project has one set of circumstances that will reproduce it, but there are others. If you slightly change any of these factors (all changes which should have no effect), then the over-release crash does not occur.

    I have read and re-read all of the text storage documentation multiple times and I am thusly confident that either some critical elusive behavior is undocumented or I've found a legitimate bug in the text system.

    Any help would be greatly appreciated. This is severely impacting my work at the moment.

    --
    Seth Willits
  • I wonder if init is too early to be setting the default paragraph style. When I changed that line in your setup method to the following, it no longer crashed on launch.

        [self performSelector:@selector(setDefaultParagraphStyle:)
                  withObject:[NSParagraphStyle defaultParagraphStyle]
                  afterDelay:0];

    --Andy

    On Mar 19, 2013, at 7:54 PM, Seth Willits <slists...> wrote:

    >
    > This has taken me a *very long time* to narrow down to its simplest bits and it's very puzzling to me.
    >
    >
    > The short story is I have an NSTextStorage subclass as I'm doing many different things, syntax coloring among them. I would swear that everything I'm doing is a legitimate implementation, but there is obviously something causing internal behavior in the text system to go haywire. The result is that the text system itself (not me!) is over-releasing the NSParagraphStyle. The memory management in my code is solid, so the problem has to be in the text system itself.
    >
    > The example project:
    > http://www.sethwillits.com/temp/PGSOverRelease.zip
    >
    > There are a number of factors involved in reliably reproducing this behavior and this project has one set of circumstances that will reproduce it, but there are others. If you slightly change any of these factors (all changes which should have no effect), then the over-release crash does not occur.
    >
    > I have read and re-read all of the text storage documentation multiple times and I am thusly confident that either some critical elusive behavior is undocumented or I've found a legitimate bug in the text system.
    >
    >
    > Any help would be greatly appreciated. This is severely impacting my work at the moment.
    >
    >
    > --
    > Seth Willits
  • Bizarrely, running your sample project in under the Zombies Instrument
    produces no crash, but it _does_ spin the pinwheel forever during
    startup, creating infinitely many instances of NSDictionary.

    --Kyle Sluder

    On Tue, Mar 19, 2013, at 04:54 PM, Seth Willits wrote:
    >
    > This has taken me a *very long time* to narrow down to its simplest bits
    > and it's very puzzling to me.
    >
    >
    > The short story is I have an NSTextStorage subclass as I'm doing many
    > different things, syntax coloring among them. I would swear that
    > everything I'm doing is a legitimate implementation, but there is
    > obviously something causing internal behavior in the text system to go
    > haywire. The result is that the text system itself (not me!) is
    > over-releasing the NSParagraphStyle. The memory management in my code is
    > solid, so the problem has to be in the text system itself.
    >
    > The example project:
    > http://www.sethwillits.com/temp/PGSOverRelease.zip
    >
    > There are a number of factors involved in reliably reproducing this
    > behavior and this project has one set of circumstances that will
    > reproduce it, but there are others. If you slightly change any of these
    > factors (all changes which should have no effect), then the over-release
    > crash does not occur.
    >
    > I have read and re-read all of the text storage documentation multiple
    > times and I am thusly confident that either some critical elusive
    > behavior is undocumented or I've found a legitimate bug in the text
    > system.
    >
    >
    > Any help would be greatly appreciated. This is severely impacting my work
    > at the moment.
    >
    >
    > --
    > Seth Willits
  • I haven't tried with Instruments, just NSZombieEnabled. I take it you do see the crash without Instruments?

    --
    Seth Willits

    On Mar 19, 2013, at 6:09 PM, Kyle Sluder wrote:

    > Bizarrely, running your sample project in under the Zombies Instrument
    > produces no crash, but it _does_ spin the pinwheel forever during
    > startup, creating infinitely many instances of NSDictionary.
    >
    > --Kyle Sluder
    >
    > On Tue, Mar 19, 2013, at 04:54 PM, Seth Willits wrote:
    >>
    >> This has taken me a *very long time* to narrow down to its simplest bits
    >> and it's very puzzling to me.
    >>
    >>
    >> The short story is I have an NSTextStorage subclass as I'm doing many
    >> different things, syntax coloring among them. I would swear that
    >> everything I'm doing is a legitimate implementation, but there is
    >> obviously something causing internal behavior in the text system to go
    >> haywire. The result is that the text system itself (not me!) is
    >> over-releasing the NSParagraphStyle. The memory management in my code is
    >> solid, so the problem has to be in the text system itself.
    >>
    >> The example project:
    >> http://www.sethwillits.com/temp/PGSOverRelease.zip
    >>
    >> There are a number of factors involved in reliably reproducing this
    >> behavior and this project has one set of circumstances that will
    >> reproduce it, but there are others. If you slightly change any of these
    >> factors (all changes which should have no effect), then the over-release
    >> crash does not occur.
    >>
    >> I have read and re-read all of the text storage documentation multiple
    >> times and I am thusly confident that either some critical elusive
    >> behavior is undocumented or I've found a legitimate bug in the text
    >> system.
    >>
    >>
    >> Any help would be greatly appreciated. This is severely impacting my work
    >> at the moment.
    >>
    >>
    >> --
    >> Seth Willits
  • On Tue, Mar 19, 2013, at 09:36 PM, Seth Willits wrote:
    > I haven't tried with Instruments, just NSZombieEnabled. I take it you do
    > see the crash without Instruments?

    Yes, I do. And as Andy pointed out, it eventually does crash in
    Instruments, but holy cow does it create a lot of NSDictionaries!!

    --Kyle Sluder
  • On Mar 19, 2013, at 9:44 PM, Kyle Sluder wrote:

    > On Tue, Mar 19, 2013, at 09:36 PM, Seth Willits wrote:
    >> I haven't tried with Instruments, just NSZombieEnabled. I take it you do
    >> see the crash without Instruments?
    >
    > Yes, I do. And as Andy pointed out, it eventually does crash in
    > Instruments, but holy cow does it create a lot of NSDictionaries!!

    Yes, the performance in this example is terrible, but it shouldn't be relevant. In the real-world there's a cached dictionary for each token type so it reuses that unless it can tell it should be different for some reason. Either way it still crashes, but I figured having the cache would be "suspect code" so I just removed it.

    On Mar 19, 2013, at 5:31 PM, Andy Lee wrote:

    > I wonder if init is too early to be setting the default paragraph style. When I changed that line in your setup method to the following, it no longer crashed on launch.
    >
    > [self performSelector:@selector(setDefaultParagraphStyle:)
    > withObject:[NSParagraphStyle defaultParagraphStyle]
    > afterDelay:0];

    I think rather than it being a case of -init being too early, it's that avoiding it in this circumstance avoids the crash. I've encountered the same crash at various times — field will load with text but deleting it and pasting new text will cause the crash, or the field loads from the nib fine but not manually,  it works with methods A B and C but not B C and D…  There are many ways to avoid the crash, but fundamentally I'm sure there's a problem still lurking, it would just be triggered later on.

    --
    Seth Willits
  • Long story short, it seems this may be caused by the retainCount value on the paragraph style object wrapping around because there are so many references to it. According to the header, it only allows for 2^19 retains (524,288), which you can see in NSParagraphStyle.h (unsigned int refCount:19;) My logging confirms that a wrap around is occurring, so I think that may be to blame.

    So now I have to spend a few hours starting back in my production code, and par it down again to figure out where all of these zillion references are coming from, since in this example it's obvious but in my production code it's not.

    Either way though, it seems in a huge document with a zillion little ranges it could easily and legitimately hit this retainCount limit. Seems kinda odd that it's limited to 19 bits.

    --
    Seth Willits
  • On Wed, Mar 20, 2013, at 11:48 AM, Seth Willits wrote:
    > Either way though, it seems in a huge document with a zillion little
    > ranges it could easily and legitimately hit this retainCount limit. Seems
    > kinda odd that it's limited to 19 bits.

    Ha!

    The reason it's limited to 19 bits is because for speed purposes
    NSParagraphStyle implements its own inline retain count rather than
    relying on NSObject's external refcount table. Which is also why you
    can't make ARC weak references to NSParagraphStyle instances. ;-)

    --Kyle Sluder
  • On Mar 20, 2013, at 12:01 PM, Kyle Sluder <kyle...> wrote:
    > The reason it's limited to 19 bits is because for speed purposes
    > NSParagraphStyle implements its own inline retain count rather than
    > relying on NSObject's external refcount table. Which is also why you
    > can't make ARC weak references to NSParagraphStyle instances. ;-)

    Yikes. Usually the inline bits are used for the *low order* bits of the retain count. That is, the full retain count is something like (internal_refs + (external_refs << internal_width). But in this case NSParagraphStyle does only seem to be using the internal bits:

    #import <Cocoa/Cocoa.h>

    int main(int argc, char *argv[])
    {
        NSParagraphStyle *pg = [NSParagraphStyle defaultParagraphStyle];
        for (;;) {
            for (NSUInteger i = 0; i < 100000; i++)
                [pg retain];
            NSLog(@"%ld", [pg retainCount]);
        }
        return 0;
    }

    2013-03-21 08:32:12.689 pgref[81462:707] 100001
    2013-03-21 08:32:12.691 pgref[81462:707] 200001
    2013-03-21 08:32:12.693 pgref[81462:707] 300001
    2013-03-21 08:32:12.695 pgref[81462:707] 400001
    2013-03-21 08:32:12.696 pgref[81462:707] 500001
    2013-03-21 08:32:12.698 pgref[81462:707] 75713  <---- Nooooooo!
    2013-03-21 08:32:12.700 pgref[81462:707] 175713
    2013-03-21 08:32:12.702 pgref[81462:707] 275713
    2013-03-21 08:32:12.704 pgref[81462:707] 375713


    -tim
  • On Mar 21, 2013, at 8:35 AM, Timothy Wood wrote:

    > On Mar 20, 2013, at 12:01 PM, Kyle Sluder <kyle...> wrote:
    >> The reason it's limited to 19 bits is because for speed purposes
    >> NSParagraphStyle implements its own inline retain count rather than
    >> relying on NSObject's external refcount table. Which is also why you
    >> can't make ARC weak references to NSParagraphStyle instances. ;-)
    >
    > Yikes. Usually the inline bits are used for the *low order* bits of the retain count. That is, the full retain count is something like (internal_refs + (external_refs << internal_width).

    Ahh. 19 seems like such an arbitrary number, but if it's _supposed_ to be extended then that'd make more sense.

    I'll be filing a radar today.

    Thanks everyone.

    --
    Seth Willits
  • On Mar 21, 2013, at 11:39 AM, Seth Willits wrote:

    >>> The reason it's limited to 19 bits is because for speed purposes
    >>> NSParagraphStyle implements its own inline retain count rather than
    >>> relying on NSObject's external refcount table. Which is also why you
    >>> can't make ARC weak references to NSParagraphStyle instances. ;-)
    >>
    >> Yikes. Usually the inline bits are used for the *low order* bits of the retain count. That is, the full retain count is something like (internal_refs + (external_refs << internal_width).
    >
    > Ahh. 19 seems like such an arbitrary number, but if it's _supposed_ to be extended then that'd make more sense.
    >
    > I'll be filing a radar today.

    #13478399

    If any employees are watching.

    --
    Seth Willits
  • Any way to make sure it also goes into the openradar site?

    On Mar 21, 2013, at 7:14 PM, Seth Willits wrote:

    > On Mar 21, 2013, at 11:39 AM, Seth Willits wrote:
    >
    >>>> The reason it's limited to 19 bits is because for speed purposes
    >>>> NSParagraphStyle implements its own inline retain count rather than
    >>>> relying on NSObject's external refcount table. Which is also why you
    >>>> can't make ARC weak references to NSParagraphStyle instances. ;-)
    >>>
    >>> Yikes. Usually the inline bits are used for the *low order* bits of the retain count. That is, the full retain count is something like (internal_refs + (external_refs << internal_width).
    >>
    >> Ahh. 19 seems like such an arbitrary number, but if it's _supposed_ to be extended then that'd make more sense.
    >>
    >> I'll be filing a radar today.
    >
    > #13478399
    >
    > If any employees are watching.
    >
    >
    > --
    > Seth Willits
previous month march 2013 next month
MTWTFSS
        1 2 3
4 5 6 7 8 9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30 31
Go to today