accessibility and iTunes?!?

  • Hello all,

    although direct sending of events would be much better for my
    purpose, since I got no luck with it so far, I am also testing the
    accessibility API. I've got a strange results though... do I overlook
    something, or does the thing just not the heck work?!?

    Whatever did I try, AXUIElementCopyElementAtPosition at the best
    found a window, much more often it found application only. So, I've
    rigged my own simple recursive go-thru-children-and-check-hits method.

    The core looks like this:

        id kind,val;
        CGPoint cgp;
        CGSize cgs;
        if (AXUIElementCopyAttributeValue(ui,kAXRoleAttribute,
    (CFTypeRef*)&kind)!=0) return NULL;
        if (AXUIElementCopyAttributeValue(ui,kAXPositionAttribute,
    (CFTypeRef*)&val)!=0) return NULL;
        if (!AXValueGetValue((AXValueRef)val,kAXValueCGPointType,&cgp))
    return NULL;
        if (AXUIElementCopyAttributeValue(ui,kAXSizeAttribute,
    (CFTypeRef*)&val)!=0) return NULL;
        if (!AXValueGetValue((AXValueRef)val,kAXValueCGSizeType,&cgs))
    return NULL;
        NSRect rr=NSMakeRect(cgp.x,cgp.y,cgs.width,cgs.height);
        NSLog(@"searching %@ in %@, %@",NSStringFromPoint
    (pt),kind,NSStringFromRect(rr));
        if (!NSPointInRect(pt,rr)) return NULL;

    Looks fair, or have I overlooked something?

    Does not work. With iTunes 6.0.4 (3) I keep getting horrible y-
    coordinates:

    2006-10-04 19:04:25.895 Media Center[8694] searching {738, 773} in
    AXWindow, {{518, 260}, {710, 556}}
    2006-10-04 19:04:25.896 Media Center[8694] found, trying 6 children...
    2006-10-04 19:04:25.898 Media Center[8694] searching {738, 773} in
    AXScrollBar, {{892, 17317}, {15, 83}}
    2006-10-04 19:04:25.903 Media Center[8694] searching {738, 773} in
    AXScrollBar, {{892, 17317}, {15, 83}}
    ...

    With iTunes 7.0 (70) the coordinates seem right, but instead of
    scrollbars and similar I keep getting some unknown elements (whose
    lists of actions are empty)...

    2006-10-04 19:11:18.021 Media Center[275] searching {403, 638} in
    AXWindow, {{190, 119}, {710, 556}}
    2006-10-04 19:11:18.021 Media Center[275] found, trying 1 children...
    2006-10-04 19:11:18.021 Media Center[275] searching {403, 638} in
    AXUnknown, {{190, 119}, {710, 556}}
    2006-10-04 19:11:18.022 Media Center[275] found, trying 1 children...
    2006-10-04 19:11:18.022 Media Center[275] searching {403, 638} in
    AXUnknown, {{190, 119}, {710, 556}}
    2006-10-04 19:11:18.022 Media Center[275] found, trying 1 children...
    ...

    Dammit. How the heck does one control another application?!?
    ---
    Ondra ÄŒada
    OCSoftware:    <ocs...>              http://www.ocs.cz
    private        <ondra...>            http://www.ocs.cz/oc
  • On Oct 4, 2006, at 10:19 AM, Ondra Cada wrote:

    > although direct sending of events would be much better for my
    > purpose, since I got no luck with it so far, I am also testing the
    > accessibility API. I've got a strange results though... do I
    > overlook something, or does the thing just not the heck work?!?
    >
    > Whatever did I try, AXUIElementCopyElementAtPosition at the best
    > found a window, much more often it found application only. So, I've
    > rigged my own simple recursive go-thru-children and-check-hits method.

    This question would probably be better directed to the accessibility-
    dev list, since it isn't specific to Cocoa.

    > The core looks like this:
    >
    > id kind,val;
    > CGPoint cgp;
    > CGSize cgs;
    > if (AXUIElementCopyAttributeValue(ui,kAXRoleAttribute,
    > (CFTypeRef*)&kind)!=0) return NULL;
    > if (AXUIElementCopyAttributeValue(ui,kAXPositionAttribute,
    > (CFTypeRef*)&val)!=0) return NULL;
    > if (!AXValueGetValue((AXValueRef)val,kAXValueCGPointType,&cgp))
    > return NULL;
    > if (AXUIElementCopyAttributeValue(ui,kAXSizeAttribute,
    > (CFTypeRef*)&val)!=0) return NULL;
    > if (!AXValueGetValue((AXValueRef)val,kAXValueCGSizeType,&cgs))
    > return NULL;
    > NSRect rr=NSMakeRect(cgp.x,cgp.y,cgs.width,cgs.height);
    > NSLog(@"searching %@ in %@,
    > %@",NSStringFromPoint(pt),kind,NSStringFromRect(rr));
    > if (!NSPointInRect(pt,rr)) return NULL;
    >
    > Looks fair, or have I overlooked something?

    Looks reasonable to me.

    > Does not work. With iTunes 6.0.4 (3) I keep getting horrible y-
    > coordinates:
    >
    > 2006-10-04 19:04:25.895 Media Center[8694] searching {738, 773} in
    > AXWindow, {{518, 260}, {710, 556}}
    > 2006-10-04 19:04:25.896 Media Center[8694] found, trying 6 children...
    > 2006-10-04 19:04:25.898 Media Center[8694] searching {738, 773} in
    > AXScrollBar, {{892, 17317}, {15, 83}}
    > 2006-10-04 19:04:25.903 Media Center[8694] searching {738, 773} in
    > AXScrollBar, {{892, 17317}, {15, 83}}
    > ...

    It wouldn't surprise me if iTunes 6 kept its scrollbars positioned far
    offscreen normally and only moved them onscreen right before drawing.

    > With iTunes 7.0 (70) the coordinates seem right, but instead of
    > scrollbars and similar I keep getting some unknown elements (whose
    > lists of actions are empty)...
    >
    > 2006-10-04 19:11:18.021 Media Center[275] searching {403, 638} in
    > AXWindow, {{190, 119}, {710, 556}}
    > 2006-10-04 19:11:18.021 Media Center[275] found, trying 1 children...
    > ...
    >
    > Dammit. How the heck does one control another application?!?

    For iTunes, I'd recommend AppleScript. I believe that's what FrontRow
    uses, for example. The problem with using Accessibility to control
    iTunes is that due to iTunes's requirements to run on both Mac and
    Windows, and have a variety of specialized and rapidly-changing UI
    features, it uses few of the standard system views, and implements
    many of its own views - and it does not implement accessibility
    support for most of its own views yet.

    -eric
previous month october 2006 next month
MTWTFSS
            1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31          
Go to today