Unit testing framework suggestions?
-
Greetings,
Sorry for all the newbie questions. But I'm afraid you're going to have to put up these for the next 6 months. ;)
I've got enough code that I want to start doing unit testing. So far I've only discovered this one:
<http://unitkit.org/>
Any comments about UnitKit (good experiences, likes, dislikes, ...)? Are there any alternatives that people would like to suggest.
Thanks,
James
--
James Bucanek <mailto:<privatereply...> -
On Sep 22, 2004, at 5:22 PM, James Bucanek wrote:
I've got enough code that I want to start doing unit testing. So far
I've only discovered this one:
<http://unitkit.org/>
Any comments about UnitKit (good experiences, likes, dislikes, ...)?
Are there any alternatives that people would like to suggest.
See Also:
http://www.sente.ch/software/ocunit/
-jcr
On Sep 22, 2004, at 5:22 PM, James Bucanek wrote:
I've got enough code that I want to start doing unit testing. So far
I've only discovered this one:
<http://unitkit.org/>
Any comments about UnitKit (good experiences, likes, dislikes, ...)?
Are there any alternatives that people would like to suggest.
See Also:
http://www.sente.ch/software/ocunit/
-jcr
John C. Randolph <jcr...> (408) 974-8819
Sr. Cocoa Software Engineer,
Apple Worldwide Developer Relations
http://developer.apple.com/cocoa/index.html -
On Wed, 22 Sep 2004 17:22:29 -0700 James Bucanek <subscriber...>
wrote:
>only discovered this one:
> I've got enough code that I want to start doing unit testing. So far I've
>there any alternatives that people would like to suggest.
> <http://unitkit.org/>
>
> Any comments about UnitKit (good experiences, likes, dislikes, ...)? Are
See also
http://testkit.sourceforge.net
Tim -
At 20:56 Uhr -0500 22.09.2004, Tim Hart wrote:
> See also
>
> http://testkit.sourceforge.net
BTW -- does anybody know some good tutorials about Unit testing? I
don't mean about how to use the frameworks, but rather on how to do
it for the most benefit, and why it's better than other methods, and
how this relates to Cocoa and Objective C.
E.g. UnitKit keeps separate "test objects", which I find kinda odd.
I'm wondering why one shouldn't put the tests in the object in need
of the testing instead... things like that.
--
Cheers,
M. Uli Kusterer
------------------------------------------------------------
"The Witnesses of TeachText are everywhere..."
http://www.zathras.de -
On Thursday, September 23, 2004, at 02:54PM, M. Uli Kusterer <Witness.of.TeachText...> wrote:
> At 20:56 Uhr -0500 22.09.2004, Tim Hart wrote:
>> See also
>>
>> http://testkit.sourceforge.net
>
> BTW -- does anybody know some good tutorials about Unit testing? I
> don't mean about how to use the frameworks, but rather on how to do
> it for the most benefit, and why it's better than other methods, and
> how this relates to Cocoa and Objective C.
A good resource for unit testing in general:
http://c2.com/cgi/wiki?UnitTestTutorial
Why is it better than other methods? I'm a unit testing fan, but 'better' is a very subjective question.
The biggest benefit I gain by keeping a suite of unit tests is this:
I have a suite that I can run on-demand, that takes a fraction of the time it would take to manually test everything, and is a reasonable measure of the overall health of the system. A breaking test is a guarantee that there is either something wrong with the code - or the test. A unit test written to 'uncover' a bug guarantees that the same bug will not reappear (although the same symptoms due to *another* bug might). Unit tests aren't really useful unless they are run on a regular, frequent basis.
Note that a passing suite of unit tests does *not* guarantee flawless code. FIT tests and user acceptance testing is still A Very Good Idea(tm). Unit tests do aid debugging by giving you a direct and immediate entry point into the method/function that isn't behaving appropriately.
There are several styles to unit testing. Martin Fowler compares 2 types of unit testing here
http://www.martinfowler.com/articles/mocksArentStubs.html
TestKit has a beginning API for mock objects, but the API is currently absent code to count/verify how many times a method is called. That will follow soon.
How does this relate to Cocoa/Objective-C? I've not seen an article relating to this. Can you be more specific with your questions?
> E.g. UnitKit keeps separate "test objects", which I find kinda odd.
> I'm wondering why one shouldn't put the tests in the object in need
> of the testing instead... things like that.
One of the main purposes of a unit testing suite is to enable the developer to concentrate on the tests, and not on the setup/identification and execution of those tests. A very convenient way to do this in Objective-C is to look for all subclasses of a particular class. In the case of TestKit, it's TWTestCase. That requirement does limit the developer from making tests part of the class being tested.
I don't know that I would want to couple my tests to the class in question so tightly, though. As a separate library, I have the choice of delivering my test suite along with the code itself to the client, or not. The client has a choice of installing the test suite on the same machine - or not. It's a bit more flexible.
NUnit, which is a unit testing kit for C# (http://www.nunit.org), would actually allow you to bind your test cases to the class being tested if your chose. The reason this is possible is due to C#'s attributes - class metadata. NUnit test cases do not have to inherit from any particular class.
> --
> Cheers,
> M. Uli Kusterer
> ------------------------------------------------------------
> "The Witnesses of TeachText are everywhere..."
> http://www.zathras.de
>
>
-
On Sep 22, 2004, at 5:22 PM, James Bucanek wrote:
> I've got enough code that I want to start doing unit testing. So far
> I've only discovered this one:
>
> <http://unitkit.org/>
>
> Any comments about UnitKit (good experiences, likes, dislikes, ...)?
> Are there any alternatives that people would like to suggest.
I have used:
a) ObjCUnit: http://oops.se/objcunit
This is a unit testing framework based on JUnit. I was pleased with it,
although have not used it recently. It is at version 1.2, circa 2002,
so I guess it hasn't changed recently either. It has some level of mock
object support as well, which I have not used.
b) OCUnit: http://www.sente.ch/software/ocunit
This is still what I use at work. It is also much like JUnit, although
I think the docs state that it is actually modeled after the SmallTalk
SUnit project (which would make OCUnit a sibling of JUnit, rather than
an adaptation of it). I have been very satisfied with it and it works
much as you'd expect, with basic Xcode integration. OCUnit is at
version 38 from June of 2004.
For mock object support, Sen:te recommends OCMock, from Mulle
Kybernetik. I have not used OCMock much yet, but it looks very
interesting:
http://www.mulle-kybernetik.com/software/OCMock
Subsequent to using the above, I have also read about, but not used:
c) TestKit: http://testkit.sourceforge.net
This is another adaptation of JUnit, which has already been mentioned
on this thread. It looks nice, and the documentation is very good. It
features mock object support and also a GUI test runner. TestKit is at
version 0.9.4 (June 2004).
d) UnitKit: http://unitkit.org
I hadn't even heard of this one until this thread. It is a bit
different than the others in that it seems to to be specific to
Objective-C and intended mainly for Xcode. It does not clone the JUnit
interface although the concepts are similar. It is at version 1.0.1
(August 2004).
----------
I would be very interested to hear people's real-world experiences with
any of these. It is pretty hard to realistically try them all in a
production environment. At Five Speed we use OCUnit because the
interface is familiar, it is authored by Sen:te and so carries their
reputation, and because it does now have Xcode integration (although
that used to be lacking). A couple years ago we used ObjCUnit, but the
main reason we switched (I think) was that OCUnit seemed more "alive"
in terms of active development.
I personally am interested to hear about real-world experiences with
UnitKit, since its approach is the most different.
Cheers,
--
Mason Mark
Five Speed Software, Inc.
http://www.fivespeed.com -
> http://www.martinfowler.com/articles/mocksArentStubs.html
>
> TestKit has a beginning API for mock objects, but the API is currently
> absent code to count/verify how many times a method is called. That
> will follow soon.
>
For an implementation of mock objects see OCMock, at
http://www.mulle-kybernetik.com/software/OCMock/
Marco Scheurer
Sen:te, Lausanne, Switzerland http://www.sente.ch -
On Sep 23, 2004, at 22:51, M. Uli Kusterer wrote:
> E.g. UnitKit keeps separate "test objects", which I find kinda odd.
> I'm wondering why one shouldn't put the tests in the object in need of
> the testing instead... things like that.
Most implementations of testing kits implements tests in a subclass of
TestCase.
One reason is that tests can have their own state and ivars, and it
wouldn't be a good idea to mix this with the class you want to test.
Another is that tests are written from the perspective of a user of the
class rather than as "internal" tests. A good practice is to write the
tests before you write the tested code. If you do so, your tests can be
seen as a formal specification of the code to be written.
Another is that a lot of stuff is usually done in the TestCase class,
so that writing a test is as simple as writing a single method in a
subclass of TestCase.
All the xUnit packages are more or less derived from the original
Smalltalk "SUnit". The seminal article is by Kent Beck, and can be
found at http://www.armaties.com/testfram.htm . So there is also an
historical reason for doing it in subclasses of Testcase.
However, this can be done differently. For instance, Marcel Weiher
advocates putting the tests in a category of the tested class.
Marco Scheurer
Sen:te, Lausanne, Switzerland http://www.sente.ch -
On Sep 23, 2004, at 1:51 PM, M. Uli Kusterer wrote:
> BTW -- does anybody know some good tutorials about Unit testing? I
> don't mean about how to use the frameworks, but rather on how to do it
> for the most benefit, and why it's better than other methods, and how
> this relates to Cocoa and Objective C.
Kent Beck, "Test Driven Development" is good. As is Martin Fowler,
"Refactoring" and Beck's original book on Extreme Programming.
Some of the references on the OCUnit site, to Beck et al's papers on
the original Smalltalk unit testing framework are also pretty good.
-- Chris -
At 4:56 Uhr +0200 24.09.2004, Marco Scheurer wrote:
> One reason is that tests can have their own state and ivars, and it
> wouldn't be a good idea to mix this with the class you want to test.
A-ha! Thanks, that makes a lot of sense. I knew there had to be some
obvious reason.
> (...)
>
> All the xUnit packages are more or less derived from the original
> Smalltalk "SUnit". The seminal article is by Kent Beck, and can be
> found at http://www.armaties.com/testfram.htm . So there is also an
> historical reason for doing it in subclasses of Testcase.
>
> However, this can be done differently. For instance, Marcel Weiher
> advocates putting the tests in a category of the tested class.
Yes, that was what I thought. It wouldn't be too hard to use e.g.
the preprocessor to remove these tests from the class for the final
production build. Since I know that I prefer things that are related
(like code and the tests that go with it, or code and its
documentation) together in one file, because otherwise it's too easy
to have them go out of sync.
But of course, if my test wants to keep state, it'd obviously be a
little bothersome to have that in the same class. It would even
violate encapsulation, come to think of it. Thanks, I think that
aspect is now a little clearer to me.
Just a thought: Does any of the kits have some macros for "default
tests"? E.g. some way to just say: Arguments to this function (or
its result, or whatever) range from X to Y, and then it would
generate code to try the edges of that range (both inside and out)
automatically?
--
Cheers,
M. Uli Kusterer
------------------------------------------------------------
"The Witnesses of TeachText are everywhere..."
http://www.zathras.de -
On Sep 24, 2004, at 2:00 PM, M. Uli Kusterer wrote:
> At 4:56 Uhr +0200 24.09.2004, Marco Scheurer wrote:
>> One reason is that tests can have their own state and ivars, and it
>> wouldn't be a good idea to mix this with the class you want to test.
>
> A-ha! Thanks, that makes a lot of sense. I knew there had to be some
> obvious reason.
>
>> (...)
>>
>> All the xUnit packages are more or less derived from the original
>> Smalltalk "SUnit". The seminal article is by Kent Beck, and can be
>> found at http://www.armaties.com/testfram.htm . So there is also an
>> historical reason for doing it in subclasses of Testcase.
>>
>> However, this can be done differently. For instance, Marcel Weiher
>> advocates putting the tests in a category of the tested class.
>
> Yes, that was what I thought. It wouldn't be too hard to use e.g. the
> preprocessor to remove these tests from the class for the final
> production build. Since I know that I prefer things that are related
> (like code and the tests that go with it, or code and its
> documentation) together in one file, because otherwise it's too easy
> to have them go out of sync.
Not really if you run your tests after each change, or frequently (at
least daily). A lot of our users actually prefer to keep the tests
separated from the code, in a different target or a different project.
One reason is to avoid deploying the test code and the testing
framework (although I think that in some cases it can be nice to be
able to run the tests at the customer's site).
> But of course, if my test wants to keep state, it'd obviously be a
> little bothersome to have that in the same class. It would even
> violate encapsulation, come to think of it. Thanks, I think that
> aspect is now a little clearer to me.
>
> Just a thought: Does any of the kits have some macros for "default
> tests"? E.g. some way to just say: Arguments to this function (or its
> result, or whatever) range from X to Y, and then it would generate
> code to try the edges of that range (both inside and out)
> automatically?
The answer is certainly not like this, AFAIK none of the kits
automatically generate code. To do this kind of tests with OCUnit you
would use one of a bunch of macros to test that exceptions are thrown
or not, for instance:
- (void) testSetRatioRange
{
STAssertNoThrow ([x setRatio: minimum], nil);
STAssertNoThrow ([x setRatio: maximum], nil);
STAssertThrowsSpecificNamed ([x setRatio:minimum - epsilon],
NSException, NSInvalidArgumentException, nil);
STAssertThrowsSpecificNamed ([x setRatio:maximum + epsilon],
NSException, NSInvalidArgumentException, nil);
}
I'm not sure that automatically generating such code is worth the
effort.
Marco Scheurer
Sen:te, Lausanne, Switzerland http://www.sente.ch -
M. Uli Kusterer wrote on Friday, September 24, 2004:
> At 4:56 Uhr +0200 24.09.2004, Marco Scheurer wrote:
>> One reason is that tests can have their own state and ivars, and it
>> wouldn't be a good idea to mix this with the class you want to test.
>
> A-ha! Thanks, that makes a lot of sense. I knew there had to be some
> obvious reason.
The documentation for UnitKit points out another reason why test classes are kept separate. Namely, the performance of an application directly impacted by the size of the code and data structures. Leaving your test code in your production code just makes for a bloated app.
And if you're thinking that you'll remove all of the test code via the pre-processor, that's a hazardous road to go down. You want to test the code that you're going to ship, not the code that you *think* will still work after you flip some compiler switches. Been there, done that, got burned.
--
James Bucanek <mailto:<privatereply...> -
At 14:51 Uhr +0200 24.09.2004, Marco Scheurer wrote:
> The answer is certainly not like this, AFAIK none of the kits
> automatically generate code. To do this kind of tests with OCUnit
> you would use one of a bunch of macros to test that exceptions are
> thrown or not, for instance:
>
> - (void) testSetRatioRange
> {
> STAssertNoThrow ([x setRatio: minimum], nil);
> STAssertNoThrow ([x setRatio: maximum], nil);
> STAssertThrowsSpecificNamed ([x setRatio:minimum - epsilon],
> NSException, NSInvalidArgumentException, nil);
> STAssertThrowsSpecificNamed ([x setRatio:maximum + epsilon],
> NSException, NSInvalidArgumentException, nil);
> }
>
> I'm not sure that automatically generating such code is worth the effort.
Well, I guess one could just have a macro that does a typical
sequence of such tests, right? After all, testing a minimum and
maximum and that it doesn't or does throw on these should be a pretty
common request.
--
Cheers,
M. Uli Kusterer
------------------------------------------------------------
"The Witnesses of TeachText are everywhere..."
http://www.zathras.de -
On Sep 24, 2004, at 5:00 AM, M. Uli Kusterer wrote:
> Just a thought: Does any of the kits have some macros for "default
> tests"? E.g. some way to just say: Arguments to this function (or its
> result, or whatever) range from X to Y, and then it would generate
> code to try the edges of that range (both inside and out)
> automatically?
Blatant plug alert:
I work for a company called Parasoft (http://www.parasoft.com) that
does this for lots of languages (C++, Java, C#), but not for
Objective-C. The technology is pretty impressive; the tools can inspect
the code and intelligently determine what tests are useful (it pulls
out constants, follows Design By Contract, keeps results for regression
tests, etc). If you can actually say that you'd be willing to buy a
theoretical ObjCTest, give the company a call and tell them so; I'd
love to work on the product!
Seth A. Roby The Amazing Llama < mail or AIM me at tallama at mac dot
com>
"Life is like an exploded clown. It's really funny until you figure out
what just happened." -
I found the book interesting, but not too useful for specifically what
Uli was asking about. I remember wishing for more examples of actual
test cases. It's more about the big project management picture, and
selling the benefits of extreme programming, and very little about
actually doing unit testing.
Daniel
On Sep 23, 2004, at 11:37 PM, Chris Hanson wrote:
> Kent Beck, "Test Driven Development" is good. As is Martin Fowler,
> "Refactoring" and Beck's original book on Extreme Programming.
-
I said:
> I found the book interesting, but not too useful for specifically what
> Uli was asking about. I remember wishing for more examples of actual
> test cases. It's more about the big project management picture, and
> selling the benefits of extreme programming, and very little about
> actually doing unit testing.
Whoops! I neglected to indicate the book I'm talking about: the Beck
book "Extreme Programming Explained."
On Sep 23, 2004, at 11:37 PM, Chris Hanson wrote:
> Kent Beck, "Test Driven Development" is good. As is Martin Fowler,
> "Refactoring" and Beck's original book on Extreme Programming.
-
Uli, I am so glad you started this discussion. I wanted to do it this
for the last two years or so, but never really found time...
On Sep 23, 2004, at 10:51 PM, M. Uli Kusterer wrote:
> E.g. UnitKit keeps separate "test objects", which I find kinda odd.
> I'm wondering why one shouldn't put the tests in the object in need of
> the testing instead... things like that.
I use testing frameworks for many years - something like 15 - long
before xUnit, XP, Agile etc were born. And I used OCUnit almost from
the day it was born, until ...
Until one of my students (a bright chap) came to me and told me that he
wasted a day before he could write his first test method. His exact
words were "... testing with OCUnit is not the simplest thing that
could possibly work" [OCUnit was at the time probably the only
framework for ObjC - but my student was criticizing the method, not the
framework]
Well, this gave me a lot of food for thinking. At that time I was
already reading a very early draft of Kent's TDD book, and happy I was
not - not because of the technique, but because of the technology... So
after thinking about this for almost two months, finally I came to the
conclusion that the problem is exactly the one you discovered -
separation of TestCase from the class being tested [well, in reality
this is not always true -- but please read my replays to Marco I intend
to write later today for more].
At the time Drew and I was starting to put our thoughts about
multi-volume text book on Cocoa programming (still work in [slow]
progress), so we had a beer together and write down (on a A6 index
card) the requirements for a testing framework to be publish in the
book. Here what we scribbled:
- the integration to any cocoa project (existing or new) should not
take longer then a minute
- one should learn the basics in max 5 mins
- test could be placed anywhere (in TestCase or directly in the class).
Later, we added three additional requirements:
- there should be no technical distinction between unit and acceptance
tests,
- the framework should be able to test the UI, and
- test runs should have memory - ignore for now, it is a subject of
completely different thread ...
We ended up writing something that at first looked like a simplified
version of xUnit. But is was not! Very soon I discovered that actually
it is completely new beast. The difference is more on how it changed my
way of testing - the flow. Now actually I can not call it testing any
more ... it feels more like compiling, or debugging - a real integrated
part of the development process. And it is part of the program you
write (as it should be - I discovered this much later) - not part of
the IDE as the original xUnit was supposed. I am sure all this has some
far reaching philosophical consequences, and although I discussed it
privately with few prominent XPers and Pragmatic Programmers, I never
found time to go deeper into this.
I had the chance to teach Cocoa to a (very) large Apple third party
software provider, and to test most of our ideas. I observed something
very surprising - developers who were exposed to testing framework for
the first time were able to write their first test within 10 min, and
after a day they were semi-experts. But there we two developers with
prior JUnit experience. They struggled really badly! Actually they
needed about a week before feel comfortable with our framework. It
might sound strange, but I went through the same initial struggle. I am
not sure, but this could be comparable to the transition of X11
developers to Cocoa - the first thing they ask for is the "while(1) {
//get next event }" loop - it takes them some time before they stop
feeling the need for such a beast...
Drew and I want to make the sources and some texts freely available,
and we have a permission already to do this, but realistically this
will not happen before the end of october. So I hope you stay tuned :-)
Georg Tuparev
Tuparev Technologies
Klipper 13
1186 VR Amstelveen
The Netherlands
Mobile: +31-6-55798196 -
It happens that I disagree with Marco once each decade (so do not
expect me to start discussing with him next 10 years) ... but I feel
this is one of these occasions...
Also read this in the context of my previous replay to Uli.
On Sep 24, 2004, at 4:56 AM, Marco Scheurer wrote:
> On Sep 23, 2004, at 22:51, M. Uli Kusterer wrote:
>
>> E.g. UnitKit keeps separate "test objects", which I find kinda odd.
>> I'm wondering why one shouldn't put the tests in the object in need
>> of the testing instead... things like that.
>
> Most implementations of testing kits implements tests in a subclass of
> TestCase.
True. As far as I know only NUnit - and I believe because of technical
reasons (I might be wrong - never spent too much time on it).
>
> One reason is that tests can have their own state and ivars, and it
> wouldn't be a good idea to mix this with the class you want to test.
You might be right is theory, but at least I and all the people I
enjoyed working with almost never kept state between two tests.
Actually, if I think very hard, I cannot find a single case where my
need of using state was not because of stinky code - or bad design, or
bad refactoring. In our testing framework, we create an instance of the
tested class for every single test method. In cases where we need
singleton we need to override the setUp and tearDown methods. For class
like App Delegate (instantiated in a nib file) we write:
+ (id)setUp { return [NSApp delegate]; }
+ (void)tearDown:(id)testInstance { }
This feels strange at the beginning - normally one needs to implement
setUp and tearDown for each TestCase class, but with our framework we
need from time to time to knock them off - most often though one never
touches them. Also note the "+" in front of these methods. This is not
a typo :-)
Here a typical test method:
- (void)testVisibleWindow {
SEConfirmIsTrue(tableWindow != nil, @"Window should not be nil");
SEConfirmIsTrue([tableWindow isVisible],
@"Window should be visible after the application is
launched");
}
UI testing made easy...
> Another is that tests are written from the perspective of a user of
> the class rather than as "internal" tests. A good practice is to write
> the tests before you write the tested code. If you do so, your tests
> can be seen as a formal specification of the code to be written.
Developer writes tests first - to help him or her to write the real
code. So you write a test not as a user, but as an author. Tests are
also the best known to me way of documenting - and you want to keep the
doc close to the code - otherwise they will get out of sync - therefore
test are so good - they must be in sync if they have to pass! [I am
sure this might sound like Knuth's Literate Programming - in retrospect
I have to acknowledge that after Drew and I wrote the first version of
our testing framework, I though that this is the closest practical way
of doing literate programming for Cocoa]
> Another is that a lot of stuff is usually done in the TestCase class,
> so that writing a test is as simple as writing a single method in a
> subclass of TestCase.
Luckily we have categories in ObjC. And no one stops you putting this
"lot of stuff" to a category of NSObject :-) But more importantly,
there is no need to have a lot of stuff in TestCase class.
[...]
Georg Tuparev
Tuparev Technologies
Klipper 13
1186 VR Amstelveen
The Netherlands
Mobile: +31-6-55798196 -
On Sep 24, 2004, at 2:51 PM, Marco Scheurer wrote:
>> Yes, that was what I thought. It wouldn't be too hard to use e.g.
>> the preprocessor to remove these tests from the class for the final
>> production build. Since I know that I prefer things that are related
>> (like code and the tests that go with it, or code and its
>> documentation) together in one file, because otherwise it's too easy
>> to have them go out of sync.
>
> Not really if you run your tests after each change, or frequently (at
> least daily). A lot of our users actually prefer to keep the tests
> separated from the code, in a different target or a different project.
Why?
> One reason is to avoid deploying the test code and the testing
> framework (although I think that in some cases it can be nice to be
> able to run the tests at the customer's site).
Some cases? Always! This saved us many debugging hours. And users are
in general impressed. When they see that your program has 3000 tests,
they are really impressed! And because we test UI too, they think it is
fun too - too see all these windows hopping around. BTW, we had one
client who told us to keep the tests always on. This "helped him to
understand better developers intentions"...
Georg Tuparev
Tuparev Technologies
Klipper 13
1186 VR Amstelveen
The Netherlands
Mobile: +31-6-55798196 -
On Sep 24, 2004, at 5:27 PM, James Bucanek wrote:
>> A-ha! Thanks, that makes a lot of sense. I knew there had to be some
>> obvious reason.
>
> The documentation for UnitKit points out another reason why test
> classes are kept separate. Namely, the performance of an application
> directly impacted by the size of the code and data structures.
> Leaving your test code in your production code just makes for a
> bloated app.
Well, we did made measurement for our last two projects - depending on
the application and on the computer, the performance deteriorates
between 0% and 3% (on average 1%). Not worth discussing - specially
because tests helped us often to have very good performance to start
with.
Georg Tuparev
Tuparev Technologies
Klipper 13
1186 VR Amstelveen
The Netherlands
Mobile: +31-6-55798196 -
On Sep 24, 2004, at 5:57 PM, M. Uli Kusterer wrote:
>> - (void) testSetRatioRange
>> {
>> STAssertNoThrow ([x setRatio: minimum], nil);
>> STAssertNoThrow ([x setRatio: maximum], nil);
>> STAssertThrowsSpecificNamed ([x setRatio:minimum - epsilon],
>> NSException, NSInvalidArgumentException, nil);
>> STAssertThrowsSpecificNamed ([x setRatio:maximum + epsilon],
>> NSException, NSInvalidArgumentException, nil);
>> }
>>
>> I'm not sure that automatically generating such code is worth the
>> effort.
>
> Well, I guess one could just have a macro that does a typical
> sequence of such tests, right? After all, testing a minimum and
> maximum and that it doesn't or does throw on these should be a pretty
> common request.
That's not the point! There are many test-coverage frameworks that are
as useful as Rational designing tool...
Good unit tests help the developer to:
- design the application,
- document the sources,
- gain confidence (courage) needed to make major refactorings,
- fulfill the promise of delivering business value to the clients (e.g.
by demonstrating that acceptance tests run).
None of these tasks is to be autogenerated by macro!
Georg Tuparev
Tuparev Technologies
Klipper 13
1186 VR Amstelveen
The Netherlands
Mobile: +31-6-55798196 -
On Sep 26, 2004, at 01:33, Georg Tuparev wrote:
> It happens that I disagree with Marco once each decade (so do not
> expect me to start discussing with him next 10 years) ... but I feel
> this is one of these occasions...
Oh! I'l have to show some disagreement too then!
>> One reason is that tests can have their own state and ivars, and it
>> wouldn't be a good idea to mix this with the class you want to test.
>
> You might be right is theory, but at least I and all the people I
> enjoyed working with almost never kept state between two tests.
> Actually, if I think very hard, I cannot find a single case where my
> need of using state was not because of stinky code - or bad design, or
> bad refactoring.
I admit that it is not frequent, and one could certainly go away
without that possibility.
> In our testing framework, we create an instance of the tested class
> for every single test method.
If I understand this correctly, this is exactly what is done in OCUnit
and SmalltalkUnit.
> In cases where we need singleton we need to override the setUp and
> tearDown methods. For class like App Delegate (instantiated in a nib
> file) we write:
>
> + (id)setUp { return [NSApp delegate]; }
> + (void)tearDown:(id)testInstance { }
>
> This feels strange at the beginning - normally one needs to implement
> setUp and tearDown for each TestCase class,
No. You can but don't have to implement setUp and tearDown. You do it
only when you want to initialize all your TestCase tests to a common
state.
> but with our framework we need from time to time to knock them off -
> most often though one never touches them. Also note the "+" in front
> of these methods. This is not a typo :-)
OCUnit also has +setUp...
> Here a typical test method:
>
> - (void)testVisibleWindow {
> SEConfirmIsTrue(tableWindow != nil, @"Window should not be nil");
> SEConfirmIsTrue([tableWindow isVisible],
> @"Window should be visible after the application
> is launched");
> }
>
> UI testing made easy...
I fail to see any differences with how this would be done using OCUnit,
except the name of the macros. Until I see more of this kit, I'm still
unconvinced: it looks to me like one of the many "I can make it
simpler" attempt, which evolves in mostly the same thing as OCUnit, a
good example of the NIH syndrome.
With an installer package, target and file templates and a README, I
doubt that it would take anyone more than 5 minutes to write a first
test using OCUnit, unless of course one is new to the whole idea.
>> Another is that tests are written from the perspective of a user of
>> the class rather than as "internal" tests. A good practice is to
>> write the tests before you write the tested code. If you do so, your
>> tests can be seen as a formal specification of the code to be
>> written.
>
> Developer writes tests first - to help him or her to write the real
> code. So you write a test not as a user, but as an author. Tests are
> also the best known to me way of documenting - and you want to keep
> the doc close to the code - otherwise they will get out of sync -
> therefore test are so good - they must be in sync if they have to
> pass!
If you test all the time (say after each change), it doesn'y matter
where your tests are, they will not get out of sync. If you don't, it
doesn't matter where they are either, they will get out of sync. So
this to "keep your tests in sync" is not a reason for having them in
one class or another.
> [I am sure this might sound like Knuth's Literate Programming - in
> retrospect I have to acknowledge that after Drew and I wrote the first
> version of our testing framework, I though that this is the closest
> practical way of doing literate programming for Cocoa]
I said that much when talking about a "formal specification".
In truth my point about writing tests first also has little to do with
where they are implemented. You can of course start writing your class
by writing the test methods. But it seems more natural to me to do so
as an "outsider", the customer who specifies what the methods are
supposed to do.
>> Another is that a lot of stuff is usually done in the TestCase class,
>> so that writing a test is as simple as writing a single method in a
>> subclass of TestCase.
>
> Luckily we have categories in ObjC. And no one stops you putting this
> "lot of stuff" to a category of NSObject :-) But more importantly,
> there is no need to have a lot of stuff in TestCase class.
Yes, as I said that's a possibility. And then in order to keep your
files shorter you may want to put the testing categories in separate
files, and if you do so you gain nothing by having the tests in a
category instead of a subclass of TestCase. However you have lost some
flexibility, like the ability to store a state when you need it.
marco
Marco Scheurer
Sen:te, Lausanne, Switzerland http://www.sente.ch -
On Sep 26, 2004, at 01:41, Georg Tuparev wrote:
> On Sep 24, 2004, at 2:51 PM, Marco Scheurer wrote:
>>
>> Not really if you run your tests after each change, or frequently (at
>> least daily). A lot of our users actually prefer to keep the tests
>> separated from the code, in a different target or a different
>> project.
>
> Why?
>
>> One reason is to avoid deploying the test code and the testing
>> framework (although I think that in some cases it can be nice to be
>> able to run the tests at the customer's site).
>
> Some cases? Always! This saved us many debugging hours.
Yes, especially for custom software, having the possibility to ask your
customer to launch the application in test mode in his environment can
be helpful.
However, most of our users told us they wanted to be able to ship
without the tests and the testing infrastructure. This also make sense,
especially for "shrink-wrap": they are some issues with installing and
depending on a 3rd party framework.
So we leave that choice to our users.
> And users are in general impressed. When they see that your program
> has 3000 tests, they are really impressed! And because we test UI too,
> they think it is fun too - too see all these windows hopping around.
> BTW, we had one client who told us to keep the tests always on. This
> "helped him to understand better developers intentions"...
Funny, some say that you write tests to better understand the customer
intentions... ;-)
marco
Marco Scheurer
Sen:te, Lausanne, Switzerland http://www.sente.ch -
On Sep 26, 2004, at 13:01, Marco Scheurer wrote:
>> Here a typical test method:
>>
>> - (void)testVisibleWindow {
>> SEConfirmIsTrue(tableWindow != nil, @"Window should not be nil");
>> SEConfirmIsTrue([tableWindow isVisible],
>> @"Window should be visible after the application
>> is launched");
>> }
>>
>> UI testing made easy...
>
> I fail to see any differences with how this would be done using
> OCUnit, except the name of the macros.
Sorry to followup on my own post, but I hit reply to soon! Another
difference is of course that by writing the tests in a category, you
have access to the internal state of the class (like the ivar
tableWindow), and you do not need access methods...
Marco Scheurer
Sen:te, Lausanne, Switzerland http://www.sente.ch -
On 25 Sep 2004, at 16:32, Georg Tuparev wrote:
>
> Well, this gave me a lot of food for thinking. At that time I was
> already reading a very early draft of Kent's TDD book, and happy I was
> not - not because of the technique, but because of the technology...
> So after thinking about this for almost two months, finally I came to
> the conclusion that the problem is exactly the one you discovered -
> separation of TestCase from the class being tested [well, in reality
> this is not always true -- but please read my replays to Marco I
> intend to write later today for more].
Yes, this was my impression as well, at first simply a conjecture and a
code smell: parallel class hierarchies. Now that I've worked with
JUnit for well over a year, I am absolutely convinced that my decision
to allow the class to test itself for MPWTest was precisely right.
> At the time Drew and I was starting to put our thoughts about
> multi-volume text book on Cocoa programming (still work in [slow]
> progress), so we had a beer together and write down (on a A6 index
> card) the requirements for a testing framework to be publish in the
> book. Here what we scribbled:
> - the integration to any cocoa project (existing or new) should not
> take longer then a minute
> - one should learn the basics in max 5 mins
> - test could be placed anywhere (in TestCase or directly in the class).
Hmm...I guess I should (have) advertise(d) MPWTestKit a little more
heavily, because it has these attributes. There are hooks for more
flexibility if you need it, but the basics are absolutely trivial.
+testSelectors
{
return [NSArray
arrayWithObjects:@"myWonderfulTest1",@"myWonderfulTest2",nil];
}
+myWonderfulTest1
{
}
+myWonderfulTest2
{
}
[Automatic test-case discovery could be added, but I actually find the
explicit listing of my tests useful]
[snip more requirements]
> We ended up writing something that at first looked like a simplified
> version of xUnit. But is was not! Very soon I discovered that actually
> it is completely new beast. The difference is more on how it changed
> my way of testing - the flow. Now actually I can not call it testing
> any more ... it feels more like compiling, or debugging - a real
> integrated part of the development process. And it is part of the
> program you write (as it should be - I discovered this much later) -
> not part of the IDE as the original xUnit was supposed.
YES! That is exactly my experience with MPWTestKit. The tests are an
integral part of each class that is written. They are always there.
They do NOT produce a dependency on the unit test framework, which only
adds infrastructure to run the tests in a controlled/grouped fashion.
In addition to the tests being automatically associated by a class,
they are automatically grouped by framework. -
On Sep 26, 2004, at 1:01 PM, Marco Scheurer wrote:
>> [I am sure this might sound like Knuth's Literate Programming - in
>> retrospect I have to acknowledge that after Drew and I wrote the
>> first version of our testing framework, I though that this is the
>> closest practical way of doing literate programming for Cocoa]
>
> I said that much when talking about a "formal specification".
>
> In truth my point about writing tests first also has little to do with
> where they are implemented. You can of course start writing your class
> by writing the test methods. But it seems more natural to me to do so
> as an "outsider", the customer who specifies what the methods are
> supposed to do.
I hate split universes! It feels schizophrenic! If you by a book, you
have the text and all illustrations together. Imagine how it will feel
if they are separated in two books (three in the case of programming:
sources, tests, documentation)... But when I separate all these three
components, soon (after a month or so) I start feeling outsider.
Also I do not want to be an outsider to my own sources! Someone needs
to be an insider after all. I also feel strong about (collective) code
ownership, and I and my colleagues like to be proud about our craft. We
cannot be proud for something we feel like outsiders. And emotions
correlate strongly with productivity.
About your comment of "yet another do it simpler framework". You are
right to suggest that every framework evolves and gets more complex
with the time. The same will happen (already did) with our framework.
But, it evolves in different direction. Our test analysis gets more
sophisticated, and our UI nicer. But tests are just tests - very
simple. They will not get more clever. There is no TestCase, or
TestSuite , there is just no place where one can add bells and
whistles.
Georg Tuparev
Tuparev Technologies
Klipper 13
1186 VR Amstelveen
The Netherlands
Mobile: +31-6-55798196 -
On Sep 26, 2004, at 5:37, Marco Scheurer wrote:
> Another difference is of course that by writing the tests in a
> category, you have access to the internal state of the class (like the
> ivar tableWindow), and you do not need access methods...
A good suite of unit tests enables refactoring. However, if the tests
access internals of the production code, the tests are getting in the
way of refactoring (because they would always need to be changed
together with the production code). That's why I think ideally unit
tests should only test the public/official api, for which categories
are not necessary.
Besides, if all unit tests were in the form of categories, how would
one test a production code category?
Christian -
On Sep 26, 2004, at 5:59, Marcel Weiher wrote:
> On 25 Sep 2004, at 16:32, Georg Tuparev wrote:
>> finally I came to the conclusion that the problem is exactly the one
>> you discovered - separation of TestCase from the class being tested
>> [well, in reality this is not always true -- but please read my
>> replays to Marco I intend to write later today for more].
>
> Yes, this was my impression as well, at first simply a conjecture and
> a code smell: parallel class hierarchies. Now that I've worked with
> JUnit for well over a year, I am absolutely convinced that my decision
> to allow the class to test itself for MPWTest was precisely right.
Parallel class hierarchies are a code smell if both hierarchies are
production code. We're dealing with one hierarchy of production code
and another one of unit testing code. To me, that doesn't smell.
In my experience, for anything but trivial systems, these two class
hierarchies are not an exact mirror of each other. Often times I have
more than one test class per production class. These test classes test
different aspects of the production class and usually have different
setups. Sometimes I also have a testclass to test the interaction
between certain production classes. And other timed I don't have a test
class for a production class at all because I only test what could
possibly fail and what requires me to do TDD.
Christian -
[sorry for the smime-less repost]
On Sep 26, 2004, at 5:37, Marco Scheurer wrote:
> Another difference is of course that by writing the tests in a
> category, you have access to the internal state of the class (like the
> ivar tableWindow), and you do not need access methods...
A good suite of unit tests enables refactoring. However, if the tests
access internals of the production code, the tests are getting in the
way of refactoring (because they would always need to be changed
together with the production code). That's why I think ideally unit
tests should only test the public/official api, for which categories
are not necessary.
Besides, if all unit tests were in the form of categories, how would
one test a production code category?
Christian -
[sorry for the smime-less repost]
On Sep 26, 2004, at 5:59, Marcel Weiher wrote:
> On 25 Sep 2004, at 16:32, Georg Tuparev wrote:
>> finally I came to the conclusion that the problem is exactly the one
>> you discovered - separation of TestCase from the class being tested
>> [well, in reality this is not always true -- but please read my
>> replays to Marco I intend to write later today for more].
>
> Yes, this was my impression as well, at first simply a conjecture and
> a code smell: parallel class hierarchies. Now that I've worked with
> JUnit for well over a year, I am absolutely convinced that my decision
> to allow the class to test itself for MPWTest was precisely right.
Parallel class hierarchies are a code smell if both hierarchies are
production code. We're dealing with one hierarchy of production code
and another one of unit testing code. To me, that doesn't smell.
In my experience, for anything but trivial systems, these two class
hierarchies are not an exact mirror of each other. Often times I have
more than one test class per production class. These test classes test
different aspects of the production class and usually have different
setups. Sometimes I also have a testclass to test the interaction
between certain production classes. And other timed I don't have a test
class for a production class at all because I only test what could
possibly fail and what requires me to do TDD.
Christian -
On Sep 26, 2004, at 19:25, Christian Pekeler wrote:
> On Sep 26, 2004, at 5:37, Marco Scheurer wrote:
>> Another difference is of course that by writing the tests in a
>> category, you have access to the internal state of the class (like
>> the ivar tableWindow), and you do not need access methods...
>
> A good suite of unit tests enables refactoring. However, if the tests
> access internals of the production code, the tests are getting in the
> way of refactoring (because they would always need to be changed
> together with the production code). That's why I think ideally unit
> tests should only test the public/official api, for which categories
> are not necessary.
Oh yes, I agree totally. I did not see this as an advantage, just a
difference.
Marco Scheurer
Sen:te, Lausanne, Switzerland http://www.sente.ch -
On Sep 26, 2004, at 13:59, Marcel Weiher wrote:
> Hmm...I guess I should (have) advertise(d) MPWTestKit a little more
> heavily, because it has these attributes. There are hooks for more
> flexibility if you need it, but the basics are absolutely trivial.
>
> +testSelectors
> {
> return [NSArray
> arrayWithObjects:@"myWonderfulTest1",@"myWonderfulTest2",nil];
> }
>
> +myWonderfulTest1
> {
> }
> +myWonderfulTest2
> {
> }
Hey, now I have to disagree with not only Georg, but also Marcel!
Don't you think that the fact that these tests need to be in class
methods smelly? Of course you keep them "close" to the tested class,
but you still have an instance of one class testing an instance of
another class.
If the idea is to have both testing and tested code in the same file,
nothing forbid having two classes in the same file either.
And I don't buy the argument that writing tests in a category rather
than a class change the whole perspective on unit testing and how it
makes it so much more integrated in the development activity.
Of course OCUnit has TestCase and TesSuite, but that's an
implementation detail with which nobody needs to care if they don't
want to. They are not getting in the way of writing tests. For
instances TestSuite are automatically created to group the test cases
of different compilation units.
I'm a fan of categories, but in that case, I don't see the point.
Writing a test using OCUnit and, I suppose, many others is just as
trivial:
@implementation MyTestCase:TestCase
- (void) testWonder1
{
}
@end
One way or another the only required activities are to write test
methods and to run them.
Marco Scheurer
Sen:te, Lausanne, Switzerland http://www.sente.ch -
On Sep 26, 2004, at 8:23 PM, Marco Scheurer wrote:
>
> On Sep 26, 2004, at 19:25, Christian Pekeler wrote:
>
>> On Sep 26, 2004, at 5:37, Marco Scheurer wrote:
>>> Another difference is of course that by writing the tests in a
>>> category, you have access to the internal state of the class (like
>>> the ivar tableWindow), and you do not need access methods...
>>
>> A good suite of unit tests enables refactoring. However, if the tests
>> access internals of the production code, the tests are getting in the
>> way of refactoring (because they would always need to be changed
>> together with the production code). That's why I think ideally unit
>> tests should only test the public/official api, for which categories
>> are not necessary.
>
> Oh yes, I agree totally. I did not see this as an advantage, just a
> difference.
>
> Marco Scheurer
> Sen:te, Lausanne, Switzerland http://www.sente.ch
Wait a second folks. I thought the discussion was about Unit Tests and
not about Acceptance Tests. Besides, if I cannot tests internals of a
class, who will give me the courage to refactor these internals? Often
these internals make more then 90% of the class. For example a
Telescope class might have a public method isTracking that simply
returns a BOOL ivar set by an internal method called 10x each second
that that corrects the geometry of the flex mirror, reads a frame from
the helper CCD, does some nasty image convolution, calls eigen vectors
based RMS routine and passes the result to a weighting function that
finally decides if the telescope is still tracking or the object was
lost. For a user of the class Telescopes these might be unimportant
details, but for me - the author of the class - these are the most
important methods, and I would like to be able to test them no matter
if someone have academic or other objections...
BTW, a good suite of tests enables refactoring by first braking few
existing tests! If I make even a simple refactoring, say "rename
method", and my test bar is still green I know I am in trouble and I do
not have enough tests! When the tests are broken, I start fixing them.
When the bar is green again, I commit. If the bar stays red for longer
then an hour, I revert.
Georg Tuparev
Tuparev Technologies
Klipper 13
1186 VR Amstelveen
The Netherlands
Mobile: +31-6-55798196 -
On Sep 26, 2004, at 7:35 PM, Christian Pekeler wrote:
> In my experience, for anything but trivial systems, these two class
> hierarchies are not an exact mirror of each other. Often times I have
> more than one test class per production class. These test classes test
> different aspects of the production class and usually have different
> setups. Sometimes I also have a testclass to test the interaction
> between certain production classes. And other timed I don't have a
> test class for a production class at all because I only test what
> could possibly fail and what requires me to do TDD.
If you read carefully my initial email, you would be surprised to
discover that I am not saying that all test code should stay in the
class being tested. But it is my experience, that this feels most
naturally for about 70% of all cases. For these 70% of all tests, I do
not want to be forced to use TestCase subclasses, because of reasons I
explained in an earlier posting. For these 70% of the tests, it is also
TSTTCPW to keep them close to the code they test, and this is a good
thing!
If I have to chose between both extremes (xUnit - all tests re in
TestCase subclasses, and Marcel's way - non of them is in a TestCase
class) I would confidently choose MPWTestKit...
Georg Tuparev
Tuparev Technologies
Klipper 13
1186 VR Amstelveen
The Netherlands
Mobile: +31-6-55798196 -
On 26 Sep 2004, at 12:12, Marco Scheurer wrote:
> On Sep 26, 2004, at 01:41, Georg Tuparev wrote:
>>
>> Some cases? Always! This saved us many debugging hours.
>
> Yes, especially for custom software, having the possibility to ask
> your customer to launch the application in test mode in his
> environment can be helpful.
>
> However, most of our users told us they wanted to be able to ship
> without the tests and the testing infrastructure. This also make
> sense, especially for "shrink-wrap": they are some issues with
> installing and depending on a 3rd party framework.
Yes, that is precisely why MPWTest does not create a dependency on the
test-framework. You can leave your tests in the code without having to
ship the testing framework.
> So we leave that choice to our users.
There is always choice: you can put the categories in a separate file,
lib or framework and ship without it. You can also create a subclass
for your tests, if you want to. But you don't have to, and that's very
important, IMHO.
Marcel -
On 26 Sep 2004, at 12:01, Marco Scheurer wrote:
> On Sep 26, 2004, at 01:33, Georg Tuparev wrote:
>>> One reason is that tests can have their own state and ivars, and it
>>> wouldn't be a good idea to mix this with the class you want to test.
>>
>> You might be right is theory, but at least I and all the people I
>> enjoyed working with almost never kept state between two tests.
>> Actually, if I think very hard, I cannot find a single case where my
>> need of using state was not because of stinky code - or bad design,
>> or bad refactoring.
>
> I admit that it is not frequent, and one could certainly go away
> without that possibility.
Precisely. So you should create a design that allows for the
infrequent case, but caters to the frequent case.
>
>>> Another is that a lot of stuff is usually done in the TestCase
>>> class, so that writing a test is as simple as writing a single
>>> method in a subclass of TestCase.
>>
>> Luckily we have categories in ObjC. And no one stops you putting this
>> "lot of stuff" to a category of NSObject :-) But more importantly,
>> there is no need to have a lot of stuff in TestCase class.
>
> Yes, as I said that's a possibility. And then in order to keep your
> files shorter you may want to put the testing categories in separate
> files, and if you do so you gain nothing by having the tests in a
> category instead of a subclass of TestCase. However you have lost some
> flexibility, like the ability to store a state when you need it.
The opposite is true. You do not lose any flexibility, because you
always *can* create a new class if you want to, with all the state you
require. You just don't *have to* create a new class, for all those
times where you don't need it. -
On 26 Sep 2004, at 00:33, Georg Tuparev wrote:
> In our testing framework, we create an instance of the tested class
> for every single test method.
You do? Automatically by the test framework? I tend to leave this up
to the test method in question (which will be a class method of the
class under test unless you configure it differently). -
On Sep 27, 2004, at 1:20 AM, Marcel Weiher wrote:
>> In our testing framework, we create an instance of the tested class
>> for every single test method.
>
> You do? Automatically by the test framework? I tend to leave this up
> to the test method in question (which will be a class method of the
> class under test unless you configure it differently).
TSTTCPW as usual. Most of the time our default behavior is all we need.
For the exceptions we override the setUp method...
Georg Tuparev
Tuparev Technologies
Klipper 13
1186 VR Amstelveen
The Netherlands
Mobile: +31-6-55798196 -
On 26 Sep 2004, at 21:57, Georg Tuparev wrote:
> On Sep 26, 2004, at 7:35 PM, Christian Pekeler wrote:
>
>> In my experience, for anything but trivial systems, these two class
>> hierarchies are not an exact mirror of each other. Often times I have
>> more than one test class per production class. These test classes
>> test different aspects of the production class and usually have
>> different setups. Sometimes I also have a testclass to test the
>> interaction between certain production classes. And other timed I
>> don't have a test class for a production class at all because I only
>> test what could possibly fail and what requires me to do TDD.
>
> If you read carefully my initial email, you would be surprised to
> discover that I am not saying that all test code should stay in the
> class being tested. But it is my experience, that this feels most
> naturally for about 70% of all cases. For these 70% of all tests, I do
> not want to be forced to use TestCase subclasses, because of reasons I
> explained in an earlier posting. For these 70% of the tests, it is
> also TSTTCPW to keep them close to the code they test, and this is a
> good thing!
>
> If I have to chose between both extremes (xUnit - all tests re in
> TestCase subclasses, and Marcel's way - non of them is in a TestCase
> class) I would confidently choose MPWTestKit...
Too much honor heaped on me: MPWTest isn't nearly that extreme. It
does allow, and I do use, completely separate test-classes. They are
just rather rare, the majority of tests do not requie, and thus do not
get, a separate class.
Cheers,
Marcel
--
Marcel Weiher Metaobject Software Technologies
<marcel...> www.metaobject.com
Metaprogramming for the Graphic Arts. HOM, IDEAs, MetaAd etc.
1d480c25f397c4786386135f8e8938e4 -
On 26 Sep 2004, at 19:57, Marco Scheurer wrote:
> On Sep 26, 2004, at 13:59, Marcel Weiher wrote:
>
>> Hmm...I guess I should (have) advertise(d) MPWTestKit a little more
>> heavily, because it has these attributes. There are hooks for more
>> flexibility if you need it, but the basics are absolutely trivial.
>>
>> +testSelectors
>> {
>> return [NSArray
>> arrayWithObjects:@"myWonderfulTest1",@"myWonderfulTest2",nil];
>> }
>>
>> +myWonderfulTest1
>> {
>> }
>> +myWonderfulTest2
>> {
>> }
>
> Hey, now I have to disagree with not only Georg, but also Marcel!
>
> Don't you think that the fact that these tests need to be in class
> methods smelly?
No. Also, they don't *need* to be in class methods. You can, if you
want to, override the +testFixture method to return any object you
please, and that will then be the test-fixture for that class.
However, the default is the class itself and I hardly every found
myself sufficiently motivated to change it.
> Of course you keep them "close" to the tested class, but you still
> have an instance of one class testing an instance of another class.
Huh? I have the class testing instance of itself. This seems
perfectly reasonable and has caused me absolutely no problems. Why do
you say an instance of one class testing an instance of another class?
Do you mean an instance of the metaclass??
> If the idea is to have both testing and tested code in the same file,
> nothing forbid having two classes in the same file either.
Sure. And if there is some reason why I *do* need a subclass to test,
then I do precisely that, for example when testing a category of a
class that already has tests. I could also do it if I actually needed
extra state.
So it's YAGNI and DTSTTCPW: yes, the things you state *can* happen,
but they are trivial to deal with when they do and for the 90% of the
time that they are not needed there is a simpler solution.
> And I don't buy the argument that writing tests in a category rather
> than a class change the whole perspective on unit testing and how it
> makes it so much more integrated in the development activity.
I wouldn't buy it from a purely theoretical point of view either, but
after a little more than 1 year of JUnit (after a number of years with
MPWTest), I can say that it has definitely been the case. YMMV.
> Of course OCUnit has TestCase and TesSuite, but that's an
> implementation detail with which nobody needs to care if they don't
> want to. They are not getting in the way of writing tests. For
> instances TestSuite are automatically created to group the test cases
> of different compilation units.
Yes, MPWTest creates both TestSuites and TestCases automatically
(though you could interfere and get your own grouping).
> I'm a fan of categories, but in that case, I don't see the point.
> Writing a test using OCUnit and, I suppose, many others is just as
> trivial:
>
> @implementation MyTestCase:TestCase
> - (void) testWonder1
> {
> }
> @end
Now I have to wonder: you complained above about an instance of one
class testing an instance of a different class (which wasn't the case),
and this is precisely the case here. How is MyTestCase related to the
class being tested?
> One way or another the only required activities are to write test
> methods and to run them.
Not quite. At the very least, you need to link in the testing
framework.
Marcel
--
Marcel Weiher Metaobject Software Technologies
<marcel...> www.metaobject.com
Metaprogramming for the Graphic Arts. HOM, IDEAs, MetaAd etc.
1d480c25f397c4786386135f8e8938e4 -
On 27 Sep 2004, at 00:33, Georg Tuparev wrote:
>
> On Sep 27, 2004, at 1:20 AM, Marcel Weiher wrote:
>
>>> In our testing framework, we create an instance of the tested class
>>> for every single test method.
>>
>> You do? Automatically by the test framework? I tend to leave this
>> up to the test method in question (which will be a class method of
>> the class under test unless you configure it differently).
>
> TSTTCPW as usual. Most of the time our default behavior is all we
> need. For the exceptions we override the setUp method...
Hmm...I have to say I don't see how this is simple. It is very
automatic and convenient, but not exactly simple. How does the unit
test framework know what message to send to get an instance of a class?
+new?
I am guessing you have a convention, and am intrigued to hear that you
say it is working well.
Cheers,
Marcel
--
Marcel Weiher Metaobject Software Technologies
<marcel...> www.metaobject.com
Metaprogramming for the Graphic Arts. HOM, IDEAs, MetaAd etc.
1d480c25f397c4786386135f8e8938e4 -
On 26 Sep 2004, at 18:43, Christian Pekeler wrote:
> [sorry for the smime-less repost]
>
> On Sep 26, 2004, at 5:59, Marcel Weiher wrote:
>> On 25 Sep 2004, at 16:32, Georg Tuparev wrote:
>>> finally I came to the conclusion that the problem is exactly the one
>>> you discovered - separation of TestCase from the class being tested
>>> [well, in reality this is not always true -- but please read my
>>> replays to Marco I intend to write later today for more].
>>
>> Yes, this was my impression as well, at first simply a conjecture and
>> a code smell: parallel class hierarchies. Now that I've worked with
>> JUnit for well over a year, I am absolutely convinced that my
>> decision to allow the class to test itself for MPWTest was precisely
>> right.
>
> Parallel class hierarchies are a code smell if both hierarchies are
> production code.
Says who??
> We're dealing with one hierarchy of production code and another one of
> unit testing code. To me, that doesn't smell.
To me it does. Because I still have to build/maintain both hierarchies.
> In my experience, for anything but trivial systems, these two class
> hierarchies are not an exact mirror of each other.
Well, that might be because it is difficult to keep them in sync. When
I use JUnit, I also have this experience. With MPWTest, I do not,
because keeping the tests with the code under test is trivially easy.
I find the latter much more natural and easy.
> Often times I have more than one test class per production class.
> These test classes test different aspects of the production class and
> usually have different setups.
Hmm..when I have different setups, I don't have to create separate
classes. I just factor out a different setup method in my test code.
It seems pretty obvious that certain rigidities in your unit testing
setup are forcing you to introduce entities that aren't strictly
necessary.
> Sometimes I also have a testclass to test the interaction between
> certain production classes.
I then tend to have an object that handles the interaction between
those production classes. However, nothing at all prevents me from
adding a test-class if I feel the need for it. It is just that I don't
have to do it if there is no need, and I rarely feel the need for it.
That I *can* put tests in categories of the class under test does *not*
imply that I *must* do so.
> And other timed I don't have a test class for a production class at
> all because I only test what could possibly fail and what requires me
> to do TDD.
Then how did the class get written, under TDD? When you look at that
particular class, how do you find the tests for it?
To me, almost all this looks like symptoms of *having* to maintain a
parallel class hierarchies, and not being able to keep them in sync.
Why you would think of them as *good* things puzzles me.
Cheers,
Marcel -
On Sep 27, 2004, at 1:47 AM, Marcel Weiher wrote:
>
>> Of course you keep them "close" to the tested class, but you still
>> have an instance of one class testing an instance of another class.
>
> Huh? I have the class testing instance of itself. This seems
> perfectly reasonable and has caused me absolutely no problems. Why do
> you say an instance of one class testing an instance of another class?
> Do you mean an instance of the metaclass??
Yes, you've got method of the class object testing methods of the
instance. These tests are no more internal than when implemented in
another class. In both cases we probably start by instantiating an
instance of the tested class. The testing code that needs to be written
is the same.
>> And I don't buy the argument that writing tests in a category rather
>> than a class change the whole perspective on unit testing and how it
>> makes it so much more integrated in the development activity.
>
> I wouldn't buy it from a purely theoretical point of view either, but
> after a little more than 1 year of JUnit (after a number of years with
> MPWTest), I can say that it has definitely been the case. YMMV.
Not fair. Comparing testing with Junit and testing with OCUnit is a bit
like comparing programming in Java and programming in Objective C. I
don't like JUnit either.
>> I'm a fan of categories, but in that case, I don't see the point.
>> Writing a test using OCUnit and, I suppose, many others is just as
>> trivial:
>>
>> @implementation MyTestCase:TestCase
>> - (void) testWonder1
>> {
>> }
>> @end
>
> Now I have to wonder: you complained above about an instance of one
> class testing an instance of a different class (which wasn't the
> case), and this is precisely the case here. How is MyTestCase related
> to the class being tested?
I was not complaining, but pointing out that having tests as class
methods of the tested class or as instance methods of another class is
not fundamentally different.
MyTestCase is a client of the tested class. It includes its interface.
Again there is no difference, if you decide to move your category in
other files, you would also need to include relevant headers.
>> One way or another the only required activities are to write test
>> methods and to run them.
>
> Not quite. At the very least, you need to link in the testing
> framework.
Well sure, and your tests run when you don't link to your framework?
Please note that I've said since the beginning that not subclassing
TestCase was also possibility. And I'm not saying that it is
necessarily worse, but that the claims that doing so makes the testing
activity so much better, simpler, easier and enjoyable are wrong in my
opinion.
marco -
>>> Of course you keep them "close" to the tested class, but you still
>>> have an instance of one class testing an instance of another class.
>>
>> Huh? I have the class testing instance of itself. This seems
>> perfectly reasonable and has caused me absolutely no problems. Why
>> do you say an instance of one class testing an instance of another
>> class? Do you mean an instance of the metaclass??
>
> Yes, you've got method of the class object testing methods of the
> instance.
Yes, that is the default case, with which I have no problems so far.
If one would want to have the test methods as instance, all one would
have to do is override +testFixture to return an instance. So far, I
haven't found the need for this.
> These tests are no more internal than when implemented in another
> class.
I didn't claim they were, except for organizational purposes.
> In both cases we probably start by instantiating an instance of the
> tested class. The testing code that needs to be written is the same.
It is very similar, but not the same. Again, I didn't claim there were
any significant differences. You just don't need an extra class, one
that is not related to the class under test.
>
>>> And I don't buy the argument that writing tests in a category rather
>>> than a class change the whole perspective on unit testing and how it
>>> makes it so much more integrated in the development activity.
>>
>> I wouldn't buy it from a purely theoretical point of view either, but
>> after a little more than 1 year of JUnit (after a number of years
>> with MPWTest), I can say that it has definitely been the case. YMMV.
>
> Not fair. Comparing testing with Junit and testing with OCUnit is a
> bit like comparing programming in Java and programming in Objective C.
> I don't like JUnit either.
I was referring to one specific aspect where OCUnit matches JUnit.
What do you not like about JUnit?
Save that that feature, I find JUnit fine (apart from having to work in
Java in the first place, but that is a completely separate issue).
So I don't think your objection is warranted.
>
> I was not complaining, but pointing out that having tests as class
> methods of the tested class or as instance methods of another class is
> not fundamentally different.
I have found that it is significantly different, because you are
leaving something out that is unnecessary and gets in the way.
> MyTestCase is a client of the tested class. It includes its interface.
> Again there is no difference, if you decide to move your category in
> other files, you would also need to include relevant headers.
This is not true. First, there still is a difference, because a
category is still part of the class. Second, I usually don't decide to
do that.
>>> One way or another the only required activities are to write test
>>> methods and to run them.
>>
>> Not quite. At the very least, you need to link in the testing
>> framework.
>
> Well sure, and your tests run when you don't link to your framework?
Yes. It is the test-tool that links to the test framework. It then
just loads the framework under test and everything is fine.
> Please note that I've said since the beginning that not subclassing
> TestCase was also possibility. And I'm not saying that it is
> necessarily worse, but that the claims that doing so makes the testing
> activity so much better, simpler, easier and enjoyable are wrong in my
> opinion.
Is your opinion backed by experience with both styles?
Marcel
--
Marcel Weiher Metaobject Software Technologies
<marcel...> www.metaobject.com
Metaprogramming for the Graphic Arts. HOM, IDEAs, MetaAd etc.
1d480c25f397c4786386135f8e8938e4 -
On 26 sep 2004, at 22.44, Georg Tuparev wrote:
> BTW, a good suite of tests enables refactoring by first braking few
> existing tests!
Your interpretation of the notion "refactoring" must be very different
than my interpretation. To me, refactoring is about changing the
internal structure of some code without changing its external behavior.
That is, a good suite of tests enables refactoring by adequately
covering the external behavior that one wants to preserve.
In my experience, testing the internals of a class (in the sense of
testing internal state by examining instance variables, etc) turns the
whole process of unit testing into a slow-flowing blob of syrup.
Personally, I'm moving farther and farther away from this kind if
testing.
To me, unit testing is a design technique that enables me to work
faster. That the internal state of one object is represented by an
intricate mixture of booleans isn't important (it might smell though,
which is important :-). What is important however, is that my test
suite--should need rise--allows me to switch to another implementation
without being afraid of breaking anything.
Regards,
Malte
--
Malte Tancred
Computer Programmer
Oops AB, http://oops.se/ -
On Sep 27, 2004, at 3:57, Marcel Weiher wrote:
> On 26 Sep 2004, at 18:43, Christian Pekeler wrote:
>> On Sep 26, 2004, at 5:59, Marcel Weiher wrote:
>>> On 25 Sep 2004, at 16:32, Georg Tuparev wrote:
>>>> finally I came to the conclusion that the problem is exactly the
>>>> one you discovered - separation of TestCase from the class being
>>>> tested [well, in reality this is not always true -- but please read
>>>> my replays to Marco I intend to write later today for more].
>>>
>>> Yes, this was my impression as well, at first simply a conjecture
>>> and a code smell: parallel class hierarchies. Now that I've worked
>>> with JUnit for well over a year, I am absolutely convinced that my
>>> decision to allow the class to test itself for MPWTest was precisely
>>> right.
>>
>> Parallel class hierarchies are a code smell if both hierarchies are
>> production code.
>
> Says who??
What? You mean if I say something, it doesn't carry enough authority
for you? ;) OK, I restate as: In my personal experience, I have never
perceived parallel class hierarchies as a code smell if one hierarchy
is for production code and the other for unit tests.
Parallel class hierarchies indicate too tight coupling, duplication, a
lack of abstraction and usually result in hard to change code. Since
tight coupling is the very purpose of unit testing (as we want to test
every public method of a specific class), and since we usually want to
test concrete implementations instead of abstractions, I'm of the
opinion that parallel class hierarchies are not a problem in the
context of unit testing.
>> We're dealing with one hierarchy of production code and another one
>> of unit testing code. To me, that doesn't smell.
>
> To me it does. Because I still have to build/maintain both
> hierarchies.
If you have to maintain a test category instead of a test class per
production class, I fail to see a significant difference for practical
purposes.
>> In my experience, for anything but trivial systems, these two class
>> hierarchies are not an exact mirror of each other.
>
> Well, that might be because it is difficult to keep them in sync.
I'm usually not even trying to keep them in sync. They just happen to
start out looking parallel in the beginning. Nothing good or bad about
this, IMHO.
>> Often times I have more than one test class per production class.
>> These test classes test different aspects of the production class and
>> usually have different setups.
>
> Hmm..when I have different setups, I don't have to create separate
> classes. I just factor out a different setup method in my test code.
Sure, that's another good way of doing this. I've recently met the
author of the upcoming XUnit Patterns book. He told me that he doesn't
like the one setUp/tearDown per test class approach because they're
invoked implicitly and therefore make the individual test methods
harder to read. I don't mind the implicit invocation as it makes my
test code shorter, but every now and then have explicit setUps as well,
especially when the test methods in a test class all have different
requirements.
>> And other timed I don't have a test class for a production class at
>> all because I only test what could possibly fail and what requires me
>> to do TDD.
>
> Then how did the class get written, under TDD? When you look at that
> particular class, how do you find the tests for it?
It can happen as a result of refactoring. For example I create a common
super class for two existing classes that have some form of code
duplication, then I usually don't create a test for the new superclass
until I have to. Or if instances of class A create instance of class B,
and B happens to be a simple container, then I usually just have a
simple test for B in the testclass for A. When working with Apple's
frameworks, I find TDD is not always practical. For example, I don't
usually test my controllers (but try to keep them as small and simple
as possible).
> To me, almost all this looks like symptoms of *having* to maintain a
> parallel class hierarchies, and not being able to keep them in sync.
> Why you would think of them as *good* things puzzles me.
I didn't mean to imply that I find them good, they just don't bother me.
I think we're splitting hairs. None of the different point of views in
this thread (not just in this mail) justify a big argument in my
opinion. I think it's great that we have more than one unit testing
solutions and I'm very interested in seeing how they're different. I'm
a bit disappointed that only the usual suspects are writing in this
thread, indicating that unit testing is still not a mainstream activity
in this community. I'm looking forward to the day that Apple releases
all their frameworks and libraries with the sources of the
corresponding unit tests and a community which wouldn't accept anything
less.
Christian -
> Parallel class hierarchies indicate too tight coupling, duplication, a
> lack of abstraction and usually result in hard to change code.
Being somewhat of a simpleton, I see parallel class hierarchies as the
same thing in two places, and thus a trivial violation of
OnceAndOnlyOnce. A lot of the other problems come as a result of this.
> Since tight coupling is the very purpose of unit testing (as we want
> to test every public method of a specific class),
So why not attach the test-code directly to the code to be tested?
Especially if it is less work to do so (the coupling is fully
automatic) and we have reduced our entities ( Occam's Razor, DTSTTCPW )
> and since we usually want to test concrete implementations instead of
> abstractions, I'm of the opinion that parallel class hierarchies are
> not a problem in the context of unit testing.
Two objections: first, your argument seems inconsistent. On the one
hand, you state that you want tight coupling, on the other hand you
don't perceive the fact that the hierarchies do get out of sync as a
problem. Second, my practical experience with both styles leads me to
conclude that it is a problem, at least relative to the solution
without quasi-parallel hierarchies.
>>> We're dealing with one hierarchy of production code and another one
>>> of unit testing code. To me, that doesn't smell.
>>
>> To me it does. Because I still have to build/maintain both
>> hierarchies.
>
> If you have to maintain a test category instead of a test class per
> production class, I fail to see a significant difference for practical
> purposes.
You don't *have* to maintain a category, it is just useful for
documentation purposes. The practical difference is that the tests are
directly, unambiguously and simply attached to the class to be tested.
So you have fewer entities in your program/framework. Less to think
about. More chunking, more abstraction...good. One very simple
benefit: if you view unit tests as a form of documentation, it is
obvious where to find the unit tests for a particular class. With
xUnit style, this is far from obvious.
Also, a class is a somewhat more heavyweight entity than a mere
category: it occupies the global namespace, you have to give it a
name, etc. It also has to be a subclass of TestCase and therefore
implies a link-time + runtime dependency on the unit testing framework.
But really, I think this boils down to you being both quite accustomed
to xUnit style, and not having practical experience with the other
style. Try to step back for a moment, and maybe consider what Georg
said about how different programmers reacted to their unit testing
framework (which I do not know). It was the ones with xUnit experience
that had the most difficulty.
>>> In my experience, for anything but trivial systems, these two class
>>> hierarchies are not an exact mirror of each other.
>>
>> Well, that might be because it is difficult to keep them in sync.
>
> I'm usually not even trying to keep them in sync.
I thought you want them to be highly coupled? How do you find the
tests that belong to a class?
> They just happen to start out looking parallel in the beginning.
> Nothing good or bad about this, IMHO.
I beg to differ.
>
>>> Often times I have more than one test class per production class.
>>> These test classes test different aspects of the production class
>>> and usually have different setups.
>>
>> Hmm..when I have different setups, I don't have to create separate
>> classes. I just factor out a different setup method in my test code.
>
> Sure, that's another good way of doing this. I've recently met the
> author of the upcoming XUnit Patterns book. He told me that he doesn't
> like the one setUp/tearDown per test class approach because they're
> invoked implicitly and therefore make the individual test methods
> harder to read.
Yep, that's also what I found. I want tests to be as obvious as
possible.
>> To me, almost all this looks like symptoms of *having* to maintain a
>> parallel class hierarchies, and not being able to keep them in sync.
>> Why you would think of them as *good* things puzzles me.
>
> I didn't mean to imply that I find them good, they just don't bother
> me.
Compared to not having unit testing: full agreement. However, coming
from MPWTest I found the separate class hierarchies of tests required
by JUnit to be bothersome.
> I think we're splitting hairs.
Hmm...I agree that you are splitting hairs, trying to pooh pooh
something that you haven't actually used that (a) is simpler and (b)
easier to use than existing unit testing. The differences may appear
small to you in theory, but I find them to be actually significant in
practice, having actually used both styles and having found xUnit style
lacking (though still absolutely superior to not unit testing).
> None of the different point of views in this thread (not just in this
> mail) justify a big argument in my opinion.
So why are you arguing?
> I think it's great that we have more than one unit testing solutions
> and I'm very interested in seeing how they're different. I'm a bit
> disappointed that only the usual suspects are writing in this thread,
> indicating that unit testing is still not a mainstream activity in
> this community.
If you go back to Georg's post, and actually also to Uli's, you will
see that the arguments presented are directly connected with trying to
make unit testing even simpler and more straightforward, which can only
help the cause.
When I came upon xUnit, I found that I didn't get it. There seemed all
this extra complexity/baggage that just baffled me, especially coming
out of YAGNI-land :-) It seemed to me that it could be a lot simpler,
especially in Objective-C. And it turns out that it *could* be a lot
simpler and more comfortable to boot.
And it seems I am not the only one baffled:
On 23 Sep 2004, at 21:51, M. Uli Kusterer wrote:
>
> E.g. UnitKit keeps separate "test objects", which I find kinda odd.
> I'm wondering why one shouldn't put the tests in the object in need of
> the testing instead... things like that.
Just because you are sufficiently clever that things like that don't
baffle you, doesn't mean that the rest of us are as well. So, if you
want more unit testing, help make it easier!
Unit testing does not *have* to be in xUnit style. In fact, the
TDD-gurus wholeheartedly recommend writing your own unit testing
framework.
> I'm looking forward to the day that Apple releases all their
> frameworks and libraries with the sources of the corresponding unit
> tests and a community which wouldn't accept anything less.
That would be great, but I am not entirely convinced that Apple
frameworks have been or are developed in TDD style...
Marcel
--
Marcel Weiher Metaobject Software Technologies
<marcel...> www.metaobject.com
Metaprogramming for the Graphic Arts. HOM, IDEAs, MetaAd etc.
1d480c25f397c4786386135f8e8938e4 -
> Wait a second folks. I thought the discussion was about Unit Tests and
> not about Acceptance Tests. Besides, if I cannot tests internals of a
> class, who will give me the courage to refactor these internals? Often
> these internals make more then 90% of the class. For example a
> Telescope class might have a public method isTracking that simply
> returns a BOOL ivar set by an internal method called 10x each second
> that that corrects the geometry of the flex mirror, reads a frame from
> the helper CCD, does some nasty image convolution, calls eigen vectors
> based RMS routine and passes the result to a weighting function that
> finally decides if the telescope is still tracking or the object was
> lost. For a user of the class Telescopes these might be unimportant
> details, but for me - the author of the class - these are the most
> important methods, and I would like to be able to test them no matter
> if someone have academic or other objections...
Now I'm not a heavywight in UnitTesting as some of you are, but I
always took such occasions as a sign that one class is doing too much
and that I should add a small class that encapsulates exactly these
internals. So the Telescope merely becomes some sort of Facade to the
real functionality.
Well, now that I think about it I'm a big fan of very little classes.
Though I'm still lacking lot's of experience.
After thinking some more: It seems to me that if you have test-classes
and source separated, it kind of enforces the use of many very small
classes. Integrating the Tests in the Classes on the other hand enables
them to do more - as it's still convenient to test.
Now I'm not sure which of the goals are more desirable to me or if
there is a middle ground to go with. But this seems to make the
engineering tradeoff more clear. Do you agree or am I overlooking
something?
> BTW, a good suite of tests enables refactoring by first braking few
> existing tests! If I make even a simple refactoring, say "rename
> method", and my test bar is still green I know I am in trouble and I
> do not have enough tests! When the tests are broken, I start fixing
> them. When the bar is green again, I commit. If the bar stays red for
> longer then an hour, I revert.
<g> When I rename a Method and a Test breaks I rather start swearing at
my stupid tools that weren't good enough to rename the method call in
the test too. </g>
Is there some sign on the horizon that we finally will be getting
SmallTalk or Eclipse Level Refactoring support? Right now the
Objective-C Tools are _very_ week. :(
cu Martin
