[lnkForumImage]
TotalShareware - Download Free Software
Confronta i prezzi di migliaia di prodotti.
Asp Forum
 Home | Login | Register | Search 


 

Forums >

comp.lang.ruby

How to use this Combinatorial testing technique

Surya

9/11/2008 5:56:00 AM

Hi Friends,

I have been in to testing for 3 yrs. In my project there are 6000 test
cases present and some of them ought to be non-value adding to the
testing effort. Around 80% of the total test cases were automated and
would like to run execersie to reduce the un important test cases from
the pool. Can any one suggest how to go about this and apply
Combinatorial/pairwise testing effectively to my situation.

TIA,

Regards,
Surya
12 Answers

Robert Klemme

9/11/2008 7:11:00 AM

0

2008/9/11 Surya <surya5kn@gmail.com>:
> I have been in to testing for 3 yrs. In my project there are 6000 test
> cases present and some of them ought to be non-value adding to the
> testing effort. Around 80% of the total test cases were automated and
> would like to run execersie to reduce the un important test cases from
> the pool. Can any one suggest how to go about this and apply
> Combinatorial/pairwise testing effectively to my situation.

It's not totally clear to me what you want. First, how do you define
an "unimportant" test case?

Then I am tempted to ask "why bother?". The only reason for reduction
I can think of is runtime of the test suite - or maybe your
requirements change so often so dramatically that you have to touch a
lot of those tests. Otherwise I do not really see the benefit. Can
you elaborate a bit more?

Kind regards

robert

--
use.inject do |as, often| as.you_can - without end

Caleb Clausen

9/11/2008 11:42:00 PM

0

On 9/10/08, Surya <surya5kn@gmail.com> wrote:
> Hi Friends,
>
> I have been in to testing for 3 yrs. In my project there are 6000 test
> cases present and some of them ought to be non-value adding to the
> testing effort. Around 80% of the total test cases were automated and
> would like to run execersie to reduce the un important test cases from
> the pool. Can any one suggest how to go about this and apply
> Combinatorial/pairwise testing effectively to my situation.

I suffer from an excess of test cases as well, but I don't have a
ready solution.
I have heard before of tools to help identify redundant test cases,
but none for
use with ruby.

I imagine such a tool would work like this:
1) run each test case by itself within a coverage analyzer, to get a
list of lines executed.
2) now work thru the list of lists of lines covered per test case,
eliminating test cases where each line covered by that case is covered
also in some other case. (A number of algorithms are possible here.)

Writing a tool like that wouldn't be too hard. Getting it to be
efficient might be challenging...

Of course there's a problem with this; line coverage is less than
perfect. So if there's a
line whose first half is tested by one test case, and the second half
tested by another
test case, the second case may be flagged as redundant when it isn't.
Expression
coverage is the gold standard when it comes to code coverage. So
ideally, you'd write the
tool above to work expression by expression rather than line by line.
There are no expression-based coverage tools for ruby that I know of.

Even expression coverage will only give you complete code coverage;
data coverage is
another matter. Is there a tool or metric for assessing data coverage?
I have no idea.

ara.t.howard

9/12/2008 12:22:00 AM

0


On Sep 11, 2008, at 5:41 PM, Caleb Clausen wrote:

> On 9/10/08, Surya <surya5kn@gmail.com> wrote:
>> Hi Friends,
>>
>> I have been in to testing for 3 yrs. In my project there are 6000
>> test
>> cases present and some of them ought to be non-value adding to the
>> testing effort. Around 80% of the total test cases were automated and
>> would like to run execersie to reduce the un important test cases
>> from
>> the pool. Can any one suggest how to go about this and apply
>> Combinatorial/pairwise testing effectively to my situation.
>
> I suffer from an excess of test cases as well, but I don't have a
> ready solution.
> I have heard before of tools to help identify redundant test cases,
> but none for
> use with ruby.
>
> I imagine such a tool would work like this:
> 1) run each test case by itself within a coverage analyzer, to get a
> list of lines executed.
> 2) now work thru the list of lists of lines covered per test case,
> eliminating test cases where each line covered by that case is covered
> also in some other case. (A number of algorithms are possible here.)
>
> Writing a tool like that wouldn't be too hard. Getting it to be
> efficient might be challenging...
>
> Of course there's a problem with this; line coverage is less than
> perfect. So if there's a
> line whose first half is tested by one test case, and the second half
> tested by another
> test case, the second case may be flagged as redundant when it isn't.
> Expression
> coverage is the gold standard when it comes to code coverage. So
> ideally, you'd write the
> tool above to work expression by expression rather than line by line.
> There are no expression-based coverage tools for ruby that I know of.
>
> Even expression coverage will only give you complete code coverage;
> data coverage is
> another matter. Is there a tool or metric for assessing data coverage?
> I have no idea.
>

path to insanity if you ask me:


cover_this = Array.new(10000){|i| define_method("foo_#{ i }"){ i } }



a @ http://codeforp...
--
we can deny everything, except that we have the possibility of being
better. simply reflect on that.
h.h. the 14th dalai lama

Trans

9/12/2008 2:52:00 AM

0



On Sep 11, 1:53=A0am, Surya <surya...@gmail.com> wrote:
> Hi Friends,
>
> I have been in to testing for 3 yrs. In my project there are 6000 test
> cases present and some of them ought to be non-value adding to the
> testing effort. Around 80% of the total test cases were automated and
> would like to run execersie to reduce the un important test cases from
> the pool. Can any one suggest how to go about this and apply
> Combinatorial/pairwise testing effectively to my situation.

Hi Surya,

I have a solution for you. And recently ported the code into the Turn
project, however I've been unable to get Tim Pease to say one way or
the other if he will accept the new code. I will try to get up with
him one more time. I hoping he will, but if not, I'll put it out as a
new project.

T.

Caleb Clausen

9/12/2008 7:03:00 PM

0

On 9/11/08, ara.t.howard <ara.t.howard@gmail.com> wrote:
>
> On Sep 11, 2008, at 5:41 PM, Caleb Clausen wrote:
>> Even expression coverage will only give you complete code coverage;
>> data coverage is
>> another matter. Is there a tool or metric for assessing data coverage?
>> I have no idea.
>>
>
> path to insanity if you ask me:
>
>
> cover_this = Array.new(10000){|i| define_method("foo_#{ i }"){ i } }

Obviously complete data coverage is out of the question, for almost
all cases. But your coverage of data doesn't have to be complete in
order to be sufficient. Usually, you want to show that you're covering
typical cases, as well as hitting all the corners and edges. But how
do you know you've covered all those edges? That's what I'm wondering
about. Is there some way to assess (for any language, tho ruby would
be nice) how close you're getting to covering a set of data which is
suitably representative of your entire set of possible data?

Lloyd Linklater

9/12/2008 7:19:00 PM

0

This seems tough to answer without knowing more about the code. If the
test cases have something in common, e.g. they descend from a common
test_case class, then you could add something there that notes every
time the test case is called.

Still, it should be mentioned that this does not answer for all. You do
not say what constitutes "value added". My first thought was that, if
it never gets called then it is not adding much, but that is not the
whole of the story.

Not knowing how it is written or what the criteria for value
determination is puts me somewhat at a loss. If you can add these
things, it would help.
--
Posted via http://www.ruby-....

ara.t.howard

9/12/2008 7:58:00 PM

0


On Sep 12, 2008, at 1:02 PM, Caleb Clausen wrote:

> Obviously complete data coverage is out of the question, for almost
> all cases. But your coverage of data doesn't have to be complete in
> order to be sufficient. Usually, you want to show that you're covering
> typical cases, as well as hitting all the corners and edges. But how
> do you know you've covered all those edges? That's what I'm wondering
> about. Is there some way to assess (for any language, tho ruby would
> be nice) how close you're getting to covering a set of data which is
> suitably representative of your entire set of possible data?


i think our minds are quite good at this - we have to select the data,
write the coverage tools, the testing tools, the ci tools, and the
code itself after all. clearly software we write cannot be more
effective that the people writing it.

it may sound heresy to some people on this list, and elsewhere, but i
think dynamic languages like ruby require a, possibly additional (on
top of tdd, bdd, whatever), rigorous process of *thought* and
*discussion* to really build good code. mechanical analysis is simply
one way to approach coverage.

a nice article summarizing what i'm saying

http://michaelfeathers.typepad.com/michael_feathers_blog/2008/06/the-flawed...

it's simply worth considering that 3 people in a room *now* maybe
quite easily be able to achieve what current tools cannot, and may
never be able to.

ymmv.

a @ http://codeforp...
--
we can deny everything, except that we have the possibility of being
better. simply reflect on that.
h.h. the 14th dalai lama

James Gray

9/12/2008 8:07:00 PM

0

On Sep 12, 2008, at 2:58 PM, ara.t.howard wrote:

> it may sound heresy to some people on this list, and elsewhere, but
> i think dynamic languages like ruby require a, possibly additional
> (on top of tdd, bdd, whatever), rigorous process of *thought* and
> *discussion* to really build good code. mechanical analysis is
> simply one way to approach coverage.

You are still my hero. I just wanted to make sure I've told you
lately. ;)

James Edward Gray II

ara.t.howard

9/12/2008 9:37:00 PM

0


On Sep 12, 2008, at 2:07 PM, James Gray wrote:

> You are still my hero. I just wanted to make sure I've told you
> lately. ;)
>
> James Edward Gray II

lol - coming from you that's a real compliment.... i heard your talk
at lone star was great.

cheers.

a @ http://codeforp...
--
we can deny everything, except that we have the possibility of being
better. simply reflect on that.
h.h. the 14th dalai lama

James Gray

9/12/2008 9:57:00 PM

0

On Sep 12, 2008, at 4:36 PM, ara.t.howard wrote:

>
> On Sep 12, 2008, at 2:07 PM, James Gray wrote:
>
>> You are still my hero. I just wanted to make sure I've told you =20
>> lately. ;)
>>
>> James Edward Gray II
>
> lol - coming from you that's a real compliment.... i heard your talk =20=

> at lone star was great.

You inspired that too. It was about NArray, RBtree, SQLite, FSDB, =85 =
;)

James Edward Gray II=