Asp Forum - How to write a test with random usage patterns?

Hallvard B Furuseth

7/27/2015 12:48:00 PM

I want to test a database by throwing random usage patterns
at it. Maybe like this, but replacing rand() with something
which exhibits random patterns in the output:

for (...)
switch (anti_rand() % CASES) {
case 0: foo(); break;
case 1: bar(randomly chosen data); break;
...
}

That is, anti_rand() might output random numbers for a while, then
switch to (X,Y,Z, X,Y,Z ...) for a while, and so on.

The point is to avoid coding up some particular patterns like
"write descending keys for a while", and instead have anti_rand()
cause that. What's a good way to write anti_rand()? How to use
it well? Or should I ask, what should I goolge for? I expect
someone has dealt with this problem before.

--
Hallvard

10 Answers

LudovicoVan

7/27/2015 2:20:00 PM

On Monday, July 27, 2015 at 1:48:18 PM UTC+1, Hallvard Breien Furuseth wrote:

> switch (anti_rand() % CASES) {

That is not a good approach: if the range of anti_rand is not an exact multiple of the number of cases, you get that some cases are hit more frequently than others. But you'll have to clarify about anti_rand before I get to suggest a better approach:

> That is, anti_rand() might output random numbers for a while, then
> switch to (X,Y,Z, X,Y,Z ...) for a while, and so on.

And so on what? Is it supposed to generate random numbers or not? Indeed, why call it anti_rand and not just rand? And why not just use some library function? (It is utterly non-trivial to code a decent random number generator.)

> I expect someone has dealt with this problem before.

I do use random input for some tests, it may be useful to automatically generate a large number of test cases with varying input: the only drawback is that tests are not repeatable unless you specify a seed for the random number generator.

Julio

LudovicoVan

7/27/2015 2:42:00 PM

On Monday, July 27, 2015 at 1:48:18 PM UTC+1, Hallvard Breien Furuseth wrote:
> I want to test a database by throwing random usage patterns
> at it. Maybe like this, but replacing rand() with something
> which exhibits random patterns in the output:
>
> for (...)
> switch (anti_rand() % CASES) {
> case 0: foo(); break;
> case 1: bar(randomly chosen data); break;
> ...
> }

Also, on a more general note, while you do not want to write and maintain millions of tests, tests that are too complex (too coarse-grained) are ineffective, i.e. when they fail you do not get enough information to pin down the problem...

Julio

Hallvard B Furuseth

7/27/2015 3:00:00 PM

Julio Di Egidio <julio@diegidio.name> writes:
> On Monday, July 27, 2015 at 1:48:18 PM UTC+1, Hallvard Breien Furuseth wrote:

[Rearranging a bit]

>> That is, anti_rand() might output random numbers for a while, then
>> switch to (X,Y,Z, X,Y,Z ...) for a while, and so on.
>
> And so on what? Is it supposed to generate random numbers or not?
> Indeed, why call it anti_rand and not just rand? And why not just use
> some library function? (It is utterly non-trivial to code a decent
> random number generator.)

I do *not* want a random number generator. I want a random _pattern_
generator. See the Subject header. Generate some pattern and output
that, then another, and so on. Some regular patterns, some half-
randomized. Or something like that.

>> switch (anti_rand() % CASES) {
> That is not a good approach: if the range of anti_rand is not an exact
> multiple of the number of cases, you get that some cases are hit more
> frequently than others. But you'll have to clarify about anti_rand
> before I get to suggest a better approach:

Not quite, but the program must be adapted to anti_rand(), yes.

>> I expect someone has dealt with this problem before.
>
> I do use random input for some tests, it may be useful to
> automatically generate a large number of test cases with varying
> input: the only drawback is that tests are not repeatable unless you
> specify a seed for the random number generator.

I expect to output the seed at the beginning of the test, and make the
program deterministic after picking the seed. Or offer that as an
option, anyway.

--
Hallvard

LudovicoVan

7/27/2015 3:43:00 PM

On Monday, July 27, 2015 at 4:00:22 PM UTC+1, Hallvard Breien Furuseth wrote:

> I do *not* want a random number generator. I want a random _pattern_
> generator. See the Subject header. Generate some pattern and output
> that, then another, and so on. Some regular patterns, some half-
> randomized. Or something like that.

You are making no sense, and "something like that" just isn't good enough as a spec.

> >> switch (anti_rand() % CASES) {
> > That is not a good approach: if the range of anti_rand is not an exact
> > multiple of the number of cases, you get that some cases are hit more
> > frequently than others. But you'll have to clarify about anti_rand
> > before I get to suggest a better approach:
>
> Not quite, but the program must be adapted to anti_rand(), yes.

Not quite what? That is plain arithmetic.

You just don't know what you are talking about, do you? Good luck with that...

Julio

Hallvard B Furuseth

7/28/2015 10:54:00 AM

Julio Di Egidio <julio@diegidio.name> writes:
> (blah)
>> Not quite, but the program must be adapted to anti_rand(), yes.
>
> Not quite what? That is plain arithmetic.

That's plain arithmetic about a random number generator, which
I keep saying is *not* want I want to generate.

> You just don't know what you are talking about, do you? Good luck with that...

Idiot. Go away.

--
Hallvard

Hallvard B Furuseth

7/28/2015 10:56:00 AM

I wrote:
> I want to test a database by throwing random usage patterns
> at it. Maybe like this, but replacing rand() with something
> which exhibits random patterns in the output:
> (...)

So, I don't want to code 20 usage patterns like this:

switch (rand() % 20) {
case 0: Do N writes in descending order; break;
....
}

when I instead can get 4*5 usage patterns like this:

switch (anti_rand() % 4) {
case 0: write(smaller item than last time); break;
....
}

unsigned anti_rand() {
switch (foo % 5) {
case 0: return the same number N times, then change foo;
...
}
}

That's still pretty unimaginative, though. So I wonder how to write
anti_rand() to produce different patterns - mixtures of order and
randomness - to guide the test program. Or what to google for,
since I hope someone has done this before.

E.g. maybe instead of returning the same number N times, it'll return
different numbers but with the same bottom bits each time for a while,
and specify that usage like above should use COUNT = power of 2.

And I'll need to know how to best use it, but I expect that must wait
until I know what anti_rand() is doing. E.g. whether to use '% COUNT'
or something else. Or instead of a "write smaller item than last time"
case, maybe write data from another anti_rand() stream and leave it to
anti_rand() to sometimes return descending values for a while.

--
Hallvard

Richard Heathfield

7/28/2015 11:27:00 AM

On 27/07/15 13:48, Hallvard Breien Furuseth wrote:> I want to test a
database by throwing random usage patterns
> at it.

Find a way to express a test case as a sequence of bits, such that *any*
sequence of bits represents a valid test case. Then generate a few
hundred random bit sequences. Test them. If possible, identify the
bitstrings that detect bugs, and flag them. Keep any that have the flag,
and throw the rest out. Then generate new test cases as follows:

1) take two test cases at random;
2) take a random number of bits from the first test case;
3) take a random number of bits from the second test case, and catenate;
4) repeat from 2) until you have a valid test case;
5) randomly flip a small but random number of bits.

Then go round again, and again, and again.

Evolution in action.

--
Richard Heathfield
Email: rjh at cpax dot org dot uk
"Usenet is a strange place" - dmr 29 July 1999
Sig line 4 vacant - apply within

LudovicoVan

7/28/2015 12:20:00 PM

On Tuesday, July 28, 2015 at 12:26:47 PM UTC+1, Richard Heathfield wrote:

> Find a way to express a test case as a sequence of bits

What test cases? What kind of testing is the OP trying to do?

> Then go round again, and again, and again.

Any convolution can be made to boil back down to just an encoding then a random number generator. But the problem remains that even that is senseless: to generate random test cases ("usage scenarios") one ends up encoding some state machine, not just a list of pure actions.

> Evolution in action.

Wishful thinking: not even for load testing one generates statements at random...

Julio

Hallvard B Furuseth

7/28/2015 1:04:00 PM

Richard Heathfield <rjh@cpax.org.uk> writes:
> Find a way to express a test case as a sequence of bits, such that *any*
> sequence of bits represents a valid test case. Then generate a few
> hundred random bit sequences. Test them. If possible, identify the
> bitstrings that detect bugs, and flag them. Keep any that have the flag,
> and throw the rest out. Then generate new test cases as follows:

That sounds pretty cool once we've found a bug. Until then though, it
seems to be just the same as "switch (rand() % COUNT)" - there'll be no
patterns in the bit strings beyond random noise. So the bit strings
don't help generate interesting test cases. That is, I'm hoping to
decompose "test cases" in small steps, then assemble them in more
interesting ways than just random order would achieve.

> 1) take two test cases at random;
> 2) take a random number of bits from the first test case;
> 3) take a random number of bits from the second test case, and catenate;
> 4) repeat from 2) until you have a valid test case;

I gather there's no need for (4) here, since you said any bit string is
a valid test case.

> 5) randomly flip a small but random number of bits.
> Then go round again, and again, and again.
>
> Evolution in action.

--
Hallvard

Richard Heathfield

7/28/2015 4:17:00 PM

On 28/07/15 14:03, Hallvard Breien Furuseth wrote:
> Richard Heathfield <rjh@cpax.org.uk> writes:
>> Find a way to express a test case as a sequence of bits, such that *any*
>> sequence of bits represents a valid test case. Then generate a few
>> hundred random bit sequences. Test them. If possible, identify the
>> bitstrings that detect bugs, and flag them. Keep any that have the flag,
>> and throw the rest out. Then generate new test cases as follows:
>
> That sounds pretty cool once we've found a bug.

Good point. :-) In the meantime, you'd need a less finicky fitness
function, wouldn't you?

> Until then though, it
> seems to be just the same as "switch (rand() % COUNT)" - there'll be no
> patterns in the bit strings beyond random noise.

At first, yes. This is typical of genetic algorithms - the interesting
behaviour takes a while to emerge.

> So the bit strings
> don't help generate interesting test cases.

I think you're correct at the start, but I also think they will help to
generate /more/ interesting test cases (once you have some that are even
mildly interesting), if you get your fitness function right.

> That is, I'm hoping to
> decompose "test cases" in small steps, then assemble them in more
> interesting ways than just random order would achieve.
>
>> 1) take two test cases at random;
>> 2) take a random number of bits from the first test case;
>> 3) take a random number of bits from the second test case, and catenate;
>> 4) repeat from 2) until you have a valid test case;
>
> I gather there's no need for (4) here, since you said any bit string is
> a valid test case.

Just making sure it's long enough, that's all - I assume (and it may not
be necessary to assume, but probably simpler if you do) that a test case
consists of precisely N bits.

--
Richard Heathfield
Email: rjh at cpax dot org dot uk
"Usenet is a strange place" - dmr 29 July 1999
Sig line 4 vacant - apply within

comp.programming

How to write a test with random usage patterns?

Hallvard B Furuseth

LudovicoVan

LudovicoVan

Hallvard B Furuseth

LudovicoVan

Hallvard B Furuseth

Hallvard B Furuseth

Richard Heathfield

LudovicoVan

Hallvard B Furuseth

Richard Heathfield

x Login to ForumsZone