Asp Forum - Embedded vs. Non-embedded Tests

Trans

9/4/2007 6:43:00 PM

As far as I know, Facets is the only large project that uses embedded
unit tests. The tests are housed in block comments, eg.

=begin test
...
=end

I have a command line tool that can run the embedded test directly.
For deployment, I have another tool that extracts all the embedded
tests and copies them to stand-alone test/ files so others can run
them. It works fairly well. And the advantage of course is that the
unit tests are right there with the libs they test.

However, considering that just about everyone else is putting units
test in their on test files, I'm wondering if I actually missing out
on superior advantages to that approach. If not why haven't others
pursued better tools for supporting embedded tests?

Thanks,
T.

12 Answers

Rick DeNatale

9/4/2007 7:07:00 PM

On 9/4/07, Trans <transfire@gmail.com> wrote:
> As far as I know, Facets is the only large project that uses embedded
> unit tests. The tests are housed in block comments, eg.
>
> =begin test
> ...
> =end
>
> I have a command line tool that can run the embedded test directly.
> For deployment, I have another tool that extracts all the embedded
> tests and copies them to stand-alone test/ files so others can run
> them. It works fairly well. And the advantage of course is that the
> unit tests are right there with the libs they test.
>
> However, considering that just about everyone else is putting units
> test in their on test files, I'm wondering if I actually missing out
> on superior advantages to that approach. If not why haven't others
> pursued better tools for supporting embedded tests?

I think that you're swimming against the tide.

Personally, it just feels right to me to separate the test code from
the code under test, most test equipment belongs in the garage instead
of the trunk of the car.

And when it gets beyond just unit testing individual classes, it's not
apparent to me which file an embedded test should call home.

From a practical tool perspective, there are lots of things which
expect a relationship between code files and test files. Two which
spring immediately to mind are autotest and rcov.

--
Rick DeNatale

My blog on Ruby
http://talklikeaduck.denh...

Rick DeNatale

9/4/2007 8:30:00 PM

On 9/4/07, Peter Cooper <peter@peterc.org> wrote:
> On 9/4/07, Rick DeNatale <rick.denatale@gmail.com> wrote:
> >
> > Personally, it just feels right to me to separate the test code from
> > the code under test, most test equipment belongs in the garage instead
> > of the trunk of the car.
>
>
> A lot of diagnostics now takes place in modern cars, however. Tire pressure
> checks, engine wear, oil, etc. Sure, you'd only work and make changes to the
> car in the garage, but a lot of the "testing" is a hidden ongoing day to day
> process.

For the most part, that's instrumentation, not testing.

In the case where sensor information is used to affect the operation
of the vehicle, such as
engine managment, abs, or traction control, it's no more testing
equipment than carburettors, brakes, and master cylinders. Yes, data
gets logged, but that's not testing, any more than a server log file
is.

> And if you get onto, say, Formula 1 cars.. some "repairs" can even take
> place on the road and the telemetry provides almost as much info to the
> engineers while the car is running out on the track as they can get in the
> garage. I've always wondered if/how to make software as easy to diagnose
> with real-time telemetry and the ability to make changes to a running
> program's parameters..

Yes, but most the testing equipment is not in the car, the engineers
in the pits have laptops, and the heavy stuff is in places like Woking
or Maranello, connected through the 'internets.'

--
Rick DeNatale

My blog on Ruby
http://talklikeaduck.denh...

Trans

9/4/2007 8:57:00 PM

On Sep 4, 12:07 pm, "Rick DeNatale" <rick.denat...@gmail.com> wrote:
> On 9/4/07, Trans <transf...@gmail.com> wrote:
>
>
>
> > As far as I know, Facets is the only large project that uses embedded
> > unit tests. The tests are housed in block comments, eg.
>
> > =begin test
> > ...
> > =end
>
> > I have a command line tool that can run the embedded test directly.
> > For deployment, I have another tool that extracts all the embedded
> > tests and copies them to stand-alone test/ files so others can run
> > them. It works fairly well. And the advantage of course is that the
> > unit tests are right there with the libs they test.
>
> > However, considering that just about everyone else is putting units
> > test in their on test files, I'm wondering if I actually missing out
> > on superior advantages to that approach. If not why haven't others
> > pursued better tools for supporting embedded tests?
>
> I think that you're swimming against the tide.
>
> Personally, it just feels right to me to separate the test code from
> the code under test, most test equipment belongs in the garage instead
> of the trunk of the car.

Interesting analogy. We do put some gages right on the dash. But
that's more like embedding the tests directly into code --eg. rescue
and try clauses. So maybe you're analogy is a good one. I can see some
reasons for doing so, it certainly make it easier to send some one a
test, that they can try against their own implementation, for
instance. But then I start to wonder about API documentation too.
Should that also be separated out into dedicated files? Why tests but
not docs?

> And when it gets beyond just unit testing individual classes, it's not
> apparent to me which file an embedded test should call home.

That's true. Embedded tests only apply for unit test. Integration
tests still need to be is separate files anyway.

> From a practical tool perspective, there are lots of things which
> expect a relationship between code files and test files. Two which
> spring immediately to mind are autotest and rcov.

That's part of what I was wondering. I've never tried these tools. I
guess I've always suspected they'd be too magical to introduce into an
already established project, and thus lead to more trouble than
they're worth --though I'm sure they're quite useful if one starts
using them from the get-go. Am I wrong about that? But even so,
couldn't such tools be adapted to handle embedded tests too?

T.

Rick DeNatale

9/4/2007 9:17:00 PM

On 9/4/07, Trans <transfire@gmail.com> wrote:
>
>
> On Sep 4, 12:07 pm, "Rick DeNatale" <rick.denat...@gmail.com> wrote:

> > Personally, it just feels right to me to separate the test code from
> > the code under test, most test equipment belongs in the garage instead
> > of the trunk of the car.
>
> Interesting analogy. We do put some gages right on the dash. But
> that's more like embedding the tests directly into code --eg. rescue
> and try clauses. So maybe you're analogy is a good one. I can see some
> reasons for doing so, it certainly make it easier to send some one a
> test, that they can try against their own implementation, for
> instance. But then I start to wonder about API documentation too.
> Should that also be separated out into dedicated files? Why tests but
> not docs?

Well , it probably seems more obvious to most to put the doc with the
code (assuming that there is any doc <G>). Note that we've just now
seemed to get tools like dcov.

It's long been pretty standard process to have tools which extract doc
from code. On the other hand there are some arguments for keeping
even these two separate. In "The Psychology of Computer Programming"
Gerry Weinberg talked about the trap of "debugging" the comments
instead of the code, and advocated tools which would not just extract
code commentary but REMOVE it.

On the other hand without having the doc with the code and tools like
rdoc we'd probably have even less documentation than we do now, just a
guess.

--
Rick DeNatale

My blog on Ruby
http://talklikeaduck.denh...

Terry Poulin

9/4/2007 10:45:00 PM

Rick DeNatale wrote:
> On 9/4/07, Trans <transfire@gmail.com> wrote:
>>
>>
>> On Sep 4, 12:07 pm, "Rick DeNatale" <rick.denat...@gmail.com> wrote:
>
>> > Personally, it just feels right to me to separate the test code from
>> > the code under test, most test equipment belongs in the garage instead
>> > of the trunk of the car.
>>
>> Interesting analogy. We do put some gages right on the dash. But
>> that's more like embedding the tests directly into code --eg. rescue
>> and try clauses. So maybe you're analogy is a good one. I can see some
>> reasons for doing so, it certainly make it easier to send some one a
>> test, that they can try against their own implementation, for
>> instance. But then I start to wonder about API documentation too.
>> Should that also be separated out into dedicated files? Why tests but
>> not docs?
>
> Well , it probably seems more obvious to most to put the doc with the
> code (assuming that there is any doc <G>). Note that we've just now
> seemed to get tools like dcov.
>
> It's long been pretty standard process to have tools which extract doc
> from code. On the other hand there are some arguments for keeping
> even these two separate. In "The Psychology of Computer Programming"
> Gerry Weinberg talked about the trap of "debugging" the comments
> instead of the code, and advocated tools which would not just extract
> code commentary but REMOVE it.
>
> On the other hand without having the doc with the code and tools like
> rdoc we'd probably have even less documentation than we do now, just a
> guess.
>
> --
> Rick DeNatale
>
> My blog on Ruby
> http://talklikeaduck.denh...
>
>

One thing I am a fan of is reading over my changes at the end of a session and
updating to-do lists, especially when it reminds me clean up my comments and
rewrite quick-fixes to proper code the next day ;-)

It's generally my opinion that the comments tell you what it does and the code
how (with comments for the less obvious). If the comments fall out of sync
with the code, the comments should be updated once the code is less likely to
change to much. Personally I think documentation about using the program
should be separate from the regular source code. Documentation about working
on the program and it's sources should be kept near by. Just how much, is a
matter of developers tastes.

I advocate people using common sense, that is to us the gray matter
approximately three feet above their romp roast.

TerryP.

--

Email and shopping with the feelgood factor!
55% of income to good causes. http://www.ip...

Stefan Rusterholz

9/5/2007 9:14:00 AM

Peter Cooper wrote:
> And if you get onto, say, Formula 1 cars.. some "repairs" can even take
> place on the road and the telemetry provides almost as much info to the
> engineers while the car is running out on the track as they can get in
> the
> garage. I've always wondered if/how to make software as easy to diagnose
> with real-time telemetry and the ability to make changes to a running
> program's parameters..

Take a look at Eiffels Design By Contract. I'm a huge fan of it. Allows
you to move most testing code into the contract, the unit tests are then
merely running your code with contracts enabled.
For an idea on how that works, take a look at the movies with "Design by
Contract" in the title here: http://eiffel.com/developers/pres...
Would be a very nice feature if ruby had that, even though I'd introduce
additional syntax and I've no idea how to integrate it nicely.

Regards
Stefan
--
Posted via http://www.ruby-....

Alex Young

9/5/2007 9:24:00 AM

Stefan Rusterholz wrote:
> Peter Cooper wrote:
>> And if you get onto, say, Formula 1 cars.. some "repairs" can even take
>> place on the road and the telemetry provides almost as much info to the
>> engineers while the car is running out on the track as they can get in
>> the
>> garage. I've always wondered if/how to make software as easy to diagnose
>> with real-time telemetry and the ability to make changes to a running
>> program's parameters..
>
> Take a look at Eiffels Design By Contract. I'm a huge fan of it. Allows
> you to move most testing code into the contract, the unit tests are then
> merely running your code with contracts enabled.
> For an idea on how that works, take a look at the movies with "Design by
> Contract" in the title here: http://eiffel.com/developers/pres...
> Would be a very nice feature if ruby had that, even though I'd introduce
> additional syntax and I've no idea how to integrate it nicely.
Allow me to shamelessly copy and paste from an old post of mine:

$ cat testable.rb
module Test; module Unit; class TestCase; end; end; end

module Testable
def test_case(&block)
Class.new(Test::Unit::TestCase).module_eval &block
end
end

$ cat builtin_tests.rb
require 'testable'

class MyWorkingClass
extend Testable

def foo(a,b)
return a+b
end

test_case {
def test_foo_null
assert_equal 2, MyWorkingClass.new.foo(1,1)
end
}
end

if __FILE__ == $0
puts MyWorkingClass.new.foo(1,1)
end

$ ruby builtin_tests.rb
2
$ testrb builtin_tests.rb
Loaded suite builtin_tests.rb
Started

Stefan Rusterholz

9/5/2007 10:07:00 AM

Alex Young wrote:
> Allow me to shamelessly copy and paste from an old post of mine:
>
> $ cat testable.rb
> module Test; module Unit; class TestCase; end; end; end
>
> module Testable
> def test_case(&block)
> Class.new(Test::Unit::TestCase).module_eval &block
> end
> end
>
> $ cat builtin_tests.rb
> require 'testable'
>
> class MyWorkingClass
> extend Testable
>
> def foo(a,b)
> return a+b
> end
>
> test_case {
> def test_foo_null
> assert_equal 2, MyWorkingClass.new.foo(1,1)
> end
> }
> end
>
> if __FILE__ == $0
> puts MyWorkingClass.new.foo(1,1)
> end
>
> $ ruby builtin_tests.rb
> 2
> $ testrb builtin_tests.rb
> Loaded suite builtin_tests.rb
> Started
> .
> Finished in 0.00095 seconds.
>
> 1 tests, 1 assertions, 0 failures, 0 errors
> $
>
>
> Something like that?

No, the whole approach is different. I'd say the movies I linked to can
explain it far better than I can.
With DBC, if contracts are enabled (in Eiffel you can disable contracts
to get better performance), your method is tested everytime you call the
method. It tests pre-conditions and/or post-conditions (providing you
the old state for your post-conditions). Also after every call to a
method the class invariant is tested to validate your object. That's the
basic idea.

One way that could probably be implemented (though, only as of ruby 1.9
due to &block params) might look this:

class DayOfTime
extend DBC
invariant {
@hour.between?(0,23)
@min.between?(0,59)
@sec.between?(0,59)
}

define(:min=) {
pre_condition { |value|
value.kind_of?(Numeric)
value.between?(0,59)
}
body { |value|
@value = value
}
post_condition { |old, value|
@hour == old.hour
@min == value
@sec == old.second
}
}
end

Regards
Stefan
--
Posted via http://www.ruby-....

Trans

9/5/2007 10:27:00 AM

On Sep 5, 2:13 am, Stefan Rusterholz <apei...@gmx.net> wrote:
> Peter Cooper wrote:
> > And if you get onto, say, Formula 1 cars.. some "repairs" can even take
> > place on the road and the telemetry provides almost as much info to the
> > engineers while the car is running out on the track as they can get in
> > the
> > garage. I've always wondered if/how to make software as easy to diagnose
> > with real-time telemetry and the ability to make changes to a running
> > program's parameters..
>
> Take a look at Eiffels Design By Contract. I'm a huge fan of it. Allows
> you to move most testing code into the contract, the unit tests are then
> merely running your code with contracts enabled.
> For an idea on how that works, take a look at the movies with "Design by
> Contract" in the title here:http://eiffel.com/developers/pres...
> Would be a very nice feature if ruby had that, even though I'd introduce
> additional syntax and I've no idea how to integrate it nicely.

http://jroller.com/dscataglini/entry/jim_weirich_test_drivern_d...

T.

ThoML

9/5/2007 10:40:00 AM

> Would be a very nice feature if ruby had that, even though I'd introduce
> additional syntax and I've no idea how to integrate it nicely.

There are a few libraries around that try to do this (google for "ruby
dbc" or similar). Most of these efforts are not entirely convincing.
Unfortunately, the ruby community seems somewhat dogmatic in arguing
against dbc. Maybe because it's not enough of a duck.

comp.lang.ruby

Embedded vs. Non-embedded Tests

Trans

Rick DeNatale

Rick DeNatale

Trans

Rick DeNatale

Terry Poulin

Stefan Rusterholz

Alex Young

Stefan Rusterholz

Trans

ThoML

x Login to ForumsZone