DevEX - reference for building teams, processes, and platforms
Metadata-driven Testing
1. Why you ever meta data you didn’t like:
Testing twisty little passages all alike.
Steven Lembark
Workhorse Computing
lemark@wrkhors.com
2. What is testing?
Smoke, white- & black-box, integration, regression...
Set up controlled environment.
See if software fails.
Lather, rinse repeat...
Controls include data, environment, handlers.
3. The usual way: Template tests
Write a test.
Make it work.
Copy it.
Edit it.
Copy it.
Edit it.
Copy...
4. “Red flags”
Cut + paste
Works until you find a bug...
Need a change...
Update 45 files.
5. Common result: A mess
Single directory.
Semi-random basenames.
Difficult to run in order.
6. First piece of metadata
Filesystem is powerful medicine.
Cures all sorts of problems.
8. Basenames
prove uses lexical order.
Might not be what you
want.
100-perlcritic.t
101-perltidy.t
102-pod.t
103-podt.t
10-init.t
200-player-TestRoo.t
201-TestRoo-Action.t
201-TestRoo-Actor.t
201-TestRoo-Adventure.t
201-TestRoo-Base.t
201-TestRoo-Item.t
201-TestRoo-Location.t
201-TestRoo-Player.t
202-TestClassMoose.t
20-player.t
300-black-hole.t
60-command.t
9. Basenames
prove uses lexical order.
Might not be what you
want.
Use consistent names.
010-init.t
020-player.t
060-command.t
100-perlcritic.t
101-perltidy.t
102-pod.t
103-podt.t
200-player-TestRoo.t
201-TestRoo-Action.t
201-TestRoo-Actor.t
201-TestRoo-Adventure.t
201-TestRoo-Base.t
201-TestRoo-Item.t
201-TestRoo-Location.t
201-TestRoo-Player.t
202-TestClassMoose.t
300-black-hole.t
10. Directories
Need every test every time?
010-init.t
020-player.t
060-command.t
100-perlcritic.t
101-perltidy.t
102-pod.t
103-podt.t
200-player-TestRoo.t
201-TestRoo-Action.t
201-TestRoo-Actor.t
201-TestRoo-Adventure.t
201-TestRoo-Base.t
201-TestRoo-Item.t
201-TestRoo-Location.t
201-TestRoo-Player.t
202-TestClassMoose.t
300-black-hole.t
11. Directories
Need every test every time?
What about:
Quick tests for check-in.
Complete for release.
010-init.t
020-player.t
060-command.t
100-perlcritic.t
101-perltidy.t
102-pod.t
103-podt.t
200-player-TestRoo.t
201-TestRoo-Action.t
201-TestRoo-Actor.t
201-TestRoo-Adventure.t
201-TestRoo-Base.t
201-TestRoo-Item.t
201-TestRoo-Location.t
201-TestRoo-Player.t
202-TestClassMoose.t
300-black-hole.t
12. Directories
Need every test every time?
What about:
prove && commit;
prove -r && release;
010-init.t
020-player.t
060-command.t
100-perlcritic.t
101-perltidy.t
102-pod.t
103-podt.t
200-Modules
300-Missions
13. Directories
Need every test every time?
What about:
prove && commit;
prove -r && release;
01-init.t
02-player.t
06-command.t
10-perlcritic.t
11-perltidy.t
12-pod.t
13-podt.t
20-Modules
30-Missions
14. Directories
Need every test every time?
What about:
prove && commit;
prove -r && release;
01-init.t
02-player.t
06-command.t
10-perlcritic.t
11-perltidy.t
12-pod.t
13-podt.t
20-Modules
30-Missions
15. Directories
Need every test every time?
Baseline & config checks.
prove t/0*t;
01-init.t
02-player.t
06-command.t
10-perlcritic.t
11-perltidy.t
12-pod.t
13-podt.t
20-Modules
30-Missions
16. Directories
Need every test every time?
Simple integration tests:
prove -v t/1*t;
01-init.t
02-player.t
06-command.t
10-perlcritic.t
11-perltidy.t
12-pod.t
13-podt.t
20-Modules
30-Missions
17. Tests don’t need to be stupid.
Adding a little logic avoids copying.
Lazy: Write once, recycle in place.
18. Choose your mission
Defines the map, monsters, goal.
The mission files start with:
my $pkg = 'Adventure';
my $path = 't/etc/EmptyMap00.yaml';
19. Choose your mission
Defines the map, monsters, goal.
One set of tests starts with:
Same tests for any config file.
my $pkg = 'Adventure';
my $path = 't/etc/EmptyMap00.yaml';
20. Choose your mission
Defines the map, monsters, goal.
Don’t copy the file:
my $pkg = 'Adventure';
my $base = basename $0, ‘.t’;
my $path = “t/etc/$base.yaml”;
21. Choose your mission
Defines the map, monsters, goal.
Don’t copy the file:
Symlink them all to “./bin/01-mission_t”.
my $pkg = 'Adventure';
my $base = basename $0, ‘.t’;
my $path = “t/etc/$base.yaml”;
22. Choose your mission
cd $(dirname $0)/../20-Missions;
for i in ../etc/*-mission.yaml;
do
base=$(basename $i .yaml);
ln -fs ../bin/01-mission_t ./01-$base.t;
done
One “test” for many configs:
23. Generic tests: well-formed Perl
“use_ok” is unfairly maligned.
It does something quite useful.
Better off if all modules pass it.
24. Generic tests: well-formed Perl
“use_ok” is unfairly maligned.
It does something quite useful.
Better off if all modules pass it.
And have a working package name.
With a version.
25. Generic tests: well-formed Perl
use Test::More;
use_ok ‘Frobnicate’;
can_ok Frobnicate => ‘VERSION’;
ok Frobnicate->VERSION,
“Frobnicate has a version”;
done_testing;
__END__
26. Generic tests: well-formed Perl
use Test::More;
use_ok ‘Frobnicate’;
can_ok Frobnicate => ‘VERSION’;
ok Frobnicate->VERSION,
“Frobnicate has a version”;
done_testing;
__END__
27. Generic tests: well-formed Perl
use Test::More;
my $package= ‘Frobnicate’;
use_ok $package;
can_ok $package => ‘VERSION’;
ok $frobnicate->VERSION,
“$package has a version”;
done_testing;
32. Testing groups of files
Runs same test on all modules:
prove 10-*.t ;
Test class & children:
prove *-Parent-*.t;
find t -name $glob | xargs prove;
33. Combine with commits
Git tags mark prove success.
Combine prove with git tags to track progress:
prove && git tag …
Merge tags into main branch for Q/A.
Use “prove -r” in master for Q/A pass.
34. Generic tests: well formed configs.
Ever fat-finger some JSON?
Leave out an XML tag?
Mis-quote an .ini?
35. Generic tests: config files.
Ever fat-finger some JSON?
Leave out an XML tag?
Mis-quote an .ini?
Then waste debugging code to find it?
36. Generic tests: config files
To the rescue: Config::Any.
If its read-able, we can check it.
At least for readabiliy...
37. Generic tests: config files
Same basic trick: symlink a reader.
my $base = basename $0;
my @partz= split /W+/, $base;
my $test = join ‘~’ => ‘01’, @partz;
symlink ‘../bin/00-config_t’ =>
$test . ‘.t’;
38. #!/bin/bash
cd $(dirname $0)/..; # run from ./t/bin
rm -f 0*.t; # remove generic tests
i='-1';
for glob in '*.yaml' '*.pm' # test config & modules
do
export j="0$((++i))";
echo "Pass: $j";
ls ./bin/$j-*_t;
find .. -name $glob |
perl -n
-E 'state $path = ( glob "./bin/$ENV{j}-*_t" )[0];'
-E 'chomp;'
-E 'my @a = split m{[/]}, substr $_, 3;'
-E 'my $b = join "-", @a;'
-E 'symlink $path => "$ENV{j}-$b.t" or warn' ;
done
exit 0;
40. Metadata: ./t is for testing
Ever test a production database?
Destructively?
Ouch...
Metadata: test configs are in ./t/etc.
41. Metadata: ./t is for testing
# tests find ./t/lib/Foo/Config.pm
# ./bin files find ./lib/Foo/Config.pm
use FindBin::libs;
use Foo::Config;
42. Metadata: ./t is for testing
# tests prefer ./t/etc
use FindBin::libs qw( base=etc export );
my $base = ‘Database.config.yaml’;
my $found
= first {-e “$_/$base” } @etc
or die “Oops... no database config.n”;
my $path = “$found/$base”;
55. All politics is local
So are values.
“local” provides scoped values.
local $ = “n”;
local *STDOUT = $fh;
56. Perl Testers Notebook
Great book, even with the fake coffee stains.
One nice technique described in detail:
Hacking CORE.
Say you want to open to fail.
57. Hack open???
sub
{
# see Perl Testing Notebook
local *CORE::open
= sub { die “No such file.n” };
$madness->$method( @_ );
}
58. Fail on one specific path
sub fail_on_open
{
my $pkg = shift;
my $path = shift;
my $open = $pkg->can( ‘open_config’ );
sub
{
$_[1] eq $path
and die “Failed open: ‘$path’n”;
goto &$open;
}
}
59. Wrap a method to fail on a specific file
my @plan
= map
{
my $sub = fail_on_open $pkg, $_;
[
[ $sub, “$_” ],undef,[ “Fail: $_” ]
]
}
glob “/etc/frobnicate/*.config.*”;
60. Testing runt reads
.ini or data files may not have bookends.
Lacking a closing marker, can you detect runts?
Bad test: Write hacked files to temp dir’s.
Better test: Hack your reader.
61. Dispatch a partial read
my $path = shift;
sub
{
$_[1] eq $path or goto &$wrapped;
my $data = &$wrapped;
substr $data, 0, rand length $data
}
62. my $method = ‘do_something’;
my @plan
= map
{
my $path = $_;
my $runt = gen_runt_read $path;
my $ref = qualify_to_ref read_cfg => $pkg;
my $sub
= sub
{
local *{ $ref } = $runt;
$object->$method( $path )
};
[
[ $sub, $path ], undef, [ “Failed read: ‘$path’” ]
]
}
glob $glob;
$object->$exercise( @plan ); # verify failing on each conf
63. Checking mods
Overload stat to return zero size, hacked mods.
Force fail on -s, -r checks for data files.
Return non-existant or zero UID, GID.
64. Mapped tests
Good: One test file, one test result.
Bad: Have to test them all each time.
Alternatives:
Symlink individual tests.
Break glob-lists into smaller pieces.
65. Aside: “Testable” code.
Monolithic code is harder to test.
Testing a find-and-check-and-validate-and-open-
and-read-and-close-and-evaluate-and-install-values-
and-use-values-and-return is hard.
Faking an open is relatively easy.
So is faking a read.
Un-testable code is less maintainable.
67. Sanity checking size
Generic wrapper: call a method, check size of
object.
Same basic wrapper:
Store size.
Call something.
Re-examine size.
68. ---
name: Empty Map 00
namespace : EmptyMap00
locations:
blackhole:
name: BlackHole
description : You are standing in a small Black Hole.
exits :
Out : blackhole
items: {}
player:
location : blackhole
items : {}
69. ---
name: Empty Map 00
namespace : EmptyMap00
locations:
blackhole:
name: BlackHole
description : You are standing in a small Black Hole.
exits :
Out : blackhole
items: {}
player:
location : blackhole
items : {}
70. Entering a black hole
use FindBin::libs qw(base=etc export scalar);
my $madness = ‘Adventure’;
use_ok $madness;
$madness->init(“$etc/blackhole.yaml);
my $player = $madness->player;
is_ok ‘blackhole’, $player->location;
$player->location_object
->use_exit('blackhole');
is_ok ‘blackhole’, $player->location;
71. Look for memory leaks
ok $player->move( blackhole ), “Can move out”
for 1 .. 1_000_000;
72. Look for memory leaks
ok $player->move( out ), “Can move out”
for 1 .. 1_000_000;
Downside: a million OK’s.
73. Avoid a million OK’s.
my $expect = ‘out’;
my $found= ‘’;
my $i;
for $i ( 1 .. 1_000_000 )
{
$player->move( $expect );
$found = $player->location;
$expect eq $found or last;
}
is $found, $expect, “’$found’-‘$expect’ at $i”;
74. How big are you?
Memory footprint:
sum_size
{
sum map { size $_ } @_
}
75. Yes, guys, size() does matter
my $loc = $player->location_object;
my $prior = 1.1*sum_size $madness, $player;
$loc->use_exit('blackhole')
for( 1 .. 1_000_000 );
my $after = sum_size $madness, $player;
ok $after < $prior, “$after < $prior”;
76. Generic Tests
Init the game with a mission file.
Check the initial location.
Write a black-hole file with 1 .. N stages.
Check that the size doesn’t grow.
77. my $base = basename $0, ‘.t’;
my $limit = ( split / W /x, $base )[ -1 ];
my $path= make_daisy_chain_map $limit;
Adventure->init( $path );
my $player = Adventure->player;
my $loc = $player->location_object;
my $prior = 1.1 * size $player;
for my $i ( 1 .. $limit )
{
my $next = ‘room_’ . $i;
$loc->use_exit( $next );
$next eq $loc->location or last;
my $after = size $player;
$prior > $after or last;
}
78. Playing with the web
Ever have test a web app?
But didn’t have a “back end”?
You are doing it wrong!