Turbo Charged Test Suites

Turbo Charged
Test Suites

Tuesday, 7 October 2008

Curtis “Ovid” Poe

• CPAN Author
• “Perl Hacks” Contributor
• Perl Foundation Member


Test Suite Performance

• Computer
• Developer


Faster Tests


Considerations

• Why speed things up?
• Trade-offs


Before You Start

• Set a goal
• No tests may fail
• No non-test output


Databases

• Don’t drop them
• Don’t fake it with transactions


Databases

• Static tables -- country_codes
• Dynamic tables -- orders


CREATE TABLE changed_table (
changed_table_id INT(11)
NOT NULL AUTO_INCREMENT PRIMARY KEY,
table_name VARCHAR(30) NOT NULL,
is_static INT NOT NULL DEFAULT 0,
inserts INT NOT NULL DEFAULT 0,
updates INT NOT NULL DEFAULT 0,
deletes INT NOT NULL DEFAULT 0
)

-- then insert table records


foreach my $action (qw/insert update delete/) {
$dbh->do(<<quot; END_SQLquot;);
CREATE TRIGGER tr_${action}_$table
BEFORE $action ON $table
FOR EACH ROW UPDATE changed_table
SET ${action}s = ${action}s + 1
WHERE table_name = '$table';
END_SQL
}


db.disable_foreign_keys()
for table in changed_tables()
table.truncate()
if table.is_static()
table.rebuild()
db.enable_foreign_keys()


Databases

• Before: 80 minutes
• After: 22 minutes


Aggregation

• Only load things once
• Dangers of shared state


use My::YAML::Test;
My::YAML::Test->run;
# t/api/customer.t
# t/api/customer.yml


use My::YAML::Test;
foreach (yaml_tests()) {
diag “running $_”;
My::YAML::Test->run($_);
}

sub yaml_tests {
return @ARGV if @ARGV;
# or return all YAML files ...
}


YAML Aggregation



use Test::Aggregate;
my $tests = Test::Aggregate->new({
dirs => ‘aggtests’,
});
$tests->run;


use Test::Aggregate;
my $tests = Test::Aggregate->new({
dirs => 'aggtests',
dump => $dump,
set_filenames => 1,
shuffle => 1,
startup => &startup,
shutdown => &shutdown,
});


Generic Aggregation



Better Aggregation

• use Test::Class


OK to Fail is OK

• POD tests
• Perl::Critic
• Anything “non-functional”


OK to Fail is OK

• Move them to xt/
• But they must be run!


Lessons Learned

• Not dropping your database is tricky
• Aggregating tests ﬁnds bugs in tests
• Aggregating tests ﬁnds bugs in code
• *CORE::GLOBAL:: is evil
• Don’t touch UNIVERSAL::


Faster Programmers


Custom Test Modules


use Test::More tests => 13;
use Test::Exception;
use Test::XML;
use Test::JSON;
use Test::Differences;


package Our::Test::More;

use Test::Builder::Module;
our ( @ISA, @EXPORT );
use Test::More;
use Test::Exception;

BEGIN {
@ISA = qw(Test::Builder::Module);
@EXPORT = (
@Test::More::EXPORT,
@Test::Exception::EXPORT,
);
}

1;


package My::Custom::Tests;

use Test::Kit (
'Test::More',
'Test::XML',
'Test::Differences',
'+explain',
);

1;


Popularity Contest
Test::More 44461
Test 8937
Test::Exception 1397
Test::Simple 731
Test::Base 316
Test::Builder::Tester 193
Test::NoWarnings 174
Test::Differences 146
Test::MockObject 139
Test::Deep 127


use Test::Most tests => 4, 'die';

ok 1, 'one is true';
is 2, 2, '... and two is two';
eq_or_diff [3], [4],
“... but three ain’t four”;
throws_ok { $foo/0 }
qr/Illegal division by zero/,
'and no-constant folding with vars';


Test From Your Editor


“ in your .vimrc

function! PerlMappings()
noremap K :!perldoc <cword> <bar><bar>
perldoc -f <cword><cr>
endfunction

function! PerlTestMappings()
noremap <buffer> ,t :!prove -vl %<CR>
endfunction

“ remember My::Test::YAML?
function! YAMLTestMappings()
noremap <buffer> ,t :!prove -vl t/yaml.t :: %<CR>
endfunction

au! FileType perl :call PerlMappings()
au! FileType yaml :call YAMLTestMappings()
au! BufRead,BufNewFile *.t :call PerlTestMappings()


vim $(ack -l --perl 'api/v1/episode' t/)

map <leader>tb :call RunTestsInBuffers()<cr>
function! RunTestsInBuffers()
let i = 1
let tests = ''
while (i <= bufnr(quot;$quot;))
let filename = bufname(i)
if match(filename, '.t$') > -1
let tests = tests . ' quot;' . filename . 'quot;'
endif
let i = i+1
endwhile
if !strlen(tests)
echo quot;No tests found in buffersquot;
else
execute ':!prove ' . tests
endif
endfunction


Advanced “prove”


Test-Harness $ prove -l t --state=hot,fast,save --timer
[20:57:19] t/yamlish-output.........ok 40 ms
[20:57:20] t/console................ok 45 ms
[20:57:20] t/utils..................ok 48 ms
<snip>
[20:57:23] t/harness................ok 300 ms
[20:57:23] t/process................ok 1020 ms
[20:57:24] t/prove..................ok 1017 ms
[20:57:25] t/regression.............ok 4217 ms
[20:57:29]
All tests successful.
Files=32, Tests=10326, 10 wallclock secs ( 1.25 usr
0.24 sys + 5.60 cusr 1.83 csys = 8.92 CPU)
Result: PASS


Test-Harness $ prove -l t --state=hot,slow,save -j 9
t/callbacks.............. ok
t/nofork................. ok
t/proverc................ ok
<snip>
t/yamlish................ ok
t/prove.................. ok
t/regression............. ok
All tests successful.
Files=32, Tests=10326, 6 wallclock secs ( 1.34 usr
0.24 sys + 5.63 cusr 1.83 csys = 9.04 CPU)
Result: PASS


# slow running tests from App::Prove::State
# http://use.perl.org/~Ovid/journal/35831

Generation 18
Number of test programs: 58
Total runtime approximately 17 minutes 35 seconds
Five slowest tests:
482.7 seconds -> t/acceptance.t
234.4 seconds -> t/aggregate.t
96.3 seconds -> t/standards/strict.t
66.6 seconds -> t/unit/db/migrations.t
56.7 seconds -> t/unit/piptest/pprove/testdb.t


Devel::CoverX::Covered


Devel::CoverX::Covered

• What tests cover this ﬁle?
• What ﬁles are covered by this test?
• (Soon) What tests cover this line?
• (Soon) What tests cover this subroutine?
• ... and more ...


function! PerlMappings()
noremap <buffer> ,cv :call Coverage()<cr>
endfunction

function! PerlTestMappings()
noremap <buffer> ,t :!prove -vl --norc %<CR>
endfunction

function! Coverage()
let file = bufname('%')
if match(filename, '.t$') > -1
execute '!covered by --test_file=quot;'.file.'quot;'
else
execute '!covered covering --source_file=quot;'.file.'quot;'
end
endfunction

au! FileType perl :call PerlMappings()
au! BufRead,BufNewFile *.t :call PerlTestMappings()


Questions?


Turbo Charged Test Suites

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Turbo Charged Test Suites

Similar to Turbo Charged Test Suites (20)

More from Curtis Poe

More from Curtis Poe (20)

Recently uploaded

Recently uploaded (20)

Turbo Charged Test Suites