=head1 Writing Tests

Z<CHP-9-SECT-13>

X<testing Parrot>
As we mentioned earlier, contributions to the Parrot project are
welcome.  Contributing tests is a good place to start. You don't have
to understand the code behind a PASM opcodeN<Or IMCC instruction.> to
test it, you only have to understand what it's supposed to do. If
you're working on some code and it doesn't do what the documentation
advertises, you can isolate the problem in a test or series of tests
and send them to the bug tracking system. There's a good chance the
problem will be fixed before the next release. Writing tests makes it
a lot easier for the developer to know when they've solved your
problem--it's solved when your tests pass. It also prevents that
problem from appearing again, because it's checked every time anyone
runs C<make> C<test>. As you move along, you'll want to write tests
for every bug you fix or new feature you add.

X<Test::Builder>
X<Parrot::Test module>
The Perl 5 testing framework is at the core of Parrot tests,
particularly F<Test::Builder>. Parrot's F<Parrot::Test> module is an
interface to F<Test::Builder> and implements the extra features needed
for testing Parrot, like the fact that PASM code has to be compiled to
bytecode before it runs.

The main Parrot tests are in the top-level F<t/> directory of the
Parrot source tree. F<t/op> contains tests for basic opcodes and
F<t/pmc> has tests for PMCs. The names of the test files indicate the
functionality tested, like F<integer.t>, F<number.t>, and F<string.t>.
Part of the F<make test> target is the command F<perl t/harness>,
which runs all the F<.t> files in the subdirectories under F</t>. You
can run individual test files by passing their names to the F<harness>
script:

  $ perl t/harness t/op/string.t t/op/integer.t

X<set opcode (PASM);test example>
Here's a simple example that tests the C<set> opcode with integer
registers, taken from F<t/op/integer.t>:

  output_is(E<lt>E<lt>CODE, E<lt>E<lt>OUTPUT, "set_i");
      set     I0, 42
      set     I1, I0
      print   I1
      print   "\\n"
      end
  CODE
  42
  OUTPUT

The code here sets integer register C<I0> to the value 42, sets C<I1>
to the value of C<I0>, and then prints the value in C<I1>. The test
passes if the value printed was 42, and fails otherwise.

The C<output_is> subroutine takes three strings: the code to run, the
expected output, and a description of the test. The first two strings
can be quite long, so the convention is to use Perl 5 here-documents.
If you look into the code section, you'll see that the literal C<\n>
has to be escaped as C<\\n>. Many tests use the non-interpolating
(C<E<lt>E<lt>'CODE>') form of here-document to avoid that problem. The
description can be any text.  In this case, it's the fully qualified
name of the C<set> opcode for integer registers, but it could have
been "set a native integer register."

If you look up at the top of F<integer.t>, you'll see the line:

  use Parrot::Test tests => 38;

(although the actual number may be larger if more tests have been added
since this book went to press).

The C<use> line for the F<Parrot::Test> module imports a set of
subroutines into the test file, including C<output_is>. The end of the
line gives the number of tests contained in the file.

The C<output_is> subroutine looks for an exact match between the
expected result and the actual output of the code. When the test
result can't be compared exactly, you want C<output_like> instead. It
takes a Perl 5 regular expression for the expected output:

  output_like(<<'CODE', <<'OUTPUT', "testing for text match");
  ...
  CODE
  /^Output is some \d+ number\n$/
  OUTPUT

F<Parrot::Test> also exports C<output_isnt>, which tests that the
actual output of the code I<doesn't> match a particular value.

There are a few guidelines to follow when you're writing a test for a
new opcode or checking that an existing opcode has full test coverage.
Tests should cover the opcode's standard operation, corner cases, and
illegal input. The first tests should always cover the basic
functionality of an opcode. Further tests can cover more complex
operations and the interactions between opcodes. If the test program
is complex or obscure, it helps to add comments. Tests should be
self-contained to make it easy to identify where and why a test is
failing.

=cut