I've been maintaining a number of Perl software packages recently.
There's
SReview
,
my video review and transcoding system of which I split off
Media::Convert
a while
back; and as of about a year ago, I've also added
PtLink
, an RSS aggregator
(with future plans for more than just that).
All these come with extensive test suites which can help me ensure that
things continue to work properly when I play with things; and all of
these are hosted on salsa.debian.org, Debian's gitlab instance. Since
we're there anyway, I configured GitLab CI/CD to run a full test suite
of all the software, so that I can't forget, and also so that I know
sooner rather than later when things start breaking.
GitLab has extensive support for various test-related reports, and while
it took a while to be able to enable all of them, I'm happy to report
that today, my perl test suites generate all three possible reports.
They are:
-
The
coverage
regex, which captures the total reported coverage for
all modules of the software; it will show the test coverage on the
right-hand side of the job page (as in
this
example
), and
it will show what the delta in that number is in merge request
summaries (as in
this
example
-
The
JUnit
report, which tells GitLab in detail which tests were run,
what their result was, and how long the test took (as in
this
example
)
-
The
cobertura
report, which tells GitLab which lines in the software
were ran in the test suite; it will show up coverage of affected lines
in merge requests, but nothing more. Unfortunately, I can't show an
example here, as the information seems to be no longer available once
the merge request has been merged.
Additionally, I also store the native perl Devel::Cover report as job
artifacts, as they show some information that GitLab does not.
It's important to recognize that not all data is useful. For instance,
the JUnit report allows for a test name and for details of the test.
However, the module that generates the JUnit report from
TAP
test suites does not make a
distinction here; both the test name and the test details are reported
as the same. Additionally, the time a test took is measured as the time
between the end of the previous test and the end of the current one;
there is no "start" marker in the TAP protocol.
That being said, it's still useful to see all the available information
in GitLab. And it's not even all that hard to do:
test:
stage: test
image: perl:latest
coverage: '/^Total.* (\d+.\d+)$/'
before_script:
- cpanm ExtUtils::Depends Devel::Cover TAP::Harness::JUnit Devel::Cover::Report::Cobertura
- cpanm --notest --installdeps .
- perl Makefile.PL
script:
- cover -delete
- HARNESS_PERL_SWITCHES='-MDevel::Cover' prove -v -l -s --harness TAP::Harness::JUnit
- cover
- cover -report cobertura
artifacts:
paths:
- cover_db
reports:
junit: junit_output.xml
coverage_report:
path: cover_db/cobertura.xml
coverage_format: cobertura
Let's expand on that a bit.
The first three lines should be clear for anyone who's used GitLab CI/CD
in the past. We create a job called
test
; we start it in the
test
stage, and we run it in the
perl:latest
docker image. Nothing
spectacular here.
The
coverage
line contains a regular expression. This is applied by
GitLab to the output of the job; if it matches, then the first bracket
match is extracted, and whatever that contains is assumed to contain the
code coverage percentage for the code; it will be reported as such in
the GitLab UI for the job that was ran, and graphs may be drawn to show
how the coverage changes over time. Additionally, merge requests will
show the delta in the code coverage, which may help deciding whether to
accept a merge request. This regular expression will match on a line of
that the
cover
program will generate on standard output.
The
before_script
section installs various perl modules we'll need
later on. First, we intall
ExtUtils::Depends
. My code
uses
ExtUtils::MakeMaker
,
which ExtUtils::Depends depends on (no pun intended); obviously, if your
perl code doesn't use that, then you don't need to install it. The next
three modules --
Devel::Cover
,
TAP::Harness::JUnit
and
Devel::Cover::Report::Cobertura
are necessary for the reports, and you should include them if you want
to copy what I'm doing.
Next, we install declared dependencies, which is probably a good idea
for you as well, and then we run
perl Makefile.PL
, which will generate
the Makefile. If you don't use ExtUtils::MakeMaker, update that part to
do what your build system uses. That should be fairly straightforward.
You'll notice that we don't actually
use
the Makefile. This is because
we only want to run the test suite, which in our case (since these are
PurePerl modules) doesn't require us to build the software first. One
might consider that this makes the call of
perl Makefile.PL
useless,
but I think it's a useful test regardless; if that fails, then obviously
we did something wrong and shouldn't even try to go further.
The actual tests are run inside a
script
snippet, as is usual for
GitLab. However we do a bit more than you would normally expect; this is
required for the reports that we want to generate. Let's unpack what we
do there:
cover -delete
This deletes any coverage database that might exist (e.g., due to
caching or some such). We don't actually expect any coverage database,
but it doesn't hurt.
HARNESS_PERL_SWITCHES='-MDevel::Cover'
This tells the TAP harness that we want it to load the Devel::Cover
addon, which can generate code coverage statistics. It stores that in
the
cover_db
directory, and allows you to generate all kinds of
reports on the code coverage later (but we don't do that here, yet).
prove -v -l -s
Runs the actual test suite, with
v
erbose output,
s
huffling (aka,
randomizing) the test suite, and adding the
l
ib directory to perl's
include path. This works for us, again, because we don't actually need
to compile anything; if you do, then
-b
(for
blib
) may be required.
ExtUtils::MakeMaker creates a
test
target in its Makefile, and usually
this is how you invoke the test suite. However, it's not the only way to
do so, and indeed if you want to generate a JUnit XML report then you
can't do that. Instead, in that case, you need to use the
prove
, so
that you can tell it to load the TAP::Harness::JUnit module by way of
the
--harness
option, which will then generate the JUnit XML report.
By default, the JUnit XML report is generated in a file
junit_output.xml
. It's possible to customize the filename for this
report, but GitLab doesn't care and neither do I, so I don't. Uploading
the JUnit XML format tells GitLab which tests were run and
Finally, we invoke the
cover
script twice to generate two coverage
reports; once we generate the default report (which generates HTML files
with detailed information on all the code that was triggered in your
test suite), and once with the
-report cobertura
parameter, which
generates the cobertura XML format.
Once we've generated all our reports, we then need to upload them to
GitLab in the right way. The native perl report, which is in the
cover_db
directory, is uploaded as a regular job artifact, which we
can then look at through a web browser, and the two XML reports are
uploaded in the correct way for their respective formats.
All in all, I find that doing this makes it easier to understand how my
code is tested, and why things go wrong when they do.
Značky: #Debian
KDE Mascot
