[[reporting]]
== Reports

This chapter covers the two types of reports that can be generated
from the TCK, an assertion coverage report and the test execution
results.

=== Jakarta Validation TCK Coverage Report

A specification can be distilled into a collection of assertions
that define the behavior of the software. This section introduces the Bean
Validation TCK coverage report, which documents the relationship between
the assertions that have been identified in the Jakarta Validation specification
document and the tests in the TCK test suite.

The structure of this report is controlled by the assertion
document, so we'll start there.

==== Jakarta Validation TCK Assertions

The Jakarta Validation TCK developers have analyzed the Jakarta Validation
specification document and identified the assertions that are present in
each chapter. Here's an example of one such assertion found in section
2.1: "Every constraint annotation must define a message element of
type String"

The assertions are listed in the XML file
`tck-audit.xml` in the Jakarta Validation TCK
distribution. Each assertion is identified by the section of the
specification document in which it resides and assigned a unique
paragraph identifier to narrow down the location of the assertion
further. To continue with the example, the assertion shown above is
listed in the `tck-audit.xml` file using this XML
fragment:

[source,xml]
----
<section id="constraintsdefinitionimplementation-constraintdefinition-properties-message" title="message" level="4">
    ...
    <!-- 3.1.1.1 - CONSTRAINTSDEFINITIONIMPLEMENTATION_CONSTRAINTDEFINITION_PROPERTIES_MESSAGE -->
    <assertion id="a">
        <text>Every constraint annotation must define a message element of type String.</text>
    </assertion>
    ...
</section>
----

The strategy of the Jakarta Validation TCK is to write a test which
validates this assertion when run against an implementation. A test case
(a method annotated with `@Test`) is correlated with an
assertion using the
`@org.jboss.test.audit.annotations.SpecAssertion`
annotation as follows:

[source, java]
----
@Test(expectedExceptions = ConstraintDefinitionException.class)
...
@SpecAssertion(section = Sections.CONSTRAINTSDEFINITIONIMPLEMENTATION_CONSTRAINTDEFINITION_PROPERTIES_MESSAGE, id = "a")
...
public void testConstraintDefinitionWithoutMessageParameter() {
    getValidator().validate( new DummyEntityNoMessage() );
    fail( "The used constraint does not define a message parameter. The validation should have failed." );
}
----

To help evaluate the distribution of coverage for these
assertions, the TCK provides a detailed coverage report. This report is
also useful to help implementors match tests with the language in the
specification that supports the behavior being tested.

==== The Coverage Report

The coverage report is an HTML report generated as part of the TCK
project build. Specifically, it is generated by an annotation processor
that attaches to the compilation of the classes in the TCK test
suite.
You can find the source code for this processor in the GitHub
repository https://github.com/jboss/jboss-test-audit[https://github.com/jboss/jboss-test-audit]
 The report is written to the file
`target/coverage-report/coverage-beanvalidation.html`.
The report itself has five sections:

. *Chapter Summary* - List the
chapters (that contain assertions) in the specification document
along with total assertions, tests and coverage percentage.

. *Section Summary* - Lists the
sections (that contain assertions) in the specification document
along with total assertions, tests and coverage percentage.

. *Coverage Detail* - Each
assertion and the test that covers it, if any.

. *Unmatched Tests* - A list of
tests for which there is no matching assertion (useful during TCK
development).

. *Unversioned Tests* - A list of
tests for which there is no `@SpecVersion`
annotation on the test class (useful during TCK development).

The coverage report is color coded to indicate the status of an
assertion, or group of assertions. The status codes are as
follows:

* *Covered* - a test exists for
this assertion

* *Not covered* - no test exists
for this assertion

* *Unimplemented* - a test
exists, but is unimplemented

* *Untestable* - the assertion
has been deemed untestable, a note, explaining why, is normally
provided

For reasons provided in the `tck-audit.xml`
document and presented in the coverage report, some assertions are not
testable.

The coverage report does not give any indication as to whether the
tests are passing. That's where the TestNG reports come in.

=== The TestNG Report

As you by now know, the Jakarta Validation TCK test suite is really
just a TestNG test suite. That means an execution of the Jakarta Validation
TCK test suite produces all the same reports that TestNG produces. This
section will go over those reports and show you were to go to find each of
them.

==== Maven, Surefire and TestNG

When the Jakarta Validation TCK test suite is executed during the
Maven test phase of the TCK runner project, TestNG is invoked
indirectly through the Maven Surefire plugin. Surefire is a test
execution abstraction layer capable of executing a mix of tests written
for JUnit, TestNG, and other supported test frameworks.

Why is this relevant? It means two things. First, it means that
you are going to get a summary of the test run on the commandline.
Here's the output generated when the tests are run using standalone
mode.

[source]
----
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running TestSuite
Tests run: 976, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.288 sec - in TestSuite

Results :

Tests run: 976, Failures: 0, Errors: 0, Skipped: 0
----

[NOTE]
====
The number of tests executed, the execution time, and the output
will differ when you run the tests using in-container mode as the Bean
Validation TCK requires.
====

If the Maven reporting plugin that compliments Surefire is
configured properly, Maven will also generate a generic HTML test result
report. That report is written to the file test-report.html in the
`target/surefire-reports` directory of the TCK runner
project. It shows how many tests were run, how many failed and the
success rate of the test run.
