Reporting

Configuration

All configurations are done in the <exporters> section of the pom.xml of your project.

A report is defined by the implementation attribute of an <exporter>. Child tags (of <exporter>) are report-dependant.

<configuration>
    ...
    <exporters>
        <exporter implementation="org.squashtest.ta.commons.exporter.surefire.SurefireSuiteResultExporter">
            <exportAttached>true</exportAttached>
            <jenkinsAttachmentMode>true</jenkinsAttachmentMode>
        </exporter>
        <exporter implementation="org.squashtest.ta.commons.exporter.html.HtmlSuiteResultExporter"></exporter>
    </exporters>
    ...
</configuration>

---

Report Status

Element :	Status :
Instruction :	NOT RUN : Instruction was not executed. SUCCESS : Instruction was successfully executed. FAILURE : An assertion has failed. ERROR : An instruction raised an unexpected error
Test case :	NOT_RUN : A phase of a test case was not executed. Otherwise its status is the severest one among status of its instructions.
Script TF : (Test)	NOT_RUN : Script was not executed. Otherwise its status is the severest one among status of its instructions.
Ecosystem :	Its status is the severest one among status of its scripts.

---

SKF HTML Report

HTML Report Configuration

In our maven archetype, HTML reports are already enabled. Otherwise, you can do it with the following implementation :

org.squashtest.ta.commons.exporter.html.HtmlSuiteResultExporter

HTML report configuration sample :

<configuration>
    ...
    <exporters>
        <exporter implementation="org.squashtest.ta.commons.exporter.html.HtmlSuiteResultExporter"></exporter>
    </exporters>
    ...
</configuration>

Since 1.7.0 version, it’s also possible to enable it with the following configuration :

<configuration>
    ...
    <exporters>
        <html/>
    </exporters>
    ...
</configuration>

Note

Both solutions are equivalent and work since 1.7.0.

HTML Report overview

The HTML Report (squash-ta-report.html) is created post-execution in directory : my.squash.ta.project/target/SquashTA/html-reports.

Here are some screenshots of an html report :

• Test suite summary :

  

• Ecosystem summary :

  

• Target Initialisation summary : (@since Squash-TA framework 1.7.0)

  

• Test script result :

  

A full sample is also available here : Execution report - execution-report-test-sample.pdf

---

HTML Summary

The HTML Summary is a less detailed version of the HTML report.

Normal HTML Summary

If you want to have an HTML Summary report, you have to add the following lines in <exporters> :

<exporters>
    <htmlSummary>
        <outputDirectoryName>directoryOfHtmlSummary</outputDirectoryName>
    </htmlSummary>
</exporters>

Lightened HTML Summary

You can lighten the report if you add to the <htmlSummary> configuration the tag :

<includeHeader>false</includeHeader>

Which gives :

<exporters>
    <htmlSummary>
        <outputDirectoryName>directoryOfHtmlSummary</outputDirectoryName>
        <includeHeader>false</includeHeader>
    </htmlSummary>
</exporters>

Note

It’s possible to have both HTML report and HTML Summary together : Put the two tags in <exporters>…</exporters>

<exporters>
  <htmlSummary>
    <outputDirectoryName>directoryOfHtmlSummary</outputDirectoryName>
    <includeHeader>false</includeHeader>
  </htmlSummary>
  <html/>
</exporters>

---

Surefire Report

Surefire Report Configuration

In our maven archetype, Surefire reports are already enabled. Otherwise, you can enable it with the following implementation :

org.squashtest.ta.commons.exporter.surefire.SurefireSuiteResultExporter

The Surefire report uses 2 properties :

• exportAttached : If you want to generate the attachments in the surefire-reports-directory :

  • Default value : true (since 1.7.0).
  • Values accepted : true/false.

• jenkinsAttachmentMode : Used with the Squash TF server for the Jenkins JUnit attachment plugin.

Surefire report configuration sample :

<configuration>
    ...
    <exporters>
        <exporter implementation="org.squashtest.tf.commons.exporter.surefire.SurefireSuiteResultExporter">
            <exportAttached>true</exportAttached>
            <jenkinsAttachmentMode>true</jenkinsAttachmentMode>
        </exporter>
    </exporters>
    ...
</configuration>

Since 1.7.0 version, there is an easy solution to activate the Surefire report :

<configuration>
    ...
    <exporters>
        <surefire>
            <exportAttached>true</exportAttached>
            <jenkinsAttachmentMode>true</jenkinsAttachmentMode>
        </surefire>
    </exporters>
    ...
</configuration>

Note

Both solutions are equivalent and works since SKF 1.7.0.

Surefire report overview

A Surefire report is an .xml file which contains all results of the scripts execution.

Each <testsuite> element represents an ecosystem and each <testcase> element a .ta script.

Surefire reports are created in directory : my.squash.ta.project/target/squashTA/surefire-reports

Here is an example of a Surefire report :

Three kind of results are possible :

• Passed : In this case the Surefire report provides only the name and the execution time of the test case :

  <testcase time="0.017" classname="tests.F01.ScriptsTF.Set1" name="UC01.01_Success.ta" />
  

• Failed (because of a false assertion) : In this case the Surefire report provides the name and the execution time of the test case. It also indicates that a ‘failure’ occured and gives the associated trace.

<testcase time="0.0090" classname="tests.Set1" name="Failure_in_setup.ta" >
  <failure message="The pattern rubbish was not found in the file."
  type="org.squashtest.ta.framework.exception.BinaryAssertionFailedException" >
Summary
SETUP :
[SUCCESS] LOAD file.txt AS file
[FAIL] ASSERT file DOES contain WITH $(rubbish)
     [SUCCESS]     $(rubbish) --> {{__temp306}}
     [FAIL] ASSERT file DOES contain WITH {{__temp306}}

Trace here

[NOT_RUN] ASSERT file DOES contain WITH $(Hello)
     [NOT_RUN]     $(Hello) --> {{__temp152}}
     [NOT_RUN]     ASSERT file DOES contain WITH {{__temp152}}
TEST :
[NOT_RUN] ASSERT file DOES contain WITH $(Hello)
     [NOT_RUN]     $(Hello) --> {{__temp597}}
     [NOT_RUN]     ASSERT file DOES contain WITH {{__temp597}}
TEARDOWN :
[SUCCESS] ASSERT file DOES contain WITH $(Hello)
     [SUCCESS]     $(Hello) --> {{__temp529}}
     [SUCCESS]     ASSERT file DOES contain WITH {{__temp529}}
  </failure>
</testcase>

• Failed (because of a technical error) : In this case the Surefire report provides the name and the execution time of the test case. It also indicates that an ‘error’ occurs and give the associated trace.

<testcase time="0.013" classname="tests.Set2" name="Error_in_setup.ta" >
  <error message="ASSERT wrongId DOES contain WITH {{__temp238}}: Cannot apply assertion :
  SCOPE_TEST:wrongId does not exist in this test context: you must load it first."
  type="org.squashtest.tf.backbone.exception.ResourceNotFoundException" >
Summary
SETUP :
[SUCCESS] LOAD file.txt AS file
[ERROR] ASSERT wrongId DOES contain WITH $(Hello)
     [SUCCESS]     $(Hello) --> {{__temp238}}
     [ERROR] ASSERT wrongId DOES contain WITH {{__temp238}}

Trace was here

[NOT_RUN] ASSERT file DOES contain WITH $(Hello)
     [NOT_RUN]     $(Hello) --> {{__temp362}}
     [NOT_RUN]     ASSERT file DOES contain WITH {{__temp362}}
TEST :
[NOT_RUN] ASSERT file DOES contain WITH $(Hello)
     [NOT_RUN]     $(Hello) --> {{__temp345}}
     [NOT_RUN]     ASSERT file DOES contain WITH {{__temp345}}
TEARDOWN :
[SUCCESS] ASSERT file DOES contain WITH $(Hello)
     [SUCCESS]     $(Hello) --> {{__temp875}}
     [SUCCESS]     ASSERT file DOES contain WITH {{__temp875}}
  </error>
</testcase>

Inside the target directory, there are also attached documents produced by SKF scripts to make easier diagnosis when an error occurs (snapshots, diff-reports, detailed logs…).