Residual Anomaly Justification: Guide for Compliance Officers

This guide explains how to justify residual anomalies (failed tests) on a Production release PR using /ra-justification comments. Residual anomalies are failed test cases from the integration/system tests (Cucumber) and unit tests (JUnit) that are accepted for release — bugs that are known and approved for production.

How It Works

When the Software Summary Report is generated, Orca parses both the Cucumber and JUnit test reports. Every failed or errored test case becomes a residual anomaly. If there are no failed tests, no justification is needed and the report renders with "No Residual anomalies to report."

If there are failed tests, the report requires a justification comment on the PR from a member of a validated GitHub team. The validation of the author, is carried out automatically. Without a valid justification, report generation and validation both fail.

Command Format

/ra-justification
<justification for anomaly 1>
<justification for anomaly 2>
...

The comment must begin with the /ra-justification marker. Everything after the marker is split on newlines into individual justifications — one per anomaly. The order in which the justifications are written in the comment, will be ordered equivalently in the generated report - It is therefore the authors responsibility to adhere to any ordering preferences, in the written comment, if any exist.

Example

If the report has two anomalies — a failed Cucumber step and a failed unit test:

Comment:

/ra-justification
Flaky integration test, tracked in SR-2701
Known unit test failure in the handlers suite, not related to this release

These appear in the Software Summary Report under the "Residual Anomalies" page, each on its own line in the "Justification" section.

Who Can Post

Currently, only members of the corticph/platform GitHub team can post a valid justification. Comments from non-team members are ignored, even if they carry the marker.

How Anomalies Are Collected

Anomalies come from two test report types:

The anomalies are combined in order: all Cucumber anomalies first, then all JUnit anomalies. Each anomaly carries the feature/suite description, the scenario/test name, and the error message from the test report.

Where Justifications Appear

In the generated Software Summary Report PDF, the "Residual Anomalies" page contains two sections:

1. Residual Anomalies table — a two-column table listing each anomaly:

   Software/System Requirement	Anomaly Description
   Feature description	Scenario name — error message

2. Justification — each justification from the comment, rendered as a separate line below the table:

   Flaky integration test, tracked in SR-2701
   Known unit test failure in the handlers suite, not related to this release
   

If there are no anomalies, the table shows "No Residual anomalies to report" and the justification section is omitted entirely.

Most Recent Comment Wins

If multiple comments on the PR carry the /ra-justification marker, the most recent one wins. This lets you correct an earlier justification by posting a new comment — no need to edit or delete the old one.

When No Justification Is Needed

If all tests pass (no failed steps in Cucumber and no failures or errors in JUnit), there are no residual anomalies and no justification comment is required. The report validates and generates without --pr-id.

Validation

The --dry-run flag validates the report without generating a PDF. If there are failed tests and no valid justification comment, validation fails with an error. The same check runs during report generation.

Edge Cases

Marker with no body

/ra-justification

This is ignored — the comment has no justification text after the marker, so it doesn't count as a valid justification.

Marker not at the start of the line

Here is my justification: /ra-justification test is flaky

This is ignored — the marker must be at the very start of the comment body (leading whitespace is trimmed, but the marker must come first).

Comment from a non-team member

/ra-justification
test is flaky

If the author is not a member of a valid team, the comment is ignored.

Empty lines between justifications

/ra-justification
test A is flaky

test C is flaky

Empty lines are dropped. This produces two justifications: test A is flaky and test C is flaky.

Quick Reference