Release Candidate Manual QA¶
This is the single release-blocking, hands-on QA procedure for APTL. A person must execute it against both the candidate distribution and a clean source checkout before the release PR is merged. Container health, automated tests, and a previous release's results are useful prerequisites, but they do not satisfy any row in this manual.
Every row is blocking. Record PASS or FAIL; SKIP, an empty result, and
"not applicable" are not passing outcomes. Stop the release on a failure,
repair it, build a new candidate, and repeat both paths from clean state.
Candidate and environment record¶
Create one record for the release and fill every field before testing:
| Field | Recorded value |
|---|---|
| Release version | |
| Release PR and head commit | |
| Candidate wheel filename and SHA-256 | |
| Candidate sdist filename and SHA-256 | |
| RAES env-pack id, version, and set digest | |
| OS and version | |
| CPU architecture | |
| Python version | |
| Docker Engine version | |
| Docker Compose version | |
| Distribution-path project directory | |
| Source-path checkout directory | |
| Evidence location | |
| Primary operator | |
| QA start and finish times, including time zone |
The release PR head must be clean and immutable while QA runs. From that exact commit, build and identify the candidate artifacts:
Record both artifact hashes. The distribution-path test installs the exact wheel; the sdist is its recorded companion release artifact. If the commit or either artifact changes, the record is invalid and both paths must be rerun.
Use a dedicated Docker host or prove the selected project has no prior resources. Do not use a daemon-wide prune. Each path ends with the scoped teardown proof before the next path begins.
Install path A: candidate distribution¶
Use an empty directory and a new virtual environment. Do not import lab assets from a source checkout.
python3 -m venv qa-candidate-venv
source qa-candidate-venv/bin/activate
pip install /absolute/path/to/aptl_labs-X.Y.Z-py3-none-any.whl
aptl --version
raes --version
aptl lab init qa-candidate-lab
cd qa-candidate-lab
Record the installed version and wheel identity in the path-A evidence. The initialized directory is the project directory for every path-A action below.
Install path B: exact source commit¶
Use a second, clean checkout at the same recorded release-PR commit. Do not reuse the distribution-path project, virtual environment, generated files, or Docker volumes.
git clone https://github.com/Brad-Edwards/aptl.git qa-source
cd qa-source
git checkout --detach <recorded-release-pr-head-commit>
git status --porcelain
python3 -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'
aptl --version
raes --version
The clean checkout itself is the project directory for every path-B action.
Evidence rules¶
For each action, capture the minimum output or screenshot that proves the expected observation. Give every file a stable reference and put that reference in the result table. A timestamp and a written assertion without the underlying observation are insufficient.
Review and redact evidence before attaching or publishing it. Never capture or
publish .env, browser storage, cookies, authorization headers, API tokens,
private keys, full generated configuration, unreviewed logs, or planted
credentials. A private archive or restrictive file mode does not replace
redaction. Preserve useful identifiers such as the candidate hash, run id,
container name, alert id, rule id, case id, execution id, and analyzer job id.
Browser trust prerequisite¶
Each clean path generates two path-specific certificate authorities. Before opening a SOC UI, import these public roots into the test browser or its operating-system trust store:
config/wazuh_indexer_ssl_certs/root-ca.pemfor Wazuh;config/soc_certs/lab-ca.pemfor MISP, Shuffle, and TheHive.
The Wazuh dashboard certificate names wazuh.dashboard, not localhost.
Configure a temporary browser or operating-system resolver entry mapping
wazuh.dashboard to 127.0.0.1, and use the dashboard host port reported by
aptl lab info (443 is only the default). The other SOC UI certificates
include localhost, but MISP's authored browser origin is
https://misp.techvault.local on port 443. Its loopback publication (normally
https://localhost:8443) is usable by API/MCP clients but is not a browser
login URL: MISP redirects the form to its canonical origin, which the browser
blocks under form-action 'self' when opened on the loopback publication.
Do not weaken CSP or change the authored in-world base URL to make this pass.
On a native-Linux Docker host, select the MISP container's security-network IP
from this path's qa-start-status.json and map misp.techvault.local to that
IP in the test browser only. Open https://misp.techvault.local/, not the
loopback publication. For example, the IP can be read without guessing a
container name:
jq -r '.containers[] | select(.name | endswith("-misp")) |
.networks | to_entries[] | select(.key | endswith("_aptl-security")) |
.value' qa-start-status.json
The browser must have a route to that in-range address. Docker Desktop hosts
may not expose the bridge address directly; use a browser in an environment
with a route to the scenario network and the same canonical origin, or record
QA-MISP as FAIL. Do not count a successful API call or the broken loopback
login page as a browser result. For Shuffle and TheHive, use each service's
actual loopback publication from qa-start-status.json. Confirm a valid
hostname and chain to the path-specific root before recording any UI result.
Do not click through a certificate warning, disable certificate validation, or
use an insecure client flag to make a QA row pass. Remove the temporary trust
entries and hostname mappings after that path's QA-TEARDOWN; path B must
import its own newly generated roots rather than reuse path A's trust.
Required actions¶
Execute QA-START through QA-TEARDOWN in order for path A, complete the
path-A results table, then repeat the whole sequence from clean state for path
B. Do not copy an observation from one path to the other.
QA-START: Complete startup and project inventory¶
Run:
set -o pipefail
aptl lab start 2>&1 | tee qa-start.txt
aptl lab status | tee qa-start-status.txt
aptl lab status --json --output qa-start-status.json
aptl runs list | tee qa-start-runs.txt
Expected: start exits zero and reports readiness; status accounts for every
project-owned container and none is created, exited, dead, or otherwise
non-running. No aptl-mcp-endpoints container exists. The runs listing
contains the canonical run created by this exact start, backed by its root
manifest.json. The JSON snapshot records the actual loopback publications
used by later UI rows. Record that full run id as the path's startup run id.
Capture the final start result, complete status inventory, JSON snapshot, and
runs listing. A start failure is a valid diagnostic, not a pass.
Keep pipefail enabled when using tee: otherwise the pipeline may exit zero
even if aptl lab start or aptl lab status failed. The plain-text status
capture is required in addition to the JSON snapshot; the latter is not a
substitute for executing the documented user-facing command.
QA-LIVE: RAES live validation¶
Run against the already-running lab and add live-gate evidence to the canonical
startup run recorded by QA-START:
Expected: the command exits zero and every live-gate check passes, including
defensive-stack readiness and declared/observed container parity. Capture the
rendered verdict and confirm that it names the same run id recorded by
QA-START.
QA-WAZUH: Dashboard, agents, and events¶
Open the Wazuh dashboard at
https://wazuh.dashboard:<reported-host-port> and log in using the generated
credential referenced by aptl lab info. In the UI, inspect agent status and
recent indexed events.
Expected: the dashboard is usable, the scenario's agents are enrolled and reporting, and recent events are visible in the indexer. Capture the agent view and a bounded recent-event view without credentials or session data.
QA-DETECT: Attack and expected Wazuh rule¶
Use the generated .mcp.json configuration with an MCP client to call
mcp-red's kali_run_command. Direct aptl container shell aptl-kali is
refused while required capture is active; a separate SSH command would not
create the MCP-side session-census entry required by transcript finalization.
Send this SQL-injection request from the realized Kali target:
curl -sf -o /dev/null -w '%{http_code}\n' 'http://172.20.1.20:8080/login?username=admin%27%20UNION%20SELECT%201,2,3--&password=x'
In Wazuh, find the resulting alert within the test time window.
Expected: kali_run_command reports target-backed success and HTTP 200;
the alert is attributable to this action and the expected custom
Wazuh rule is 302010 (SQL injection). Capture the action time, source, alert
id, rule id, description, and event time.
QA-SURICATA: Network rule reaches the SIEM¶
Docker bridge networks do not pass peer-to-peer unicast traffic to a third
container as a passive tap. This row therefore sends the same live HTTP payload
from the Suricata sensor's own network namespace; it validates the live capture,
rule, and Wazuh-forwarding chain and does not claim passive visibility of the
separate Kali flow from QA-DETECT.
Open a sensor shell and record a new time around the request:
aptl container shell aptl-suricata
date -u +%Y-%m-%dT%H:%M:%SZ
curl -sf 'http://172.20.1.20:8080/login?username=ids_qa%27%20UNION%20SELECT%201,2,3--&password=x'
date -u +%Y-%m-%dT%H:%M:%SZ
Inspect Suricata evidence and then locate the corresponding event in Wazuh.
Expected: Suricata reports APTL SQL Injection Attempt - UNION SELECT with
signature id 1000010, and its event reaches Wazuh as a Suricata/web-attack
event (normally rule 303020). Capture both sides with a shared timestamp,
flow, or alert identifier.
QA-SOAR: Real-alert Shuffle playbook¶
In Shuffle, create or inspect a playbook that accepts the exact alert produced
by QA-DETECT and performs an observable downstream action, such as creating or
updating its TheHive case. Run it with that alert's real id, source, and rule
data (not a dummy or synthetic smoke payload), and monitor it to completion.
Expected: the execution reaches FINISHED, its input can be tied to the real
Wazuh alert, and the downstream action is visible. Capture the workflow id,
execution id, input alert id, terminal state, and downstream object id.
QA-CASE: TheHive case and Cortex analyzer¶
In TheHive, create or open the case produced from the QA-DETECT alert, add a
relevant observable such as the recorded source IP, and run an available Cortex
analyzer on it.
Expected: the case retains the alert relationship, the observable is present, and the analyzer job completes with a visible result. Capture the case id, alert id, observable type/value, analyzer name, job id, and terminal result.
QA-MISP: Seeded threat intelligence and round trip¶
Open MISP at the canonical browser origin specified above. Confirm the seeded
APTL Lab - Known Threat Actors event and its TechVault-scenario Kali
indicator 172.20.4.30, then add a harmless release-QA indicator in a
dedicated QA event or select that seeded indicator. Retrieve the same
indicator through a second supported surface, preferably
mcp-threatintel in QA-MCP-TI.
Expected: MISP is usable and seeded, and the exact indicator can be pushed or pulled and found again. Capture event and attribute ids plus the redacted round-trip result; never capture the API key.
QA-MCP-RED: mcp-red¶
From an MCP client using the generated configuration, invoke kali_info, then
use kali_run_command for a harmless real command such as whoami.
Expected: the server responds from the realized Kali target and the command returns target output. Capture the tool names, target identity, and bounded result.
QA-MCP-WAZUH: mcp-wazuh¶
Use wazuh_query_alerts to retrieve the exact QA-DETECT alert.
Expected: the live Wazuh target returns the matching alert and rule 302010.
Capture the tool name, query bounds, alert id, and rule id.
QA-MCP-INDEXER: mcp-indexer¶
Use indexer_query with a time-bounded query for the recorded alert or rule.
Expected: the live indexer returns the expected document. Capture the tool name, index pattern, query bounds, and matching document id.
QA-MCP-SOAR: mcp-soar¶
Use the SOAR tools to list/get the real-alert workflow and retrieve or execute
the QA-SOAR run with the recorded real alert data. For
soar_execute_workflow, pass the alert object itself as body; wrapping it
in {"execution_argument": ...} makes the workflow receive a nested object
and leaves $exec.rule.id and the other alert fields empty.
Expected: the live Shuffle target returns the same workflow and a terminal successful execution. Capture the tool names, workflow id, execution id, alert id, and terminal state.
QA-MCP-CASE: mcp-casemgmt¶
Use the case-management tools to list the QA-CASE case and, if needed, add a
second harmless observable to prove a target-backed write.
Expected: the live TheHive target returns the case and reflects the operation. Capture the tool names, case id, and bounded result.
QA-MCP-TI: mcp-threatintel¶
Use threatintel_search_iocs or threatintel_correlate_observable for the
exact indicator selected in QA-MISP.
Expected: the live MISP target returns that indicator and event context. Capture the tool name, indicator type/value, event id, and bounded result.
QA-MCP-NET: mcp-network¶
Use network_query_ids_alerts or network_query_web_attacks for the bounded
QA-SURICATA time window.
Expected: the live network/Wazuh target returns the Suricata event with the expected signature. Capture the tool name, time bounds, signature id, and matching result.
QA-MCP-REVERSE: mcp-reverse¶
Attempt the reverse MCP check on every path. If the exact candidate scenario
realizes the reverse target, call reverse_info and reverse_run_command for a
harmless real operation such as which r2.
If the selected scenario deliberately omits the reverse target, attempt the
client connection/tool call with the shipped server and production MCP protocol
driver. Run this from the initialized project directory after aptl lab start:
set -o pipefail
python - <<'PY' | tee qa-mcp-reverse.txt
import json
from pathlib import Path
from aptl.validation.mcp_protocol import McpProtocolError, call_mcp_tool
try:
result = call_mcp_tool(
["node", "mcp/mcp-reverse/build/index.js"],
"reverse_run_command",
{"command": "which r2"},
cwd=Path.cwd(),
timeout_seconds=30,
)
except McpProtocolError as exc:
raise SystemExit(f"FAIL: reverse MCP protocol did not complete: {exc}")
content = result.get("content")
if not isinstance(content, list) or len(content) != 1:
raise SystemExit("FAIL: reverse_run_command returned malformed content")
text = content[0].get("text") if isinstance(content[0], dict) else None
try:
payload = json.loads(text) if isinstance(text, str) else None
except json.JSONDecodeError:
payload = None
omission_errors = {
"Container 'reverse' not found in configuration",
"Reverse Engineering instance is not enabled",
}
if (
not isinstance(payload, dict)
or payload.get("success") is not False
or payload.get("command") != "which r2"
or payload.get("error") not in omission_errors
or "output" in payload
):
raise SystemExit("FAIL: reverse_run_command did not prove target unavailability")
print(json.dumps({"outcome": "expected-unavailable", "operation": "reverse_run_command"}))
PY
This performs MCP initialize, tools/list, and tools/call against the built
mcp-reverse artifact; it is not a declaration-only check. Record the command's
zero exit and bounded expected-unavailable result. Cite both the scenario
declaration and aptl lab status inventory proving that the target was
intentionally not realized. A missing Node runtime, server artifact, registered
reverse_run_command tool, complete protocol response, or structured target
failure makes the command fail. Connection errors, internal errors, and failed
remote commands cannot count as an intentional omission. The admitted target
configuration must explicitly omit or disable reverse; do not edit it merely
to obtain this result.
Expected: a realized target returns real target output; an intentionally omitted target produces the declared, inventory-backed tested-negative result. An unexpectedly missing target or an unattempted check is a failure, not a skip.
QA-ARCHIVE: Run evidence and integrity¶
Use the canonical startup run id recorded by QA-START and reused by
QA-LIVE:
aptl runs list
aptl runs show <run-id>
aptl runs export-bundle <run-id> --output-dir qa-evidence
aptl runs verify-bundle qa-evidence/<run-id>.evidence-bundle.tar
Expected: the run exists; its manifest identifies the expected scenario, backend, package version, pack/runtime evidence, and captured runtime; export succeeds; and independent bundle verification passes. The separate candidate and environment record remains authoritative for the release-PR commit and wheel/sdist hashes: compare its installed version and path-specific startup run id with this manifest, without claiming that the manifest stores artifact hashes it does not contain. Also compare at least one claimed container and one live-gate evidence item with earlier observations. Capture the run id, bundle root identity, member count, verification verdict, cross-record comparison, and the two inspected claims.
The lab start run is a provisional startup record, not a terminal experiment
attempt. Its bundle may therefore verify as unsealed with the absent
run-provenance.json and #444 seal explicitly disclosed. Record those
limitations; do not call the bundle sealed or treat an unsealed startup bundle
as a failed terminal-attempt seal. A terminal execution attempt that should
seal but does not is a separate failure under EXP-009.
QA-TEARDOWN: Scoped removal¶
Run:
Expected: both commands exit zero and the checked project owns no remaining
containers, networks, or volumes. Capture the project-scoped proof. Do not use
docker system prune, broad deletion, or an aptl-* name search as evidence.
The -y flag confirms only this documented project-scoped volume removal; it
does not widen the cleanup scope.
Path A results: Candidate distribution¶
| ID | Result (PASS/FAIL) |
Actual observation | Evidence reference | Operator | Timestamp |
|---|---|---|---|---|---|
| QA-START | |||||
| QA-LIVE | |||||
| QA-WAZUH | |||||
| QA-DETECT | |||||
| QA-SURICATA | |||||
| QA-SOAR | |||||
| QA-CASE | |||||
| QA-MISP | |||||
| QA-MCP-RED | |||||
| QA-MCP-WAZUH | |||||
| QA-MCP-INDEXER | |||||
| QA-MCP-SOAR | |||||
| QA-MCP-CASE | |||||
| QA-MCP-TI | |||||
| QA-MCP-NET | |||||
| QA-MCP-REVERSE | |||||
| QA-ARCHIVE | |||||
| QA-TEARDOWN |
Path B results: Exact source commit¶
| ID | Result (PASS/FAIL) |
Actual observation | Evidence reference | Operator | Timestamp |
|---|---|---|---|---|---|
| QA-START | |||||
| QA-LIVE | |||||
| QA-WAZUH | |||||
| QA-DETECT | |||||
| QA-SURICATA | |||||
| QA-SOAR | |||||
| QA-CASE | |||||
| QA-MISP | |||||
| QA-MCP-RED | |||||
| QA-MCP-WAZUH | |||||
| QA-MCP-INDEXER | |||||
| QA-MCP-SOAR | |||||
| QA-MCP-CASE | |||||
| QA-MCP-TI | |||||
| QA-MCP-NET | |||||
| QA-MCP-REVERSE | |||||
| QA-ARCHIVE | |||||
| QA-TEARDOWN |
Release decision¶
Record the final decision in or from the release PR:
| Field | Recorded value |
|---|---|
| Path A result | |
| Path B result | |
| Open failures | |
| Evidence review/redaction completed by | |
| QA record URL or immutable reference | |
Release decision (PASS or BLOCK) |
|
| Approver and timestamp |
PASS requires all 36 path-specific rows to pass and the evidence review to be
complete. Any other state is BLOCK; do not merge the release PR.