How-to guide

Validation and reports

Source and package versions

One AIWAF repository. Python: py/aiwaf/ · Node.js: js/ · Rust/WASM: rust/ · Java: java/. Source and canonical README. Versions shown describe the source checkout; package releases are independent. Current Python/Node source targets are aiwaf 1.1.8 (Python) and aiwaf 1.1.5 (npm), with Rust/Python 0.2.7 and WASM 0.2.5 and Java 1.3.5. Sandbox guides retain their separately pinned builds.

Run the source validation workflow

Start the fifteen integrations from the tutorial, then the optional reference services. Run checks sequentially to avoid build/scan load distorting request timeouts.

docker compose --profile assessment up -d --build security-fixture aiwaf-assessment
python -m pip install requests
python -m unittest discover -s examples/sandbox -p 'test_*.py'
node --test examples/sandbox/common/redis-cache.test.js examples/sandbox/common/proxy.test.js
python examples/sandbox/run-source-packages.py

The runner verifies source labels and versions, then executes smoke, SQL login, payload matrix, post-matrix SQL, reference checks and nine production npm audits: eight proxy trees and the SDK. It excludes PHP. --verify-only performs installed-source checks. --skip-completed accepts prior suite reports only when the recorded source URL/commit matches; omit it for a fresh run.

Interpret each check

Check	What success establishes	Limit
Smoke	Normal/lowercase headers succeed; automation and excess traffic are denied; benign JSON login reaches the app.	Not a complete attack evaluation. Burst checks need accepted and denied requests, so blanket blocking fails.
SQL login	Two SQL payloads cannot issue a token; invalid credentials supply the control.	Specific exploit outcomes only. Tokens are omitted.
Payload matrix	All selected observations complete without request errors.	Exit 0 does not mean every payload was denied or every exploit prevented.
Reference fixture	Status, response fields and state changes enforce each tested application rule.	Application-owned controls; an unrelated WAF 403 cannot count as success.
npm/image audit	Advisory query and matched dependency findings are recorded.	Package locations and exploitability still need triage.

The legacy attack suite classifies 403/405/409/429 as blocked; these statuses can come from another layer. Default legacy probes can also be rejected by header validation before payload inspection. A block percentage is not a demonstrated detection rate. The newer matrix uses browser headers except in scenarios designed to alter them.

Rate-limit and brute-force scenarios retain one public client IP throughout, then generate a fresh IP for the next scenario. Other probes rotate per request. Redis is not flushed, so retained reputation may reveal false positives after attack traffic.

Target one runtime or resume observations

python examples/sandbox/smoke-test.py --target django=http://127.0.0.1:3009 --burst-workers 16
python examples/sandbox/validate-login-bypass.py --target java=http://127.0.0.1:8080
python examples/sandbox/validate-runtime-matrix.py --exclude laravel --exclude symfony --exclude wordpress --output examples/sandbox/results_local_payload.json

The matrix's default per-request timeout is five seconds. --timeout changes that requirement; disclose any change when reporting results. --resume retains saved baselines and targets without errors, and reruns failed targets. Keep resumes within the same source/configuration; the matrix alone does not verify commit provenance.

Extended checks with TLS

Caddy exposes only loopback ports 8443/8444. Its internal CA stays in a dedicated Docker volume. Export the public root certificate; pass it explicitly to the runner without changing host trust.

docker compose --profile assessment --profile assessment-tls up -d --build security-fixture aiwaf-assessment assessment-tls
New-Item -ItemType Directory -Force examples/sandbox/assessment/.tls
docker compose --profile assessment --profile assessment-tls cp assessment-tls:/data/caddy/pki/authorities/local/root.crt examples/sandbox/assessment/.tls/root.crt
python examples/sandbox/assessment/run.py --extended --fixture https://localhost:8443 --fixture https://localhost:8444 --ca-file examples/sandbox/assessment/.tls/root.crt --tls-url https://localhost:8443 --tls-url https://localhost:8444 --npm-audit --local-runtime-scope --output examples/sandbox/results_local_owasp_extended.json

Tests require trusted TLS 1.2/1.3, rejection of old protocols, untrusted roots and incorrect hostnames. Client-side inability to test an old protocol is not_assessed, not a server pass. Both profiles are needed because TLS depends on the fixture/proxy. Caddy preserves synthetic client IPs for loopback assessment; that trust arrangement is not a production proxy policy.

For HTTP-only checks, use both --fixture http://localhost:8090 and --fixture http://localhost:3020 with --extended --allow-local-http. HTTP transport remains unassessed. Omitting --npm-audit leaves the advisory query unassessed.

Isolated failures and dependency scans

python examples/sandbox/assessment/outages.py
python examples/sandbox/audit-published-proxies.py --include-sdk --output examples/sandbox/results_local_npm_audits.json
python examples/sandbox/assessment/scan_images.py
python examples/sandbox/assessment/render_report.py

Outage tests run an isolated Express process with actual Redis/upstream connection refusals. They require generic 503 with no application call for Redis, and generic 502 for upstream failure. They do not stop shared Redis or demonstrate failover in all runtimes. Python outage behavior has separate SDK unit/HTTP regressions.

Grype 0.120.0 is pinned by image digest, scans fifteen built images including OS packages, mounts the Docker socket and uses sandbox_grype_db for its advisory database. It exits 1 on high/critical findings or scan errors; lower findings are retained. --service selects images; --resume reuses raw reports only when filesystem layers and runtime configuration match, retaining original scan dates.

Report reference

Files under examples/sandbox/	Contents
`results_local_versions.json`, `results_local_run.json`, `results_local_validation.md`	Source runner provenance, suite exits, aggregate validation. Existing reports may describe an earlier run; inspect timestamps/source.
`results_local_smoke.json`, `results_local_login*.json`, `results_local_payload.json`	Runtime requests, SQL outcomes, matrix observations.
`results_local_owasp*.json`, `results_local_outages.json`, `results_local_npm_audits.json`	Reference/TLS outcomes, isolated failures and npm audits.
`results_local_image_advisories.json`, `results_owasp_scan_*.json`	Image finding summaries and raw package-location evidence.
`results_local_owasp_summary.md`	Category totals, failed image checks, remaining scope and grouped high/critical findings.

The report renderer reads the results_local_* files together; it does not reconcile timestamps or source provenance for every component. Generate fresh components before treating an aggregate as current. Reports are ignored by Git and excluded from build images.

Assessment statuses are pass, fail, error, and not_assessed. Failures/errors cause exit 1. A clean npm audit can coexist with OS image advisories. OWASP scope explains the remaining work.