Python Operations and Migration

Rust-backed CSV operations and training

Python 1.1.8 requires aiwaf-rust>=0.2.7 as a base dependency. Framework extras select Django, Flask or FastAPI. The optional rust extra remains an installation alias. Legacy capability selection is automatic unless disabled; the new complete pipeline requires the explicit adapter.

pip install "aiwaf[flask,rust]"
# Run these repository scripts with the matching installed packages.
python scripts/check_native_csv_runtime.py
python scripts/check_csv_release_concurrency.py

# FastAPI: import the live application and use its routes/runtime stores.
aiwaf fastapi train --app app:app --log-dir aiwaf_logs --disable-ai
aiwaf fastapi train --app app:app --log-dir aiwaf_logs --model-path model.json
# Django retains its management-command workflow.
python manage.py detect_and_train

FastAPI also exposes aiwaf.fast.trainer.train_from_logs(app, ...). --data-dir selects the shared policy CSV directory and imports an existing legacy runtime_store.csv once. Training writes shared keyword and blocklist files; keyword frequencies are retained in auxiliary runtime metadata. AI training obeys its volume threshold; --force-ai explicitly overrides that threshold. Training optionally writes a JSON model artifact. Reload the model explicitly or restart the application: existing FastAPI anomaly middleware instances do not automatically adopt the new forest.

Flask, Django CSV, and FastAPI stream plain/gzip CSV directly in bounded batches, including quoted newlines and seconds/milliseconds timing fields. Supported Apache logs use batched extraction and timestamp parsing. Django preserves its legacy CSV timezone/microsecond truncation; Flask and FastAPI preserve ISO offsets. Unique route checks, per-IP aggregation, and keyword persistence are batched. Native feature handles keep the matrix, forest fitting/prediction, and anomaly statistics in Rust, without a Pandas DataFrame. Custom parsers, missing capabilities, and Unicode keyword learning retain compatibility paths.

CSV access logs are written and flushed synchronously under OS locks. Cached writer handles detect rotation, replacement, and truncation. After requests stop, call the Flask logger's close() or aiwaf.core.native_runtime.close_csv_logs() before deleting temporary directories, especially on Windows. Handles also close at process shutdown. AIWAF_CSV_LOG_FSYNC=True adds per-row disk sync; without it, a system crash can lose recently flushed OS-buffered entries. There is no application-side logging queue.

Native pipeline performance and deployment

Release candidate: this page documents Python 1.1.8 and Rust/Python 0.2.7 in the current source checkout. Source versions do not establish registry availability. Python 1.1.8 requires aiwaf-rust>=0.2.7; framework dependencies remain extras.

Matched WSL Linux measurements used all eight stages, immediate CSV logging, 1,000 blocklist entries, 80% GET / 20% JSON POST (30 fields), and one concurrent client. Three alternating 300-request runs per framework/mode used a seeded 100-tree Rust forest and 200 initial history records per client when AI was enabled. GeoIP reflects a local fixture.

These are added server processing latency: median of per-run mean protected times minus the corresponding framework-only baseline. They are not total request latency, network latency or production throughput. Memory reduced measured overhead by 73–79% versus persistent CSV timing, with different persistence and worker-sharing semantics. All 13,500 measured requests returned 200; protected PHP/SQL probes returned 403.

Reproduce from the monorepo with scripts/benchmark_release_http.py --pipeline; add --honeypot-backend memory for memory mode. Include a framework-only baseline using the same handler, workload and server. Compare your production filesystem, worker count and real GeoIP/model workload before setting latency expectations.

Model and worker lifecycle

Pass a trained Rust IsolationForest as model=, or use Django's AIWAF_CSV_PIPELINE_MODEL. Load a new snapshot with pipeline.native.set_model(model) where you hold the engine reference, or reload workers. Native initialization does not automatically train or load the legacy adapter's model artifact. The runtime's diagnostics() reports model activation and state counts.

Release order

Publish aiwaf-rust 0.2.7 before aiwaf 1.1.8. Native wheels target Linux x86_64/aarch64, musl x86_64, macOS x86_64/arm64 and Windows x86_64. Windows and WSL builds were checked locally; all remaining wheel targets must pass release CI. Local validation passed eight test groups and ten independent Windows native API smoke checks. These local checks do not replace platform CI.

Continue with the request lifecycle, the native profile, migration, and performance and deployment.

Shared Python CSV storage

List APIs and command help now use blocklist and allowlist. Earlier import/method/command spellings remain compatibility aliases. Existing CSV filenames and Django model identity are retained so upgrades preserve stored blocks and exemptions; use the package storage constants rather than constructing filenames.

Python 1.1.4 standardizes Flask, FastAPI, Django CSV mode, and the explicit native pipeline on blacklist.csv, whitelist.csv, keywords.csv, geo_blocked_countries.csv, and path_exemptions.csv. Keyword frequencies and non-policy runtime history are retained in runtime_metadata.csv. Memory, JSON-file, database, and Django ORM modes keep their existing storage behavior.

Startup imports legacy runtime CSV data once, preserving block expiry and metadata. Original sources remain intact; backups and completion markers support recovery and prevent removed entries from returning on restart. Existing canonical policy wins conflicts. Stop old workers before upgrading and use the same absolute directory for the application and CLI. See migration paths and recovery.

Flask management searches existing project aiwaf_data and .aiwaf/data before user/package directories. Explicit AIWAF_DATA_DIR or --data-dir takes priority. The five policy files use UTF-8 across Python and Rust. Sharing these files does not make memory rate buckets, model history, or optional memory form timing distributed.

Flask and FastAPI native honeypot cache

Python 1.1.8 with aiwaf-rust 0.2.7 adds opt-in Redis and supplied-cache timing to both native adapters. CSV remains their default. Rust retains timing rules and exemptions; only timestamp I/O uses the cache.

import os
profile = {
    "data_dir": "aiwaf_data",
    "honeypot_backend": "redis",
    "honeypot_redis_url": os.environ["AIWAF_REDIS_URL"],
    "honeypot_cache_namespace": "myapp:honeypot:v1",
}
# Flask:
from aiwaf.flask.csv_pipeline import NativeAIWAF
NativeAIWAF(app, config=profile)
# FastAPI alternative:
# from aiwaf.fast.csv_pipeline import NativeAIWAFMiddleware
# app.add_middleware(NativeAIWAFMiddleware, config=profile)

For Flask-Caching or a compatible custom object, use honeypot_backend="cache" and pass honeypot_cache=cache to the native adapter. It must provide synchronous get(key) and set(key, value, timeout=seconds), preserve the timestamp/expiry pair, honor TTL and propagate errors. A false write result means failure; silent/no-op backends are unsuitable. Local caches remain worker-local. Use a shared backend and the same namespace across workers; isolate unrelated applications with different namespaces.

Records have a 300-second TTL. Missing/evicted timestamps allow the timing check, while cache errors and invalid records return generic HTTP 503. Expired-page POSTs return 409 until a new GET or TTL expiry; old snapshots are not deleted. FastAPI runs cache-backed native checks in its worker thread pool so synchronous cache I/O does not block the event loop. Other policy stores and legacy middleware behavior are unchanged.

Django native honeypot cache

Python 1.1.8 with aiwaf-rust 0.2.7 defaults Django native honeypot timing to Django's configured cache. Set honeypot_cache_alias in AIWAF_CSV_PIPELINE to select an alias; it defaults to default. Rust calls a small Python cache adapter for timestamp I/O and retains timing decisions, logical expiry and exemptions. Other policy CSVs and rate-limit configuration are unchanged.

AIWAF_CSV_PIPELINE = {
    "data_dir": "aiwaf_data",
    "honeypot_cache_alias": "default",
}
# Configure CACHES["default"] with a shared backend for multiple workers.

Redis or Memcached can share timestamps across workers. LocMemCache is supported with a startup warning; it remains worker-local. DummyCache is rejected when honeypot protection is enabled. Cache errors, failed writes and invalid records produce generic HTTP 503; known ignored-cache-error modes are rejected.

Entries use a 300-second TTL. Expired-page POSTs return 409 until a new GET or TTL expiry. No eager deletion occurs, so an old POST cannot erase a concurrent refresh. A missing or evicted timestamp allows the POST timing check, as before; cache failure is not treated as a miss. Backends do not transfer existing timestamps. Install the matching Rust extension and restart workers together.

Explicit honeypot_backend="csv" retains filesystem timing; "memory" uses Rust's per-engine map. Flask, FastAPI and direct native construction keep CSV as the default. See the Django cache documentation.

Training replay protection

Python 1.1.6 with aiwaf-rust 0.2.5 uses Rust-owned receipts in Flask, FastAPI, and Django. Reusing requests already handled by training cannot extend or recreate a block. Fresh requests must independently satisfy enforcement rules; models and keyword learning can still reuse historical logs.

Keep training_evidence.json across expiry, manual unblocking, log rotation, and restarts. It lives in the policy data directory unless AIWAF_TRAINING_STATE_DIR selects another durable directory. Trainers sharing a blocklist must share that directory. Receipts start with this release and cannot reconstruct decisions made by older versions. Missing native support and corrupt state stop training visibly. Block and receipt writes are separate, so a crash between them can require recovery.

Flask CLI listing now filters expired entries without rewriting an older snapshot over blocks concurrently added or refreshed by a trainer. The CLI displays the canonical added_date, with the older timestamp as a fallback. This listing fix and training replay protection address different causes of block changes. A first temporary block defaults to 900 seconds; later reputation decisions can increase its duration. Check the stored expiry rather than assuming every block lasts 24 hours.

Python 1.1.4 CSV migration checks (historical)

This source release adds installed-framework CLI detection, the explicit fastapi alias, reuse of existing Flask data directories, and shared policy CSV startup migration. It keeps aiwaf-rust>=0.2.3; these changes require no new native API.

# Show the deployed interpreter's installed versions.
python -m pip show aiwaf aiwaf-rust
python -c "import aiwaf; print(aiwaf.__version__)"

# Flask-style policy management can inspect the shared policy directory.
aiwaf flask --data-dir /absolute/path/aiwaf_data status

# Route-aware FastAPI training uses the same policy directory.
aiwaf fastapi --data-dir /absolute/path/aiwaf_data train --app app:app --log-dir aiwaf_logs --disable-ai

Stop older workers before importing legacy CSV data. Original sources, pre-import backups, and completion markers remain on disk. Correct invalid legacy input and restart to retry an incomplete migration; subsequent writes to a completed legacy source are not replayed. Use the same directory for application, training, and management commands.

Local validation passed a broad 869-test core/Flask/FastAPI run, 284 tests through Django's test runner, and focused migration/UTF-8 compatibility checks. The 1.1.4 wheel and source archive were built and the packaged version, CSV backend, and required Rust dependency were verified. Source artifacts do not establish that a registry release has been published.

The benchmark tables on this page predate the 1.1.4 migration changes. Rerun the same baseline-subtracted benchmark for the deployed version and filesystem.

October 2026 release upgrade

The prepared source versions are Python 1.1.6, Rust/Python 0.2.5, Node.js 1.1.4, WASM 0.2.4, and Java 1.3.4. These are source targets; verify registry availability before installing. Python requires aiwaf-rust>=0.2.5, including the TrainingEvidence API. Node offline training requires the matching WASM receipt API.

1. Back up policies, logs, models, and training evidence. Use the same absolute data and evidence directories for application workers and CLI trainers.
2. Install the matching native companion before upgrading its host package: Rust before Python, WASM before Node. Preserve training_evidence.json and node_training_evidence.json across expiry, unblocking, restarts, and log rotation.
3. Stop older file writers and restart all workers on the new code. Mixing old unlocked writers with new writers can still lose updates.
4. In staging, train twice on identical logs, then verify handled evidence does not recreate a removed or expired block. Add genuinely fresh requests and confirm they must independently satisfy the enforcement threshold.
5. Verify that listing and expiry checks preserve concurrently refreshed blocks and unrelated IPs. Check the raw expiry and backend location if entries disappear unexpectedly.

Java CSV and file storage

Java 1.3.4 reloads current storage under a shared JVM and OS sidecar lock for each operation. Reads filter expiry without rewriting disk; mutations preserve other instances' updates and report persistence failures. This adds file I/O. Java model training still does not add blocklist entries. Rate buckets and other per-engine memory state retain their existing isolation.

Lock recovery

Python training uses an OS file lock; Java CSV/file OS locks also release on process exit. Keep their stable sidecar lock files in place while workers run. Node uses lock directories for training and legacy CSV mutations; a crash can leave them behind. Confirm the owning process has stopped before removing a Node lock directory. Do not delete training receipts to clear a lock: doing so loses handled-evidence history.

Release checks

Local validation passed 247 Node tests across 67 suites with the coverage gate, 215 Java tests with 3 skips, and 38 Rust/WASM native tests. After the native dependency bump, 11 affected Python packaging tests passed through the appropriate Django and pytest runners. These checks do not replace full release CI, dependency audits, signing, or operating-system wheel builds. Build Linux/macOS Rust wheels using the release matrix; the local bundle contains a Windows x86-64 wheel.

For release maintainers: publish the WASM companion, refresh the Node lockfile from the registry to record its actual integrity, then run npm ci and the native integration checks. Do not substitute an older native dependency to satisfy a stale packaging test. See the changelog.