Python Architecture and Request Lifecycle

Rust-owned CSV request lifecycle

Release candidate: this page documents Python 1.1.8 and Rust/Python 0.2.7 in the current source checkout. Source versions do not establish registry availability. Python 1.1.8 requires aiwaf-rust>=0.2.7; framework dependencies remain extras.

Install the explicit native adapter instead of the existing AIWAF middleware chain. Installing the required Rust dependency does not automatically switch existing applications to this pipeline. Legacy adapters remain available for custom stores and hooks; their configuration and fallback behavior are separate.

Rust owns the native CSV pipeline configuration, enforcement order, SQL and keyword inspection, geo/blocklist decisions, rate buckets, form timing, UUID scores, anomaly history/inference, keyword learning, and CSV logging. Python supplies framework routes, authentication facts, GeoIP lookup, bounded request buffering, and response construction. Training and management still have Python interfaces.

1. Initialize onceValidate the native API, immutable profile and CSV schemas; optionally load a trained model snapshot.
2. Extract framework factsPython resolves routes, methods, decorators, authentication and country; buffers a bounded body and normalizes headers.
3. Begin in RustOne engine call applies enabled request stages and returns an opaque context plus an allow or denial decision.
4. Run the applicationPython dispatches allowed requests or constructs the rejection response.
5. Finish in RustOne response call updates UUID/anomaly state, learns eligible 404 keywords and appends the CSV log.

Enforcement order

geo_block → ip_keyword_block → rate_limit → ai_anomaly → honeypot → uuid_tamper → header_validation → logging. Request checks run before the handler; response-dependent UUID/anomaly observation and logging run during finalization. Model scoring sees the response outcome and can update state for later requests; it cannot retract a response already sent.

Boundaries and guarantees

The engine has no Python policy callbacks or Python execution fallback. Missing native API, invalid schemas, unknown profile keys and unsupported settings fail at startup. Bounded structured JSON/form inspection covers nested escaped strings; byte/traversal limits prevent unlimited parsing. Rust owns a request context, validates engine ownership and prevents duplicate finalization.

The pure ASGI adapter preserves body replay and streaming. Finalization occurs at response start; logged duration measures time to headers, not stream completion. Response elapsed time uses a Rust monotonic clock so wall-clock adjustments do not invalidate requests.

Routes and policies

An unknown path is still evaluated by applicable checks; it is not automatically denied just because no route exists. Full/partial exemption decorators and required middleware supply endpoint facts. Required decorators override decorator exemptions; profile-level path/IP exemptions remain global. Native paths are literal case-insensitive prefixes, not globs. Legacy PATH_RULES, generated manifests, custom stores and uppercase settings are not imported.

State and workers

Policy CSVs retain shared allowlist, blocklist, keyword, geo and path schemas with native locked mutations and validated snapshots. Rate buckets default to process memory; rate_backend="csv" shares persistent buckets. UUID scores and anomaly history are per worker. Redis and custom ORM stores are unsupported in this profile.

Honeypot timestamps default to shared persistent CSV with immediately synced writes. honeypot_backend="memory" uses an expiring Rust map per engine/worker; GET updates and POST evaluation/clearing avoid the durable CSV rewrite. Memory timestamps reset on restart and are not shared with other workers. Switching backend does not migrate existing timestamps.

Request logs append immediately; log_fsync defaults to false and optionally syncs each row. This setting does not weaken synchronization of changed policy-state CSV writes.

Training and inference

Pass a trained six-feature aiwaf_rust.IsolationForest as model= in Flask/FastAPI, or set AIWAF_CSV_PIPELINE_MODEL in Django. Without a loaded model, history and eligible 404 learning continue, but model-based blocking is absent. pipeline.native.set_model(model) copies an owned snapshot; set_model(None) disables inference. Updating the original model does not change the installed snapshot.

CSV/gzip training uses TrainingCSVReader, retained NativeTrainingBatch and NativeTrainingBuilder. Python still coordinates training, compatible timestamps, route discovery and artifact storage. Installing Rust does not mean the entire package has been rewritten.

Source map

• rust/src/csv_pipeline.rs: profile, state and begin/finish enforcement.
• py/aiwaf/core/csv_pipeline.py: startup validation and transport facts.
• py/aiwaf/flask/csv_pipeline.py: Flask hooks.
• py/aiwaf/fast/csv_pipeline.py: pure ASGI transport.
• py/aiwaf/django/csv_pipeline.py: Django request/response adaptation.

Continue with the request lifecycle, the native profile, migration, and performance and deployment.

Shared and legacy module boundaries

Initialization and policy planning

Initialization normalizes settings, selects storage, and resolves the requested middleware set. middleware_plan.py expands canonical selections and applies disabled checks. Route discovery supplies hints such as UUID routes and application intent, while path_manifest.py loads generated path policies.

The resulting middleware plan determines which framework hooks are registered. A route plan then decides whether each registered protection applies to a particular request. Global enablement and per-route applicability are different questions.

How each adapter executes

FastAPI's orchestrator documents the effective available chain as Geo, IP/keyword, Rate, AI, Honeypot, UUID, Header, and Logging. Disabled or inapplicable layers are skipped. Do not infer execution order from the order of add_middleware calls alone.

Following one request

1. Resolve contextRead request metadata and the resolved endpoint where available.
2. Apply route policyCheck path rules and endpoint exemption or protection metadata for each layer.
3. Enforce selected checksConsult block state, counters, validation rules, and the available anomaly model.
4. Observe outcomeAllow the handler or return a denial, then record the response through applicable hooks.

The exact shortcut behavior belongs to each middleware. An exempt path, an exempt client, a geo-only exemption, and a decorator disabling one check are distinct policies. Use the layer-specific decision to explain a bypass rather than assuming that every exemption disables the complete stack.

Live routes, manifests, and decorators

The CLI imports the actual application object or Django URL configuration and enumerates registered routes. Where handler source is available, static analysis adds method, API, form, and authentication signals. The generated .aiwaf/paths.json is a policy input, not a replacement router.

Manifest compilation creates path rules for middleware enablement and rate overrides. Explicit decorators carry endpoint intent: aiwaf_exempt, aiwaf_exempt_from, aiwaf_only, and aiwaf_require_protection. They complement global configuration. Regenerate a manifest when live registrations change so the policy does not describe an older application.

Storage and reputation

The shared runtime storage layer provides memory, JSON-file, SQLite, and canonical policy CSV backends. Flask, FastAPI, Django CSV mode, and the native Rust pipeline use blacklist.csv, whitelist.csv, keywords.csv, geo_blocked_countries.csv, and path_exemptions.csv. Keyword frequencies and non-policy runtime history are stored separately in runtime_metadata.csv. Framework database modes and model storage remain separately configured.

Reputation-based blocklisting stores weighted offenses, temporary-block escalation, expiration, and metadata. A later request reads that state rather than recomputing an entire traffic history. Startup imports legacy runtime key/value CSV data once, retaining original files and a completion marker. Read the shared CSV migration guide before upgrading.

Memory is local to a process. Persisting an operational store does not automatically make every rate bucket or recent-request buffer distributed. FastAPI can supply a Redis-backed rate cache; review the storage semantics of the other enabled checks separately.

Training and scoring

The feature order is part of the model contract. A trainer and scorer must agree on the order, status mapping, time units, and window semantics. Summary fields ending in _ms use milliseconds; the feature vector uses seconds. Sharing feature names alone does not make every model artifact interchangeable.

1. Request logsCompleted responses supply paths, client identity, status, and timing.
2. Feature extractionRecords are normalized and combined with per-client activity windows.
3. Train and reviewVolume thresholds gate model training; route exemptions filter learned keywords.
4. Publish artifactsThe runtime loads persisted model state and updated learned keywords.

The training job is separate from ordinary application request handling. Reading logs and rebuilding a forest does not need to happen for every request. Runtime keyword learning, where enabled, is a separate mechanism that can use completed-response signals.

The trainer parses logs, extracts behavior, filters legitimate route tokens from keyword learning, and trains only when configured volume thresholds permit. Request-time scoring reads an available model; lack of a usable model does not remove deterministic header, rate, keyword, or geo controls.

Model artifacts carry forest state and metadata in JSON. Current Python persistence does not load pickle or joblib objects. That serialization boundary is separate from the choice of file, database, or cache for storing the artifact.

The Rust acceleration boundary

In legacy adapters, rust_backend.py discovers installed aiwaf_rust capabilities. Supported native operations accelerate matching, header validation, feature extraction, and forest operations while the Python adapter owns request/response dispatch. Python 1.1.4 declares the Rust package as a required installation dependency.

The compatibility bridge normalizes HTTP and WSGI header names and retains guarded Python paths for custom integrations or unavailable legacy capabilities. The explicit CSVPipeline instead requires NativeCSVPipeline at startup and keeps its enforcement core in Rust; it does not use a Python policy fallback. These are separate execution choices.

Worked example: an API request and a scanner

A JSON API handler can carry route metadata that omits HTML form timing. Header and rate checks still apply when selected. Allowed requests reach the handler, and the logger records its actual result.

A scanner requesting many missing paths creates a different record: suspicious keyword hits, repeated 404s, and burst activity. Keyword or rate checks may reject it immediately. Sufficient logs later give the trainer behavior data for the anomaly path. Each decision should be explainable by its enabled layer and recorded reason.

System overview connects these components. The setup guides show wiring, and operations covers migration and training commands.

Trace the implementation

Follow these files in the monorepo to connect the architecture to the code:

• py/aiwaf/core/middleware_plan.py
• py/aiwaf/core/path_manifest.py
• py/aiwaf/core/runtime_fastapi.py
• py/aiwaf/flask/flask_integration.py
• py/aiwaf/django/middleware.py
• py/aiwaf/core/runtime_blocklist.py
• py/aiwaf/core/rust_backend.py

Shared Python CSV storage

List APIs and command help now use blocklist and allowlist. Earlier import/method/command spellings remain compatibility aliases. Existing CSV filenames and Django model identity are retained so upgrades preserve stored blocks and exemptions; use the package storage constants rather than constructing filenames.

Python 1.1.4 standardizes Flask, FastAPI, Django CSV mode, and the explicit native pipeline on blacklist.csv, whitelist.csv, keywords.csv, geo_blocked_countries.csv, and path_exemptions.csv. Keyword frequencies and non-policy runtime history are retained in runtime_metadata.csv. Memory, JSON-file, database, and Django ORM modes keep their existing storage behavior.

Startup imports legacy runtime CSV data once, preserving block expiry and metadata. Original sources remain intact; backups and completion markers support recovery and prevent removed entries from returning on restart. Existing canonical policy wins conflicts. Stop old workers before upgrading and use the same absolute directory for the application and CLI. See migration paths and recovery.

Flask management searches existing project aiwaf_data and .aiwaf/data before user/package directories. Explicit AIWAF_DATA_DIR or --data-dir takes priority. The five policy files use UTF-8 across Python and Rust. Sharing these files does not make memory rate buckets, model history, or optional memory form timing distributed.

Flask and FastAPI native honeypot cache

Python 1.1.8 with aiwaf-rust 0.2.7 adds opt-in Redis and supplied-cache timing to both native adapters. CSV remains their default. Rust retains timing rules and exemptions; only timestamp I/O uses the cache.

import os
profile = {
    "data_dir": "aiwaf_data",
    "honeypot_backend": "redis",
    "honeypot_redis_url": os.environ["AIWAF_REDIS_URL"],
    "honeypot_cache_namespace": "myapp:honeypot:v1",
}
# Flask:
from aiwaf.flask.csv_pipeline import NativeAIWAF
NativeAIWAF(app, config=profile)
# FastAPI alternative:
# from aiwaf.fast.csv_pipeline import NativeAIWAFMiddleware
# app.add_middleware(NativeAIWAFMiddleware, config=profile)

For Flask-Caching or a compatible custom object, use honeypot_backend="cache" and pass honeypot_cache=cache to the native adapter. It must provide synchronous get(key) and set(key, value, timeout=seconds), preserve the timestamp/expiry pair, honor TTL and propagate errors. A false write result means failure; silent/no-op backends are unsuitable. Local caches remain worker-local. Use a shared backend and the same namespace across workers; isolate unrelated applications with different namespaces.

Records have a 300-second TTL. Missing/evicted timestamps allow the timing check, while cache errors and invalid records return generic HTTP 503. Expired-page POSTs return 409 until a new GET or TTL expiry; old snapshots are not deleted. FastAPI runs cache-backed native checks in its worker thread pool so synchronous cache I/O does not block the event loop. Other policy stores and legacy middleware behavior are unchanged.

Django native honeypot cache

Python 1.1.8 with aiwaf-rust 0.2.7 defaults Django native honeypot timing to Django's configured cache. Set honeypot_cache_alias in AIWAF_CSV_PIPELINE to select an alias; it defaults to default. Rust calls a small Python cache adapter for timestamp I/O and retains timing decisions, logical expiry and exemptions. Other policy CSVs and rate-limit configuration are unchanged.

AIWAF_CSV_PIPELINE = {
    "data_dir": "aiwaf_data",
    "honeypot_cache_alias": "default",
}
# Configure CACHES["default"] with a shared backend for multiple workers.

Redis or Memcached can share timestamps across workers. LocMemCache is supported with a startup warning; it remains worker-local. DummyCache is rejected when honeypot protection is enabled. Cache errors, failed writes and invalid records produce generic HTTP 503; known ignored-cache-error modes are rejected.

Entries use a 300-second TTL. Expired-page POSTs return 409 until a new GET or TTL expiry. No eager deletion occurs, so an old POST cannot erase a concurrent refresh. A missing or evicted timestamp allows the POST timing check, as before; cache failure is not treated as a miss. Backends do not transfer existing timestamps. Install the matching Rust extension and restart workers together.

Explicit honeypot_backend="csv" retains filesystem timing; "memory" uses Rust's per-engine map. Flask, FastAPI and direct native construction keep CSV as the default. See the Django cache documentation.

Training replay protection

Python 1.1.6 with aiwaf-rust 0.2.5 uses Rust-owned receipts in Flask, FastAPI, and Django. Reusing requests already handled by training cannot extend or recreate a block. Fresh requests must independently satisfy enforcement rules; models and keyword learning can still reuse historical logs.

Keep training_evidence.json across expiry, manual unblocking, log rotation, and restarts. It lives in the policy data directory unless AIWAF_TRAINING_STATE_DIR selects another durable directory. Trainers sharing a blocklist must share that directory. Receipts start with this release and cannot reconstruct decisions made by older versions. Missing native support and corrupt state stop training visibly. Block and receipt writes are separate, so a crash between them can require recovery.

Flask CLI listing now filters expired entries without rewriting an older snapshot over blocks concurrently added or refreshed by a trainer. The CLI displays the canonical added_date, with the older timestamp as a fallback. This listing fix and training replay protection address different causes of block changes. A first temporary block defaults to 900 seconds; later reputation decisions can increase its duration. Check the stored expiry rather than assuming every block lasts 24 hours.