Rules Pipeline

Goal

Keep the secret detection catalog up to date without trusting live upstream downloads at runtime.

Files

• resources/security/rules/canonical_rules.json: human-maintained source of truth
• resources/security/rules/imports/: optional upstream packs in TOML, JSON, or YAML
• resources/security/generated_rules.json: generated runtime bundle embedded into the binary
• bin/rules_compiler.rs: local generator and CI checker
• .github/workflows/rules.yml: CI validation workflow

Workflow

1. edit resources/security/rules/canonical_rules.json
2. run cargo run --bin rules_compiler
3. commit the regenerated resources/security/generated_rules.json
4. let GitHub Actions run cargo run --bin rules_compiler -- --check
5. run the test suite

Supported import shapes

• canonical SecretRule arrays in JSON, TOML, or YAML
• RulePack { rules: [...] }
• Gitleaks-style [[rules]] packs with id and regex

Why this is safer

• the runtime never fetches untrusted rules on startup
• CI can verify the bundle before merge
• the embedded bundle is deterministic and versioned
• external packs can be added later as signed overlays

Test strategy

Use a small set of seed secrets and generate many positive/negative combinations on the fly:

• text wrappers: plain, key=value, key: value, logs
• structured data: nested JSON, YAML, TOML, config snippets
• negative controls: common words, docs phrases, and benign token-like strings

This catches false positives and false negatives without growing the fixture set too fast.

Runtime filtering

Refer to security_and_compliance.feature for business rules regarding strictness and confidence thresholds.

Crates

In Rust, a crate is a package unit. A crate can be:

• a library crate: reusable code exposed to other crates
• a binary crate: an executable built by Cargo

This project now uses both: the gateway library and the rules_compiler binary.