AI Assistance

Overview

CraftedSignal uses AI to assist detection engineers — never to auto-deploy or make autonomous decisions. AI is optional, transparent, and can be disabled entirely.

What AI does

Rule generation

Describe what you want to detect. AI generates:

• The detection rule (SPL, KQL, or FalconQL)
• Positive and negative test cases
• MITRE ATT&CK mapping
• Context (rationale, assumptions, noise expectations)
• Runbook and playbook draft, when response content is enabled

AI-generated rules are created in the web UI. Describe the threat, select your target platform, and the AI produces a complete rule with tests, MITRE mapping, and optional response steps for you to review before pushing.

Runbook and playbook drafts

AI can draft Markdown response steps for a rule. The draft uses this structure:

• Runbook: alert intent, triage steps, evidence to collect, false-positive checks, and escalation criteria.
• Playbook: containment, eradication and recovery, communications and handoff, and follow-up detection work.

Drafts are grounded in the rule logic, platform, data sources, fields, severity, MITRE mapping, tags, tests, and company context. They remain suggestions until a human reviews and saves them.

Existing runbooks and playbooks can also inform later AI refinement of the rule, tests, escalation criteria, and follow-up detection work. The goal is to keep the rule and response steps aligned in both directions.

Translation linting

When a rule is translated across platforms (e.g., SPL to KQL), AI highlights semantic differences that could affect detection accuracy.

Health insights

AI analyzes rule performance and suggests improvements:

• Tuning recommendations for noisy rules
• Query optimization suggestions
• Coverage gap recommendations from threat intel
• Missing runbook and playbook recommendations for rules that do not yet explain the analyst response
• Stale runbook and playbook recommendations when rule or hunt context changes

Autofix

AI can suggest fixes for rules that fail validation or testing. You review and approve the suggestion before it’s applied.

Threat actor adjudication

When the threat feed ingests a brief that names an actor not in the catalog, AI normalizes the name against the existing threat-actor catalog . It returns one of three structured decisions: alias an existing actor, create a new entry, or skip when the string isn’t a threat actor. The decision and confidence score are recorded in the LLM usage log.

When AI is disabled, the catalog stops growing — exact-match still works, unmatched actor names just stay unlinked.

Hunt outcome and digest summaries

After a hunt completes, AI summarizes the evidence into a human-readable paragraph stored on the risk’s lifecycle timeline. Campaign closes and the threat-feed digest are summarized the same way. These summaries are advisory; the underlying clusters, verdicts, and briefs are the source of truth.

Usage tracking

Every AI call is logged with model, input/output token counts, cached-token counts, cost estimate, and the activity that triggered it. Tracked activities include:

• actor_adjudication — name normalization in the feed bridge.
• novel_chain_extraction — attack-chain analysis from briefs.
• hunt_outcome_summary — post-hunt evidence narrative.
• campaign_close_summary — campaign-level wrap-up.
• digest_narrative — feed digest copy.

The log table is queryable per-company per-time-window for cost analytics and audit. Surfaces in the AI Quality screen for owners (/ai-quality), where you can see per-activity volume, cost, and the prompt → response history for spot-checking the model.

Cost is best-effort: providers that don’t return native cost data (e.g., self-hosted Ollama) record token counts and a $0 estimate. Token counts are always recorded.

Guardrails

Human approval required

AI suggestions are never auto-deployed. Every AI-generated or modified rule requires explicit human approval before it reaches your SIEM.

Explainability

Every AI suggestion includes:

• The prompt that was used
• The diff between current and suggested rule
• The suggested runbook/playbook draft, when one is generated
• A confidence score
• Reasoning for the suggestion

Data minimization

• No raw logs leave your boundary
• PII is redacted before processing
• AI sees only rule logic and metadata, never customer telemetry

Safety checks

AI-generated rules go through the same validation pipeline as human-written rules: lint, test, shadow eval, approval. AI-generated runbooks and playbooks remain editable Markdown and should be reviewed against the rule before analysts use them.

Self-hosted AI

Run AI features entirely on your infrastructure using Ollama:

ai:
  enabled: true
  ollama_url: "http://localhost:11434"
  ollama_model: "qwen2.5-coder:14b"

When self-hosted, no data leaves your network. CraftedSignal never sends rule data to external AI services unless you explicitly configure it. See Configuration for all AI settings.

Disable AI

If your security policy prohibits AI, disable it entirely:

ai:
  enabled: false

All AI features are removed from the UI and CLI. The platform works fully without AI — it’s an enhancement, not a dependency. Runbooks and playbooks can still be written and reviewed manually.

Data policy

• CraftedSignal never trains on your data
• AI interactions are logged in the immutable audit trail
• You control which AI model is used and where it runs

See Runbooks & Playbooks for sync behavior, response content structure, and the review checklist.