Threat Intelligence Lifecycle in DKTrace — From IOC Ingest to Alert

The Problem With Raw Threat Intelligence

Raw TI is noise. An IP address that was malicious 18 months ago is probably a recycled residential IP today. A malware hash from 3 years ago is likely in every AV database already. The value of TI is in how you ingest, score, deduplicate, and operationalise it — at detection speed.

DKTrace's TI Pipeline

Stage 1: Ingest (15+ Feeds Supported)

Stage 2: Confidence Scoring With Age Decay

def effective_confidence(ioc):
    age_days = (now() - ioc.first_seen).days
    
    # Half-lives by IOC type (how quickly IOCs go stale)
    half_lives = {
        "ip":      7,    # IPs recycle fast
        "domain":  30,   # Domains held longer by attackers
        "hash":    365,  # File hashes are permanent
        "cve":     None, # CVEs don't decay
    }
    half_life = half_lives.get(ioc.type, 30)
    
    if half_life:
        decay = math.exp(-age_days / half_life)
    else:
        decay = 1.0
    
    return ioc.base_confidence * ioc.source_reliability * decay

IOCs with effective_confidence < 0.3 are suppressed from alerting but retained for threat hunting queries.

Stage 3: Sub-Millisecond Lookup

All active IOCs (confidence ≥ 0.3) are loaded into Redis bloom filters at startup and refreshed every 60 seconds.

The enrichment-engine checks each canonical event's IPs, domains, hashes, and JA3 fingerprints against these bloom filters in < 0.5ms per event — without touching PostgreSQL for the hot path.

False positive rate of bloom filter: < 0.01% (tuned to 1M+ expected elements, 12 hash functions).

Stage 4: Deduplication

IOCs are deduplicated on (ioc_type, ioc_value). When the same indicator appears in multiple feeds:

This prevents the same IP being stored 8 times from 8 feeds, and inflating confidence through double-counting.