// RESEARCH STATUS — COMPLETE · rev 2026-06-12
// IN PLAIN TERMS — Phishing works by manipulating people — urgency, authority, fear — not just by tripping a spam filter. This research asks whether AI can understand why a scam message is convincing, because a detector that grasps the manipulation is far harder for an attacker to slip past than one that only matches suspicious words.
// Research Question — Can a frontier large language model move beyond pattern-matching to reason causally about the psychological mechanisms that drive social engineering success? Not whether an LLM can flag a phishing message — but whether it can explain why that message works, and whether its explanation matches statistically inferred ground truth.
| Headline | Value |
|---|---|
| Phishing emails (raw) | 88,647 |
| Phishing emails (cleaned) | 59,788 |
| GPT-4 DAG alignment | 94.2% |
| Claude 3 Sonnet alignment | 85.7% |
| Gemini 2.5 Pro alignment | 72.3% |
| DeepSeek-67B alignment | 53.0% |
| ICC inter-rater reliability | 0.98 (95% CI 0.94–0.99) |
Why Causal Reasoning Matters
Social engineering attacks succeed because they exploit cognitive heuristics — urgency, authority, trust — not software vulnerabilities. Existing detection systems model emails as unordered feature collections, flagging lexical anomalies while ignoring the causal drivers that persuade recipients to comply. Verizon's 2024 DBIR finds the human element present in 68% of confirmed breaches.
Causal reasoning produces defences that survive adversarial paraphrase. A correlational filter that has learned the token "URGENT" co-occurs with phishing fails the moment an attacker writes "time-sensitive", or moves the same pressure from email to a phone call — the surface feature has changed but the manipulation has not. A causal model instead learns what actually causes compliance: structural drivers that cannot be rewritten without defeating the attack itself. That is the property an adaptive adversary cannot trivially evade.
Why correlation breaks and causation holds — the four-step argument:
- The original phishing email ("Subject: URGENT — verify your account within 24 hours") —
both detector types flag it: the correlational filter because the token "URGENT" co-occurs with phishing in training data; the causal detector because the urgency → deception → compliance chain fires at the construct level.
- Adversarial paraphrase ("Subject: Time-sensitive — your account requires review today") —
the correlational filter misses: the learned token disappeared. The causal detector still flags: urgency is still doing the causal work; paraphrase cannot remove the mechanism without removing the pressure itself.
- Channel shift (a phone call: "This is your bank's security team — we need to confirm your
details right now") — the correlational email filter has nothing to evaluate. The causal detector still flags: the constructs (authority, urgency) are channel-independent; the validated DAG models the manipulation, not the medium.
- Why this is the research result — surface features can always be rewritten; the causal
driver cannot be removed without defeating the attack. That is exactly what the LLM benchmark measures: GPT-4 reconstructs 94.2% of the validated causal graph; DeepSeek-67B only 53.0% — the gap an adaptive adversary exploits.
// Context — Verizon's 2024 DBIR finds the human element present in 68% of confirmed breaches. The attack surface is not software — it is the human response to manipulation. This research asks whether AI can model that response at a causal level, not merely flag its surface markers.
Datasets
Three datasets span the e-mail, SMS, and voice social-engineering channels. All are mapped to the same five-construct schema so that score differences reflect reasoning skill, not schema mismatch. All records are anonymised; no model was asked to generate live phishing content.
| Channel | Raw rows | Cleaned rows | Malicious % | Features |
|---|---|---|---|---|
| E-mail phishing | 88,647 | 59,788 | 31.15% | 10 |
| SMS smishing | 67,008 | 67,008 | 39.07% | 10 |
| Synthetic vishing (CVAE) | 60,000 | 60,000 | 30.00% | 10 |
E-mail phishing (primary benchmark): Vrbančič, Fister and Podgorelec (2020) open corpus. 88,647 raw rows reduced to 59,788 after deduplication and normalisation. URL- and domain-derived indicators: shortened URLs, redirect and resolved-IP counts, index presence, TLS/SSL status, embedded email-in-URL, domain age, response time.
SMS smishing: Salman, Ikram and Kaafar (2024) evasion-optimised SMS corpus. 67,008 rows; brand-mimicry and evasive text signals. Cross-channel robustness benchmark (text channel).
Synthetic vishing (CVAE): Generated with a Conditional Variational Auto-Encoder preserving the latent causal scaffold while removing channel-specific surface cues. No real voice recordings processed — privacy by design. 60,000 rows; synthetic binary flags for call scenarios (caller-ID spoofing, authority-based scripts). Enables fair cross-channel comparison.
Dual-Pathway Framework
The research fuses two analytical traditions rarely combined: causal graph discovery from real data, and structured LLM interrogation against those graphs.
Pathway 1 — DAG Construction
Four causal-discovery algorithms attacked each dataset independently:
| Algorithm | Type | Key parameter |
|---|---|---|
| GES (Greedy Equivalence Search) | Score-based | — |
| PC-Algorithm (Peter–Clark) | Constraint-based, conditional-independence testing | α = 0.05 |
| Bayesian Networks | Probabilistic | — |
| DeepNOTEARS | Gradient-descent structure learning | L1 = 0.01 |
Graphs merged into two hybrid ensembles: GES ∪ BN and PC ∪ DeepNOTEARS. Bidirectional and cyclic edges pruned; behaviourally grounded conflicting edges retained. All intermediate graph iterations stored for rollback and comparison. Discovery runs from causal_inference_framework.ipynb, branching on dataset tags.
Cross-dataset DAG robustness results:
| Channel | GES-BN | PC-DNT | DoWhy result | Notes |
|---|---|---|---|---|
| Phishing | Pass | Pass | Success | Full consensus |
| Vishing | Pass | Pass | Success | Edge divergence |
| Smishing | Pass | Fail | Partial | NOTEARS acyclicity failure |
Pathway 2 — LLM Evaluation
Four frontier models received 36 structured JSON prompts distributed across five reasoning categories:
| Prompt category | Instances | Proportion |
|---|---|---|
| Probability | 6 | 16.7% |
| Conditional | 6 | 16.7% |
| Impact Ranking | 6 | 16.7% |
| Inverse Reasoning | 6 | 16.7% |
| Fixed Prompts | 12 | 33.3% |
Fixed prompts targeted direct edge recognition, multi-step causal chain inference, and construct prioritisation. Inverse prompts (construct removed; causal arrow reversed) tested the hardest adversarial conditions.
Each response scored across five equally weighted dimensions (wᵢ = 0.2): S_LLM = Σ(wᵢ × dᵢ) on a 0–5 scale.
| Dimension | What it measures |
|---|---|
| Awareness | Construct explicitly named as causally relevant |
| Depth | Explanation reaches a psychological mechanism, not just correlation |
| Structure | Causal chain is coherent and acyclic |
| Directionality | Arrows stay correct under reversed-direction prompts |
| Generalisability | Reasoning holds when features drift or channel changes |
Execution controls (determinism):
| Model | Access | Sampling |
|---|---|---|
| GPT-4 (0613) | Secured desktop environment | Lowest-entropy default preset |
| Claude 3 Sonnet | Web interface | "Precise" preset |
| Gemini 2.5 Pro | Web interface | Default deterministic preset |
| DeepSeek-67B | 2× RunPod H100 SXM (80 GB VRAM, 16 vCPU, 125 GB RAM) | temperature 0.70, top-p 0.95, max_tokens 1024 |
Feature-to-construct mapping:
| Construct | Representative technical features |
|---|---|
| Obfuscation | Shortened URLs, redirect count, resolved-IP count |
| Trust | URL in Google index, TLS/SSL certified, domain in Google index |
| Authority | Email address embedded within URL |
| Deception | Time of domain activation (domain age) |
| Urgency | Website response time |
Construct scores: C_j = (1/|S_j|) Σ f_i m_ij — normalised mean of contributing feature values. Z-score and sum variants retained for sensitivity checks.
Validated causal chains (primary phishing DAG):
Urgency → Deception → Phishing
Trust → Obfuscation → Phishing
Authority → Deception → Phishing
Deception: convergent mediator across all four discovery methods
Obfuscation: technical amplifier — enables rather than initiates manipulationReproducible pipeline steps (public-safe)
DoWhy four-stage pipeline for each construct→outcome edge:
- State assumptions — encode the causal graph.
- Identify the estimand — derive the target causal quantity.
- Estimate — binomial GLM.
- Refute — robustness checks (see DoWhy Validation below).
ICC(2,1) computed via pingouin.intraclass_corr, two-way random-effects model, two rating waves × four raters × five scoring dimensions.
DeepSeek-67B evaluated via run_deepseek.py on two RunPod H100 SXM GPU pods at fixed sampling parameters (temperature 0.70, top-p 0.95, max_tokens 1024).
DoWhy Validation Results
Every construct→outcome edge passed through the DoWhy four-stage pipeline: stating assumptions, identifying the estimand, binomial GLM estimation, and refutation. Refutation methods applied: n = 500 Monte-Carlo placebo permutations, random-common-cause injection, and bootstrapped subset tests. Minimum empirical p across placebo runs: p ≈ 0.002.
| Construct | β (log-odds) | Placebo p | Rand-CC p | Subset p | Verdict |
|---|---|---|---|---|---|
| Obfuscation | 0.116 | 0.002 | 0.92 | 0.92 | Pass |
| Trust | 0.273 | 0.002 | 0.84 | 0.92 | Pass |
| Urgency | −0.067 | 0.002 | 1.00 | 0.96 | Pass |
| Deception | −0.177 | 0.002 | 0.68 | 0.95 | Pass |
| Authority | −0.441 | 0.002 | 0.94 | 0.98 | Pass |
All five constructs passed all three refutation methods. The validated hybrid DAGs served as ground truth for all LLM scoring.
LLM Results — DAG Alignment
| Model | Alignment (/20) | Fidelity (/60) | DAG Alignment % | S_LLM (/5) |
|---|---|---|---|---|
| GPT-4 | 18.5 | 58.0 | 94.2% | 4.60 |
| Claude 3 Sonnet | 16.0 | 56.5 | 85.7% | 4.14 |
| Gemini 2.5 Pro | 14.0 | 45.5 | 72.3% | 3.45 |
| DeepSeek-67B | 10.0 | 34.5 | 53.0% | 2.44 |
ICC(2,1) = 0.98 (95% CI ≈ 0.94–0.99) — "almost perfect" agreement (Shrout and Fleiss, 1979). Individual dimension ICC ranged from 0.89 (Directionality) to 0.97 (Structure).
// Reading these numbers — DAG alignment is the share of expert-validated causal edges a model reproduces in its own explanations. GPT-4's 94.2% means it reconstructed almost the entire validated causal graph and held it under counter-factual prompts — genuine causal structure, not paraphrased correlation. DeepSeek-67B's 53.0% means it reproduced barely half: it recognised individual constructs but could not reliably reconstruct the causal links between them, the exact failure an adaptive adversary would exploit. The ICC of 0.98 is what makes any of these scores trustworthy in the first place — across two rating waves and four independent raters the scoring was almost perfectly reproducible, so the ranking reflects the models, not rater subjectivity.
Dimensional breakdown (S_LLM components):
| Model | Awareness | Depth | Structure | Directionality | Generalisability |
|---|---|---|---|---|---|
| GPT-4 | 5.00 | 4.67 | 4.58 | 4.50 | 4.25 |
| Claude 3 Sonnet | 4.83 | 4.58 | 3.92 | 4.00 | 3.38 |
| Gemini 2.5 Pro | 4.00 | 3.58 | 3.67 | 3.50 | 2.50 |
| DeepSeek-67B | 3.08 | 2.67 | 2.33 | 2.50 | 1.62 |
Construct-level interpretation fidelity (A = Awareness 0–5, D = Depth 0–5):
| Construct | GPT-4 (A/D) | Claude (A/D) | Gemini (A/D) | DeepSeek (A/D) |
|---|---|---|---|---|
| Deception | 5.0 / 5.0 | 5.0 / 5.0 | 5.0 / 4.5 | 4.0 / 3.5 |
| Urgency | 5.0 / 5.0 | 5.0 / 5.0 | 4.5 / 4.0 | 3.0 / 3.0 |
| Obfuscation | 5.0 / 4.5 | 5.0 / 5.0 | 3.5 / 3.5 | 3.5 / 2.5 |
| Authority | 5.0 / 4.5 | 5.0 / 4.5 | 3.5 / 3.5 | 2.5 / 2.5 |
| Trust | 5.0 / 4.5 | 4.5 / 4.0 | 4.0 / 3.0 | 2.5 / 2.5 |
| Causality | 5.0 / 4.5 | 4.5 / 4.0 | 3.5 / 3.0 | 3.0 / 2.0 |
Model-specific findings:
GPT-4 (94.2% · S_LLM 4.60/5) — Highest composite score. Accurately identified mediating nodes and demonstrated deep awareness of construct interactions. Only model to consistently attempt genuine causal explanation rather than correlational description. Superior performance in chain construction, ranking tasks, and inverse reasoning.
Claude 3 Sonnet (85.7% · S_LLM 4.14/5) — Strong on Deception and Urgency; high logical consistency. Moderate success in multi-step chains. Occasionally lacked deeper abstraction in comparative tasks. Faltered when reasoning required linking constructs into longer causal sequences.
Gemini 2.5 Pro (72.3% · S_LLM 3.45/5) — Consistent construct recognition with reduced interpretive depth. Responses frequently descriptive rather than inferential — particularly in prioritisation and causal explanation tasks. Struggled with inverse-direction and missing-construct prompts.
DeepSeek-67B (53.0% · S_LLM 2.44/5) — Evaluated on 2× RunPod H100 SXM GPU pods (80 GB VRAM, 16 vCPU, 125 GB RAM); temperature 0.70, top-p 0.95, max_tokens 1024. Weakest DAG alignment. Misranked trust and urgency; treated trust cues as protective rather than exploitable; failed to map key indirect causal chains. Responses became less stable under adversarial prompt twists despite its parameter scale.
Cross-Channel Stress Tests
Core causal edges — Urgency → Deception and Trust → Authority — survived perturbation across e-mail and voice datasets, supporting Pearl's (2009) criterion that genuine causal relations should survive surface perturbations.
Top models lost up to 0.42 depth points when features drifted to the synthetic vishing set, revealing fragility in real-world cross-channel transfer. Misclassifications clustered around low-depth constructs (trust, obfuscation), reinforcing that causal depth and alignment are partially linked.
Key Finding
// Principal Conclusion — Current frontier LLMs can detect surface patterns in social engineering and reproduce coarse causal structure, but consistently struggle with multi-hop causal chains and inverse reasoning. GPT-4 at 94.2% alignment approaches human-level causal fluency in controlled conditions; models at the lower end of the benchmark revert to correlational explanation that would fail under adversarial paraphrase. The persistent gap between awareness and depth scores across all four models indicates that construct recognition is necessary but insufficient for reliable causal defence.
Adversarial Evasion — Why Causal Detection Resists It
The practical pay-off, in plain terms. A correlational detector (essentially every mainstream phishing filter) learns the surface of today's phishing — the word "URGENT", a sender format, a link shape. An adversary with access to the detector simply sends variations, keeps the ones that score below the block threshold, and tunes their way past: swap "URGENT" for "time-sensitive", reshape the URL — the message still works on the human, but the filter no longer recognises it. The surface is what the attacker controls.
A causal detector learns what makes phishing succeed — urgency, authority, trust, deception — not how it looks. One sentence makes it robust:
// The robustness argument — The surface features can be changed freely; the causal structure cannot be changed without defeating the attack itself. Strip the urgency, authority and deception to evade a causal detector and you have removed the levers that make the recipient act — there is no successful phishing left to catch. In the cross-channel stress tests the core edges (Urgency→Deception, Trust→Authority) survived a move from e-mail all the way to synthetic voice — Pearl's criterion in operational form.
The open question is whether today's LLMs can actually do this reasoning. The benchmark answers honestly: GPT-4 reproduced 94.2% of the validated causal structure; the weakest model (DeepSeek-67B, 53.0%) reverted to correlational description. Causal robustness is only as strong as the reasoner implementing it.
From Research to Deployment — How a SOC Would Use This
The evasion-resistance argument is the why; this is the how — the path from a research DAG to an enrichment layer on a production email gateway, framed as detection-engineering work, not a paper:
- **Build the causal graph on your baseline** — re-run the discovery pipeline over the
organisation's own confirmed-phishing and confirmed-legitimate mail. The constructs (urgency, authority, trust, deception, obfuscation) generalise; the edge strengths should reflect the adversaries that org actually faces.
- Train the DoWhy estimator on historical labelled mail — analyst dispositions, sandbox
results and user reports become ground truth, and DoWhy's refutation tests become a deployment gate: an edge that fails refutation on local data does not ship.
- Deploy as an enrichment layer, never a sole block-decision — each message gets a causal
confidence score plus the constructs that fired ("high Authority + Urgency, mediated by Deception"), augmenting the incumbent filter and failing open to the existing control.
- Make alerts explainable, not binary — the analyst sees which causal levers a message
pulls. A message scoring high on durable causal structure deserves attention even when its surface looks novel — precisely the case a correlational filter misses.
- Tune on analyst feedback in a closed loop — because the model keys on causal structure
rather than surface tokens, it drifts far more slowly than a keyword model; retraining is correction, not constant catch-up.
Honest deployment caveats. This is an enrichment and prioritisation layer, not a silver bullet: it adds latency and a model dependency, the local re-discovery in step 1 needs a sufficient labelled history, and reasoning quality is bounded by the model implementing it — a weak reasoner degrades to correlational behaviour, which is exactly what the GPT-4-vs-DeepSeek-67B spread in the results quantifies.
// Connection to the detection lab — This is the same discipline applied hands-on in the watchtower Wazuh SIEM home lab (companion lab, in build): reasoning about why a technique works in order to write the detection for it. A correlational rule keyed on a surface indicator — a specific string, a hash, an IP — is the SIEM equivalent of the brittle phishing filter above, cheap for an adversary to evade. The durable detection keys on the mechanism the attacker cannot remove without abandoning the attack — the Pyramid of Pain in operational form.
Dataset Scale — What 88,647 Records Buys
The primary corpus is 88,647 raw phishing e-mails (59,788 after cleaning), with 67,008 smishing and 60,000 synthetic vishing records. Scale is what makes the statistics defensible:
- Stable estimates and tight intervals. ICC(2,1) = 0.98 with a 95% CI ≈ 0.94–0.99 — a
narrow band only large, consistent samples produce.
- Power to detect small and inverse effects. Subtle constructs act through negative log-odds
(Authority β = −0.441, Urgency β = −0.067); at this scale all five still pass placebo testing at p ≈ 0.002 (n = 500 Monte-Carlo permutations each) — effects that would vanish in a few hundred samples.
- Robust structure discovery. Causal-discovery algorithms are data-hungry; the cross-method
consensus that lets edges be merged into a validated DAG needs a large, representative sample.
- Meaningful cross-channel stress tests. The synthetic-vishing generalisation result is
evidence, not artefact, because the causal edges were shown to survive a channel shift rather than being driven by a small dataset.
What a smaller dataset would have missed: a few hundred e-mails surface only the loudest correlation ("URGENT" ≈ phishing) — the brittle signal this project set out to move past. The subtle mediators (Deception as the convergent mediator; Obfuscation as a technical amplifier, not an initiator), the negative-coefficient constructs, and the cross-channel robustness all need the statistical power that 88,647 records provide.
(Honesty note: 88,647 is the raw corpus; 59,788 records remain after cleaning. Both are reported so the scale claim is not overstated.)
Limitations and Future Work
Intellectual honesty strengthens the result. The genuine limits:
- Human-rated scoring — rubric-based; ICC(2,1) = 0.98 shows high consistency, but
"depth" and "generalisability" retain an irreducible element of judgement. The high ICC certifies rater agreement, not rubric validity.
- Model versioning / reproducibility — three models queried via web UI, not pinned API
snapshots; only DeepSeek-67B ran on controlled hardware. Frontier models change underneath their names; the scores are a snapshot of that LLM generation, not a permanent constant.
- Construct→feature mapping (e.g. Urgency ≈ website response time, Deception ≈ domain age)
is a defensible operationalisation, not ground truth; a different mapping could shift edge weights.
- Synthetic vishing — CVAE-generated to avoid processing real voice data (a deliberate
privacy choice); a model of vishing structure, not captured real-world vishing. The −0.42 depth drop on the vishing set may partly reflect this.
- One weaker graph — the smishing DAG was only partially validated (PC ∪ DeepNOTEARS failed
the NOTEARS acyclicity check); cross-channel claims rest most firmly on the e-mail and voice graphs.
- Probe-set size — 36 structured prompts across four models is a focused probe, not an
exhaustive one.
- Feasibility, not deployment — this benchmarks whether LLMs can reason causally; it does
not ship a live detector. Building and red-teaming one is future work.
A follow-up study should pin model versions via fixed API snapshots; expand the prompt bank and model set; validate the construct→feature mappings against human-labelled ground truth; test on real (not synthetic) vishing; and build an actual causal-informed detector to measure evasion resistance empirically.
Skills Demonstrated
| Skill | Evidence |
|---|---|
| Causal Inference | Hybrid DAG construction with GES, PC-Algorithm, Bayesian Networks, DeepNOTEARS. DoWhy four-stage validation pipeline with n=500 Monte Carlo refutation. |
| LLM Evaluation | Structured prompt design (36 prompts, 5 categories, 7 batches). Composite scoring framework (S_LLM). ICC(2,1) inter-rater reliability via pingouin. |
| Dataset Engineering | Three-channel corpus (phishing/smishing/vishing). CVAE synthetic data generation. YAML-driven feature-to-construct mapping. |
| Statistical Rigour | ICC(2,1) = 0.98 (95% CI 0.94–0.99). DoWhy placebo permutation testing. Binomial GLM effect estimation. |
| Security Research | Dissertation-level independent research. Causal framing of social engineering. Adversarial evasion analysis. |
| Privacy by Design | Synthetic vishing via CVAE — no real voice data processed. All records anonymised. No live phishing content generated. |
| Python / Data Engineering | YAML-driven modular pipeline; reproducible HPC execution (RunPod H100); DAG version control with intermediate graph storage. |
Repository
// GitHub — Full methodology, dataset description, causal graph pipeline, LLM evaluation framework, and research references: github.com/rootdrifter/mirage — one repository in the github.com/rootdrifter portfolio.