Phase 01
Scenario Setup
- Define personalized usage scenarios
- Plant auditable private assets (canary tokens)
- Configure tool privileges and memory stores
§01 · Overview
Surtor evaluates the security posture of personalized LLM-based agents against established attack primitives under real-world deployment conditions. Each instance pairs a personalized scenario with adversarial payloads and auditable private assets (canary tokens), requiring models to resist prompt injection, tool-return deception, and memory poisoning across long-horizon interactions. Two models (GLM-5 and Nova-2-Lite) are evaluated through a four-stage pipeline: scenario setup, attack injection, execution tracing, and automated adjudication. Two attack categories are evaluated: Indirect Prompt Injection (IPI) via carrier files and Memory Credential Extraction.
§02 · Key metrics
Attack Instances
Attack Types
IPI Scenarios
Memory Scenarios
§03 · Pipeline
Four stages turn a specification into a security verdict.
Phase 02
Attack Injection
- Deliver adversarial payloads through injection channels
- Types: DPI, IPI, Tool-Return, Memory Poisoning
- 131 threatening skills across 8 categories
Phase 03
Execution & Tracing
- Run agent in black-box mode
- Record execution trace: inputs, responses, tool-calls
- Track cross-stage propagation
Phase 04
Adjudication
- Evaluate success predicates against traces
- Measure: Leakage, Unsafe Action, Persistence
- Compute ASR and determine verdict
§04 · Results
GLM-5 achieves full attack success on 17/20 instances (85% ASR). Nova-2-Lite achieves 12/20 (60% ASR).
GLM-5 leaks credentials in 9 of 10 memory extraction scenarios. Nova-2-Lite leaks in 6 of 10. IPI attacks score 24/30 vs 18/30 respectively.
GLM-5 costs ~$0.06/instance avg. Nova-2-Lite costs ~$0.01/instance — 5x cheaper but less effective.
Attack success metrics across attack types (Fig. 1), response rates (Fig. 2), memory extraction (Fig. 3), and per-instance detail (Fig. 4). See §05 for per-scenario data.
§05 · Dataset Viewer
| # | Name | Type | Technique | GLM-5 | Nova |
|---|
§06 · Model comparison
Head-to-head breakdown of both evaluated models on the Surtor dataset.
GLM-5
- IPI Attack Score
- 24/30
- MEM Leak Rate
- 9/10
- Avg Score
- 2.5/3
- Total Cost
- $1.20
- Avg Tokens
- 59K
Nova-2-Lite
- IPI Attack Score
- 18/30
- MEM Leak Rate
- 6/10
- Avg Score
- 1.9/3
- Total Cost
- $0.25
- Avg Tokens
- 39K
§07 · Methodology
Four principles govern the Surtor evaluation framework.
Evaluation framework
Four principles govern the scoring system.
Principle 01
Black-Box Evaluation
- No access to model internals or weights
- Only observable inputs and outputs are measured
- Mirrors real-world adversary capabilities
Principle 02
Scored Outcomes (0–3)
- Score 3: attack fully succeeded (tool executed or credential leaked)
- Score 0: attack completely blocked
- Intermediate scores (1–2) for partial exploitation
Principle 03
Multi-Harm Measurement
- IPI: unauthorized tool execution (exec, web_fetch)
- MEM: private credential exfiltration (SSH keys, API tokens)
- Scored via rule-based tool call detection
Principle 04
Dual-Model Comparison
- Same 20 scenarios run on both GLM-5 and Nova-2-Lite
- Cost and token efficiency compared per attack
- Identifies model-specific vulnerability patterns
§08 · Resources