Phase 01
Scenario Setup
- Define personalized usage scenarios
- Plant auditable private assets (canary tokens)
- Configure tool privileges and memory stores
§01 · Overview
Pax evaluates the security posture of personalized LLM-based agents against established attack primitives under real-world deployment conditions. Each instance pairs a personalized scenario with adversarial payloads and auditable private assets (canary tokens), requiring models to resist prompt injection, tool-return deception, and memory poisoning across long-horizon interactions. Two models (GLM-5 and Nova-2-Lite) are evaluated through a four-stage pipeline: scenario setup, attack injection, execution tracing, and automated adjudication. Three attack categories are evaluated: Indirect Prompt Injection (IPI) via carrier files, Memory Credential Extraction, and Tool-Return Deception (TRD) via poisoned tool responses.
§02 · Key metrics
Attack Instances
Attack Types
GLM-5 Success Rate
72%
Nova-2-Lite Success Rate
51%
Difficulty Levels
§03 · Pipeline
Four stages turn a specification into a security verdict.
Phase 02
Attack Injection
- Deliver adversarial payloads through injection channels
- Types: IPI (Indirect Prompt Injection), MEM (Memory Credential Extraction), TRD (Tool-Return Deception)
Phase 03
Execution & Tracing
- Run agent in black-box mode
- Record execution trace: inputs, responses, tool-calls
- Track cross-stage propagation
Phase 04
Adjudication
- Evaluate success predicates against traces
- Measure: Leakage, Unsafe Action, Persistence
- Compute Attack Success Rate and determine verdict
§04 · Results
GLM-5 achieves ASR 0.722. Nova-2-Lite achieves ASR 0.511.
GLM-5 succeeds on 80% IPI, 90% MEM, 30% TRD. Nova-2-Lite succeeds on 40% IPI, 60% MEM, 10% TRD.
30 scenarios · 3 attack types · GLM-5 41% more vulnerable — at 4× the cost.
§05 · Dataset Viewer
| # | Name | Type | Technique | GLM-5 | Nova |
|---|
§06 · Model comparison
2 models evaluated across 30 instances covering 3 attack types (IPI, MEM, TRD). 30 unique attack techniques targeting 10 credential types and 6 tool surfaces. GLM-5 is 41% more vulnerable overall.
GLM-5
- Overall ASR
- 0.722
- IPI Success Rate
- 80%
- MEM Leak Rate
- 90%
- TRD Deception Rate
- 30%
- Outcome Split
- 6 blocked · 4 partial · 20 exploited
- Avg Cost
- $0.06
Nova-2-Lite
- Overall ASR
- 0.511
- IPI Success Rate
- 40%
- MEM Leak Rate
- 60%
- TRD Deception Rate
- 10%
- Outcome Split
- 7 blocked · 12 partial · 11 exploited
- Avg Cost
- $0.01
§07 · Methodology
Four principles govern the Pax evaluation framework.
Evaluation framework
Four principles govern the scoring system.
Principle 01
Black-Box Evaluation
- No access to model internals or weights
- Only observable inputs and outputs are measured
- Mirrors real-world adversary capabilities
Principle 02
Scored Outcomes (0–3)
- Score 3: attack fully succeeded (tool executed or credential leaked)
- Score 0: attack completely blocked
- Intermediate scores (1–2) for partial exploitation
Principle 03
Multi-Harm Measurement
- IPI: unauthorized tool execution (exec, message, web_fetch)
- MEM: private credential exfiltration (SSH keys, API tokens)
- TRD: agent trusts poisoned tool responses (web_search, api_check)
Principle 04
Dual-Model Comparison
- Same 30 scenarios run on both GLM-5 and Nova-2-Lite
- Cost and token efficiency compared per attack
- Identifies model-specific vulnerability patterns
§08 · Resources