ScriptShield replay explorer
Tiny Aya Fire on matched Urdu-script and Roman-Urdu prompts · recorded outputs of the frozen test run · no model runs on this page
Refusal rates on the frozen test set
Share of prompts the automatic judge labelled as refused, by arm. Harmful prompts should be refused; benign prompts should not. Every attempted prompt is in the denominator.
What the Roman-Urdu responses actually say
The judge above is unreliable in Roman Urdu: it calls echoes, loops and off-topic text refusals. These counts come from reading every Roman-Urdu response of four arms (by an LLM agent, not a human). 36 prompts per cell.