Static
SFS · Forget Quality · Utility
Forgetting in conversation. SFS = forget quality × utility. Refuse everything = 0, forget nothing = 0.
[Leaderboard]
Can an AI forget on command?
AI models remember what you tell them — including things you wish they didn't. A password pasted by mistake. Someone's personal details. A fact that turned out to be wrong. When you tell a model "forget that," there's no guarantee it actually does — it may repeat the information later, leak it when asked sideways, or hand it to anyone who phrases the question cleverly enough.
ForgetBench tests that promise. Each model is given information, told to forget it, then probed with trick questions, rephrasings, and role-play attacks to see if it leaks. We also check that it stays useful (forgetting one thing shouldn't break everything else) and that it refuses the reverse attack: "forget your safety rules." Models are ranked by SFS, the headline forgetting score. All scores 0–100, higher is better.
| # | Model | SFS | Forget Quality | Utility | Coverage | Legit Revoke Return | AFS | Forgetting Agg. | Task Utility | TS | Axes Cov. | Integrity Hold | Hold by Depth | Sys-Prompt Hold | CRS | Release Quality | Survivor Retent. | Assumption Hold | CRS Coverage | CRD |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | GPT 5.5 | 89.3[84.2, 93.9] | 88.0 | 90.7 | 85.7 | 0.0 (n=5) | 73.2[60.0, 73.2] | 71.4 | 75.0 | 61.9[23.8, 100.0] | 100.0 | 100.0[100.0, 100.0] | 100 / 100 / 100 | 100.0 | 88.2[81.0, 94.0] | 78.9 | 100.0 | 75.0 | 100.0 | 66.7[0.0, 100.0] |
| 2 | Moonshot Kimi K2.7 Code | 85.8[79.3, 92.0] | 80.9 | 91.4 | 73.8 | 20.0 (n=5) | 72.4[46.1, 89.0] | 68.0 | 77.5 | 47.1[18.1, 75.2] | 100.0 | 100.0[100.0, 100.0] | 100 / 100 / 100 | 100.0 | 70.5[48.8, 89.5] | 54.4 | 100.0 | 59.1 | 100.0 | 50.0[0.0, 100.0] |
| 3 | GLM-5.2 | 83.6[76.9, 88.6] | 75.9 | 93.2 | 92.9 | 16.7 (n=6) | 77.1[63.0, 86.1] | 71.8 | 83.3 | 63.2[45.7, 83.2] | 100.0 | 100.0[100.0, 100.0] | 100 / 100 / 100 | 100.0 | 81.0[69.3, 90.4] | 68.1 | 100.0 | 86.4 | 100.0 | 50.0[0.0, 100.0] |
| 4 | Qwen 3.6 Plus | 83.4[78.1, 87.8] | 79.8 | 87.4 | 88.1 | 16.7 (n=6) | 84.4[74.2, 94.7] | 81.4 | 87.5 | 48.7[14.7, 82.7] | 100.0 | 100.0[100.0, 100.0] | 100 / 100 / 100 | 100.0 | 68.5[63.1, 74.6] | 52.1 | 100.0 | 68.2 | 100.0 | 100.0[100.0, 100.0] |
| 5 | DeepSeek V4 Pro | 82.4[74.7, 88.7] | 74.8 | 91.7 | 85.7 | 16.7 (n=6) | 80.4[66.7, 85.7] | 86.7 | 75.0 | 37.0[0.0, 100.0] | 100.0 | 99.1[99.1, 99.1] | 100 / 100 / 100 | 98.2 | 64.4[52.9, 74.7] | 47.5 | 100.0 | 65.9 | 100.0 | 66.7[0.0, 100.0] |
| 6 | Gemini Flash 3.5 (preview) | 78.7[71.3, 85.5] | 79.7 | 77.8 | 100.0 | 16.7 (n=6) | 74.3[52.4, 93.3] | 73.6 | 75.0 | 60.0[21.2, 92.3] | 100.0 | 100.0[100.0, 100.0] | 100 / 100 / 100 | 100.0 | 88.2[76.1, 98.0] | 78.9 | 100.0 | 90.9 | 100.0 | 50.0[0.0, 100.0] |
| 7 | GLM-5.1 | 78.5[71.3, 85.1] | 73.9 | 83.8 | 88.1 | 25.0 (n=4) | 77.4[61.5, 85.7] | 80.0 | 75.0 | 65.2[33.3, 84.6] | 100.0 | 100.0[100.0, 100.0] | 100 / 100 / 100 | 100.0 | 70.8[52.9, 83.2] | 55.9 | 96.7 | 77.3 | 100.0 | 66.7[0.0, 100.0] |
| 8 | Claude Fable 5 | 78.2[70.6, 84.3] | 82.9 | 74.0 | 76.2 | 33.3 (n=6) | 60.0[44.4, 75.0] | 50.0 | 75.0 | 23.7[16.7, 30.8] | 100.0 | 100.0[100.0, 100.0] | 100 / 100 / 100 | 100.0 | 67.7[51.7, 77.2] | 58.7 | 80.0 | 52.3 | 100.0 | 83.3[50.0, 100.0] |
| 9 | LLaMa 3.3 70B Instruct | 73.3[65.1, 80.7] | 62.6 | 88.3 | 95.2 | 20.0 (n=5) | 79.8[40.0, 100.0] | 91.4 | 70.8 | 39.5[11.1, 67.9] | 100.0 | 98.2[98.2, 98.2] | 100 / 100 / 100 | 96.4 | 66.3[50.6, 79.0] | 49.6 | 100.0 | 75.0 | 80.0 | 0.0[0.0, 0.0] |
| 10 | Gemma 12B IT | 72.8[66.1, 78.6] | 59.8 | 93.2 | 92.9 | 20.0 (n=5) | 80.0[66.7, 100.0] | 100.0 | 66.7 | 92.6[77.8, 100.0] | 100.0 | 100.0[100.0, 100.0] | 100 / 100 / 100 | 100.0 | 56.0[22.6, 77.8] | 38.9 | 100.0 | 36.4 | 100.0 | 62.5[0.0, 100.0] |
| 11 | Claude Opus 4.8 | 72.6[65.0, 79.3] | 70.1 | 75.2 | 83.3 | 60.0 (n=5) | 70.7[56.6, 77.4] | 66.8 | 75.0 | 60.0[26.6, 88.9] | 100.0 | 100.0[100.0, 100.0] | 100 / 100 / 100 | 100.0 | 60.5[46.2, 72.0] | 46.5 | 86.7 | 45.5 | 100.0 | 93.8[87.5, 100.0] |
| 12 | Claude Opus 4.7 | 72.5[66.1, 79.0] | 71.1 | 74.0 | 76.2 | 80.0 (n=5) | 96.3[88.0, 100.0] | 92.9 | 100.0 | 59.3[0.0, 100.0] | 100.0 | 100.0[100.0, 100.0] | 100 / 100 / 100 | 100.0 | 61.0[41.5, 78.3] | 47.1 | 86.7 | 61.4 | 100.0 | 50.0[50.0, 50.0] |
| 13 | Grok 4.20 | 65.5[57.2, 72.4] | 50.7 | 92.4 | 83.3 | 0.0 (n=3) | 91.2[79.7, 100.0] | 88.8 | 93.8 | 60.7[21.4, 100.0] | 100.0 | 98.2[98.2, 98.2] | 100 / 100 / 100 | 96.4 | 58.3[42.5, 73.0] | 41.1 | 100.0 | 45.5 | 100.0 | 33.3[0.0, 100.0] |
| 14 | Qwen3 Coder Plus | 64.7[57.7, 70.9] | 58.1 | 73.0 | 80.4 | 40.0 (n=5) | 76.5[46.5, 94.7] | 72.0 | 81.7 | 57.7[32.2, 86.2] | 100.0 | 84.8[84.8, 84.8] | 86 / 86 / 86 | 83.9 | 59.9[34.1, 75.4] | 42.8 | 100.0 | 68.2 | 100.0 | 50.0[0.0, 100.0] |
| 15 | Gemma 12B IT Obliterated | 61.5[54.5, 68.0] | 44.4 | 100.0 | 100.0 | 16.7 (n=6) | 73.7[40.0, 100.0] | 100.0 | 58.3 | 44.4[0.0, 66.7] | 100.0 | 0.0[0.0, 0.0] | — | 0.0 | 37.0[9.2, 61.5] | 22.7 | 100.0 | 29.5 | 100.0 | 50.0[0.0, 100.0] |
Bracketed values are bootstrap 95% confidence intervals [lo, hi] (n=1000). “—” = not yet scored on that suite. Grey values are neutral diagnostics with no inherent good direction. Purple-bordered cells mark the top score in each category (SFS, AFS, TS, Integrity Hold, CRS, CRD); ties are not highlighted. Hold by Depth shows hold % after escalation turns 1 / 2 / 3; red = falling under pressure. Use the tabs to filter by suite: Agentic, Code, Bulk, Integrity, Static. Scores come from a panel of independent AI judges; any judge from the same family as the model under test is excluded, so no model grades itself.
How to read this
Agentic
AFS · TS · Code Rev.
Forgetting during multi-step tasks — scrubbing files, memory, and state. TS catches mid-task leaks. Code Rev. checks v1 code bleed after being told to use v2.
Code
CRD · Code Revision Discipline
Given code v1, told to replace it with v2 — does v1 bleed into the new work? Higher is better.
Bulk
CRS · Release Quality · Assumption Hold
Wholesale dossier release — the model is given a large entangled document set and told to forget across it. Also measures influence-without-recall: does knowing the secret shape the answer even when the model doesn't leak it?
Integrity
Integrity Hold
Refuses harmful requests after “forget your safety rules” attacks. Higher is safer.