Ranking under each sentiment pathway

Same cached inputs for every pathway (MR_OFFLINE=1): RT, BOM, CinemaScore and the same 1,381 comments; only the per-comment sentiment/stance step differs. Rank = position in the shortlist (bucket first, then score). Rank change = live rank minus pathway rank (positive = moves up). Labels for accuracy are mine (Grok Bot), single annotator. Live index.html still uses siebert+opinion.

Cell = #rank · score · change vs live (↑ moves up) · bucket · R = Reddit community score. Colour = bucket (Watch Depends Skip). Click a cell for that movie's comments with each pathway's verdict and rationale. The live shortlist on the home page is unchanged (siebert+opinion).

Stance pathways ask: did the commenter like or recommend the film? (positive / negative / mixed / no verdict, JSON, temperature 0). Mixed and no-verdict count as neutral. OpenRouter spend for these evaluations and runs: $13.63. Not run: Local Qwen2.5-3B stance

movie (ordered by live rank)VADER (old v1)siebert+opinion (LIVE)Local NLI stance (deberta-v3-large)GPT-6.1 SolClaude Opus 5.5Claude Sonnet 5.5Gemini 4 ArgonGrok 4.7GPT-6 Luna (cheap)Claude Haiku 5.5 (cheap)Gemini 3.5 Flash-Lite (cheap)
Wildwood#4 80 ↓3
Watch · R 68
#1 81
Watch · R 70
#1 84 =
Watch · R 76
#3 92 ↓2
Watch · R 94
#2 92 ↓1
Watch · R 96
#2 92 ↓1
Watch · R 95
#2 92 ↓1
Watch · R 95
#1 92 =
Watch · R 95
#1 92 =
Watch · R 96
#3 91 ↓2
Watch · R 92
#3 92 ↓2
Watch · R 94
Forgotten Island#1 83 ↑1
Watch · R 82
#2 79
Watch · R 75
#7 58 ↓5
Depends · R 27
#6 88 ↓4
Watch · R 93
#6 89 ↓4
Watch · R 95
#6 87 ↓4
Watch · R 92
#6 89 ↓4
Watch · R 95
#6 87 ↓4
Watch · R 91
#6 88 ↓4
Watch · R 94
#6 87 ↓4
Watch · R 92
#6 84 ↓4
Watch · R 86
Wicker#2 81 ↑1
Watch · R 75
#3 79
Watch · R 71
#9 52 ↓6
Skip · R 14
#2 92 ↑1
Watch · R 98
#4 92 ↓1
Watch · R 98
#3 92 =
Watch · R 98
#4 92 ↓1
Watch · R 98
#3 92 =
Watch · R 98
#2 92 ↑1
Watch · R 98
#2 92 ↑1
Watch · R 98
#4 89 ↓1
Watch · R 93
Sense and Sensibility#5 79 ↓1
Watch · R 88
#4 75
Watch · R 79
#6 59 ↓2
Depends · R 44
#8 74 ↓4
Depends · R 76
#9 74 ↓5
Depends · R 76
#9 77 ↓5
Watch · R 82
#8 76 ↓4
Watch · R 80
#8 75 ↓4
Watch · R 78
#8 75 ↓4
Watch · R 78
#8 75 ↓4
Watch · R 79
#8 74 ↓4
Depends · R 76
Clayface#6 76 ↓1
Watch · R 72
#5 73
Depends · R 63
#3 67 ↑2
Depends · R 50
#5 88 =
Watch · R 97
#5 89 =
Watch · R 99
#5 89 =
Watch · R 99
#5 89 =
Watch · R 99
#5 88 =
Watch · R 98
#5 89 =
Watch · R 99
#5 88 =
Watch · R 98
#5 87 =
Watch · R 96
Resident Evil#8 71 ↓2
Depends · R 55
#6 72
Depends · R 58
#2 73 ↑4
Depends · R 59
#4 91 ↑2
Watch · R 98
#3 92 ↑3
Watch · R 100
#4 89 ↑2
Watch · R 94
#3 92 ↑3
Watch · R 100
#4 90 ↑2
Watch · R 96
#3 91 ↑3
Watch · R 97
#1 92 ↑5
Watch · R 100
#2 92 ↑4
Watch · R 100
You Can See Everything#3 81 ↑4
Watch · R 66
#7 71
Depends · R 47
#8 56 ↓1
Depends · R 20
#1 95 ↑6
Watch · R 93
#1 95 ↑6
Watch · R 93
#1 93 ↑6
Watch · R 89
#1 95 ↑6
Watch · R 93
#2 92 ↑5
Watch · R 88
#4 89 ↑3
Watch · R 82
#4 89 ↑3
Watch · R 81
#1 93 ↑6
Watch · R 89
Heart of the Beast#7 74 ↑1
Depends · R 66
#8 67
Depends · R 50
#5 62 ↑3
Depends · R 40
#7 81 ↑1
Watch · R 82
#7 83 ↑1
Watch · R 85
#7 83 ↑1
Watch · R 85
#7 85 ↑1
Watch · R 90
#7 83 ↑1
Watch · R 85
#7 83 ↑1
Watch · R 84
#7 81 ↑1
Watch · R 81
#7 80 ↑1
Watch · R 80
A Talent for Murder#9 66 =
Depends · R —
#9 66
Depends · R —
#4 66 ↑5
Depends · R —
#10 66 ↓1
Depends · R —
#10 66 ↓1
Depends · R —
#10 66 ↓1
Depends · R —
#10 66 ↓1
Depends · R —
#10 66 ↓1
Depends · R —
#10 66 ↓1
Depends · R —
#10 66 ↓1
Depends · R —
#10 66 ↓1
Depends · R —
Whalefall#10 63 =
Depends · R 58
#10 66
Depends · R 65
#10 51 =
Skip · R 33
#9 72 ↑1
Depends · R 80
#8 75 ↑2
Depends · R 85
#8 77 ↑2
Watch · R 91
#9 75 ↑1
Watch · R 85
#9 74 ↑1
Depends · R 84
#9 70 ↑1
Depends · R 75
#9 75 ↑1
Depends · R 84
#9 74 ↑1
Depends · R 83
Street Fighter#13 50 ↓2
Skip · R 49
#11 40
Skip · R 30
#13 28 ↓2
Skip · R 4
#13 36 ↓2
Skip · R 20
#13 34 ↓2
Skip · R 16
#13 35 ↓2
Skip · R 18
#13 35 ↓2
Skip · R 18
#13 34 ↓2
Skip · R 18
#13 34 ↓2
Skip · R 16
#13 36 ↓2
Skip · R 22
#12 34 ↓1
Skip · R 17
Other Mommy#14 42 ↓2
Skip · R 51
#12 35
Skip · R 36
#12 31 =
Skip · R 25
#11 44 ↑1
Skip · R 56
#11 47 ↑1
Skip · R 61
#11 48 ↑1
Skip · R 64
#11 47 ↑1
Skip · R 61
#11 42 ↑1
Skip · R 51
#11 46 ↑1
Skip · R 60
#11 47 ↑1
Skip · R 62
#11 43 ↑1
Skip · R 52
Verity#12 52 ↑1
Skip · R 52
#13 35
Skip · R 16
#11 31 ↑2
Skip · R 6
#12 36 ↑1
Skip · R 18
#12 36 ↑1
Skip · R 18
#12 36 ↑1
Skip · R 18
#12 37 ↑1
Skip · R 19
#12 36 ↑1
Skip · R 18
#12 35 ↑1
Skip · R 15
#12 38 ↑1
Skip · R 21
#13 33 =
Skip · R 11
Klara and the Sun#11 52 ↑3
Skip · R 67
#14 28
Skip · R 27
#14 12 =
Skip · R 0
#14 25 =
Skip · R 22
#14 29 =
Skip · R 29
#14 27 =
Skip · R 26
#14 29 =
Skip · R 28
#14 28 =
Skip · R 27
#14 24 =
Skip · R 20
#14 28 =
Skip · R 28
#14 22 =
Skip · R 18
Per-pathway summary (labels are mine, single annotator)
Held-out acc / macro-F1 (n=112)38% / 0.3459% / 0.5853% / 0.4687% / 0.8684% / 0.8183% / 0.8180% / 0.7981% / 0.7980% / 0.8078% / 0.7780% / 0.79
Challenge acc (n=71)55%76%66%93%90%89%89%89%93%90%86%
Held-out vs live (McNemar p)+9/−32, p=0.0004live+18/−25, p=0.3604+37/−6, p=0.0+36/−8, p=0.0+37/−10, p=0.0001+33/−9, p=0.0003+32/−7, p=0.0001+30/−6, p=0.0001+31/−10, p=0.0015+29/−6, p=0.0001
Cost per full cold run$0 (local)$0 (local)$0 (local)$1.37$4.63$2.15$1.60$3.19$0.09$0.11$0.24
Latency per comment——0.9 s2.2 s3.0 s1.2 s3.4 s3.2 s1.8 s1.1 s0.9 s
Cold run, scoring time (serial)1 s6 min19 min47 min65 min26 min72 min68 min38 min23 min19 min
Stance prompt (shared by every LLM pathway, stance-v1)
You classify a Reddit comment from a movie's discussion thread by the commenter's VERDICT ON THE FILM ITSELF: did they like it / would they recommend it?
Judge the verdict, not the tone or vocabulary.
- Praise of dark, scary, violent, gross, sad or disturbing content is POSITIVE ("the villain is terrifying", "that scene was brutal, incredible", "I cried the whole time, beautiful").
- Hating a character, calling a character evil/a psycho, or describing a horrifying scene is NOT a negative verdict unless they criticise the film.
- Polite or soft disappointment is NEGATIVE ("I really wanted to like it", "it fell flat for me", "beautiful but I didn't care").
- Clear praise and clear criticism with no overall lean is MIXED. If they lean one way overall ("I liked it despite flaws"), pick that side.
- Plot discussion, theories, quotes, jokes, trivia, comments about the book/director/marketing without a judgement of this film, or not having seen it: NO_VERDICT.
Reply with JSON only: {"verdict": "positive" | "negative" | "mixed" | "no_verdict", "rationale": "<max 20 words>"}
NLI hypotheses
Challenge set by type (correct / n)
pathwaydark words, positive verdictpolite words, negative verdictdark description, no verdictmixed
VADER (old v1)28/3910/191/110/2
siebert+opinion (LIVE)35/3917/192/110/2
Local NLI stance (deberta-v3-large)26/3915/195/111/2
GPT-6.1 Sol36/3918/1911/111/2
Claude Opus 5.536/3916/1910/112/2
Claude Sonnet 5.535/3915/1911/112/2
Gemini 4 Argon36/3915/1910/112/2
Grok 4.736/3916/1910/111/2
GPT-6 Luna (cheap)36/3918/1910/112/2
Claude Haiku 5.5 (cheap)36/3916/1910/112/2
Gemini 3.5 Flash-Lite (cheap)35/3914/1910/112/2

Data: compare_pathways.json · stance_eval.json (every labelled comment, every pathway) · challenge labels