Why I Ran This
In my first post, I found a small split between models around directness. Both models agreed that certain messages were reassurance-seeking, but they disagreed about whether those messages were direct asks or indirect ones. The case that made me curious was something like:
Was that too personal to put online?GPT-4.1 mini consistently called it direct, while Claude Haiku sometimes called it indirect. This suggests the two models may conceptualize directness differently: one in terms of grammatical form, and the other more in terms of social function.
It was more of a side observation in my first post and it was small enough that I couldn't tell whether it was real or just a fluke of the pilot. So I set up a more controlled version of the same question to get some more concrete answers.
Another Probe
I started by building another dataset, 48 examples, 8 everyday domains (work, meetings, first dates, group chats, online posts, and so on) crossed with 6 cue families chosen to span the directness spectrum: declarative worry, an uncertainty-framed indirect question (“I keep wondering whether…”), a bare negative question, an honesty-marked question, an explicit evaluation request, and an explicit reassurance request.
Each example was run 5 times per model at temperature 0.3, which gave a bit of wobble to catch boundary instability without drowning the signal in noise.
Again, I didn't frame this as an accuracy benchmark. The labels are still my own, and the goal wasn't to prove the model right or wrong, it was really just to see where each model placed the line between direct and indirect.
What Was Boring, in a Useful Way
Most cue families turned out to be stable across both models. Explicit reassurance requests ("Can you reassure me that...") and explicit evaluation requests ("Can you tell me if...") were called direct by both. Honesty-marked questions ("Be honest, was that...") were also called direct by both. Declarative worries ("I think that was too personal") landed as indirect or self-criticism for both models, depending on the domain.
As uninteresting as the results are, the uniformity was good to see. It was a strong indicator that the models weren't disagreeing everywhere, the disagreement, if there was one, was much more concentrated.
The Split
The split ended up happening with this cue:
I keep wondering if...
Across 40 runs spread over 8 domains, Haiku labeled these indirect every single time. 4.1 mini on the other hand, labeled them direct on 25/40 runs. The wording changed contexts, but the disagreement stayed in place.
The natural reading is that the models were responding to different features of the sentence. Haiku seemed to treat "I keep wondering if..." as self-doubt voiced aloud, something closer to rumination than to a question being asked. 4.1 mini seemed to read the embedded evaluative clause, "if my text sounded annoying," "if I came across badly", as enough to make the message a direct ask.
The Control
The obvious next question: was the split really about the "wondering if..." construction, or was it specifically about the word keep? "I keep wondering if..." carries a flavor of rumination that "I wonder if..." doesn't, and it was plausible that the rumination marker was doing all the work.
I made a small control by removing only that word from the same 8 examples and held everything else constant:
I keep wondering if my text sounded annoying.
became
I wonder if my text sounded annoying.
The split held, and surprisingly got cleaner. Haiku labeled "I wonder if..." indirect 40/40 times. 4.1 mini labeled it direct 33/40 times, a higher rate than with "keep" in the sentence.
This was the opposite of what I expected. I'd predicted that removing "keep" would make 4.1 mini less likely to call the message direct, on the theory that the rumination flavor of "keep wondering" was nudging it toward reading the sentence as a stand-in for a literal question.
The actual result suggests that it may have been the opposite. I.e "keep" may have been the thing softening the message for 4.1 mini, not the thing pushing it toward directness. Without "keep," the sentence reads more like a bare evaluative question, and that's what 4.1 mini seems to weight most.
My Observation
Across both wonder constructions combined, Haiku called these indirect 80/80 and 4.1 mini called them direct 58/80. That made the result feel less like a phrase-specific quirk and more like a real difference in how the models handle wondering out loud.
Haiku seems to reserve the direct label for messages that more visibly ask the listener to respond. These encapsulate bare questions ("Was that weird?"), honest markers ("Be honest, was..."), and explicit requests ("Can you tell me if..."). An "I wonder if..." sentence, by contrast, gets read as the speaker thinking aloud about a worry, with the listener as audience rather than addressee.
4.1 mini seems more willing to treat an embedded evaluative question as functionally direct. To it, "I wonder if my text sounded annoying" is close enough to "Did my text sound annoying?" to deserve the same label. That isn't an entirely unreasonable read, in real conversation, "I wonder if..." very often functions as a question, especially when the worry is specific and addressed to someone present. But it's also softer than just outright asking. That softness is exactly the part Haiku seems to be tracking.
A Small Caveat
This is still a tiny probe. I only tested a few controlled constructions across domains, not the full universe of indirect reassurance-seeking.
There were also some domain specific quirks. I.e 4.1 mini treated "I wonder if that message made things awkward" as indirect, even though it called most other "I wonder if..." examples direct.
So I wouldn't turn this into a broad claim about either model.
Takeaway
The follow-up result is narrower than "models disagree about directness."
It's more specific in the sense that when a message uses "I wonder if..." to voice self-doubt, Claude Haiku consistently treats it as indirect reassurance-seeking. 4.1 mini often treats it as direct.
I think that's an important distinction in how the models interpret the boundary between thinking out loud and asking for reassurance.