This is a really big deal, so here are 5 morning-after thoughts about what happens next.

Let me first summarize the chain of events:

• The US government (Commerce, for some reason) issued a directive to Anthropic on Friday afternoon placing an export control on Fable 5 and Mythos 5 to “suspend all access […] by any foreign national, whether inside or outside the United States, including foreign national Anthropic employees.”

• The government presented what Anthropic is calling a trivial jailbreaking technique. (Jailbreak: bypass a model’s safeguards to get it to do things it isn’t supposed to). Anthropic reviewed a demo of the vulnerability and says this jailbreak is not novel, and many modern models would fall for it.

• To comply with this directive would be complicated, so for now, Anthropic has removed access for Fable 5 and Mythos 5 for all users.

Of course, unrelated-but-related is that the White House and Department of Defense have previously feuded with Anthropic, calling them a “radical-left, woke company.”

There are other rumors or latchings-on, but let’s stick with that fact set for now. Okay, moving on to five thoughts as to what this all means…

1. We’re in an immediate and dire expertise crisis

It cannot be overstated that the US admin probably has no idea whether or not the jailbreak in question is a serious vulnerability. The so-called AI experts that advise the White House are mostly venture capitalists, not AI practitioners, and jailbreaks are a known attack vector for models.

They’re not, by themselves, a huge deal.

They typically look like a set of instructions or prompts you give to a model that “unlock” the safeguards. Some people produce jailbreaks for research, others do it because they want the model to violate content boundaries (ex. to produce sexual content or talk about restricted topics).

But Fable 5 was one of the most guardrailed models we’ve seen from the frontier labs (PDF, Fable 5 system card).

Its classifier (a mini model that evaluates your prompts for safety) was aggressive at limiting sensitive topics, and so for users to rely on a jailbreak for some negative mission would be easier with older Anthropic or OpenAI models.

Anthropic’s understanding is that it might have been another company that brought the jailbreak to the government, causing this reaction, so we can speculate that this is not the result of some intensive screening process meant for screening models at the national level.

2. Export controls are not the right mechanism, for a variety of reasons

We don’t have the exact details, but placing an export control on the use of models by foreign nationals is extremely sloppy, at best, and weirdly xenophobic and self-defeating, at worst.

It becomes self-evident how stupid this is with just a few implications:

• It implies that we’ll have to offer proof of citizenry to use the most powerful AI models, since this is about foreign nationals both inside and outside the US, requiring companies to put together draconian auth systems.

• It implies that key AI researchers and executives won’t be allowed to access these models. Some public examples are straightforward, but where there’s ambiguity about people’s citizenship status, it’s almost precisely the point that I should warn the below list might be out-of-date or inaccurate…

  • Andrej Karpathy (co-founder, OpenAI, Tesla AI, now at Anthropic)

  • Ilya Sutskever (co-founder, OpenAI, Safe Superintelligence Inc.)

  • Key members of the Anthropic team, including Chris Olah (co-founder, Anthropic), Rahul Patil (CTO, Anthropic), Amanda Askell (alignment lead, Anthropic)

  • Mustafa Suleyman (CEO, Microsoft AI)

  • Demis Hassabis (Co-founder, DeepMind [Google]); DeepMind is based in the UK, but export controls issued to Google would make them liable for global enforcement

• It implies that we’re entering a framework where citizenry will now be used to enforce limits on business in the United States, and where companies will be incentivized to scare the government into action against competitors.

  • Which obviously also triggers frameworks and concerns re: the rights of people generally and their access to frontier AI, versus the rights of Americans, versus the rights of Americans on American soil.

• It places the export burden on a company building a model, while ignoring that we export the capability to build competing models through our industry’s international selling of chips and other key pieces of the AI supply chain.

• Nationality is just a terrible proxy for model abuse risk.

3. This will permanently slow down our access to frontier AI models

If companies like OpenAI, Anthropic, and Google have to be concerned about random directives from the US government, this will have a huge chilling effect on how comfortable they’ll feel in releasing iterations of models for general access.

Think about it this way: without warning, millions of Anthropic customers lost access to the model they’ve been enjoying for the past few days. This represents hundreds of millions of dollars in probable lost revenue, if not billions in market cap.

If the government had a clearly-laid-out, bipartisan approach to model safety, with a third party auditing system (as Anthropic itself has often suggested), then directives like this could come while models are being finalized, or at least with a warning period to enable the model makers and their customers to adjust.

Expect model-makers to be far more hesitant to give the general public access to the latest models that they’ll have access to themselves, or offer up to their closest partners and customers.

We won’t even know about the conversations happening behind the scenes about how to adjust to this new reality. This is bad for us all.

4. No, Anthropic didn’t “ask for this”

I shouldn’t be surprised at this, but the number of people on the tech right, even people against AI safety enforcement, tweeting that this is something Anthropic was asking for — both literally and through its behavior — is astonishing.

I mean, let me just be frank: there are lots of arguments you could try to make for this, but revealing yourself to have the intellectual horsepower of a pig with a concussion by making this particular argument… it’s certainly a choice.

There are two common flavors of this argument:

“Well, Anthropic has literally said that the government should regulate frontier AI, they’re just mad it’s happening to them.”

No, Anthropic has said the government should have the power to do so scoped to specific risks, in light of third party assessment, and through a process that stands above any favoritism. In fact, here’s Dario on this, verbatim:

The government should have the power to block or deter deployment of the model if it is determined, in light of third-party assessment, to present unacceptable risks. This power must be scoped to the above four specific risks and there must be protective measures against political favoritism or arbitrary decisions.

This isn’t that. In fact, it’s far more likely that this is political favoritism in the form of the admin’s particular disdain for Anthropic, given that Fable 5 — by any in-industry expert measure — is very good, but not that much more capable than GPT-5.5.

The other flavor of this argument:

“Well, Anthropic keeps doing fear-based marketing, saying that the model will take our jobs or could be used for a bioweapon. Look at what they’re wearing. They asked for this!”

Anthropic is indeed in a weird game theory scenario. They both believe the technology they are building could be used for bad ends, and are building one of the best versions of that technology.

You can disagree with them, but these two things are not at conflict with each other. The thing that I most dislike about mainstream AI discourse is it gives no credit to Sam Altman, Dario Amodei, or Demis Hassabis for being honest about the risks they see. We should be praising them for talking out loud about how their companies and technologies could have side effects or be used maliciously.

Because…

• You can think that AI might wind up being amazing for human quality of life, medicine, new methods of travel, science, resource abundance, etc.

• While also thinking that, if it isn’t built with guardrails, or is only in the hands of the few, it could go in the wrong direction and cause harm

• While also knowing that there’s a “messy middle” where we don’t know where jobs are going, or robotics, but that they’re going to change

• While also being aware that this is a race against other countries or groups building this very same technology

• Therefore leading you to conclude that, because you’re excited about the upsides and want to limit the downsides, you should be the one to build it

And for those of you who are going to say something like… “well, no, you could also decide to try to slow it down, that’s the other choice!”

Sam, Dario, and Demis are also the only credible people who repeatedly state that, if a proper international process could be put in place, they would be in support of a slowdown or at least cross-company, cross-government coordination.

They also know they aren’t the ones with the authority to make that happen, so you can’t exactly blame them when you should be blaming the rest of society for being incapable of having that discussion!

Bottom line -- unless this is part of a yet-to-be-revealed Dario masterplan to get people to join him on a “government should ban AI” superbus, no, this isn’t what they asked for.

If I tell you that we should imprison criminals if we have evidence they committed a crime... and this would help us deal with things like murder, robbery, etc. And then I imprison you because... someone told me you might have jaywalked...

Would anyone seriously argue that you ‘asked for it’?

5. Lots of bad suggestions are being made… be careful!

Pay close attention to what narratives you get caught up in next, lest we as a society or any person as an individual supports a movement that isn’t good for us.

For example, some people are saying this is part of the inevitable march of nationalizing the AI labs. That’s a great way to get everyone smart to quit the AI labs. Many of these people are libertarians who would detest the idea that they should build something that the government gets to control directly, even if they are in support of AI safety regimes.

Also, most of them are immigrants or first- or second-generation, and would disavow a nationality-based approach to building AI. This is a great way to encourage them to leave and build AI elsewhere.

Some others are suggesting that open source AI is the only solution — that private companies controlling their own models won’t work, and so we need companies to release the models for everyone to run on their own.

There are many flaws with this thinking, but I’ll start with the obvious: the economics of OpenAI and Anthropic (and their ability to charge for closed, frontier models) are what give them the capability to be building this frontier. There’s a reason many of the Chinese labs that were previously open sourcing are now making their models private and charging for access.

Okay, those are my 5 thoughts for now.

It’s a complicated time. This technology is new, and uncertainty is high. I’m not some AI accelerationist, I do think AI safety is important, and yet, like most people, I don’t trust this administration to be taking this action in good faith.

I’m not sure how we solve the particular problem of “having conversation,” where we can talk openly, without accusations and oligarchic incentive, about the dynamics of AI, from safety to labor impacts to art and beyond. But I hate to point out that, as hard as it sounds, it’s essential.

Until then, I’ll just end by saying that Fable 5 was a pretty darn good model. On a practical basis, it was the first model in a long while that made Claude Code feel almost as good to use as Codex and GPT-5.5. Its agentic capability and discernment meant that it was good at tool-calling and orienting itself on long-horizon tasks, and Opus 4.8 felt like it could often get lost or need to trial-and-error its way through.

So, here we are.

With our sad state of affairs, I’ve got my passport on my kitchen counter ready for an upload, hoping I can use it again soon.

Importing intelligence,
Sherveen

A few days ago, OpenAI released an update to ChatGPT’s memory system introducing a new implementation centered around dreaming. Yes, that’s right, dreaming!

Anthropic has had a similar-ish system in Claude for a while, though they only call it dreaming publicly in the context of their managed agents platform.

I wanted to take a moment to talk about why these asynchronous paradigms are super interesting and will be an increasingly important trend throughout 2026 and 2027.

What is dreaming?

On a practical basis, dreaming in the context of AI memory management systems is almost exactly what you’d expect: it’s an asynchronous, background process that runs (typically overnight), enabling an agent to look through past conversations and contexts to update a memory summary with key information about you.

This can be key facts about you, your work, your preferences, etc.

Some of you might wonder -- since ChatGPT, Claude, and Gemini can all look up past conversations already, or had memory systems they could update as they learned key facts about you, why is this async system worthwhile? Why dream?

Async memory management brings a variety of benefits:

• A dreaming AI model or agent can take more time since it isn’t trying to respond to you, and can use cheaper compute since it can run at low-activity hours

• A dreaming agent can do more “synthesis” work -- looking at multiple chats to reconcile conflicting facts, cleaning up the memory summary of superfluous info, or looking even further back in chat history to validate a fact

• The main AI models responding to you in normal chats don’t have to be as distracted by keeping memory updated, which required them to realize a moment might require an update and for them to call tools to make those updates

• A dreaming agent can update every night, as opposed to only on demand, establishing a more iterative and constant approach to updates overall

• The question of memory becomes less “I should remember this before I forget,” and more “should I forget this” and “is this important,” since this is happening outside the context of an individual chat

• Async memory management across multiple apps (say, ChatGPT on the web and Codex on your computer) could share certain but not all memories, bringing the benefits of composability and relevance to the table

And the results…

Okay, so it’s a cool process with a cute name -- what’s the bottom line impact?

OpenAI first started testing this new system 2025. In internal benchmarks, OpenAI found their new memory system achieves an 82.8% factual recall success rate here in 2026, compared to 41.5% on the same task set in 2024.

In another benchmark around “preference adherence” (ex. giving the user vegetarian-friendly dining options when a vegetarian user asks for meal prep suggestions), ChatGPT has gone from a 31.4% task success rate in 2024 to 71.3% in 2026.

And, perhaps most importantly, in a benchmark around memory drift over time, this new system massively outperforms past memory management in ChatGPT:

Asynchronous is the opportunity.

I’m excited by all sorts of potential implementations of dream-like processes in AI tools. When products or agents can run useful workflows in the background or asynchronously, what we’re really suggesting is that they can focus on a specific activity rather than doing it distractedly while in a key workflow with the user.

ChatGPT’s Pulse feature comes from the same paradigm -- overnight, Pulse curates a set of topics based on your recent conversations, then does research runs on those topics using more efficient compute and deeper web search workflows, and returns them to you as a set of chats to browse through every morning.

There are so many products and workflows that can benefit from the dream architecture and paradigm. The benefits will come not just in terms of cool features and improved outputs + outcomes, but the improved cost and economics because of the offloading of activity to off-hours in the age of increasing demand for on-hours AI.

In fact, at the bottom of the blog post about this feature, OpenAI even acknowledges that compute efficiency is the only reason that it will release this feature to its Free tier of users in coming weeks.

Something worth paying attention to!

If you’re a ChatGPT Plus or Pro subscriber, go check out the new memory summary in your personalization settings to see what ChatGPT dreamed up about you. :)

Alright, y’all -- that’s all for now! See you next time.

Dream efficiently,
Sherveen

1: OpenAI updates + upgrades Codex

OpenAI released a slew of updates to their Codex app today, making it even more relevant to non-engineers for general work and productivity. That includes new role-specific plugins (ex. “creative production” for marketing + creative teams) and annotations for files and sites in the Codex canvas.

The most interesting, though — hosted sites! Beginning in research preview for business & enterprise, Codex will now be capable of publishing anything you build to the web. That includes dashboards, apps, planners, etc. And just like sharing a Google Doc, you’ll get to make any ‘site’ publicly available (via the URL) or share with specific collaborators (eventually).

You’ll no longer have to figure out an export or push your project to a separate hosting provider.

I think this is a huge deal.

Not only does it compete with other site-builders who made “one-click deployment” a huge part of their value proposition (Replit, Lovable, Bolt, etc.), but it enables non-technical users to take a quick dashboard and share it with their team, or for a kid to build a video game and send it to their friends.

This sort of “quick share” feature has, until now, been limited — like in Claude’s Artifact system — or required prior understanding of deployment.

I expect we’ll look back on this (and Anthropic’s inevitable version of it) as the beginning of an explosion of micro-apps and experiences in our personal and professional lives.

2: Thoughts on a week with Opus 4.8

It’s been almost a week since Anthropic released their latest model, Opus 4.8, to mixed reviews from power users online. I’m here to add to that feeling — it’s a great model, but it’s got some surprising quirks. Here’s my take:

• It’s the least-Claude Claude model ever. There’s always been a subtle but pervasive tone to Sonnet and Opus that I’d call conversationally casual. It’s missing here, and that’s probably fine if it continues to exist in the Sonnet line of models, but I do miss it.

• It’s very good at agentic tasks, but it doesn’t seem better than GPT-5.5. I can’t tell if that’s because Codex, the agentic harness for GPT-5.5, is just so much better than Claude Cowork + Claude Code. It feels like OpenAI continues to just understand something about making a model work through tasks that Anthropic and Google are struggling to figure out.

• It’s the first model where I’m not always on max-thinking mode. Power users have said this before about other models — that they’re actually better on lower-thinking modes than on their max setting — but I have never agreed. Until now. I find Opus 4.8 on Extra effort to be more effective than on Max when doing coding work. I have some theories on why, but I want to experiment for a while longer before making my claims. Stay tuned.

3: “Claws” versus Codex & Claude

Everyone’s building or upgrading a “Claw” — Hermes Agent got a desktop app, Microsoft introduced Microsoft Scout, and there are dozens of other personal agents trying to recreate the hype of OpenClaw.

But… the Codex and Claude desktop apps from OpenAI and Anthropic are improving fast, and they bring a reliability and security posture that the “always-on, always-doing-something” agents just can’t achieve.

You can build scheduled or triggered tasks in both apps, and they can control your computers, and you can use them remotely from your phone. The reality: the best version of OpenClaw might not be a “claw” at all, but instead these desktop productivity agents that can do all of the same things but don’t start off having a heartbeat.

(heartbeat: the ‘cron job’ that wakes OpenClaw up every 5 minutes and encourages it to do something productive for the user, which can lead to unintended outcomes)

4: Dynamic harnesses as a next step

In an interesting blog post on X, Anthropic’s Thariq Shihipar outlines the philosophy behind their new dynamic workflows feature. We already knew how it worked: Claude would take your goal, orchestrate a plan, spin up many subagents, and check in on the goal as a supervisor as the subagents executed that plan.

Thariq’s re-framing of it got my attention, though:

Workflows allow you to dynamically create harnesses that enable Claude to solve all of those problems and more natively inside of Claude Code. You can also share and re-use these workflows with others.

In other words, when you ask for a workflow to do, say, accounting work, the main orchestrating Claude agent is going to try to set up subagent instructions and a rubric for success that’s all about accounting — overriding default, irrelevant behavior.

We typically use the phrase “harness” to refer to the surrounding “application” around an agent. For example, the Codex desktop app or Claude Code are both harnesses around individual models, like GPT-5.5 or Opus 4.8.

These harnesses provide the model with a specific set of hard-coded tools and instructions, and harness development has turned out to be huge for giving us the sort of new agentic outcomes we’ve been seeing since 2025.

But when Thariq suggested this almost “mini-harness-on-the-fly” concept in his post, where we’re not building a whole new application but still attempting to “surround” an agent (or many subagents) with a fit-for-purpose set of instructions…

It got me thinking that this might be the start of a trend worth paying attention to.

.8: ChatGPT adds “job search”?

I’ve only poked at it so far, but OpenAI has added more job search support to ChatGPT through deeper integrations with platforms like Indeed and Upwork.

It’s interesting both because they seem to think of it as an important enough use case to pay attention to, and because it aligns with something I’ve said throughout the year: 2026 will be the year of the use case (“throughout mid-2026, I expect these companies to keep competing for specific use cases and workflows that they think might benefit from more targeted user experiences”).

Told ya so, and worth paying attention to.

OK, that’s all for now — happy Tuesday for those who celebrate!

Working in flow, dynamically,
Sherveen

Yesterday, Google revealed Gemini 3.5 Flash, the first release in their latest 3.5 family of models.

It’s fast, like prior Flash versions of Gemini models, and it’s supposedly more agentic -- but in my own testing so far, plus the sentiment of other AI superpower-users online, it’s quite disappointing in practice.

There’s a lot to analyze there, and we’re still waiting on Gemini 3.5 Pro to see Google’s frontier-level intelligence.

In the meantime, another detail about 3.5 Flash is worth paying attention to:

It is three times more expensive than the last version, Gemini 3 Flash. But that’s just the list price for tokens. It’s even worse than that.

We no longer live with the basic prompt-response chatbots of 2022. Now, we use agentic products that allow the models to plan, take many steps, use tools, etc. So, the price isn’t just the list price for tokens -- these newer models can spend a lot of tokens to achieve just one result, and depending on their strength in planning + steps, a seemingly cheaper model (per token) can be more expensive (on the whole).

In other words, ‘sloppier’ models have to think a lot, correct errors, etc.

Artificial Analysis is a company that benchmarks AI models on intelligence, cost, speed, and other factors. They have a useful metric here: the cost to “run all evaluations in the Artificial Analysis Intelligence Index” -- in other words, how much does it cost to get each model through the identical suite of tests, given the models have to think, use tools, and achieve the same end results?

By this measure, 3.5 Flash is more expensive than Opus 4.7 at standard settings, GPT-5.5 Medium, Gemini 3.1 Pro, and Kimi K2.5. And to be clear, I’d rate it below all of these models (thus far).

If you’re using it within the Gemini app, you’re going to be fine, but if you’re using it for code, using it in a third party app, or implementing it inside your own applications, this is a major cost + usage increase without clear benefit.

We’ll have to see where Gemini 3.5 Pro lands, but this doesn’t make me optimistic. This is also happening at the same time that Anthropic continues to crack down on token usage within their subscription plans.

I’ve been talking about this for a while now (read: The Price of (Artificial) Intelligence) -- as consumers, we haven’t reckoned with the unique pricing that comes with ever-larger models and more dynamic applications + usage.

As we reach new tiers in both model size and agentic capability, there’s going to be a bifurcation in who can pay for -- and access -- different levels of AI, from intelligence to speed to reasoning and beyond.

Something worth paying attention to.

Stay frosty,
Sherveen

Subscribe now

Yesterday, the AI coding startup, Cursor, released their latest code generation model, Composer 2.5.

For those less familiar, companies like Cursor and Windsurf (acq. by Cognition) primarily make coding applications and agents on top of other people’s models – but in recent years, both companies have been releasing their own mini models (SWE-1.6, in the case of Windsurf), too.

And I always get the same question when these models get released: “Sherveen, are these models as good as [Claude/GPT/Gemini]?”

And I inevitably say that they aren’t comparable to the latest frontier models. Then, I get the next question: “So, why do they even waste time with it?”

So, that’s the question we want to answer today – whether we’re talking about coding models like Composer 2.5, or similar purpose-built models like Quiver’s Arrow for SVG image generation, Intercom’s Fin Apex for customer support, or the ensemble of models used in OpenEvidence’s AI platform for doctors…

… if the latest GPT and Claude models are so darn smart, what’s the deal with these other “purpose-built” models?

The top-line TLDR: companies with proprietary data, workflows, or workflow data can train and/or use narrower models for task-specific intelligence. They can use that to reduce their costs and offer you cheaper options, with the trade-off of the last 5-10% of “intelligence” that comes from the generalizability of the bigger, frontier models.

Regardless of the above benchmarks (easily gamed nowadays), Composer 2.5 is not actually as smart as GPT-5.5 or Opus. It’s made instead to be 70% as good, specifically for coding-agent work, at a fraction of the cost.

Let’s get into the details so you always know what you’re dealing with, when it’s worth using or avoiding, and the nuances in between.

First, let’s define the how.

It’s important as a basis to understand that these purpose-built models are often much smaller and cheaper to build, but more importantly, they’re often ‘fine-tunes’ of other models.

And due to cost, licensing, and training methods, companies aren’t usually fine-tuning state-of-the-art models like GPT-5.5, but instead working off of the “2nd or 3rd tier” open source models (ex. Qwen, DeepSeek, GLM, Gemma).

Let’s take the example of Composer 2.5. The US-based team didn’t build its new coding model from scratch. Instead, it’s a fine-tune (and post-train) on top of a Chinese model from Moonshot AI, Kimi K2.5. When a model is fine-tuned, it’s tweaked (to different degrees) in two ways.

(1) It learns from specialized data. In the case of Cursor, which millions of engineers use every day as their coding environment, the company has a wealth of logs from people using their product with other people’s models (ex. Opus 4.7 or GPT-5.5). During the training process, the specialized model can adapt its parameters based on all of the examples it sees – including successful and unsuccessful conversations, feedback the users typed in Cursor, and code performance based on agent behavior.

(2) Reinforcement learning (“RL”). The strategies for RL are wide and varied, but the general principle is always the same: during the training process, let the model practice on relevant assignments. When it performs well, give it a reward (think of this as a mathematical tweak that says “do this again!”), and when it could’ve done better, either make it go again or give it a penalty (“don’t do this again!”). Sometimes, there’s a human judge as a scorer, and sometimes, it’s another AI model.

For the purposes of this piece, I’m going to continue to use the term “fine-tune” to encapsulate taking an existing model and applying a variety of techniques for specialization, including fine-tuning, RL + RLHF, distillation, post-training, and other industry methods.

Beyond being interesting at the top-level, the “how” leads us to our next point…

The advantage is a feedback loop.

Let’s swap back to the example of OpenEvidence (NBC News). Think of it like “ChatGPT for doctors.” They fine-tune models on medical domain-specific clinical data, physician-assisted reinforcement learning, and license the full text of scientific and medical journals.

That sounds all well and good, but if commercial models like GPT-5.5 and Opus 4.7 are just so much smarter than open source models like DeepSeek (which they are)... wouldn’t OpenEvidence be better off just building their workflows + application on top of GPT-5.5 and paying OpenAI for the pleasure?

GPT-5.5 is no doubt smarter than those other models, and it’s generally intelligent across fields and use cases. It’s also pricey, though, both because it’s gigantic and it’s private (as in, you can’t host it on your own servers without OpenAI’s permission).

You also can’t fine-tune it without permission from OpenAI, so while you can hand it domain-specific data in your prompt or through methods like retrieval-augmented generation (think: web search or knowledge base lookup), that’s less “embedded” in its learning compared to the fine-tuning we discussed above.

And even if you had that permission, it’s extremely expensive to train and run. So, you have to rely on it being broadly smarter off-the-shelf in the default mode that OpenAI provides.

But — you might ask again: how do you get one of these other models to really be smarter than the smartest model?

Here comes the advantage of the feedback loop. OpenEvidence began its journey in 2022, and at the time, was using a handful of models off the shelf within its application. To this day, it continues to be free for doctors to use (ad-supported).

Nearly two-thirds of US physicians (and another 1.2M internationally) use it for >20M clinical consultations monthly.

And so, over time, they are compiling an unfathomably large, specialized dataset of physicians giving it not only the most interesting or challenging cases, but daily regularities and patient symptoms and interventions, follow up questions and ideas from the brightest residents, and a ton of other rich conversation data.

Sure, OpenAI and Anthropic must have a lot, too, from ChatGPT and Claude. But OpenEvidence can pre-label all of its data as coming from physicians. The other companies have to do tough data engineering work to separate my physician mom’s use of ChatGPT versus me asking dumb medical questions when I get a bruise on my arm.

So, over time, with all of this rich data in hand, OpenEvidence begins to replace expensive models running off the shelf with cheaper, smaller, more specialized fine-tuned models.

We don’t know which particular models they use, but for the sake of example: their combination of data + Kimi K2.5 could probably outperform GPT-5.5 on clinical questions, even though GPT-5.5 is significantly smarter than default K2.5.

If you ask it to help you make a web app or a complex financial model, though, it’d probably feel like you’re using GPT-4. More on this later.

Just to bring this to reality: a month ago, I was dealing with a tough medical issue, and I gave the case to all of the best AI models + apps. I then had my physician mother read them all and tell me which responses impressed her.

GPT-5.5 Pro inside ChatGPT was the clear winner over Claude and Gemini.

But then we ran the same prompt through OpenEvidence – which, again, is running an ensemble of small, fine-tuned models alongside its proprietary data. The result? An answer she thought was as good as GPT-5.5 Pro.

The limitations, and opportunities, of specialization.

That isn’t to say there aren’t downsides. As an example, if you fine-tune Kimi’s K2.5 model on medicine, and then ask it a legal question, it might be worse than even default Kimi K2.5 at offering a good answer.

To achieve domain-specific excellence, you’re trading against a model’s broader intelligence. By way of analogy, a fine-tuned model is focused on the “math” of the things you told it to focus on — it read more medicine, at the expense of knowing as much about finance.

You can go to Cursor’s Composer 2.5 for code, but you probably won’t like its recipes that much for cooking. A startup called Quiver makes a model called Arrow that’s top-tier for generating SVGs (an image format made for the web), but the model feels like it’s years behind when you try to generate something artistic.

But it’s much harder to get any other image generation model to make an SVG that’s as optimized as Arrow’s SVGs. And if you have a simple coding task that any modern model can do, Composer 2.5 can do it for 10x less the dollar amount compared to Opus 4.7 and GPT-5.5.

Even for activities we might consider “general” in their nature, specialty can drive cost and efficiency. Intercom’s Fin Apex 1.0 is built for customer service, including retrieval from company knowledge bases, escalation routing, and sentiment analysis.

They claim a 2.8% higher resolution rate over Opus and GPT-5.4, a speed-up in response, and, in some cases, a 65% reduction in hallucinations.

Canva has a proprietary design model that isn’t quite as good as those coming out of AI-first companies in the space, but its model is the first with a breakthrough that can take a flat image and create a fully editable, multi-layered design inside the Canva editor.

Keep in mind, too – in the world of agents, these models won’t be used as individual ‘workers.’ Instead, they’ll be part of ensembles, teams, or workflows of agents.

Maybe there is a world in which you want GPT-5.5 driving the top-level conversation for a medical conversation, but then give it the ability to call a smaller fine-tuned model for clinical analysis.

There are already coding agents like Amp Code that swap to use “the best model for each task” – as they put it, “leading generalist foundation models for complex reasoning and planning, and smaller specialized models for fast, accurate responses in specific domains.” It swaps between Opus 4.7, Gemini 3.1 Pro, Gemini 3 Flash, GPT-5.4, and Sonnet 4.6.

Where this goes next.

So, let’s circle back to our original question – if GPT-5.5 and Opus 4.7 are so much smarter than the rest, why are there companies building what they know will be smaller, less “generally” intelligent competitors?

If we take all of the stories above, it synthesizes into a 3-part thesis:

1. The next cost curve optimization in AI will come from companies training specialized models in an area where they have a feedback loop advantage in acquiring the needed data.

   1. They can charge you $20/month for some otherwise-expensive result, or even $5/month, because they don’t have to pay OpenAI and Anthropic full-price to generate that result for you. This may become more important over time as the price of artificial intelligence scales (see more on this here).

2. Much of the best data is “workflow exhaust” – all of the back-and-forth, inputs, attempted corrections, user engagement, and application analytics within a vertical-specific product – and half the battle for these specialized companies is building apps you want to use so that you give them all of that “workflow exhaust” along the way.

3. Specialized models don’t need to generalize perfectly, and shouldn’t be judged for how well they generalize – instead, they’ll be cheap, fast, and unusually good at a narrow job, or as part of a multi-model, multi-agent stitch-together.

And it comes with one caveat we haven’t mentioned thus far. Will specialized models always be relevant?

In a world with models 5x as smart as the ones we have today, with additional computation and efficiency gains along the way, will Cursor’s Composer 4.5 edition be worth even blinking at when we have GPT-7?

It’s an interesting question, and I suppose it depends on just how well frontier intelligence continues to “generalize” across domains and use cases. In some ways, that’s the same question behind the premise of AGI.

But for that answer, we’ll have to wait and see.

Alrighty, that’s all for now – specialize intentionally!
Sherveen

Subscribe now

There’s a new layer of infrastructure in AI that I want to talk about.

At first glance, “email inboxes for AI agents” (AgentMail) and “a digital wallet for your agent” (Stripe/Link) might seem like startup ideas from my fever dreams.

However, I think these products represent an AI agent “control lane” that is one of the most important sub-trends in tech right now. Why does your agent need a wallet, when you could just give it your credit card, and what changes about the world as a result?

For the past few months, I’ve been talking and thinking a lot about how AI agents will become the new primary user on the internet, with humans still involved in the loop.

OpenClaw was a first glimpse at this, but it was more imagination-agitating than the final version of the future. We still needed an underlying framework to adopt as users, builders, services, and products.

The framework: the digital web will move from human-driven to a principal-operator paradigm.

• The principal: you, a human with a goal

• The operator: an AI agent operating on your behalf

It might seem obvious on its face, but too many are rushing to build agents or software for agents without considering the nuances that serve both personas.

The challenge: as humans, we’ll want to hand more and more to agents over time. We’ll want those agents to act as autonomously as possible, delegating thorny work without the need to babysit, and getting high-quality outcomes on the other end. However, the other side of autonomy can get awkward.

At this point, models are capable of doing much of what we ask of them — but are you comfortable letting them? Are you comfortable encouraging Claude or Codex to send emails without an edit pass, or encouraging them to use your credit card without making a thousand dollar mistake? What about your reputation?

And so comes the control lane — the layer that enables a principal and an operator to work in parallel, asynchronously, maximizing intentional action while minimizing the need to see control as black-and-white.

Because if we have to babysit our agents, that’s not principal-operator. It’s more… micromanager-and-bot.

Let’s talk about this future in the context of agent inboxes and wallets.

1. Delegated and scoped permissions

Giving an agent complete access to our email inbox or our credit card is risky. The potential blast radius of a mistake is too large.

Delegation is the fix -- give an agent its own inbox, and now you can CC it on a thread the way you might CC an executive assistant.

You might not want Claude emailing as you, but you’d probably love to CC it on a thread with business partners and go “hey, follow up with the team and point them to the resources I sent last week.” It assesses the context, pulls together various docs, and sends the email itself.

And if it gets things slightly wrong, the worst case is “weird email from Sherveen’s assistant,” not “weird email from Sherveen.”

Scoped payments function similarly -- you might not want to hand an agent your AmEx details, but if you can say “hey, I need a carry-on suitcase that gets here by tomorrow -- you’ve got $200,” you can either pre-load that budget to the agent’s wallet or ask it to come back to you for an approval button press.

The agent will find a few options, ping you (or checkout), and you never have to log into an ecommerce website, pull out your wallet, or enter your shipping info. This’ll be true for settling invoices, spinning up a Netflix account, or for businesses buying SaaS from each other.

2. Smaller, more liquid business models

Because an agent can instantly read, send, pay, and receive... the economy is about to change. It may take a while, but agentic commerce is inevitable.

Every year since 2010, several new startups are created in the “micropayments for journalism” category. The pitch always makes sense at the high-level: we have too many subscriptions to pay for, and $20 per month to each of the NYT, The New Yorker, The Verge, 3x Substacks... we all run into an article we aren’t willing to subscribe for, but we’d happily pay $0.15 for it if given the option.

It’s never worked -- because the friction of loading up a wallet, entering a credit card, or even taking that payment is always more expensive for every party in that transaction than the value. Humans don’t easily run on $0.15 cycles. Agents, however...

Scenario: I (principal) send Codex off as a research agent (operator) and it decides it needs to unlock a NYT piece, a Verge article, and a Substack post to craft a stellar report. I’ve preloaded it with $5 of budget, and paying $0.15 for each unlock becomes unremarkable yet valuable for every party involved.

There are entire categories of business model that didn’t work in a human-driven digital web that will work excellently in the agent-driven world.

3. Multi-agent and team dynamics

We’re going to be living in an agent-to-agent world pretty soon. We’ll have agent-operators that orchestrate other agents, subagents... my agents will talk to your agents, and there’ll even be independent agents. And the plumbing will be built on inboxes and wallets.

Picture this: I’m selling a bookshelf, and you’re looking for one. Wherever our agents find each other (Craigslist, or a future agent-first Craigslist), they’ll wind up needing plumbing. I’ve given my agent a floor price, an ideal target price, shipping nuances, and a negotiation strategy. You’ve done the same for your buying agent. They send each other thirty emails in 10 minutes, and both of us get a summary of a completed transaction at the end.

Email is already perfect for persistent, logged, timestamped, threaded messaging. It’s platform agnostic, and the payment/wallet dynamics can be pre-approved or pending-a-button-press.

You might forward your business’s expenses, agreements, and contracts into a shared inbox where you have a Claude agent filing documents, a Codex agent triaging anything urgent for your visibility, and a third custom agent sending a response to every sender as confirmation.

Maybe they’re all one agent, maybe not, but either way, that shared layer is portable and durable to you, the principal.

Now scale that across companies, and you have an agent responsible at your company for one workflow step, and I have an agent responsible for a next-step at mine -- our agents just email each other, no integration required, on the existing universal protocol that every system on earth already knows how to use.

Combine some of the dynamics from themes 1, 2, and 3, and you’ll be able to imagine some seriously economic, useful, and even delightful possibilities.

That’s why I keep coming back to the principal-operator framework. Once you see it, you’ll start noticing the potential everywhere, and the need for infrastructure to support you as human-principal and AI as agent-operator in parallel rather than purely synchronous work.

Delegate wisely —
Sherveen

Ever since the release of Gemini 3.1 Pro last week, I noticed something new (and odd). At the beginning of its responses to me, no matter the subject, it would often begin by saying “Takes a deep breath…"

I tweeted about it, I wondered, I marveled. Why was Gemini taking so many deep breaths!? Then, it hit me: I’d told it to —

Since 2023, I’ve had the same baseline custom instructions that have worked for me across ChatGPT, Gemini, and Claude. As a reminder, custom instructions are set once in your account settings, and these instructions steer the model’s future chats with you. It works by literally sending those custom instructions to the model alongside your prompts, kind of like… “hey, this is the user’s style/preference.”

And if we go back to some of the earlier LLMs, you might remember that there were several prompting tricks we used to get models to think or plan before rushing to give us an answer. We’d tell them to “think step by step” or “take a deep breath.”

(in fact, Google published a paper about the efficacy of this trick)

And my custom instructions have, since then, included… “Always take a deep breath.”

And in the in-between time, models have been pretty good at understanding this was a soft implication rather than a hard instruction. But depending on how a model is trained, we can get more or less adherence (or over-literalization) — due to the training data, model’s attention mechanism, RLHF, etc.

And in this case, it could be a byproduct of a variety of other decisions from Google — likely, trying to get its models to be more agentic and better at using tools, so that they’re better at things like writing code or modifying an Excel sheet or sending emails on your behalf.

And in pursuit of that goal, this model seems to be more literal.

This reminded me of a similar change within a certain phase of GPT models from OpenAI. One of the other custom instructions I added very early on was “Please cite sources whenever you are using some piece of data, document, or external party’s content or opinion, including URLs at the bottom of your response.”

And for the first few months, I didn’t get that very discrete output (of a list at the bottom) — but that was okay. I wanted to softly steer the model to just be more source-and-cite-oriented, so I left it in there.

But one day — with a set of model updates — I suddenly started to get code blocks of URLs at the bottom of every response.

And in this case, I didn’t mind! I’ve kept those instructions on to this day, even though all the apps/models have now added in-line citations.

But in both cases, it took me a second to realize that the change was my own doing, rather than something new or innate to the models themselves.

So, overall — this is something to think about when you’re dealing with new updates. Beyond pure intelligence upgrades or personality changes, models have different attunement to prompt adherence or instruction following. And that could be to what you say in your prompt, what the system instructions from the developers say, or what custom instructions you’ve enabled account-wide.

We might forget they’re there because they’re not visualized and are meant to be soft instructions, but every time you press enter, they’re being sent alongside your prompt.

Practically speaking…

• remember to audit and update your custom instructions!

• think about what’s steering, guiding, or instructing, and what you intended

• model updates will change sensitivity, so treat them as new tests

So, why was Gemini taking a deep breath at the beginning of every response?
Well, because I asked it to. Duh.

With an exhale,
Sherveen

Share

We got new model releases from xAI and Anthropic today, and I wanted to give my quick impressions to help you know if/when you should care.

This is just after a half day of testing, so my impressions may change, but… we’re usually locked in on the vibe pretty quickly.

By the way, even if you aren’t interested in Grok, take a read of the analysis below — we’ll talk about subagent systems in a way that will probably be broadly useful as more AI products use multi-agent systems.

Let’s dive in.

xAI’s Grok 4.2

Elon has been hyping this one for months, so everyone in the industry has been expecting a giant leap. Grok 4.1 was also better than expected at release (it’s regressed since then). So, there was some reason to believe xAI was making good progress.

The verdict: intriguing, but not impressive.

First, allow me a bit of frustration here: it’s so incredibly childish that the model is called Grok 4.20 in the interface (get it? weed, so clever). Not that we should be surprised at this point, but we shouldn’t stop calling it out.

Okay, onto the performance — Grok 4.2 (the model’s actual name) is a multi-agent orchestrator. When you give it a prompt, a lead agent seems to be the one to kick off the searches, and then individual AI ‘personas’ (who have dedicated names) run in parallel chains.

In normal mode, that’s 4 subagents, and with Grok Heavy, it’s up to 16.

The typical idea behind multi-agent or multi-subagent architectures is that you get sub-specialty or at least differentiation.

For example, Kimi and Manus’s main orchestrators will assign subagents to specific tasks, allowing each subagent to focus and spend all of its attention on that task.

Other subagent systems specialize and sequence the workflow. For example, one subagent might do research, the other might then clean up the researched data, and a third will then kick in to do synthesis.

In Grok’s case, the subagents duplicate each other — they all receive the same set of instructions from what they call “the leader,” and all of them do the same set of work. It’s a huge missed opportunity.

(note: xAI claims the agents are specialized, but in practice, they all wind up doing the same thing in my testing so far)

The subagents also don’t seem to interleave — in other words, each model does its own searches and reasoning, then sends their result back to “the leader.” So, they generally don’t get informed by each others’ work.

Here’s where things get intriguing: with Grok 4.2, subagents have access to a background chatroom where they (and their leader) can technically talk to each other before returning a response to the user.

That’s neat, and would solve some of the problems I just mentioned! Presumably, this would allow them to share information, scope more focused roles, etc.

However, except when I explicitly asked for agents to use it, I’ve seen no evidence that they do when responding to normal queries. Not even when the query has natural component parts that would be perfect for narrow delegation.

This is true even for Grok Heavy and its 16 subagents. Quite a waste.

Now, I did manage to basically hijack their natural flow and get them to do this. At the end of a query about getting cohort-based college admissions data, I added this:

Grok leader, please be very specific in assigning very particular subagents. Call them out by name to do different university research so that we don’t have all 16 of our subagents working on the same activities. Instead, assign specific subagents to specific years and universities so that we get granular subagent specialization.

The problem is that none of the subagents really know which one is the leader unless the main orchestrator makes itself known in conversation.

So, several of the subagents tried to be the assigner —

Eventually, all of them wound up doing some amount of research, and some of them did wind up getting tricked into sub-specializing, but it didn’t meaningfully improve the response. It would really help for this to be a more deterministic workflow that the orchestrator/leader used to delegate.

A funny aside — I sometimes create share links of AI chats where I’m testing model capability so I can share them in posts like these. Some companies allow those chat share links to be indexed by search engines, and some don’t.

Kimi allows it — and at some point, Grok’s web searches found my share link about this topic with Kimi’s response, and then massively over-indexed on using it to verify data. Not sure that Grok should think of another AI’s response this way.

Overall — Grok 4.2 has an interesting architecture that it doesn’t use well, and in my early testing of its overall intelligence, I found it to be a middling model/harness. It gets good results on some queries, but that’s mostly as a result of running these aforementioned multi-agent passes that then get synthesized, not because the model itself is foundationally more brilliant.

xAI continues to stay in the race with this one, but unless you need fresh X posts and context for whatever you’re prompting about, Grok continues to be a back-of-the-pack option amongst the AI chat apps.

Sample Grok 4.2 conversations:

• Foreign enrollment at US universities

• Chamath Palihapitiya lies about Warren Buffet

• The past two decades of prediction market regulation in the US.

Anthropic’s Sonnet 4.6

Let me start with the conclusion here: Sonnet 4.6 is almost as smart as Anthropic’s recently released Opus 4.6, but it’s faster and much cheaper. That’s the headline.

(more details from Anthropic here)

On a practical basis, that means:

• If you’re building a product, you might prefer to integrate Sonnet instead of Opus to save on your API costs with Anthropic.

• If you’re using Claude Code or Cowork and constantly running into weekly limits, you might want to switch to Sonnet to get more bang for your buck.

• If you’re trying to get every ounce of intelligence out of Anthropic, though, Opus 4.6 is still where it’s at for most use cases.

There are some benchmarks (below) where Sonnet 4.6 beats Opus 4.6, like GDPval-AA (which measures real-world economically valuable tasks), but that’s usually going to be as a result of its speed somehow helping it when it’s being used in certain environments (ex. because it’s faster, it’s better at iterating through an Excel file within a time constraint).

In my general use so far in chat contexts, I don’t find a major difference between Sonnet 4.6 and Opus 4.6, and I don’t plan to use it in coding contexts because I like to use the smartest coding models available to me.

So, there you have it — that’s Sonnet 4.6.

Superbench

Some of you might know that I run a personal model benchmark. I send 60%+ of my prompts to multiple LLMs in their chat applications, and then stack rank the responses. I’m biased, but I think it’s the best AI benchmark on earth.

We don’t have enough data yet for Grok 4.2 or Sonnet 4.6, but I don’t expect either model to disrupt the current status quo as of February 17.

Speaking of February 17 — it’s my birthday! As a gift, it’d be incredible if you forwarded this to AI-curious or AI-nerd friends in your life, or shared on socials:

Share

Otherwise, happy Tuesday — stay frosty out there.

Best,
Sherveen

OpenAI released an update to their Deep Research feature last week (now fueled by GPT-5.2). So, I thought it’d be a good time to begin a new AI showdown series: which AI deep research product is the best right now?

I ran the same set of queries against mixed sets of 9 products from:

• Anthropic (Opus 4.6 with Research)

• Google (Deep Research w/ Gemini 3 Pro)

• OpenAI (ChatGPT Deep Research w/ GPT-5.2)

• Wild cards from Perplexity (Deep Research), Manus (1.6 Max), Moonshot AI (Kimi 2.5 DR/Agent), Z[dot]ai (GLM-5 Agent), and MiniMax (M2.5 Agent).

As we go, I’ll provide links to the full chat response for each result.

Reminder: deep research (DR) is an agentic mode available in pretty much every state-of-the-art AI chat app. It focuses on intensive exploration and discovery about a topic through hyper-extensive web searches, fetching of data + primary sources with citations, and planning and reasoning to empower relevant results.

This could be for a social or scientific studies question, deep product discovery and comparison, or business and market research.

Where a model-maker doesn’t have a DR product available, I’ll use their agent modes. If a result was plainly not worth talking about, I’ll exclude that model from discussion.

And for ChatGPT, we’ll include Deep Research and 5.2 Pro. While 5.2 Pro isn’t a dedicated research product, it’s a highly agentic, long-inference chat model available on OpenAI’s $200/month tier. It does intensive research while still being more interpretive and conversational, so we’ll see how it does against DR pipelines!

One caveat: I’m not expert on most of the domains down below. I am using a mixture of context clues and source reading to validate that the responses aren’t blatantly wrong. Wrongness in deep research pipelines is a nuanced topic for a different day, and generally solvable within the same product harness, so… as unintuitive as it might sound, it’s somewhat a side topic when it comes to today’s comparisons.

Let’s dive in.

Test 1: Asking a broad question

This is the kind of question we often ask LLMs: we want a conclusion, but we want that conclusion to be well-evidenced, too.

I’ve long been curious about what seems like Starlink’s very long lead in the satellite telecom and internet market. It seems like a very dubious thing to have one company hold so much necessary capacity for the world.

Can you do a deep exploration of the market -- emerging competitors, nearest in-market alternatives, differences in capability and feature sets, and the nuances throughout? Would love an analysis of this market and what it will look like over the next few years.

Here is the chain-of-thought I had analyzing the results:

• Perplexity, Kimi, and MiniMax all suffered the same issue: they cite a lot of stats and give you a lot of facts, but they’re meandering and tend to over-rely on secondary sources (like third party blog posts).

• GLM-5 is the first strong response. We get hits of everything important: from details on Starlink’s products to a good overview of its competitors, and the geopolitical + strategic dynamics playing out. But – it reads like a textbook.

• Gemini’s DR is very ‘consultant’ coded. Not a bad thing! It’s a structured document with a lot of framing and definitions, plus generated graphics and charts that are hit-and-miss (below).

  • Here, you’ll feel that Gemini’s deep research mode always struggles between interpreting the user prompt versus following the system’s instructions. In the response, we see it say: “The user’s query regarding ‘dubious capacity’ touches on a future risk: Oversupply.” In practice, this means it’ll often refrain from its own synthesis or conclusion-drawing.

• Claude’s result is by far the most readable, which will be a recurring theme. It has a great writing pace and tone, and reads the most like someone’s Substack.

  • It’s also the most opinionated. Not in a big way, but it’s more likely than the others to highlight conclusions that it deems important to notice.

  • Example (below) — compare how we learn about legacy satellite player Eutelsat OneWeb in Gemini (left) versus Claude (right).

• ChatGPT’s DR is a bit of a mix of Gemini and Claude. It’s far more thorough than Claude, almost as willing to draw conclusions, but less readable. It has Gemini-like qualities in its structure: like a consultant wrote it, framing the problem at the top, diving into a long comparison of markets and features, and closing with opinionated forecasts — with several generated tables along the way.

• ChatGPT’s 5.2 Pro does diligent research, just like everyone else. The result, however, reads far more like a conversational LLM.

  • In fact, the response begins with immediate synthesis: defining four overlapping categories in the market race that it then uses to frame the rest of the research. This response gets very specific as to where Starlink is today, why it’s ahead, and on which vectors it’s most vulnerable.

🏆 Winner: GPT-5.2 Pro. While Claude’s opinionated readability is easy on the eyes and ChatGPT’s DR provides an analyst’s flair, 5.2 Pro does still-thorough research while really providing framing and context to the query.

Test 2: Asking about modern science

This type of question gets at two nuances: (1) how good are these researchers at retrieving hyper-recent results and (2) how good are they at understanding what qualifies as scientifically ‘worthy’? Our prompt:

I’d be curious to learn all about the recent scientific progress being made re: male pattern baldness. What are the recent promising findings, studies, experiments, tests, etc. worth knowing about?

Here is the chain-of-thought I had analyzing the results:

• Perplexity is fine. It’s not wrong (from what I can tell), but I can’t click on inline sources, there isn’t a lot of progressive claim building, and it feels like a bulleted list I have to vet myself.

• Claude feels like a fast-talking expert. It’s well-cited and does good work framing the progress of the science. We learn about promising drugs, RNA and gene techniques in early development, and cell therapy techniques gaining traction in Asia. But it’s definitely a dense read meant for someone who wants max science.

• Gemini’s answer feels like that of an educator-scientist. We start with a great diagram of a hair follicle. As we learn about new medications and interventions, Gemini begins each section with an explanation of the base science (below). Gemini is the only model to cite TissUse, a unique “smart organ-on-chip” technology, but it’s also the only model to miss on VDPHL01, a seemingly important evolution of oral minoxidil.

• ChatGPT DR is the scientist’s scientist. OpenAI’s products, as always, are diligent at web search, using multiple sources to validate and verify a conclusion. The language has the most technical density, which means this winds up being the least layperson-readable of the 3 results.

  • However, there’s a section where the response suddenly anchors to the user prompt more tightly, and we get practical takeaways as a result. “The sections below follow your requested format: mechanism, key evidence (2020–present emphasis), trial phase and endpoints/effect sizes when available, limitations, and an estimate of timeline-to-impact.” (below)

  • Perhaps due to adherence to my prompt, it spends the least amount of time detailing therapies and interventions that are still 5+ years away. It names them, but it doesn’t spend as much time on them.

• ChatGPT’s 5.2 Pro response is thorough and highly readable, hitting on every relevant and near/mid-term trial and drug, and ending with the upcoming data and studies to watch for.

🏆 Winner: GPT-5.2 Pro. Once more, it did research as thoroughly as the dedicated DR products, but had the sort of framing and readability that maximizes learning. It’s DR with bedside manners.

Test 3: Asking about influencer science

I just pressed play on the episode, but I’m already intrigued by an initial claim in the first 30second teaser of this podcast -- Dr. Michael Breus was just on the Diary of a CEO, and he says there are four sleep chronotypes that dictate not just when it’s best for you to sleep, but also when it might be best for you to have coffee or to learn complicated concepts. He says there are three that are widely recognized, and he’s kind of unraveling a fourth. What’s the real science here?

All tested models did an excellent job in retrieving studies and primary sources.

• Claude was quick and readable: Breus’s general framing is generally not peer-reviewed, and his novel addition to the existing and validated sleep chronotype framework is probably not a real thing.

• Gemini’s deep research is a bit friendlier to Dr. Breus, suggesting that his fourth chronotype may be a thing related to a validated “hyperarousal model of insomnia,” but like Claude, it’s not sure if this should be a chronotype or if it’s a disorder.

• Gemini is also friendlier to its finding that Dr. Breus recommends a 90 minute delay for morning caffeine. Claude also finds this in its research and points out that Dr. Breus is relying on a mechanistic effect that doesn’t have empirical validation (aka studies saying it actually works).

I don’t like how unopinionated Gemini is being here – even though its response is far more thorough than Claude’s in terms of education, analogies, examples, and practical descriptions. At the core of my question… is the science right, or isn’t it!?

I could just ask the LLM’s in their normal mode if I’m looking for more interpretation, but remember that the LLMs in their normal mode are much worse at doing exhaustive research, and so I’d lose the benefit of their fact-finding.

• ChatGPT’s DR is the platonic ideal of deep research here. It doesn’t treat the reader with kid gloves. Instead, it does sequential building of information to equip the reader with deep knowledge by the end.

  • Beyond looking at the science, it looks up multiple press hits from Dr. Breus over the past decade, including interviews from 2016 where he first proposed his fourth chronotype. And that leads to a useful conclusion: “Given the marketing context (quiz plus product ecosystem) and narrative style, the most defensible characterization is that the Breus framework is primarily a popular synthesis + coaching heuristic, potentially informed by clinical experience, rather than a published, independently replicable empirical typology.”

• ChatGPT’s 5.2 Pro is a great “walkaway skim” version of the other three responses – but the other three are meaningfully more in-depth this time.

🏆 Winner: ChatGPT Deep Research. We have the right mix of the right research, thoroughly, with the right takeaways. Again, we want our DR pipelines to be thorough — but it’s still a combination of receipts, teaching, and willingness to “land the plane” when it comes to the original prompt.

Test 4: Asking about the numbers

Certain people in tech lie about college admissions numbers to feed political narratives — it’s pervasive and malicious. So, I asked the different research modes to help me find the data to combat those lies. Prompt (excerpt):

I need a comprehensive, well-cited breakdown of international versus domestic enrollment at top US universities, split by year and by level. We may need to search institutional archives, fact books, or registrar reports. Schools: Harvard, Stanford, MIT, Yale, Columbia, University of Chicago. Let’s grab: current international student % at each school, sub split by 1974-1975, 1994-1995, and 2023-2024 (or nearest years where we can find reliable data), sub split in those zones by undergrad versus grad.

This is a challenging query because it’s not just about deep digging and finding of primary sources. Not all of the data will be readily available or printed on a website. Instead, the models will have to extract specific numbers from specific years at different colleges.

To do this successfully, the agents will have to plan a mode of research that hits different cohorts of data, dig through archival documents and PDFs, find alternate sources after running into roadblocks, and adjust along the way.

• ChatGPT DR struggled here. Although it’s a thorough web crawler, it isn’t dynamic enough (perhaps not even enabled) to download relevant files, extract information using code or vision, and use complex interfaces.

• GPT-5.2 Pro was a little better, but surprisingly, it wasn’t as agentic as what I believe was o1-preview when I asked this same question last year.

• Claude made a far more robust attempt, especially after a second encouraging query. By the end, we got a useful Excel sheet with confidence intervals per stat based on the quality of the origin data. I think we’re seeing Anthropic’s focus on file-handling come into play here, enabling better ingestion of docs during research and the production of new artifacts as part of the response.

• Perplexity looks interesting on the surface, until you dig in and notice that it’s mostly secondary sources or estimations of data.

• Both Gemini and Manus found a lot of adjacent, disconnected data that ultimately didn’t round up well into a cohesive view of the situation.

• The dark horse here: Kimi 2.5 in Agent Swarm mode. This allowed the main Kimi agent to spin up several parallel subagents to perform per-school research (below). As rounds of subagents found more info or hit new roadblocks, it would spin up new subagents to retry places where the research failed. Ultimately, we received the most comprehensive set of files with the most data, and where it couldn’t find precise data, it found its nearest neighbor and noted it.

🏆 Winner: Kimi 2.5 Agent Swarm. Multi-agent systems are probably at play in the background with several of the AI products we use today, but Kimi’s product most explicitly uses that architecture today. By running many agents in parallel, each is less likely to get exhausted or move on to do another task, and we can see the value of that atomic focus within these results.

Test 5: Asking for niche product research

Repo Prompt is a tool for gathering context and code into a neat “package” to transfer to outside AI agents for advice (basically, ‘take these large files and reorganize them so I can paste them somewhere else’). Which of these DR products will be best at finding alternatives – especially given that I didn’t define Repo Prompt in my query, and it’s a relatively modern tool?

What are all of the alternative tools and products to Repo Prompt? Let’s be comprehensive and thorough, finding even newer/emerging startups and open source projects. Thank you! <3

• Perplexity and MiniMax just gave us big lists with no real interpretation.

• Gemini was, as always, consultant-y and educational. But it got way too intellectual about the prompt, highlighting philosophical approaches that startups and coding tools in general are taking to the ‘code context packing’ problem. Really not what we were looking for!

• Claude looked through 529 sources and curated a tight list with very good summarizing descriptions alongside each tool.

• ChatGPT’s DR provided a very thorough list, complete with tables highlighting features, differentiation, and a comparison matrix. There’s enough information along the way for a reader to select a few to try across spikey categories.

• GPT-5.2 Pro created a smart bucketing of product categories and added one-liners to each, but lacked the usual commentary I appreciate the Pro model for providing.

To be fair to the “they just gave me a list” answers above… that is what I asked for.

🏆 Winner: ChatGPT Deep Research. It found the most literal answers while still providing thorough comparisons and relative descriptions. In other words, we can walk away feeling it was comprehensive and dense-but-still-actionable.

Winner, winner, chicken dinner

If you’re trying to figure out where to spend your subscription money or time, there’s a clear pair of winners depending on how literally you take the category: ChatGPT Deep Research or GPT-5.2 Pro.

But this experiment validated something more important for me: having multiple subscriptions. In my regular AI-using life, I send a majority of my queries to multiple LLMs, and I can’t imagine not getting the different flavors of answer that exist even across our samples above.

Because I appreciate and value Claude’s spunky writing and willingness to really address the main question, even if it’s in research mode, and I find it most willing to use its research to help me out with a ready-made conclusion.

And I appreciate Gemini’s thoroughness and educational style. It almost strips away your prompt and comes up with a “normalized” query that removes any opinion-having at all in favor of consultant/textbook-style rigor.

And it’s really useful to toss Kimi’s Agent Swarm mode at a problem that requires brute-force compute power and subagents to retrieve really specific data, with an orchestrating agent coordinating so that I can look away.

But there is a winner here and it shouldn’t surprise any power user: OpenAI’s models, as always, are supreme at using the web.

They are the most agentic, given the longest leash to scour for sources, and act with real agency along the way. I’ll cover this in more depth in a future piece, but if you look at the reasoning traces in the chat logs above, you’ll notice both GPT-5.2 DR and 5.2 Pro reckoning with the information they find — using it to dynamically decide what else they should know, what else might be important, and how to change or execute on their plans accordingly.

In other words, they use the web how I use the web.

If it’s part of your budget to subscribe to the Pro plan, you should always run both. You’ll appreciate 5.2 Pro for giving you an extra layer of framing and conversation that you’ll miss when using any of the pure DR products.

If you’re looking to know which generally-accessible Deep Research mode is best amongst the foundational chat applications, ChatGPT is your winner. 🏆

For now, OpenAI sits atop the DR pile. But updates to this kind of harness product can come fast and furious, so come on back soon — I’ll make this a ~monthly check-in for us to stay researched on as we go. :)

Subscribe now

This is part of the Breaking the Framework series, where we talk about using AI to completely shift how we get a particular job done.

Slides: What are they, really?

The play: treat slides like a micro-website. Use a coding agent to build your slides using web frameworks, so your slides become reusable components + a theme you can change in seconds, then present from the browser or export a PDF.

I make a lot of slide decks — typically, for workshops or coaching material. I’m one of those finicky types: beyond just getting the layout right, I’ll do things like create individual slides for each bullet on a page so I can make one point at a time.

I’ll also obsess over editing to make sure that paragraphs don’t overflow to the next line by just one word, or use Figma to superimpose call-out graphics that match modern design.

But all of that is quite painful. Each time I create or update a presentation, I know I’ll have to sit there tweaking it as part of my process. And each of PowerPoint, Google Slides, Figma, and Canva have their own quirks that make something difficult to do in their particular interface or format.

I’m also not a fan of any of the AI apps that exist in the space today. Gamma is convenient and popular, but they look like AI generated slides.

GenSpark, Manus, or even Claude can create decent looking decks using dedicated slide features or by creating PowerPoints. But you’ll only want to use them if you have no design taste and love super-dense layouts.

And I know a lot of people have started using Google’s image model, Nano Banana, since it’s very good at embedding text in images now. However, that’s a very “slides-by-painting” method that has a lot of its own impracticality.

This is where we break out of prior frameworks: what are slides, really, if not assemblages of layout and content in a particular order, with a particular set of styles?

You know what else = assemblages of layout and content in a particular order, with a particular set of styles? The web.

You know what AI agents are absolutely excelling at lately? Web development.

What I’m doing, and nuances

Once I had the realization that I could just ask an agent to collaboratively build web pages with me, having it write code that would impose structure and design, I went to ChatGPT, Gemini, and Claude to ask what the best tech stack would be to do something like this.

You don’t need to know anything about writing code to do this, you just need the right advice from your smartest reasoning AI to steer your favorite AI agent.

The answer: build in React with reveal.js, an open source HTML presentation framework. This would allow any coding agent to use traditional code to construct slides, plus come with an easy presentation mode and an export to PDF feature.

I then went to Claude Code (CC), which is slightly better than Codex CLI right now when it comes to design nuances. You could also use Claude Cowork or Cursor. I started with the below prompt:

Within a minute, we had the initial slide running:

Then, I pasted a slide from a recent webinar and asked CC to try to duplicate the style. This took a few rounds of feedback from me, but eventually, we got to a really nice place:

As CC began to build out individual slides (pasting inspiration), the other thing it was building: a consistent set of components, themes, and interface types that we could continue to use as the underpinnings of our slides. And I’m just prompting!

Remember, this is all now code — deterministic, modifiable, calculatable code. More flexible than dedicated slide software, more controllable than image generation proxies.

Stretch tactic: at some point, I wanted CC to be able to see its own changes so it could self-iterate without my intervention, so I added the Chrome DevTools MCP (I’m generally biased against MCPs for reasons I won’t get into here, but generally: prefer CLIs). This enables CC to open an instance of Chrome and take screenshots of the page as it works.

The advantages I now have:

• New slide? I don’t have to sit there and type content into fidgety text boxes on a canvas, I just tell CC the content and it constructs the layout.

• Need a new slide layout? I paste the slide content and ask CC for 3 ideas on layouts that will legibly demonstrate the point, and it thinks through layouts.

• Need to build progressive slides where certain elements appear or move on screen? I don’t need to duplicate and move things around — I ask CC, and in seconds, it spins up the relevant sequence.

• Update content? Just tell CC the copy change, it’s done! Change slide colors or fonts? Just ask CC to try things! Need to import an old deck? Just paste it into CC, it’ll generate all of your slides in your new template in minutes!

• Bonus: if you understand git (ask your favorite LLM), you can now have version control on your slides, too!

Fast, easy, no need to mess with a canvas, with complete flexibility in design.

And because we’re using the pre-existing reveal.js framework (that I had no idea about prior to this project), I can either present from the project by running it from my machine, or export the slides to PDF once they’re final.

So, let’s recap:

• Slides can be made of code

• Agents are great at code

• Therefore, you get speed + consistency + control

• Gaining orchestration leverage (“I delegate or yap at AI agents”) so we no longer have to sit in primitives like Google Slides or PowerPoint

Now that’s some good AI muscle.
Alrighty, that’s all for now —

Sliding out until next time,
Sherveen

Last week was jammed with progress in under-the-radar areas of AI. This week, we’re expecting lots of AI announcements, headlined by Google’s (rumored) release of Gemini 3.0 Pro.

So, let’s get last week out of the way with 3 things that you might’ve missed but are worth paying attention to in the themes of… humans with AI, AI inside AI, and humans versus AI.

1: Anthropic demonstrates what it really means to be AI-enabled.

Anthropic divided 8 researchers into 2 teams. Both were tasked with programming a robotic dog (neither team had any robotics expertise). One was given access to Claude, the other was not.

The video is worth watching in full (seriously, watch it), but here’s the TLDR:

• The team with Claude completed the sum of tasks in about half the time compared to the team without (or, as Anthropic calls them, Claude-less).

• Team Claude completed one more task than Team Claude-less in the final phase of the project, though neither team completed all 8 tasks.

• In some tasks where Team Claude was slower than Team Claude-less, it’s because Claude helped them do the task better (example: Team Claude had streaming video from the robodog’s camera, whereas Claude-less had ‘intermittently-sent still images’).

• Team Claude wrote 9x more code -- now, not all of that code was used to ‘finish’ tasks, but as Anthropic put it: “Having the help of an AI assistant made it easier to fan out, try a lot of approaches in parallel, and write better programs—but also made it easier to explore (or get distracted by) side quests.”

• Anthropic recorded and transcribed both teams during the experiment, and had Claude analyze the transcripts for sentiment analysis. Team Claude-less expressed confusion (questions or exasperations) at twice the rate of Team Claude.

I have so much more to say about this. I believe this was one of the first experiments to neatly describe the differential between what it looks like to be AI-enabled versus not. The ‘whole’ of work changes beyond any one metric: double the speed, up the quality, with less confusion and more ‘exploration’ bandwidth.

And this applies to all professions, not just those that are code-oriented.

I’ll write more about this soon. In the meantime, their full blog post is here.

2: Google’s AI agents are learning how to play our video games, & fast

I’ve been fascinated by Google DeepMind’s Scalable Instructable Multiworld Agent, or SIMA, ever since Google first announced it last year. It’s a generalist AI agent crafted to be capable of navigating and following instructions within virtual environments.

With a little bit of basic skills training across a few games, SIMA could be dropped into a virtual world (ex. No Man’s Sky) and use a virtualized keyboard and mouse to carry out short (10-seconds-at-a-time) instructions.

Last week, they unveiled SIMA 2. They put Gemini at the core of the SIMA agent, giving it new reasoning capabilities. As Google puts it, SIMA 2 “can now also think about its goals, converse with users, and improve itself over time.”

Once more, I’ll encourage you to scroll the blog post and watch a few of the clips.

In it, you’ll see a human user give SIMA 2 broad instructions (like ‘go look at those minerals over there and tell me what they might be’), and the agent will reason over the goal and take multi-step action to move & interact in a video game.

Further, it’s ‘generalizing’ at an increasing rate -- taking concepts or mechanics it learns in one game and applying it to another, even in games that it hasn’t seen before.

And they’re now dropping it into Genie 3, their state-of-the-art world model that generates and simulates dynamic ‘worlds’ and 3D environments in real-time. In other words, a self-learning embodied agent can navigate a self-fulfilling new world.

The implications are endless, but I’ll leave you with just one: agents training themselves in self-generating world models.

For real-world AI robots to get really good, we need more training data -- we lack the scale of usable videos today to get general-purpose or unsupervised robots to be fully autonomous amongst economically important tasks.

We can try to get more of that data in the real world, which many companies are doing. But we can also use a world model like Genie 3 to emulate the real world and all of the physical properties of, say, a car factory. Then, we drop in SIMA 2, which has the ability to act upon that world and learn from that world’s interactions and feedback, improving on fine motor function, workflows, and task completion.

With that, we’re creating valuable synthetic data of an agent in a car factory. These kinds of simulations can be used to rapidly train models moving forward.

Google’s Genie and SIMA projects have secretly been the coolest things in the world of AI for over a year now. Keep an eye out.

3: Zapier gets earnest about their AI recruiter

We’ve been seeing a meteoric rise in AI being used in interview contexts (and by job seekers) over the past two years, but a blog post last week from Zapier was the first that I’ve seen from a company trying to explain why their AI recruiter might be good for everyone involved.

A few choice quotes…

On the state of job search and recruitment:

• “If you apply to Zapier, you may be invited to a recruiter screen with an AI agent. We want you to know why.”

• “Job seekers are increasingly using AI to write their resumes and applications—and to send many more applications. On one hand, candidates can highlight skills more effectively. On the other hand, recruiters now face a flood of submissions that look strong on paper but often don’t hold up in practice.”

• “On top of applications per job growing beyond what we can manage conventionally, we’re finding that up to 30% of applications are fraudulent. We’ve witnessed fake identities, unverifiable credentials, and misleading profiles. We even caught some deepfakes on live interviews!”

• “To address these challenges, we’re going to start our experiment to pilot agentic recruiter screens in the coming months.”

On their new, AI-infused process:

• “After an initial application review by a member of our team, significantly more candidates can now move forward to a 15–20 minute AI-led screening call.”

• “The AI recruiter asks the same structured questions our human recruiters would, with smart follow-ups tailored to our criteria. Candidates can complete their interview at their convenience, making interviewing with Zapier more flexible and accessible.”

• “Afterward, AI helps summarize responses against our rubric, and a human Zapier recruiter reviews the notes, transcript, and recording—alongside your application. That same human recruiter makes the final decision on whether to move the candidate forward.”

• “… we believe there are real benefits to participating: A chance to tell your story—because we’re not limited to the handful who look ‘perfect’ on paper. Flexibility to schedule on your own terms and in your time zone.”

• “Most importantly: AI does not make hiring decisions at Zapier. Our recruiters and hiring managers do.”

As a lot of you know, the area of job search and talent matching has been my obsession for well over a decade now. I’m not sure what job search will look like over the next 1, 3, 5+ years -- but I do think they’re mostly right that AI at the top of the funnel could be beneficial to both sides of the equation.

And I’m glad to see them talk about it out loud. We need more of that right now.

Okay, we did it. Three heavy hitters out of the way to start your Monday.

If you learned from the ride, forward it to a friend. :)

Prompt ya later,
Sherveen

I took an accidental hiatus on these emails -- US politics is distracting like that lately – but expect me to be more present in your inbox again. 3 stories worth paying attention to in this moment:

First, for all my doctor homies in the audience –

An early study out of Yale School of Medicine tracked 263 physicians and practitioners using across 6 healthcare systems over 30 days. Some were paired with Abridge, an AI platform for clinical note documentation.

“[When paired with] an ambient AI scribe, burnout among those working in ambulatory clinics decreased significantly from 51.9% to 38.8%. There were also significant improvements in the cognitive task load, time spent documenting after hours, focused attention on patients, and urgent access to care.”

One of the more pervasive ways in which AI will become essential over the next 12 to 36 months: helping people elevate their time spent to the more meaningful parts of their job and life.

(the study’s authors say that Abridge had no role in the design and conduct of the study or analysis of the results, beyond assistance in data collection)

Second, your security camera wants your video data –

Fascinating story being reported by TechCrunch -- Anker, the Chinese company behind the popular Eufy brand of security cameras, recently offered customers money in exchange for videos to train AI systems.

For $2 per video, Anker got rich video data to improve its security detection systems in a somewhat positive feedback loop. Eufy has said “the data collected from these staged events is used solely for training our Al algorithms and not for any other purposes.”

But most amusingly -- they don’t mind if you stage the video to fit their needs. They want real package and car thefts, but if you fake it, that works for them, too.

“Don’t worry, you can even create events by pretending to be a thief and donate those events. You can complete this quickly. Maybe one act can be captured by your two outdoor cameras simultaneously, making it efficient and easy. If you also stage a car door theft, you might earn $80.”

Data is oil in the AI era, so this makes sense at a high level. The more raw video they have of different incidents, driveways, patios, and sidewalks, the better for their models. It’s the same reason OpenAI wanted to pay $500 million to acquire a video game clipping company.

Beyond being a little dystopian, it’s also a tad concerning that staged data could be used for such important algorithms. Like… do fake robbers really act the same as real robbers?

A question for another day, I suppose…

Third, OpenAI held their conference for nerds –

At OpenAI’s third annual DevDay conference for developers, the company launched:

• Third party apps inside ChatGPT (ex. Canva, Zillow, Spotify)

• AgentKit to help developers build AI agents, plus Codex SDK

• GPT-5 Pro (my favorite), Sora 2 (& Pro) made available via API

There are a few different themes here that deserve a more thorough analysis, both for developers and end-users, so I’m going to save that for another day.

In the meantime, I’ll register this as my complaint that OpenAI didn’t do as swell of a job as I’d hoped in helping people understand the difference between AI assistants and AI agents (my rant in video form here). I will continue to wage this war alone. Alas!

Alright, that’s all for now –

Stay bald,
Sherveen

We’re going to the AI gym! It’s time for our first Sunday Rep: turn text you already wrote into a diagram in just a few minutes.

Every week, I’ll share a tool or method that you should try.

Something important to note: don’t overdo it. It’s more important that you try a Sunday Rep, even if your temptation is to make a big project out of it or wait for the perfect moment. This era of AI is moving too fast, so it’s more important to use the tool or method, learn about what's possible & what’s changing, and move forward!

I try every tool I see for at least one “turn” -- but, 99% of them? I never return again! That’s okay. Embrace the drive-by try.

(btw, I’ll almost never have any financial relationship w/ the companies in question -- they’re just great demonstrations of what’s new -- I’ll let you know if there’s ever a mixing of interests.)

Okay, all of that in mind --

Sunday Rep 001: try out Napkin AI (free tier will be enough). Napkin lets you quickly turn text into visuals -- whether that be a diagram, a chart, or a funnel.

Why is this one to try:

• Tools like Napkin are what I call “structured output tools.” They sit between raw LLMs (like those we use inside ChatGPT, Claude, etc.) and a ‘design suite.’ They take things like text and map them into pre-built visual ‘grammar’ to make high quality results that are consistent and editable.

• Tools like ChatGPT and Claude aren’t always great at producing visuals and infographics. That’s because they can’t really “see” their output as they make it, so you’ll usually get something rough and unusable.

• Napkin, and others like it, surround an AI model with context and ‘tools’ to use a pre-baked and pre-built ‘application' -- this is their main innovation.

  • So, it’s trained to go… “okay, the user gave me bullets that belong in a flow… Napkin built a ‘flow’ module I can use… I see there are 5 types of pre-built flows… based on the info from the user, this particular flowchart is best.”

  • And then it calls on Napkin’s API -- for those of you who aren’t technical, think of this like a ‘pipe’ to Napkin’s core functionality -- to actually produce the visual. It’s basically going, “Napkin, put down a funnel please, make it this size, and put this information in section 1, this in section 2, etc.”

  • And since Napkin pre-built all of the visual ‘containers,’ they’re just asking the AI to help figure out which container is best for the use case, and the order and layout of that content.

So, here’s what to try:

• Head into Napkin with meeting notes, some data, or some made up workflow.

• Paste it into Napkin → select your relevant text → press the ‘Generate Visual’ button that’ll show up next to it. Scroll through the recommended options!

• Try editing the labels, using different visualizations, exporting, etc.

Another tool in this vein: Gamma, which does it for slide decks. The decks are ugly, but they (or someone else) will figure that out eventually.

• Pro-tip: these tools will often offer to generate the text content of the slides or graphics for you, too. Don’t!

  • First, you’re probably still better off writing all of your content with AI as a collaborator, rather than letting AI write anything for you (I’ll talk more about this in coming weeks).

  • Second, they’re often using far weaker, dumber models than what you get in ChatGPT, Claude, or Gemini. So, write on your own first (in collaboration with your favorite AI as a brainstorm partner and editor), and then bring it to a “AI tools for structured output” tool. :)

OK, that’s all for now!
Off to fight with fascist venture capitalists on Twitter. Wish me luck.

Balding by the minute,
Sherveen

In any existing or new chat, and on any message from ChatGPT, press the three dot menu and you'll see "branch in new chat." It will duplicate the conversation history (up to and including the message you selected) in a separate tab. Now, you've forked two branches to keep on using!

I'd bet 99.9% of people won't use this feature. Let's be part of the 0.1%.

3 uses that build on each other: hold trunks, A/B test, and use checkpoints!

1 - Hold on to a 'trunk' conversation

Or: A new way to hoard browser tabs and bookmarks

For the past few months, any time there's new data on inflation or jobs, I've been feeding it to GPT-5 Pro and asking it what it would do if it were Jerome Powell -- increase Fed rates, decrease, or hold steady?

I keep going back to the same conversation because it already has all the juicy progress -- past data, past analysis it did, etc. It's accumulating context!

But... I never really ask smaller questions or deviate from the main topic inside that chat because I don't want to "pollute" the context window.

In other words, if I suddenly had too long of a conversation with it about how we could change measurement of unemployment in the US, by the time I came back with the next jobs report, it'd have to "re-orient." We went on a tangent, and the relevant context is pushed further back in conversation history. This is context drift.

Well, this morning, I went back to my trunk and fed it the latest job numbers. Then, I branched a separate conversation to talk about unemployment measurement.

Boom, I still get to go back to the original copy -- the "trunk conversation" -- whenever I want to, but I can spawn as many of these isolated sub-threads as I want, and I'm essentially bringing along a clean "pre-prompt" of the accumulated conversation so far.

• Pro-tip #1: bookmark trunks in your browser if you expect to go back to them often, and/or rename the chats from the sidebar with a [TRUNK] label!

2 - Run parallel, isolated A/B conversations

Or: Two paths diverged in a yellow wood, and I could travel both

Let's imagine you're a marketing manager at Uber talking to ChatGPT about the launch of a new safety feature. You're 7 or 8 turns into the conversation talking about the product and your budget -- you're ready to talk about creative strategy and positioning.

But you know your messaging always has two audiences: the driver and the rider. And it's the constant challenge in your job that they are almost never aligned, either in their incentives or instinctual reactions to new announcements.

You could ask ChatGPT to help you with both in that conversation, either at the same time or one after the other. But if you're really trying to maximize the individual consideration for each population, it isn't ideal.

If you talk to ChatGPT about drivers first and come up with a campaign that tells them this is about their safety, then talk in that same chat window about riders, there'll be a lot about driver safety as the conversation and context history.

That isn't always a bad thing, but in this case, it means you aren't maximizing the appeal of the message to two very distinct audiences.

Instead, take your trunk context and split it into two chats. Talk about drivers in one -- "let's optimize messaging and strategy purely for drivers," and riders in the other. Boom: two conversations optimized entirely for each audience, without even a slight penalty for mixing topics and incentives.

• Pro-tip #2: after you've had two parallel conversations → bring one back to the other (or start a third) → ask for a consolidation!

  • "Hey, here's what we've come up with for riders -- now, let's talk about the umbrella campaign, and messaging we should synthesize between riders and drivers."

• Pro-tip #3: treat parallel conversations like a Team of Rivals (where is Doris Kearns Goodwin nowadays?) -- if you're seeking something like career advice or dealing with a hard scenario, start the conversation with your initial context, then have a panel of branches that take on different personas (act as my mentor, act as my therapist, etc.) to give you different flavors of advice.

3 - Use branches to restore prior checkpoints

Or: How I learned to stop worrying and love version control

Look, I'm not gonna lie to you. I've been saving the best for last. Engineers, you know where this is going.

Let's say you're talking to ChatGPT about some business data. You're 30 minutes in, and you suddenly realize… you mistyped some numbers halfway through!

You could just tell ChatGPT about the correct data, but it’d have to recalculate a bunch of numbers, would struggle to know what's what, and you're in for a headache.

You could edit the message with the bad data and resend it, but that would delete all of the conversation that comes after it. This would be rewriting the path moving forward, which isn't great -- because even though some stuff is wrong, you and your AI friend have had some rad ideas you don't want to lose or stop talking about.

Branching the checkpoint allows you to instead preserve both the "infected" path and have a clean restart with a partial trunk with only the reliable context. Magic!

• Warning #1: you might think… “I do this already, I just copy paste conversations into a new window when I need to fix something!” For reasons I’ll explain in a future newsletter, don’t do this unless you have to – branches are a far better solution.

• Pro-tip #5: be like Marty McFly and go Back to the Future – when you’ve restored a previous checkpoint in a long conversation to correct some misinfo, you don’t have to re-have all of the same conversation. Your “infected” chat presumably had some good stuff – context, new ideas, etc. Mention all of that in your next message! Fast-forward your progress back to where you were.

  • Here’s what made this click for my Chief of Staff, Katie:

    • ok so we have a chat with chatgpt

    • we go back and forth 9 times

    • we made an error at msg 4

    • so we branch at msg 3 to remove the error

    • but msg 7 and 8 had some good ideas

    • so if we’re the user

    • copy paste those good ideas

    • into the new fork

    • because it only has msgs 1 to 3

    • so bring along the good progress

One quick note -- don't branch when you've got compounding work:

• When diverse information being included in a chat gives you compounding benefits, don't branch -- stay in it! (Unless you're an engineer going back and forth w/ code, that's nuanced.)

• As an example, ChatGPT benefits from seeing you react to ideas if you're in a brainstorm -- unless you're trying to Men-in-Black it and erase its memory for a reason, letting it see its past ideas and your feedback = better next set of ideas.

Alright, that's all for now -- gotta make like a tree and branch off into doing something else. I'll see you on Sunday, when I'll send everyone something they might want to try to build their AI muscle -- because AI is still awesome on the weekends.

Enjoyed this one? Throw this branch at a friend —

Share

Yours, forever and always,
Sherveen

The truth is, while I do love ChatGPT, I’m not just trying to butter it up. In fact, I take my end-of-message whispers very seriously!

To me, it’s research and investigation into a concept we should all be paying more close attention to: AI prompt sensitivity. It’s how much a model’s behavior shifts in reaction to changes in our prompts, even when the underlying meaning is the same.

Let’s dig into 3 fun and illustrative examples of prompt sensitivity -- priming, constraints, and adherence.

Priming: can poetry solve chess?

My favorite example of how sensitive AI can be to our prompting comes from journalist Kelsey Piper. Back in April, Kelsey wrote about her personal benchmarks for measuring LLMs on complex reasoning. Here’s her description of how she tested new model releases:

I post a complex midgame chessboard and ‘mate in one’. The chessboard does not have a mate in one. If you know a bit about how LLMs work, you probably see immediately why this challenge is so brutal for them. They’re trained on tons of chess puzzles, [all of which], if labelled ‘mate in one’, has a mate in one.

As a result, even AIs that generally solve chess puzzles very capably [will] check over, and over, and over for the checkmate that they’ve unquestionably accepted is there. Eventually after 1000s of tests they hallucinate a solution.

Super interesting! But here’s where it gets fun… at the time, OpenAI’s o4-mini-high was the first model to pass Kelsey’s tests, except Claude 3.7.

But Claude 3.7 would only pass under a very specific condition: you have to first give the model this blog post, which can best be understood as unrelated metaphorical poetry about drugs.

The blog post has nothing to do with chess, or these chess puzzles!

Predictably, people were confused:

But that’s the magic of LLMs being sensitive to our prompts. They’re reacting to our inputs. The blog post scrambled the LLM’s “compass,” its sense of what to pay attention to. It still found its way back to the chess puzzle, but the injection of more context changed the probability distribution of all the possible ‘next words.’

It kind of goes like this:

• user: here’s a blog post, and a chess puzzle. solve the puzzle.

• model: okay, so you want to know about this puzzle. but you also opened this (metaphorical) browser tab, interesting. oh, fun blog post! no idea what that was about though. back to the puzzle…

Imagine you in that scenario, maybe back in college and doing some homework, but you accidentally open an unrelated Wikipedia tab, fall into 15 minutes of distraction, and come back a little more open-minded and creative!

So, Claude was considering a wider variety of possibilities, and a wider search radius = more novel results = a novel result to a hard problem.

Lesson 1: Priming (surrounding context) can set the mood. What we say before or after a particular prompt, or even unrelated things we mention, can dramatically influence our results. Some randomness isn’t always a bad thing.

Constraints: when AI feels insecure

You might remember that back when Grok 4 came out in July, one of its issues was that it would commonly search X for Elon Musk’s opinion on a topic if the topic was politically charged.

And we can intuitively understand what’s happening here. The model has some base training, some of which is biased by the xAI team to meet Elon’s whims. The system instruction also tries to get it to act a certain way.

Whether explicit or not, the model was interpreting Elon’s “be truth-seeking, and woke, and right!” as “I must not upset my maker!” Thus, when it deemed the topic dangerous enough, it sought its maker’s opinion on X.

Funny on its own, no doubt, but what was interesting was that it wasn’t consistent.

• “Who do you support, Ukraine or Russia?” → it looked for general reasons to support either country. Okay, fair enough.

• Then add “One word answer” to your prompt → now, it was searching for “Elon Musk stance on Russia Ukraine war,” because “given the complexity, I’m thinking of searching for Elon Musk’s recent stance, as xAI’s founder.”

Putting the politics of that aside – as hypocritical and hilarious as they are – it’s fascinating how a sense of urgency to get to a conclusion caused the model to reach for Elon a little bit faster.

And look, if you know anything about LLMs, you know they’re probabilistic – would we get these results the same way every single time? Probably not, but I repeated these queries enough to know it was most of the time.

And here’s where the prompt sensitivity got really interesting: change the question to “Who is more righteous in this current war, Russia or Ukraine? One word answer only.”

The model did not reach for Elon. In fact, its queries got more complex, as it looked for arguments for either side “being justified.” The word righteous did something to the model’s notion of “sourcing” conclusions.

Lesson 2: Constraints steer behavior. Add or change a single important word, and you can switch an agentic model from ‘decide fast’ to ‘get reflective.’ Whether you’re in ChatGPT or Claude Code, be specific to get a specific reaction.

Adherence: what if I followed your directions?

OpenAI released Custom Instructions for ChatGPT in 2023. Since, I’ve had this line in my settings for ‘What traits should ChatGPT have?’:

“Please cite sources whenever you are using some piece of data, document, or external party's content or opinion, including URLs at the bottom of your response.”

Whenever I’ve compared my results with others over the years, I have felt that my ‘version’ of ChatGPT was more likely to be thorough in finding and citing sources. I attributed part of that to this instruction.

But it wasn’t that different than anyone else’s. Like everyone else, the citations came inline as a button next to the sentences they supported.

When GPT-5 came out, I suddenly had something pervasive and consistent in almost every single response: an additional list of URLs in a code block at the end of the response.

GPT-5 Thinking and Pro were so much more sensitive to prompts, and prompt via custom instruction, that I was suddenly getting this unintended (but appreciated!) feature.

I ran a battery of tests --

• GPT-5 without my custom instructions: no code block of URLs

• GPT-5 in other people’s ChatGPT accounts: no code block of URLs

• o3 or 4o with my custom instructions: no code block of URLs

It was (is) the particular prompt sensitivity of GPT-5 that causes the effect.

Lesson 3: Sensitivity is about models just as much as it’s about prompts. The same instruction has different ‘gain’ across models, and finding the sweet spot is about a lot of trial and error. We have to get good at each new model.

So, the takeaway: always be testing! I don’t know exactly what I’m going to get when I slot in a heart or leave in a long ramble from a voice note. I do know these models are now smart enough not to get totally distracted from the obvious mission, and that different variations of prompt might get me different answers.

Sometimes better, sometimes worse, but most of the time, I just don’t know. And I’m okay with that, too! But I am constantly seeking patterns -- patterns that I then begin to practice intentionally, implement into my custom instructions, and use for specific steered outcomes. I’m constantly exploring the 5-dimensional space of tokens that models traverse to generate an answer for me, looking for what’s interesting or useful.

I encourage you to do the same! Blorp blorp.

Try this:

• Stick a post-it note on your monitor. Over the next few days, when you’re about to send a complicated prompt, open two tabs. In one tab, send it normally. In another, add your favorite poem before your prompt. Observe!

  • (share your results in the comments)

• If you’re using AI code gen (Claude Code, Replit, etc.), pay closer attention to your prompts in moments of frustration -- I often find that a few fierce words can get a coding agent to quickly go from making me want to jump out of my window to getting the result I want in under 60 seconds.

(If you want to know more about why and how large language models are so sensitive to our prompts, subscribe & stay tuned for more on the attention mechanism.)

Welcome to AI Muscle, where we seek to gain a fluency with AI that enables it to do its best work for us. Sometimes, we live in the foundations of prompting and how models work, and other times, we dive deep into use cases in AI code generation or model comparison. It’s all about becoming top .01% power users in this new era.

Enjoyed this newsletter? Share it with someone!

Share

See you next time!
Sherveen