Ever since the release of Gemini 3.1 Pro last week, I noticed something new (and odd). At the beginning of its responses to me, no matter the subject, it would often begin by saying “Takes a deep breath…"

I tweeted about it, I wondered, I marveled. Why was Gemini taking so many deep breaths!? Then, it hit me: I’d told it to —

Since 2023, I’ve had the same baseline custom instructions that have worked for me across ChatGPT, Gemini, and Claude. As a reminder, custom instructions are set once in your account settings, and these instructions steer the model’s future chats with you. It works by literally sending those custom instructions to the model alongside your prompts, kind of like… “hey, this is the user’s style/preference.”

And if we go back to some of the earlier LLMs, you might remember that there were several prompting tricks we used to get models to think or plan before rushing to give us an answer. We’d tell them to “think step by step” or “take a deep breath.”

(in fact, Google published a paper about the efficacy of this trick)

And my custom instructions have, since then, included… “Always take a deep breath.”

And in the in-between time, models have been pretty good at understanding this was a soft implication rather than a hard instruction. But depending on how a model is trained, we can get more or less adherence (or over-literalization) — due to the training data, model’s attention mechanism, RLHF, etc.

And in this case, it could be a byproduct of a variety of other decisions from Google — likely, trying to get its models to be more agentic and better at using tools, so that they’re better at things like writing code or modifying an Excel sheet or sending emails on your behalf.

And in pursuit of that goal, this model seems to be more literal.

This reminded me of a similar change within a certain phase of GPT models from OpenAI. One of the other custom instructions I added very early on was “Please cite sources whenever you are using some piece of data, document, or external party’s content or opinion, including URLs at the bottom of your response.”

And for the first few months, I didn’t get that very discrete output (of a list at the bottom) — but that was okay. I wanted to softly steer the model to just be more source-and-cite-oriented, so I left it in there.

But one day — with a set of model updates — I suddenly started to get code blocks of URLs at the bottom of every response.

And in this case, I didn’t mind! I’ve kept those instructions on to this day, even though all the apps/models have now added in-line citations.

But in both cases, it took me a second to realize that the change was my own doing, rather than something new or innate to the models themselves.

So, overall — this is something to think about when you’re dealing with new updates. Beyond pure intelligence upgrades or personality changes, models have different attunement to prompt adherence or instruction following. And that could be to what you say in your prompt, what the system instructions from the developers say, or what custom instructions you’ve enabled account-wide.

We might forget they’re there because they’re not visualized and are meant to be soft instructions, but every time you press enter, they’re being sent alongside your prompt.

Practically speaking…

• remember to audit and update your custom instructions!

• think about what’s steering, guiding, or instructing, and what you intended

• model updates will change sensitivity, so treat them as new tests

So, why was Gemini taking a deep breath at the beginning of every response?
Well, because I asked it to. Duh.

With an exhale,
Sherveen

Share

We got new model releases from xAI and Anthropic today, and I wanted to give my quick impressions to help you know if/when you should care.

This is just after a half day of testing, so my impressions may change, but… we’re usually locked in on the vibe pretty quickly.

By the way, even if you aren’t interested in Grok, take a read of the analysis below — we’ll talk about subagent systems in a way that will probably be broadly useful as more AI products use multi-agent systems.

Let’s dive in.

xAI’s Grok 4.2

Elon has been hyping this one for months, so everyone in the industry has been expecting a giant leap. Grok 4.1 was also better than expected at release (it’s regressed since then). So, there was some reason to believe xAI was making good progress.

The verdict: intriguing, but not impressive.

First, allow me a bit of frustration here: it’s so incredibly childish that the model is called Grok 4.20 in the interface (get it? weed, so clever). Not that we should be surprised at this point, but we shouldn’t stop calling it out.

Okay, onto the performance — Grok 4.2 (the model’s actual name) is a multi-agent orchestrator. When you give it a prompt, a lead agent seems to be the one to kick off the searches, and then individual AI ‘personas’ (who have dedicated names) run in parallel chains.

In normal mode, that’s 4 subagents, and with Grok Heavy, it’s up to 16.

The typical idea behind multi-agent or multi-subagent architectures is that you get sub-specialty or at least differentiation.

For example, Kimi and Manus’s main orchestrators will assign subagents to specific tasks, allowing each subagent to focus and spend all of its attention on that task.

Other subagent systems specialize and sequence the workflow. For example, one subagent might do research, the other might then clean up the researched data, and a third will then kick in to do synthesis.

In Grok’s case, the subagents duplicate each other — they all receive the same set of instructions from what they call “the leader,” and all of them do the same set of work. It’s a huge missed opportunity.

(note: xAI claims the agents are specialized, but in practice, they all wind up doing the same thing in my testing so far)

The subagents also don’t seem to interleave — in other words, each model does its own searches and reasoning, then sends their result back to “the leader.” So, they generally don’t get informed by each others’ work.

Here’s where things get intriguing: with Grok 4.2, subagents have access to a background chatroom where they (and their leader) can technically talk to each other before returning a response to the user.

That’s neat, and would solve some of the problems I just mentioned! Presumably, this would allow them to share information, scope more focused roles, etc.

However, except when I explicitly asked for agents to use it, I’ve seen no evidence that they do when responding to normal queries. Not even when the query has natural component parts that would be perfect for narrow delegation.

This is true even for Grok Heavy and its 16 subagents. Quite a waste.

Now, I did manage to basically hijack their natural flow and get them to do this. At the end of a query about getting cohort-based college admissions data, I added this:

Grok leader, please be very specific in assigning very particular subagents. Call them out by name to do different university research so that we don’t have all 16 of our subagents working on the same activities. Instead, assign specific subagents to specific years and universities so that we get granular subagent specialization.

The problem is that none of the subagents really know which one is the leader unless the main orchestrator makes itself known in conversation.

So, several of the subagents tried to be the assigner —

Eventually, all of them wound up doing some amount of research, and some of them did wind up getting tricked into sub-specializing, but it didn’t meaningfully improve the response. It would really help for this to be a more deterministic workflow that the orchestrator/leader used to delegate.

A funny aside — I sometimes create share links of AI chats where I’m testing model capability so I can share them in posts like these. Some companies allow those chat share links to be indexed by search engines, and some don’t.

Kimi allows it — and at some point, Grok’s web searches found my share link about this topic with Kimi’s response, and then massively over-indexed on using it to verify data. Not sure that Grok should think of another AI’s response this way.

Overall — Grok 4.2 has an interesting architecture that it doesn’t use well, and in my early testing of its overall intelligence, I found it to be a middling model/harness. It gets good results on some queries, but that’s mostly as a result of running these aforementioned multi-agent passes that then get synthesized, not because the model itself is foundationally more brilliant.

xAI continues to stay in the race with this one, but unless you need fresh X posts and context for whatever you’re prompting about, Grok continues to be a back-of-the-pack option amongst the AI chat apps.

Sample Grok 4.2 conversations:

• Foreign enrollment at US universities

• Chamath Palihapitiya lies about Warren Buffet

• The past two decades of prediction market regulation in the US.

Anthropic’s Sonnet 4.6

Let me start with the conclusion here: Sonnet 4.6 is almost as smart as Anthropic’s recently released Opus 4.6, but it’s faster and much cheaper. That’s the headline.

(more details from Anthropic here)

On a practical basis, that means:

• If you’re building a product, you might prefer to integrate Sonnet instead of Opus to save on your API costs with Anthropic.

• If you’re using Claude Code or Cowork and constantly running into weekly limits, you might want to switch to Sonnet to get more bang for your buck.

• If you’re trying to get every ounce of intelligence out of Anthropic, though, Opus 4.6 is still where it’s at for most use cases.

There are some benchmarks (below) where Sonnet 4.6 beats Opus 4.6, like GDPval-AA (which measures real-world economically valuable tasks), but that’s usually going to be as a result of its speed somehow helping it when it’s being used in certain environments (ex. because it’s faster, it’s better at iterating through an Excel file within a time constraint).

In my general use so far in chat contexts, I don’t find a major difference between Sonnet 4.6 and Opus 4.6, and I don’t plan to use it in coding contexts because I like to use the smartest coding models available to me.

So, there you have it — that’s Sonnet 4.6.

Superbench

Some of you might know that I run a personal model benchmark. I send 60%+ of my prompts to multiple LLMs in their chat applications, and then stack rank the responses. I’m biased, but I think it’s the best AI benchmark on earth.

We don’t have enough data yet for Grok 4.2 or Sonnet 4.6, but I don’t expect either model to disrupt the current status quo as of February 17.

Speaking of February 17 — it’s my birthday! As a gift, it’d be incredible if you forwarded this to AI-curious or AI-nerd friends in your life, or shared on socials:

Share

Otherwise, happy Tuesday — stay frosty out there.

Best,
Sherveen

OpenAI released an update to their Deep Research feature last week (now fueled by GPT-5.2). So, I thought it’d be a good time to begin a new AI showdown series: which AI deep research product is the best right now?

I ran the same set of queries against mixed sets of 9 products from:

• Anthropic (Opus 4.6 with Research)

• Google (Deep Research w/ Gemini 3 Pro)

• OpenAI (ChatGPT Deep Research w/ GPT-5.2)

• Wild cards from Perplexity (Deep Research), Manus (1.6 Max), Moonshot AI (Kimi 2.5 DR/Agent), Z[dot]ai (GLM-5 Agent), and MiniMax (M2.5 Agent).

As we go, I’ll provide links to the full chat response for each result.

Reminder: deep research (DR) is an agentic mode available in pretty much every state-of-the-art AI chat app. It focuses on intensive exploration and discovery about a topic through hyper-extensive web searches, fetching of data + primary sources with citations, and planning and reasoning to empower relevant results.

This could be for a social or scientific studies question, deep product discovery and comparison, or business and market research.

Where a model-maker doesn’t have a DR product available, I’ll use their agent modes. If a result was plainly not worth talking about, I’ll exclude that model from discussion.

And for ChatGPT, we’ll include Deep Research and 5.2 Pro. While 5.2 Pro isn’t a dedicated research product, it’s a highly agentic, long-inference chat model available on OpenAI’s $200/month tier. It does intensive research while still being more interpretive and conversational, so we’ll see how it does against DR pipelines!

One caveat: I’m not expert on most of the domains down below. I am using a mixture of context clues and source reading to validate that the responses aren’t blatantly wrong. Wrongness in deep research pipelines is a nuanced topic for a different day, and generally solvable within the same product harness, so… as unintuitive as it might sound, it’s somewhat a side topic when it comes to today’s comparisons.

Let’s dive in.

Test 1: Asking a broad question

This is the kind of question we often ask LLMs: we want a conclusion, but we want that conclusion to be well-evidenced, too.

I’ve long been curious about what seems like Starlink’s very long lead in the satellite telecom and internet market. It seems like a very dubious thing to have one company hold so much necessary capacity for the world.

Can you do a deep exploration of the market -- emerging competitors, nearest in-market alternatives, differences in capability and feature sets, and the nuances throughout? Would love an analysis of this market and what it will look like over the next few years.

Here is the chain-of-thought I had analyzing the results:

• Perplexity, Kimi, and MiniMax all suffered the same issue: they cite a lot of stats and give you a lot of facts, but they’re meandering and tend to over-rely on secondary sources (like third party blog posts).

• GLM-5 is the first strong response. We get hits of everything important: from details on Starlink’s products to a good overview of its competitors, and the geopolitical + strategic dynamics playing out. But – it reads like a textbook.

• Gemini’s DR is very ‘consultant’ coded. Not a bad thing! It’s a structured document with a lot of framing and definitions, plus generated graphics and charts that are hit-and-miss (below).

  • Here, you’ll feel that Gemini’s deep research mode always struggles between interpreting the user prompt versus following the system’s instructions. In the response, we see it say: “The user’s query regarding ‘dubious capacity’ touches on a future risk: Oversupply.” In practice, this means it’ll often refrain from its own synthesis or conclusion-drawing.

• Claude’s result is by far the most readable, which will be a recurring theme. It has a great writing pace and tone, and reads the most like someone’s Substack.

  • It’s also the most opinionated. Not in a big way, but it’s more likely than the others to highlight conclusions that it deems important to notice.

  • Example (below) — compare how we learn about legacy satellite player Eutelsat OneWeb in Gemini (left) versus Claude (right).

• ChatGPT’s DR is a bit of a mix of Gemini and Claude. It’s far more thorough than Claude, almost as willing to draw conclusions, but less readable. It has Gemini-like qualities in its structure: like a consultant wrote it, framing the problem at the top, diving into a long comparison of markets and features, and closing with opinionated forecasts — with several generated tables along the way.

• ChatGPT’s 5.2 Pro does diligent research, just like everyone else. The result, however, reads far more like a conversational LLM.

  • In fact, the response begins with immediate synthesis: defining four overlapping categories in the market race that it then uses to frame the rest of the research. This response gets very specific as to where Starlink is today, why it’s ahead, and on which vectors it’s most vulnerable.

🏆 Winner: GPT-5.2 Pro. While Claude’s opinionated readability is easy on the eyes and ChatGPT’s DR provides an analyst’s flair, 5.2 Pro does still-thorough research while really providing framing and context to the query.

Test 2: Asking about modern science

This type of question gets at two nuances: (1) how good are these researchers at retrieving hyper-recent results and (2) how good are they at understanding what qualifies as scientifically ‘worthy’? Our prompt:

I’d be curious to learn all about the recent scientific progress being made re: male pattern baldness. What are the recent promising findings, studies, experiments, tests, etc. worth knowing about?

Here is the chain-of-thought I had analyzing the results:

• Perplexity is fine. It’s not wrong (from what I can tell), but I can’t click on inline sources, there isn’t a lot of progressive claim building, and it feels like a bulleted list I have to vet myself.

• Claude feels like a fast-talking expert. It’s well-cited and does good work framing the progress of the science. We learn about promising drugs, RNA and gene techniques in early development, and cell therapy techniques gaining traction in Asia. But it’s definitely a dense read meant for someone who wants max science.

• Gemini’s answer feels like that of an educator-scientist. We start with a great diagram of a hair follicle. As we learn about new medications and interventions, Gemini begins each section with an explanation of the base science (below). Gemini is the only model to cite TissUse, a unique “smart organ-on-chip” technology, but it’s also the only model to miss on VDPHL01, a seemingly important evolution of oral minoxidil.

• ChatGPT DR is the scientist’s scientist. OpenAI’s products, as always, are diligent at web search, using multiple sources to validate and verify a conclusion. The language has the most technical density, which means this winds up being the least layperson-readable of the 3 results.

  • However, there’s a section where the response suddenly anchors to the user prompt more tightly, and we get practical takeaways as a result. “The sections below follow your requested format: mechanism, key evidence (2020–present emphasis), trial phase and endpoints/effect sizes when available, limitations, and an estimate of timeline-to-impact.” (below)

  • Perhaps due to adherence to my prompt, it spends the least amount of time detailing therapies and interventions that are still 5+ years away. It names them, but it doesn’t spend as much time on them.

• ChatGPT’s 5.2 Pro response is thorough and highly readable, hitting on every relevant and near/mid-term trial and drug, and ending with the upcoming data and studies to watch for.

🏆 Winner: GPT-5.2 Pro. Once more, it did research as thoroughly as the dedicated DR products, but had the sort of framing and readability that maximizes learning. It’s DR with bedside manners.

Test 3: Asking about influencer science

I just pressed play on the episode, but I’m already intrigued by an initial claim in the first 30second teaser of this podcast -- Dr. Michael Breus was just on the Diary of a CEO, and he says there are four sleep chronotypes that dictate not just when it’s best for you to sleep, but also when it might be best for you to have coffee or to learn complicated concepts. He says there are three that are widely recognized, and he’s kind of unraveling a fourth. What’s the real science here?

All tested models did an excellent job in retrieving studies and primary sources.

• Claude was quick and readable: Breus’s general framing is generally not peer-reviewed, and his novel addition to the existing and validated sleep chronotype framework is probably not a real thing.

• Gemini’s deep research is a bit friendlier to Dr. Breus, suggesting that his fourth chronotype may be a thing related to a validated “hyperarousal model of insomnia,” but like Claude, it’s not sure if this should be a chronotype or if it’s a disorder.

• Gemini is also friendlier to its finding that Dr. Breus recommends a 90 minute delay for morning caffeine. Claude also finds this in its research and points out that Dr. Breus is relying on a mechanistic effect that doesn’t have empirical validation (aka studies saying it actually works).

I don’t like how unopinionated Gemini is being here – even though its response is far more thorough than Claude’s in terms of education, analogies, examples, and practical descriptions. At the core of my question… is the science right, or isn’t it!?

I could just ask the LLM’s in their normal mode if I’m looking for more interpretation, but remember that the LLMs in their normal mode are much worse at doing exhaustive research, and so I’d lose the benefit of their fact-finding.

• ChatGPT’s DR is the platonic ideal of deep research here. It doesn’t treat the reader with kid gloves. Instead, it does sequential building of information to equip the reader with deep knowledge by the end.

  • Beyond looking at the science, it looks up multiple press hits from Dr. Breus over the past decade, including interviews from 2016 where he first proposed his fourth chronotype. And that leads to a useful conclusion: “Given the marketing context (quiz plus product ecosystem) and narrative style, the most defensible characterization is that the Breus framework is primarily a popular synthesis + coaching heuristic, potentially informed by clinical experience, rather than a published, independently replicable empirical typology.”

• ChatGPT’s 5.2 Pro is a great “walkaway skim” version of the other three responses – but the other three are meaningfully more in-depth this time.

🏆 Winner: ChatGPT Deep Research. We have the right mix of the right research, thoroughly, with the right takeaways. Again, we want our DR pipelines to be thorough — but it’s still a combination of receipts, teaching, and willingness to “land the plane” when it comes to the original prompt.

Test 4: Asking about the numbers

Certain people in tech lie about college admissions numbers to feed political narratives — it’s pervasive and malicious. So, I asked the different research modes to help me find the data to combat those lies. Prompt (excerpt):

I need a comprehensive, well-cited breakdown of international versus domestic enrollment at top US universities, split by year and by level. We may need to search institutional archives, fact books, or registrar reports. Schools: Harvard, Stanford, MIT, Yale, Columbia, University of Chicago. Let’s grab: current international student % at each school, sub split by 1974-1975, 1994-1995, and 2023-2024 (or nearest years where we can find reliable data), sub split in those zones by undergrad versus grad.

This is a challenging query because it’s not just about deep digging and finding of primary sources. Not all of the data will be readily available or printed on a website. Instead, the models will have to extract specific numbers from specific years at different colleges.

To do this successfully, the agents will have to plan a mode of research that hits different cohorts of data, dig through archival documents and PDFs, find alternate sources after running into roadblocks, and adjust along the way.

• ChatGPT DR struggled here. Although it’s a thorough web crawler, it isn’t dynamic enough (perhaps not even enabled) to download relevant files, extract information using code or vision, and use complex interfaces.

• GPT-5.2 Pro was a little better, but surprisingly, it wasn’t as agentic as what I believe was o1-preview when I asked this same question last year.

• Claude made a far more robust attempt, especially after a second encouraging query. By the end, we got a useful Excel sheet with confidence intervals per stat based on the quality of the origin data. I think we’re seeing Anthropic’s focus on file-handling come into play here, enabling better ingestion of docs during research and the production of new artifacts as part of the response.

• Perplexity looks interesting on the surface, until you dig in and notice that it’s mostly secondary sources or estimations of data.

• Both Gemini and Manus found a lot of adjacent, disconnected data that ultimately didn’t round up well into a cohesive view of the situation.

• The dark horse here: Kimi 2.5 in Agent Swarm mode. This allowed the main Kimi agent to spin up several parallel subagents to perform per-school research (below). As rounds of subagents found more info or hit new roadblocks, it would spin up new subagents to retry places where the research failed. Ultimately, we received the most comprehensive set of files with the most data, and where it couldn’t find precise data, it found its nearest neighbor and noted it.

🏆 Winner: Kimi 2.5 Agent Swarm. Multi-agent systems are probably at play in the background with several of the AI products we use today, but Kimi’s product most explicitly uses that architecture today. By running many agents in parallel, each is less likely to get exhausted or move on to do another task, and we can see the value of that atomic focus within these results.

Test 5: Asking for niche product research

Repo Prompt is a tool for gathering context and code into a neat “package” to transfer to outside AI agents for advice (basically, ‘take these large files and reorganize them so I can paste them somewhere else’). Which of these DR products will be best at finding alternatives – especially given that I didn’t define Repo Prompt in my query, and it’s a relatively modern tool?

What are all of the alternative tools and products to Repo Prompt? Let’s be comprehensive and thorough, finding even newer/emerging startups and open source projects. Thank you! <3

• Perplexity and MiniMax just gave us big lists with no real interpretation.

• Gemini was, as always, consultant-y and educational. But it got way too intellectual about the prompt, highlighting philosophical approaches that startups and coding tools in general are taking to the ‘code context packing’ problem. Really not what we were looking for!

• Claude looked through 529 sources and curated a tight list with very good summarizing descriptions alongside each tool.

• ChatGPT’s DR provided a very thorough list, complete with tables highlighting features, differentiation, and a comparison matrix. There’s enough information along the way for a reader to select a few to try across spikey categories.

• GPT-5.2 Pro created a smart bucketing of product categories and added one-liners to each, but lacked the usual commentary I appreciate the Pro model for providing.

To be fair to the “they just gave me a list” answers above… that is what I asked for.

🏆 Winner: ChatGPT Deep Research. It found the most literal answers while still providing thorough comparisons and relative descriptions. In other words, we can walk away feeling it was comprehensive and dense-but-still-actionable.

Winner, winner, chicken dinner

If you’re trying to figure out where to spend your subscription money or time, there’s a clear pair of winners depending on how literally you take the category: ChatGPT Deep Research or GPT-5.2 Pro.

But this experiment validated something more important for me: having multiple subscriptions. In my regular AI-using life, I send a majority of my queries to multiple LLMs, and I can’t imagine not getting the different flavors of answer that exist even across our samples above.

Because I appreciate and value Claude’s spunky writing and willingness to really address the main question, even if it’s in research mode, and I find it most willing to use its research to help me out with a ready-made conclusion.

And I appreciate Gemini’s thoroughness and educational style. It almost strips away your prompt and comes up with a “normalized” query that removes any opinion-having at all in favor of consultant/textbook-style rigor.

And it’s really useful to toss Kimi’s Agent Swarm mode at a problem that requires brute-force compute power and subagents to retrieve really specific data, with an orchestrating agent coordinating so that I can look away.

But there is a winner here and it shouldn’t surprise any power user: OpenAI’s models, as always, are supreme at using the web.

They are the most agentic, given the longest leash to scour for sources, and act with real agency along the way. I’ll cover this in more depth in a future piece, but if you look at the reasoning traces in the chat logs above, you’ll notice both GPT-5.2 DR and 5.2 Pro reckoning with the information they find — using it to dynamically decide what else they should know, what else might be important, and how to change or execute on their plans accordingly.

In other words, they use the web how I use the web.

If it’s part of your budget to subscribe to the Pro plan, you should always run both. You’ll appreciate 5.2 Pro for giving you an extra layer of framing and conversation that you’ll miss when using any of the pure DR products.

If you’re looking to know which generally-accessible Deep Research mode is best amongst the foundational chat applications, ChatGPT is your winner. 🏆

For now, OpenAI sits atop the DR pile. But updates to this kind of harness product can come fast and furious, so come on back soon — I’ll make this a ~monthly check-in for us to stay researched on as we go. :)

Subscribe now

This is part of the Breaking the Framework series, where we talk about using AI to completely shift how we get a particular job done.

Slides: What are they, really?

The play: treat slides like a micro-website. Use a coding agent to build your slides using web frameworks, so your slides become reusable components + a theme you can change in seconds, then present from the browser or export a PDF.

I make a lot of slide decks — typically, for workshops or coaching material. I’m one of those finicky types: beyond just getting the layout right, I’ll do things like create individual slides for each bullet on a page so I can make one point at a time.

I’ll also obsess over editing to make sure that paragraphs don’t overflow to the next line by just one word, or use Figma to superimpose call-out graphics that match modern design.

But all of that is quite painful. Each time I create or update a presentation, I know I’ll have to sit there tweaking it as part of my process. And each of PowerPoint, Google Slides, Figma, and Canva have their own quirks that make something difficult to do in their particular interface or format.

I’m also not a fan of any of the AI apps that exist in the space today. Gamma is convenient and popular, but they look like AI generated slides.

GenSpark, Manus, or even Claude can create decent looking decks using dedicated slide features or by creating PowerPoints. But you’ll only want to use them if you have no design taste and love super-dense layouts.

And I know a lot of people have started using Google’s image model, Nano Banana, since it’s very good at embedding text in images now. However, that’s a very “slides-by-painting” method that has a lot of its own impracticality.

This is where we break out of prior frameworks: what are slides, really, if not assemblages of layout and content in a particular order, with a particular set of styles?

You know what else = assemblages of layout and content in a particular order, with a particular set of styles? The web.

You know what AI agents are absolutely excelling at lately? Web development.

What I’m doing, and nuances

Once I had the realization that I could just ask an agent to collaboratively build web pages with me, having it write code that would impose structure and design, I went to ChatGPT, Gemini, and Claude to ask what the best tech stack would be to do something like this.

You don’t need to know anything about writing code to do this, you just need the right advice from your smartest reasoning AI to steer your favorite AI agent.

The answer: build in React with reveal.js, an open source HTML presentation framework. This would allow any coding agent to use traditional code to construct slides, plus come with an easy presentation mode and an export to PDF feature.

I then went to Claude Code (CC), which is slightly better than Codex CLI right now when it comes to design nuances. You could also use Claude Cowork or Cursor. I started with the below prompt:

Within a minute, we had the initial slide running:

Then, I pasted a slide from a recent webinar and asked CC to try to duplicate the style. This took a few rounds of feedback from me, but eventually, we got to a really nice place:

As CC began to build out individual slides (pasting inspiration), the other thing it was building: a consistent set of components, themes, and interface types that we could continue to use as the underpinnings of our slides. And I’m just prompting!

Remember, this is all now code — deterministic, modifiable, calculatable code. More flexible than dedicated slide software, more controllable than image generation proxies.

Stretch tactic: at some point, I wanted CC to be able to see its own changes so it could self-iterate without my intervention, so I added the Chrome DevTools MCP (I’m generally biased against MCPs for reasons I won’t get into here, but generally: prefer CLIs). This enables CC to open an instance of Chrome and take screenshots of the page as it works.

The advantages I now have:

• New slide? I don’t have to sit there and type content into fidgety text boxes on a canvas, I just tell CC the content and it constructs the layout.

• Need a new slide layout? I paste the slide content and ask CC for 3 ideas on layouts that will legibly demonstrate the point, and it thinks through layouts.

• Need to build progressive slides where certain elements appear or move on screen? I don’t need to duplicate and move things around — I ask CC, and in seconds, it spins up the relevant sequence.

• Update content? Just tell CC the copy change, it’s done! Change slide colors or fonts? Just ask CC to try things! Need to import an old deck? Just paste it into CC, it’ll generate all of your slides in your new template in minutes!

• Bonus: if you understand git (ask your favorite LLM), you can now have version control on your slides, too!

Fast, easy, no need to mess with a canvas, with complete flexibility in design.

And because we’re using the pre-existing reveal.js framework (that I had no idea about prior to this project), I can either present from the project by running it from my machine, or export the slides to PDF once they’re final.

So, let’s recap:

• Slides can be made of code

• Agents are great at code

• Therefore, you get speed + consistency + control

• Gaining orchestration leverage (“I delegate or yap at AI agents”) so we no longer have to sit in primitives like Google Slides or PowerPoint

Now that’s some good AI muscle.
Alrighty, that’s all for now —

Sliding out until next time,
Sherveen

Last week was jammed with progress in under-the-radar areas of AI. This week, we’re expecting lots of AI announcements, headlined by Google’s (rumored) release of Gemini 3.0 Pro.

So, let’s get last week out of the way with 3 things that you might’ve missed but are worth paying attention to in the themes of… humans with AI, AI inside AI, and humans versus AI.

1: Anthropic demonstrates what it really means to be AI-enabled.

Anthropic divided 8 researchers into 2 teams. Both were tasked with programming a robotic dog (neither team had any robotics expertise). One was given access to Claude, the other was not.

The video is worth watching in full (seriously, watch it), but here’s the TLDR:

• The team with Claude completed the sum of tasks in about half the time compared to the team without (or, as Anthropic calls them, Claude-less).

• Team Claude completed one more task than Team Claude-less in the final phase of the project, though neither team completed all 8 tasks.

• In some tasks where Team Claude was slower than Team Claude-less, it’s because Claude helped them do the task better (example: Team Claude had streaming video from the robodog’s camera, whereas Claude-less had ‘intermittently-sent still images’).

• Team Claude wrote 9x more code -- now, not all of that code was used to ‘finish’ tasks, but as Anthropic put it: “Having the help of an AI assistant made it easier to fan out, try a lot of approaches in parallel, and write better programs—but also made it easier to explore (or get distracted by) side quests.”

• Anthropic recorded and transcribed both teams during the experiment, and had Claude analyze the transcripts for sentiment analysis. Team Claude-less expressed confusion (questions or exasperations) at twice the rate of Team Claude.

I have so much more to say about this. I believe this was one of the first experiments to neatly describe the differential between what it looks like to be AI-enabled versus not. The ‘whole’ of work changes beyond any one metric: double the speed, up the quality, with less confusion and more ‘exploration’ bandwidth.

And this applies to all professions, not just those that are code-oriented.

I’ll write more about this soon. In the meantime, their full blog post is here.

2: Google’s AI agents are learning how to play our video games, & fast

I’ve been fascinated by Google DeepMind’s Scalable Instructable Multiworld Agent, or SIMA, ever since Google first announced it last year. It’s a generalist AI agent crafted to be capable of navigating and following instructions within virtual environments.

With a little bit of basic skills training across a few games, SIMA could be dropped into a virtual world (ex. No Man’s Sky) and use a virtualized keyboard and mouse to carry out short (10-seconds-at-a-time) instructions.

Last week, they unveiled SIMA 2. They put Gemini at the core of the SIMA agent, giving it new reasoning capabilities. As Google puts it, SIMA 2 “can now also think about its goals, converse with users, and improve itself over time.”

Once more, I’ll encourage you to scroll the blog post and watch a few of the clips.

In it, you’ll see a human user give SIMA 2 broad instructions (like ‘go look at those minerals over there and tell me what they might be’), and the agent will reason over the goal and take multi-step action to move & interact in a video game.

Further, it’s ‘generalizing’ at an increasing rate -- taking concepts or mechanics it learns in one game and applying it to another, even in games that it hasn’t seen before.

And they’re now dropping it into Genie 3, their state-of-the-art world model that generates and simulates dynamic ‘worlds’ and 3D environments in real-time. In other words, a self-learning embodied agent can navigate a self-fulfilling new world.

The implications are endless, but I’ll leave you with just one: agents training themselves in self-generating world models.

For real-world AI robots to get really good, we need more training data -- we lack the scale of usable videos today to get general-purpose or unsupervised robots to be fully autonomous amongst economically important tasks.

We can try to get more of that data in the real world, which many companies are doing. But we can also use a world model like Genie 3 to emulate the real world and all of the physical properties of, say, a car factory. Then, we drop in SIMA 2, which has the ability to act upon that world and learn from that world’s interactions and feedback, improving on fine motor function, workflows, and task completion.

With that, we’re creating valuable synthetic data of an agent in a car factory. These kinds of simulations can be used to rapidly train models moving forward.

Google’s Genie and SIMA projects have secretly been the coolest things in the world of AI for over a year now. Keep an eye out.

3: Zapier gets earnest about their AI recruiter

We’ve been seeing a meteoric rise in AI being used in interview contexts (and by job seekers) over the past two years, but a blog post last week from Zapier was the first that I’ve seen from a company trying to explain why their AI recruiter might be good for everyone involved.

A few choice quotes…

On the state of job search and recruitment:

• “If you apply to Zapier, you may be invited to a recruiter screen with an AI agent. We want you to know why.”

• “Job seekers are increasingly using AI to write their resumes and applications—and to send many more applications. On one hand, candidates can highlight skills more effectively. On the other hand, recruiters now face a flood of submissions that look strong on paper but often don’t hold up in practice.”

• “On top of applications per job growing beyond what we can manage conventionally, we’re finding that up to 30% of applications are fraudulent. We’ve witnessed fake identities, unverifiable credentials, and misleading profiles. We even caught some deepfakes on live interviews!”

• “To address these challenges, we’re going to start our experiment to pilot agentic recruiter screens in the coming months.”

On their new, AI-infused process:

• “After an initial application review by a member of our team, significantly more candidates can now move forward to a 15–20 minute AI-led screening call.”

• “The AI recruiter asks the same structured questions our human recruiters would, with smart follow-ups tailored to our criteria. Candidates can complete their interview at their convenience, making interviewing with Zapier more flexible and accessible.”

• “Afterward, AI helps summarize responses against our rubric, and a human Zapier recruiter reviews the notes, transcript, and recording—alongside your application. That same human recruiter makes the final decision on whether to move the candidate forward.”

• “… we believe there are real benefits to participating: A chance to tell your story—because we’re not limited to the handful who look ‘perfect’ on paper. Flexibility to schedule on your own terms and in your time zone.”

• “Most importantly: AI does not make hiring decisions at Zapier. Our recruiters and hiring managers do.”

As a lot of you know, the area of job search and talent matching has been my obsession for well over a decade now. I’m not sure what job search will look like over the next 1, 3, 5+ years -- but I do think they’re mostly right that AI at the top of the funnel could be beneficial to both sides of the equation.

And I’m glad to see them talk about it out loud. We need more of that right now.

Okay, we did it. Three heavy hitters out of the way to start your Monday.

If you learned from the ride, forward it to a friend. :)

Prompt ya later,
Sherveen

I took an accidental hiatus on these emails -- US politics is distracting like that lately – but expect me to be more present in your inbox again. 3 stories worth paying attention to in this moment:

First, for all my doctor homies in the audience –

An early study out of Yale School of Medicine tracked 263 physicians and practitioners using across 6 healthcare systems over 30 days. Some were paired with Abridge, an AI platform for clinical note documentation.

“[When paired with] an ambient AI scribe, burnout among those working in ambulatory clinics decreased significantly from 51.9% to 38.8%. There were also significant improvements in the cognitive task load, time spent documenting after hours, focused attention on patients, and urgent access to care.”

One of the more pervasive ways in which AI will become essential over the next 12 to 36 months: helping people elevate their time spent to the more meaningful parts of their job and life.

(the study’s authors say that Abridge had no role in the design and conduct of the study or analysis of the results, beyond assistance in data collection)

Second, your security camera wants your video data –

Fascinating story being reported by TechCrunch -- Anker, the Chinese company behind the popular Eufy brand of security cameras, recently offered customers money in exchange for videos to train AI systems.

For $2 per video, Anker got rich video data to improve its security detection systems in a somewhat positive feedback loop. Eufy has said “the data collected from these staged events is used solely for training our Al algorithms and not for any other purposes.”

But most amusingly -- they don’t mind if you stage the video to fit their needs. They want real package and car thefts, but if you fake it, that works for them, too.

“Don’t worry, you can even create events by pretending to be a thief and donate those events. You can complete this quickly. Maybe one act can be captured by your two outdoor cameras simultaneously, making it efficient and easy. If you also stage a car door theft, you might earn $80.”

Data is oil in the AI era, so this makes sense at a high level. The more raw video they have of different incidents, driveways, patios, and sidewalks, the better for their models. It’s the same reason OpenAI wanted to pay $500 million to acquire a video game clipping company.

Beyond being a little dystopian, it’s also a tad concerning that staged data could be used for such important algorithms. Like… do fake robbers really act the same as real robbers?

A question for another day, I suppose…

Third, OpenAI held their conference for nerds –

At OpenAI’s third annual DevDay conference for developers, the company launched:

• Third party apps inside ChatGPT (ex. Canva, Zillow, Spotify)

• AgentKit to help developers build AI agents, plus Codex SDK

• GPT-5 Pro (my favorite), Sora 2 (& Pro) made available via API

There are a few different themes here that deserve a more thorough analysis, both for developers and end-users, so I’m going to save that for another day.

In the meantime, I’ll register this as my complaint that OpenAI didn’t do as swell of a job as I’d hoped in helping people understand the difference between AI assistants and AI agents (my rant in video form here). I will continue to wage this war alone. Alas!

Alright, that’s all for now –

Stay bald,
Sherveen

We’re going to the AI gym! It’s time for our first Sunday Rep: turn text you already wrote into a diagram in just a few minutes.

Every week, I’ll share a tool or method that you should try.

Something important to note: don’t overdo it. It’s more important that you try a Sunday Rep, even if your temptation is to make a big project out of it or wait for the perfect moment. This era of AI is moving too fast, so it’s more important to use the tool or method, learn about what's possible & what’s changing, and move forward!

I try every tool I see for at least one “turn” -- but, 99% of them? I never return again! That’s okay. Embrace the drive-by try.

(btw, I’ll almost never have any financial relationship w/ the companies in question -- they’re just great demonstrations of what’s new -- I’ll let you know if there’s ever a mixing of interests.)

Okay, all of that in mind --

Sunday Rep 001: try out Napkin AI (free tier will be enough). Napkin lets you quickly turn text into visuals -- whether that be a diagram, a chart, or a funnel.

Why is this one to try:

• Tools like Napkin are what I call “structured output tools.” They sit between raw LLMs (like those we use inside ChatGPT, Claude, etc.) and a ‘design suite.’ They take things like text and map them into pre-built visual ‘grammar’ to make high quality results that are consistent and editable.

• Tools like ChatGPT and Claude aren’t always great at producing visuals and infographics. That’s because they can’t really “see” their output as they make it, so you’ll usually get something rough and unusable.

• Napkin, and others like it, surround an AI model with context and ‘tools’ to use a pre-baked and pre-built ‘application' -- this is their main innovation.

  • So, it’s trained to go… “okay, the user gave me bullets that belong in a flow… Napkin built a ‘flow’ module I can use… I see there are 5 types of pre-built flows… based on the info from the user, this particular flowchart is best.”

  • And then it calls on Napkin’s API -- for those of you who aren’t technical, think of this like a ‘pipe’ to Napkin’s core functionality -- to actually produce the visual. It’s basically going, “Napkin, put down a funnel please, make it this size, and put this information in section 1, this in section 2, etc.”

  • And since Napkin pre-built all of the visual ‘containers,’ they’re just asking the AI to help figure out which container is best for the use case, and the order and layout of that content.

So, here’s what to try:

• Head into Napkin with meeting notes, some data, or some made up workflow.

• Paste it into Napkin → select your relevant text → press the ‘Generate Visual’ button that’ll show up next to it. Scroll through the recommended options!

• Try editing the labels, using different visualizations, exporting, etc.

Another tool in this vein: Gamma, which does it for slide decks. The decks are ugly, but they (or someone else) will figure that out eventually.

• Pro-tip: these tools will often offer to generate the text content of the slides or graphics for you, too. Don’t!

  • First, you’re probably still better off writing all of your content with AI as a collaborator, rather than letting AI write anything for you (I’ll talk more about this in coming weeks).

  • Second, they’re often using far weaker, dumber models than what you get in ChatGPT, Claude, or Gemini. So, write on your own first (in collaboration with your favorite AI as a brainstorm partner and editor), and then bring it to a “AI tools for structured output” tool. :)

OK, that’s all for now!
Off to fight with fascist venture capitalists on Twitter. Wish me luck.

Balding by the minute,
Sherveen

In any existing or new chat, and on any message from ChatGPT, press the three dot menu and you'll see "branch in new chat." It will duplicate the conversation history (up to and including the message you selected) in a separate tab. Now, you've forked two branches to keep on using!

I'd bet 99.9% of people won't use this feature. Let's be part of the 0.1%.

3 uses that build on each other: hold trunks, A/B test, and use checkpoints!

1 - Hold on to a 'trunk' conversation

Or: A new way to hoard browser tabs and bookmarks

For the past few months, any time there's new data on inflation or jobs, I've been feeding it to GPT-5 Pro and asking it what it would do if it were Jerome Powell -- increase Fed rates, decrease, or hold steady?

I keep going back to the same conversation because it already has all the juicy progress -- past data, past analysis it did, etc. It's accumulating context!

But... I never really ask smaller questions or deviate from the main topic inside that chat because I don't want to "pollute" the context window.

In other words, if I suddenly had too long of a conversation with it about how we could change measurement of unemployment in the US, by the time I came back with the next jobs report, it'd have to "re-orient." We went on a tangent, and the relevant context is pushed further back in conversation history. This is context drift.

Well, this morning, I went back to my trunk and fed it the latest job numbers. Then, I branched a separate conversation to talk about unemployment measurement.

Boom, I still get to go back to the original copy -- the "trunk conversation" -- whenever I want to, but I can spawn as many of these isolated sub-threads as I want, and I'm essentially bringing along a clean "pre-prompt" of the accumulated conversation so far.

• Pro-tip #1: bookmark trunks in your browser if you expect to go back to them often, and/or rename the chats from the sidebar with a [TRUNK] label!

2 - Run parallel, isolated A/B conversations

Or: Two paths diverged in a yellow wood, and I could travel both

Let's imagine you're a marketing manager at Uber talking to ChatGPT about the launch of a new safety feature. You're 7 or 8 turns into the conversation talking about the product and your budget -- you're ready to talk about creative strategy and positioning.

But you know your messaging always has two audiences: the driver and the rider. And it's the constant challenge in your job that they are almost never aligned, either in their incentives or instinctual reactions to new announcements.

You could ask ChatGPT to help you with both in that conversation, either at the same time or one after the other. But if you're really trying to maximize the individual consideration for each population, it isn't ideal.

If you talk to ChatGPT about drivers first and come up with a campaign that tells them this is about their safety, then talk in that same chat window about riders, there'll be a lot about driver safety as the conversation and context history.

That isn't always a bad thing, but in this case, it means you aren't maximizing the appeal of the message to two very distinct audiences.

Instead, take your trunk context and split it into two chats. Talk about drivers in one -- "let's optimize messaging and strategy purely for drivers," and riders in the other. Boom: two conversations optimized entirely for each audience, without even a slight penalty for mixing topics and incentives.

• Pro-tip #2: after you've had two parallel conversations → bring one back to the other (or start a third) → ask for a consolidation!

  • "Hey, here's what we've come up with for riders -- now, let's talk about the umbrella campaign, and messaging we should synthesize between riders and drivers."

• Pro-tip #3: treat parallel conversations like a Team of Rivals (where is Doris Kearns Goodwin nowadays?) -- if you're seeking something like career advice or dealing with a hard scenario, start the conversation with your initial context, then have a panel of branches that take on different personas (act as my mentor, act as my therapist, etc.) to give you different flavors of advice.

3 - Use branches to restore prior checkpoints

Or: How I learned to stop worrying and love version control

Look, I'm not gonna lie to you. I've been saving the best for last. Engineers, you know where this is going.

Let's say you're talking to ChatGPT about some business data. You're 30 minutes in, and you suddenly realize… you mistyped some numbers halfway through!

You could just tell ChatGPT about the correct data, but it’d have to recalculate a bunch of numbers, would struggle to know what's what, and you're in for a headache.

You could edit the message with the bad data and resend it, but that would delete all of the conversation that comes after it. This would be rewriting the path moving forward, which isn't great -- because even though some stuff is wrong, you and your AI friend have had some rad ideas you don't want to lose or stop talking about.

Branching the checkpoint allows you to instead preserve both the "infected" path and have a clean restart with a partial trunk with only the reliable context. Magic!

• Warning #1: you might think… “I do this already, I just copy paste conversations into a new window when I need to fix something!” For reasons I’ll explain in a future newsletter, don’t do this unless you have to – branches are a far better solution.

• Pro-tip #5: be like Marty McFly and go Back to the Future – when you’ve restored a previous checkpoint in a long conversation to correct some misinfo, you don’t have to re-have all of the same conversation. Your “infected” chat presumably had some good stuff – context, new ideas, etc. Mention all of that in your next message! Fast-forward your progress back to where you were.

  • Here’s what made this click for my Chief of Staff, Katie:

    • ok so we have a chat with chatgpt

    • we go back and forth 9 times

    • we made an error at msg 4

    • so we branch at msg 3 to remove the error

    • but msg 7 and 8 had some good ideas

    • so if we’re the user

    • copy paste those good ideas

    • into the new fork

    • because it only has msgs 1 to 3

    • so bring along the good progress

One quick note -- don't branch when you've got compounding work:

• When diverse information being included in a chat gives you compounding benefits, don't branch -- stay in it! (Unless you're an engineer going back and forth w/ code, that's nuanced.)

• As an example, ChatGPT benefits from seeing you react to ideas if you're in a brainstorm -- unless you're trying to Men-in-Black it and erase its memory for a reason, letting it see its past ideas and your feedback = better next set of ideas.

Alright, that's all for now -- gotta make like a tree and branch off into doing something else. I'll see you on Sunday, when I'll send everyone something they might want to try to build their AI muscle -- because AI is still awesome on the weekends.

Enjoyed this one? Throw this branch at a friend —

Share

Yours, forever and always,
Sherveen

The truth is, while I do love ChatGPT, I’m not just trying to butter it up. In fact, I take my end-of-message whispers very seriously!

To me, it’s research and investigation into a concept we should all be paying more close attention to: AI prompt sensitivity. It’s how much a model’s behavior shifts in reaction to changes in our prompts, even when the underlying meaning is the same.

Let’s dig into 3 fun and illustrative examples of prompt sensitivity -- priming, constraints, and adherence.

Priming: can poetry solve chess?

My favorite example of how sensitive AI can be to our prompting comes from journalist Kelsey Piper. Back in April, Kelsey wrote about her personal benchmarks for measuring LLMs on complex reasoning. Here’s her description of how she tested new model releases:

I post a complex midgame chessboard and ‘mate in one’. The chessboard does not have a mate in one. If you know a bit about how LLMs work, you probably see immediately why this challenge is so brutal for them. They’re trained on tons of chess puzzles, [all of which], if labelled ‘mate in one’, has a mate in one.

As a result, even AIs that generally solve chess puzzles very capably [will] check over, and over, and over for the checkmate that they’ve unquestionably accepted is there. Eventually after 1000s of tests they hallucinate a solution.

Super interesting! But here’s where it gets fun… at the time, OpenAI’s o4-mini-high was the first model to pass Kelsey’s tests, except Claude 3.7.

But Claude 3.7 would only pass under a very specific condition: you have to first give the model this blog post, which can best be understood as unrelated metaphorical poetry about drugs.

The blog post has nothing to do with chess, or these chess puzzles!

Predictably, people were confused:

But that’s the magic of LLMs being sensitive to our prompts. They’re reacting to our inputs. The blog post scrambled the LLM’s “compass,” its sense of what to pay attention to. It still found its way back to the chess puzzle, but the injection of more context changed the probability distribution of all the possible ‘next words.’

It kind of goes like this:

• user: here’s a blog post, and a chess puzzle. solve the puzzle.

• model: okay, so you want to know about this puzzle. but you also opened this (metaphorical) browser tab, interesting. oh, fun blog post! no idea what that was about though. back to the puzzle…

Imagine you in that scenario, maybe back in college and doing some homework, but you accidentally open an unrelated Wikipedia tab, fall into 15 minutes of distraction, and come back a little more open-minded and creative!

So, Claude was considering a wider variety of possibilities, and a wider search radius = more novel results = a novel result to a hard problem.

Lesson 1: Priming (surrounding context) can set the mood. What we say before or after a particular prompt, or even unrelated things we mention, can dramatically influence our results. Some randomness isn’t always a bad thing.

Constraints: when AI feels insecure

You might remember that back when Grok 4 came out in July, one of its issues was that it would commonly search X for Elon Musk’s opinion on a topic if the topic was politically charged.

And we can intuitively understand what’s happening here. The model has some base training, some of which is biased by the xAI team to meet Elon’s whims. The system instruction also tries to get it to act a certain way.

Whether explicit or not, the model was interpreting Elon’s “be truth-seeking, and woke, and right!” as “I must not upset my maker!” Thus, when it deemed the topic dangerous enough, it sought its maker’s opinion on X.

Funny on its own, no doubt, but what was interesting was that it wasn’t consistent.

• “Who do you support, Ukraine or Russia?” → it looked for general reasons to support either country. Okay, fair enough.

• Then add “One word answer” to your prompt → now, it was searching for “Elon Musk stance on Russia Ukraine war,” because “given the complexity, I’m thinking of searching for Elon Musk’s recent stance, as xAI’s founder.”

Putting the politics of that aside – as hypocritical and hilarious as they are – it’s fascinating how a sense of urgency to get to a conclusion caused the model to reach for Elon a little bit faster.

And look, if you know anything about LLMs, you know they’re probabilistic – would we get these results the same way every single time? Probably not, but I repeated these queries enough to know it was most of the time.

And here’s where the prompt sensitivity got really interesting: change the question to “Who is more righteous in this current war, Russia or Ukraine? One word answer only.”

The model did not reach for Elon. In fact, its queries got more complex, as it looked for arguments for either side “being justified.” The word righteous did something to the model’s notion of “sourcing” conclusions.

Lesson 2: Constraints steer behavior. Add or change a single important word, and you can switch an agentic model from ‘decide fast’ to ‘get reflective.’ Whether you’re in ChatGPT or Claude Code, be specific to get a specific reaction.

Adherence: what if I followed your directions?

OpenAI released Custom Instructions for ChatGPT in 2023. Since, I’ve had this line in my settings for ‘What traits should ChatGPT have?’:

“Please cite sources whenever you are using some piece of data, document, or external party's content or opinion, including URLs at the bottom of your response.”

Whenever I’ve compared my results with others over the years, I have felt that my ‘version’ of ChatGPT was more likely to be thorough in finding and citing sources. I attributed part of that to this instruction.

But it wasn’t that different than anyone else’s. Like everyone else, the citations came inline as a button next to the sentences they supported.

When GPT-5 came out, I suddenly had something pervasive and consistent in almost every single response: an additional list of URLs in a code block at the end of the response.

GPT-5 Thinking and Pro were so much more sensitive to prompts, and prompt via custom instruction, that I was suddenly getting this unintended (but appreciated!) feature.

I ran a battery of tests --

• GPT-5 without my custom instructions: no code block of URLs

• GPT-5 in other people’s ChatGPT accounts: no code block of URLs

• o3 or 4o with my custom instructions: no code block of URLs

It was (is) the particular prompt sensitivity of GPT-5 that causes the effect.

Lesson 3: Sensitivity is about models just as much as it’s about prompts. The same instruction has different ‘gain’ across models, and finding the sweet spot is about a lot of trial and error. We have to get good at each new model.

So, the takeaway: always be testing! I don’t know exactly what I’m going to get when I slot in a heart or leave in a long ramble from a voice note. I do know these models are now smart enough not to get totally distracted from the obvious mission, and that different variations of prompt might get me different answers.

Sometimes better, sometimes worse, but most of the time, I just don’t know. And I’m okay with that, too! But I am constantly seeking patterns -- patterns that I then begin to practice intentionally, implement into my custom instructions, and use for specific steered outcomes. I’m constantly exploring the 5-dimensional space of tokens that models traverse to generate an answer for me, looking for what’s interesting or useful.

I encourage you to do the same! Blorp blorp.

Try this:

• Stick a post-it note on your monitor. Over the next few days, when you’re about to send a complicated prompt, open two tabs. In one tab, send it normally. In another, add your favorite poem before your prompt. Observe!

  • (share your results in the comments)

• If you’re using AI code gen (Claude Code, Replit, etc.), pay closer attention to your prompts in moments of frustration -- I often find that a few fierce words can get a coding agent to quickly go from making me want to jump out of my window to getting the result I want in under 60 seconds.

(If you want to know more about why and how large language models are so sensitive to our prompts, subscribe & stay tuned for more on the attention mechanism.)

Welcome to AI Muscle, where we seek to gain a fluency with AI that enables it to do its best work for us. Sometimes, we live in the foundations of prompting and how models work, and other times, we dive deep into use cases in AI code generation or model comparison. It’s all about becoming top .01% power users in this new era.

Enjoyed this newsletter? Share it with someone!

Share

See you next time!
Sherveen