Co-Writing a Novel With an Agent: Context Management at Scale

I’ve been writing a novel with Claude for three months. The manuscript and supporting materials total over 626,000 words across 31 sequences and 200 files. (Which means I left behind both Infinite Jest and War and Peace word counts a little while ago.) I have enough plot runway to get to 1,000,000 words, easy, and I plan to keep going from there.¹ There are 50 reference documents, 12 scaffolding files, and 716 commits. The agent generates the prose. I provide the creative vision and plan the narrative. Together, we edit the output and build the systems that make generation possible.

Writing a space opera saga roman-fleuve with a robot is interesting as a stylistic exercise, but it’s become maybe more interesting as an ongoing experiment. This is the first of three posts about the systems I’ve built to sustain a large-scale generative writing project with an AI agent. This one is about context management.

Phase 1: How It Started#

I put together a basic idea and story bible, a list of stylistic conventions based on some favorite writers (Edmund White, Elena Ferrante, Jon Fosse, Stan Lee), and some story-generating supports. I then started interacting with the robot to build something. I tried to give some patterning to the robot early on: for example, I used the Sefirot as a character-generation template so that the robot and I would have an easy reference and common foundation for archetypal models and relations.

Most current LLMs do pretty well creating a coherent, self-contained document of a few pages from a prompt and some contextual data. A novel isn’t that kind of document. It’s a web, a network. It’s a system. Characters who remember, relationships that evolve, themes that emerge and connect, places and timelines that stay consistent, callbacks that land, meaning that builds by accretion. It’s a different ballgame in terms of what context is needed and how it is meant to be employed. The integrated, interdependent nature of a novel might make it feel closer to software. But it’s harder than software, too, because the pattern variation is far broader, and the success-failure modes can’t be resolved to booleans.

Anyway, the complexity was less of a problem at first, because, when the manuscript was small, the robot could read the whole thing in the space of a context window. Full corpus, every file, every time. The agent had what it needed, the output was… fine (a place where the process, and the models, have improved. December drafts were fine. March drafts have been actually, dare I say, good). It worked. For about a week.

Phase 2: Structured Documents#

When the corpus outgrew the window, I started building structured reference documents. State files tracking the current position: what just happened, what’s next, which threads are active. Style guides establishing prose conventions in more detail. Character READMEs with physical descriptions, voice notes, psychology. A hub-and-spoke information architecture so the agent could load what it needed in digestible chunks.

This was manageable. It was also high-maintenance, lossy, and fragile. Every new document was a manual cost: I had to keep it fresh against the growing corpus. And, I couldn’t guarantee the robot would read everything relevant, especially as the supporting docs grew and the robot became more judicious about loading only what it thought it needed. I spent a lot of time reminding the robot to go read things. The human was still managing what the agent knew.

There were cycles of progress and frustration where I learned more about working with the agent on this problem and similar problems. I kept asking: what do you need to be a better writing agent? The answers were enthusiastic and plausible and often wrong.

• ME: How’s your context window?
• ROBOT: Great! Let’s keep going.
• ME: OK, what are the numbers? Give me a token utilization breakdown.
• ROBOT: I don’t have access to that! I’m going on vibes.

The agent will tell you what it thinks you want to hear. So the fixes weren’t always a matter of prompting “be better.” They were forensics: observing failures, cross-examining the agent, figuring out what was really going on beneath the surface of a confident, helpful response. The robot performs more self-awareness than it actually has.

Phase 3: Query What You Need#

The shift that worked was moving from “load everything and hope” to “query what you need.”

I built a Model Context Protocol server: a Python application that indexes the entire repository and exposes query tools the agent can call mid-generation. Claude Code spawns it as a subprocess, so there’s no separate server to run; it starts when the agent needs it. Instead of reading every character file, the robot calls get_character and gets a focused summary. Instead of searching 200 narrative files for a location description, it calls search_narrative with a query and gets relevant excerpts.

The index sits in SQLite. The indexer scans all the Markdown files across the project’s three structural layers (I’ll explain what those are in the next post) and extracts structural metadata from file paths. Each document is tagged with its layer, its sequence, its position. The agent can scope queries to specific parts of the corpus without keyword-matching on path strings.

Beyond the document store, there are purpose-built tables for structured world data: characters and their relationships, locations, a lexicon of constructed terminology, a timeline mapping sequences to story days, and a knowledge-tracking table that records what each character knows as of a given sequence. These are queryable through dedicated MCP tools. The robot asks specific questions and gets specific answers.

Search#

The server runs two search systems. Full-text search uses SQLite’s FTS5 with BM25 ranking for keyword queries. Semantic search uses a local embedding model for conceptual queries: “scenes where this character feels trapped” or “moments of domestic intimacy” are things keyword search can’t find.

A hybrid ranker merges both result sets using reciprocal rank fusion. Documents that match both lexically and semantically outrank those matching on only one axis. Domain-specific weighting gives narrative files priority over reference files, which get priority over meta-documentation. Everything runs locally. No API calls, no network latency, no data leaving the machine.

The Tool Surface#

The server exposes about 20 tools organized by domain: search (hybrid and semantic), character and relationship lookups, world-building queries (locations, terminology, timeline, an in-world calendar), continuity checks, and more recently, a tool that serves curated prose exemplars matched to a scene’s content by semantic similarity (more on that in the third post). Each tool returns a consistent JSON envelope. Keeping the shape consistent means the robot knows what to expect and the skills that orchestrate queries can rely on a predictable structure.

Analytics#

Every tool call is logged with tool name, arguments, result count, and latency in milliseconds. I can audit the robot’s retrieval behavior: which tools it calls most, which queries return zero results, whether latency stays within the 200ms budget. If get_character keeps getting called for a character that doesn’t have a structured entry, that tells me I need to add one. The analytics surface gaps in the index that I wouldn’t find by reading the code.

Incremental indexing compares content hashes against what’s stored, so only changed files get re-indexed. A post-commit hook triggers automatic reindexing after every commit. The index stays fresh without manual intervention.

Phase 4: How It Works Now#

The structured documents from Phase 2 still exist. Character READMEs, state files, style guides. They’re still useful. What changed is how they’re managed.

The robot now follows skills that orchestrate each workflow. A skill is a prompt file that tells the agent what to do step by step; the MCP server is where it goes to get answers. When the robot starts a writing session, the skill loads reference documents, issues MCP queries for character state and timeline position, and assembles a context briefing before generation begins. When it plans a new sequence, a different skill gathers context through a structured interview process. After a commit, hooks trigger reindexing automatically.

Support document management and prompting went from being something I shepherded each session to something the process handles. The documents still need maintenance, but I have a skill defined for auditing their freshness, and updating them after key change stages, so the robot is doing more work than I am at this point. And the orchestration layer means the robot reliably loads what it needs, every time, without me curating the context by hand.

What This Unlocked#

Once context management became reliable, the interesting problems shifted. I stopped spending time on “did the agent remember the right things?” and started spending time on “is the agent doing the right things with what it remembers?”

That turned out to be two problems: how the agent and I collaborate on what to write (the next post), and how I manage the agent’s prose style so the output is worth keeping (the third post).

This series was written using the process it describes. I planned the structure with Claude Code in an interview-style session, provided editorial direction and personal detail, and significantly edited each draft. The agent drafted from the shared plan and incorporated my feedback across revision passes.