Helping AI assistants use software without guessing

A person opening unfamiliar software looks around for clues before touching anything: where they are, what they can do here, and whether the button in front of them is going to save something. A coding agent, the kind of AI assistant that can read and change code, runs into the same questions whenever it works through a tool, even though the answer may arrive as data in a terminal instead of a visible screen.

Take a hypothetical request: change a screen so it matches the layout a reviewer picked from a few options. To finish it, an agent would have to collect the reviewer's choice, inspect that layout in the design file, read the team document that describes the screen, and update the task record once the work is done. At every stop it has only the tool's reply to work out:

• what just happened, and whether the action finished or only showed a preview,
• how much of what came back it actually needs to read,
• whether the next command will only look or will change something,
• and, if the work is cut off halfway, what it needs in order to pick it up again.

My recent side projects grew out of those questions. The idea behind them is that whatever a tool sends back is the interface the agent sees, so it deserves the care we give a screen built for a person: it should help the caller understand what happened and what it can do next.

Four tools came out of that idea, one for each stop in a request like the one above. Relay keeps the material under review beside a person's feedback on it, the way a draft and the notes about it sit on the same desk. Figma Lens helps an agent inspect a design before it builds from it. Outline CLI and Jira CLI connect the agent to a team's documents and task records, so it can read the same pages and tickets the team works from.

When the reviewer's choice comes back

Imagine asking someone to find a document and getting back a long page of search results without a clear link: you still have work to do before you can use the answer. An agent is left with the same extra work when a tool replies loosely, which is why I care about stable fields in a structured result. A command can state whether it completed an action or only produced a preview, and a review result can keep the chosen option together with the note attached to it, so the next step can inspect those fields directly.

Relay made this especially clear to me, and it is the first stop in the screen request. A reviewer may select a layout and leave a comment asking for one adjustment, and an agent that reads only the selected answer walks away with a different meaning from the one the reviewer gave. The tool's instructions therefore ask the agent to read every feedback channel.

When the design and the document are bigger than the task

With the layout chosen, the agent needs to look at it in the design file and read what the team wrote about the screen. Too much information at this point makes the next decision harder, while too little can send the work in the wrong direction, so I try to give the agent a smaller first look and let it choose where to go deeper. Figma Lens splits inspection into stages: the agent can look over a board, choose a screen and then inspect a few important parts at a useful size, while larger files stay in a cache and the returned result helps it navigate them.

Outline CLI takes a related approach for documents. The agent starts with IDs or summaries and then opens the page it picked, and although a title it remembers from earlier helps it find the source, the current wording still needs a current read. In both tools the overview helps the agent pick the screen or document worth a closer look before it opens the details.

When the agent is ready to update the task

Up to here the agent has only been reading, but updating the task record at the end changes a record the team works from. Reading a page and changing it have different consequences, and I want that difference to show in the command itself. Figma Lens is read-only. Outline CLI requires explicit action intent before it writes, and deleting a document also needs a recent read receipt tied to that document. Jira CLI previews changes by default and only sends them when the agent adds an action flag.

These conditions help keep an ordinary inspection from turning into an accidental update. Whether a change is authorized still depends on what the user asked for, and the tool's job is to make that boundary easier to follow without deciding it for the person.

When a valid command keeps getting misused

An argument list can explain what a command accepts, but it says little about when to use it or how the agent knows it already has enough information. Because of that gap, each project also includes an operating guide for the agent, called a skill. The Figma guidance asks it to open the returned images and inspect the details before building the screen, the Outline guidance asks it to read before editing, and Relay's guide explains when a board helps and when a short chat answer is enough.

Even with a guide, an agent sometimes keeps misusing a command that is perfectly valid. When that happens I look at the defaults and the sequence of steps around the command, because the missing piece may be a clearer result or an example of what to do after a failure, whereas adding more instructions to the end of a long prompt can leave the original ambiguity untouched.

When the work stops halfway

A request like the screen change can also be interrupted, so each tool holds on to something the next step needs in order to resume. Relay can keep a board and its draft feedback available after a wait ends, Outline checks the expected revision of a document before applying a patch, and Figma evidence already in the cache can support a narrower follow-up.

When agents write most of the code

I lead development of Figma Lens, Outline CLI and Jira CLI. The work starts from the places where an agent gets stuck, and from there I define how each tool should behave and respond. Coding agents produced most of the implementation, while my part is reviewing their code changes and testing the workflow across the tools, then refining the requirements whenever each command works on its own but the whole process is still awkward.

These personal tools give me practical interface decisions to test in my own work. The detailed notes on Relay, Figma Lens, Outline CLI and Jira CLI cover the commands and limits behind each of those decisions.

Contracts behind the workflow

Outline CLI has ids, summary and full views and requires performAction:true for mutations. Jira CLI uses --perform-action. Figma Lens exposes stages including scout, focus or focus-set, and detail. Returning a screenshot path does not establish that the agent viewed it. Relay returns feedback as JSON and supports detached review, keeping its board identity across waits. These personal projects let me test interface decisions in my own work.