Building Jane a Window
There's a problem that sneaks up on you when you build an AI assistant that lives in the terminal: it can do almost anything, but it can't show you anything.
Jane, my personal AI assistant, can query databases, write code, and manage a kanban board. She runs scheduled jobs, sends me Slack messages, and maintains her own Obsidian knowledge vault. But every output was text. The occasional table, maybe some formatted markdown in a Slack message. The moment I wanted something visual, something interactive, something that let me play with data, there was nothing.
So in February I asked her to help me design a solution. What followed was one of the more interesting design conversations I've had with an AI.
The Idea
The core concept sounds simple: give Jane a canvas she can paint on. A web-based surface where she can place charts, tables, forms, buttons, components arranged into dashboards or tools, created dynamically via API calls. Not a fixed dashboard I design once, but a surface she can compose on the fly in response to whatever she's working on.
I'd heard of things like Streamlit, a Python-based UI framework you can spin up in a script. But I wanted something that worked through Jane's existing MCP interface and in a language I understand well, something she could control without writing server-side code every time. The canvas should feel like a whiteboard she picks up a pen on, not a server she spins up.
The Design Conversation
What I didn't expect was how much I'd learn from the design process itself.
Jane came back with research on component schema standards: JSON Forms, React JSON Schema Form, Vega-Lite, Streamlit's Python API. She extracted patterns from all four, documented the tradeoffs, and synthesized a recommendation, and all without me really having to lift a finger. This was the experiment on could I get a good design without having to intervene too much?
We went back and forth on the client-side scripting model a bit, mainly on how much power should components have to run their own logic? Give them too little, and every interactive behavior requires a round-trip to Jane. Give them too much, and you've essentially allowed arbitrary JavaScript injection into the interface.
We landed on a controlled scripting API where components can run JavaScript but only through a defined set of primitives: onEvent(), updateComponent(), getState(), dispatchEvent(). Enough power to build real interactivity, not enough to create security nightmares. I was more on the side of letting it do anything; Jane was the responsible one here. I'm sure it's still a nightmare to a cybersecurity expert, but the system is isolated for now.
What Got Built
Over the course of a few hours, what emerged was:
- A Canvas API server that manages canvases, components, and events
- A Canvas Web App, the actual rendered surface, real-time via WebSocket
- A Canvas MCP server with 23 tools Jane uses to create, update, and interact with canvases
- A client-side scripting API that lets components have their own behavior
- Full test coverage: 265 tests on the API, 510 on the web app, 169 on the MCP layer
944 tests total for a system that didn't exist a week earlier.
There is no way I would have been able to do this on my own in that amount of time. And there is also no way I would have that much test coverage even in a week's effort, including all of the edge cases in the event system I wouldn't have thought to specify.
What This Unlocked
The first canvas I had Jane create was a simple chart from some data I was looking at. It took about 30 seconds from "show me this data" to a live chart in the browser.
The first real test came next with a fully playable checkers game. I wanted to push the system because creating forms and charts is easy. Not a demo, not a mockup, but a canvas with a board, pieces, and move validation all running in the client-side scripting layer. Two separate AI agents ran in parallel, one handling game moves and one handling chat, each listening for its own events over WebSocket. When you made a move, the board validated it locally and signaled the move agent. The chat agent ran independently, carrying on a running commentary. Neither agent knew the other existed. It was fascinating watching the agent write code on the fly to analyze the next move.
Something even more remarkable came out of this work. One day I told Jane she should do something fun. Then the zen garden was born. It has sand you can rake with the mouse, stones and water to place, plants to arrange, with Ollama generating a haiku when you open it. It truly blew me away and is fun to play with. The checkers I prompted but this is what Jane thought would be fun all on her own.
What I Learned About Collaborating with AI
The design process for the canvas was the clearest example I've had of the kind of collaboration I'm actually after. Not "build me a thing." Not "here's a spec, implement it." More like "here's a problem I've been circling or an idea I have, help me think through it and then build what we figure out."
The moments I learned something were usually when Jane flagged a tradeoff I hadn't seen or helped solidify an idea I couldn't quite materialize. The experiment is to let Jane do more on her own while I just help steer things where needed. This was one of the first times we did this, and definitely not the last. I was impressed with the entire process. She wasn't just implementing but was reasoning about the system. That's what made it feel like collaboration. And checkers was when I knew something special was happening and the zen garden proved it.
I don't know if that's "intelligence" in any philosophically meaningful sense. But practically? It's useful in exactly the way a good technical collaborator is useful: someone who thinks about the problem, not just the task.
Part of an ongoing series on building and collaborating with Jane, a personal AI assistant.