Familiar
A native desktop assistant that connects screen context with guided and agent-driven workflows.
- Software
Familiar uses screen context to help someone work through an application. My contributions to the native client cover screen context, guided interactions, and explicit controls for changing modes or stopping a task.
- When
- May to June 2026
- Where
- Built with a teammate
- Role
- Developer of the native client and agent loop
- Platform
- Native macOS app with a Node backend
- Modes
- Guide, Agent, and Chat
- Stopping
- Global stop shortcut, per-action time cap, and a watchdog
- Tools
- Swift, SwiftUI, AppKit, Accessibility API, ScreenCaptureKit, Node.js, TypeScript, Supabase
Credit. Built with a teammate, who designed much of the notch interface and the run history and replay.
Problem
Instructions for desktop software usually live in a separate window from the software itself. Familiar reads what is on screen and helps from inside the task, either by pointing to the next step or by carrying the steps out.
My role
My work covered the native Swift client, screen-context and targeting code, the agent loop, and cancellation and recovery logic. A teammate built much of the notch interface, the theme, and the run history and replay.
How context reaches the assistant
The Mac app reads the frontmost window through the macOS Accessibility API and captures the screen with ScreenCaptureKit. It sends the goal and that context to a backend, which calls a model and returns one step at a time. Each step’s target is found by its Accessibility label first, with a vision fallback that is checked against the real interface element before it is used. A privacy setting lists apps Familiar will not read.
Three modes
- Guide only points. A ghost cursor shows where to click and a short bubble says what to do. The app waits for my real click, or for the expected app to open, before moving to the next step.
- Agent moves the real cursor and performs the step. It stops on an approval card when the model marks a step as irreversible.
- Chat has no cursor. Answers draw on the current screen and a searchable history of earlier screens.
Staying oriented and stopping
A widget at the top of the screen shows the current mode and progress. A global shortcut cancels any run and releases the cursor. Every action has an 8-second cap, and a watchdog notices when a run stops making progress and shows a card with Retry, Skip, and Stop instead of leaving the app frozen. Accessibility calls have a short timeout so one unresponsive app cannot stall the loop.
Decisions
- Pointing and acting are separate modes. Guide never moves the cursor. Taking control is an explicit switch to Agent mode.
- Accessibility before vision. Targets come from the Accessibility tree. Vision is a fallback, and its answer has to match a real element on screen before it is used.