Ten Ways to Let an Agent Test an Electron App

An agent here means a program, such as a coding assistant, that starts your app, drives it, reads what happened and decides whether it passed. These ten ideas make that possible without a person watching. They come from one app, Tortie, that was built this way. Each section explains the idea in plain words and then gives one example of how Tortie does it.

The ten are in the order a test run happens, from launch to verdict.

One flag turns a normal launch into a test run
A test run gets its own folders and refuses the real ones
Drives live inside the app and take a person's path
The app says when it is ready
Print a number, not only a picture
Report on standard output with fixed prefixes
Every native surface has one way to answer it
Point the app at stand ins for the outside world
One launcher starts the app and always ends it
The app reports facts and the script decides

Words used in this post

Main process. The part of an Electron app that runs in Node. It owns the windows, the menus and the connection to the operating system. It draws nothing itself.
Renderer. The part of an Electron app that draws one window. It is a web page running inside Chromium. It cannot reach the operating system on its own.
Profile. The folder where the app keeps its settings and saved state. Electron calls this the user data directory.
Harness. Code inside the app that exists only for testing. It starts the app, performs a scenario and reports what happened. It never runs for an ordinary user.
Drive. A set of steps the harness performs inside a window, e.g. open a project and then click a row. A drive is written into the app and is chosen from outside by name and given its arguments.
Probe. A script outside the app. It starts the app, asks for a drive, reads what the app printed and decides pass or fail.
Knob. An environment variable that turns a harness feature on or gives it a value. The app reads it once at launch.
Marker line. A line the app prints to standard output with a fixed prefix, so a probe can find it among everything else the app prints.
Seam. A place where the app can be pointed at a stand in for something outside it, e.g. a file in place of a network service.
Gate. A check that runs at build time or before a commit. It fails when a rule is broken, so the rule cannot erode quietly.

1. One flag turns a normal launch into a test run

The app reads one environment variable when it starts. When the variable is empty, the app starts the way it always does. When the variable names a scenario, the app boots its real code, performs that scenario, prints what happened and exits. The exit code is the verdict.

The same binary runs in both cases. There is no separate test build, so the test build cannot drift away from the product.

Both rows start the same binary. Only the environment variable differs. The exit code of the lower row is the verdict a probe reads.

A useful shape for tests of saved state is a pair of scenarios that run in two separate processes. The first process creates something and exits. The second process starts fresh and checks that the thing is still there. A test inside one process cannot prove that state survives a quit.

In Tortie. The variable GMUX_SMOKE selects one of 37 scenarios. A second variable, GMUX_SHOT, holds a file path and makes the app boot, photograph its window to that path and exit. The scenario named create makes a terminal session and exits leaving it running. The scenario named verify starts a new process and checks that the session is still there.

2. A test run gets its own folders and refuses the real ones

A test launch must never touch the profile a person uses or the services their work runs on. So every test launch gets a throwaway profile in a temporary folder, and a throwaway copy of any background service the app talks to.

Two separate checks enforce this. The launcher script refuses to start the app with a profile inside the repository or inside the home folder. The app then checks again from the inside. Before it creates anything, the harness confirms that its profile sits inside the temporary folder it was told about and that it is not talking to the real service. If either check fails, the app stops with a message that names the missing setting.

Both checks exist because either one alone can be bypassed. A person can run the app by hand without the launcher, and a launcher can be edited.

The launcher and the app each refuse on their own. A launch has to pass both before a single file is created.

One detail on macOS is easy to miss. A launch whose home folder was redirected has no keychain. The system then shows a dialog and waits, and every later keychain request on the whole machine queues behind that dialog. So a harness launch starts Chromium with a mock keychain.

Compare folder paths after resolving symbolic links. On macOS the temporary folder is reported as /var/folders by Node and as /private/var/folders by Electron. A plain string comparison refuses a launch that is in fact isolated.

In Tortie. The launcher creates a folder under the temporary directory and passes it as GMUX_HARNESS_DIR. It also names a private tmux socket for the run. Inside the app, every destructive scenario calls one function that confirms the profile is under that folder and that the tmux socket is not the real one. The check on the socket happens before anything is killed, because an earlier version killed first and checked after, and that cost the owner 48 live sessions.

3. Drives live inside the app and take a person's path

A drive is the set of steps a test performs inside a window. Write the drives inside the renderer, next to the code they drive. A drive there can call the app's own functions and send the app's own events. A script outside the window would have to find elements by their appearance and could only guess at what happened.

Two rules keep this honest.

The driving code never loads for a user

The main process adds a flag to the window's address only on a harness launch, e.g. index.html?harness=1. The renderer reads that flag once when it starts, and imports the driving code only if the flag is present. The build tool then places the driving code in its own file, because nothing else imports it. A gate reads the built output and fails the build if any of the driving code ends up in the file a normal launch loads.

The flag on the address is the only gate. The build tool separates the files on its own, and a gate keeps them separate.

A drive uses the same path a click would

A drive calls the same function the app calls when a person clicks, and sends the same browser events. It never writes state directly. If a drive had a shortcut into state, a screenshot would show what the shortcut produced and not what the app does. Any bug in the path between the click and the state would stay hidden.

The drive enters at the same place a person does. The dashed path is the one to refuse.

In Tortie. The file probe-registry.ts holds every drive. A launch loads it only when the window address carries harness=1. The drive that switches projects by number sends a key event to the window, which is where the app's keyboard handler listens. The reason written in that file is that a loader with a second path would make every screenshot a photograph of the harness.

4. The app says when it is ready

A test that waits a fixed number of seconds is slow on a fast machine and wrong on a slow one. Instead, the drive raises a flag on the window when the screen it staged is drawn. Before every change of view it lowers the flag, and it raises the flag again after the change lands. The main process polls the flag every quarter second up to a deadline, and captures or measures only when the flag is up.

Poll an error slot in the same expression. When a drive throws, it stores the error in that slot. The main process then fails within a quarter second and prints the stack, instead of waiting for the deadline and reporting nothing useful.

Main reads the flag and the error slot in one expression. The flag goes down during every change of view, so main cannot capture in the middle of one.

A run with no drive has nothing to raise a flag. That run needs a delay, and the delay should be a knob rather than a constant, because a cold start on a busy machine takes longer than a warm one.

Before the capture itself, bring the window to the front and wait for two animation frames. Chromium returns the last painted frame when the window is not in front, and a window in the background also slows every timer to one tick per second. A drive measured at 18 seconds in front took 52 seconds behind another window.

In Tortie. The drive sets window.__gmuxShotReady when it finishes. Every extra step in the layout wrapper lowers that flag and raises it again, about twenty pairs in all. Main polls the flag and window.__gmuxShotError together with a sixty second deadline, and it pins the window on top for the length of the drive.

5. Print a number, not only a picture

A screenshot is evidence a person can read. An agent reads it less reliably, and a picture cannot be compared to a threshold. So give the harness a way to run one JavaScript expression inside the window after the drive and before the capture, and print the result as JSON on a marker line.

Each drive leaves its readings in a named global on the window, e.g. the width of a column and the count of rows drawn. The expression names that global. The probe finds the marker line, parses the JSON and compares it to what it expected. The screenshot is kept beside the number as supporting evidence.

The expression is one line the probe chooses. The answer comes back as JSON on a line the probe can find by its prefix.

In Tortie. The knob GMUX_SHOT_JS holds the expression. A drive that reads file history leaves its 31 rows in window.__gmuxP198FileHistory, and the probe compares those rows to what git prints for the same file. Around 38 such globals exist, one per drive.

6. Report on standard output with fixed prefixes

A probe cannot see inside the app. It sees only what the app prints. So the app prints every harness message on standard output with a fixed prefix, and the probe waits for lines by pattern and parses them. Keep the set of prefixes small and never change one, because a probe that waits for a line that no longer prints waits until its deadline and then reports nothing.

The kinds of lines an app needs are few.

A numbered step line for each stage of a scenario, so a failed run shows how far it got.
A result line with JSON after a fixed prefix, for anything a probe will compare.
A line that says a seam was installed, so a probe can wait for it before acting.
A failure line with a reason in words.

Add one knob that copies the window's console to standard output. The drive runs inside the window, and without this a drive that stalls leaves no trace of where it stopped.

[smoke] 1/5 window shown
[smoke] 2/5 session created id=$7
[shot] usage fixture installed, the vendor is a file
[shot] popup-pick {"pick":"History","id":"hist","labels":["Open","History","Copy path"]}
[shot] probe {"columnWidth":184,"rows":31}
[shot] wrote /tmp/run-8f31/window.png
[shot][renderer] drive step: open file
[smoke] FAIL: drive never finished

Give the launcher one function that waits for a line by pattern with a timeout. Most of a probe is then a short list of waits and parses.

In Tortie. Every harness line starts with a bracketed prefix such as [gmux-shot] or [gmux-smoke]. The launcher hands each probe a waitForLine function. The knob GMUX_SHOT_VERBOSE copies the renderer console to standard output with its own prefix.

7. Every native surface has one way to answer it

A context menu, a file picker, a message box and a tray menu are windows the operating system owns. The renderer cannot click them. A screenshot of the app window does not include them. A test that opens one by accident stops and waits forever.

So for each native surface, give the main process one knob that answers it without showing it. For a context menu, the knob holds the label of the item to choose. When the renderer asks for the menu, main does not show it. Instead main finds the item with that label, resolves the request with that item's id, and prints every label the menu would have shown. The probe can then confirm the item was offered and not only run.

The same knob set to a label that no item has closes every menu without choosing anything. That is how an unattended run avoids getting stuck on a menu it did not expect.

Under the knob the menu is never shown. The request is answered as if the person had chosen the item, and the printed labels prove the item was on the menu.

Other native surfaces need their own answer. An external program the app would open can be recorded to a file instead of started, so it does not take focus from the window. A message box can be replaced for the length of one scenario and restored after. The application menu and the tray menu can be read from the real menu objects and printed as rows, since they cannot be photographed.

In Tortie. The knob GMUX_SHOT_POPUP_PICK answers context menus by label. Several probes set it to a label no item has, which closes every menu and still prints every label for inspection. The knob GMUX_OPEN_WITH_RECORD makes an external launch append one JSON line and start nothing. One scenario walks the application menu and the tray menu out of the real menu objects and prints every row with its shortcut.

8. Point the app at stand ins for the outside world

An app talks to things it does not own, e.g. a vendor's network service. A test that reaches those things is slow, costs money and gives a different answer each time. So give the app a seam for each one. A seam is a knob that points the app at a stand in, and the app reads the stand in through the same code it uses for the real thing.

Three kinds of stand in cover most cases.

A file in place of a network service. The knob names a JSON file, and the app reads the answer from the file instead of making the request. The file can also hold a delay, so the slow case can be tested.
A stub program in place of a real one. The probe writes a short shell script with the real program's name. The script answers a version check, records the environment it was given to a file, and then sleeps. The app launches it exactly as it would launch the real program, and the probe reads the record to see what the program was handed.
A seed file in place of history. The knob names a JSON file of rows, and the app inserts them through its own write functions before the window opens. The screen then shows state that would take hours to build by hand.

Every seam refuses to install unless the launch is isolated and the profile sits inside the scratch folder. A seam that installs on a real profile is a way to feed a real user false data.

The column on the left never changes. Only what sits on the far side of it changes, and only under a knob that refuses on a real profile.

In Tortie. The knob GMUX_USAGE_FIXTURE makes the usage meter read a file and refuses the keychain outright. Probes write stub scripts named claude and codex that record the pane's environment and sleep, so no vendor program runs and no token is spent. Four seed knobs insert rows through the app's own store functions. Each of these refuses unless the profile is under the harness folder.

9. One launcher starts the app and always ends it

An agent runs many probes in a row, and it retries the ones that fail. A probe that ends the app only when it passes leaves one app running every time it fails. Each Electron app holds a few hundred megabytes. Enough of them take the machine down. Tortie's owner lost a day of uncommitted work this way.

The fix is one function that owns every launch. Every script starts the app through it and never on its own. The function hands the running app to the script's code, and then ends the whole process tree in a finally block whatever that code did. It also registers handlers for process exit and for unhandled errors, because a finally block does not run when a script calls process.exit.

Ending an Electron app correctly takes several steps.

Send SIGTERM to the process the launcher started. When the app was started through the electron command, that process is a small Node forwarder and not the app. The forwarder passes SIGTERM on. It cannot pass SIGKILL on, so a SIGKILL to it leaves the app running with no parent.
Read the list of child processes before sending any signal as well as after, because a dead parent's children get a new parent and can no longer be found from it.
Send SIGKILL to every child by process id.
Find any process whose command line names this launch's own profile path and end it. That catches the crash handler, which changes parent immediately.
Return the count of processes the teardown had to end. Zero is the normal case, and the probe can print it.

Three steps in order. The forwarder gets SIGTERM, the descendants get SIGKILL by id, and a sweep by the launch's own profile path catches what already escaped the tree.

Add a gate that scans every script and fails if any of them starts Electron without going through the launcher. The gate should also confirm that the launcher's own kill sits inside a finally block, by matching braces rather than by searching for the word.

In Tortie. The file build/electron-run.mjs is the one place an Electron starts. Fifty scripts reach it. The gate named gate:electron runs inside every build and found a new leak in a probe from another phase the week it was written.

10. The app reports facts and the script decides

The harness inside the app is the thing being tested. It should not also decide whether it passed. So the harness writes readings, e.g. the status it showed at each moment while a network link was cut. The probe outside reads those readings and decides. A harness that grades itself has the same bugs as the code it grades.

The grader in the probe should be a pure function. It takes readings and returns a verdict, and it touches no process and no file. Give every probe a --self-test flag that runs the grader over a set of saved readings that should pass and a set that should fail, and starts no app. A grader that has never been seen to fail proves nothing when it passes.

The same grader reads live readings and saved ones. The saved bad set is what proves the grader can say no.

Keep the same habit for the gates. A gate that scans files for a rule should run itself over a few small files it writes on the spot, some that break the rule and some that keep it. When someone finds a case that walks past the gate, the case goes into that set in the same commit as the fix.

In Tortie. Nine harness files carry the sentence that the app never grades itself. The scenario that cuts a network link samples the status every quarter second and writes rows, and the script that started it reads the rows and decides. Fifteen probes accept --self-test. The gate on SSH known hosts keeps 36 sample files beside it, 32 of which must make it fail, after a reviewer wrote 21 hostile scripts and nine of them walked past the first version.