Skip to content
All work

Ops Automation · Reliability

Exhibit Auto-Recovery

A recovery system for MaskBot, a face- and voice-interaction robot exhibit on the museum floor. I installed it on the exhibit machine in July 2026, and it still runs unattended in daily operation.

MaskBot's control software would sometimes freeze or exit mid-operation. The only fix was a manual restart through a remote session from one specific machine. Until someone got to that machine, visitors stood in front of an exhibit that could not hear them.

Now a watchdog checks the program every 20 seconds and relaunches it within a minute when the process dies or its window stops responding. Scheduled restarts also run twice a day. Any staff member can trigger recovery from their own phone on a single-button page. Enrolled devices reach it through Tailscale, a WireGuard-based mesh VPN, with no port open to the internet.

One failure cannot be detected automatically: speech recognition stops while the process stays alive and the window keeps responding. Scheduled restarts and the phone button cover that case. I also wrote a handover document on configuration, logs, routine maintenance, a diagnosis playbook, and known limitations, so the maintenance technician can own the system without me.

Year

2026

Role

Solo: diagnosis, build, rollout, handover

Focus

Exhibit uptime · self-initiated

Highlights

  • Every freeze or crash needed a manual restart, and it could only be done from one specific machine.
  • A watchdog relaunches the exhibit program within a minute of a crash or hang it detects.
  • Any staff member can restart the exhibit from their own phone on a one-button page served from the exhibit machine.
  • Shipped with a maintenance handover document: configuration, logs, diagnosis playbook, and known limitations.

Architecture

How the app is structured: screens, roles, and data flow.

Exhibit Auto-Recovery architecture flow

This site keeps anonymous visit statistics. Privacy