Ops Automation · Reliability
Exhibit Auto-Recovery
A recovery system for MaskBot, a face- and voice-interaction robot exhibit on the museum floor. I installed it on the exhibit machine in July 2026, and it still runs unattended in daily operation.
MaskBot's control software would sometimes freeze or exit mid-operation. The only fix was a manual restart through a remote session from one specific machine. Until someone got to that machine, visitors stood in front of an exhibit that could not hear them.
Now a watchdog checks the program every 20 seconds and relaunches it within a minute when the process dies or its window stops responding. Scheduled restarts also run twice a day. Any staff member can trigger recovery from their own phone on a single-button page. Enrolled devices reach it through Tailscale, a WireGuard-based mesh VPN, with no port open to the internet.
One failure cannot be detected automatically: speech recognition stops while the process stays alive and the window keeps responding. Scheduled restarts and the phone button cover that case. I also wrote a handover document on configuration, logs, routine maintenance, a diagnosis playbook, and known limitations, so the maintenance technician can own the system without me.
Highlights
- Every freeze or crash needed a manual restart, and it could only be done from one specific machine.
- A watchdog relaunches the exhibit program within a minute of a crash or hang it detects.
- Any staff member can restart the exhibit from their own phone on a one-button page served from the exhibit machine.
- Shipped with a maintenance handover document: configuration, logs, diagnosis playbook, and known limitations.
Architecture
How the app is structured: screens, roles, and data flow.
This site keeps anonymous visit statistics. Privacy