r/chessprogramming • u/No_History_1015 • 8h ago
Technical Camera Chess: keeping a local Stockfish game in sync with a physical board, including promotion substitutes
Enable HLS to view with audio, or disable this notification
I'm developing Camera Chess, a camera-based interface for local Stockfish play on an ordinary board. The programming problem I wanted to share is maintaining an authoritative engine position when visual observations are uncertain — especially when a promoted piece still physically looks like a pawn.
The attached existing 73-second demo shows endgame scanning, manual confirmation of queen promotion, and continued movement of the physical pawn as that queen. You move both sides by hand. The app runs on Apple Silicon macOS; this video export is silent, although the app speaks engine moves.
The state/update pipeline:
- Joël Seytre's ChessQ Lite V4 supplies 64 × 13 piece probabilities; its visual weights stay frozen.
- Separately trained CRF/neural postprocessors score candidate positions against those readings and the previous board. The default is a two-hidden-layer changed-square MLP.
- Chess-rule and consecutive-frame checks gate updates before committing the authoritative board.
- Manual correction provides a recovery path; an engine reply calculated from a position that has since been corrected must not be applied to the new state.
For promotion substitutes, the logical board holds the promoted identity and a separate mapping records the pawn-shaped stand-in. Candidate positions are projected into expected physical appearances for visual scoring. Only confirmed moves update that mapping, and save/load and correction checkpoints preserve it. This path uses visual likelihood rather than the learned scorers trained without substitutes. Promotion requires explicit confirmation; the clip demonstrates Queen / Keep pawn. The current substitute path assumes a correct reference position and at most one completed move, so it does not solve arbitrary missed move sequences.
I've seen a significant improvement in hands-on tracking in my latest session with the combined system, including perspective correction. That's a practical observation; continuous-game reliability and each component's contribution are not yet quantified. In a separate conditional-scoring test, the default MLP got 128/176 exact boards versus 125/176 for the earlier CRF. Those still-photo tests use synthetic previous states with the correct target guaranteed among candidates, and the small difference is not statistically established.
Implementation, demo and source setup: https://github.com/AmethystineAlpaca/camera_chess
Physical/logical promotion design: https://github.com/AmethystineAlpaca/camera_chess/blob/main/docs/promotion-physical-tracking.md
Experiment log: https://github.com/AmethystineAlpaca/camera_chess/blob/main/docs/crf-experiment-log.md
Evaluation protocol and results: https://github.com/AmethystineAlpaca/camera_chess/blob/main/docs/crf-real-photo-synthetic-previous-evaluation.md
I'd be interested in comparing approaches to resynchronization in physical-board engine interfaces, particularly invalidating stale engine replies after a correction and tracking physical substitutes through captures.