Arabic Sign Language Digital Twin
Reconstructing dual hand 3D motion from ordinary video, to build a virtual smart glove without the glove.
The problem
Sign language recognition gloves put hardware on every user's hands. Expensive, fragile, and a barrier to adoption. The question was whether the glove's signal could be reconstructed from plain RGB video instead.
Approach
- A staged pipeline: RGB video, dual hand 3D reconstruction, temporal tracking, anatomical joint angles, virtual glove signal, then recognition.
- Four pose backends benchmarked against each other behind a shared interface instead of committing to one early: MediaPipe, HaMeR, WiLoR, and OmniHands.
- Anatomical kinematics computed in a canonical hand frame, so joint angles are comparable across signers and camera positions.
- Alongside it the physical path: a recognition glove on IMU sensors and ESP32 microcontrollers, with the network embedded on device for real time translation.
Key decisions
- 01
Stop at the sensor design, not at the demo
The milestone deliberately ends after reliable 3D data and its evaluation, before recognition, Arabic NLP, or speech. Building recognition on unvalidated pose data would have produced a number that looked like progress and taught us nothing about where the error came from.
- 02
Raw extraction outputs are immutable
Every stage writes forward and never overwrites its input, so any result traces back to the frames that produced it. Reproducibility over convenience.
- 03
Benchmark on an RTX 2060 Super
The target hardware is a consumer GPU, not a datacenter card. A method that only works on an A100 is not a solution to this problem.
Results
- Reproducible research codebase with enforced stage separation, mandatory reporting, and a shared evaluation harness across four pose backends.
Saden
Want the detail behind any of this? Get in touch.