Project Overview

Vision-Language-Action (VLA) models unify visual perception, natural language understanding, and physical manipulation. However, standard end-to-end VLAs struggle with ambiguous instructions and non-English prompts. This project investigates a two-stage approach: using intent classification as a pre-processing frontend to decouple linguistic ambiguity before executing physical actions with a lightweight SmolVLA (0.45B) policy on a 6-DOF SO-101 robotic arm.

The Challenge

Deploying monolithic end-to-end VLA models in real-world human-robot interaction environments revealed critical points of failure:

• Severe Prompt Ambiguity Sensitivity: Standard VLA policies fail when instructions are informal, conversational, or slightly ambiguous, causing physical success rates to drop to zero without explicit structured commands.
• Multilingual Language Failure: Providing prompts in languages other than English (such as Dutch or German) causes the model to fail completely or execute random motor actions because the vision-language policy lacks non-English grounding.
• Data Scarcity for Full End-to-End Training: Collecting enough multimodal demonstration data to teach a monolithic model all linguistic nuances, grammar variations, and multi-language instructions directly is prohibitively expensive.

Engineering Solution & Two-Stage Architecture

To solve prompt fragility, a decoupled two-stage architecture is developed that separates intent interpretation from physical motor control:

• Stage 1: Lightweight Intent & VLM Pre-Processing: Designed a lightweight first stage that introduces minimal computational overhead, serving as a proof of concept for using a low-frequency VLM frontend to preprocess the scene and retranslate/standardize ambiguous or multilingual prompts (such as Dutch and German) into structured goals. Evaluated across 2,600 prompt variations, fine-tuned BERT achieved an 85% F1-score.
• Stage 2: SmolVLA Manipulation Execution: Fed standardized intent targets to the SmolVLA 0.45B policy running within Hugging Face LeRobot, controlling the 6-DOF SO-101 follower arm equipped with dual camera feeds (overhead fixed camera and wrist gripper camera).
• Robust Task Disambiguation: Breaking complex commands into clean, standardized subtasks prevents error propagation and allows the downstream VLA policy to maintain high physical accuracy without linguistic distraction.

Physical Robot Arm Video Demonstration

Experimental Results and Success Rates

Physical experiments on the SO-101 setup proved that monolithic end-to-end VLA models degrade steeply under linguistic variations, confirming the necessity of intent classification:

Technologies