Project Overview

The JIWY Embedded Tracker is a 2-axis pan-tilt robotic vision platform. The project focuses on the complete cyber-physical development and hardware/software co-design: integrating custom Verilog modules on FPGA fabric with high-level software running on an ARM core to track moving targets with minimal latency.

The Challenge

The primary challenge was architecting the end-to-end cyber-physical co-design between the FPGA fabric and the ARM host processor across distinct hardware platforms (evaluating the Intel Cyclone V SoC on DE10-Nano versus a Raspberry Pi with Lattice iCE40 FPGA). Key engineering hurdles included:

• Hardware/Software Partitioning: Designing custom Verilog modules on the FPGA to handle high-frequency quadrature encoder decoding and 20 kHz PWM generation, while bridging them seamlessly to the ARM processor.
• Inter-Domain SPI Communication: Implementing high-speed SPI communication between the FPGA and the ARM processor to transmit encoder counts and control signals deterministically without timing jitter or packet loss.
• Resource-Constrained Vision: Due to limited on-board processing capacity and specific raw camera output formats, standard color-space conversions were too costly, requiring optimized, high-throughput image processing techniques to sustain real-time frame rates.

Engineering Solution & Co-Design Architecture

I implemented a complete hardware/software co-design pipeline connecting sensor acquisition, real-time image processing, and closed-loop motor actuation:

• Custom FPGA Hardware Modules (Verilog): Developed hardware blocks on the Cyclone V FPGA fabric, including dual-channel quadrature encoder counters and high-frequency 20 kHz PWM motor drivers equipped with hardware-level duty-cycle limits and dead-time safety protection for the H-bridge drivers.
• Memory-Mapped Bus Communication: Mapped FPGA hardware registers directly into the Linux user-space application via the Avalon Memory-Mapped bus, enabling microsecond-level register reads and writes with zero driver overhead.
• Automated Self-Calibration Routine: Because the physical configuration and zero positions are unknown at startup, developed a self-calibration routine. It drives motors at a safe duty cycle, detects mechanical end-stops by monitoring encoder feedback variations, measures physical axis ranges, calculates the true center, and sets deterministic zero references.
• Optimized 3D LUT Color Segmentation: Overcame camera format and compute bottlenecks by building a quantized 3D Look-Up Table (LUT) in YUV color space. This reduced per-frame processing time from 45.27 ms down to 12.05 ms (3.75x faster), eliminating frame drops and comfortably fitting within the 33.3 ms (30 FPS) real-time budget while saving memory.
• Closed-Loop 2-Axis Motion Control: Modeled the pan-tilt dynamics and implemented dual digital PID controllers in embedded C++, running at 100 Hz with software limit switches and safety watchdog monitoring to keep the target centered smoothly.

Real-Time Pan-Tilt Visual Tracking Demonstration

Hardware & Software Co-Design Architecture

System data flow across the ARM host application and custom FPGA fabric:

Embedded Benchmarks and Results

The co-design achieved fast frame processing and smooth target tracking:

Technologies

Open Source Repository