深度影像機械手臂人機互動介面Depth-Vision Robotic-Arm HMI

本計畫開發一套結合六軸機械手臂與深度影像辨識（搭配深度學習）的人機互動介面。設計理念是讓「取什麼、放到哪」由人決定 — 使用者只要在螢幕上點一下物體，手臂便會自動將它移到指定位置。計畫期間完成了手臂通訊、正逆向運動學、像素轉實際長度的座標校正與物體辨識，並成功以純深度影像（不依賴顏色資訊）辨識並抓取物體，後續再朝物體堆疊、非平面與圓錐形等複雜情境擴展。This grant project built a human–machine interface combining a six-axis arm with depth-vision recognition (plus deep learning). The idea is to let a human decide what to pick and where to place it — tap an object on screen and the arm moves it there automatically. The project implemented arm communication, forward/inverse kinematics, pixel-to-real-length calibration, and object recognition, and successfully grasped objects from depth images alone (no color cues), with later extensions toward stacked, non-planar, and conical objects.

工業機器人的安裝量持續攀升（2021 年全球新安裝量達 517,385 台，創歷史新高），智慧工廠對「結合影像感測器、能即時操控」的自動化需求也隨之提高。傳統視覺抓取容易受光線干擾，因此本計畫改以深度影像搭配深度學習，在弱化顏色與光照依賴的前提下完成辨識與抓取，並以直覺的人機介面降低操作門檻。Industrial-robot installations keep climbing (a record 517,385 units installed worldwide in 2021), raising demand for vision-sensor-driven automation with real-time control. Conventional vision grasping is sensitive to lighting, so this project pairs depth imaging with deep learning to recognize and grasp with reduced color/illumination dependence, behind an intuitive interface that lowers the operating barrier.

• 深度影像感測Depth sensing
  以 Intel RealSense D435 取得對齊的深度影像，作為辨識與抓取的主要輸入。Intel RealSense D435 provides aligned depth images as the primary input for recognition and grasping.
• 座標校正Coordinate calibration
  依相機內參與深度動態計算水平／垂直視野（HFOV／VFOV），將像素座標轉換為實際空間長度，並以 OpenCV 進行輪廓分析。Using intrinsics and depth, HFOV/VFOV are computed dynamically to map pixels to real-world lengths, with OpenCV contour analysis.
• 運動學求解Kinematics
  以 DH 模型定義手臂參數，完成正向／逆向運動學與多軸角度控制，將目標座標轉為各關節角度。A DH model defines the arm; forward/inverse kinematics and multi-axis angle control turn target coordinates into joint angles.
• 抓取規劃（GGCNN）Grasp planning (GGCNN)
  以 GGCNN 演算法處理抓取場景，估計物體角度並辨識物體堆疊狀況，產生可行的抓取姿態。GGCNN handles the grasping scene — estimating object angle and detecting stacking — to produce feasible grasp poses.
• 手勢辨識（YOLO）Gesture recognition (YOLO)
  以 YOLO（TensorFlow／Keras）辨識手勢，作為操作指令的另一種輸入方式。YOLO (TensorFlow/Keras) recognizes hand gestures as an alternative command input.
• ROS2 整合與模擬ROS2 integration & simulation
  於 ROS2 架構下整合各模組，以 MoveIt 進行路徑規劃，並在 Webots 中模擬驗證後再與實機互通。Modules are integrated under ROS2, with MoveIt for path planning, validated in Webots simulation before bridging to the real arm.

• 完成手臂通訊、正逆向運動學、像素轉實際長度校正與物體辨識的完整流程。Completed the full pipeline of arm communication, forward/inverse kinematics, pixel-to-length calibration, and object recognition.
• 成功以純深度影像辨識並抓取物體，降低對顏色與光照的依賴。Grasped objects from depth images alone, reducing color/illumination dependence.
• 人機介面支援點選取放：指定 A、B 兩點取放，並可在搬運途中改變目的地。The HMI supports click-to-place: designate A/B pick-and-place points and re-target the destination mid-transfer.
• 於 Webots 完成模擬驗證，並達成 ROS2 與模擬軟體互通，朝非平面、堆疊等複雜場景擴展。Validated in Webots with ROS2–simulation interoperability, extending toward non-planar and stacked scenes.

手臂Arm

AR3 六軸機械手臂AR3 six-axis arm

相機Camera

Intel RealSense D435 (Depth)

深度學習Deep learning

GGCNN（抓取）、YOLO 手勢（TensorFlow／Keras）GGCNN (grasping), YOLO gesture (TensorFlow/Keras)

影像處理Vision

OpenCV 輪廓分析、HFOV／VFOV 視野計算OpenCV contours, HFOV/VFOV mapping

機器人Robotics

ROS2、MoveIt 路徑規劃、Webots 模擬ROS2, MoveIt planning, Webots simulation

運動學Kinematics

DH 模型、正／逆向運動學DH model, forward/inverse kinematics