ROS2 Simulation and Vision

Simulation and Vision Overview

This module focuses on building perception workflows in simulation first, then moving them to real hardware with minimal changes.

Key goals:

  • Validate robot behavior in a controllable environment
  • Integrate camera/lidar data pipelines
  • Benchmark perception and control loops

Simulation Workflow

Environment setup

Typical stack:

  • Gazebo for physics simulation
  • RViz2 for visualization
  • ROS 2 nodes for control and perception

Basic loop

  1. Launch simulator
  2. Spawn robot model
  3. Start sensor and controller nodes
  4. Verify topics/TF in RViz2
  5. Record and replay with rosbag2 if needed

Useful checks:

bash
ros2 node list
ros2 topic list
ros2 topic hz /scan

Vision Pipeline Basics

Camera topics

Common outputs:

  • /camera/image_raw
  • /camera/camera_info

Inspect metadata:

bash
ros2 topic echo /camera/camera_info

Image transport and debugging

Recommended tools:

  • RViz2 Image display
  • rqt_image_view
  • ros2 topic hz for frame rate checks

Calibration and Coordinate Consistency

Camera calibration

Calibration provides intrinsic matrix and distortion parameters. Without calibration, pose estimation and 3D projection are unreliable.

TF alignment

Ensure camera frame is correctly connected in TF tree:

bash
ros2 run tf2_ros tf2_echo base_link camera_link

Misaligned TF leads to incorrect overlay and localization drift.

Practical Example: Sim-to-Real Validation

A practical validation checklist:

  1. Run perception node in simulation.
  2. Confirm output topic exists and updates at expected rate.
  3. Verify TF and visualization output.
  4. Replay the same input with rosbag2.
  5. Move to hardware and compare metrics.

Track at least:

  • Frame rate
  • End-to-end latency
  • CPU/GPU usage
  • Detection/tracking stability

Common Issues and Fixes

  1. No image in RViz2:
  • Check topic name and QoS profile.
  1. Delayed or dropped frames:
  • Reduce image resolution or processing load.
  1. Wrong object position:
  • Recheck camera calibration and TF chain.
  1. Works in sim but fails on hardware:
  • Confirm exposure, lighting, and sensor driver settings.

Next steps

After this module, continue with:

  1. Multi-sensor fusion
  2. Visual SLAM
  3. Navigation stack integration
  4. Real robot deployment and performance tuning