Chapter 4: Digital Twin Simulation (Gazebo + Isaac)
Learning Objectives
By the end of this chapter, you will be able to:
- Understand the concept and benefits of digital twin simulation
- Set up and configure Gazebo for robot simulation
- Create robot models using URDF and SDF formats
- Simulate sensors (cameras, LIDAR, IMU) in Gazebo
- Understand NVIDIA Isaac Sim capabilities and use cases
- Compare Gazebo and Isaac Sim for different applications
- Implement sim-to-real transfer strategies
- Debug and optimize simulation performance
4.1 What is Digital Twin Simulation?
A digital twin is a virtual replica of a physical system that mirrors its behavior, appearance, and dynamics. In robotics, digital twins enable:
4.1.1 Key Benefits
1. Safe Testing
- Test dangerous scenarios without risk
- Experiment with extreme conditions
- Validate edge cases
2. Rapid Iteration
- No hardware setup time
- Instant environment changes
- Parallel testing of multiple configurations
3. Cost Reduction
- Reduce hardware wear and tear
- Minimize prototype iterations
- Lower testing infrastructure costs
4. Scalability
- Train thousands of agents simultaneously
- Test fleet coordination
- Simulate large-scale deployments
5. Data Generation
- Generate labeled training data
- Create diverse scenarios
- Augment real-world datasets
4.1.2 Simulation Fidelity Levels
| Level | Physics | Graphics | Sensors | Use Case |
|---|---|---|---|---|
| Low | Basic | Wireframe | Ideal | Early prototyping |
| Medium | Rigid body | Textured | Simplified | Algorithm development |
| High | Contact, friction | Realistic | Ray-traced | Pre-deployment validation |
| Photorealistic | Deformable | Path-traced | Physical | Sim-to-real transfer |
4.2 Introduction to Gazebo
Gazebo is an open-source 3D robotics simulator that integrates tightly with ROS 2.
4.2.1 Gazebo Architecture
Gazebo Classic (Gazebo 11): Monolithic architecture
- Single process
- Mature, stable
- Large plugin ecosystem
Gazebo (Ignition/Gz): Modular architecture
- Separate libraries: gz-sim, gz-physics, gz-rendering
- Modern design
- Better performance
- Recommended for new projects
4.2.2 Key Features
✅ Physics Engines: ODE, Bullet, DART, Simbody
✅ Rendering: OGRE 1.x/2.x for graphics
✅ Sensors: Camera, depth, LIDAR, IMU, GPS, contact
✅ Plugins: Extend functionality (custom sensors, controllers)
✅ ROS 2 Integration: Native support via ros_gz_bridge
✅ Distributed Simulation: Primary/replica for multi-robot
4.2.3 Installation
Gazebo (Ignition) Fortress or later:
# Ubuntu 22.04 (Humble)
sudo apt install ros-humble-ros-gz
# Or standalone
sudo apt install ignition-fortress
Verify installation:
gz sim --version
4.3 Robot Modeling with URDF
Unified Robot Description Format (URDF) is an XML format for describing robot kinematics and dynamics.
4.3.1 URDF Structure
Basic elements:
<robot>: Root element<link>: Rigid body with visual, collision, inertial properties<joint>: Connection between two links<gazebo>: Gazebo-specific tags (plugins, sensors)
4.3.2 Simple Robot Example
<?xml version="1.0"?>
<robot name="simple_robot">
<!-- Base Link -->
<link name="base_link">
<visual>
<geometry>
<box size="0.5 0.3 0.2"/>
</geometry>
<material name="blue">
<color rgba="0 0 1 1"/>
</material>
</visual>
<collision>
<geometry>
<box size="0.5 0.3 0.2"/>
</geometry>
</collision>
<inertial>
<mass value="10.0"/>
<inertia ixx="0.1" ixy="0" ixz="0" iyy="0.2" iyz="0" izz="0.2"/>
</inertial>
</link>
<!-- Wheel Link -->
<link name="left_wheel">
<visual>
<geometry>
<cylinder radius="0.1" length="0.05"/>
</geometry>
<material name="black">
<color rgba="0 0 0 1"/>
</material>
</visual>
<collision>
<geometry>
<cylinder radius="0.1" length="0.05"/>
</geometry>
</collision>
<inertial>
<mass value="1.0"/>
<inertia ixx="0.01" ixy="0" ixz="0" iyy="0.01" iyz="0" izz="0.01"/>
</inertial>
</link>
<!-- Joint connecting base to wheel -->
<joint name="left_wheel_joint" type="continuous">
<parent link="base_link"/>
<child link="left_wheel"/>
<origin xyz="0.15 0.2 -0.1" rpy="1.57 0 0"/>
<axis xyz="0 0 1"/>
</joint>
</robot>
4.3.3 URDF Components Explained
Visual: What you see
- Geometry: box, cylinder, sphere, mesh
- Material: color, texture
Collision: What interacts physically
- Often simpler than visual (performance)
- Should enclose visual geometry
Inertial: Mass and inertia
- Required for physics simulation
- Calculated from geometry or measured
Joint Types:
fixed: No movementrevolute: Rotation with limitscontinuous: Unlimited rotation (wheels)prismatic: Linear motion (sliders)floating: 6 DOF (rarely used)planar: 2D motion in a plane
4.3.4 Using Xacro for Modularity
Xacro extends URDF with macros, variables, and math:
<?xml version="1.0"?>
<robot xmlns:xacro="http://www.ros.org/wiki/xacro" name="modular_robot">
<!-- Properties -->
<xacro:property name="wheel_radius" value="0.1"/>
<xacro:property name="wheel_width" value="0.05"/>
<!-- Macro for wheel -->
<xacro:macro name="wheel" params="name x y">
<link name="${name}_wheel">
<visual>
<geometry>
<cylinder radius="${wheel_radius}" length="${wheel_width}"/>
</geometry>
</visual>
<collision>
<geometry>
<cylinder radius="${wheel_radius}" length="${wheel_width}"/>
</geometry>
</collision>
<inertial>
<mass value="1.0"/>
<inertia ixx="0.01" ixy="0" ixz="0" iyy="0.01" iyz="0" izz="0.01"/>
</inertial>
</link>
<joint name="${name}_wheel_joint" type="continuous">
<parent link="base_link"/>
<child link="${name}_wheel"/>
<origin xyz="${x} ${y} 0" rpy="1.57 0 0"/>
<axis xyz="0 0 1"/>
</joint>
</xacro:macro>
<!-- Instantiate wheels -->
<xacro:wheel name="left_front" x="0.2" y="0.15"/>
<xacro:wheel name="right_front" x="0.2" y="-0.15"/>
</robot>
Convert to URDF:
xacro my_robot.urdf.xacro > my_robot.urdf
4.4 Simulation Description Format (SDF)
SDF is Gazebo's native format, more expressive than URDF.
4.4.1 URDF vs SDF
| Feature | URDF | SDF |
|---|---|---|
| Purpose | Robot description | World + robot description |
| Scope | Single robot | Multiple models, environments |
| Sensors | Limited | Extensive (camera, LIDAR, IMU, GPS, etc.) |
| Plugins | Via <gazebo> tags | Native <plugin> tags |
| Physics | Basic | Advanced (multiple engines, custom properties) |
| Nested Models | No | Yes |
4.4.2 Simple SDF World
<?xml version="1.0"?>
<sdf version="1.9">
<world name="simple_world">
<!-- Physics -->
<physics name="1ms" type="ignored">
<max_step_size>0.001</max_step_size>
<real_time_factor>1.0</real_time_factor>
</physics>
<!-- Lighting -->
<light type="directional" name="sun">
<pose>0 0 10 0 0 0</pose>
<diffuse>0.8 0.8 0.8 1</diffuse>
<specular>0.2 0.2 0.2 1</specular>
<direction>-0.5 0.5 -1</direction>
</light>
<!-- Ground plane -->
<model name="ground_plane">
<static>true</static>
<link name="link">
<collision name="collision">
<geometry>
<plane>
<normal>0 0 1</normal>
</plane>
</geometry>
</collision>
<visual name="visual">
<geometry>
<plane>
<normal>0 0 1</normal>
<size>100 100</size>
</plane>
</geometry>
<material>
<ambient>0.8 0.8 0.8 1</ambient>
</material>
</visual>
</link>
</model>
<!-- Include robot model -->
<include>
<uri>model://my_robot</uri>
<pose>0 0 0.5 0 0 0</pose>
</include>
</world>
</sdf>
4.5 Simulating Sensors
4.5.1 Camera Sensor
URDF with Gazebo plugin:
<gazebo reference="camera_link">
<sensor name="camera" type="camera">
<camera>
<horizontal_fov>1.047</horizontal_fov>
<image>
<width>640</width>
<height>480</height>
<format>R8G8B8</format>
</image>
<clip>
<near>0.1</near>
<far>100</far>
</clip>
</camera>
<always_on>1</always_on>
<update_rate>30</update_rate>
<visualize>true</visualize>
<topic>camera</topic>
</sensor>
</gazebo>
ROS 2 Bridge (publish to /camera/image):
ros2 run ros_gz_bridge parameter_bridge /camera@sensor_msgs/msg/Image@gz.msgs.Image
4.5.2 Depth Camera
<sensor name="depth_camera" type="depth_camera">
<camera>
<horizontal_fov>1.047</horizontal_fov>
<image>
<width>640</width>
<height>480</height>
</image>
<clip>
<near>0.1</near>
<far>10</far>
</clip>
</camera>
<always_on>1</always_on>
<update_rate>30</update_rate>
<topic>depth_camera</topic>
</sensor>
4.5.3 LIDAR (Laser Scan)
<sensor name="lidar" type="gpu_lidar">
<lidar>
<scan>
<horizontal>
<samples>360</samples>
<resolution>1</resolution>
<min_angle>-3.14159</min_angle>
<max_angle>3.14159</max_angle>
</horizontal>
</scan>
<range>
<min>0.1</min>
<max>30.0</max>
<resolution>0.01</resolution>
</range>
<noise>
<type>gaussian</type>
<mean>0.0</mean>
<stddev>0.01</stddev>
</noise>
</lidar>
<always_on>1</always_on>
<update_rate>10</update_rate>
<topic>lidar</topic>
</sensor>
4.5.4 IMU (Inertial Measurement Unit)
<sensor name="imu" type="imu">
<imu>
<angular_velocity>
<x>
<noise type="gaussian">
<mean>0.0</mean>
<stddev>0.01</stddev>
</noise>
</x>
<y>
<noise type="gaussian">
<mean>0.0</mean>
<stddev>0.01</stddev>
</noise>
</y>
<z>
<noise type="gaussian">
<mean>0.0</mean>
<stddev>0.01</stddev>
</noise>
</z>
</angular_velocity>
<linear_acceleration>
<x>
<noise type="gaussian">
<mean>0.0</mean>
<stddev>0.1</stddev>
</noise>
</x>
<y>
<noise type="gaussian">
<mean>0.0</mean>
<stddev>0.1</stddev>
</noise>
</y>
<z>
<noise type="gaussian">
<mean>0.0</mean>
<stddev>0.1</stddev>
</noise>
</z>
</linear_acceleration>
</imu>
<always_on>1</always_on>
<update_rate>100</update_rate>
<topic>imu</topic>
</sensor>
4.5.5 Sensor Noise Modeling
Adding realistic noise improves sim-to-real transfer:
Types:
gaussian: Normal distribution (most common)uniform: Random within rangesalt_pepper: Random outliers (image noise)
Parameters:
mean: Biasstddev: Standard deviationprecision: Inverse variance
4.6 Gazebo Plugins
Plugins extend Gazebo functionality for custom behaviors.
4.6.1 Differential Drive Plugin
<gazebo>
<plugin filename="gz-sim-diff-drive-system" name="gz::sim::systems::DiffDrive">
<left_joint>left_wheel_joint</left_joint>
<right_joint>right_wheel_joint</right_joint>
<wheel_separation>0.4</wheel_separation>
<wheel_radius>0.1</wheel_radius>
<odom_publish_frequency>50</odom_publish_frequency>
<topic>cmd_vel</topic>
<odom_topic>odom</odom_topic>
</plugin>
</gazebo>
Control from ROS 2:
ros2 topic pub /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 0.5}, angular: {z: 0.2}}"
4.6.2 Joint State Publisher
<gazebo>
<plugin filename="gz-sim-joint-state-publisher-system"
name="gz::sim::systems::JointStatePublisher">
<joint_name>left_wheel_joint</joint_name>
<joint_name>right_wheel_joint</joint_name>
<topic>joint_states</topic>
<update_rate>50</update_rate>
</plugin>
</gazebo>
4.6.3 Contact Sensor Plugin
<gazebo reference="foot_link">
<sensor name="foot_contact" type="contact">
<contact>
<collision>foot_collision</collision>
</contact>
<update_rate>100</update_rate>
<topic>foot_contact</topic>
</sensor>
</gazebo>
4.7 Running Gazebo with ROS 2
4.7.1 Launch File Integration
from launch import LaunchDescription
from launch.actions import IncludeLaunchDescription
from launch.launch_description_sources import PythonLaunchDescriptionSource
from launch_ros.actions import Node
from ament_index_python.packages import get_package_share_directory
import os
def generate_launch_description():
pkg_share = get_package_share_directory('my_robot_gazebo')
# Gazebo launch
gazebo = IncludeLaunchDescription(
PythonLaunchDescriptionSource([
os.path.join(get_package_share_directory('ros_gz_sim'),
'launch', 'gz_sim.launch.py')
]),
launch_arguments={'gz_args': '-r empty.sdf'}.items()
)
# Spawn robot
spawn_robot = Node(
package='ros_gz_sim',
executable='create',
arguments=[
'-name', 'my_robot',
'-topic', '/robot_description',
'-z', '0.5'
],
output='screen'
)
# Robot state publisher
robot_state_publisher = Node(
package='robot_state_publisher',
executable='robot_state_publisher',
parameters=[{'robot_description': open('my_robot.urdf').read()}]
)
# ROS-Gazebo bridge
bridge = Node(
package='ros_gz_bridge',
executable='parameter_bridge',
arguments=[
'/cmd_vel@geometry_msgs/msg/Twist@gz.msgs.Twist',
'/odom@nav_msgs/msg/Odometry@gz.msgs.Odometry',
'/camera@sensor_msgs/msg/Image@gz.msgs.Image',
'/lidar@sensor_msgs/msg/LaserScan@gz.msgs.LaserScan'
],
output='screen'
)
return LaunchDescription([
gazebo,
robot_state_publisher,
spawn_robot,
bridge
])
4.7.2 Running the Simulation
# Launch
ros2 launch my_robot_gazebo simulation.launch.py
# View topics
ros2 topic list
# Control robot
ros2 topic pub /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 1.0}}"
# Visualize in RViz
rviz2
4.8 Introduction to NVIDIA Isaac Sim
NVIDIA Isaac Sim is a photorealistic robot simulator built on NVIDIA Omniverse.
4.8.1 Key Advantages
1. Photorealism
- Real-time ray tracing (RTX)
- Physically-based rendering (PBR)
- High-fidelity materials and lighting
2. Physics Accuracy
- NVIDIA PhysX 5
- GPU-accelerated simulation
- Deformable objects, fluids, soft bodies
3. AI Integration
- Synthetic data generation (domain randomization)
- Isaac Gym for RL training
- TensorRT for inference
4. Scalability
- Parallel environments (thousands of robots)
- Cloud deployment
- Distributed simulation
4.8.2 Use Cases
✅ Perception Training: Generate labeled synthetic data
✅ Reinforcement Learning: Train policies in parallel
✅ Manipulation: High-fidelity contact simulation
✅ Autonomous Vehicles: Photorealistic sensor simulation
✅ Warehouse Automation: Fleet coordination testing
4.8.3 System Requirements
- GPU: RTX 2070 or higher (RTX 3080+ recommended)
- RAM: 32 GB minimum
- OS: Ubuntu 20.04/22.04 or Windows 10/11
- Storage: 50 GB
4.8.4 Installation
# Download Isaac Sim from NVIDIA Omniverse
# Install via Omniverse Launcher
# Or headless (cloud/server)
wget https://developer.nvidia.com/isaac-sim
./isaac-sim.sh --headless
4.8.5 Python API Example
from omni.isaac.kit import SimulationApp
# Initialize
simulation_app = SimulationApp({"headless": False})
from omni.isaac.core import World
from omni.isaac.core.objects import DynamicCuboid
from omni.isaac.core.robots import Robot
# Create world
world = World()
# Add robot
robot = world.scene.add(
Robot(
prim_path="/World/Robot",
name="my_robot",
usd_path="path/to/robot.usd"
)
)
# Add object
cube = world.scene.add(
DynamicCuboid(
prim_path="/World/Cube",
name="cube",
position=[1.0, 0, 0.5],
size=0.1
)
)
# Reset world
world.reset()
# Simulation loop
for i in range(1000):
world.step(render=True)
# Get robot state
position, orientation = robot.get_world_pose()
print(f"Robot at: {position}")
simulation_app.close()
4.9 Gazebo vs Isaac Sim Comparison
| Feature | Gazebo | Isaac Sim |
|---|---|---|
| Cost | Free, open-source | Free for individuals/research |
| Hardware | CPU-friendly | Requires RTX GPU |
| Physics | ODE, Bullet, DART | PhysX 5 (GPU-accelerated) |
| Graphics | OGRE (good) | RTX ray tracing (photorealistic) |
| ROS Integration | Native (ROS 1/2) | Via Isaac ROS |
| Sensors | Standard robotics | Photorealistic + standard |
| Parallel Sim | Limited | Thousands of environments |
| Domain Randomization | Manual | Built-in tools |
| Learning Curve | Moderate | Steep |
| Community | Large, mature | Growing |
| Best For | General robotics dev | Perception, RL, manipulation |
4.9.1 When to Use Gazebo
✅ CPU-only machines
✅ Standard mobile robots
✅ ROS 2-centric workflows
✅ Quick prototyping
✅ Community plugins/models
4.9.2 When to Use Isaac Sim
✅ RTX GPU available
✅ Perception pipeline development
✅ Reinforcement learning training
✅ High-fidelity manipulation
✅ Synthetic data generation
✅ Large-scale parallel training
4.10 Sim-to-Real Transfer
The sim-to-real gap is the difference between simulated and real-world performance.
4.10.1 Sources of Sim-to-Real Gap
Physics Modeling:
- Friction coefficients
- Contact dynamics
- Deformation and compliance
- Actuator delays and backlash
Sensor Differences:
- Noise characteristics
- Calibration errors
- Lighting variations
- Occlusions and reflections
Environmental Factors:
- Surface textures
- Object properties
- Weather conditions
- Human interactions
4.10.2 Strategies to Bridge the Gap
1. Domain Randomization Vary simulation parameters during training:
- Lighting (intensity, color, direction)
- Textures (floors, walls, objects)
- Object properties (mass, friction, size)
- Sensor noise (mean, variance)
- Actuator dynamics (delays, gains)
Example in Isaac Sim:
from omni.isaac.core.utils.prims import create_prim
from omni.isaac.core.utils.semantics import add_update_semantics
# Randomize lighting
light_intensity = random.uniform(500, 2000)
light_color = [random.uniform(0.8, 1.0) for _ in range(3)]
# Randomize object properties
mass = random.uniform(0.5, 2.0)
friction = random.uniform(0.3, 0.9)
2. System Identification Measure real robot parameters:
# Measure friction coefficient
measured_friction = measure_friction_coefficient(robot)
# Update simulation
set_friction_coefficient(sim_robot, measured_friction)
3. Sim-to-Real Adaptation Fine-tune in real world:
- Collect small real-world dataset
- Fine-tune sim-trained policy
- Online adaptation during deployment
4. Robust Control Design controllers tolerant to model errors:
- Adaptive control
- Model predictive control with uncertainty
- Robust optimization
5. Realistic Sensor Modeling Add noise that matches real sensors:
<noise type="gaussian">
<mean>0.0</mean>
<stddev>0.02</stddev> <!-- Measured from real sensor -->
</noise>
4.11 Performance Optimization
4.11.1 Gazebo Optimization
Reduce Visual Complexity:
- Use simple collision meshes
- Lower polygon count for visuals
- Disable shadows if not needed
Physics Tuning:
<physics name="fast" type="ode">
<max_step_size>0.01</max_step_size> <!-- Larger = faster -->
<real_time_factor>1.0</real_time_factor>
<max_contacts>10</max_contacts> <!-- Fewer = faster -->
</physics>
Sensor Update Rates:
- Camera: 10-30 Hz (not 60 Hz unless needed)
- LIDAR: 10-20 Hz
- IMU: 100-200 Hz
Parallel Simulation: Use multiple Gazebo instances for different scenarios.
4.11.2 Isaac Sim Optimization
Use GPU Acceleration:
from omni.isaac.core.utils.extensions import enable_extension
enable_extension("omni.isaac.core_nodes")
Headless Mode (no GUI):
simulation_app = SimulationApp({"headless": True})
Reduce Rendering Quality (if graphics not critical):
simulation_app.update_app_setting("rtx/post/aa/op", 0) # Disable anti-aliasing
simulation_app.update_app_setting("rtx/reflections/enabled", False)
4.12 Chapter Summary
Digital twin simulation enables safe, rapid, cost-effective robot development through virtual replicas. Gazebo is an open-source simulator with tight ROS 2 integration, supporting various physics engines and sensors. NVIDIA Isaac Sim offers photorealistic graphics and GPU-accelerated physics for perception and reinforcement learning applications.
Robot models are described using URDF (ROS standard) or SDF (Gazebo native). URDF defines links, joints, and basic properties, while Xacro adds modularity through macros. SDF extends capabilities with advanced sensors, plugins, and world descriptions.
Sensor simulation includes cameras, depth sensors, LIDAR, and IMUs with realistic noise modeling. Gazebo plugins enable differential drive, joint state publishing, and contact sensing. ROS 2 integration uses ros_gz_bridge to connect simulation topics with robot software.
Isaac Sim excels in photorealism, parallel simulation, and AI integration, requiring RTX GPUs. Gazebo is CPU-friendly and better for general robotics development. Choose based on hardware, use case, and required fidelity.
The sim-to-real gap arises from physics modeling, sensor differences, and environmental factors. Bridge it through domain randomization, system identification, sim-to-real adaptation, robust control, and realistic sensor modeling. Optimize performance by tuning physics parameters, reducing visual complexity, and leveraging GPU acceleration.
Key Takeaways
✅ Digital twins enable safe, scalable, cost-effective robot testing
✅ Gazebo: Open-source, ROS 2-native, CPU-friendly, mature ecosystem
✅ Isaac Sim: Photorealistic, GPU-accelerated, AI-focused, requires RTX
✅ URDF describes robot kinematics; SDF adds sensors and world modeling
✅ Xacro provides modularity through macros and variables
✅ Simulate cameras, depth, LIDAR, IMU with realistic noise
✅ Gazebo plugins enable custom behaviors (diff drive, joint states)
✅ ros_gz_bridge connects Gazebo topics to ROS 2
✅ Sim-to-real gap: Address with domain randomization, system ID, adaptation
✅ Optimize: Tune physics, reduce complexity, use GPU acceleration
Further Reading
-
Gazebo Documentation:
- Gazebo (Ignition) Tutorials (gazebosim.org/docs)
- ROS 2 + Gazebo Integration Guide
- SDF Specification (sdformat.org)
-
Isaac Sim:
- NVIDIA Isaac Sim Documentation
- Isaac ROS Documentation
- Omniverse USD Documentation
-
Books & Papers:
- "Learning Dexterous Manipulation from Suboptimal Demonstrations" (OpenAI)
- "Sim-to-Real Transfer of Robotic Control with Dynamics Randomization" (Peng et al.)
-
Community:
- Gazebo Community (community.gazebosim.org)
- NVIDIA Isaac Forums
- ROS Discourse Simulation Category
Next Chapter: Vision-Language-Action Systems – We'll explore how modern AI models enable robots to understand natural language commands and perform complex tasks through multimodal perception.