Skip to main content

Chapter 4: Digital Twin Simulation (Gazebo + Isaac)

Learning Objectives

By the end of this chapter, you will be able to:

  • Understand the concept and benefits of digital twin simulation
  • Set up and configure Gazebo for robot simulation
  • Create robot models using URDF and SDF formats
  • Simulate sensors (cameras, LIDAR, IMU) in Gazebo
  • Understand NVIDIA Isaac Sim capabilities and use cases
  • Compare Gazebo and Isaac Sim for different applications
  • Implement sim-to-real transfer strategies
  • Debug and optimize simulation performance

4.1 What is Digital Twin Simulation?

A digital twin is a virtual replica of a physical system that mirrors its behavior, appearance, and dynamics. In robotics, digital twins enable:

4.1.1 Key Benefits

1. Safe Testing

  • Test dangerous scenarios without risk
  • Experiment with extreme conditions
  • Validate edge cases

2. Rapid Iteration

  • No hardware setup time
  • Instant environment changes
  • Parallel testing of multiple configurations

3. Cost Reduction

  • Reduce hardware wear and tear
  • Minimize prototype iterations
  • Lower testing infrastructure costs

4. Scalability

  • Train thousands of agents simultaneously
  • Test fleet coordination
  • Simulate large-scale deployments

5. Data Generation

  • Generate labeled training data
  • Create diverse scenarios
  • Augment real-world datasets

4.1.2 Simulation Fidelity Levels

LevelPhysicsGraphicsSensorsUse Case
LowBasicWireframeIdealEarly prototyping
MediumRigid bodyTexturedSimplifiedAlgorithm development
HighContact, frictionRealisticRay-tracedPre-deployment validation
PhotorealisticDeformablePath-tracedPhysicalSim-to-real transfer

4.2 Introduction to Gazebo

Gazebo is an open-source 3D robotics simulator that integrates tightly with ROS 2.

4.2.1 Gazebo Architecture

Gazebo Classic (Gazebo 11): Monolithic architecture

  • Single process
  • Mature, stable
  • Large plugin ecosystem

Gazebo (Ignition/Gz): Modular architecture

  • Separate libraries: gz-sim, gz-physics, gz-rendering
  • Modern design
  • Better performance
  • Recommended for new projects

4.2.2 Key Features

Physics Engines: ODE, Bullet, DART, Simbody
Rendering: OGRE 1.x/2.x for graphics
Sensors: Camera, depth, LIDAR, IMU, GPS, contact
Plugins: Extend functionality (custom sensors, controllers)
ROS 2 Integration: Native support via ros_gz_bridge
Distributed Simulation: Primary/replica for multi-robot

4.2.3 Installation

Gazebo (Ignition) Fortress or later:

# Ubuntu 22.04 (Humble)
sudo apt install ros-humble-ros-gz

# Or standalone
sudo apt install ignition-fortress

Verify installation:

gz sim --version

4.3 Robot Modeling with URDF

Unified Robot Description Format (URDF) is an XML format for describing robot kinematics and dynamics.

4.3.1 URDF Structure

Basic elements:

  • <robot>: Root element
  • <link>: Rigid body with visual, collision, inertial properties
  • <joint>: Connection between two links
  • <gazebo>: Gazebo-specific tags (plugins, sensors)

4.3.2 Simple Robot Example

<?xml version="1.0"?>
<robot name="simple_robot">

<!-- Base Link -->
<link name="base_link">
<visual>
<geometry>
<box size="0.5 0.3 0.2"/>
</geometry>
<material name="blue">
<color rgba="0 0 1 1"/>
</material>
</visual>
<collision>
<geometry>
<box size="0.5 0.3 0.2"/>
</geometry>
</collision>
<inertial>
<mass value="10.0"/>
<inertia ixx="0.1" ixy="0" ixz="0" iyy="0.2" iyz="0" izz="0.2"/>
</inertial>
</link>

<!-- Wheel Link -->
<link name="left_wheel">
<visual>
<geometry>
<cylinder radius="0.1" length="0.05"/>
</geometry>
<material name="black">
<color rgba="0 0 0 1"/>
</material>
</visual>
<collision>
<geometry>
<cylinder radius="0.1" length="0.05"/>
</geometry>
</collision>
<inertial>
<mass value="1.0"/>
<inertia ixx="0.01" ixy="0" ixz="0" iyy="0.01" iyz="0" izz="0.01"/>
</inertial>
</link>

<!-- Joint connecting base to wheel -->
<joint name="left_wheel_joint" type="continuous">
<parent link="base_link"/>
<child link="left_wheel"/>
<origin xyz="0.15 0.2 -0.1" rpy="1.57 0 0"/>
<axis xyz="0 0 1"/>
</joint>

</robot>

4.3.3 URDF Components Explained

Visual: What you see

  • Geometry: box, cylinder, sphere, mesh
  • Material: color, texture

Collision: What interacts physically

  • Often simpler than visual (performance)
  • Should enclose visual geometry

Inertial: Mass and inertia

  • Required for physics simulation
  • Calculated from geometry or measured

Joint Types:

  • fixed: No movement
  • revolute: Rotation with limits
  • continuous: Unlimited rotation (wheels)
  • prismatic: Linear motion (sliders)
  • floating: 6 DOF (rarely used)
  • planar: 2D motion in a plane

4.3.4 Using Xacro for Modularity

Xacro extends URDF with macros, variables, and math:

<?xml version="1.0"?>
<robot xmlns:xacro="http://www.ros.org/wiki/xacro" name="modular_robot">

<!-- Properties -->
<xacro:property name="wheel_radius" value="0.1"/>
<xacro:property name="wheel_width" value="0.05"/>

<!-- Macro for wheel -->
<xacro:macro name="wheel" params="name x y">
<link name="${name}_wheel">
<visual>
<geometry>
<cylinder radius="${wheel_radius}" length="${wheel_width}"/>
</geometry>
</visual>
<collision>
<geometry>
<cylinder radius="${wheel_radius}" length="${wheel_width}"/>
</geometry>
</collision>
<inertial>
<mass value="1.0"/>
<inertia ixx="0.01" ixy="0" ixz="0" iyy="0.01" iyz="0" izz="0.01"/>
</inertial>
</link>

<joint name="${name}_wheel_joint" type="continuous">
<parent link="base_link"/>
<child link="${name}_wheel"/>
<origin xyz="${x} ${y} 0" rpy="1.57 0 0"/>
<axis xyz="0 0 1"/>
</joint>
</xacro:macro>

<!-- Instantiate wheels -->
<xacro:wheel name="left_front" x="0.2" y="0.15"/>
<xacro:wheel name="right_front" x="0.2" y="-0.15"/>

</robot>

Convert to URDF:

xacro my_robot.urdf.xacro > my_robot.urdf

4.4 Simulation Description Format (SDF)

SDF is Gazebo's native format, more expressive than URDF.

4.4.1 URDF vs SDF

FeatureURDFSDF
PurposeRobot descriptionWorld + robot description
ScopeSingle robotMultiple models, environments
SensorsLimitedExtensive (camera, LIDAR, IMU, GPS, etc.)
PluginsVia <gazebo> tagsNative <plugin> tags
PhysicsBasicAdvanced (multiple engines, custom properties)
Nested ModelsNoYes

4.4.2 Simple SDF World

<?xml version="1.0"?>
<sdf version="1.9">
<world name="simple_world">

<!-- Physics -->
<physics name="1ms" type="ignored">
<max_step_size>0.001</max_step_size>
<real_time_factor>1.0</real_time_factor>
</physics>

<!-- Lighting -->
<light type="directional" name="sun">
<pose>0 0 10 0 0 0</pose>
<diffuse>0.8 0.8 0.8 1</diffuse>
<specular>0.2 0.2 0.2 1</specular>
<direction>-0.5 0.5 -1</direction>
</light>

<!-- Ground plane -->
<model name="ground_plane">
<static>true</static>
<link name="link">
<collision name="collision">
<geometry>
<plane>
<normal>0 0 1</normal>
</plane>
</geometry>
</collision>
<visual name="visual">
<geometry>
<plane>
<normal>0 0 1</normal>
<size>100 100</size>
</plane>
</geometry>
<material>
<ambient>0.8 0.8 0.8 1</ambient>
</material>
</visual>
</link>
</model>

<!-- Include robot model -->
<include>
<uri>model://my_robot</uri>
<pose>0 0 0.5 0 0 0</pose>
</include>

</world>
</sdf>

4.5 Simulating Sensors

4.5.1 Camera Sensor

URDF with Gazebo plugin:

<gazebo reference="camera_link">
<sensor name="camera" type="camera">
<camera>
<horizontal_fov>1.047</horizontal_fov>
<image>
<width>640</width>
<height>480</height>
<format>R8G8B8</format>
</image>
<clip>
<near>0.1</near>
<far>100</far>
</clip>
</camera>
<always_on>1</always_on>
<update_rate>30</update_rate>
<visualize>true</visualize>
<topic>camera</topic>
</sensor>
</gazebo>

ROS 2 Bridge (publish to /camera/image):

ros2 run ros_gz_bridge parameter_bridge /camera@sensor_msgs/msg/Image@gz.msgs.Image

4.5.2 Depth Camera

<sensor name="depth_camera" type="depth_camera">
<camera>
<horizontal_fov>1.047</horizontal_fov>
<image>
<width>640</width>
<height>480</height>
</image>
<clip>
<near>0.1</near>
<far>10</far>
</clip>
</camera>
<always_on>1</always_on>
<update_rate>30</update_rate>
<topic>depth_camera</topic>
</sensor>

4.5.3 LIDAR (Laser Scan)

<sensor name="lidar" type="gpu_lidar">
<lidar>
<scan>
<horizontal>
<samples>360</samples>
<resolution>1</resolution>
<min_angle>-3.14159</min_angle>
<max_angle>3.14159</max_angle>
</horizontal>
</scan>
<range>
<min>0.1</min>
<max>30.0</max>
<resolution>0.01</resolution>
</range>
<noise>
<type>gaussian</type>
<mean>0.0</mean>
<stddev>0.01</stddev>
</noise>
</lidar>
<always_on>1</always_on>
<update_rate>10</update_rate>
<topic>lidar</topic>
</sensor>

4.5.4 IMU (Inertial Measurement Unit)

<sensor name="imu" type="imu">
<imu>
<angular_velocity>
<x>
<noise type="gaussian">
<mean>0.0</mean>
<stddev>0.01</stddev>
</noise>
</x>
<y>
<noise type="gaussian">
<mean>0.0</mean>
<stddev>0.01</stddev>
</noise>
</y>
<z>
<noise type="gaussian">
<mean>0.0</mean>
<stddev>0.01</stddev>
</noise>
</z>
</angular_velocity>
<linear_acceleration>
<x>
<noise type="gaussian">
<mean>0.0</mean>
<stddev>0.1</stddev>
</noise>
</x>
<y>
<noise type="gaussian">
<mean>0.0</mean>
<stddev>0.1</stddev>
</noise>
</y>
<z>
<noise type="gaussian">
<mean>0.0</mean>
<stddev>0.1</stddev>
</noise>
</z>
</linear_acceleration>
</imu>
<always_on>1</always_on>
<update_rate>100</update_rate>
<topic>imu</topic>
</sensor>

4.5.5 Sensor Noise Modeling

Adding realistic noise improves sim-to-real transfer:

Types:

  • gaussian: Normal distribution (most common)
  • uniform: Random within range
  • salt_pepper: Random outliers (image noise)

Parameters:

  • mean: Bias
  • stddev: Standard deviation
  • precision: Inverse variance

4.6 Gazebo Plugins

Plugins extend Gazebo functionality for custom behaviors.

4.6.1 Differential Drive Plugin

<gazebo>
<plugin filename="gz-sim-diff-drive-system" name="gz::sim::systems::DiffDrive">
<left_joint>left_wheel_joint</left_joint>
<right_joint>right_wheel_joint</right_joint>
<wheel_separation>0.4</wheel_separation>
<wheel_radius>0.1</wheel_radius>
<odom_publish_frequency>50</odom_publish_frequency>
<topic>cmd_vel</topic>
<odom_topic>odom</odom_topic>
</plugin>
</gazebo>

Control from ROS 2:

ros2 topic pub /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 0.5}, angular: {z: 0.2}}"

4.6.2 Joint State Publisher

<gazebo>
<plugin filename="gz-sim-joint-state-publisher-system"
name="gz::sim::systems::JointStatePublisher">
<joint_name>left_wheel_joint</joint_name>
<joint_name>right_wheel_joint</joint_name>
<topic>joint_states</topic>
<update_rate>50</update_rate>
</plugin>
</gazebo>

4.6.3 Contact Sensor Plugin

<gazebo reference="foot_link">
<sensor name="foot_contact" type="contact">
<contact>
<collision>foot_collision</collision>
</contact>
<update_rate>100</update_rate>
<topic>foot_contact</topic>
</sensor>
</gazebo>

4.7 Running Gazebo with ROS 2

4.7.1 Launch File Integration

from launch import LaunchDescription
from launch.actions import IncludeLaunchDescription
from launch.launch_description_sources import PythonLaunchDescriptionSource
from launch_ros.actions import Node
from ament_index_python.packages import get_package_share_directory
import os

def generate_launch_description():
pkg_share = get_package_share_directory('my_robot_gazebo')

# Gazebo launch
gazebo = IncludeLaunchDescription(
PythonLaunchDescriptionSource([
os.path.join(get_package_share_directory('ros_gz_sim'),
'launch', 'gz_sim.launch.py')
]),
launch_arguments={'gz_args': '-r empty.sdf'}.items()
)

# Spawn robot
spawn_robot = Node(
package='ros_gz_sim',
executable='create',
arguments=[
'-name', 'my_robot',
'-topic', '/robot_description',
'-z', '0.5'
],
output='screen'
)

# Robot state publisher
robot_state_publisher = Node(
package='robot_state_publisher',
executable='robot_state_publisher',
parameters=[{'robot_description': open('my_robot.urdf').read()}]
)

# ROS-Gazebo bridge
bridge = Node(
package='ros_gz_bridge',
executable='parameter_bridge',
arguments=[
'/cmd_vel@geometry_msgs/msg/Twist@gz.msgs.Twist',
'/odom@nav_msgs/msg/Odometry@gz.msgs.Odometry',
'/camera@sensor_msgs/msg/Image@gz.msgs.Image',
'/lidar@sensor_msgs/msg/LaserScan@gz.msgs.LaserScan'
],
output='screen'
)

return LaunchDescription([
gazebo,
robot_state_publisher,
spawn_robot,
bridge
])

4.7.2 Running the Simulation

# Launch
ros2 launch my_robot_gazebo simulation.launch.py

# View topics
ros2 topic list

# Control robot
ros2 topic pub /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 1.0}}"

# Visualize in RViz
rviz2

4.8 Introduction to NVIDIA Isaac Sim

NVIDIA Isaac Sim is a photorealistic robot simulator built on NVIDIA Omniverse.

4.8.1 Key Advantages

1. Photorealism

  • Real-time ray tracing (RTX)
  • Physically-based rendering (PBR)
  • High-fidelity materials and lighting

2. Physics Accuracy

  • NVIDIA PhysX 5
  • GPU-accelerated simulation
  • Deformable objects, fluids, soft bodies

3. AI Integration

  • Synthetic data generation (domain randomization)
  • Isaac Gym for RL training
  • TensorRT for inference

4. Scalability

  • Parallel environments (thousands of robots)
  • Cloud deployment
  • Distributed simulation

4.8.2 Use Cases

Perception Training: Generate labeled synthetic data
Reinforcement Learning: Train policies in parallel
Manipulation: High-fidelity contact simulation
Autonomous Vehicles: Photorealistic sensor simulation
Warehouse Automation: Fleet coordination testing

4.8.3 System Requirements

  • GPU: RTX 2070 or higher (RTX 3080+ recommended)
  • RAM: 32 GB minimum
  • OS: Ubuntu 20.04/22.04 or Windows 10/11
  • Storage: 50 GB

4.8.4 Installation

# Download Isaac Sim from NVIDIA Omniverse
# Install via Omniverse Launcher

# Or headless (cloud/server)
wget https://developer.nvidia.com/isaac-sim
./isaac-sim.sh --headless

4.8.5 Python API Example

from omni.isaac.kit import SimulationApp

# Initialize
simulation_app = SimulationApp({"headless": False})

from omni.isaac.core import World
from omni.isaac.core.objects import DynamicCuboid
from omni.isaac.core.robots import Robot

# Create world
world = World()

# Add robot
robot = world.scene.add(
Robot(
prim_path="/World/Robot",
name="my_robot",
usd_path="path/to/robot.usd"
)
)

# Add object
cube = world.scene.add(
DynamicCuboid(
prim_path="/World/Cube",
name="cube",
position=[1.0, 0, 0.5],
size=0.1
)
)

# Reset world
world.reset()

# Simulation loop
for i in range(1000):
world.step(render=True)

# Get robot state
position, orientation = robot.get_world_pose()
print(f"Robot at: {position}")

simulation_app.close()

4.9 Gazebo vs Isaac Sim Comparison

FeatureGazeboIsaac Sim
CostFree, open-sourceFree for individuals/research
HardwareCPU-friendlyRequires RTX GPU
PhysicsODE, Bullet, DARTPhysX 5 (GPU-accelerated)
GraphicsOGRE (good)RTX ray tracing (photorealistic)
ROS IntegrationNative (ROS 1/2)Via Isaac ROS
SensorsStandard roboticsPhotorealistic + standard
Parallel SimLimitedThousands of environments
Domain RandomizationManualBuilt-in tools
Learning CurveModerateSteep
CommunityLarge, matureGrowing
Best ForGeneral robotics devPerception, RL, manipulation

4.9.1 When to Use Gazebo

✅ CPU-only machines
✅ Standard mobile robots
✅ ROS 2-centric workflows
✅ Quick prototyping
✅ Community plugins/models

4.9.2 When to Use Isaac Sim

✅ RTX GPU available
✅ Perception pipeline development
✅ Reinforcement learning training
✅ High-fidelity manipulation
✅ Synthetic data generation
✅ Large-scale parallel training


4.10 Sim-to-Real Transfer

The sim-to-real gap is the difference between simulated and real-world performance.

4.10.1 Sources of Sim-to-Real Gap

Physics Modeling:

  • Friction coefficients
  • Contact dynamics
  • Deformation and compliance
  • Actuator delays and backlash

Sensor Differences:

  • Noise characteristics
  • Calibration errors
  • Lighting variations
  • Occlusions and reflections

Environmental Factors:

  • Surface textures
  • Object properties
  • Weather conditions
  • Human interactions

4.10.2 Strategies to Bridge the Gap

1. Domain Randomization Vary simulation parameters during training:

  • Lighting (intensity, color, direction)
  • Textures (floors, walls, objects)
  • Object properties (mass, friction, size)
  • Sensor noise (mean, variance)
  • Actuator dynamics (delays, gains)

Example in Isaac Sim:

from omni.isaac.core.utils.prims import create_prim
from omni.isaac.core.utils.semantics import add_update_semantics

# Randomize lighting
light_intensity = random.uniform(500, 2000)
light_color = [random.uniform(0.8, 1.0) for _ in range(3)]

# Randomize object properties
mass = random.uniform(0.5, 2.0)
friction = random.uniform(0.3, 0.9)

2. System Identification Measure real robot parameters:

# Measure friction coefficient
measured_friction = measure_friction_coefficient(robot)

# Update simulation
set_friction_coefficient(sim_robot, measured_friction)

3. Sim-to-Real Adaptation Fine-tune in real world:

  • Collect small real-world dataset
  • Fine-tune sim-trained policy
  • Online adaptation during deployment

4. Robust Control Design controllers tolerant to model errors:

  • Adaptive control
  • Model predictive control with uncertainty
  • Robust optimization

5. Realistic Sensor Modeling Add noise that matches real sensors:

<noise type="gaussian">
<mean>0.0</mean>
<stddev>0.02</stddev> <!-- Measured from real sensor -->
</noise>

4.11 Performance Optimization

4.11.1 Gazebo Optimization

Reduce Visual Complexity:

  • Use simple collision meshes
  • Lower polygon count for visuals
  • Disable shadows if not needed

Physics Tuning:

<physics name="fast" type="ode">
<max_step_size>0.01</max_step_size> <!-- Larger = faster -->
<real_time_factor>1.0</real_time_factor>
<max_contacts>10</max_contacts> <!-- Fewer = faster -->
</physics>

Sensor Update Rates:

  • Camera: 10-30 Hz (not 60 Hz unless needed)
  • LIDAR: 10-20 Hz
  • IMU: 100-200 Hz

Parallel Simulation: Use multiple Gazebo instances for different scenarios.

4.11.2 Isaac Sim Optimization

Use GPU Acceleration:

from omni.isaac.core.utils.extensions import enable_extension
enable_extension("omni.isaac.core_nodes")

Headless Mode (no GUI):

simulation_app = SimulationApp({"headless": True})

Reduce Rendering Quality (if graphics not critical):

simulation_app.update_app_setting("rtx/post/aa/op", 0)  # Disable anti-aliasing
simulation_app.update_app_setting("rtx/reflections/enabled", False)

4.12 Chapter Summary

Digital twin simulation enables safe, rapid, cost-effective robot development through virtual replicas. Gazebo is an open-source simulator with tight ROS 2 integration, supporting various physics engines and sensors. NVIDIA Isaac Sim offers photorealistic graphics and GPU-accelerated physics for perception and reinforcement learning applications.

Robot models are described using URDF (ROS standard) or SDF (Gazebo native). URDF defines links, joints, and basic properties, while Xacro adds modularity through macros. SDF extends capabilities with advanced sensors, plugins, and world descriptions.

Sensor simulation includes cameras, depth sensors, LIDAR, and IMUs with realistic noise modeling. Gazebo plugins enable differential drive, joint state publishing, and contact sensing. ROS 2 integration uses ros_gz_bridge to connect simulation topics with robot software.

Isaac Sim excels in photorealism, parallel simulation, and AI integration, requiring RTX GPUs. Gazebo is CPU-friendly and better for general robotics development. Choose based on hardware, use case, and required fidelity.

The sim-to-real gap arises from physics modeling, sensor differences, and environmental factors. Bridge it through domain randomization, system identification, sim-to-real adaptation, robust control, and realistic sensor modeling. Optimize performance by tuning physics parameters, reducing visual complexity, and leveraging GPU acceleration.


Key Takeaways

✅ Digital twins enable safe, scalable, cost-effective robot testing
✅ Gazebo: Open-source, ROS 2-native, CPU-friendly, mature ecosystem
✅ Isaac Sim: Photorealistic, GPU-accelerated, AI-focused, requires RTX
✅ URDF describes robot kinematics; SDF adds sensors and world modeling
✅ Xacro provides modularity through macros and variables
✅ Simulate cameras, depth, LIDAR, IMU with realistic noise
✅ Gazebo plugins enable custom behaviors (diff drive, joint states)
✅ ros_gz_bridge connects Gazebo topics to ROS 2
✅ Sim-to-real gap: Address with domain randomization, system ID, adaptation
✅ Optimize: Tune physics, reduce complexity, use GPU acceleration


Further Reading

  • Gazebo Documentation:

    • Gazebo (Ignition) Tutorials (gazebosim.org/docs)
    • ROS 2 + Gazebo Integration Guide
    • SDF Specification (sdformat.org)
  • Isaac Sim:

    • NVIDIA Isaac Sim Documentation
    • Isaac ROS Documentation
    • Omniverse USD Documentation
  • Books & Papers:

    • "Learning Dexterous Manipulation from Suboptimal Demonstrations" (OpenAI)
    • "Sim-to-Real Transfer of Robotic Control with Dynamics Randomization" (Peng et al.)
  • Community:

    • Gazebo Community (community.gazebosim.org)
    • NVIDIA Isaac Forums
    • ROS Discourse Simulation Category

Next Chapter: Vision-Language-Action Systems – We'll explore how modern AI models enable robots to understand natural language commands and perform complex tasks through multimodal perception.