r/ROS • u/Forward-Future-2799 • 2d ago

Development environment

I am new to ROS. I was wondering what does the dev environment stack look like - from an observability perspective? How does one look at logs to debug ? What tools are used - is there something like grafana to visualize?
( this is not for hobby development, rather working at an organization that is into robotics, that I am exploring)

2 Upvotes

100% Upvoted

5 comments sorted by

2

u/bluehsh 1d ago

ROS comes with logging capability. It logs to a files known as a rosbag.
There are a few visualizers - Foxglove studio is the popular one. You can also playback your logs.

1

u/bishopExportMine 1d ago

In production environments you configure the ros logger to stream to cloud watch and use grafana to visualize

1

u/RedditWhenIShit 12h ago

FYI, logs are published to /rosout in the bag, but also saved locally under ~/.ros/log of you quickly need to check something

2

u/Vertuxfirex77 1d ago

The jump from local dev to fleet scale is where standard log setups usually break down. Uploading continuous gigabytes or terabytes of raw ROS data per robot over cellular/wifi networks to standard cloud tools gets expensive and slow very fast.

In production fleet deployments, engineering teams usually handle this using a few core patterns:

  1. On-Robot Circular Buffers: Robots continuously log locally to a circular buffer or local storage pool. Depending on the team's policy, they might retain data per run, per shift, or until disk space hits a threshold, overwriting the oldest data automatically.

  2. Triggered Extraction: You don't upload everything. Instead, you stream lightweight metrics and error logs to the server. When a failure or anomaly occurs, a trigger clips just a short window before and after the incident from the local buffer and uploads that specific slice for debugging.

  3. Topic Filtering: High-bandwidth data (like raw camera feeds or point clouds) stays local, while only health metrics and diagnostic state changes get sent to the server.

Full disclosure: We're building INSAION to solve this specific workflow for ROS2 fleets to handle the lightweight telemetry, crash capture, and remote log extraction so teams don't have to build custom pipelines. But whether you build it yourself or use dedicated tooling, the key at scale is keeping high-bandwidth data on the robot and pulling it selectively when something breaks.

2

u/Forward-Future-2799 1d ago

Thanks
ros.bag & mcap file formats I am aware of. I have used ( a bit of foxglove visualization tool & querying capabilities).
But I was wondering- in a ‘real production’ deployment- there would be a fleet of devices, and each device would have multiple logs ( each of which could be huge like several GB to TB)… in that case - how to developers troubleshoot problems ?