Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Extra Credit]: Added extensive robotics guides covering vision, SLAM and deep learning optimization #196

Open
wants to merge 20 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
aea7837
Added article for ZED setup
shahram95 Dec 6, 2024
0d36c0d
Added ZED ROS wrapper setup and troubleshooting article
shahram95 Dec 6, 2024
b1ae0ae
Added article for ZED neural depth mode
shahram95 Dec 6, 2024
fd1de2f
Added article for 6d pose estimation and tracking for velocity estima…
shahram95 Dec 6, 2024
931acff
Added article for model quantization and tensorrt optimization
shahram95 Dec 6, 2024
5a04376
Updated navigation.yml
shahram95 Dec 6, 2024
ab9b68e
Added CUDA PCL article
shahram95 Dec 6, 2024
e0f724a
Added yolo installation for arm architectures
shahram95 Dec 6, 2024
a86615b
Added ORB SLAM3 custom installation guide for ubuntu 20.04 and above
shahram95 Dec 6, 2024
518926d
Added comprehensive guide to albumentations
shahram95 Dec 6, 2024
f2fac8c
Added article for Kornia
shahram95 Dec 6, 2024
e58fb72
Added OLLAMA Franka Arm interfacing guide
shahram95 Dec 6, 2024
8cb06bd
Added article for interfacing ROS2 websocket visualization
shahram95 Dec 7, 2024
bb3973b
Added article on multitask learning starter code
shahram95 Dec 7, 2024
2aafc62
Added article for Kalman filtering and applications in visual tracking
shahram95 Dec 7, 2024
68e5d6d
Added article for knowledge distillation
shahram95 Dec 7, 2024
5d9effe
Added article for neural network model pruning
shahram95 Dec 7, 2024
39a9324
Added article for learning techniques for 3d datasets
shahram95 Dec 7, 2024
66d934a
Added article on optical flow classical to dl methods
shahram95 Dec 7, 2024
2ad48bd
Addressed review comments: Added page meta data; updated PR title and…
shahram95 Dec 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions _data/navigation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,12 @@ wiki:
url: /wiki/sensing/azure-block-detection/
- title: DWM1001 UltraWideband Positioning System
url: /wiki/sensing/ultrawideband-beacon-positioning.md
- title: Setting Up the ZED Camera with ROS and troubleshooting steps
url: /wiki/sensing/setting-up-the-zed-camera-with-ros.md
- title: Neural Depth Sensing in ZED Stereo Cameras
url: /wiki/sensing/neural-depth-sensing-in-zed.md
- title: Accelerating Point Cloud Processing with CUDA PCL (cuPCL)
url: /wiki/sensing/accelerating_point_cloud_processing_with_cuda_pcl.md
- title: Actuation
url: /wiki/actuation/
children:
Expand Down Expand Up @@ -162,6 +168,30 @@ wiki:
url: /wiki/machine-learning/mediapipe-live-ml-anywhere.md/
- title: NLP for robotics
url: /wiki/machine-learning/nlp_for_robotics.md/
- title: 6D Pose Estimation with YOLO and ZED
url: /wiki/machine-learning/6d_pose_tracking_yolo_zed.md
- title: Practical Guide to Model Quantization and TensorRT Optimization
url: /wiki/machine-learning/practical-guide-to-model-quantization-and-tensorrt-optimization.md
- title: Installing YOLO on ARM Architecture Devices
url: /wiki/machine-learning/installing-yolo-on-arm-architecture-devices.md
- title: Comprehensive guide to albumentations
url: /wiki/machine-learning/comprehensive-guide-to-albumentations.md
- title: Kornia technical guide
url: /wiki/machine-learning/kornia-technical-guide.md
- title: Integrating OLLAMA LLMs with Franka Arm
url: /wiki/machine-learning/integrating-ollama-llms-with-franka-arm.md
- title: Multi-task learning A starter guide
url: /wiki/machine-learning/multitask-learning-starter.md
- title: Understanding Kalman Filters and Visual Tracking
url: /wiki/machine-learning/understanding-kalman-filters-and-visual-tracking.md
- title: Knowledge Distillation practical implementation guide
url: /wiki/machine-learning/knowledge-distillation-practical-implementation-guide.md
- title: Neural Network optimization using model pruning
url: /wiki/machine-learning/neural-network-optimization-using-model-pruning.md
- title: Deep learning techniques for 3D datasets
url: /wiki/machine-learning/deep-learning-techniques-for-3d-datasets.md
- title: Optical Flow - Classical to Deep Learning Implementation
url: /wiki/machine-learning/optical-flow-classical-to-deep-learning-implementation.md
- title: State Estimation
url: /wiki/state-estimation/
children:
Expand All @@ -179,6 +209,8 @@ wiki:
url: /wiki/state-estimation/Cartographer-ROS-Integration/
- title: Externally Referenced State Estimation for GPS Lacking Environments
url: /wiki/state-estimation/gps-lacking-state-estimation-sensors.md
- title: Complete Guide to Installing ORB SLAM3
url: /wiki/state-estimation/orb-slam3-setup-and-troubleshoot-guide.md
- title: Programming
url: /wiki/programming/
children:
Expand Down Expand Up @@ -235,6 +267,8 @@ wiki:
url: /wiki/interfacing/microros-for-ros2-on-microcontrollers/
- title: ROS 1 - ROS 2 Bridge
url: /wiki/interfacing/ros1_ros2_bridge/
- title: Interfacing Streamlit, ROS2, and HTML/CSS/JS for visualization
url: /wiki/interfacing/interfacing-streamlit-ros2-and-html-css-js-for-visualization.md
- title: Computing
url: /wiki/computing/
children:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,218 @@
# Interfacing Streamlit, ROS2, and HTML/CSS/JS for Visualizations

## Introduction

Modern robotics applications demand sophisticated visualization and control interfaces that can handle real-time data while providing rich interactive features. This guide explores the integration of Streamlit, ROS2, and web technologies to create a powerful, real-time robotics visualization dashboard. By combining these technologies, we can create interfaces that are both functional and user-friendly, while maintaining the robust communication capabilities required for robotics applications.

## System Architecture Overview

The integration of Streamlit with ROS2 and web visualizations creates a multi-layered architecture that balances performance with functionality. At its core, the system uses ROS2 for reliable robotics communication, Streamlit for rapid interface development, and custom web components for rich visualizations. This architecture enables real-time data flow while maintaining system responsiveness and user interaction capabilities.

The key components interact through a carefully designed communication layer that consists of:

The ROS2 Backend manages robot communication and data flow, serving as the foundation for all robotics operations. The Streamlit Frontend provides the user interface framework, enabling rapid development of interactive features. Custom Web Components enable rich interactive visualizations, while the State Management system coordinates data flow between all components.

### Technology Stack Deep Dive

The system relies on several key technologies, each serving a specific purpose in the architecture. ROS2 provides the foundation for robotic system communication, offering reliable publish-subscribe patterns and service-based interactions. Streamlit serves as the primary web framework, chosen for its Python-native approach and rapid development capabilities. The WebSocket protocol enables real-time communication between the web interface and ROS2 system, while custom JavaScript components provide rich visualization capabilities.

## Implementation Guide

### 1. Foundation Setup

The project structure needs to support both development workflow and runtime requirements. Here's the recommended organization:

```plaintext
project_root/
├── src/
│ ├── frontend/
│ │ ├── components/
│ │ ├── pages/
│ ├── ros/
│ │ ├── nodes/
│ └── shared/
├── static/
└── config/
```

This structure provides clear separation of concerns while maintaining easy access to shared resources. Each directory serves a specific purpose in the application's architecture, allowing for modular development and easy maintenance.

### 2. ROS2 Integration Layer

The ROS2 integration represents a critical aspect of the system. Here's a comprehensive implementation:

```python
class ROSInterface:
def __init__(self):
# Initialize ROS2 node
rclpy.init()
self.node = Node('visualization_interface')

# Thread-safe state management
self.state_lock = threading.Lock()
self.shared_state = {}

# Set up message filters and synchronization
self.sync_filters = {}
self.subscribers = {}

# Initialize communication interfaces
self._setup_communications()

def _setup_communications(self):
"""Configure ROS2 publishers/subscribers with thread safety"""
# Set up main data subscribers
self.subscribers['telemetry'] = self.node.create_subscription(
TelemetryMsg,
'/robot/telemetry',
self._telemetry_callback,
qos_profile=qos.QoSProfile(
reliability=qos.ReliabilityPolicy.BEST_EFFORT,
durability=qos.DurabilityPolicy.VOLATILE,
history=qos.HistoryPolicy.KEEP_LAST,
depth=10
)
)
```

### 3. WebSocket Bridge Implementation

The WebSocket bridge enables real-time communication between ROS2 and the web interface:

```python
class WebSocketBridge:
def __init__(self, host='localhost', port=9090):
self.host = host
self.port = port
self.connections = set()
self.message_handlers = {}

# Set up asyncio event loop
self.loop = asyncio.get_event_loop()
self.server = None

async def start_server(self):
"""Initialize WebSocket server with error handling"""
try:
self.server = await websockets.serve(
self._handle_connection,
self.host,
self.port,
ping_interval=20,
ping_timeout=30
)
print(f"WebSocket server running on ws://{self.host}:{self.port}")
except Exception as e:
print(f"Failed to start WebSocket server: {e}")
raise
```

### 4. Interactive Visualization Component

The visualization component handles rendering and user interaction:

```javascript
class RobotVisualizer {
constructor(config) {
// Initialize canvas layers
this.mainCanvas = document.createElement('canvas');
this.overlayCanvas = document.createElement('canvas');
this.setupCanvasLayers();

// Initialize WebGL context
this.gl = this.mainCanvas.getContext('webgl2');
if (!this.gl) {
throw new Error('WebGL2 not supported');
}

// Set up rendering pipeline
this.setupShaders();
this.setupBuffers();
this.setupInteraction();
}

setupCanvasLayers() {
// Configure canvas properties
this.mainCanvas.style.position = 'absolute';
this.overlayCanvas.style.position = 'absolute';

// Set up high-DPI support
this.setupHighDPI();

// Configure canvas container
this.container = document.createElement('div');
this.container.style.position = 'relative';
this.container.appendChild(this.mainCanvas);
this.container.appendChild(this.overlayCanvas);
}
}
```

### 5. State Management System

A robust state management implementation:

```python
class SharedState:
def __init__(self):
self._state = {}
self._callbacks = {}
self._lock = threading.Lock()
self._history = {}
self._max_history = 1000

def subscribe(self, key, callback):
"""Register callback for state changes"""
with self._lock:
if key not in self._callbacks:
self._callbacks[key] = set()
self._callbacks[key].add(callback)

def update(self, key, value, store_history=False):
"""Thread-safe state update with history tracking"""
with self._lock:
self._state[key] = value

if store_history:
if key not in self._history:
self._history[key] = []
self._history[key].append({
'timestamp': time.time(),
'value': value
})

# Maintain history size
if len(self._history[key]) > self._max_history:
self._history[key].pop(0)

# Notify subscribers
self._notify_subscribers(key, value)
```

## Best Practices

### Error Handling

Implement comprehensive error handling throughout the system:

- Create error boundaries at component boundaries
- Provide clear, actionable error messages
- Handle network failures gracefully
- Implement appropriate fallback behaviors
- Log errors appropriately for debugging

### Resource Management

Proper resource management prevents memory leaks and ensures system stability:

- Clean up WebSocket connections when they're no longer needed
- Manage canvas and WebGL resources efficiently
- Implement proper memory management strategies
- Handle component lifecycle events appropriately
- Monitor system resource usage

## Conclusion

The integration of Streamlit, ROS2, and web visualizations provides a powerful foundation for building sophisticated robotics interfaces. Success with this architecture requires careful attention to threading and state management, implementation of appropriate optimization strategies, adherence to best practices for resource management, and regular monitoring and performance optimization.

Remember to adapt these patterns to your specific use case while maintaining the core principles of performance, reliability, and maintainability. Regular testing and monitoring of the system will help ensure it continues to meet the demands of modern robotics visualization requirements.
Loading