Teleoperation · Robotics systems · Human-in-the-loop control

Teleoperation Systems Integration

I implemented 4G/5G-based teleoperation on an ADAS-enabled Droid setup on an NVIDIA Jetson, integrating ROS control, four-camera GStreamer feedback, and cross-system fault diagnosis.

TeleoperationROSNVIDIA Jetson AGX XavierGStreamerEmbedded LinuxSystems reliability

Sensor-equipped robotic platform used for the teleoperation integration work

00 / Experimental platform

A physical platform for closing the remote-control loop.

The ADAS Droid served as the experimental platform for integrating remote commands, onboard execution, visual feedback, and physical actuation. The hardware made teleoperation a real systems problem rather than a software-only demonstration.

Green and white sensor-equipped Droid outdoors — **Robotic platform**Sensor-equipped vehicle used during integration.

Side view showing the Droid body and onboard hardware — **Onboard integration**Compute, sensing, and actuation context.

ComputeNVIDIA Jetson AGX Xavier

Robotics layerROS + embedded Linux

Visual feedbackFour fisheye cameras + GStreamer

Platform contextLiDAR · ultrasonic sensing · actuation

01 / Research problem

Connected automated vehicles need a remote-intervention path.

Automated-driving functions can encounter unusual or complex situations that they cannot resolve safely on their own. The project explored teleoperation as a human fallback: connect a remote operator to an ADAS-enabled vehicle, transfer steering and acceleration commands over a mobile/cloud link, and return enough live visual context for informed intervention.

01 / Automation limit

Handle the edge case

Provide human intervention when onboard automation cannot confidently continue.

02 / Connected control

Move intent across the network

Translate remote input into bounded vehicle-side steering and acceleration commands.

03 / Situational awareness

Return the scene to the operator

Use four live camera streams so the operator can understand and navigate the environment.

02 / My implementation

I implemented teleoperation at the robot-side integration boundary.

Valeo owned the platform and wider architecture. My implementation focused on the Droid-side behavior and the interfaces that turned remote operator commands into a working physical response with visual feedback.

Remote controlImplemented teleoperation behavior at the communication-to-ROS actuation boundary.
ROS softwareMaintained and extended the robot-side components used by the integrated control path.
Jetson + LinuxResolved runtime, build-environment, and deployment issues on NVIDIA Jetson.
Video feedbackIntegrated four-camera GStreamer streaming for operator navigation feedback.
Fault diagnosisTraced problems across command handling, compute, actuation, camera feedback, and hardware interfaces.
Operational readinessStabilized the end-to-end prototype for a live public demonstration.

03 / Concept architecture

Separate control, video, safety, and system state.

The implemented prototype used mobile connectivity, UDP-to-ROS command conversion, and GStreamer video. The diagram below extends that experience into a public, conceptual teleoperation architecture: time-sensitive control stays separate from video, every command carries freshness information, and loss of communication is handled locally by the vehicle.

Conceptual architecture

A production-oriented connected-vehicle teleoperation loop

Vehicle edge

ADAS-enabled vehicle

Camera · radar · LiDAR · CAN
ROS 2 / DDS on the local network
Hardware H.264 camera encoding
Vehicle-side bridge to selected topics

Communication

5G / edge network

Primary and secondary provider paths
QoS / network-slice policy
On-site edge relay where possible
Independent uplink and downlink health

Control centre

Operator station

Wheel, pedals, or game controller
Video decode + small jitter buffer
Views · latency · link health · state
Mirrored session state machine

Control downlink Controller input→Separate UDP channel→Sequence + timestamp→UDP-to-ROS bridge→Actuation

Video uplink Four cameras→H.264 / GStreamer→RTP over UDP→RTCP loss + jitter→Decode + HMI

Vehicle-local safety gate Heartbeat watchdog·Hard command timeout·Reject stale or repeated commands·Vehicle-decided minimal-risk manoeuvre

Session authorityNo command accepted outside an active, authenticated session.

Time alignmentGNSS PPS / PTP timestamps align vehicle and control-centre records.

RecordingTriggered logs on both sides support incident reconstruction and evaluation.

Implemented core. Mobile-network control, UDP-to-ROS conversion, bounded steering/acceleration commands, and four-camera GStreamer feedback.

Conceptual extension. RTP/RTCP, redundant connectivity, synchronized recording, mirrored sessions, and a vehicle-local safety gate show how the prototype could evolve; they are not claimed as implemented project features.

04 / Reliability engineering

Teleoperation failures emerge between layers.

The reliability work was end to end: separate a communication symptom from a robot-side software fault, confirm that commands reached actuation, and verify that the operator still received usable feedback. Debugging the interfaces was as important as debugging the modules.

01 / Control path

Command-to-actuation tracing

Investigated steering and acceleration behavior across the ROS-to-physical-control boundary.

02 / Feedback path

Camera-stream continuity

Separated capture, encoding, transport, and display problems that affected operator visibility.

03 / Onboard runtime

Compute stability

Resolved Jetson and Linux environment issues that prevented the integrated stack from operating.

ConnectionMobile-network teleoperation

DemonstrationIntegrated robot-side control + video feedback

AudienceManagement · local government · press

Outcome

The complete teleoperation loop was demonstrated live to technical and public stakeholders.

05 / Prototype boundary

The prototype established integration—not production readiness.

This stage focused on making remote control and visual feedback work reliably on the physical platform. Quantitative latency characterization and communication security were intentionally excluded from the project scope, as were production safety validation and fail-operational behavior.

Delay and jitter

Variable round-trip time can destabilize human control and make precise manoeuvres harder.

Limited situational awareness

Camera-only feedback narrows depth, peripheral context, and the operator’s understanding of local risk.

Network dependence

Loss, bandwidth variation, and handovers can interrupt either command delivery or visual feedback.

Security exposure

Remote actuation requires authenticated, integrity-protected communication and a carefully defined trust boundary.

Operator workload

Continuous manual control scales poorly and can increase fatigue during long or complex missions.

Safe fallback

A production system needs a deterministic response when commands, feedback, or connectivity become unreliable.