Baja AR — Sensor Fusion Drive Lab

What to look for

P1 · HEAD TRACKING

Transfer-aligned vs yaw-only

Over the whoops your head nods against the cab by a few degrees. With transfer-aligned tracking the road-edge lines and the ghost stay on the real road. Switch to Yaw-only, which ignores the nod, and watch them bounce off it.

P2 · PREDICTION

20 ms is enough to miss a whoop

With None, the overlay trails your head by the latency. Gyro extrapolation fixes smooth motion but misses the start of every hit. Terrain-predictive runs a suspension model forward over the mapped ground. In the P2 prototype, the suspension model does most of the work at 20 ms; the map ahead starts to matter at 30–50 ms. It cannot know about the new washout.

P3 · CHANGE DETECTION

Only what changed is bright

Ruts, a fallen rock, a washout, a stuck truck and a moved marker are drawn in red; known terrain stays faint. In dust the LiDAR finds the ruts only metres ahead, while the radar still flags the stuck truck as soon as it is in line of sight.

P4 · SYNTHETIC VISION

Honestly blurred vs confidently wrong

Untick Thermal fusion: in the dust the badge reads MAP-ONLY and the pre-run terrain takes over as a wireframe. Radar Doppler keeps it within half a metre and softens the lines to show its uncertainty. Switch to Wheel speed and it stays sharp while sliding metres off the road.

P5 · GHOST CAR

See the line past the crest

A ghost RZR drives the optimal line about 2 s ahead at the profile speed, dashed where terrain hides it. Before the blind crest it runs ahead into the hidden valley and shows the left turn early. At the hairpin, turn on Naive ghost projection and watch it jump when the driver cuts inside.

P6 · THERMAL FUSION

Sees the road through the dust and past the headlights

Long-wave IR scatters far less in fine dust than the LiDAR's laser. In the dust the camera keeps the road edges and the truck ahead, drawn from the map's depth. Switch to COTI outlines and every thermal edge is traced, tire tracks included. At night the outlines run to 180 m where the lights stop at 100, and a cow on the course is flagged from 120 m.

P7 · OPTIMAL LINE

Brake here, this hard

The orange chevrons are the braking zone on the min-curvature line for the RZR's grip; the bar across the ribbon is where braking must start at your current speed, and it slides as you slow. The bar at the right shows how hard: the required deceleration against what the surface gives. In dust the grip is unverified and the bar says so.

Model notes

The course is rebuilt in JavaScript from the DEFAULT_SEGMENTS list (ds 0.5 m). Terrain uses its own noise, so heights match the Python world in layout and feature positions, not cell for cell.
Transfer-aligned tracking is emulated as the true head attitude plus about 0.2° of slow and fast noise, matching the calibrated case in the P1 prototype. Without accelerometer calibration P1 measured about 0.7° RMS. Yaw-only gets the true head yaw, so its error is exactly the head's pitch and roll against the cab.
Terrain-predictive replays the motion model on the pre-run map (whoops 13% lower, no race-day changes) across the latency window, starting from the delayed measurement. Position latency is assumed to be removed by velocity extrapolation.
Dead-reckoning errors and detection ranges are scripted curves shaped like the P3 and P4 results, not live filters. GNSS is assumed blocked while dust blinds the LiDAR; map registration pulls the error back once the LiDAR is live again.
The driver cuts the hairpin across its inside (up to 16 m off the line). A 7 m apex cut does not make a nearest-point lookup jump on this course; the demo's deeper cut does, so the localizer's course station is the path's own (what a heading-gated projection recovers); the Naive ghost projection shows the jump.
Vehicle (RZR Pro R Ultimate, 2-seat). Published: 225 hp, 2,339 lb dry, 104.5 in wheelbase, 74 in wide, 32 in tyres, 27/29 in travel, 90 mph, 0–60 in 5 s. Assumed: 1,190 kg race weight, 143 kW at the wheels, CdA 1.8 m², eye 1.40 m above ground, CG 0.75 m (rollover threshold 1.09 g, so tyre grip governs). The suspension filter runs at 1.25 Hz heave and 1.45 Hz pitch.
Optimal line and speed profile (assumptions). The line is the min-curvature path inside a ±1.75 m usable corridor (road half-width 3 m less half the car and a 0.3 m margin), with local bounds to straddle the ruts, pass the rock on the left and the stuck truck on the berm; it is embedded from the P7 prototype's converged solution. Grip: hardpack 0.75 g, silt 0.55, whoops 0.45, berm 0.60; caps: 40 m/s absolute, 22 m/s in the whoops, 24 m/s over the crest (to stay grounded), 8 m/s at the washout. The profile is the friction-ellipse limited three-pass solution (lateral limit, braking, traction and power). On this course the line itself is worth about 1 s a lap; the grip model and the caps set the rest. The driver runs 0–5 % above the profile so the cue has something to say.
Brake cue (assumption). Required deceleration to reach the next apex or cap speed over the remaining distance, less a 0.4 s reaction allowance unless the car is already decelerating, divided by the available braking on the current surface at the current lateral load. The brake point is where that ratio would reach 90 %; the alarm sounds above 110 %. In the dust the grip is unverified and the cue uses 0.60 g with an amber stripe.
COTI outlines (assumption). Contours are taken from the world's thermal features at the true camera pose, limited by the camera's 0.078°/px resolution (nothing under 2 px) and the NETD-limited contrast after the plume (3σ), then cast to the first map intersection from the estimated pose. A real system would extract them from the image. Thermal contrast assumed: fresh tracks +1.5 K by day and +0.8 K at night, race-day ruts −1.2 K by day and +0.6 K at night, rocks +3 / +1 K, bushes −4 / +2 K, cacti −5 / +3 K, course markers neutral. The HUD budgets 12 road contours and 12 hot contours; road contours appear only in dust or darkness, beyond 55 m at clear night. Vegetation and rock contours remain in the inset. Tracks are optional. Occluded samples and large projected discontinuities break lines.
Night (assumption). Time of day follows the lap: full day to s 700, dusk through the hairpin, night from s 830 (Force night puts the whole lap in the dark). Headlights: a 15° half-angle beam plus a 32° light bar, road bright to 60 m and faint at 100 m; markers are retro-reflective. At night hardpack holds 26 °C, silt cools to 20 °C and scrub to 18 °C, so the silt bed at s 870–905 reads cool where by day it read warm; sky −40 °C. A cow (38 °C skin, 1.4 m) stands on the course at s 925 and is flagged by the IR from 180 m (its pixel limit) and confirmed by radar from 120 m; a chase truck with a warm engine is parked 12 m off the line at s 880 and stays a violet advisory. The LiDAR is an active sensor and stays LIVE in the dark.
Over a blind crest the ghost's lead stretches (up to 120 m) while the line ahead is hidden, so it can show the turn beyond. Elsewhere it follows the 2 s rule.
3D presentation. The visible scene uses a depth-buffered WebGL mesh of the existing heightfield (1 m near terrain, 8 m distant terrain), with solid scenery and a chassis-attached cockpit. A column renderer remains as the graphics fallback and supplies the thermal inset's depth buffer. Gravel texture and the distant skyline are visual only; they do not change vehicle motion.
Sensor horizon and green samples. Sparse 10 Hz beams stop at their first terrain intersection, limited by the modeled LiDAR range in dust; object returns are omitted. These samples illustrate geometry and do not run the P3 detector. Radar and thermal sectors show modeled reach, not guaranteed coverage or free space. Target symbols use the demo's existing detection state.
Dust plume (assumption). The dust is thrown by a race truck ahead on the pre-run line. Its gap to the driver is scripted: 120 m entering the zone (s 470), 50 m at the whiteout core (s 662–668), 55 m at the exit, then it pulls away. The plume is a function of distance behind that truck: rising over the first 15 m, a core of β_LiDAR 0.70 m⁻¹ (0.61 after the bursts) at 15–35 m, easing to 0.28 at 50 m and then thinning with a 45 m e-fold (0.08 at 100 m), only where the road is silt (s 470–668, then half as much residual dust on hardpack to 692, clear by 702), scaled by the truck's throttle (it lifts for the washout, which opens a thin gap at s 575–615) with ±25 % bursts, and drifting right at 0.5 m/s in a crosswind. Fog in the scene integrates this along the road ahead, so it is thickest right behind the truck ahead.
Thermal camera (assumption). 640×512 LWIR microbolometer, 12 µm pitch, 50°×40° field, 0.078°/px, 30 Hz, NETD 40 mK plus a small fixed column pattern, mounted 0.15 m right of the LiDAR ([1.20, −0.15, 2.10] m in the vehicle frame) and pitched 5° down. Its extinction is taken as 0.15× the LiDAR's (10 µm against 905 nm in fine dust). The scene is an afternoon one: hardpack road 46 °C, silt 52 °C, verge 36 °C, sun-facing slopes warmer and shaded berms cooler, sky −30 °C; the cloud itself radiates at 34 °C, which lifts the floor and costs contrast rather than blacking out. Truck engine bays 95 °C, exhausts 180 °C, tyres 60 °C, bodywork 48 °C; course markers neutral.
Effective ranges (assumption). LiDAR range is where the optical depth ahead reaches 1.25 (1.25/β in uniform dust, as before). IR edge range is where road/verge contrast falls to 30 % of clear: optical depth 1.2 at 10 µm, which is 1.204/(0.15 β) in uniform dust (about 100 m at β 0.08, 40 m at 0.2). Because the plume thickens toward the truck ahead, the integrated IR range inside it runs 60–85 m early on (it reaches through the thin gap), 30–45 m later, and under 30 m in the core. Hot spots stay detectable to an IR optical depth of 4 (a 55 K engine still 1 K, 25× NETD). The inset reuses the eye's depth buffer and the vehicle's yaw and pitch; the 1.3 m offset between eye and camera is ignored.
Fusion (assumption). Thermal edges are angles: each edge pixel's ray is cast from the estimated pose onto the map terrain and the 3-D point is re-projected, so the drawn edge is right laterally whatever the dead-reckoning error, and only its assigned depth carries the along-track error. While the state is THERMAL the cross-track error is pulled to a 0.3 m registration residual with a 0.4 s time constant (wheel odometry has no lateral observability, so the edge is the only lateral measurement; with radar Doppler the two are weighted by their σ); along-track keeps the wheel or radar drift. Hot spots take radar range and Doppler when the radar has a return in the same direction, otherwise the map ground under the cluster's bottom edge.
State thresholds: THERMAL when the LiDAR range is under 32 m (dust fraction, 1 − range/80, of 0.6 or more; the 15 m rule is inside this) and the IR edge range is at least 30 m; MAP-ONLY when the IR range is under 30 m too; the 0.3 s worse / 1.0 s better hysteresis is unchanged. With Thermal fusion off the machine is the old one. Through the dust, with fusion on: DEGRADED from s 413 (the LiDAR sees the plume's face 60 m ahead), THERMAL from 470, MAP-ONLY in the whiteout core 643–674, THERMAL again as the plume thins, LIVE from 708; with fusion off: DEGRADED 413, MAP-ONLY 473, LIVE 708.