Baja AR — Sensor Fusion Drive Lab

What to look for

P1 · HEAD TRACKING

Transfer-aligned vs yaw-only

Over the whoops your head nods against the cab by a few degrees. With transfer-aligned tracking the road-edge lines and the ghost stay on the real road. Switch to Yaw-only, which ignores the nod, and watch them bounce off it.

P2 · PREDICTION

20 ms is enough to miss a whoop

With None, the overlay trails your head by the latency. Gyro extrapolation fixes smooth motion but misses the start of every hit. Terrain-predictive runs a suspension model forward over the mapped ground. In the P2 prototype, the suspension model does most of the work at 20 ms; the map ahead starts to matter at 30–50 ms. It cannot know about the new washout.

P3 · CHANGE DETECTION

Only what changed is bright

Ruts, a fallen rock, a washout, a stuck truck and a moved marker are drawn in red; known terrain stays faint. In dust the LiDAR finds the ruts only metres ahead, while the radar still flags the stuck truck as soon as it is in line of sight.

P4 · SYNTHETIC VISION

Honestly blurred vs confidently wrong

Untick Thermal fusion: in the dust the badge reads MAP-ONLY and the pre-run terrain takes over as a wireframe. Radar Doppler keeps it within half a metre and softens the lines to show its uncertainty. Switch to Wheel speed and it stays sharp while sliding metres off the road.

P5 · GHOST CAR

See the line past the crest

A ghost RZR drives the optimal line about 2 s ahead at the profile speed, dashed where terrain hides it. Before the blind crest it runs ahead into the hidden valley and shows the left turn early. The driver follows the same optimized race line, including the full hairpin.

P6 · THERMAL FUSION

Read thermal contrast beyond the headlights

This model gives long-wave IR lower dust attenuation than LiDAR. Sparse road contours preserve the view; their angles come from thermal contrast and their depth depends on the map. Hot targets stay advisory until radar confirms range. Terrain blocks all three sensors, and sufficiently dense dust removes useful thermal edges too. Compare the same night scene with AR on and off.

RACE PLANNING

Brake, rotate, drive out

The planner trades corner distance against exit speed over the whole stage. Orange is braking; green is available drive. Watch the brake release as lateral grip builds, then the throttle return on exit. The speed strip previews the next 220 m, including grip changes and crests.

Model notes

Extended stage. The first 961.7 m retains the Python course layout; ten additional clothoids extend the browser route to 3,841.7 m. The extension adds a 600 m silt basin, a stalled vehicle at s 1970, a 240 m whoop field and a 13 m ridge. Course sampling is 0.5 m; the ground grid is 1 m. Heights are synthetic, not a copy of the Python terrain. The return straight from s 830–970 is graded to isolate the night comparison from crest occlusion.
Transfer-aligned tracking is emulated as the true head attitude plus about 0.2° of slow and fast noise, matching the calibrated case in the P1 prototype. Without accelerometer calibration P1 measured about 0.7° RMS. Yaw-only gets the true head yaw, so its error is exactly the head's pitch and roll against the cab.
Terrain-predictive replays the motion model on the pre-run map (whoops 13% lower, no race-day changes) across the latency window, starting from the delayed measurement. Position latency is assumed to be removed by velocity extrapolation.
Dead-reckoning errors are scripted curves shaped like the P4 results, not live filters. The two dust sectors include an independent, scripted GNSS interruption to exercise drift; dust itself does not block GNSS. Map registration pulls the error back when observations return. The wheel baseline retains P4's slip error and overconfident covariance, and is not a claim about every wheel-aided filter.
The driver follows the planned line and speed, including the complete hairpin; no shortcut or scripted overspeed is added. This is a prescribed replay, not a closed-loop vehicle controller. Nearest-point projection remains available as a localization comparison.
Vehicle (RZR Pro R Ultimate, 2-seat). Published: 225 hp, 2,339 lb dry, 104.5 in wheelbase, 74 in wide, 32 in tyres, 27/29 in travel, 90 mph, 0–60 in 5 s. Assumed: 1,190 kg race weight, 143 kW at the wheels, CdA 1.8 m², eye 1.40 m above ground, CG 0.75 m (rollover threshold 1.09 g, so tyre grip governs). The suspension filter runs at 1.25 Hz heave and 1.45 Hz pitch.
Race line (assumptions). A deterministic local search minimizes full-stage time with smooth lateral changes, corridor bounds and vehicle-footprint clearance. It evaluates the same powertrain and grip model for each candidate and for the centreline-with-detours baseline. This is a local solution, not a proven global optimum. Stationary hazards and escape shoulders are supplied in advance; the plan is not evidence of real-time obstacle avoidance. The animal is passed slowly on a designated shoulder.
Speed and pedals (assumptions). Forward/backward passes converge on one shared speed envelope for the driven path and HUD. Combined braking/cornering and acceleration/cornering share a friction circle with 10% grip reserve. Assumed μ: hardpack 0.75, silt 0.55, whoops 0.45, shoulders at most 0.60. Grade, 143 kW wheel power, drag and 0.025 rolling resistance affect longitudinal force. A 12 m smoothed terrain envelope models crest unloading; a minimum 35% normal load is retained, while roughness uses explicit speed/grip caps. Bank angle, tyre transients, brake temperatures, gear/CVT dynamics and wheel-level load transfer are not simulated. Brake percentages use straight-line tyre capacity and throttle uses engine capacity (wheel power with a 1 g launch-force ceiling), so both reduce as cornering consumes grip. These are force estimates, not actual pedal calibration. No constant minimum braking grip is invented at the cornering limit.
Brake cue. The moving braking point integrates backward from the upcoming target across intervening grip, curvature and grade, with 0.4 s preparation allowance before braking. Current pedal demand follows the feasible profile; overspeed adds braking and suppresses throttle. Trail braking releases longitudinal force as lateral demand rises. The predicted finish time uses the plan; it is not a measured race result.
COTI outlines (assumption). Contours are taken from the world's thermal features at the true camera pose, limited by the camera's 0.078°/px resolution (nothing under 2 px) and the NETD-limited contrast after the plume (3σ), then cast to the first map intersection from the estimated pose. A real system would extract them from the image. Thermal contrast assumed: fresh tracks +1.5 K by day and +0.8 K at night, race-day ruts −1.2 K by day and +0.6 K at night, rocks +3 / +1 K, bushes −4 / +2 K, cacti −5 / +3 K, course markers neutral. The HUD budgets 12 road contours and 12 hot contours; road contours appear only in dust or darkness, beyond 55 m at clear night. Vegetation and rock contours remain in the inset. Tracks are optional. Occluded samples and large projected discontinuities break lines.
Night (assumption). Lighting stays fixed through a run. The night preset selects darkness; other presets select daylight. Headlights have a 15° half-angle beam plus a 32° light bar, useful road contrast to an assumed 60 m and faint illumination at 100 m. Night temperatures: hardpack 26 °C, silt 20 °C, scrub 18 °C, sky −40 °C. The cow at s 925 has 38 °C skin and a 1.4 m height. Its thermal range cap is 180 m, radar cap 120 m; terrain and field of view can shorten either. The off-line chase truck at s 880 stays advisory. LiDAR remains active at night.
Over a blind crest the ghost's lead stretches (up to 120 m) while the line ahead is hidden, so it can show the turn beyond. Elsewhere it follows the 2 s rule.
3D presentation. The visible scene uses a depth-buffered WebGL mesh of the existing heightfield (1 m near terrain, 8 m distant terrain), with solid scenery, terrain-following trucks and a chassis-attached cockpit. Soft ground-contact shading is an appearance approximation, not traced shadows. Off-screen scenery is culled in bounded cells. A column renderer remains as the graphics fallback and supplies the thermal inset's depth buffer. Gravel, tyre wear, sky and bounded dust puffs are visual only; they do not alter optical-depth calculations, sensor reach or vehicle motion. Split comparison clips only the AR layer; both sides use the same scene, pose and sensing. Guided highlights select five intervals of this same replay, with a separate night comparison.
Sensor horizon and green samples. Sparse 10 Hz beams stop at their first terrain intersection, limited by the modeled LiDAR range in dust; object returns are omitted. These samples illustrate geometry and do not run the P3 detector. Radar and thermal sectors show modeled reach, not guaranteed coverage or free space. Target symbols use the demo's existing detection state.
Same-target warning comparison. Stationary vehicle radar uses a 150 m cap, terrain visibility, a forward field of view and three consecutive 20 Hz confirmations. The animal uses a 120 m cap. Unaided acquisition requires terrain visibility and an assumed 8% contrast threshold, capped at 200 m by day or 60 m at night. Warning lead is the difference between first acquisition times on the complete, identical replay. These values do not establish driver reaction, lap-time or safety gains. The stopping panel is a constant-deceleration estimate with 1.2 s reaction allowance, friction-circle corner load and road grade; it is separate from the anticipatory racing brake cue.
Hardware context. The clear-air LiDAR cap follows the Livox HAP specification. Thermal attenuation and contrast values here are scenario assumptions; FLIR describes how atmospheric conditions limit thermal reach. Dust-tolerant does not mean unlimited visibility.
Dust plume (assumption). The dust is thrown by a race truck ahead on the pre-run line. Its gap to the driver is scripted: 120 m entering the zone (s 470), 50 m at the whiteout core (s 662–668), 55 m at the exit, then it pulls away. The plume is a function of distance behind that truck: rising over the first 15 m, a core of β_LiDAR 0.70 m⁻¹ (0.61 after the bursts) at 15–35 m, easing to 0.28 at 50 m and then thinning with a 45 m e-fold (0.08 at 100 m), only where the road is silt (s 470–668, then half as much residual dust on hardpack to 692, clear by 702), scaled by the truck's throttle (it lifts for the washout, which opens a thin gap at s 575–615) with ±25 % bursts, and drifting right at 0.5 m/s in a crosswind. Fog in the scene integrates this along the road ahead, so it is thickest right behind the truck ahead.
Thermal camera (assumption). 640×512 LWIR microbolometer, 12 µm pitch, 50°×40° field, 0.078°/px, 30 Hz, NETD 40 mK plus a small fixed column pattern, mounted 0.15 m right of the LiDAR ([1.20, −0.15, 2.10] m in the vehicle frame) and pitched 5° down. Its extinction is taken as 0.15× the LiDAR's (10 µm against 905 nm in fine dust). The scene is an afternoon one: hardpack road 46 °C, silt 52 °C, verge 36 °C, sun-facing slopes warmer and shaded berms cooler, sky −30 °C; the cloud itself radiates at 34 °C, which lifts the floor and costs contrast rather than blacking out. Truck engine bays 95 °C, exhausts 180 °C, tyres 60 °C, bodywork 48 °C; course markers neutral.
Effective ranges (assumption). LiDAR range is where the optical depth ahead reaches 1.25 (1.25/β in uniform dust, as before). IR edge range is where road/verge contrast falls to 30 % of clear: optical depth 1.2 at 10 µm, which is 1.204/(0.15 β) in uniform dust (about 100 m at β 0.08, 40 m at 0.2). Because the plume thickens toward the truck ahead, the integrated IR range inside it runs 60–85 m early on (it reaches through the thin gap), 30–45 m later, and under 30 m in the core. Hot spots stay detectable to an IR optical depth of 4 (a 55 K engine still 1 K, 25× NETD). The inset reuses the eye's depth buffer and the vehicle's yaw and pitch; the 1.3 m offset between eye and camera is ignored.
Fusion (assumption). Thermal edges are angles: each edge pixel's ray is cast from the estimated pose onto the map terrain and the 3-D point is re-projected, so the drawn edge is right laterally whatever the dead-reckoning error, and only its assigned depth carries the along-track error. While the state is THERMAL the cross-track error is pulled to a 0.3 m registration residual with a 0.4 s time constant (wheel odometry has no lateral observability, so the edge is the only lateral measurement; with radar Doppler the two are weighted by their σ); along-track keeps the wheel or radar drift. Hot spots take radar range and Doppler when the radar has a return in the same direction, otherwise the map ground under the cluster's bottom edge.
State thresholds: THERMAL when the LiDAR range is under 32 m (dust fraction, 1 − range/80, of 0.6 or more; the 15 m rule is inside this) and the IR edge range is at least 30 m; MAP-ONLY when the IR range is under 30 m too; the 0.3 s worse / 1.0 s better hysteresis is unchanged. With Thermal fusion off the machine is the old one. Through the dust, with fusion on: DEGRADED from s 413 (the LiDAR sees the plume's face 60 m ahead), THERMAL from 470, MAP-ONLY in the whiteout core 643–674, THERMAL again as the plume thins, LIVE from 708; with fusion off: DEGRADED 413, MAP-ONLY 473, LIVE 708.

See beyond the dust.

Read the road ahead.

Set up the exit.

More time to make the next decision.