Simulation environment

Docs index

There are two simulators, and which one a run uses changes what is real about it. Neither is the subject of the project; both exist to put the stack under something that moves.

1. The headless model

kraken_sim/headless_sim is a unicycle with about a hundred lines of physics in it. It exposes the same topic contract as the O3DE robot — ground truth, GNSS, IMU, wheel odometry, ground-speed radar — so scenarios do not know which one they are running against.

It owns /clock and steps at real_time_factor times wall speed, so the suite is fast and needs no GPU. The factor is load-bearing rather than a convenience: at 5× the EKF could not keep up with its own input and the same scenario returned 0.46 m one run and 1.46 m the next. The reasoning, and why the navigation scenarios run at 1.0 instead, is in design.

What it models:

   
Motion unicycle, cmd_vel integrated directly, no steering limit
GNSS 10 Hz NavSatFix about a configurable datum, which must match navsat_transform’s
IMU 50 Hz, yaw-rate noise and a small constant bias
Wheels 50 Hz odometry, speed noise, lateral slip noise, a 1% scale error
Radar 20 Hz Doppler ground speed, its own RNG stream so adding it leaves other sensors’ noise sequences untouched
Terrain kraken_sim/terrain: a traction scalar in $(0,1]$ per rectangular patch, scaling forward and yaw rate together

What it does not model: slope, load transfer, lateral drift, obstacles, and anything you could range off. There is no lidar here, so the obstacle layer fed from one stays empty — see design.

Terrain is the one physical fault. Every other failure in the suite is a lie told to the filter by the injector; slip has to change where the robot actually is, so it lives in the simulator. See design.

2. The O3DE scene

Project/ is the O3DE side, descended from o3de/ROSConDemo. It gives real physics, real trees and a real sensor: the Kraken carries an RGL (Robotec GPU Lidar) that raycasts the level’s meshes, which is what the costmap obstacle layer and row perception read.

It is a skeleton, not a finished level. The project registers and the ROS 2 Gem wiring is described, but the scene itself has to be authored in the Editor. That is the biggest open piece of this repo.

Three things the Docker setup does to the upstream demo, worth knowing because they explain otherwise baffling behaviour:

Which level, and the flag that does not work

Upstream ships three, and Levels/ spells them Main, playground and Test while the argument that loads them wants main, playground and test.

  main playground
What it is the orchard: five rows, terrain, trees a flat grid with a sun and one Kraken
Named entities Trees, Terrian, SurfaceMaterials, 18× RowSpawnPoint, line1–line5, 53× GatherPoint* Sun, Ground, Grid, Global Sky, Camera, and the robot
Rows five none
Terrain yes none
Robots none placed; you spawn them on a line* point one, already there
Namespace you get kraken1, from the spawn name base_link, there being no spawn name to inherit
Real time factor 1.000x 1.000x
GPU 100% ~40%

(test is the third: 25 entities, no Kraken at all, and nothing here uses it.)

Measured on an RTX 3500 Ada, profile build, nothing else running.

Both levels hold real time. The orchard costs far more GPU to do it, and on a smaller card that headroom is what would go first, but on this one there is no speed difference to trade. Pick the level on what you need to see, not on what it costs: playground has no trees and no terrain, so it suits work where the scene is irrelevant — sensor rates, DDS transport, the filter, fault injection — while anything that has to see something needs main: row perception, the costmap obstacle layer, and the coverage geometry in navigation, which is read out of Main.prefab and has nowhere else to run.

Do not believe a real time factor sampled in the first minute. main chains through intermediate loads before it arrives, and on this machine the orchard landed about 45 seconds after launch. Sample before that and /clock is either absent or advancing against a level that is still coming up; three consecutive 20-second windows read 0.000x and then 0.999x once it had settled. sim_perf --settle has to clear the whole chain, not just the process start. Confirm against the launcher log, which prints the level it actually reached:

docker logs kraken_launcher 2>&1 | grep -a "spawnable loaded"

The namespace row is the one that catches people twice. main’s Krakens come out of a spawner, so they are named when you ask for them and the rest of the stack’s kraken1 default lines up. playground’s Kraken is placed in the level with no name to take, so it publishes under base_link and every tool has to be told:

ros2 run kraken_scenarios sim_perf --namespace base_link --duration 30

playground used to carry two. The level instantiates both apple_kraken_v1 and apple_kraken_v2, and each takes its namespace from its root entity’s name — base_link in both — so every sensor topic had two publishers. Ground truth arrived at 100 Hz from a 50 Hz sensor and traced 1710 m of path in 25 seconds while the robot drove 21, and the stereo pair was free to take its left image from one machine and its right from the other. kraken_sensors.py drops the second one now. ros2 topic info <topic> | grep Publisher is the check, and it is worth running before believing any number off this simulator at all.

Re-baking assets, and one spawn per launch

Two ways to ruin a working simulator without being told.

Source ROS before re-baking. cmake --build build/linux --config profile --target ROSConDemo.Assets must run with /opt/ros/$ROS_DISTRO/setup.bash sourced, exactly as roscondemo-setup.sh does it. Without it the AssetProcessor cannot load libROS2.Editor.so — libament_index_cpp.so is not on the loader path — and writes a spawnable with every ROS 2 component dropped, while reporting Number of Assets Failed to Process: 0. The level then loads, /clock ticks, and the robot publishes nothing whatsoever. The tell is the file size: about 140 KB intact against 122 KB stripped. Products are compared by content hash, so touch will not force a rebuild and a stripped one looks current indefinitely.

One spawn per launch. Deleting a robot and spawning another inside a running simulator broke it twice in an afternoon: once a segfault preceded by repeated Failed to initialize PipelineState for DynamicDrawContext, once a hard freeze. Each robot carries three cameras and therefore three render pipelines — GPU use went 1832 → 4096 MiB for a single one against a 12 GB card — and DeleteEntity removes the ROS side while leaving the mesh behind, so the cost is not returned until the process restarts. Those leftovers are not merely idle. They are robot-shaped objects standing near the spawn point, and stereo visual odometry tracks features on them happily, which flatters precisely the measurements a clean scene would fail. Restart the simulator between measurement cases.

A frozen simulator looks alive from every angle that does not involve a message. docker ps reports Up, docker inspect reports exit code 0, and ros2 topic info /clock reports a publisher — because that is discovery metadata, not liveness. Only asking whether messages are arriving tells the truth, which is what results/_sim_health.py checks alongside the publisher counts.

A launch that lands on main when you did not ask for it means the setup is half applied. docker/roscondemo-setup.sh defaults DEMO_LEVEL to playground and step 11 writes it into both of the mechanisms the project uses to pick a startup level — Registry/autoexec.game.setreg and autoexec.cfg — then greps both back and exits 1 if either did not take. But the script restores those two files to upstream at the top of every run, and upstream points at main. Interrupt it in between and you are left with a checkout that is patched everywhere else and silently starts a level you did not ask for. git -C /o3de/ROSConDemo diff --stat is the check: if Project/autoexec.cfg is missing from the list, re-run the setup script.

+loadlevel playground overrides it either way, and -LoadLevel=playground does not. The - form, which upstream’s README prints, is silently ignored: CSystem::ExecuteCommandLine only executes + prefixed arguments, so the - form is parsed as an unknown switch and dropped without a warning, leaving whatever autoexec.cfg says to stand. --console-command-file= does not override it either, whichever directory the file is written to.

/o3de/ROSConDemo/Project/build/linux/bin/profile/ROSConDemo.GameLauncher \
    +loadlevel playground --r_width=1280 --r_height=720

The launcher is windowed: the engine defaults r_fullscreen to false and r_width/r_height to 1920x1080, and nothing grabs the pointer, so it does not take the desktop over while it runs.

Give it thirty seconds before believing the topic list. Loading chains through main twice before it reaches playground, and the sensors do not appear until the last of those settles. Measured too early it looks like a level with no robot in it.

The cameras

Three, all colour only at 640×480, 60° vertical field of view — about 75° horizontal at 4:3. Configured for 10 Hz; measured at 12 on a loaded scene and 32 on the empty one, for the reason below.

  camera_front_left camera_front_right camera_ground
Pose on base_link x 2.35, y +0.06, z 1.40, 10° down x 2.35, y −0.06, z 1.40, 10° down x 2.65, z 0.90, 45° down
Sees down the row down the row a ~3.1 m strip of floor, 2.9–6.0 m ahead
For stereo pair, and people and hazards stereo pair spare

The tilt is measured from vertical, not from the horizon. kraken_sensors.py writes Rotate: [0, term, −90], and the term that looks level is 90, because elevation is 90 − term. It was authored as 0 and 45 for as long as the cameras existed, which pointed all three at the sky.

That is worth dwelling on, because it survived a stereo calibration checked end to end — baseline read from tf, $P_{03} = -f_xB$ agreeing to five figures — and it survived because none of those numbers depend on where a camera points. The rendered image is the only thing that does. Two cheap statistics settle it without looking: Laplacian variance and an ORB keypoint count read about 50 and 87 on sky, and about 670 and 1000 on the orchard floor.

The front pair is tilted 10° down rather than left level because a level camera 1.4 m up spends its upper half on sky, which has no texture to track and no disparity to triangulate.

The front two are a stereo pair on a 0.12 m baseline, and that number was reached by measurement after a 0.5 m one was reached by argument.

The argument for the wide pair was sound and about the wrong quantity. Depth error goes as $Z^2\sigma_d/(fB)$, so at $f = 416$ px with a quarter-pixel matcher 0.5 m gives 1 cm at 3 m and 3 cm at 5 m, four times better than 0.12 m at every range. That is the error on the depth of one point. It is not the error on the robot’s motion, and visual odometry estimates the second.

What a wide baseline costs is the minimum range, roughly ten times the baseline: about 5 m at 0.5 m, and 1.2 m at 0.12 m. Down a row that looks affordable, because the row recedes ahead of the machine. Turning out of the row it is not — across the rows there is a wall of trunks two to four metres off and nothing visible behind them, so the wide pair had nothing left to match. Same drive, one robot, real time confirmed:

case 0.5 m baseline 0.12 m baseline
straight, 1 m/s −0.71%, 0/155 unsolved −0.36%, 0/155
straight, 2.5 m/s −0.40%, 0/141 −0.30%, 0/141
turn to 80° −99.58%, 179/396 +0.29%, 0/456
turn to 111° −99.73%, 43/124 −0.30%, 0/152
turn to 137° −99.78%, 79/131 −0.61%, 0/157

The narrow pair is not merely better on turns. It is better on the straights too — 4.03 mm of per-step error against 6.41 — and its covariance is closer to honest, 0.64 to 0.90 of the error it makes against 0.37 to 0.56. Near features carry more parallax per metre travelled than far ones, so a pair that cannot see them discards the measurements that matter and keeps precision on the ones that matter least.

Stereo rather than one camera because monocular has no scale. A single camera recovers a trajectory only up to an unknown factor which then drifts, so there is nothing to hand a filter. Depth from disparity between two views taken at one instant is metric by construction.

camera_ground is a leftover and no longer has a job. It was placed for a monocular ground-flow scheme that recovers scale from an assumed plane height — which fails exactly where this robot works: 25 cm of grass under a 90 cm camera is a 28% speed error, the same order as the wheel slip it would have been there to catch, and invisible in a simulator whose ground is bare and windless. Turn it off with KRAKEN_CAMERAS to save 15 MB/s.

Each is a child entity rather than another component on base_link, because the pose is the whole point and a component cannot carry one. That also means their extrinsics arrive on /tf_static for free, unlike base_link itself, whose transform is deliberately silenced.

Depth is off. A rendered depth buffer is exact — no baseline, no matching failures, no noise — and a sensor that cannot be wrong teaches nothing. The depth publishers stay in the configuration because the component looks each of its four publishers up by name; only the channel is disabled.

That argument is about the depth buffer, not about depth. Stereo matched from two rendered colour images is honest: the matcher fails where real matchers fail — textureless dirt, the repeated texture of a row, occlusion boundaries — and those failures are properties of the scene and the algorithm, so they transfer to hardware. Asking the simulator for per-pixel ground truth is what does not.

Nothing consumes the images yet, but the pair is describable as one. The ROS 2 Gem publishes every CameraInfo identically, with the fourth column of the projection matrix left at zero — and that element is Tx, which REP 104 requires to be $-f_xB$ for the right camera. Zero says the two cameras are in the same place, so anything deriving depth from disparity divides by a baseline of nothing. R is never set either, and default-constructs to all zeros rather than identity.

kraken_camera’s stereo_camera_info corrects both. The render is an ideal pinhole with no distortion, so the images are already rectified and the right answer is exactly R = identity and Tx = -fx * baseline. It takes the baseline from /tf_static rather than a parameter — the extrinsics are authored in docker/kraken_sensors.py, and a parameter would be that number written down a second time, where a disagreement yields depth wrong by a constant factor nothing downstream can detect. Measured on the live pair:

stereo baseline base_link/camera_front_left -> base_link/camera_front_right: 0.1200 m
LEFT  P = [415.692, 0, 320,    0.000, ...]
RIGHT P = [415.692, 0, 320, -49.883, ...]

The two cameras are exactly synchronised, but not for a reason that would survive contact with hardware. Measured on camera_info — same component, same tick, same header as the images, and a few hundred bytes rather than 1.23 MB, so it measures the stamps and not the socket buffer:

front 968 @ 32.33 Hz | ground 968 @ 32.33 Hz
  pair offset ms: mean +0.000  sd 0.000  min +0.000  max +0.000
  exactly equal stamps: 968 of 968

Nothing triggers them together. Each ROS2CameraSensorComponent calls StartSensor with its own frequency and its own callback, and stamps inside that callback with GetROSTimestamp(), which is now() called live rather than a value cached for the frame. Two independent sensors, two independent stamps.

They come out identical because the clock is quantised to the tick: under use_sim_time the node clock only advances when /clock does, so two live now() calls inside one tick return the same number. The pair is synchronised by the simulator’s sense of time, not by anything a camera is doing.

So an exact-time message_filters policy will work here, and that is a trap. Real cameras never produce bit-identical stamps; they need a hardware trigger and a global shutter, and a pipeline tuned against exact-time in this simulator would fall over on the first real pair. Configure approx_sync anyway — it costs nothing here and it is what the hardware will require.

Both also run at 32 Hz on the empty playground level despite being configured for 10, which is the same render-tick snapping that produced 12 Hz on a loaded scene. The configured frequency is an upper bound the gem rounds to a divisor of the frame rate, not a rate it holds.

KRAKEN_CAMERAS picks which of them publish — all (the default), none, or a comma separated list of frame names — and is read when the prefab is generated, so changing it means re-running the script and restarting O3DE.

KRAKEN_CAMERAS=camera_front_left python3 docker/kraken_sensors.py "$PROJECT"
ros2 run kraken_scenarios sim_perf --label camera_front_left --duration 30

KRAKEN_HAZARD is the same idea for the ISO 18497 test object — x,y in world metres stands one in the orchard, unset leaves it out — and it is read when the level prefab is generated, so it also means re-running the script and restarting.

KRAKEN_HAZARD=-49.25,20 python3 docker/kraken_hazards.py "$PROJECT"

Unset is not a different level, it is this one with the object removed, which is what makes it usable as the blank run: a false-positive check is only worth anything if the scene it clears is the scene it was tripped by. Aisle centres are x = -63.25 + 3.5k for k in 0..17, running along y.

The object is the standard’s, not ours: two coaxial cylinders 800 mm tall, a 380 mm body and a 140 mm head, representing a seated human. It is built from the engine’s _Cylinder_1x1 primitive, which is 1 m across and tall and centred rather than based. Do not read that size off the source FBX — its vertices span only ±0.05 m, and taking that at face value once built an object ten times too big, a 3.8 m drum filling the aisle that the lidar dutifully reported as a wall.

sim_perf reads the cost back off a running level without consuming a frame. It times the IMU instead: O3DE renders against wall clock, so a render pass it cannot afford shows up as its tick slipping, and every sensor slips with it. Camera rate comes off camera_info rather than the image — same component, same tick, a few hundred bytes instead of very nearly a megabyte. --images subscribes to the images too, which measures transport rather than the render, and adds load while doing it.

Measured, playground level, one Kraken, RTX 3500 Ada, images subscribed:

reference /kraken1/imu/data          50.04 Hz of 50.0 nominal  (100%)
/kraken1/camera_front/camera_info    12.00 Hz
/kraken1/camera_front/image_color    12.00 Hz  14.74 MB/s
/kraken1/camera_ground/camera_info   12.00 Hz
/kraken1/camera_ground/image_color   12.00 Hz  14.74 MB/s

Both cameras cost the sim tick nothing measurable: the IMU holds its full 50 Hz and the clock stays at 1.0x while 29.5 MB/s of pixels crosses the graph. The 12 Hz is not the 10 Hz that was asked for — the gem services the sensor on a render tick boundary, so the rate snaps to a divisor of the frame rate rather than honouring the request exactly. 12 is the nearest one above 10.

This needs a broker running, and without it you lose most of the frames silently. A frame is 640×480×4 = 1.23 MB. Against the stock net.core.rmem_max of 208 kB it does not fit in a socket buffer, and since the publisher is BEST_EFFORT a fragment lost to the overrun is never retransmitted and takes its whole frame with it — image_color measured 0.95 Hz while camera_info sat at 12. Nothing logs an error; the topic is simply mostly empty.

The fix was to keep the frames out of the socket, by turning on CycloneDDS shared memory so each frame is handed over as a segment brokered by iceoryx’s RouDi. That is currently switched off, and the cameras are slow because of it. Shared memory breaks static TF badly enough to be the worse of the two faults — see the transport trade below.

docker compose -f docker/docker-compose.yml up -d roudi

RouDi arrives as a dependency of ros-jazzy-cyclonedds, so it is already in both images, and one instance serves every container running with ipc: host.

Measured on an untuned host, 640×480 RGBA published at 12 Hz, subscriber rate:

transport received
CycloneDDS over UDP 1.4 Hz
CycloneDDS over UDP, config asking for 16 MB buffers 3.9 Hz
FastDDS, defaults 1.0 Hz
CycloneDDS over shared memory 11.95 Hz

The second row is the useful one: the config alone cannot fix this, because the kernel clamps the request.

The transport trade

Shared memory buys those 12 Hz and costs static TF, and the second bill is the larger one. With SHM enabled at both ends, CycloneDDS 0.10.5 cannot deliver a TRANSIENT_LOCAL topic to a subscriber that joins after the publisher — such a reader never matches the writer at all, so it gets neither the retained sample nor any later live one. /tf_static is the only TRANSIENT_LOCAL topic here, which is exactly why this hid.

What it cost: base_link → imu_link never reached the EKF, so robot_localization could not transform the IMU into base_link and discarded every angular-velocity measurement — at RF_DEBUG, so nothing appeared at any normal log level. ekf_robust.yaml takes yaw rate from the IMU alone, so heading quietly stopped being observable and robust behaved as naive:

  total_gnss_dropout, 10 seeds passes its 2.0 m bound
shared memory on 19.27 ± 6.43 m 1/10
shared memory off 0.39 ± 0.24 m 10/10

Republishing the transforms does not rescue it — that was tried, and a late subscriber got nothing in three runs out of three, because the problem is matching rather than replay. So shared memory is off, and the image path needs a fix that is not SHM.

What the image path needs instead

The frames do not need a different transport. They need the socket buffer the kernel is refusing to give, which is what the ROS 2 DDS tuning guide prescribes for this exact symptom — “Cyclone DDS is not delivering large messages reliably, despite using reliable settings”.

Measured here without a simulator, by publishing 640×480 RGBA at 12 Hz between two ordinary nodes — a 1.23 MB frame is a 1.23 MB frame whether O3DE renders it or a test node fabricates it:

window frames rate
0–4.2 s 23 5.5 Hz
4.2–9.2 s 42 8.4 Hz
9.2–14.2 s 5 1.0 Hz

It does not fail immediately. It starts nearly healthy and collapses once the buffer saturates, landing on the same ~1 Hz the camera measurement found. A short test says everything is fine.

The fix is net.core.rmem_max, and it belongs in min rather than max. The config here asks for 16 MB in max, which the kernel silently clamps to whatever it allows — 425984 bytes, against a frame of 1228800. Asking in min instead makes Cyclone refuse to start and say so:

failed to increase socket receive buffer size to at least 4194304 bytes,
current is 425984 bytes

That is the whole difference. max degrades quietly, min fails loudly, and everything else in this repository argues for the second. It is deliberately still max until something actually consumes images, because the headless scenario suite needs no such buffer and should not be made unrunnable on an untuned host. Whatever turns the cameras on turns this on with it, and the host needs:

sudo sysctl -w net.core.rmem_max=16777216

Two things that turned out not to be the answer, having been checked: imagezero_transport is the right algorithm — lossless, and 20× faster to compress than PNG — but it is ROS 1 only and four years stale. ros2_shm_msgs gets true zero copy, but requires the shared memory that breaks static TF above, and requires the publisher to use its own message types, which the ROS 2 Gem does not. Lossy transports are ruled out separately: JPEG artefacts land on sub-pixel disparity, which is exactly what stereo depth accuracy rests on.

RouDi is not needed while shared memory is off. It is not started, not consulted, and gated behind a compose profile; everything goes over the loopback socket. It is kept, along with ensure-roudi.sh, because re-enabling shared memory needs all of it back. The rest of this section is what it costs when it is on.

With SHM on, RouDi is not optional, and it fails hard. There is no fallback to UDP. A Cyclone participant that cannot reach it waits 60 s and then aborts with IPC_INTERFACE__REG_ROUDI_NOT_AVAILABLE — and that is every node using this config, not only the ones touching cameras. So the compose services all depends_on: [roudi], RouDi runs restart: unless-stopped, and the devcontainer starts one from .devcontainer/ensure-roudi.sh on every start rather than at creation.

It needs a shared /tmp as much as it needs ipc: host. The segments live in /dev/shm, which ipc: host shares, but clients register over a Unix socket at /tmp/roudi, and that path is compiled into iceoryx — iox-roudi has no option to move it. A container with a private /tmp therefore never finds the broker and hangs, even though the shared memory itself is perfectly visible. So every DDS container mounts the host /tmp, and so does the devcontainer via runArgs. Measured container-to-container on a stock kernel: 11.95 Hz.

If ROS nodes start hanging for a minute and dying, check ls -l /tmp/roudi first — not pgrep, which cannot see into another container’s PID namespace. Starting a second RouDi is refused rather than damaging (ICEORYX_ROUDI_MEMORY_MANAGER__ROUDI_STILL_RUNNING).

And every participant has to share a uid:gid. RouDi scopes each segment to the group that created it, so a client in a different group is refused at registration — which is the same 60 s hang, not a warning. A root client and a uid-1000 RouDi reject each other in both directions. All three images therefore take USER_UID/USER_GID build arguments and run as that user rather than root; the devcontainer gets there on its own through updateRemoteUserUID. It has to be a build argument and not --user at run time, because RouDi names the segment after getgrgid(), so the gid must exist in /etc/group. If your host is not 1000:1000, export them before building:

export USER_UID=$(id -u) USER_GID=$(id -g)
docker compose -f docker/docker-compose.yml build

The alternative, and why it is not the default. Raising the host limit reaches the same 12 Hz:

sudo sysctl -w net.core.rmem_max=16777216 net.core.wmem_max=16777216
# lost on reboot; to keep it:
printf 'net.core.rmem_max=16777216\nnet.core.wmem_max=16777216\n' \
  | sudo tee /etc/sysctl.d/30-kraken-dds.conf

That needs root, applies machine-wide, and lives outside the repository, so a fresh checkout on an untuned machine reproduces the bug with nothing to suggest why. It is harmless to anything else sharing the machine — rmem_max is a ceiling, not an allocation, so raising it makes no process consume more — but the shared-memory route is self-contained, which is why it is the one wired up. The socket buffer sizes in docker/cyclonedds.xml no longer carry the images at all, but they still govern every other topic: min stays at 64 kB deliberately, because Cyclone treats a min it cannot reach as fatal, so a hard 16 MB floor would make the launcher exit with “failed to increase socket receive buffer size” on an untuned host rather than merely running lossy.

Recorded as not fixes, because they look like ones: raising MaxMessageSize to 65500 B to cut 686 fragments per frame down to 15 measured worse at 0.25 Hz, since coarser datagrams overrun a small buffer faster; and FastDDS, whose shared memory is on by default but whose default 512 kB segment is too small to hold a frame, so it silently falls back to UDP. Given a 64 MB segment it matches Cyclone at 11.95 Hz, but adopting it would mean re-deriving the loopback discovery pinning for the whole stack.

What none actually buys. It leaves the camera entities in place and clears Publishing Enabled. In the gem that flag is tested in the tick callback before FrequencyTick, which is the only thing that asks for a frame, and the pipeline is taken off the render tick when it is built so it draws only on request. A disabled camera therefore does no per-frame GPU work: none is a genuine zero-render baseline.

What it does not free is memory. The CameraSensor and its render-to-texture pipeline are built during Activate whatever the flag says, so a disabled camera still holds its render target. Read the reference rate as the cost of rendering, not as the cost of existing.

3. Which one to use

  headless O3DE
GPU no yes, Vulkan
Clock owns /clock, up to 3× wall speed real time, both levels
Ranging none RGL lidar
Cameras none two, colour, 12 Hz
Physics kinematic; traction as a scalar contact, friction, load transfer
Used by the whole scenario suite the coverage runs
Bring-up seconds hours, the first time

The fault injector sits between the sensors and the filter in both cases and neither simulator knows it exists. That is what lets the same scenario file run against either — and it is why simulator:=o3de is a launch argument rather than a different stack.

Magnitudes do not carry across. The headless model scales commanded velocity by a traction factor; O3DE gives real friction instead. The mechanism a scenario demonstrates carries over, the numbers do not.

4. Running them

Both are in getting started. Image layout, GPU flags and the ROS distribution build arguments are in docker/README.md; what an authored level has to provide is in Project/README.md.