Real-time stress harness (recorded-data replay)¶
gnss realtime-stress replays a recorded RINEX run as RTCM3 over local TCP or a minimal
local NTRIP caster, paced against the wall clock, runs gnss live against those sockets,
injects link faults and measures what gnss live does. There is no receiver in the loop.
What this is and is not
This is a recorded-data replay on loopback, not a field soak. It shows how gnss live
behaves when a faithful byte stream is late, missing, corrupted, split or cut. It does not
show behaviour on a real network (MTU splitting, bufferbloat, caster quirks), with a real
receiver, or over days. The fault, soak and pacing results below are GPS-only;
multi-constellation MSM was checked separately (see
Multi-constellation MSM). The "soak" below is a 60 minute wall-clock run at 60x that loops the 40 minute
PPC run about 90 times (about 62 hours of stream time), not a 60 hour test.
Commands¶
cmake --build build --target gnss_live gnss_rtcm_encode gnss_solve
# 1. Encode a recorded run: rover/base RINEX -> RTCM3 MSM7 + 1005 + 1019, plus an epoch index.
python3 apps/gnss.py realtime-stress prepare \
--run-dir data/PPC-Dataset/tokyo/run1 --out-dir output/rt/prep --duration 420
# 2. One scenario: pacing 1x/5x/20x, TCP or NTRIP, any number of faults, parity vs gnss_solve.
python3 apps/gnss.py realtime-stress run --prepared output/rt/prep --out-dir output/rt/r1 \
--speed 1 --transport ntrip --batch-parity \
--fault base_outage@100+10 --fault corrupt_rover@200+0=0.05
# 3. A named matrix (smoke | faults | faults_ntrip), several scenarios in parallel.
python3 apps/gnss.py realtime-stress suite --prepared output/rt/prep --out-dir output/rt/faults \
--speed 1 --preset faults --jobs 4
python3 apps/gnss.py realtime-stress report --suite-dir output/rt/faults
# 4. Soak: loop a window with time-shifted (orbit-consistent) data until a wall-clock cap
# (prepare without --duration for the whole run).
python3 apps/gnss.py realtime-stress run --prepared output/rt/full --out-dir output/rt/soak \
--speed 60 --loops 0 --max-wall 3660 --compact
gnss rtcm-encode is the underlying encoder (--systems G,R,E,J,C, --format msm7|msm5|msm4|legacy,
--station, --rinex-out, --eph-systems). prepare --eph-systems G,R,E,J,C puts Galileo, QZSS and BeiDou
ephemerides (1045/1046, 1044, 1042) in the base stream next to 1019/1020; it is passed to the base encode only
and the default (not passed, so the encoder's G,R) leaves the command line and the bytes unchanged. Only
systems that are also in --systems are broadcast. See Ephemerides in the base
stream. --loops time-shifts GPS MSM and 1019 frames only, so a looped
soak of a stream with 1042/1044/1045/1046 keeps their original times. The ctest lane python_realtime_stress_tests replays a tracked 40 s
fixture (tests/fixtures/realtime_stress) at 20x against the built gnss_live in about 17 s
and checks survival plus detection of every fault kind.
Fault syntax¶
--fault kind@at+dur=value, times in stream seconds from the start of the window (so a scenario has
the same shape at 1x and 20x):
| kind | effect |
|---|---|
base_outage@t+d / rover_outage@t+d |
link up, data dropped for d s |
base_stall@t+d |
data held for d s, then delivered in a burst |
base_latency@t=v |
constant extra delay v s on the base stream from t |
base_jitter@t=v |
random delay U(0,v) s per base epoch from t; epochs can be reordered |
corrupt_base@t=p, corrupt_rover@t=p |
flip one random bit in a fraction p of frames (CRC failures, bad lengths) |
rover_disconnect@t+d, base_disconnect@t+d |
server closes the connection and refuses reconnects for d s |
server_restart@t+d |
both servers go down for d s |
base_blackhole@t, rover_blackhole@t |
from t the server stops sending on the connection that exists at t but keeps it open (no FIN, no RST) until the client closes it or the run ends. A connection made after t is served normally from the live position in the schedule. Open ended: dur must be 0 (base_blackhole@100+10 is rejected). Exercises gnss live --stall-timeout-ms. |
--supervise relaunches gnss_live when it exits early (what an init system would do);
--reconnect runs it with its own --reconnect (--reconnect-delay-ms sets its delay, default 200;
the attempt budget is sized by the harness). Any other gnss_live option goes through --live-arg=..., e.g.
--live-arg=--stall-timeout-ms --live-arg=5000 for the stall watchdog.
Every fault report also carries first_solution_after_s (stream time between the fault start and the moment
gnss_live wrote the first row for data at or after it), first_solution_data_s (data time of that row minus the fault start)
and solutions_from_30s_after (rows with data time from t+30 to the end of the stream). A blackhole report adds
rover_stalls_reported / base_stalls_reported (from the summary: line) and the server's epochs_blackholed.
What is measured¶
| metric | definition |
|---|---|
| survival | one gnss_live process ran to the end of the stream and exited 0 (soak: still running when the feed was halted) |
| output vs input | solved epochs written / rover epochs sent |
| latency | solution row first seen in --out minus the time that rover epoch was scheduled on the socket (p50/p95/p99/max). The scheduled time is when the data was available to the client |
| recovery | stream seconds from the end of a fault to the first FLOAT / FIXED output, next to the same figure in the clean run |
| accuracy | ENU error against reference.csv for the 30 s before, during, 0-30 s and 30-90 s after each fault, next to the clean run in the same windows; wrong fixes are FIXED epochs with a 3D error above 0.5 m |
| detection | whether gnss_live itself reports the fault (decoder errors, held/skipped epochs, output gap, process exit, reconnect) |
| RSS / CPU | /proc/<pid> once per second, linear RSS slope after a warm-up, CPU seconds |
| parity | the same epochs from gnss_solve on the encoded RINEX (vs_gnss_solve) and from gnss_live over the same RTCM files with no sockets (vs_gnss_live_files) |
Output per scenario: result.json, epochs.csv (one row per solution with wall time, latency, ENU error),
resources.csv, server_{rover,base}.csv (what was sent and when, and which fault touched it), the
.pos and gnss_live log. A suite also writes suite_summary.json. Nothing needs matplotlib.
Method notes¶
- Data: PPC Tokyo run1, rover 5 Hz and base 1 Hz, GPS only (L1/L2/L5) encoded as MSM7, 1005 every 10 s and 1019
round-robin so each satellite repeats every 30 s. These runs predate multi-constellation epoch assembly, when
gnss_liveconsumed one observation message per epoch, so only one constellation per stream was usable. - Loops shift every timestamp by a multiple of 16 s and rotate the right ascension of each 1019 by
OMEGA_E * shift, so the shifted broadcast orbit gives the same ECEF track: each loop is the same physics at a later time. The loop gap is 60 s of silence. - Two
gnss_liveoption sets are used.moving(the harness default) adds--max-pos-jump-min 5 --max-pos-jump-rate 30, the motion-aware fixed-jump gate a vehicle needs;defaultis baregnss_live. The bare static 5 m gate refuses fixes for a moving rover (first 300 s: 15 % fixed against 79 %), butmovingaccepts more wrong fixes in the urban canyon. The stress numbers are about the streaming behaviour; neither profile is a statement about RTK tuning. - Hardware: 4 vCPU VM, Release build, Python 3.13 server threads, loopback only.
Results¶
All numbers: PPC Tokyo run1 (2390 s, 5 Hz rover, 1 Hz base), GPS-only MSM7, loopback, 4 vCPU, Release build.
"Before" is gnss_live as of origin/develop (ae177d37); "after" includes the fixes
listed below. moving option set unless noted. Raw JSON/CSV are produced by the commands above.
Pacing runs: 1x for 16.7 min, 20x for the full run¶
| run | wall | survives | solutions / rover epochs | fixed | accuracy vs reference | latency p50 / p95 / max | CPU, RSS |
|---|---|---|---|---|---|---|---|
| 1x, first 1000 s | 1000.8 s | yes, exit 0 | 3441 / 4996 | 70.4 % | 3D p50 6.5 cm, p95 22.8 m, 425 wrong fixes (> 0.5 m) | 0.41 / 0.80 / 0.83 s | 0.5 % of a core, RSS 27 MB, slope after warm-up 0.06 MB/h |
| 20x, full run | 120.3 s (19.87x) | yes, exit 0 | 7829 / 11919 | 49.5 % | 3D p50 3.29 m, p95 36.8 m, 1055 wrong fixes of 3874 | 0.025 / 0.044 / 0.066 s | 8.8 % of a core, RSS 27 MB |
20x, full run, bare gnss_live |
120.3 s | yes, exit 0 | 7611 / 11919 | 2.9 % | 3D p50 2.77 m, p95 36.7 m, 0 wrong fixes | 0.025 / 0.044 / 0.075 s | 8.4 % of a core, RSS 27 MB |
- Solutions are fewer than epochs because GPS alone often has fewer than 4 usable satellites in this urban run.
- The latency is the wait for the next 1 Hz base epoch, which
gnss liveneeds for interpolation: at 1x it is uniformly 0 to 0.8 s (p50 0.41 s), at 20x it is the same 0.5 s of stream time shrunk to 25 ms. It is not processing time.gnss liveover the files processes the whole run in 9.6 s (about 250x real time) on one core.
Parity¶
| comparison | epochs | result |
|---|---|---|
gnss_live over sockets (paced) vs gnss_live over the same RTCM files, 1x/1000 s, 20x/full (moving) and 20x/full (bare) |
3441 / 7829 / 7611 | byte-identical .pos rows in all three |
vs batch gnss_solve --mode kinematic on the same encoded epochs, bare, full run |
6244 common (1367 only live, 93 only batch) | status agreement 98.5 %; both fixed (223): position difference p50 1.2 cm, p95 4.0 cm, max 39 cm; all statuses p50 0.67 m, p95 14.2 m |
same, moving, full run |
5813 common (2016 only live, 88 only batch) | status agreement 76.4 %; both fixed (2755): p50 0.9 cm, p95 8.6 m (a wrong fix on one side), max 60 m |
same, moving, first 1000 s |
2762 common (679 only live) | status agreement 91.3 %; both fixed (2014): p50 0.8 cm, p95 9.5 m |
Streaming does not change the solution. The differences against gnss_solve come from its extra batch logic
(non-fix drift guard, height-step guard, FLOAT bridge-tail guard, SPP fallback and so on, printed in its summary),
which gnss live does not have, and they grow with the looser moving fixed-jump gate.
Fault matrix, TCP, 1x, 420 s window, first fault at 100 s¶
"Before" and "after" are gnss_live before and after the fixes below; *_reconnect rows run
gnss live --reconnect. Times are stream seconds. "Recovery" is the time from the end of the fault to the first
FLOAT/FIXED output; "post-fault accuracy" is the 30 s after a windowed fault (everything after the start for
open-ended faults) with the clean run in brackets; "wrong" counts fixed epochs with a 3D error above 0.5 m.
Outage, stall, disconnect and restart last 10 s ("long": 30 s); latency 2 s; jitter up to 3 s; corruption flips one bit in 5 % of frames.
| scenario | before: survives | after: survives | out epochs (clean 2086) | latency p50/p95/max (s) | max gap (s) | recovery FLOAT/FIX (s) | post-fault accuracy | reported before/after |
|---|---|---|---|---|---|---|---|---|
| base_jitter | yes | yes | 1807 | 1.29/3.00/3.76 | 3.0 | 0.0/0.0 | 63% fix, 15 cm p50, 119 wrong (clean 75%, 5 cm, 117) | Y/Y |
| base_latency | yes | yes | 2086 | 2.21/2.80/2.84 | 0.2 | 0.0/0.0 | 75% fix, 5 cm p50, 117 wrong (clean 75%, 5 cm, 117) | Y/Y |
| base_outage | yes | yes | 2032 | 0.41/0.81/0.87 | 11.0 | 0.0/0.0 | 100% fix, 3 cm p50, 0 wrong (clean 100%, 3 cm, 0) | Y/Y |
| base_outage_long | yes | yes | 1932 | 0.41/0.81/0.84 | 31.0 | 0.0/0.0 | 100% fix, 3 cm p50, 5 wrong (clean 100%, 3 cm, 0) | Y/Y |
| base_stall | yes | yes | 2086 | 0.41/0.81/10.81 | 2.0 | 0.0/0.0 | 100% fix, 3 cm p50, 0 wrong (clean 100%, 3 cm, 0) | Y/Y |
| baseline | yes | yes | 2086 | 0.41/0.81/0.84 | - | - | - | -/- |
| combined | yes | yes | 1989 | 0.80/1.80/2.81 | 11.0 | 0.0/0.0 | 100% fix, 3 cm p50, 0 wrong (clean 100%, 3 cm, 0) | N/Y |
| corrupt_base | yes | yes | 1969 | 0.41/0.81/3.80 | 5.0 | 0.0/0.0 | 74% fix, 5 cm p50, 111 wrong (clean 75%, 5 cm, 117) | N/Y |
| corrupt_rover | yes | yes | 1965 | 0.41/0.81/0.83 | 1.0 | 0.0/0.0 | 74% fix, 5 cm p50, 90 wrong (clean 75%, 5 cm, 117) | N/Y |
| rover_disconnect | no | no | 500 | 0.41/0.80/0.81 | 0.0 | -/- | n/a (process ended) | Y/Y |
| rover_disconnect_reconnect | n/a | yes | 2035 | 0.41/0.80/0.84 | 10.4 | 0.2/0.2 | 100% fix, 3 cm p50, 0 wrong (clean 100%, 3 cm, 0) | -/Y |
| rover_disconnect_supervised | no (10 relaunches) | no (10 relaunches) | 1968 | 0.41/0.80/0.82 | 23.4 | 13.2/14.0 | 38% fix, 246 cm p50, 32 wrong (clean 100%, 3 cm, 0) | Y/Y |
| server_restart | no | no | 497 | 0.41/0.80/0.82 | 0.0 | -/- | n/a (process ended) | Y/Y |
| server_restart_reconnect | n/a | yes | 2025 | 0.41/0.80/0.81 | 12.4 | 1.4/1.4 | 100% fix, 3 cm p50, 0 wrong (clean 100%, 3 cm, 0) | -/Y |
What the table shows:
- A base outage costs exactly the outage plus the interpolation look-ahead and nothing else. With a 10 s outage 54 epochs are lost (the 50 inside it and the 4 before it that were waiting for the next base epoch), the filter returns FIXED at the first epoch after the base is back, and the next 30 s are as good as the clean run (100 % fixed, 3 cm). After a 30 s outage 5 of the first 150 epochs are wrong fixes (clean: 0).
- Stalls and latency are absorbed by TCP. A 10 s stall is delivered as a burst and every epoch is solved with the clean-run accuracy; the price is latency (max 10.8 s, i.e. the stall). A constant 2 s base delay shifts every latency by 1.8 s and changes nothing else.
- Jitter that reorders epochs loses data: base epochs that arrive after a later one are unusable, 279 of 2086 solutions are lost, 63 % fixed afterwards instead of 75 %, 3D p50 15 cm instead of 5 cm.
- Corrupt frames do not crash
gnss_live; 5 % bad frames cost about 6 % of the solutions. The loss is bigger than the corrupted fraction because a corrupted length field swallows the next frames before the parser re-synchronises. Before the fix the run reportedrover_decoder_errors=0andbase_decoder_errors=0; now 118 and 38. - Losing the link ends the run unless
--reconnectis used: 500 of 2086 solutions, exit 0, no error. With--reconnectit resumes after 0.2 s (rover only) or 1.4 s (both servers restarted) with no loss of quality. Relaunching from outside (--supervise, one attempt per second; 9 of the 10 launches could not connect while the server was down) recovers FLOAT 13 s and FIXED 14 s after the link returns, but the cold start is poor: in the first 30 s only 84 epochs are solved, 38 % of them are fixed and every one of those fixes is wrong (32 epochs, median 3D error 2.5 m; clean: 100 % fixed, 0 wrong), because the loosemovinggate accepts early fixes.
Fault matrix, NTRIP, 1x¶
The same faults served by the local caster (ntrip://127.0.0.1:port/MOUNT, ICY 200 OK).
Outputs of 2086 are the clean run.
| scenario | before fixes | after fixes | after fixes + --reconnect |
recovery FLOAT/FIX with --reconnect (s) |
|---|---|---|---|---|
| baseline | runs to end, 2086 | runs to end, 2086 | runs to end, 2086 | - |
| base_outage | ends early, 497 | ends early, 497 | runs to end, 2032 | 0.0/0.0 |
| base_outage_long | ends early, 497 | ends early, 497 | runs to end, 1932 | 0.0/0.0 |
| base_stall | ends early, 497 | ends early, 497 | runs to end, 2086 | 0.0/0.0 |
| base_latency | ends early, 497 | ends early, 497 | runs to end, 2086 | 0.0/0.0 |
| base_jitter | ends early, 497 | ends early, 497 | runs to end, 1807 | 0.0/0.0 |
| corrupt_base | ends early, 522 | ends early, 522 | runs to end, 2037 | 0.0/0.0 |
| corrupt_rover | ends early, 600 | runs to end, 1985 | runs to end, 1985 | 0.0/0.0 |
| rover_disconnect | ends early, 500 | ends early, 500 | runs to end, 2035 | 0.2/0.2 |
| server_restart | ends early, 497 | ends early, 497 | runs to end, 2025 | 1.4/1.4 |
| combined | ends early, 497 | ends early, 497 | runs to end, 2018 | 0.0/0.0 |
- The NTRIP client reads with a 1 s select timeout and
gnss livetook one empty read for end of stream, so any gap longer than a second (a base epoch lost to a CRC error, a 2 s latency step, any outage) ended the run, 497 outputs in, with exit 0. A base epoch is 1 s apart, so even one dropped base frame is enough. - A partial frame in one TCP read was treated the same way (see the fixes), which is why 5 % corrupt rover frames stopped the NTRIP run at 600 outputs before and let it finish (1985) after.
- With
--reconnectevery NTRIP scenario above runs to the end.
Soak: 60 min wall at 60x, the 2390 s run looped with a 60 s gap (about 62 h of stream)¶
Same stream, same moving option set, --compact, RSS and CPU sampled each second from /proc.
| build | RSS 10 / 30 / 60 min | RSS slope | CPU | achieved speed | end |
|---|---|---|---|---|---|
origin/develop (before) |
69 / 145 / 209 MB | +175 MB/h | 43 % at 10 min, 98 % from 30 min (mean 86 %, 3226 CPU s) | 60x, then 51x at 30 min, 39x at 40 min, 30x at 60 min | feed halted at 60 min with a backlog (latency up to 14 s); 70 loops, 830k of 1.05M epochs delivered; did not exit within the drain timeout, killed |
| + ephemeris de-duplication | 132 / 133 / 133 MB | +0.4 MB/h after warm-up | 28 to 31 % (1063 CPU s) | 60x all hour | exit 0, summary line; 90 loops (221 686 s of stream), 1 062 244 rover epochs, 676 472 solutions |
| + bounded satellite-state memo (final) | 27 / 28 / 29 MB | +2.6 MB/h | 25 to 27 % (946 CPU s) | 60x all hour | exit 0, summary line; 90 loops, 1 062 237 rover epochs, 676 466 solutions |
The 132 MB plateau of the middle row is the satellite-state memo, which only empties when a new ephemeris is added (once per loop here); the last row caps it. The remaining 2.6 MB/h is the 90 loops' worth of genuinely new ephemerides (a real stream adds one set about every two hours).
gnss live was ingesting an ephemeris rebroadcast every 30 s per satellite as the stream really carries them
(60x means one every half second of wall time). Before the fixes every repeat was appended, sorted and invalidated
the satellite-state cache: memory grew by 175 MB/h, and the per-epoch ephemeris scan grew with it until the
process was CPU-bound and fell from 60x to 30x. The loops are physically consistent (in a 3-loop test of a 300 s window every loop reproduced the same 1179 fixes),
so the growth is receiver-side bookkeeping and not the data.
Other measurements¶
--no-base-interp --base-hold-seconds 1.0(the low-latency-looking option) has the same latency (p50 0.41, p95 0.81, max 0.81 s) because the loop still reads ahead to the next base epoch before it chooses between exact, interpolated and held; it also loses RTK completely with a 1 Hz base (0 % fixed, 3D p50 8.8 m).- Heavy corruption (30 % and 90 % of frames on both streams) neither crashes nor hangs
gnss_live: it ends withbase_source_exhausted(exit 0) after 39 solutions, or with exit 1 and no solution. - A 5-minute window encoded as G+R+E+J+C MSM7 gives 1 solution from 6004 rover messages (see Limits).
Bugs found and fixed¶
Each fix is its own commit with a regression test that uses the tracked recorded fixture
(tests/fixtures/realtime_stress) and needs no hardware or external data. For every fix the default
gnss live output on the full PPC run is byte-identical before and after (.pos and summary line apart from
timings, gnss_solve output unchanged); the behaviour changes only in the fault case.
| commit | problem found by the harness | evidence |
|---|---|---|
fix(live): report RTCM decoder errors ... |
rover_decoder_errors / base_decoder_errors (gnss live) and crc_errors / reconnect_count (gnss stream) were always 0: both tools read the reader's counters after close() had reset them |
TCP corrupt-base scenario: 0 reported before, 38 after (26 frames corrupted); the fault was undetectable from the summary |
fix(live): do not re-add rebroadcast RTCM ephemerides ... |
every rebroadcast 1019/1020 was appended to the navigation database, re-sorted, and cleared the satellite-state cache; memory and lookup cost grow without bound | soak: +175 MB/h and 43 % to 98 % CPU before; RSS flat afterwards (below) |
feat(live): opt-in --reconnect ... |
a dropped tcp:// link, or one NTRIP read that returns nothing for 1 s, ended the run; NTRIP auto-reconnect existed but gnss live never enabled it and tcp:// had none |
rover disconnect: 500 of 2086 solutions before, 2035 after with --reconnect; restart of both servers 497 to 2025; every NTRIP outage scenario runs to the end |
fix(live): bound the satellite-state memo ... |
NavigationData memoises satellite states per (satellite, time) and only clears the memo when an ephemeris is added; with a navigation RINEX and no RTCM ephemerides it grows with every epoch |
opt-in setSatelliteStateCacheLimit (default unbounded, so batch multi-pass solvers keep their cache), 16384 entries in gnss live; flat RSS in the final soak |
fix(ntrip): a partial RTCM frame in one read ... |
NTRIPClient::readMessages returned "no message" when a recv() ended inside a frame, and gnss live took that for the end of the stream; any TCP segmentation of a frame, or a corrupted length byte, stopped the run |
recorded stream with every 10th epoch split over two segments: 10 solutions before, 201 of 201 after; NTRIP corrupt-rover 600 to 1985 outputs |
--reconnect[-delay-ms N][-attempts N] is opt-in on purpose: without it a finite NTRIP stream still ends
the run, and the summary line only gains rover_reconnects= / base_reconnects= when it is given.
Wrong FIX on the real-time path¶
Do not trust a gnss live FIX in a dense urban canyon. Even with all
constellations, gnss live --preset low-cost published FIX epochs that were more
than 0.5 m off on two of the four UrbanNav Tokyo runs at rates above 5 %. These
are offline replays of the 2018-12-19 runs: G+R+E+J+C MSM7, --nav-rinex, the
surveyed base and the real host clock, measured against a contract frozen
before the run on 2026-10-12. The numbers below were measured on develop
52853076; the contract was first run on b9a6e243, and the multi-GNSS
numbers changed between the two (see the last bullet). The Odaiba u-blox row
was measured again on 2026-10-15, after the
Doppler slip fix; the other
rows are unchanged by it.
UrbanNav Tokyo, --preset low-cost |
FIX, GPS-only → multi | wrong FIX, GPS-only → multi | Wrong/FIX, GPS-only → multi | FIXED 3D p95, multi |
|---|---|---|---|---|
| Shinjuku, Trimble rover | 3511 → 13636 | 1711 → 905 | 48.7 % → 6.6 % | 0.78 m |
| Shinjuku, u-blox rover | 623 → 4988 | 202 → 258 | 32.4 % → 5.2 % | 0.51 m |
| Odaiba, Trimble rover | 4419 → 7704 | 125 → 81 | 2.8 % → 1.1 % | 0.17 m |
| Odaiba, u-blox rover | 77 → 3456 | 4 → 43 | 5.2 % → 1.2 % | 0.17 m |
- Multi-GNSS fixes 3.9 and 8.0 times as many epochs on the two Shinjuku runs, 1.7 times as many on Odaiba Trimble and 45 times as many on Odaiba u-blox, and cuts Wrong/FIX on all four. The number of wrong fixes falls on Shinjuku Trimble (1711 → 905) and Odaiba Trimble (125 → 81) and rises on Shinjuku u-blox (202 → 258) and Odaiba u-blox (4 → 43). On Odaiba u-blox, GPS alone fixes 1.2 % of epochs and multi-GNSS 56 %.
- Most of these wrong fixes are decimetres to a few metres off, not tens of metres: see the FIXED 3D p95 column. That still matters for anything that trusts the FIX flag.
--kinematic-post-filter(opt-in) lowers multi-GNSS Wrong/FIX only slightly (6.6 → 5.0 %, 5.2 → 4.3 %, 1.1 → 0.4 %, 1.2 → 1.0 %). It cuts the all-solution 3D p95 several-fold (Shinjuku Trimble 65 → 5 m) by withholding 19–33 % of solutions. It does not remove wrong fixes.- On the PPC runs the multi-GNSS low-cost Wrong/FIX was 1.2–3.7 % (see Multi-constellation MSM), measured before #615 and not re-measured since.
gnss livelacksgnss solve's whole-solution post-filters.- The GPS-only numbers (48.7 % and 32.4 %) come from the GPS-only replay described below.
- Change since the first run of this contract (
b9a6e243, which gave Wrong/FIX 12.1 %, 5.5 %, 1.0 % and 22.2 % in the table order above, and 11.2 %, 4.8 %, 0.4 % and 20.5 % with--kinematic-post-filter): the encoded streams and all GPS-only arms are identical in the two runs, and the multi-GNSS low-cost arms differ. Rebuilding develop atf32ea3b5(before #615) reproduced all four earlier multi-GNSS low-cost results, and9d58ef51(#615, deterministic same-toe ephemeris order, Galileo I/NAV first) gave the new ones. The decision rule is unchanged: three of the four runs were still above 5 %, and two are after the Doppler slip fix.
A frozen-contract check (2026-10-11) of
--preset low-cost --kinematic-post-filter plus FixRecoveryGuard was
No-Go. The combination was chosen on PPC Tokyo run1 only and checked once on
nine other GPS-only replays: PPC Tokyo run2–3 and Nagoya run1–3, and UrbanNav
Shinjuku and Odaiba with the Trimble and u-blox rovers (surveyed base).
| Pooled over the 9 check runs | emitted | FIX | correct | wrong | Wrong/FIX |
|---|---|---|---|---|---|
--preset low-cost |
59472 | 26836 | 23470 | 3366 | 12.5 % |
+ --kinematic-post-filter + FixRecoveryGuard |
47064 | 13578 | 12787 | 791 | 5.8 % |
| Gate | Result |
|---|---|
| Wrong FIX −50 % | pass (−76 %) |
| Pooled Wrong/FIX ≤ 10 % | pass |
| Correct FIX kept ≥ 60 % | fail (54.5 %) |
| No run with a higher Wrong/FIX | fail: Tokyo run2 7.0 → 14.4 %, Shinjuku Trimble 48.7 → 98.8 % |
Per-flag ablation on the same nine runs, run after the gates (not a gate):
| Arm | emitted | FIX | correct | wrong | Wrong/FIX |
|---|---|---|---|---|---|
--kinematic-post-filter only |
47077 | 25654 | 23178 | 2476 | 9.7 % |
| FixRecoveryGuard only | 59467 | 14124 | 12909 | 1215 | 8.6 % |
- FixRecoveryGuard. It removes almost half of all FIX epochs. On Shinjuku
Trimble it keeps the wrong ones (346 FIX, 343 wrong), so it is not offered for
gnss live. --kinematic-post-filter. It is offered as an opt-in flag (see Interfaces). It keeps 98.8 % of correct FIX, makes no run's Wrong/FIX higher, and removes 26 % of wrong FIX.- Status of these numbers. They come from the ablation run after the No-Go
decision on the same check runs, so they describe the flag rather than prove
it. They are not evidence for a default. All of these GPS-only numbers were
measured before the
Doppler slip fix, which
changes the Odaiba u-blox run: GPS-only
--preset low-costnow fixes 77 epochs there, 4 of them wrong, against none before.
Doppler slip test and time-shifted base epochs¶
Fixed on 2026-10-15 in the RTK filter (src/algorithms/rtk_bias.cpp) and checked under a contract frozen before the
run. It changed gnss live and gnss replay on UrbanNav Odaiba with the u-blox rover. The numbers for that run in
this document, in gnss live and in Interfaces were measured
after the fix unless a table says otherwise.
- Cause.
gnss liveandgnss replayuse a base epoch within 1 ms of the rover epoch as it is and interpolate the others to the rover time. The Odaiba u-blox rover tags 6069 of its 6206 epochs 1 ms after the second. With a 5 Hz rover and a 1 Hz base the two kinds alternate, so the base epoch advances 0.199 s or 0.201 s while the rover advances 0.2 s. The Doppler slip test predicted the single-difference phase change with the rover's span for both receivers. At a Doppler of 1 kHz that is about one cycle off, against a threshold of 0.2 cycles. With--preset low-cost --hold-fix-min-matched-pairs 12, the test flagged false slips on about five satellites at a time at 93.5 % of the epochs at .0 s and .2 s, and an L1 ambiguity was reset on 41 % of all epochs. - Fix. The test now predicts the change with each receiver's own span. When the two spans agree within 1 µs, or the base span is not known, it evaluates the old expression, so time-matched data gives byte-identical output.
- Not affected:
gnss solve, which treats a base epoch as time-matched only within 1 µs and interpolates the others; base streams without Doppler (MSM4), where the test does not run; and rovers whose epochs fall on the base epochs. - Byte-identical
.posbefore and after: gnss solveon PPC Tokyo run1 and Nagoya run3, with and without--preset low-cost, and with--preset low-coston the seven urban development runs of Interfaces and the four UrbanNav Tokyo runs.gnss liveon the other five runs (PPC Tokyo run1 and Nagoya run3, UrbanNav Shinjuku Trimble and u-blox, Odaiba Trimble) with--preset low-cost, alone and with--hold-fix-min-matched-pairs 12, with that and--min-fix-satellites 4, and with--hold-fix-unchanged-set.gnss liveon the other three UrbanNav runs, GPS-only and multi-GNSS, without options, with--preset low-costand with--kinematic-post-filteradded.gnss replayon three UrbanNav runs.- Odaiba u-blox, multi-GNSS
gnss live --preset low-cost. Each cell is FIXED more than 0.5 m off / FIXED.gnss solveon the RINEX of the same run is shown for comparison.
| Options added | Before the fix | After the fix | gnss solve |
|---|---|---|---|
| none | 592 / 2588 (22.9 %) | 43 / 3456 (1.24 %) | 36 / 3693 (0.97 %) |
--hold-fix-min-matched-pairs 12 |
495 / 2476 (20.0 %) | 43 / 3456 (1.24 %) | 33 / 3690 (0.89 %) |
--hold-fix-min-matched-pairs 12 --min-fix-satellites 4 |
2 / 1201 (0.17 %) | 8 / 3253 (0.25 %) | 26 / 3606 (0.72 %) |
- Correct FIXED epochs went from 1996, 1981 and 1199 to 3413, 3413 and 3245.
- With
--hold-fix-min-matched-pairs 12, the epochs with a Doppler slip went from 37.6 % to 0.29 % (gnss solve: 0.7 %). gnss replay(rover RINEX, base RTCM,--preset low-cost) on the same run: 6 / 290 to 6 / 291.- Six-run totals (PPC 2 + UrbanNav Tokyo 4, multi-GNSS,
--preset low-cost): 2179 / 38903 (5.60 %) to 1630 / 39771 (4.10 %); correct FIXED epochs 36724 to 38141. - Without the preset (outside the contract):
- Multi-GNSS went from 9 / 219 to 0 / 461.
- With GPS only, one wrong fix lasted longer: 150 / 150 to 460 / 460, all about 1.1 m off with 4 to 6 satellites. The first 150, from the start of the run, are the same in both. Before the fix false slips ended that episode. After it the solution stayed about 1.1 m off for 310 more FIXED epochs within the next 73 s.
- GPS alone with
--preset low-costfixed 77 epochs, 4 of them wrong (none before).
Multi-constellation MSM (epoch assembly)¶
A receiver sends one MSM message per constellation with the same time tag. gnss live used to treat every
observation message as an epoch (limit 1 below), so only GPS-only streams worked. Two changes fix this, both on
by default:
- Messages are merged into one epoch until a message has the RTCM "multiple message" bit cleared, which adds no
latency. A time change also closes the epoch, so a lost last message never merges two epochs. The escape
hatch is
--no-rtcm-epoch-assembly(Interfaces).gnss rtcm-encodenow sets the bit on every observation message of an epoch except the last. - The decoder dates epochs from a GPST reference taken from the stream instead of the host clock (limit 4 below). Without that, a 2024 recording replayed on a 2026 host gave 1 solution, because GLONASS epochs took the host week.
Frozen-contract check (2026-10-12). It covered PPC Tokyo run2–3 and Nagoya run1–2, encoded G+R+E+J+C MSM7
(rover 5 Hz, base 1 Hz), with --nav-rinex base.nav and the real host clock. A first version without the
reference time was No-Go: the head epoch of every stream split in two, and the runs only worked with the
host clock set to the data date.
| gate | result |
|---|---|
| GPS-only path unchanged | 30/30 byte-identical comparisons against the previous develop (encoder, gnss live bare and low-cost, gnss convert, gnss solve) |
| assembled epochs | 8 streams: assembled = sent epochs, 0 duplicate times, 100 % equal to the per-message decoded union, 0 wrong week/TOW (one epoch per message before: 4x the epochs, ≤ 0.07 % equal) |
gnss live vs gnss solve on the same decoded epochs (bare) |
100 % of batch epochs common; status agreement 99.32–99.96 %; both-FIXED horizontal difference p50 0.22–0.42 cm |
| paced over TCP (Tokyo run3, 300 s, 20x) | byte-identical to the file run, exit 0 |
| host-clock independence | byte-identical .pos with the real clock and with the clock at the data date: multi-GNSS + --nav-rinex (Tokyo run3), and G+R with 1019/1020 in the base stream and no navigation RINEX (Nagoya run2, 98.1 % of rover epochs solved) |
Multi-GNSS against GPS-only, gnss live --preset low-cost (all six PPC runs, report only, wrong FIX = FIXED with 3D
error > 0.5 m):
| run | solutions / rover epochs, GPS → multi | FIX % of epochs, GPS → multi | wrong FIX, GPS → multi | Wrong/FIX, GPS → multi |
|---|---|---|---|---|
| Tokyo run1 | 0.66 → 0.99 | 34.3 → 66.1 | 451 → 247 | 11.0 % → 3.1 % |
| Tokyo run2 | 0.74 → 1.00 | 44.6 → 78.0 | 201 → 141 | 4.9 % → 2.0 % |
| Tokyo run3 | 0.79 → 1.00 | 49.6 → 64.7 | 530 → 274 | 7.0 % → 2.8 % |
| Nagoya run1 | 0.79 → 0.99 | 35.9 → 69.1 | 122 → 137 | 4.5 % → 2.6 % |
| Nagoya run2 | 0.60 → 1.00 | 30.6 → 45.7 | 126 → 52 | 4.4 % → 1.2 % |
| Nagoya run3 | 0.56 → 1.00 | 17.6 → 37.8 | 152 → 72 | 16.6 % → 3.7 % |
- Wrong/FIX was lower with multi-GNSS in 11 of the 12 run/profile pairs, bare and low-cost, and equal in 1. Wrong fixes do not go away: there are still tens to hundreds per run with the low-cost preset, and on Nagoya run1 the count rose (122 → 137) because many more epochs were fixed.
- Multi-GNSS costs CPU. The offline realtime factor fell from about 220–335x to about 31–71x, measured with four runs in parallel on a 4 vCPU VM. At 20x pacing the first 300 s of Tokyo run3 had a latency p50 of 0.35–0.84 s (0.05 s elsewhere). At 1x there is ample margin.
- Tokyo run1 was used while developing the change. Tokyo run2 and Nagoya run1 were check runs of the No-Go first version.
Ephemerides in the base stream (--eph-systems)¶
Until now the encoded base stream carried 1019 and 1020 only, so a G+R+E+J+C replay without --nav-rinex solved with
G+R only (Tokyo run3, 120 s, 20x: FIX 0.519 against 0.952 with --nav-rinex). gnss rtcm-encode --eph-systems
G,R,E,J,C adds 1045/1046 (Galileo F/NAV, I/NAV), 1044 (QZSS) and 1042 (BeiDou), and the library now decodes
1042/1044 and routes 1042/1044/1045/1046 like 1019/1020 (Interfaces).
Frozen-contract check (2026-10-12), against develop 9d58ef51 (which includes the deterministic same-toe ephemeris
order of #615): Go, all of G0 to G4 pass. It ran end to end on the four UrbanNav Tokyo runs (2018-12-19: Shinjuku
and Odaiba, Trimble and u-blox rovers; base base_trimble.obs, the DLR BCEmerge base.nav, Applanix reference.csv,
surveyed --base-ecef -3961904.9530 3348993.7578 3698211.7553), data that was not used while building the feature.
G+R+E+J+C MSM7, rover at its native rate (10 Hz Trimble, 5 Hz u-blox), base 1 Hz, gnss live --preset low-cost, offline on
the files. Arm S: base stream with --eph-systems G,R,E,J,C, no --nav-rinex. Arm N: the same streams with
--nav-rinex base.nav. Arm O: today's base stream (G,R), no --nav-rinex. The BCEmerge file has no
IONOSPHERIC CORR header lines, so neither S nor N applies a broadcast ionosphere model and the comparison isolates
the ephemeris path.
| gate | result |
|---|---|
| default output unchanged | 23 of 23 files byte-identical to develop 9d58ef51: the regenerated tracked fixture (harness prepare, equal to the tracked bytes), gnss rtcm-encode of Tokyo run1 base with --systems G and G,R,E,J,C (bytes, index, summary), the G+R+E+J+C rover stream, gnss live on the fixture (bare, --preset low-cost) and on Tokyo run1 multi-GNSS with --nav-rinex, gnss convert of the fixture (the wall-clock PGM / RUN BY / DATE header line excluded) |
encode, decode, compare (UrbanNav base.nav) |
5436 Galileo (4645), BeiDou (696) and QZSS (95) records: 100 % encoded, 100 % within 0.053 mm and 5e-16 s of the source record at nine times, toe-2 h to toe+2 h |
| RTKLIB 2.4.3 b34 parity, both directions | 100 % of the same records (our encoder into RTKLIB's decoder: 0.63 mm / 5e-15 s at worst; RTKLIB's encoder into ours: 0.053 mm / 5e-16 s); 693 of 696 BeiDou 1042 frames bit-identical to RTKLIB's |
| end to end, S against N | pass in all 4 runs: status agreement 99.4 to 100 % (criterion 97 %), S has 1.000 of N's solutions, both-FIXED horizontal difference p50 0 m (criterion 0.01 m) |
| safety, pooled | S Wrong/FIX 6.42 % (1867 of 29,059 FIXED), N 6.35 % (1836 of 28,916): +0.08 pp (limit +1.0 pp) |
| paced over TCP (Shinjuku Trimble, first 300 s, 20x), arm S | exit 0, 2987 of 2987 rows, .pos byte-identical to the offline run on the same files (latency p50 0.04 s, p95 0.05 s) |
Per run, gnss live --preset low-cost (report only for the O, N and S columns; wrong FIX = FIXED with 3D error above
0.5 m against reference.csv, nearest epoch within 0.25 s; ns is the mean NumSat column of the .pos):
| run | rover epochs | S/N solutions | S vs N status agreement | mean ns O, N, S |
FIX % O, N, S | wrong FIX O, N, S | Wrong/FIX O, N, S |
|---|---|---|---|---|---|---|---|
| Shinjuku Trimble | 20,783 | 1.000 | 99.4 % | 5.7, 10.7, 10.7 | 15.7, 65.6, 66.2 | 1649, 905, 898 | 50.5, 6.6, 6.5 % |
| Shinjuku u-blox | 10,476 | 1.000 | 99.8 % | 5.4, 10.8, 10.8 | 5.9, 47.6, 47.6 | 202, 258, 264 | 32.4, 5.2, 5.3 % |
| Odaiba Trimble | 12,399 | 1.000 | 100.0 % | 5.4, 11.3, 11.3 | 35.6, 62.1, 62.1 | 125, 81, 81 | 2.8, 1.1, 1.1 % |
| Odaiba u-blox | 6206 | 1.000 | 99.6 % | 5.6, 11.8, 11.8 | 0.0, 41.7, 42.1 | 0, 592, 624 | 0.0, 22.9, 23.9 % |
- These per-run numbers and the gates above were measured before the Doppler slip fix of 2026-10-15, which changes the Odaiba u-blox run. They were not measured again.
- All five criteria pass in every run, including mean
nsof S at least 5.1 above O: Galileo, QZSS and BeiDou satellites are used from the stream alone. S and N agree on the status of 99.4 to 100 % of the epochs. The both-FIXED horizontal difference is 0 m at p50 and p95 in every run; the largest single difference is 2.1 m (Odaiba u-blox). - The Wrong/FIX levels are those of the urban canyon, not of the ephemeris source: S and N differ by at most 1.0 pp in a run (Odaiba u-blox 23.9 against 22.9 %), and pooled O has 23.8 % (1976 of 8308) on fewer FIXED epochs. Keep the caution in Wrong FIX on the real-time path for these receivers.
- Base stream size: O 6.7 to 7.2 kbit/s, S 7.9 to 8.3 kbit/s, 1.2 kbit/s more with the five systems.
- Realtime factor offline (three
gnss_liveprocesses in parallel on a 4 vCPU VM, so it is noisy): O 156 to 314x, N 33 to 96x, S 35 to 106x.
The first version of this check (v1, PPC Tokyo run2-3 and Nagoya run1-3, against develop f32ea3b5) was No-Go on one
criterion: status agreement of S against N was 77 to 91 % against 97 %. The cause was not the ephemerides. The PPC
navigation files carry an IONOSPHERIC CORR header that only the --nav-rinex arm reads (without a navigation RINEX
gnss live applies no Klobuchar model in its SPP seed, because nav.ionosphere_model.valid is false), and the Galileo
F/NAV against I/NAV choice at an equal toe depended on insertion order, which #615 fixed. A stream carries no ionosphere
parameters, so with a PPC-style navigation file --nav-rinex is not equivalent to ephemerides in the stream.
QZSS: the UrbanNav base.nav holds 95 real QZSS records (J01, J02, J03, J07) and the PPC navigation files hold
26 to 75 per file; they were used in arms S (as 1044) and N (from the file), and in the round trips and RTKLIB parity
above. No 1044 stream from a real receiver or caster was available, so no 1044 message has been checked as a
live broadcast; QZSS rests on the real records through the encode, decode and compare round trips and the bit-level
comparison with RTKLIB.
Base-outage SPP fallback¶
gnss live --base-outage-fallback spp (Interfaces)
answers the remainder of limit 3 below: during a base outage the rover epochs that cannot get a base are solved with the
library's SPP solver and written with status SPP, never FLOAT/FIXED, and the RTK state is not touched.
Each live source gets a reader thread so the rover keeps flowing while the base is silent. Measured on the same setup as the stall
watchdog (PPC Tokyo run1, GPS only, 420 s window, 5 Hz rover / 1 Hz base, moving profile, TCP, first fault at 100 s, one run each, four
runs in parallel on a 4 vCPU VM; the gates were fixed before the first scored run):
| run | result |
|---|---|
base_outage@100+30, no flag |
0 of 141 rover epochs in [101 s, 129 s] get a row; 154 rover epochs skipped without output |
base_outage@100+30, --base-outage-fallback spp |
141 of 141 rows, all SPP; latency p50 0.00 s, p95 0.00 s, max 0.04 s; SPP 3D error vs the reference p50 8.5 m, p95 9.4 m, max 9.5 m (code-only, urban GPS-only, ~7 satellites) |
| recovery | first FLOAT row after the outage written at 130.00 s (no flag: 130.00 s); FLOAT+FIXED rows from 160 s to the end: 1100 against 1100 without the flag |
base_blackhole@100 + --stall-timeout-ms 5000 --reconnect --reconnect-delay-ms 2000, no fallback |
0 of 26 rover epochs in [101 s, 106 s] get a row |
| the same with the fallback | 26 of 26, all SPP; the run ends with exit 0; FLOAT+FIXED rows from 130 s: 1250, as in the clean run |
| clean 1x and 20x, with the flag | .pos byte-identical to the run without it, fallback_spp_epochs=0 |
Details worth knowing: the flag changes nothing without an outage, and nothing at all when it is off (the .pos of the default
path is byte-identical to the previous release on the fixture, on Tokyo run1 and on a clean TCP run). Epochs within --base-hold-seconds of the
last base epoch are still RTK epochs with the held base; only the epochs after that become SPP, a silence timeout of
max(1 s, hold + base interval) (1.5 s here) after the newest base epoch, so the first rows of an outage are written about 1.3 s late
(those are the held FLOAT rows) and the SPP rows after that on arrival. The timeout is wall time on the main thread, so a base that is merely
slower than 1.5 s per epoch would be treated as in outage.
Limits found and not fixed¶
Found by the harness, deliberately not changed here (each needs a design decision or is larger than a bug fix):
gnss livehandles one observation message per epoch, so real multi-constellation MSM does not work. (Addressed: MSM epochs are now assembled using the multiple-message bit, see Interfaces; the text below describes the behaviour before that change.) Each constellation arrives as its own MSM message with the same time tag, andreadNextObservationtreats every observation message as an epoch. A 5 min window encoded as G+R+E+J+C MSM7 produced 1 solution (6004 rover messages, 4800 skipped epochs, a bogus 7e7 s solution span).gnss replayhas the same assumption. The harness therefore replays GPS only (--systems G). The fix is an epoch assembler that merges messages with the same time, using the MSM "multiple message" bit that the decoder currently discards (no extra latency) or the next time tag.- The latency floor is one base interval. A rover epoch is only solved after the next base epoch has been read
(p50 0.41 s, max 0.82 s with a 1 Hz base), also with
--no-base-interp, which only changes which base is used. A real low-latency mode would solve on arrival with the held or extrapolated base (see online base extrapolation).
A frozen-contract check of such a mode (2026-10-12) was No-Go, and the code was not merged. The candidate:
* solved each rover epoch on arrival;
* used the exact base if one was already read, otherwise the latest earlier base carried to the rover time by the
causal geometry-corrected hold rtk_base_alignment::holdBaseEpoch, up to 2.0 s old.
Check runs were PPC Tokyo run2–3 and Nagoya run1–3: G+R+E+J+C MSM7, --preset low-cost, --nav-rinex.
| default (interpolating) | solve on arrival | |
|---|---|---|
| latency p50 / p95, 1x TCP, 300 s of Tokyo run2 | 0.416 / 0.816 s | 0.007 / 0.022 s |
| FIX, pooled | 28,576 | 28,131 (98.4 %) |
| wrong FIX (> 0.5 m), pooled | 676 (2.37 %) | 695 (2.47 %) |
| Wrong/FIX, Nagoya run3 | 72 / 1964 = 3.67 % | 153 / 2148 = 7.12 % (gate: ≤ +2 pp) |
| median horizontal error of correct FIX | 0.0280 m | 0.0294 m |
- The latency goal was met, and the pooled numbers barely moved.
- Per run, Wrong/FIX changed by −0.8 to +3.5 pp, and Nagoya run3 nearly doubled.
- Tokyo run1 was used during development and was not scored. On it, Wrong/FIX went from 3.1 % to 10.8 %.
- Holding a base forward over a sub-second gap is therefore not safe enough on this data to ship, even opt-in.
- A follow-up, such as a shorter maximum age or never declaring FIXED on a held base, needs check runs that have not been seen. These PPC runs now have been.
- TCP reads have a watchdog only when asked for. A silent but connected
tcp://source blocks inread()with no timeout, so a dead peer that never sends a FIN blocksgnss live; thebase_blackhole/rover_blackholefaults reproduce it. Opt-in fix:gnss live --stall-timeout-ms <ms>(with--reconnect, see Interfaces); the default0keeps the blocking read. Still the case: the flag is off by default; it coverstcp://and serial/device reads but not NTRIP (which keeps its ownselect()timeout and does not reconnect on a quiet caster) or files; serial/device sources cannot reconnect, a stall ends them; the rover is not read whilegnss livewaits on the base (and vice versa), so during a base outage the rover socket just fills and recovery starts only after the stall timeout plus the reconnect delay; rover epochs that cannot get a base within 2 s are skipped without any output. Fixed, opt-in:gnss live --base-outage-fallback sppreads every live source in its own thread and writes an SPP row (statusSPP, neverFLOAT/FIXED, RTK state untouched) for each rover epoch that has no base aftermax(1 s, base hold + base interval)of wall time (see Base-outage SPP fallback). Still the case: the flag is off by default; the fallback is single-point positioning (meters, no dead reckoning), a base that never delivers its first epoch or whose source has ended still stops the run, NTRIP reads keep their own bounded select() wait, and the rover is still waited for when it is the silent source. - The GPS week. Fixed for
gnss live:RTCMStreamDecoderkeeps one GPST reference (seeded from--nav-rinex, else from the first GPS/Galileo/BeiDou/QZSS ephemeris in either stream), resolves TOW-only observations to the nearest week, follows the stream across a week rollover, and dates GLONASS MSM epochs and 1020 ephemerides from it instead of the host clock, so replaying old recordings no longer depends on the host clock when a reference exists. Not fixed: a stream with neither--nav-rinexnor any GPS/Galileo/BeiDou/QZSS ephemeris before its first epochs has no reference (head epochs stay week 0, GLONASS is host-clock dated); a GLONASS 1020 that arrives before any reference is still dated from the host clock; and the 10-bit week of a 1019 is still expanded around the host clock until a reference exists.gnss replay --base-rtcmnow uses the same decoder and reference (seeded from--nav-rinex, else from the first in-stream ephemeris); before that its rawRTCMProcessorleft GPS/Galileo/QZSS/BeiDou parts in week 0 and dated GLONASS from the host clock, so a GPS-only base never aligned to a RINEX rover and a multi-GNSS one only matched on the GLONASS parts, and only when the host clock was near the data date. - Cold start after a restart is optimistic. With the
movingoption set the first 30 s after a relaunch contained only wrong fixes (32 epochs, median error 2.5 m), and the bare static gate refuses most fixes of a moving rover (2.9 % against 49.5 % fixed over the run, but 0 against 1055 wrong fixes).gnss livehas no option set that is good at both. gnss liveandgnss solveare different algorithms. Streaming is exact (byte-identical to files), butgnss_solveadds batch guards, so the same epochs agree in status only 98.5 % (bare) or 76 % (moving).
Limitations of this harness¶
- Loopback only: no real packet loss, reordering, MTU splitting, congestion or caster behaviour. The partial-frame bug was found through fault injection (corrupted length bytes) and confirmed with a hand-split stream, not seen on a network. The NTRIP caster is minimal (no authentication, no source table).
- One dataset run (PPC Tokyo run1) and a GPS-only subset of it; Nagoya and runs 2 and 3 are not run. The encoder
quantises to MSM7 and writes only the satellites
gnss_livecan use. - Faults are injected at the server; the receiver side of the link is real code, the server side is Python (timing jitter about 1 ms, GIL contention at hundreds of epochs per second).
- Latency uses the scheduled send time, so at speeds where
gnss_livefalls behind (the unfixed soak) it understates the lag; use the achieved stream speed there. - The soak is accelerated (60x) and time-shifted; it exercises memory and CPU growth per ephemeris rebroadcast and per epoch, not hours of real signal conditions. Looped fix statistics are not accuracy figures.
- Wall-clock figures come from a shared 4 vCPU VM with other jobs running; compare runs by ratio, not by the last digit.