September 2026 — Same recipe, different bytes

The previous post converted the litter detector on a CPU and found that a detection count is not a parity check. This one takes the same graph to two places it had never been: a TensorRT engine on a desktop GPU, and a Raspberry Pi 5 — the board CrustBot actually carries, where /benchmarks has since May said that no litter-model number exists.

Nothing below was deployed. Production still serves the same fp32 ONNX under onnxruntime-node on an arm64 Lambda, and no artifact here has ever answered a request.

Two results turned out to matter more than the conversions themselves, and they are mirror images of each other:

• Same recipe, different bytes. One graph, one builder config, three builds — three different plan files, three different sizes, none bit-identical.
• Same bytes, different decisions. One int8 file, byte-identical input tensors, two CPU architectures — 18 query-level decisions land differently.

Between them they say that neither the recipe nor the artifact hash is sufficient on its own to identify what a model will do.

The engines exist

Built with a scripted builder (export.trt_build) rather than trtexec, which is not shipped in the tensorrt pip wheel. The ONNX parsed with no unsupported layers — RF-DETR’s TopK and attention blocks converted without complaint into a 3,863-layer network.

The first surprise is a version change, not a measurement. TensorRT 11 removed the FP16 and INT8 builder flags. The only type-related creation flag left is STRONGLY_TYPED, and precision now rides on the ONNX graph itself. “Build an fp16 engine” has become “convert the ONNX, then build” — precision is an artifact-level decision, not a builder-level one.

So the two engines built here differ on the one precision-adjacent thing that is still a builder flag, TF32, which is on by default:

Host for everything in this section: Windows 11, NVIDIA GeForce RTX 4070, compute capability 8.9 (Ada), 46 SMs, driver 591.86, CUDA runtime 13.1, TensorRT 11.2.1.2. A desktop GPU under WDDM. Not an Orin, not an embedded part, not the arm64 Lambda that serves today. Every latency number below names its host for that reason.

Build the same graph three times, get three different files

The point of a release registry is that a set of bytes has an identity. So: same ONNX, same builder config, build it repeatedly.

Without a timing cache — not deterministic, and not equivalent.

Three hashes and three different sizes, spread 71,856 bytes. Different sizes mean the builder picked genuinely different tactics, not that it serialised the same engine differently. Run all three over 20 real fixtures: 0 of 20 images bit-identical between any pair, max absolute delta on labels up to 3.487 and on dets up to 0.908.

For scale: the fp32→fp16 conversion measured on CPU last month had a max labels delta of 12.17. Two builds of one graph from one config are the same order of magnitude apart as a precision conversion.

With a timing cache supplied — behaviourally deterministic, still not byte-identical.

Identical sizes, three different hashes. Over those same 20 fixtures the three plans were bit-identical on 20 of 20 images — max absolute delta exactly 0.0 on both outputs. Build time fell from 15.5 s to 4.4 s as a bonus.

The bytes that differ are 0.005% of the file — 4,658 to 6,476 bytes out of 118.9 MB, all of it between offsets 5,402 and 1,922,153. The roughly 117 MB weight region is byte-identical across all three.

What that licenses. On this host, this TensorRT and this graph, a plan file’s hash is not a usable identity for the model: two builds from an identical recipe never produced identical bytes, cache or no cache. A timing cache did something stronger and narrower than byte-reproducibility — it pinned tactic selection, which made the engines numerically identical while leaving the files different.

The consequence for the registry is direct. It cannot content-address a plan. It has to address the recipe plus the builder environment plus the timing cache, and verify equivalence behaviourally rather than by hash. That is a statement about TensorRT 11.2.1.2 on sm_89 with this graph, not a general claim about TensorRT.

One honest qualifier, because it cuts against the drama: across those 20 fixtures, the three cacheless builds produced zero threshold flips at 0.4. The bytes and the raw logits moved; the served decisions, on that small sample, did not.

TF32 is a default that quietly changes the numbers

A “fp32” engine on this GPU is not doing fp32 matmuls unless somebody says so. TF32 carries a 10-bit mantissa and is on by default. Measured over 200 real photographs from the model’s held-out validation split — 100 litter-bearing, 100 hard negatives — against the ONNX under ONNX Runtime 1.29.0:

Clearing the flag drops the mean labels delta by about 195× — 0.182 to 0.00093 — and takes every threshold flip to zero. The largest confidence shift on any single query goes from 0.7696 to 0.0317.

It is not free. On the RTX 4070, batch 1, device execution plus synchronize only:

Clearing TF32 costs 29% of engine latency, far outside the engine’s own p50 noise floor of 0.7% over three runs. So the default is a precision decision hiding inside a builder flag, and it is worth 29% of the frame time. Nobody in this project had made that decision; the default had made it for us.

For context on the same box, and not the same measurement — the engine number excludes host-to-device copies while ONNX Runtime’s is session.run wall time — ONNX Runtime’s CUDA EP (ORT 1.20.2, CUDA 12.4) ran 7.93–8.74 ms p50 with 10.2% run-to-run noise, and its CPU EP ran 139–154 ms. Against ORT’s best CUDA p50, TensorRT is 1.55× with TF32 and 1.20× without; the second of those is close enough to ORT’s own noise to be read as a range rather than a number.

Two gates, opposite verdicts, both correct

Polygraphy is the standard tensor-level parity tool, and its failure rule is permissive: an element mismatches only when the absolute difference exceeds atol and the relative difference exceeds rtol. Fed 25 of the same real fixtures as identical saved tensors — not Polygraphy’s random default inputs — it says:

The TF32 engine fails every iteration at every tolerance tried, including 1e-2. By that gate the conversion is a total failure and should never ship.

Now the same engine, same 200 fixtures, decoded exactly the way the backend decodes — argmax over the 43 named columns, sigmoid, threshold at the sidecar’s recommended 0.4, top-k 100, no NMS. Because engine and ONNX are the same graph, query slot q means the same thing in both, so the comparison is query-aligned and exact:

Two changed decisions out of 183, on one image out of 200.

Both numbers are right. They answer different questions. A tensor-level tolerance gate on this graph would block a conversion that barely moves anything the product serves — and the same gate, at 1e-3, also fails the TF32-off engine on 3 of 25 images while it produces zero changed decisions. The tensor gate and the decision gate disagree, and here they disagree in the direction of blocking a shippable artifact rather than passing a broken one. That is why the operating-point comparison is the one worth gating on.

Note also what the totals conceal, for the third time in this project. 183 against 183 is a perfect match on count, with two queries disagreeing — one in each direction. The int8 conversion did this at 202 against 202 concealing 112 flips; the fp16 conversion did it at 202 against 203 concealing 19. A count-based check waves all three through. Different hardware, different runtime, same trap.

Why the engines were not built from the registered ONNX

They could not be. The graph they were built from is not the graph in the registry, and finding out why produced the most uncomfortable result here.

The registered fp32 ONNX is a59fc417, 120,039,167 bytes, exported 2 August. Re-exporting for the GPU work produced c055e87c, 120,111,594 bytes, on 7 September. Same checkpoint. Same exporter. Same opset. Same producer, pytorch 2.6.0. Same Windows host, same virtualenv — confirmed by counting the absolute local paths that PyTorch’s tracer bakes into every node’s doc_string: this export carries 60,832 frames rooted at C:\tmp\venv-rfdetr and 4,677 at its checkout, and zero POSIX paths.

They are not the same file.

This is not platform variance, which everyone expects and nobody worries about. It is one export failing to reproduce on one machine across five weeks.

The three differences run in three unrelated directions, and the ranking is the finding:

1. Debug metadata churn — about 214 KB, and it dominates the byte delta. Node doc_strings are 4.80% of this artifact (5.77 MB). They are tracing debris: absolute source paths and line numbers. 98.4% of the doc_string difference between the two exports is accounted for by the checkout having moved — the recorded root is 76 characters here against 31 before, and 45 bytes × 4,677 frames is 210,465 bytes. Moving a directory changed the artifact’s hash.
2. Shape-inference metadata — about 138 KB. The older graph carries 1,575 value_info entries; this one carries zero. One export ran shape inference and the other did not. Reconciling the whole gap leaves 23 unaccounted bytes. Nothing executes differently, but any consumer that reads shapes off the graph sees a different graph.
3. Three Cast nodes — about 250 bytes, and semantic. Weights are bit-identical and every op type except Cast matches exactly. Those three nodes decide whether the repo’s fp16 conversion produces a model that will load at all: the same fp16 pipeline that worked on the 39-Cast export produces, on the 36-Cast export of the same checkpoint from the same machine, a graph that neither ONNX Runtime nor TensorRT will load — a third fp16 defect distinct from the two the CPU work already documented, and not covered by the existing repair. No repair was attempted; inventing one would be a new claim needing its own validation.

Byte size and behavioural risk are anti-correlated here. About 350 KB of changed path strings and shape annotations changed nothing that runs. Roughly 250 bytes of Cast nodes made an artifact unloadable in two independent runtimes. Any registry policy that ranks artifact diffs by size ranks these exactly backwards.

The cheapest fix available, worth taking on its own: strip doc_strings before hashing, or export without them. They are 4.80% of the file, they embed absolute local paths into something published to S3, and they destroy hash stability for no semantic gain. They are not a fix for the Cast nodes.

The Pi 5 finally has litter-model numbers

Since May, /benchmarks has carried a Pi 5 row for a stock COCO YOLOv8n and an explicit admission that no litter-model number existed for that board. It exists now.

Everything in this section is Raspberry Pi 5 Model B Rev 1.1, aarch64, Cortex-A76 ×4, 987 MiB RAM, ONNX Runtime 1.29.0 CPU execution provider. An embedded-class board with no GPU and no DLA. Nothing here transfers to an Orin or to any automotive runtime.

The prior assumption was that a 120 MB fp32 graph would not fit in 1 GB, and that int8 was what made the model runnable at all. That is not what happened.

fp32 loaded and ran at 380 MiB peak RSS with no swap touched. The hypothesised roughly 500 MB fp32 working set never materialised: at 512×512 batch 1, RF-DETR-Small’s activation footprint is small enough that the graph size dominates, and 120 MB of weights plus runtime overhead peaks under 400 MiB. int8’s memory advantage is real but modest — 76 MiB, about 20% — so on this board int8 is a latency optimisation, not a fit-enabler. It is 1.53× faster.

Two things that number is not:

• It is not a frame rate for the robot. 745 ms per frame on the Pi is a measurement, not a plan. The edge path is still gated on a distilled or quantization-aware-trained student, which is what six post-training recipes failed to substitute for last month.
• It is not a thermal result. This was 100 inferences in short bursts, not the 1-hour soak the YOLOv8n row is based on, so no peak temperature is claimed for this model. A single reading after the runs was 41.1 °C at 1.6 GHz with no live throttling — and vcgencmd get_throttled returned 0x50000, meaning under-voltage and throttling had both occurred at some point since boot. That is the power-delivery condition the repo’s own Pi bring-up sequence exists to fix, and it is worth stating rather than reporting the 41.1 °C alone.

Against the published YOLOv8n row, this is a substantially heavier edge load — 2.2× (int8) to 2.7× (fp32) the memory, and much slower per frame, being an RF-DETR with a DINOv2 backbone rather than a YOLO. The comparison is memory-and-architecture only. The two models are different models, and the fp32 percentiles above come from 5 images rather than 100 (peak RSS is set by the session plus one inference, so that figure is representative; min 1100 ms, max 1193 ms).

The same bytes decide differently on ARM and x86

This is the result the Pi trip was actually for.

The same int8 ONNX ran on the Pi and on an x86 reference host — same file, same ONNX Runtime version, same execution provider, same 100 fixtures. To rule out preprocessing, the preprocessed input tensors were saved on both machines and compared:

max |Δinput| = 0.000e+00  on every sampled image

Byte-identical inputs. So everything below is the runtime and the architecture.

Raw outputs diverge first: max absolute delta 1.698 on dets and 9.063 on labels.

85 against 84 looks like agreement. It is 18 query-level decisions across 9 images, hidden behind a count difference of one. A cross-platform check comparing detection counts would report “identical” and ship. So would mAP against a shared ground truth, which would blur 18 decisions into near-noise.

The mechanism is not mysterious: dynamic int8 computes with int8 matmuls and fp32 accumulation, and the kernels that do that differ between AVX-512/VNNI on x86 and NEON dot-product on Cortex-A76, as do their rounding and accumulation order. Different arithmetic on the same weights, and a query sitting near p=0.4 lands on either side of it.

This is decision instability across architectures. It is not an accuracy difference, and nothing here says which platform is more nearly correct — that would need scoring both against ground truth, which this measurement does not have. The blind human labels from last month’s update cannot be borrowed for it either: those cover a different 200-image Reddit fixture set, and these 100 images are drawn from the held-out validation split. Different corpora, so the flips here cannot be scored against that ground truth at all.

What it does establish is narrower and, for anything that ships one artifact to more than one CPU architecture, more useful: “we validated the int8 model” is not a claim about the model. It is a claim about the model on one architecture.

int8 changes sign with the host

Comparing dynamic int8 against its own fp32 baseline within each machine — same conversion, same mechanism, so the only variable is the host:

The two ARM hosts agree in direction. The x86 host inverts it. “Is int8 faster?” has no answer that does not name a host.

The confound is large and has to be stated. These are three different machines, three operating systems, and no guarantee that the ONNX Runtime build or the CPU feature set is matched across them. Two ARM points agreeing is suggestive of a kernel-availability effect. It is not proof of an architecture law, and this measurement cannot support one.

It is also not the easy explanation. The obvious story — “x86 lacks int8 acceleration” — is wrong on the facts: this x86 box is an AMD Ryzen 7 7700 (Zen 4) whose CPU flags include avx512vnni, so the int8 dot-product instructions are present. int8 is slower there anyway. What that points at is dynamic quantization’s per-inference activation-scaling overhead outweighing the int8 arithmetic saving on this graph — the runtime cost of recomputing activation ranges on every call — rather than missing hardware. The clean version of this experiment, one artifact and one ONNX Runtime build across otherwise matched ARM and x86 hosts, has not been run.

Static PTQ: fastest, and least faithful

Which raises the obvious question. If dynamic quantization pays an activation-scaling cost on every inference, bake the scales in ahead of time. That is static post-training quantization, and it is exactly the path the exporter refuses without a calibration set. So it got one: 48 real images drawn from the other half of the validation split, disjoint from the 100 evaluation images, balanced 12 per cell across density (litter-bearing vs hard negative) and lighting (below/above each class’s median measured luminance). Chosen for coverage, not count.

All figures below are x86 AMD64, ONNX Runtime 1.29.0 CPU execution provider, QDQ format, per-channel int8 weights and per-tensor uint8 activations.

Static is faster than both — about 1.75× faster than the dynamic int8 the repo already builds. Then the operating point, against fp32’s 90 detections over the same 100 images:

Static PTQ loses about a third of the detections and disagrees with fp32 on about a third of the images — against dynamic’s 14%. Max absolute labels delta versus fp32 tells the same story: 10.33 dynamic, 12.50 static MinMax, 13.30 static Percentile. It is not a uniform confidence shift; 25 queries survive in both static-MinMax and fp32 and are assigned a different class.

The reason is structural rather than a bad setting. DETR activations — LayerNorm outputs, attention scores — have wide, input-dependent dynamic range. A per-tensor static activation scale fixed from a handful of calibration images clips that range identically for every input. Dynamic quantization recomputes it per inference, and that adaptivity is exactly what static gives up. It is also, exactly, the overhead that made dynamic slower on x86. The thing that costs the latency is the thing that buys the fidelity.

Two operational findings came out of the calibration itself, and both are worth more than the headline:

• The Percentile calibrator ran out of memory. It augments the graph to emit every intermediate tensor and keeps a full 2,048-bin histogram for each of 958 tensors. At 48 calibration images it died with bad allocation. It completed only after dropping to 8 images with chunked collection. That is a real operational property of the Percentile and Entropy calibrators on a graph this size, not a footnote — and it means the two calibrators below were not built on equal footing.
• The calibrators disagree with each other. At matched calibration size — 8 images each — MinMax and Percentile produce 65 and 61 detections, with 34 threshold flips, 13 class flips and disagreement on 25 of 100 images. Same calibration data, different scales, different decisions. MinMax at 8 images also recovered slightly over MinMax at 48 (65 against 58), which means the static result is sensitive to calibration-set composition as well as calibrator. That is a caution, not a knob: chasing a calibration set that happens to restore detections would be fitting the metric.

The honest recommendation is the unglamorous one. If int8 is wanted for fidelity, dynamic is the better post-training quantization here despite being slower on x86. If static’s speed is wanted, the accuracy has to be bought back with quantization-aware training — not with a bigger calibration set and not with a looser threshold. “Static costs too much accuracy on this model” is the result.

torch.compile does not compile this model

The other standard lever, tried on the same checkpoint on the RTX 4070 box under PyTorch 2.6.0+cu124. It produced no runnable compiled model, for two independent reasons. They are worth keeping separate, because only one of them is a property of this model.

Blocker one is the platform. torch.compile(model) and mode="reduce-overhead" both raise BackendCompilerFailed: RuntimeError: Cannot find a working triton installation. TorchInductor generates Triton kernels and Triton has no official Windows build, so on this box the default backend — the one every “just wrap it in torch.compile” recommendation assumes — is simply unavailable, and max-autotune was not attempted for the same reason. A Linux box with Triton gets past this line.

Blocker two is the model, and it is the one that matters. Compile time and graph-break counts could not be produced because the model never forms a graph. Every capture path tried — CompileCounter, backend="aot_eager", backend="cudagraphs", and fullgraph=True — fails at the same two constructs, none of which involve Triton or Inductor:

• A NestedTensor built from Python shape ints. LWDETR.forward wraps its input with nested_tensor_from_tensor_list, which allocates a padded tensor and builds a boolean mask by Python-integer slice assignment. Dynamo cannot trace a slice driven by .shape ints used as Python values; it sees a torch.Size where it expects a tensor and raises.
• A torch.stack on a tuple, which is where fullgraph=True reports the first graph break.

fullgraph=True failing is the direct statement that RF-DETR-Small has graph breaks, and the capture errors say those breaks are not benign fall-back points but constructs that abort tracing outright. Even with Triton present, this model would shatter at the NestedTensor boundary on every forward — precisely the case where torch.compile quietly stops helping.

The contrast is the useful part. The same forward that Dynamo cannot capture exports to ONNX cleanly — it is the 1,570-node graph every TensorRT result above is built on. torch.onnx.export runs the Python once and specialises on the concrete shapes it sees, so the host-side mask construction executes during the trace and bakes its result into constants. Export tolerates host-side Python because it is leaving Python; the artifact never runs that code again. torch.compile has to preserve a runnable Python callable with dynamism, so it must trace symbolically the same constructs export was happy to constant-fold.

They are not competing options. torch.compile keeps a model fast inside a Python loop on a host that has the framework; export ships an artifact to a device that does not. Everything this project deploys — the Lambda ONNX Runtime path, the engines above — is the export case. For this model, export is the tool that fits, and it already works.

For completeness, the eager baseline on the RTX 4070 was 11.8–13.9 ms p50 with 12–18% run-to-run noise on the identical call. Any speedup claim would have had to clear that. The TensorRT engine’s own p50 noise, for comparison, is 0.7%.

What the registry did with all of this

The registry now holds five records, and getting the engines into it broke the same rule twice, one level apart.

The first attempt collided. Registry.register refused any second artifact under an existing (model, version) pair, whatever target it named — even though identity() had always included the target and already rated the ONNX and its engine as two distinct releases. The module’s identity model said “two releases”; its uniqueness rule said “collision”. That was found by trying it, not by reading it: the registry refused a release it was built to record.

Keying uniqueness on (model, version, target-digest) fixed the cross-runtime case and missed the same contradiction one level down. The TF32-on and TF32-off engines share a target digest exactly — same GPU, same driver, same CUDA, same runtime, same execution provider — because TF32 is a builder-config decision recorded in the conversion, not a property of the destination. Two builds with measurably different numerics still collided.

The fix that holds keys uniqueness on the release key: the full identity tuple with the artifact hash removed. The recipe and the destination, without the bytes. Both engines now coexist with the ONNX and with each other.

Three narrower repairs came with it, and two of them exist only because a plan is an opaque blob:

• A TensorRT plan has no ONNX initializers, so the registry’s “precision is read off the graph, not taken from a label” guarantee simply cannot apply. Rather than storing a bare null — which also means “an ONNX we forgot to inspect” — the engine records carry PrecisionEvidence.opaque("tensorrt plan: weights are not inspectable"). Unobtainable evidence and absent evidence are different claims and now serialise differently.
• Target gained structured computeCapability, cudaVersion and driverVersion fields. Those three change the numbers a plan produces, so they belong in the identity digest as comparable fields rather than string-packed into a host label.
• A parity report from another runtime is now refused rather than relabelled. Transcribing a TensorRT report used to stamp runtime: "onnxruntime" onto a measurement made under something else.

And the six determinism builds from earlier in this post still collide, now by design. Same recipe, same target, different bytes — one release key, so the second registration is refused. That is the registry surfacing the non-reproducibility rather than storing six rows each claiming to be the same build. Recording all six would need a build identifier the recipe does not carry, which is a deliberate design change and not a workaround.

One open item is named and deliberately not fixed. verify reports MISSING artifact for both engine records, because a 119 MB plan is gitignored and not kept in the tree — which is the intended state, given that the determinism result above is the argument that storing the plan would not have established identity anyway. The gap is real: verify cannot tell “this artifact is deliberately not retained” from “this artifact should be here and is gone.” For a 119 MB plan the first is the normal case. The fix is a retention field on the record and a distinct status for it, which is a schema change that would invalidate two records written against the current schema for no gain today. So it is written down with its fix named rather than rushed.

What none of this licenses

• These are not embedded numbers. Every GPU figure is an RTX 4070 desktop part under WDDM; every Pi figure is a Cortex-A76 CPU. Neither describes an Orin, an automotive runtime, or the arm64 Lambda that serves production today. For any other accelerator this is a method, not a result.
• No thermal claim for the litter model. No soak was run on the Pi. The 73.6 °C on /benchmarks belongs to a stock COCO YOLOv8n, a different and lighter model.
• The Pi flips are not an accuracy difference. Nothing here scores either architecture against ground truth, and the fixture set differs from the one that has blind human labels.
• The int8 sign flip is not an architecture law. Three unmatched machines and three operating systems. One case of each direction, and a named confound.
• Nothing was promoted. fp32 still serves. The parity gate is wired but has never run against real artifacts in continuous integration — it needs credentials and the fixture corpus first — so no conversion here has been accepted or rejected by anything except a person reading a table.
• Held-out validation images are closer to production than the previous 200 Reddit photographs, and still are not the deployed input distribution. The GPU work used 200 of them; the Pi and static-PTQ work used 100. Both are different sets from last month’s, so flip counts here are not comparable to those in absolute terms.

What has changed is the shape of the problem. Last month’s post argued that a conversion needs a decision-level gate rather than a count. This month says the artifact needs an identity, and that neither half of the obvious answer works alone: the recipe does not determine the bytes, and the bytes do not determine the decisions. A release record has to carry the recipe, the environment, the timing cache, the artifact hash and a behavioural measurement made on the target — or it is describing something other than what will run.