ET-SoC-1 · per-sensor temperatures ·

Heat from a block of shires spreads across the die and shows on the new per-sensor readings. One shire stays under the whole-degree step.

← AI Foundry lab dashboard

With Roman's new firmware on aifoundry3's card, my agents heated chosen parts of the die in on/off cycles and read all 35 temperature sensors twenty times a second. Blocks of four or more shires light up their corner of the die and the heat visibly spreads; one core, or one shire of 32 cores, never moves its own sensor by a whole degree. The movies below average every repeat of a cycle, which is what pulls sub-degree changes out of whole-degree readings. The sensor map was measured first, and the movies use it.

35temperature sensors read, every 50 ms
83heating cycles, 8 s on and 12 s off
0.58fit of separate runs with the measured sensor map, against 0.41 with the latency map
1 °Cstep of each reading; a shire's own rise is about 0.5 °C

Watch it: one heating cycle, averaged over every repeat

0 s4 s8 s12 s16 s-0.5+0+0.5+1+1.5+2+2.5heatedrest of die

Outlined cells are the heated shires. The red band is the heater; time 0 is the start of the heater's process. Cells follow the lab's die map from on-chip latency. With "measured sensor map" on, each sensor sits where the sensor-map runs below placed it; off, sensor n sits at shire n.

Blocks of shires show; one core or one shire does not

Every block run raised its own region above the rest of the die, by up to +1.26 °C (the east half (14 shires)). The single core and the three single shires never moved their own sensor: in every cycle it stayed on the same whole degree. A shire's expected rise is about half a degree, and whole degrees can't show that without a dither the idle card doesn't provide. For single shires it is not granular enough: that needs the raw 12-bit codes, 0.061 °C a step, which Roman is adding to the firmware.

RunCyclesHeated, peakWhole die, peakHeated minus restSensors that movedSeen?
One core (1 of 32) in shire S14, the centre3+0.00+0.03+0.036 / 34No: one core draws about 20 mW
One shire, S0: the NW corner4+0.00+0.14-0.0012 / 34No: under one whole degree
One shire, S31: the SE corner4+0.00+0.10+0.017 / 34No: under one whole degree
One shire, S14: the centre4+0.00+0.17-0.0012 / 34No: under one whole degree
A 2×2 block in the centre (S13, S14, S21, S22)4+1.20+0.74+0.5234 / 34Yes
The NW quadrant (9 shires)5+2.00+1.55+0.6434 / 34Yes
The NE quadrant (6 shires)5+1.49+0.86+0.8032 / 34Yes
The SW quadrant (9 shires)5+1.92+1.40+0.8834 / 34Yes
The SE quadrant (8 shires)5+1.88+1.11+1.1434 / 34Yes
The west half (18 shires)4+3.29+3.07+0.6734 / 34Yes
The east half (14 shires)4+2.68+2.00+1.2634 / 34Yes

Where each sensor sits: the latency map is broadly right, and the measured map is better

To place the sensors, my agents heated every column of the die in turn, then every row, then two corners, and compared each sensor's responses with a heat-spread model (0.52 °C at the source, 0.24 °C one tile away, 0.10 °C two away). The assumed layout, sensor n at shire n's place on the latency map, fits these 14 patterns far better than chance (total correlation 13.7, against 0.0 for random relabelings), so heat sources and sensors broadly line up. It is not exact: only 15 of 35 sensors have their assumed cell among their three best fits. Sensors 13 and 14 are the clearest: they stay cold when the west half, with shires 13 and 14, heats, and warm with the east half.

The best placement fitted on these patterns alone was then checked on the main session's eight separate runs: it explains them better than the assumed layout (correlation 0.58 against 0.41 on average; 0.78 against 0.42 for the west half). The movies use it. Any single sensor's place is approximate, since whole degrees and three cycles a stripe leave room; raw codes would settle it.

column 1 heated
column 2 heated
column 3 heated
column 4 heated
column 5 heated
column 6 heated
row 0 heated
row 1 heated
row 2 heated
row 3 heated
row 4 heated
row 5 heated

Each small map is the die at the end of heating, minus the whole-die rise, drawn on the latency map: red where a sensor rose more than average. Outlined: the heated shires.

The measured placement

✓: the sensor sits where the latency map put it. Dashed: one cell away. Amber: further; the cell says where the latency map had it.

Every run at its sharpest moment

One core (1 of 32) in shire S14, the centre
3 cycles · at +8.75 s · heated +0.00 °C, rest of die -0.03 °C
One shire, S0: the NW corner
4 cycles · at +0.00 s · heated +0.00 °C, rest of die +0.00 °C
One shire, S31: the SE corner
4 cycles · at +0.25 s · heated +0.00 °C, rest of die -0.01 °C
One shire, S14: the centre
4 cycles · at +0.00 s · heated +0.00 °C, rest of die +0.00 °C
A 2×2 block in the centre (S13, S14, S21, S22)
4 cycles · at +7.25 s · heated +1.19 °C, rest of die +0.67 °C
The NW quadrant (9 shires)
5 cycles · at +6.25 s · heated +1.92 °C, rest of die +1.28 °C
The NE quadrant (6 shires)
5 cycles · at +7.00 s · heated +1.45 °C, rest of die +0.65 °C
The SW quadrant (9 shires)
5 cycles · at +5.25 s · heated +1.79 °C, rest of die +0.91 °C
The SE quadrant (8 shires)
5 cycles · at +3.25 s · heated +1.63 °C, rest of die +0.49 °C
The west half (18 shires)
4 cycles · at +6.25 s · heated +3.04 °C, rest of die +2.37 °C
The east half (14 shires)
4 cycles · at +7.75 s · heated +2.68 °C, rest of die +1.42 °C

No hidden way to finer readings: it takes a firmware change

The sensors resolve 0.061 °C in hardware, but every route from the host gets whole degrees or nothing. Checked on Roman's build on aifoundry3 and in the firmware and driver source:

RouteWhat it givesWhy
Command 74, temperature group (Roman's build)Whole degreesThe firmware converts each 12-bit code with integer arithmetic and divides by 1000 before it answers (pvt_controller.c:567-585).
Command 74, other groupsErrors (−9007, no data)Roman's patch defines groups 0-6; none returns raw temperature codes.
Process detectors (ring oscillators)A constant (47620), not countsTheir counts would track temperature, but the firmware builds them with measurement disabled (PVT_PD_OSC_SELECT).
The other temperature commands (21, 22, 27, 28, 38)Whole degreesMeans, peaks, thresholds and throttle state, all in whole °C.
The SP's debug traceNothing per sensorThe per-shire temperature print exists, but nothing calls it (pvt_print_all has no caller).
Debug-interface memory reads (MDI)Not reachableAllowed only in host-managed DRAM and minion stack regions; the sensor registers are the SP's own.
The PCIe windows the driver mapsNot reachableQueues, scratch, traces and the SP and MM statistics; the sensor controllers (R_SP_PVT*) are SP-only.
Python and other host softwareNothingNo Python package for the cards on the box; libDM and /opt/et hold no other path; et-pvt-top uses command 74.

What would fix it: return the raw 12-bit code from the same register read that command 74 already makes, with no rounding. Roman has agreed to add it. Until then, averaging repeated cycles is the only way under a degree, and it only works for patterns large enough to cross a degree boundary.

How the runs were made
  • Card: aifoundry3 card 0, firmware release 1.4.0 (BL1/BL2 0.22.0, minions 0.25.1), Roman's build with command 74. Idle at about 57 °C; the governor is pinned at 600 MHz on this card.
  • Readings: DM_CMD_GET_ASIC_PVT_SAMPLES group 0 (34 minion-shire sensors) every 50 ms and group 4 (the I/O shire) once a second, sent from Python straight to the management node (42 µs on the card per call). 34138 samples in all, no read errors.
  • Heat: the lab's fp32 FMA heater (sparsity_host --test fma) on the chosen shires, 32 cores each (one core for the single-core run), 8 s on, 12 s off, 3 to 6 cycles per run, under the card's lock. The hottest reading anywhere was 66 °C.
  • Movies: every cycle aligned on the heater start, each sensor minus its own mean over the 2 s before, averaged over the cycles in 0.25 s frames.
  • Layout: the die frame from the lab's latency map (tools/claims-v3/hp/placements.json), with the sensors placed by the measured map above.
  • Code and raw data: heatmovie.py, analyze.py, sensormap2.py (with sensormap.py), pvt_ctl.py, data.json.