How to log 27 treadmill sessions when nothing involved has an API

The music study needed heart rate data from nine people using four different devices. This is the part of the work that does not appear in the write-up, and it took longer than the study.

What the vendors provide

Two of the four offer a CSV export, buried two levels into a settings page and rate-limited to one request per hour for no reason anyone could explain to me. One offers a PDF. One offers a screenshot feature.

The PDF was the interesting problem. It contains a chart image and no numbers, so the data exists only as pixels. I spent an evening writing something to read values off the rendered line before noticing that the same account's web dashboard fetches the underlying series as JSON to draw that chart, and that the endpoint is not authenticated beyond a session cookie.

The shape of the mess

Four devices, four sampling rates, three timestamp formats, and two different opinions about whether a dropped signal should be a null, a zero, or the previous value repeated. The last one matters enormously and is documented by none of them.

device_a: 1Hz,  ISO-8601 UTC,     null on dropout
device_b: 0.2Hz, epoch millis,     0 on dropout
device_c: 1Hz,  local time, no TZ, last value held
device_d: ?,    "14:32:07",        unclear

Device C is the one that quietly lies. A held value looks exactly like a person with a very steady heart rate, and it is invisible unless you go looking for runs of identical samples. There were 340 such runs.

What I would build next time

A single ingest step that normalises to one sample rate and one dropout convention before anything else touches the data, with the raw file kept untouched alongside. I did this halfway through, having already analysed a week of data that turned out to be four percent held values.

The scraper and the normaliser are both in the repo. Neither is good.