Row vs Column: Sparse Robot Logs
A robot log stored as one wide table is mostly empty cells. Pick a sensor, drag a time window, and see what each layout reads to answer one query.
Robots record every sensor they have, dozens of streams at once, so engineers can replay a run later and find out what went wrong. How that recording is laid out on disk decides how fast you can ask it questions.
What this shows
A robot log is a handful of sensors, each reporting at its own rate. The top of the drawing is four seconds of one: an IMU at 100 Hz, joint angles at 50, a camera at 15, a lidar at 10, and a battery monitor once a second. No two readings land on the same instant.
Below that, the same readings are stored two ways. The first is one wide table, the shape you get from a spreadsheet or a naive export: a row for every timestamp and a column for every sensor. Every reading has its own timestamp, so every row has exactly one filled cell. The second gives each sensor its own pair of columns, time and value, cut into chunks of 32 readings. Each chunk remembers its first and last timestamp, and that short list of ranges is the index.
Both layouts answer the same question: this sensor’s readings in this window. The outlined regions show what each has to read to answer it. The table has no way to know which rows hold your sensor, so it reads every row in the window across every column. The per-sensor layout checks the index, opens only the chunks whose range overlaps the window, and never touches the other sensors.
Things to try
Query the battery. The table reads hundreds of cells to return one reading, or none at all.
Query the IMU and drag the window slowly. The per-sensor layout reads a little past the window’s edges, because it opens whole chunks. That over-read is the price of keeping the index small.
Add the force sensor. One fast sensor multiplies the table’s rows for everyone, so every query gets more expensive, the battery’s included. On the per-sensor side it is one more strip that no other query ever reads.
What it leaves out
This counts cells, not bytes. Real formats compress, store an empty cell as a single bit, and keep large payloads like images out of line, so a wide table is less wasteful on disk than it looks here. None of that changes the read pattern. A query against the table still walks every row in its window, and the row count still grows with every sensor you add. I have been building a columnar store in Rust for exactly this kind of data, and that read pattern is the reason.