How Much Data Are We Losing, and Does It Matter? by Sathvik

How Much Data Are We Losing, and Does It Matter? by Sathvik - October 2026 Scholarship Essay

The research question that changed how I think was deceptively simple: how much data are we losing, and does it matter?

I was working at Berkeley RadWatch, building data pipelines for IoT radiation sensors. The sensors were deployed in the field, collecting readings continuously. My job was to make sure the data got from the sensor to the database reliably. It sounded straightforward. It was not.

When I started measuring, I found we were losing 8.2 percent of readings. Almost one in twelve data points was vanishing somewhere between collection and storage. Network drops, buffer overflows, timing issues, silent failures. The system was designed by smart people, but nobody had quantified the loss because nobody had looked closely enough.

That 8.2 percent bothered me. In radiation monitoring, missing data is not just an inconvenience. It is a gap in your understanding of what is happening. If you are trying to detect anomalies, and you are missing nearly a tenth of your observations, your conclusions are built on an incomplete picture. I became obsessed with finding every point of failure.

What followed was months of systematic debugging. I instrumented every stage of the pipeline. I found that the biggest losses came from a buffer that overflowed during network outages, silently discarding readings instead of queuing them. I redesigned the buffering logic, added persistent local storage as a fallback, and built monitoring that alerted us to drops in real time. When I finished, data loss was 0.05 percent. We went from losing one in twelve readings to losing one in two thousand.

This project deepened my understanding in two ways. Technically, I learned that reliability is not a feature you add at the end. It is a property that has to be designed into every layer. A pipeline is only as strong as its weakest buffer. Philosophically, I learned that the most important problems are often the ones nobody is looking at. Everyone assumed the data was fine. It took someone asking a basic question to discover it was not.

That experience shapes everything I build now. My company exists because I asked a similar question about AI training data: who has access, and who does not? The answer revealed an asymmetry I could not unsee. Just like the sensor pipeline, the problem was hiding in plain sight, waiting for someone to measure it.

Curiosity is not just about asking interesting questions. It is about asking basic ones and refusing to accept 'probably fine' as an answer.