A Summer With CycleQA

Problem → Idea → Design → Improvement → Product

Jeløy, this summer — where all of this actually happened.

Problem

It started with a question I couldn't quite let go of: if two things both claim to have seen the same event, how do you actually know they're telling the truth?

I'd been standing on my balcony on Jeløy some evenings, watching ships cross the fjord, checking what I saw against a live AIS map on my phone — the way you check a gut feeling against a second opinion. It's a small, human thing to do. But it's also, almost exactly, what has to happen on a production line at 2am, when nobody's there to watch: a camera sees a part, a tester measures it, and something, somewhere, has to decide whether those two stories agree.

Most quality systems don't really ask that question. They trust one sensor at a time and hope. I wanted to know what it would take to actually check.

Idea

The idea, once I found it, was almost embarrassingly simple: neither sensor should be allowed to grade its own homework.

A camera should say what it saw. A tracker should say what it saw. Neither one should be trusted to also decide whether it was right. That decision belongs to a third, independent step — one that never touches either sensor directly, and whose only job is to compare their evidence honestly, including when the answer is "these don't agree" or even "I'm not sure."

It sounds obvious written down. It wasn't obvious to build, and it took me a while to actually trust the shape of it enough to try.

Design

So I tried it on the smallest, most honest version of the problem I could find: my own balcony, a laptop webcam, and a free maritime traffic feed.

Two sensors, reporting independently. A zoom-webcam watching the fjord, and a public AIS feed watching the same water. Neither one aware the other existed. A separate piece of code — untouched by either — was the only place a verdict got made: did the same vessel show up in both, close enough in time, to count as confirmed.

Four times, one evening, it did. Color Fantasy, Polaris Liberty, Hagland Progress, Nordic Crown — a passenger ship, a car carrier, a bulk carrier, and another passenger ship, four real vessels correctly matched while I stood there enjoying the evening instead of doing the checking myself.

It also caught a seagull. At one point, a bird drifted past at exactly the right moment to get logged alongside a real ship's crossing — a false match, honestly recorded, evidence and all, instead of quietly slipping through. I liked that it happened. It's the difference between a system that trusts a match and moves on, and one that shows you its work.

The vision side kept improving as the summer went on, too. It started on a public YOLO model that could manage "boat" or "bird" and not much more. By the time I wrote this, it was running on my own trained model instead, telling actual vessel classes apart — cargo, cruise ferry, sailboat — and cross-checking two independent identification methods against each other: object detection, and OCR reading a vessel's name straight off its hull. I didn't sink real weeks into the machine-learning side of it; training a model properly is its own rabbit hole, and for now I wanted to understand its principles and its honest limitations, not chase a leaderboard score.

There's an architecture choice sitting underneath all of this that's easy to miss but matters a lot: every sensor node does its own processing locally, at the edge, on whatever's actually running it — a laptop, eventually something smaller. Only the result ever travels onward — a class name, a confidence score, a decoded vessel name, a position fix. The video frames themselves never leave the machine that captured them, and CycleQA never stores a single one. What lands in the system is structured evidence a person or another piece of software can actually query and compare, not a video archive nobody has time to rewatch. It's not just tidier. It's the same reason the system doesn't care whether it's watching one sensor or a hundred — it was never built to be a video-storage problem in the first place.

I wrote that evening up as its own story back in June — "Four Ships, One Seagull," if you want the honest, occasionally-wrong-on-purpose version. This is the sequel: what happened when I stopped asking whether the idea worked once, and started asking whether it actually generalizes.

It's also, I realized once I said it out loud, exactly the same pattern a PCB production line already runs, without necessarily calling it that — Automated Optical Inspection cross-checked against an independent electrical test, two different senses of the same board, neither one trusted alone. I hadn't set out to reinvent AOI. I'd just built the smallest possible version of the same idea, on a fjord instead of a conveyor belt.

Improvement

Because a nice result on one evening with two sensors proves an idea can work. It doesn't prove it's a product.

So this summer, I set out to break it — on purpose, with two problems that don't look anything like ships and cameras at all.

The first: a signal moving through five stages of a real hardware pipeline — a sensor, an FPGA, a protocol layer, an application, a display — the way an actual measurement travels from silicon to screen. Not "do two things agree," but "does one thing survive the whole trip, in order, without silently getting lost at stage three while stages four and five never even notice."

The second: a small fleet of test devices, logging their own health continuously, hour after hour, the way real hardware does during a long validation run. Not "did two things agree," but a completely different question: does any one of them quietly drift out of spec over time, even when no single reading ever crosses a line? And separately — does it ever just go silent, and can the system tell the difference between "this sensor crashed" and "this sensor is fine and genuinely has nothing to report"? Those sound like the same failure. They're not, and a system that can't tell them apart is lying to you either way.

One honest referee, three genuinely different kinds of evidence to referee.

Neither of those is a variation on the ships-and-camera idea. They're structurally different problems. And the same engine — the one built for a webcam and a public ship feed — handled both without needing to be rebuilt underneath.

None of that matters much, though, without somewhere to actually run it — every day, unattended, at a real customer's site, not just once on my own balcony. So a good part of the summer also went into designing the Scheduler: what I've come to think of as der Hebel, the lever, for putting CQA-AUTO to work daily in a real production environment, not just proving an idea once and calling it done.

There was a real bug along the way, and I'd rather tell you about it than not: an early version of the grouping logic made an assumption about how test data gets organized that turned out to be wrong once I checked it against real production data instead of my own mental model. Caught before it shipped, fixed, verified against the real thing — but it's the kind of mistake that's easy to miss if you only ever test against your own assumptions. Worth saying out loud, the same way the seagull was worth keeping in the first story.

Product

Which brings me to what this was actually for.

CQA-AUTO is the part of CycleQA meant to run exactly this kind of test — scheduled, unattended, fully automatic — for teams who don't want to write custom infrastructure every time they need to know whether their hardware is behaving. The idea isn't exotic lab equipment. It's the same instruments most small hardware teams already have on a bench: a multimeter, a power supply, an oscilloscope. Standard equipment, talking to a system that knows how to ask it honest questions and check its own answers.

That last part — actually talking to real instruments instead of simulating them — is the piece I haven't built yet, and I'd rather say that plainly than pretend otherwise. Everything this story describes has been proven against simulated hardware: real logic, real bugs, real fixes, but not yet a real oscilloscope on a real desk. That's next.

And the ships haven't gone anywhere, for what it's worth — I still don't know if they end up staying inside CQA-AUTO or find their own home somewhere else in CycleQA down the line. I'm leaving that one open on purpose. Some questions are worth not answering before you have to.

Most of this, if I'm honest, didn't get worked out at a desk. It got worked out the way most of my ideas do — walking the beach near home, stone to stone, trusting my weight to each one before reaching for the next. I didn't plan it as a metaphor. It just turned out to already be one: you don't get to the next stone by leaping for it. You get there by trusting the one you're already standing on.