White Hairs & Binary Protocols

At the end of an excellent talk on binary protocols from Jason Smith of Container Solutions at Kubecon last week, one of the youthful attendees asked an interesting question. “Are modern binary protocols as painful as I hear ASN.1 was?” The answer was “no” but there are still some interesting lessons from the evolution of binary protocols that are worth pondering.

Binary vs Text — Text Won

The downside of binary was messages were hard to read by eye, so it was hard to diagnose problems. You often ended up decoding messages by hand. When bandwidth and cpu became more abundant in the late ’90s, text-based encodings like XML and JSON took over. They were expensive but much simpler to debug — you could easily see if something was malformed or merely unexpected.

Binary is Back

What Lessons Does ASN.1 Teach Us About Modern Binary Protocols?

  • ASN.1 pre-dated common open source. There was no standard library for encoding and decoding ASN.1, everyone wrote their own encoder. When inter-operating you frequently decoded binary messages that had been encoded by a different encoder. This mean’t lots of mismatches that had to be resolved with manual decoding and workarounds in your encoder implementation.

Lesson one: if you’re going to inter-operate it’s a lot easier if everyone uses the same encoding library. Don’t rely on compliance with a standard.

  • To compound this, there were seemingly endless encoding variants of ASN.1, which made it even harder to write consistent encoders.

Lesson two: if you are going to invent a binary protocol, keep it simple. The more encoding variants you have and the more encoders there are out there the harder it’s going to be to get smooth interoperability.

And that’s the story of where my white hair comes from.

End note — looking at Linkerd, which I’ve never tried, it sounds like it might effectively enforce the use of a consistent encoder version??? That sounds handy?

SciFi author interested in tech, engineering, science, art, SF, economics, psychology, startups. Chaotic evil.

SciFi author interested in tech, engineering, science, art, SF, economics, psychology, startups. Chaotic evil.