This is a more personal post than I usually write, and it was prompted by several people asking what I had been up to lately. It turns out that it is somewhat of a story. It is a long story too.

Since the beginning of 2018 day to day management of PowerDNS is now truly in the hands of professionals. I’m very proud to report that PowerDNS has not only survived the transition to Open-Xchange but is actually thriving & has achieved sufficient revenues & talented staff that we’ve solved the famous open source “what if Bert gets hit by a bus” problem.

Mind you, I’m still around, but now with a broader remit, and I get to spend less time personally signing contracts and opining on office arrangements.

It is pretty rare for acquisitions to go well and I can highly recommend getting acquired by Open-Xchange.

An open internet

PowerDNS (which I co-founded in 1999) has always been about keeping the internet open, although it took us until 2001 or so to actually open source our software. In 2015 we joined forces with Open-Xchange, a company with a storied history, that now encompasses Dovecot, PowerDNS and App Suite. App Suite is relatively unknown but it powers the webmail for subscribers of companies like Comcast, Liberty Global, Cox, Orange and many others. It is mostly open source but definitely 100% open standards.

Keeping the internet open means that new products can ‘join in’, and that those products can be used by new entrants to the market to sell or provide services. These services should then be available to everyone, without barriers. It should not be the case that if you show up as a new kid your stuff doesn’t work.

For email for example this is now effectively broken. Unless you run a huge mail server, Outlook/Microsoft, Google and others will frequently drop your email, without any explanation or recourse. Email is no longer an open market unless you are a very large player. If your email is hosted by such a large party, you may not be aware of this problem. But even Jamie Zawinski, one of the original Netscape/Mozilla people who knows Everyone, recently found out he could no longer reliably send email from his own server. I’ve had a similar experience, and to this day my “perfect” domain name (hubertnet.nl) goes straight to spam over at Outlook.com, despite signing contracts with Microsoft and complying with all their demands.

This happened because email became too centralized: the dominant providers are now in a position where they can ignore the rest of the world, to their mutual benefit. This has effectively closed down the email market, unless you accept spotty reliability (or ‘inbox rate’).

Centralization is now sadly also coming for DNS – both Google and Mozilla are making moves that may well end up with their browsers sending DNS queries straight to servers picked by these vendors.

I spent a good part of this year pointing out the risks associated with centralizing DNS, but also making sure the (superior) technology used to justify this centralization is also available for local service providers.

DNS: Documentation & Software

Keeping the internet open is not only a matter of power play & centralization, it also requires access to the technology required to take part. If the technology is hard to understand, that itself acts as a gatekeeper. In March of this year, somewhat by accident, I ended up presenting at the IETF in London on how hard DNS had become to understand.

In my research for that presentation, mysteriously called ‘The DNS Camel’ (also by accident), I found out that understanding DNS now requires reading around 2000 pages of arcane texts. Yet we all wondered why new DNS software was so dire. Turns out no one was reading those 2000 pages - who would’ve thought? My message was, please stop adding further complexity, because one more RFC might break the camel’s back.

The presentation really struck a chord with people, and the associated blogpost attracted around 14000 views. Some people did note that while I highlighted the problem, I had not offered any solutions.

As a followup to the DNS Camel presentation, I created the following things:

The DNS Camel Viewer, a centralised place where all relevant DNS documents can be found in one place. With the help of many contributors we’ve now categorised most documents as ‘core’ or ‘uses of DNS’ etc.
“Hello-DNS”, a description of the DNS protocol for newcomers. We’ve tried real hard to keep this accessible, and have for example split up what you need to know if all you want to do is implement a stub resolver. This page has attracted over 50 thousand different visitors so it appears there is real demand.
“tdns”, software to go with “Hello-DNS”, implementing a standards compliant authoritative server and a workable resolver in a few thousands lines of well documented code.
tdns & hello-dns were presented at DNS-OARC and also received attention on the IETF blog.
Based on the tdns resolver (tres), I checked a list of popular domain name on if they can be resolved by a naive (ie, correct) resolver. The results can be seen here.

Programming Languages

In 2018, I spent more time learning to appreciate modern C++. Although Go and Rust are getting a lot of attention, it appears large swathes of open source communities are still plugging away in C. C programmers are urged incessantly to migrate to Rust or Go, and I think these languages do have an important role to play. But they aren’t the only game in town.

I therefore authored a six-part introduction to Modern C++ for C Programmers.

These pages attracted over a hundred thousand views, which was pleasing. Less pleasing were the attacks from the ‘Rust evangelism strikeforce’ who told me in no uncertain terms that no one should program in anything other than Rust, ever. As far as I can tell, none of these complaints came from people that ship any software in Rust.

Based on this, I decided to pen some admittedly biased words on the relative merits of C++, Go and Rust. This post led to some robust discussion here and there.

In hindsight, while I think Go and Rust are interesting, Go is yet another Google project having an outsize influence on the open source world. Promoting C++ is therefore important to maintain a healthy ecosystem of modern programming languages.

It may be worthwhile to note, most of the ‘high traffic’ articles I wrote included a note ‘if you like this stuff, come work with me’. This has done wonders for PowerDNS & Open-Xchange recruiting efforts and has contributed to the hiring of several people.

Education

I can’t help it, I like to explain things. Earlier this year, NASA landed its InSight mission on Mars, filled with international instruments for learning more about the interior of the red planet. The landing happened between 8 and 9PM local time, but I wanted to share the excitement with children from my kids’ school.

My goal was to host a live blog with the NASA webcast front and center, but with the ability for viewers to ask questions, while I also added written notes (live & in Dutch) to the transmission.

I searched high and low but could not find a downloadable or ‘cloud based’ liveblogging platform that was suitable so I wrote my own, tentatively called powerblog.

The landing & the liveblog were a success, and I wrote down my notes in Liveblog: a Mars landing watch party with school kids.

Documenting & Enhancing other people’s software

Sometimes people write great software but spend more time on the software than on explaining, documenting and (for lack of a better word) ‘selling’ it. I’ve previously contributed documentation to the pretty impressive LMDB key/value store. LMDB is as impressive as it is demanding to use - it is super fast and super stable, as long as you use it exactly right.

I had a brief conversation with Howard Chu of LMDB on if he’d be interested in an ‘LMDB easy mode’ that is harder to misuse, but his opinion is that wrappers should do that work. So, I wrote an LMDB wrapper called lmdb-safe. It tries to be fast but also safe.

Similarly, earlier this year, we picked the very impressive H2O webserver & library for DNS over HTTPS/2 in dnsdist. The webserver library, libh2o is a lot of fun to use but the documentation is very spread out. To further the use of this excellent library, I wrote ‘hello-h2o’, somewhat of a pattern there. The H2O authors have been extremely helpful in answering questions, and I hope the documentation we wrote will be of use to people.

Much like with LMDB, I then also felt the urge to write a wrapper for H2O so I could use it easily, but still at blazing speed, from modern C++. So far, this library can be found hiding within powerblog as mentioned above.

With this wrapper, lightning fast HTTPS/2 support is now easily available to C++ programmers.

PowerDNS Scalability

As noted we care about keeping the internet open. PowerDNS Authoritative Server is dominant among “pretty large” DNS hosters (up to a few million domains), but most of the super large operators either use technology developed at home or have worked proprietary miracles to make PowerDNS scale to their numbers.

This year, we authored two notable technologies that should enable any PowerDNS operator to go head to head with dominant cloud providers. First, our ‘Lua records’ enable smooth and powerful geographical load balancing and high-availability, delivering functionality present in Route 53 or Akamai services for example.

While first presented in December 2017, we only merged and polished this code in the course of 2018.

Secondly, when running PowerDNS with tens of millions of zones, replication becomes a challenge. We had long wanted to do an LMDB backend but lacked the technology to make this robust. This explains our LMDB efforts outlined above.

Based on the new LMDB stuff, PowerDNS Authoritative Server will soon have the ability to serve from a compact database that can easily be distributed or updated in a decentralized fashion. More details can be found here.

Projects that didn’t go anywhere: ipcipher

2018 was of course the year of GDPR. PowerDNS and Open-Xchange take privacy seriously, but we also know that to keep stuff running well we sometimes need to store IP addresses. I got pretty fired up about ways to encrypt IPv4 addresses in such a way that they stay IPv4 addresses. In other words, you encrypt 1.2.3.4 to 232.123.24.127. Because an IP address stays an IP address, you can keep using your existing tools for analysis and storage.

And to decrypt an IP address, you then go to your privacy officer and ask them to do it for you.

The idea is pretty neat and it got some traction. I even presented about it remotely to the NDSS conference, but it was later pointed out that the technology I had chosen was not quite good enough for the purpose.

I do know what the right way of doing it is, but some of these ways appear to be patent encumbered, and I simply ran out of attention and time to finish this project. Sorry.

But do read about it here and here.

(If used correctly, the ipcipher protocol is still safe, so we’ll continue to support it).

Notable blog posts

I wrote a bit on a recently republicized idea on how to spoof DNS with fragments. As far as I know this is still the only standalone explanation. I first presented about this idea in 2009, but from experts I understand the concept was then already well known in some circles.
Almost ancient history by now, but earlier this year, news broke of Meltdown & Spectre, which I described in this post. This got

100k visitors and landed me on Dutch radio, which was nice.
Some words on how to recruit people, in Dutch. Nothing groundbreaking, but common sense is not as common as one would hope. I wrote this because I kept having to explain the same thing to companies I help with recruiting (including my own).

Small bits of software

Lots of people now scan the internet full time, to the point that even an “empty” internet connection now sees a lot of traffic. For most things, such 100% scans aren’t necessary. I love statistics, and I attempted a sub-random scan of a tiny part of the internet to count the number of nameservers. Sub-random in this context means not randomly trying IP addresses but using a pattern that gradually covers more of the address space. I ran the scans, which confirmed that you don’t need that much traffic to get good numbers. Sadly, it turned out the vaunted Sobol sub-random scan is only slightly better than an actual random scan.
- Tweet one, Tweet two.
I care deeply about email. It is an open, federated, standards based way of exchanging information. It is also full of bad stuff. To explore some ideas, I authored software to study email flows. It can be found through this tweet. Nothing interesting was discovered, but the pictures sure are nice.
mcounter. As our code gets to be more high performance, ever smaller things turn out to be bottlenecks. One of these seemingly trivial things is incrementing counters that are shared between threads. Once the rest of your code is as fast as it could be, you can easily spend 20% of your time on this task. mcounter is a simple way in C++ to get counters that can be updated with no overhead from many threads, and safely read for reporting.

What’s up for 2019?

Predictions are hard, especially if they are about the future. But for now, I expect what I’ve been doing to continue - creating and fostering technology that allows people to communicate using open standards, without having their every move be monetized by cloud providers.