History of PowerDNS: 2013-2020 (Technology)
In this part of the history of PowerDNS, I talk about the technical developments from 2013 to 2020. Over these years many fascinating business & hiring developments also happened, but these are described in a separate post, part 3B of the PowerDNS History.
For context, the 2013-2020 era is what I would describe as “PowerDNS 3.0” - how we grew from a one (or two) person company to a healthy and thriving department of Open-Xchange. It should be realised that all the technical changes described below happened while we were merging with Open-Xchange. Mergers & acquisitions are rarely good for a company, neither in the short or longer runs. But here it did work out.
Technology: The Big Cleanup
At the start of this part 3, PowerDNS is already 14 years old. Over time, software accumulates complexity. In addition, after so many years, even parts of the core code that are “suboptimal” are now generally well tolerated - and by now, some of the product might even be relying on the oddities. These are the so-called ’load bearing bugs'.
Over time, the cognitive load of all of these oddities may become too much to bear. It is however rare for mature software products to take a step back and focus on the internals. Most software development remains driven by product development and customer requirements.
For some reason, PowerDNS did manage to perform an internal cleanup that exposed many many bugs and oddities. I estimate we spent over half a year on it. During that time we fixed the PowerDNS original sin - treating domain names as ASCII strings.
DNS is in some ways a rather odd protocol. ‘www.powerdns.com’ looks surprisingly like an ASCII string, but the semantics are way different. For example, if we want to sort ‘aaa.powerdns.net’ and ‘bbb.powerdns.com’ in DNS order, it turns out that ‘bbb.powerdns.com’ sorts before ‘aaa.powerdns.net’: DNS sorts based on ’labels’, starting at the last one.
Also, WwW.poWerDnS.CoM and www.powerdns.com are equivalent in all ways (except when DNSSEC standards explicitly say they aren’t). Then there is the matter of escaping - contrary to what people often think, DNS is actually 8 bit clean, you can put anything in there, including dots and spaces. Not everything in DNS is a host name, for which more stringent rules apply. You can ignore this for a long time, but eventually someone registers a SOA record with a space in the email address, and things must not break at that point.
PowerDNS 2.x and 3.x struggled mightily with this, and performed escaping and unescaping in all kinds of places, mostly correctly too. But only mostly - and this would not get us towards correct DNSSEC.
For the shift to 4.x, we allowed ourselves to replace all the DNS name innards of PowerDNS with a ‘DNSName’ class that would get everything right. In addition, we moved on to C++ 2011, which would prove to be extremely helpful. Although PowerDNS 4.0 probably had more bugs than the last 3.x releases, from 4.1 and onwards things were definitely better.
I’m very proud that we allowed ourselves the time to revamp our inner workings - I can recommend this highly to other projects as well, even if they feel the time just isn’t there. It’ll pay off in the long run, and it will make your life a lot more fun.
DNSSEC forced us to take this step - other software projects might also find a compelling event (“excuse”) to trigger an internal cleanup.
It should be noted however that a big cleanup is something very different from starting from scratch. This has a far worse track record, which has been attributed to the second system effect. Among other things, this means that you’ll tend to want to fix everything that was broken in the original- while forgetting a lot that was great in ‘version 1’.
Whether a rewrite-from-scratch or a rip-out-the-bad-stuff operation is appropriate depends on the project, and you’ll only find out afterwards if you got it right.
DNSSEC
DNSSEC in 3.x was my baby, and it was.. tolerable. In most common situations, it would do the right thing. It most certainly was easy to use! But if you tried exotics, you’d run into problems. Partially this was due to the ASCII nature of domain names in 3.x, but partially it was also due to PowerDNS simply not being stringent enough internally to deal with all the corner cases. DNS is a very difficult protocol and DNSSEC builds on getting all the DNS details right.
As an example, if you have a name called ‘www.france.powerdns.com’, this summons ‘france.powerdns.com’ into existence as well. This had been true in DNS for a very long time (empty non-terminals), but getting it wrong carried no penalty. The operator of .COM and .NET for example famously got this wrong for a decade+. With the advent of DNSSEC, empty non-terminals suddenly became very important.
In 2015 we had hired Pieter Lexis, which again proved to be a very good development. At that time, PowerDNS consisted of Peter, Pieter and Bert much to the consternation of the outside world “do you have to be called Peter to work there?”. That we also frequently worked with Jan-Piet Mens only added to the confusion.
We knew our Pieter already because earlier he’d fixed up our documentation simply because he could not tolerate how sloppy it was. This is one of the tremendous advantages of being an open source project
- not only do you get people fixing your stuff, it is also a very good way of recruiting new employees.
Pieter did not disappoint and succeeded in doing a PowerDNS release on his first day of work.
Similarly, some time later, Rémi Gacogne (who was already a valued open source contributor to other projects) sent out an open application. Subsequently, Otto Moerbeek (the OpenBSD malloc maintainer) sent us an almost identical email if he could join us.
Peter, Pieter, Rémi and open source contributor & certified consultant Kees Monshouwer subsequently professionalised my toy DNSSEC implementation into the real thing - expanding DNSSEC from the authoritative server to the recursor. These days Otto is also in firm command of the most complex areas of (Power)DNS, including qname minimization.
I also want to highlight the work of allround friend of PowerDNS Jan-Piet Mens who built some difficult DNSSEC setups with PowerDNS and just .. kept on finding things that were not good. To this day I fear his ‘big.aa’ zonefile! But it did massively contribute to a more robust and correct PowerDNS.
Dnsdist
For a long time, people ran DNS as just a single process. It was called named, and it was indeed ’the naming daemon’. This single process (from BIND) performed both authoritative and resolving roles, and did a lot of other things too. Djbdns already split up resolving and authoritative, and PowerDNS followed suit. Because it turns out that while the DNS protocol is a common standard, resolving DNS and authoritative DNS are in fact very different disciplines. They may share code and infrastructure of course, but if you mix up both disciplines on a single IP address, strange things can happen.
At the end of part 2, I told about the new difficult customer, where we had a very hands-on role during migration and rollout. This customer was not as flexible as earlier customers and we faced the need to forward DNS queries from one IP address to another IP address - without changing anything. It turns out this is surprisingly hard to do without special infrastructure. You can fake this with elaborate iptables rules, but it is not very robust. There is no ‘xinetd’ that allows you to forward UDP bidirectionally. For TCP this is easy to do because TCP packets are robustly related to each other - unlike UDP datagrams.
To solve this problem, I had whipped up a small program called ‘dnsdist’
- it performed just this function, forwarding DNS queries to one or more IP addresses and also relaying responses to the original client. This small tool was slated to help with the migration, and we left it there. In the end it turned out we did not need dnsdist to do that migration, but the code did get written.
The Scandinavian Problem
This is related to dnsdist, I promise. As noted, PowerDNS had become very big in The Netherlands, powering over half of all .NL domain names, with DNSSEC. Something similar was going on in Scandinavia, but there people had somehow learned that “PowerDNS sucked”. It appears Scandinavian webhosters are tight with each other, because they all learned this news.
This stuff is lethal for your company.
Bad news spreads like wildfire, even if it is not true. Some of these incidents were due to a very old PowerDNS bug, long since fixed. But that does not stop people from saying PowerDNS sucks, because it did give them problems. Also, people love blaming free open source software for their problems - it is a lot safer to do this than (say) to blame Oracle.
I hope people think a little bit about what this means - big expensive companies ship far worse software and users will carefully only say ‘vendor C’ or ‘vendor J’ had a problem. Yet when there is an open source bug, we can all vent our rage in public, and perhaps not mention we’re running the version that shipped with CentOS 5, and are refusing to upgrade.
Anyhow, in this context, Loopia from Sweden started to complain about PowerDNS, and we simply could not have that. We had to defend ourselves against unspecified problems from Scandinavia too much already. We enlisted the help of Kees Monshouwer, one of the PowerDNS certified consultants. And we had to break out everything to get to the bottom of the issue, which manifested itself as a steady slowdown in response times. After a short while, PowerDNS would become unusable, and had to be restarted.
Now, the Loopia people worked very well with us, and they were willing to run as many experimental PowerDNS versions as we sent them. And we sent them a lot before we found the core of the problem. But I was impressed by their flexibility and we asked them why they were so willing to stop and start things.
The answer turned out to be OpenBSD’s relayd. Relayd is an impressive piece of load balancing software that within itself contains a quite capable DNS module. Because of Relayd, the Loopia people were fine with individual backends going up and down like a yoyo - either Relayd would find that out itself, or they’d reconfigure the daemon in advance.
And I realised I wanted that functionality for all our users.
In the end we found that the Loopia problem was related to an unfortunate interaction between PowerDNS, an uncommon encryption library & and its compilation settings, leading to massive memory fragmentation. I think this is the most epic bug hunt we’d ever been on.
The results were pretty good however - the hunt for the bug caused us to add a lot of new performance metrics. Lots of unrelated issues got fixed along the way. We also sharded some caches because we mistakenly thought lock contention was a problem. It wasn’t, but PowerDNS came out a lot better anyhow.
Dnsdist birth
Inspired by Relayd’s success over at Loopia, I expanded the primordial dnsdist until it could do basic DNS load balancing using several different algorithms (round robin, ‘first available’, weighted, hashed etc). I also added a console and snazzy statistics. This was all possible based on our earlier good experiences with Lua as an embedded programming language. By making dnsdist configurable and commandable from Lua, we very quickly gained a lot of functionality ‘out of thin air’.
The earliest versions of dnsdist provided load balancing, simple rules for sending traffic in different directions and ample statistics. A very early discovery was that load balancing as performed for HTTP is detrimental for DNS. If queries are divided over many backends, the effective cache hit rate goes down. For DNS, it is best to spread traffic over as few servers as possible. Often this turns out to be *one* server!
Interestingly enough, if left alone, dnsdist would often figure this out by itself, if configured to distribute traffic to the server that is fastest on average. Once a server has a tiny edge over the rest, it attracts yet more traffic, and becomes even faster etc. In this way, dnsdist slowly converges on an optimal traffic distribution. This is akin to Google reporting in 2013 that their ‘Omega’ cluster management system started to exhibit emergent behaviour.
Even when dnsdist was still exceptionally fresh, people were already running it. I have never experienced any piece of software being in such high demand. Long before we released 1.0 live from UKNOF34, dnsdist was already powering big setups - people ripped it out of our hands, code still warm from Github.
We can wonder why this is. The distribution capabilities of dnsdist indeed give tremendous flexibility, something that was not available before to most users. There were generic (hardware) load balancers with DNS features of course. As noted, dnsdist sort of started because we needed it for a specific situation. But while doing this, we did wonder if the world needed another piece of software. There is no shortage of software.
We asked the DNS-OARC audience in 2015 if they thought the world needed a ‘dnsdist’, and overwhelmingly they thought it did.
It also turned out we could not find anyone that was happy with their DNS load balancer solution (except a few relayd users!). Dnsdist suddenly provided operators with the possibility to move traffic around to new servers, something that had previously required either delegation or IP address gymnastics.
One addictive thing about dnsdist is the console. While it is running, you can interrogate the system, get statistics, plot graphs and even change rules on the fly.
While full featured name server software like BIND also had abilities to forward traffic to backends, this perhaps felt more like a reconfiguration than logging in to a console. I’m not aware of many setups previously routing their traffic through an existing nameserver to backends.
One explicit design goal was that ‘dnsdist’ should be vendor neutral. We wanted this to be ‘dnsdist’ and not ‘pdnsdist’. To ensure this, for a long time we put all releases and documentation on dnsdist.org and not on powerdns.com. The vendor neutral aspect was quite deliberate - when we realised there was a lot of demand for dnsdist, we thought it might be a good ‘gateway drug’ to get people to learn about us, “without having to migrate to powerdns, even mentally”.
Over time, dnsdist grew into its frontend role, which quickly encompassed doing DNSCrypt, DNS over TLS and then also DNS over HTTPS/2. I am somewhat sad to see other nameserver products attempting to stuff all these technologies into one big process - it appears to be somewhat more natural to do certificate things in a separate place and not burden the main DNS codebase with that.
I built the initial versions of dnsdist, but handed the program over to Rémi Gacogne very shortly after he joined the company, and things have only improved from there. It is Rémi’s success but I am happy we decided to give dnsdist to him real early in his PowerDNS tenure.
Mentioned earlier in this history, PowerDNS picked the GPL version 2 as its license. Uncommon for the time, we removed the suggested wording that you could pick version 2 or at your choice any later version of the GNU Public License. So we got stuck on GPLv2. The problem now is that a lot of good libraries these days are being released under the Apache 2.0 license, which conflicts with GPLv2. This bites for dnsdist particularly as it wants to use some modern libraries.
Shifting to GPLv3 would solve this, but it would be a massive undertaking trying to get consent from all copyright holders (and their heirs - some of them are sadly no longer with us). But there is an additional problem: GPLv3 contains some watered down anti-TiVo-ization clauses. But even in their weak state, these clauses would likely preclude modem/router vendors from shipping dnsdist, which is something we’d been hoping they might do (to offer DNS over HTTPS or DNS over TLS).
Looking back, dnsdist quickly took the world by storm - and I think this is an example of how a mature software ecosystem (DNS) can still miss significant functionality, even though operators have not yet articulated this explicitly. No one had asked for dnsdist, but once we made it, it got deployed straight from github long before we were doing releases.
Other fields may have similar gaps waiting to be filled by someone.
Dnsdist offshoot: weakforced
During the time of our merger (see part 3B when it arrives), Open-Xchange was facing a challenge where a customer could not migrate to Dovecot since it lacked integrated password brute force detection over multiple servers. Eager to show how useful we were, I forked dnsdist into ‘weakforced’ which was designed to detect password brute forcing attempts using clever sets of rules.
Although only a tiny part of dnsdist ended up in weakforce, I still think this was a worthwhile addition. Timo Sirainen (the founder of Dovecot) quickly whipped up modules for the IMAP daemon, which would report login attempts to weakforced. And if weakforced determined a login was weird or part of a brute forcing attack, the daemon would silently block the login.
Rather quickly Weakforce maintenance was taken over by Neil Cook, one of the people who joined us after the Open-Xchange merger.
LUA records
PowerDNS had long supported Lua modules in the Recursor. Lua is a lovely minimal programming language that can be embedded easily. Its unique model makes deep integration both possible and easy between the C or C++ host process and the Lua modules. And not only is the integration deep, it can also be very fast.
During one of our business adventures (which will be mentioned in the final chapter of this PowerDNS history series), it became necessary to provide DNS-based load balancing & automated answer generation.
PowerDNS Authoritative Server already had several ways of embedding dynamic backends, including a Lua based one. This was pretty good of course, but what you’d really like is to express dynamic rules within a regular (database) backend. It is nice you can host ‘dyn.powerdns.com’ in a Python-based backend, but it is nicer if you can do your dynamic stuff directly in the powerdns.com zone.
As one of my last significant contributions to PowerDNS, I created the LUA record type. Now, the Lua people (who have given us a great language to use) are rather fussy about the name of their language. It is Lua, which is Portuguese for moon. It is not LUA or lua. This put us in somewhat of a bind because DNS record types have always been ALL CAPS. Luckily enough, they forgave us.
With the LUA record, you can simply put some Lua code in a zone and have that generate answers. And within that record, powerful functions are available that check if IP addresses or URLs are available. Operators can also define pools and priorities. Finally, AS and geographical location tables can be used to direct traffic to specific pools or servers (with priorities).
The LUA records were an instant success. Much like dnsdist, people were using the LUA records before they had ever made it to an official PowerDNS release. At one point we found out that significant parts of the Formula 1 racing season were being streamed using a LUA record based CDN - a few months before our first supported release. This did help to very rapidly shake out some bugs though.
Based on many suggestions we were able to add powerful traffic engineering features to the PowerDNS LUA records, and it’ll be interesting to see what people will be building with this feature. “Be your own Akamai”.
Metrics
As mentioned before, over the years we became ever bigger fans of metrics. As somewhat of an innovation, we even opened up a facility where anyone could send us PowerDNS statistics, and we’d turn them into pretty graphs, https://metronome.powerdns.com/. I can’t recommend this highly enough. Every project should have something like this. When open source users report problems, it is very often useful to have them send you telemetry.
Over time, we added ever more metrics, including ones that showed us problems with your backend database, but also virtualization efficiency. This latter one settled many issues on “it used to be fast, now we virtualized, now it is slow and this is PowerDNS’ fault”.
Our own “metronome” server was nice (and secure), but far more powerful graphing and management was made possible by integrating with Grafana and Prometheus.
I am particularly proud of one of our dedicated homegrown graphs though
- the logarithmic percentile performance histogram. In short, this allows operators to instantly see if the bulk (average) performance is good, but also what the slowest percentile experience is like. This was not an original PowerDNS invention by a long shot, but having it available ’live’ was rather innovative, and was later also adopted by other DNS groups.
Open source collaboration
One of the things that make open source great is how easy it is to collaborate. Here I want to highlight a few instances. Early on we fell in love with LMDB, the “lightning database”. We did trip over the rather special and then somewhat underdocumented semantics of LMDB. Most things were written down somewhere, but there was no single place. Together with main author Howard Chu, we worked on gathering and documenting a lot of the semantics. These are now part of the official LMDB documentation.
Later while PowerDNS itself was under a (weak) denial of service attack, I decided we needed an LMDB-powered backend in PowerDNS. I wrote one and was rather pleased with myself. Later, Kees Monshouwer and Robin Geuze of TransIP found the work wasn’t truly done and they finished the backend.
TransIP is one of the largest webhosters of The Netherlands, and they are now planning to phase out their custom DNS software after many years, and replace it by PowerDNS & the LMDB backend. And this is the result of fruitful cooperation between Howard Chu, PowerDNS, Kees Monshouwer and TransIP.
Similarly, for many years Cloudflare and PowerDNS have been feuding somewhat. But that did not stop Cloudflare’s Pavel Odintsov from taking dnsdist and finishing up some features authored earlier by XS4ALL. Pavel also resolved problems he found while scaling dnsdist up to Cloudflare volumes. In this way everyone benefited - XS4ALL started good work, Pavel finished it, and now Cloudflare & everyone benefits from an improved dnsdist.
Further big changes
At the beginning of this period we had some continuous integration going on, and a few automated builds. Today, PowerDNS automatically builds and distributes packages for a whole host of Linux distributions and architectures. Every commit gets taken through a zoo of integration tests, whereby a bunch of PowerDNS processes transfer zones to each other & run tests on the resulting situation.
When looking back it is hard to realise that back in the early 3.0 days releases were a mostly manual process. It is even harder to believe that we mostly got it right.
The team has worked very hard over the past few years to make releasing new versions almost painless. And since I am dispensing some advice here and there, I can much recommend this approach.
Summarising
During this “third period” of PowerDNS we grew a lot in terms of code size, but we simultaneously managed to perform a big cleanup. DNSSEC came of age, both in Authoritative and Recursor.
We also more or less invented the concept of dnsdist and discovered that this appeared to be what DNS was missing.
And all this happened while we were merging & integrating with Open-Xchange, more about which in part 3B of the PowerDNS history.