History of PowerDNS: 1999-2003

Hi everyone,

I have often told the history of PowerDNS, or at least how I recall it, during talks and presentations. As I am about to truly leave DNS and PowerDNS, I thought it was high time to document my recollections.

Since this is a 20+ year old story right now, it is very possible I have forgotten or misremembered important things. Do let me know if you miss important things, or if you recall things differently!

The launch of PowerDNS should be seen in the context of the “dot com boom”. Back in 1997, I became part of the somewhat rocky start of Casema Internet, one of the earliest Cable ISPs in The Netherlands (currently part of VodafoneZiggo). There was a great shortness of Internet talent at the time, to the point that even my mediocre system administration skills were in high demand.

Over at Casema, I joined a very small crew that helped scale the fledgling ISP from 50 to 50,000 users in a very short time. While there, I worked on building the authentication/authorization platform, creating a cable modem performance management system, optimizing the mail platform and many other things. In many ways it was a wild ride.

Because the shortage of developers and engineers was so dire, eventually Casema paid me very significant money (at least for a physics student) to hang around for longer than I originally intended. After a while this also led me to give up my studies of physics. The money I made there would later enable me to launch PowerDNS.

While at Casema, I had learned a few things about doing business and operations. But I also really wanted to do something for myself. I did manage to somewhat commercialise the cable modem monitoring platform I had written (“Pinguin”), it was eventually also used at CAI Westland. I had also tried to sell it to the somewhat odd producer of the somewhat odd Casema cable modems, but this did not work out.

Somewhere in 1999, I was contacted by the people behind the then famous V3 web redirection service. At the time, you could get a free homepage in many places, but you’d end up with a ghastly URL like http://geocities.com/something/else/yourname. The folks behind V3 had bought way more interesting names like ‘come.to’, ‘go.to’, ‘surf.to’ and ‘browse.to’. This means you could get http://come.to/bert and it would redirect to your ugly Geocities name.

The people behind V3 were themselves very interesting entrepreneurs. A big airplane consortium (Fokker) had gone bust some years earlier and it represented one of the largest bankruptcies in Dutch history. The non-airplane assets of Fokker were so large that no one really knew what it all was. The assets were therefore offered in bulk for a pittance. The founders of V3 recognized there was way more value there, if only someone would be able to inventory it all in time.

They built databases and infrastructure very rapidly and were able to somehow finance the acquisition of the assets and sell it all before the interest was due. In 2020 this all sounds simple, but back in the day there WERE no websites or databases you could just use for this kind of thing. This was the era of the fax machine.

V3 was founded with the profits of this transaction, and was headed by Hans-Poul Veldhuijzen van Zanten (“Z” for short), Eric Visser and Paul de Groot. This team was the “ZEP Holding”. This holding not only did IT things, it also dabbled in a (very nice) fish restaurant, but also eventually invested in kitchen ventilation technology and probably a lot more.

In true “dot com boom” fashion, V3 had been sold in exchange for vaporous stocks in a US company called Fortune City. Fortune City wanted to consolidate the servers that performed the come.to/go.to/surf.to redirection to the US. This did not sit well with the ZEP Holding as they knew how that would end.

An argument from Fortune City was that servers in the US would deliver better service to US users of the internet. The V3 servers in Europe were indeed rather far removed from the US and this caused noticeable slowdowns.

V3 employee Taco Kemna, who knew me, then came up with the idea to ask me if I perhaps had an idea to improve this situation, possibly via DNS. I went to the very pretty V3/ZEP Holding office (which they shared with DPI Animation House, a co-venture) at the Javastraat in The Hague.

And because this was 1999, instead of me quoting some software development to them, we decided to launch a company that would create geographical load balancing DNS software. 1999 was a wild time.

A bit on DNS

It may be good at this point to discuss DNS a bit. For a very long time, the DNS protocol had only one viable implementation, BIND. And at times even BIND was not very viable - the software had long periods of instability and security problems. There were good reasons for this at the time, and I hasten to say that these days BIND is excellent software, and ISC (the BIND authors) and PowerDNS people get along famously.

At the time, the inestimable Dan J Bernstein (DJB) of qmail fame had also written a nameserver. This nameserver was a somewhat acquired taste, deviating substantially from how software was generally written and operated.

In hindsight, I can only say that Dan Bernstein’s DNS products are works of sheer brilliance. Many times over the subsequent 20 years of software development, I would think I had found some kind of vulnerability in DNS and found that Dan had already thought about that. DJB’s unique style of communications and somewhat unusual software operational practices led to his software not being widely adopted.

But the stunning quality of his work shows that we maybe should have accepted more of his stringent development style and software infrastructure. Now 20 years later, most software processes are actually launched in ways that DJB pioneered, for example.

The first version of the geographical load balancing software I wrote was in fact based on a hacked up version of djbdns, and it made it possible to send US users of a domain name to US servers, and European users to European servers.

For a variety of reasons however, this software was never actually deployed for V3. But by then we had found a name, “PowerDNS”, and had actually founded a legal entity with me as the majority shareholder. Other names considered for PowerDNS were SuperDNS and UltraDNS. The PowerDNS.COM domain name was also already taken and we had to shell out $9000 or so to acquire it. We later found that SuperDNS was an internal name of a Verisign server that is now called Atlas. UltraDNS would later also become a DNS company.

We then said goodbye to the djbdns prototype and started over with ‘AhuDNS’, named after my irc nickname ‘ahu’. This was to be my first C++ project, and I think it showed. The quality was initially not too great. AhuDNS was a closed source authoritative nameserver, with working modular backends, both for databases and for other things.

Offices

PowerDNS first shared an office with ZEP Holding and DPI Animation House on the Javastraat in The Hague. This was a very nice monumental building. Because of this monumental status, certain tax advantages were available. But it turned out that exploiting these was such a hassle that in the end only a tiny amount of money was saved, but lawyers made out like bandits. This happened slightly before PowerDNS really got going there, but it is a lesson I have never forgotten. If a complicated scheme allows you to save some taxes, just pay the taxes.

Later we moved to the Duinweg, also in The Hague. This was also a rather grand building, but it was standing on its last legs. Formerly an embassy, it still had a somewhat mysterious air around it. Probably because it had been an embassy, there were weird connectors in places, where communication equipment had likely been connected originally. There was also a mysterious button on one wall. We pressed it a lot and joked that whenever we did, somewhere in the world a light went on.

The building was very old and slated to be demolished, but the ground floor had been made very representative and was nice. PowerDNS lived upstairs and it was a lovely dump.

We got a really cheap lease on it, on the condition we’d vacate quickly if they would ever start construction work. In the basement was an apartment that was rented out to a couple, and we somewhat shared the kitchen that also was in that basement. Their cat would sometimes venture into our office - it was all very domestic. One morning I arrived early for a call, went down to the basement kitchen, and ran into our downstairs neighbour - who was naked. She said hi, I got my coffee and went on with my day. It was that kind of place.

The business opportunity

This part is likely very educational for anyone pondering launching a startup. I can tell you we messed this up bigtime. The initial story was good - there was only one more or less viable DNS product (BIND), but it did not meet the needs of the market. It did not scale to large numbers of domains. Every larger scale DNS hoster employed a whole team dedicated to making BIND deal with thousands or even tens of thousands of domain names.

BIND was not database driven, so whenever a domain name was changed, a text file had to be regenerated, and BIND had to be convinced to reload, and while reloading it would often not answer questions.

In short, there was an unmet need in the market for better software that would serve DNS from a relational database, instead of from text files.

It may be hard to imagine, but at the time, it wasn’t even clear if a nameserver like that would be legal! Here and there the DNS standards seemed to imply that DNS could only be served from “zone files”, which were text based. In addition, everyone was sure databases would not be fast enough.

These are ideal conditions to disrupt, where competition is held back because the feeling exists that 1) what you are doing might be good, but is not normal 2) your idea will never work anyhow. If you ever encounter these two conditions, consider yourself very lucky.

Yet we messed up everything. Because I had studied physics (even though I dropped out), I was blissfully unaware of how businesses operate, yet very convinced that since I had mastered at least part of quantum mechanics, doing marketing and sales would come naturally to me. Hard to believe, but studying physics (or any other “hard” science) does this to people.

Our pitch briefly went like this: “getting BIND to work with large amounts of zonefiles sucks, so you need to spend a lot of effort on that. PowerDNS is much more efficient, and allows you to do your work with far fewer employees”. As pitches go, this one is not the worst.

Except if your only business contacts are the VERY system administrators that will lose their job if they buy into your premise!

We got nowhere. A pitch that says “buy our software and lose your job” is dead on arrival. Yet the alternative was out of our reach - we could have sold this to ISP management, but we utterly failed to do that as well. Not only did we not have meaningful contacts in positions of authority, we could check none of the boxes senior management was looking for.

We did not have any solid sales people. Our homepage was a joke. Our manner of communication was nothing like that of other vendors that were successfully selling things (like Cisco, Netscape, Oracle etc).

PowerDNS was a bit lost at this time, and we went looking for business. These days you’d call that a pivot. V3 and ZEP Holding had meanwhile changed somewhat, and gained a UK sister organization. Together these launched Betalab, and in The Netherlands, there was Trilab, which would provide software development services for internet things.

Through these other ventures we did gain some possible business. The UK Betalab folks worked hard to deliver us projects. This led to typical startup things where we suddenly promised to launch a whole web-based DNS management platform for registrars. We also ended up hosting this ourselves for customers and our own web assets. This was “PowerDNS Express”.

One of the more poignant events was a big demo over at Ascio in Denmark, where Nikolaj Nyholm somehow had faith in us. We had slaved all through the night on the demo and flew out to Copenhagen. There we told the Ascio people that we had worked 24/7 to deliver their demo as promised, and I will never forget the pitch perfect Scandinavian response “then you did not start in time”. They were right too.

Things were not going that well with the company. We tried a few things, but nothing really delivered solid revenues, let alone the growth we needed.

One notable product we tried was PowerMail (now open source). This never went anywhere commercially, but was an interesting distributed, redundant, high performance email storage solution. The design meant that every incoming message was stored on at least N backends, and these could also be remote. On accessing email, PowerMail would gather all messages from all available nodes. It was a very fast and modular system which delivered spectacular availability numbers.

Even though nothing came from it, we eventually found out a major UK bank and South Africa’s largest insurance company actively relied on the software

• because of its distributed and robust nature. PowerMail is one of these things that I think could have gone far had it been developed in the context of sufficient business expertise. That it found important deployment even without any sales efforts is testament to that.

As things were heading south, my interests had already started to diverge a bit from DNS. Being in startup mode, you think about nothing but your company. Unless it starts to really sour on you. So, I had shifted one letter on the keyboard and became interested in DNA. I wrote a page “DNA for coders” that still exists and that attracted some attention online. News had somehow reached a Californian venture capitalist, and he wanted to discuss some of his DNA & computer programming related ideas.

I can no longer find a trace of that email, but it was supposedly the VC that had originally funded Quantum hard drives (or it may have been his brother). Anyhow, the VC and I get on a call (an actual hours long international POTS phone call, this being 2002 or so). We discussed his idea but sadly it didn’t fly. However, he was somewhat intrigued by our PowerDNS story and asked me to tell him our pitch and plans etc.

This would turn out to be a very important phone call. It is quite enlightening to sit down and truly explain your business idea to a good and willing listener. He asked many questions and in the end summarised our situation as “You built a product for people with no money that don’t want to buy it”. And I was stunned. Not that I had felt that things were going well, but hearing it this way in one sentence really brought it home to me.

At this time, PowerDNS had swollen to 12 people or so. But eventually, we realised we could not make it work. There were no revenues to support this many people. My memories of this time (2001-2002) are not very concrete or complete - I remember being extremely sad at the company not delivering on our hopes. I remember feeling awful about having to fire people (or not renew contracts, or no longer paying contractors). I don’t think we actually fired anyone in the strict legal sense - but not having your time limited contract renewed feels like the same thing of course.

As we shrank, in my mind, this marked the end of “PowerDNS 1.0”.

Here I want to mention some of the key people from that time.

We hired Stefan Arentz as a contractor, and he attempted to professionalise our software development practices, something we weren’t always ready for, but he did move us in the right direction. He also contributed code but should also definitely be remembered as the designer of the PowerDNS logo. This was so nice that we never felt the need to change it! Stefan’s exit was not smooth and I am sorry we made a mess of things.

We also benefited from the work of my good friend Floor Plikaar who now runs the successful Dutch educational audio book publisher Home Academy. When PowerDNS was waning and we shrunk to only a few employees, Floor and I did not coordinate anything in terms of references and as a result when I was asked for a reference by a potential employer, I told them exactly the wrong thing. I still feel bad about this, but perhaps that job wasn’t meant for her anyhow. (Update: she confirms this was the case!).

Hartger Ruijs also joined us and allowed us to share in his enthusiasm and energy. Like Floor, he is now also a successful entrepreneur, heading Dutch security company Computest.

Michel Stol also did fun things for us, like porting PowerDNS to Windows. This did teach us some very interesting things. The Windows port was a very interesting proof of concept, and Michel had delivered it at great speed. We never made a formal release, but you could download a .EXE with a “SuperPimp”-technology powered installer.

However, it was not anywhere NEAR something you could take into production. And yet we found out that people did. On a massive scale even. For years, very important domain names ran based on this toy product. Even though it would leak memory like a sieve and required frequent restarts, it was in full production.

The Windows version also led to an unending stream of inane support requests. Installing software on Linux or Unix takes a little bit of thought. But it appears anyone with a pair of hands is able to get a Windows .EXE going. The questions we got were spectacular, where people clearly had no idea what they were doing.

Eventually we removed the Windows binary from the website, but then threats came in that we had to restore it. One guy even went so far to hammer our website with requests for the URL that used to contain the .EXE. Eventually we put it back briefly so he would stop.

Not all hires were as grand - early in PowerDNS, the holding company hired someone who only showed up for one day, then mysteriously fell ill for a month and then tried to sue us for unfair dismissal. Turned out this person had a history of this, and we eventually won this case in court. But it took a while.

We also had an employee who left after our downsizing and took his company phone with him. Being the kind of place we were we did not keep track of phones and subscriptions, but at one point we realized something was up. So we called him on his old office phone number and he did indeed answer. Turns out he’d been making good use of that phone and apparently thought the phone and the subscription were some kind of parting gift.

The end of PowerDNS 1.0

As we could not make it work, eventually no one was left. Trilab was still around and continued to employ people who did work on PowerDNS Express - but this was not part of “PowerDNS-the-company”. In order to keep us viable, we had decided earlier to open source the PowerDNS software under a license that would make sure no one else would commercialise it. GPLv2 it was - without the possibility to upgrade to v3, a decision that would come to haunt us later on.

“i saw that it was written in C++ and licensed under the GPL, so i stupidly assumed it would fail and paid no attention.” –Paul Vixie

During this period, Hans-Poul Veldhuijzen van Zanten and I explored various other things. We met with the people that were building Skype. They even offered us a very affordable stake in the company, but Hans and I could not see how this could be commercialized. Anyhow.

We also looked into what is quite popular right now, zero cost stock trading. This ended when we found out the chosen broker platform could do one whole transaction per second at most.

We also looked into promising new DSL modulation technology that, to this day, remains just that - promising. It never worked out.

What did work out was keeping the PowerDNS software alive and keeping up appearances. The website was still there, the phone was being answered, software releases kept appearing. To the outside world, the company was not dead. Later this would turn out to be key.

What other people were doing

Could we have turned PowerDNS 1.0 into a success? Even in 2002, PowerDNS was at best a 3 year old company (if you counted it charitably). Our potential customers were web hosters, who are notoriously bad at spending money, and large scale communications providers, who’d never bet their operations on a fresh startup. And not only were we fresh, we looked nothing like what telcos were looking for.

Meanwhile, two other companies had entered the scene. Infoblox built an appliance based on BIND. This did not really address any of the issues with BIND, but this appliance was a lot easier to buy and understand in telco circles.

Secondly, a company called Nominum came into existence, with an office right next to ISC that had previously been developing the BIND software. I do not know a lot about Nominum’s origins, but eventually they did what anyone with any business sense would have done. They built decent DNS software with a credible team around it, doing all the things that the paying market of large scale telecommunication companies likes. Good sales people, solid documents, expensive website.

In hindsight this was obviously what anyone wanting to enter the market should have done. There’s no way to make money from web hosters, and telecommunications companies will only part with their money if you take part in their rituals. And Nominum did and it was a pretty big success eventually.

Again in hindsight, I feel bad we did not ourselves come to this conclusion. However, even if we had (and maybe unconsciously we did), we could not have acted on it. There are people that can turn your software shop into something that telcos want to drop big money on. I now know a lot of these people even. But none of them would have joined our scrappy startup - their talents were being well remunerated elsewhere.

So at least in my mind, PowerDNS 1.0 failed because 1) we did not have the strategic vision and 2) even we had, we would not have been able to deliver on it.

The good news is that PowerDNS 1.0 turns out not to have been the end of it. Head over to episode two of this riveting tale to find out what happened next!