This part of the history of PowerDNS is mostly about our business adventures & merger with Open-Xchange. The technical history of this period is described in part 3a.

Business

In 2013, PowerDNS was in a crucial phase. Existing business was going well, but we were receiving pointed signals there was trouble on the horizon.

Some existing customers were close to deserting us, because we could not provide them with the top-down attention other vendors were lavishing on them. If upper management keeps on coming down to the technical department to ask why you aren’t using vendor X, eventually going with vendor X starts to look attractive.

We had previously been able to sell “support-only” deals, and this was marvelous. Our open source users did not need private support, so they were happy with our products and public support. Meanwhile, our commercial customers were happy with that same open source software, but also with the private dedicated support. We also implemented any features commercial users needed and made sure migrations were successful.

We explained this situation in a blog post called ‘Open Source Support: out in the open’, and it worked pretty well for us. Help our open source users in public, provide private support and assurance for our commercial customers.

It became clear however that this happy situation was fading. Modern telecommunication companies were (and are) losing skills at a rapid clip. I have written about this elsewhere, but a modern telco can mostly be seen as a financing operation for vendors to roll out equipment, and for a government to sell spectrum licenses. Some fiber may also get laid.

But actually running, installing, operating, configuring, monitoring etc of a large scale access network is not something a modern telco sees as its core business. The vendor should do all the installing and deploying, and will likely also be asked to run the whole setup. Even if the vendor doesn’t do it, a big outsourcing company is usually tasked with this role.

For PowerDNS, this changing landscape was difficult. We’d always been very technology driven & bottom up in our customer approach. Our strategy, if you want to call it that, was to win the hearts and minds of engineers by working with them, and from there get management to sign off on a rather affordable deal. I hesitate to call this a strategy though, it was simply how we liked to work.

But now we started to learn that at many potential new customers, there barely were any “working engineers” to talk to! Our whole approach fell flat there. In addition, not only would we have a hard time engaging with such customers, we also weren’t selling what they needed: turn-key solutions.

The big new customer

Sadly, this part of the PowerDNS story is too recent for me to mention customer names. But we were approached by a very well known and huge telecommunication company, one that needed far more than PowerDNS had originally shipped: centralised monitoring, rollouts of new versions, a web control panel, graphing, anycast load balancing, plus some very niche 3gpp features.

We did not think we’d win this deal, but we decided to give it a very good shot. We engaged with some of our consultants and friends and created a control panel with graphs and it was pretty nice. And indeed, we did not win the deal. But it did advance us towards potentially winning a similar one later.

Then a slightly smaller telco came along, closer to The Netherlands, and they wanted something similar. This one looked winnable except it was clear we’d actually have to do a LOT of work for these people. Still, the alternative was not engaging with any new modern telco customers at all, so Peter and I got to work, even though it was clear this would not be a fun project. But we had to do it.

And lo, with not too much trouble, we actually won this deal, at a price point that was a lot higher than we’d previously managed to get. We later found out that even our “seriously enhanced pricing” was a factor of 8 lower than the competition - a pattern that would later repeat itself.

We bought equipment and tools, got involved with picking hardware, and became part of regular project meetings. Peter and I spent ages on this customer, and even with the help of one of our certified consultants, we only barely managed to deliver this project. This did lead to another popular blog post, “How many hours for Multithreading the Server?”

This proved to Peter and me that we could actually be successful with modern telco customers, but we also realized it came at tremendous personal effort, doing things we really didn’t like.

Scaling PowerDNS-the-company

We spent a lot of time thinking how we might scale the business. One problem is that you can’t hire just one salesperson. Sales is a hit and miss affair and you can’t count on getting the perfect all in one sales person in one go. I only know two such one-person-saleseforces (do hire them if they are on the market!), and people like them don’t go work for small-scale companies. Also in general, if you want to scale a company, go from 2 to 12 employees. Anything in between is iffy - if one person leaves or becomes ill, the whole company is in peril. After several meetings with promising people, we gave up on the idea of scaling the business ourselves.

What we really wanted was to join forces with a company that already had the things we needed, so we could continue to focus on software development and support.

Some years ago we had already been approached by a competitor that wanted to acquire us, but luckily this did not work out - the cultural fit was terrible. They admired Oracle. From work unrelated to PowerDNS I had experience with mergers & acquisitions, so we took PowerDNS through the required exercise to see if it would be viable to sell, and what we would need to clean up first. Selling a company is a lot like selling a house - you need to make sure you have the title to everything, that there are no hidden problems etc. From this work we knew the company was something that could be acquired (with some work).

Then, out of the blue, a small Finnish outfit sent us an email, if we could help with a secure email initiative involving DNSSEC? And when I looked into them, I immediately fell in love - they had exactly what we needed, they were doing complicated software projects already, and clearly were working on the kind of customer intimacy that we needed.

When Mikko Linnamäki and I had our second phone call, we already discussed doing a merger.

This was the Finnish Dovecot IMAP company, which had found a bunch of people to turn the free open source Dovecot server into a business, and it was working out. As we spent more time talking with them about a possible merger when we learned something new.

Dovecot itself was in the process of being acquired.

We did not let that stop us however, and conversations continued. Pretty soon however we learned that “Open-Xchange” was planning to acquire Dovecot.

Open-Xchange was not a widely known name in the open source business, but it should have been. This company provided an open source webmail solution to very large internet access providers and hosters. They also developed an online “Google Docs”-like environment.

As noted, I had been in the mergers and acquisition business before, so I knew a little bit how these things worked. And I wanted to make sure that this would work. Open-Xchange is a very open company (‘Ruthlessly open’ was a motto for a long time), so I invited myself to the Open-Xchange Summit in Munich to spy on what they were doing.

While there, I liked everything I saw. Because the thing is - good software does not sell itself. Almost no good product does. You actually do need customer events, smooching, expensive drinks. At some point you actually do need “Rolex wearing sales guys” (do read this post).

At the Open-Xchange Summit in Munich, I had a meeting scheduled with Rafael Laguna de la Vera, then the OX CEO. And we hit it off famously - which was not much of a surprise because from the OX background and Rafael’s presentations I could find online, I knew we believed many of the same things.

And it bears some repeating. We had this meeting in September 2014 but the themes still resonate. The internet is careening towards massive centralization and ditto massive centralised control. Email is being provided by 3 parties, social media and instant messaging by two (in the West). This was and is a big theme over at Open-Xchange. Not only is this centralization bad for the world, it also is bad for many decentralized companies that previously were in the business of doing telecommunications.

The upshot of this centralization that if you operate outside of Google or Microsoft, your email randomly stops arriving for people. For the past 5 years my private email server has been unable to send email to outlook.com users, despite massive attempts to change this situation. I have even signed contracts with Microsoft to gain insight into what is going on, but nothing helps. Later I spoke to people with firsthand knowledge of how Microsoft accepts email (or doesn’t) and they confirmed small scale senders are not a priority for outlook.com (they do a better job on Office365, probably on purpose).

During my first meeting with Rafael I pitched to him that DNS would soon be going the same way. If left unchecked, the Googles of this world would attempt to also take DNS away from the decentralized internet, and that the large cloud providers would take this over as well. This turned out to have been a prophetic meeting.

We quickly agreed that our companies should merge. Over at PowerDNS we were sure Open-Xchange would honour our open source culture, and in hindsight we can now say this was a correct assessment. We also found our company’s cultural values to be very similar, although it turned out we could not simply merge our development teams. We did work together well though.

Eventually it turned out it would take more than a year to actually finish the acquisition - there were legal loose ends on the PowerDNS side, and Open-Xchange also had to jump through some regulatory hoops. But that did not stop us, from the second phone call we started working together as if we were one team already.

During this time we also found the need to get an actual office again. More significant customers were coming along and our excuses were no longer cutting it. The office space market in The Netherlands is very weird. It is likely weird in other places too, but I have no experience there.

We found various antiseptic offices we could rent for five years, but they lacked any kind of character. In addition, we wanted something really close to a major train station. And also, we wanted it right now. I performed some searches but found nothing promising. I then asked a friend with some experience in the real-estate business and she found us a quite rundown office that was however immediately available. We did indeed move in immediately - the location was marvelous, close to The Hague CS train station, right next to the Dutch parliament and ministries. It was however challenging to keep warm in winter. There was also a rodent problem, and if you turned on the wrong appliance in the kitchen a loud bang would happen. Still, I personally liked it.

The place was however so shabby that I think it did cost us a very corporate style potential customer. “How can the software be good if the office is so bad”. Later Bob Brandt & team found us a better office, and I really miss it.

Joining Open-Xchange

This was a very smooth process in many practical ways. The trick with acquisitions is for the smaller company to stop doing things it is really inefficient at, and transfer those to HQ. But the flip side of that is to not transfer things that make the smaller company work. Processes that work at a scale of 200 employees do not work for 3 employees, and will in practice be extremely painful.

Open-Xchange mostly left us alone, either by conscious choice or because they had better things to do. Probably a mix.

Where we did decide to work together immediately was on the subject of sales - this being the whole goal of the merger in the first place. The first thing we did, spearheaded by my brother Mikko Linnamäki was to fix up some of the extremely cheap & frankly dangerous contracts we had signed back when we were a small company. As long as you are small it is not that bad to sign contracts with unlimited liabilities. This changes when you scale!

Sales & Product Management

As I told in part 1 of this PowerDNS history, my few years of studying physics had not prepared me for the world of business. I find that many technical people suffer from the same problem, often willfully. Technical people personally tend to have terrible experiences when approached by salesfolks. This does nothing to turn you into an adequate salesperson yourself.

Over at Open-Xchange and elsewhere, I have benefited from a masterclass in what doing sales to large organizations means. And beyond that, what should your actual product be?

When we joined Open-Xchange, we had a steady business going, but as noted, we were on the proverbial “burning platform”. We were selling services to a market that was rapidly vanishing. New and even existing customers would need different products but also different kinds of support and services.

Despite this, I was feeling pretty good about things - business was brisk, profits were good. By myself, I’m a pretty risk averse business person - I tend to let growth fund itself, for example.

Open-Xchange was far more ambitious and wanted to figure out how to quickly scale up the PowerDNS business. And this was a somewhat painful process.

Sales is always a matter of 1) Why buy something 2) Why buy it from me 3) Why now? And the problematic thing with DNS is that every company on the internet already uses DNS. Every new deal you sign has to displace some existing supplier or piece of software - the pie is not getting a lot bigger.

It is a lot easier if you can grow your company into a green field where there is lots of excitement about new things. DNS tends to not be like that.

We tried various things - would access providers be interested in faster DNS? You’d think this would fly. We performed a lot of measurements and found that many ISPs could do with better DNS. There’s a reason so many people changed their own DNS settings to 8.8.8.8.

This idea went exactly nowhere! And the reasons are somewhat instructive. Why would an ISP run badly performing DNS? Not because they want to, obviously. If DNS is bad, there has likely been sustained underinvestment in people, hardware or simply maintenance. Something is badly wrong in that case - DNS is relatively easy to fix.

So we would show up with reports showing just how much better performance would be with PowerDNS, and this resonated in no way. And the reason is obvious - they already knew things sucked! And our sales pitch only served as a reminder of how much stuff sucked. So we sold nothing based on “we can make things better”. This in itself is an indictment of the Internet access industry by the way.

Now, in small markets with not too many suppliers, there’s always one way to gain market share: customers that are being bilked by their supplier, or receiving very bad service. Large incumbents sometimes do get lazy, and they will in any case increase pricing regularly - they know many of their customers would internally cost a migration at millions of euros/pounds/dollars of work. As long as an incumbent supplier carefully calibrates their price increases to stay below that point, they can make a lot of money over a decade. But eventually it ends.

The problem with this as a challenger is that you have to wait for it to happen. And once it happens, you do have to be ready with everything the customer needs, including a team that can do the migration.

We were lucky that a number of very large scale internet access providers hit this point of dissatisfaction with their suppliers. In some cases we did not win the deal, but were able to drive down prices for the incumbent party a lot. This in itself does not help you of course, but it does make sure your competition has to work a lot harder.

In addition, from such engagements (which include demos and proof of concept deployments) you do learn if your product is a good fit for the market.

Over several failed attempts to displace an incumbent, we honed our technology to the point where it became very competitive. One key factor there, which I recommend highly, is to actually believe your competitor’s marketing materials.

As a case in point, closed source DNS suppliers kept telling their customers they had great reporting engines. Management sure love getting reports with colourful graphs etc. We even found some marketing materials online describing what these reports supposedly looked like, so we made sure we too could create reams of plots showing all kinds of statistics. Later we found out that the competition had actually only made marketing materials, and that there were no actual reports!

Believing competitor marketing can be a great way to exceed customer expectations.

Closing deals

Here I want to pay tribute to the talents Open-Xchange brought to bear on PowerDNS. Our new customer base consisted of very large telecommunications & cable companies. And let me tell you, you don’t just sell something to these kinds of places. This section may perhaps be useful for other open source projects pondering going commercial.

PowerDNS did have some reasonably good business going with very large places, but in almost all cases, these happened by being in the right place at just the right time (as outlined in part 2 of the history of PowerDNS). Also we asked for very little money. I can now disclose that one of the largest cable companies in Europe was paying us 8000 euros a year at one point. Other customers were on more reasonable pricing plans, but still the sort of money that flies under the radar.

Large companies almost exclusively pick new vendors via cumbersome Request For Proposals (RFP) processes. In theory these are meant to objectively pick the best solution, both in terms of pricing and capabilities. In practice, the RFP process is a rotten mess in most places. I’ve written about this before, so I won’t repeat all the pain here.

In short, in an RFP a company gathers a whole bunch of requirements for a solution. These requirements come from different departments, and often are completely incoherent. The solution must scale horizontally, vertically, be totally secure, yet operable from everywhere, live in the cloud and be on premise, be 99.9999% available yet make 100% use of all hardware without redundancy.

Also, various departments have already selected favorite (and different) suppliers. So within the RFP, you’ll find a lot of “feature f*cking”. This is where a department requires that in-memory caches use a lock-free implementation of a skip-hash trie, which happens to be an implementation detail of one vendor only.

And another department has found a similar requirement that only their favorite vendor features.

These conflicting and competing requirements mean that most RFPs are effectively dead on arrival, or at least should be.

To actually win such an RFP requires masterful techniques. Here I want to pay tribute to people who taught me so much.

For starters our original sales manager Thomas Bioud. Customers are often heavily conflicted internally, to the point it can be hard to even pick a venue for a meeting. One customer was so messed up internally they could not settle on what building to meet in. Thomas somehow managed to get on the phone with the facilities people of that customer, squared it, and emailed round “the meeting is at 15:00 on the third floor in building 5”. And everyone fell in line and showed up. Thomas is also such a consummate professional that he brings along his own projector. No technical problem will disrupt his presentation.

Our head of sales Frank Hoberg (also a co-founder of Open-Xchange) provided teeth in negotiations. Sometimes a customer has an actual budget crunch and then you have to be there for them to see what can be done. There is also the flip side. Sometimes you as a vendor must make sure you get paid for the value you deliver. Frank was our backbone and I think he saved the company many times by balancing out the occasions where we helped out a customer with money problems. In addition, his long term strategic relation with many customers allowed him to negotiate very credibly.

But it is not only money. Customers worry. A big contract is a lot like getting married. You tie the knot for at least a few years. Companies and especially individual employees know they are betting their careers on getting it right. If doubts remain, it is always possible to not buy anything, or at least not now.

Closing a deal fundamentally means credibly conveying to the customer that your company can deliver what is required in terms of technology and support. And it is also good to know that most large scale customers are very aware that they have not written down everything you need to know. There will be surprises post signing. Will you stay the course?

So how do you convince the customer that it will work? First you have to listen. Here I want to thank Stuart Paton, the single best telecommunication salesperson I know. Listening sounds easy, just sit there and listen. But that is not the whole story. No customer will come out and tell you the three previous migrations were a disaster. Not unless they trust you.

So how do you make that happen? Be a part of the world of the customer. You should be at the events they frequent. Visit industry fairs. And if you visit the customer on site, create ample opportunity for people to voice their worries and things they care about especially. Listening is an active process.

Once it is understood what is needed, it is also necessary to demonstrate that the requirements will actually be delivered. From the technical side, we geeks know how to do that. Actual hands-on demonstrations & sharing war stories from previous deployments. But that is not enough - even if the technology is fine, how do you convince the customer that the rollout will go well?

Here I want to mention Stephan Martin. I am not exactly sure how this works, but somehow it is possible to radiate (truthfully) that you’ve done 20 of these projects already, and that this 21st project will also work out fine. At one point with a customer we got hung up on training. How would we train the customer staff? Now, everyone dreads this - training system administrators in a classroom pretty much sucks. It is painful to do, it is painful to endure. Eight hours of slides.

Stephan then explained our vision on training and that we vastly prefer to deploy all the software together with customer staff, “in the beginning we do all the typing, in the end your staff does most of it”. And if you have enough experience, the customer can hear your experience and will believe it too. This saved us weeks of painful training & quickly allowed us to move on. Stephan was very often able to share his wide experience with customers to get us over hurdles.

Part of a sales process is working with the customer to deliver proof of concepts and (trial) deployments. A large customer has many moving parts and is not easy to pin down. Your own people are not available at a moment’s notice, but large corporations will very often suddenly change plans and expect you to adapt. Maintaining order while keeping your promises is a very challenging task. Pablo Martinez-Almeida of Open-Xchange delivered this for us - his steadying & calming influence helped us win several deals I am sure.

Large scale customers also lead to large scale contracts. There is no shortage of legal people in the world. Many of them can spot dozens of problems per day they look at your paperwork. But that is not what we need. Customer contracts are often highly skewed to favour the buyer. And likewise, vendor contracts are full of get-out-of-jail terms for the supplier. These things collide.

Given the mountains of paper, a large deal might never get off the ground. A thousand pages is nothing here.

The Open-Xchange legal team is full of professionals that can deal with this stuff. I want to mention Maxim Letski, Open-Xchange’s general counsel. Unlike many legal professionals, Maxim was quick to grasp technical details, and also able to reach out in time if things were not clear. And unlike most legal departments, Maxim & coworkers were able to accept risk. I vividly remember one upsetting paragraph in a contract that simply would not work. Yet the customer was very attached to it.

So Maxim asked if the situation described in that paragraph would ever happen. And for technical reasons, it never would. “So let’s not worry about it then”. This is a rare skill in legal circles.

Contracts also contain a lot of words on risks, penalties, liabilities etc. Eventually that turns into a judgment call for a company. We need to hold the customer harmless on intellectual property claims, in theory for unlimited amounts of money. Will that happen? What level of liability do we accept? These are the big calls. Open-Xchange’s COO Carsten Dirks was the one to make these calls, and like Maxim, he made serious studies of the contacts, but was then able to decide, accepting reasonable levels of risk.

The bigger the customer the more difficult it can be. It is little known, but two of Europe’s largest telecommunication companies have actually bundled their procurement departments. I should credit Daniel Halbe for his Sitzfleisch and experience that actually got one of our most important deals across the line with this gargantuan organization. Similarly, one customer decided it would be easier to do business with us through a network vendor. This wasn’t the case, but Oliver Loeschner managed to close the deal anyhow.

Large companies these days are also rightfully worried about security. Is your software safe? Do you have a paper trail of every change that has ever been made to your software & the approval process? Is there open source in there? Doesn’t that mean you are hacked already? We worked hard to keep our software secure, but you do need to work with the customer to show you’ve thought of everything, and also have the specific solutions their security policies require. Here we benefited tremendously from Neil Cook’s efforts - much like Stephan Martin mentioned above, he was able to convey to customers his decades long experience in this field.

Telecommunication companies can be difficult to understand if you don’t grasp their strategic environment. What drives them? CAPEX? OPEX? Consolidation on edge computing? I’ve benefited greatly from Alberto Camerlo’s insights. Errol Vanderhorst provided a lot of insights into customer processes and thinking as well.

And finally, all these efforts come with a lot of administration, travel arrangements & accounting. I know I caused Monika Schroeder and Michael Knapstein a lot of pain from time to time, but it was great to know we could be sure the numbers and books were all well kept! Also a huge round of thanks to everyone else that supported our efforts.

Summarising, if you want to make your project a success with very large scale customers, you’ll of course need capable developers & people that can deploy your software. But you’ll also need:

• A sales organization that can truly listen to customers & find out what they need - even beyond what they write down formally
• Folks that provide steel to your pricing negotiations
• A legal team that does not just say “no” but can actively weigh risks
• A management team willing to sign off on those risks
• People that can radiate experience to customers in terms of delivering projects
• Credible security staff that can similarly convey confidence & work with and around (!) requirements
• Have rock solid project management that can calm often erratic large corporations
• Staff that knows about the strategic environment a customer is in
• A great administrative and support organization for all the people mentioned above!

At the beginning of this article I noted that that Peter and I initially looked into scaling PowerDNS ourselves without merging with another company. Given this list above, it is abundantly clear that we took the right decision to join forces with Open-Xchange and Dovecot.

Parting words

So if everything is so great over at PowerDNS, why did I leave? I wrote about this more extensively in my “Goodbye DNS” blogpost. But in short - the goal for me in 2013 was to get PowerDNS into steady waters. This meant finding a partner, merging with them and making the acquisition a success. This took several years - and once things were settled, it really was time for me to move on.

But I’ll forever remain grateful for the many people that believed in the company & how Open-Xchange lavished resources on the project until the business truly scaled.

The key moment was when Rafael (the Open-Xchange CEO) and I met in Munich - I know that we both felt that it would work. If you ever ponder selling your company & you don’t feel the love, think very hard before you proceed!

Similarly, I’d like to thank many people over at customers who stuck the course with us, or who believed in our small company when they could have also stayed with their very large existing supplier.