History of PowerDNS: 2003-2013

This is part two of the PowerDNS history as I recall it. As noted in part one, some of this stuff is rather old, and it is entirely possible I am misremembering things. Please do let me know if you find any important omissions or mistakes!

At the end of 2002, beginning of 2003, PowerDNS as a going concern was gone. There were no more employees, there were no paying customers. Trilab did keep PowerDNS Express, a DNS hosting platform, alive, but it was not part of PowerDNS-the-company. In addition, the PowerDNS.COM website was also still there, with a working phone number even.

At that time, PowerDNS was not making any money, not even enough money to keep one person housed and fed. I clearly needed to do something else.

I had given myself some room to learn new things in the interim. I had never been very fond of HTML and ‘web software’, but I forced myself to learn that. This would later prove to be extremely useful. Many developers get “old before their time” and reject new technologies, often for the best of reasons, but the result is not good. However, HTML and the web in general were not going away, so it was useful to learn more about that.

Post-PowerDNS I explored a few things, but nothing really went anywhere. In 2002 the world was still recovering from the “Dot Com Boom”. There wasn’t a lot of fun development going round.

If you’ve ever been an entrepreneur, setting your own hours and goals, going back to a normal job is an incredibly difficult step. I could not see myself do that. But I did need to find something.

A friend of mine, and an early mentor, jokingly pointed me at two job vacancies at the Dutch General Intelligence and Security Service (AIVD), sort of like a combined MI5/MI6 style agency. Although it was meant as a joke, it did seem like the thing for me. It asked for all kinds of skills that I actually had, and working in such an exciting and secret place looked to be a lot more fun than getting a normal job. Turns out, it was indeed a lot more fun.

By this time, PowerDNS as a company was dormant, but the software was open source and quite thriving. I did worry if I would be able to work for a “secret service” and continue to work on this software in public. The intelligence service agreed that I could, as long as I would not be doing any cryptography or security stuff in there.

But I did worry a lot about what the open source community would think if some kind of spy (which I wasn’t) turned out to have been writing their software. These worries were not unfounded. In a way I lucked out

  • it was custom at the time (and still is) for AIVD employees to not tell anyone where they work. So I didn’t. In a way, working for an intelligence agency is almost as bad as working for Apple - you virtually disappear.

It may be good to know I never mixed “work and pleasure” - PowerDNS software was never in any way influenced by my government work.

Saturday, 10th of January 2004, 13:37:04 UTC

This was an important moment for PowerDNS, and I know exactly where I was. Within the PowerDNS Authoritative Server there is a loop that checks if the zones we follow from other servers are still fresh. Some zones need to be checked more frequently than others. So the loop calculates what the next wake up moment is. It does so by calculating the domain that needs to be checked soonest, and in order to do so, it goes over all domain names, and every time it finds one with an earlier check time, that becomes the next wake-up moment. For safety, you compare all these times against a ‘date very far in the future’.

The typical date for ‘very far in the future date’ is the famous UNIX epoch rollover in 2038, which happens 2³¹ seconds after midnight UTC January 1st 1970. Sadly I made the mistake of picking Saturday, 10th of January 2004, 13:37:04 UTC, which is 2³⁰ seconds after January 1st 1970. Note the 30.

This in turn meant that when that date arrived, the loop never slept anymore

  • it would always immediately wake up and see which domains needed checking, thousands of times per second. This effectively broke PowerDNS for many installations.

And on this Saturday, I was at the birthday party of my girlfriend, where ever more people were frantically trying to contact me why their PowerDNS was going crazy. This was not a good day for PowerDNS, or the birthday party. We still got married just over a year later, so that was good.

XS4ALL

The first Dutch mass market ISP sprang from the Hacktic hacker community. XS4ALL was highly idealistic and kept up strong standards for user privacy, encryption and generally keeping the internet free and open.

For context, DNS has two halves - Authoritative DNS publishes DNS data, Resolving (or caching or recursing) DNS looks up such published data. Both halves are very important. Originally, PowerDNS only offered authoritative service.

A DNS resolver is a spectacularly difficult piece of software that has to answer end-user questions at a very high clip, while looking up answers through far flung and slowly responding authoritative servers. Authoritative servers that, it should be noted, often give ludicrous answers, or refer to yet other servers where you should continue your quest for results. And very often, these directions prove to be circular or actually go nowhere.

Writing resolvers is therefore very hard. I had read the source code of the two big open source resolvers (BIND and dnscache from DJB), and both had left my eyes bleeding. Although I may have achieved a few things as a software developer, in one important way I’m actually not that good at it. I can’t deal with too much complexity, it is too much for me. And the way these resolvers were written, as big event-driven state machines, was not something I could cope with. I’m in awe of developers who can.

Earlier, I had written a piece on ‘linear programming’ which stated that people have a far easier time thinking about code if it is laid out linearly in time, and I thought I should bring this to DNS resolving. A DNS resolver frequently has to deal with thousands of questions being ‘in the air’ simultaneously. This normally precludes any kind of multithreading, which would enable linear code.

My innovation was to write a userspace cooperative multitasking system that would be lightweight enough to support all these queries in parallel, but written without state machines that would confuse me.

This had turned out to be a success, although the PowerDNS Recursor had not yet seen any production use. But XS4ALL really wanted to use it, and for good reason. At the time, late 2005, BIND was going through a bad phase where it was not that stable and performance also disappointed. It should be realised that if DNS lookups are slow, the whole internet service is slow. And even worse, if the DNS resolver is down, everything is down. This makes DNS rather vital.

Back then the only reliable resolver came from Nominum, who were also mentioned in 1999-2003 part of the PowerDNS history. The problem for XS4ALL, or more exactly, the problem for Nominum, was that XS4ALL did not want to be running any closed source software that came with expensive sales guys.

In 2005, XS4ALL offered 7000 euros for me to take the PowerDNS Recursor to ‘production quality’, and that is what happened. In hindsight, it is fun to realize that Nominum was competing against 7000 euros. A single visit of their US sales force to XS4ALL would’ve cost more.

Some time after we had proved our scaling capabilities over at XS4ALL, a sort of similar process happened over at AOL, but we added the sort of things you need if you have millions and millions of subscribers. At that time, AOL was still a huge deal. Based on the improved PowerDNS Recursor software, AOL was able to decommission 120 servers or so. I sometimes think about the energy savings this caused when I get on a plane.

Stuff you learn when you scale up

One in a million problems will on average happen several times if your user is a country-sized ISP. So when we deployed at XS4ALL, a very small number of their users reported losing internet access at almost precisely the time that the PowerDNS Recursor was deployed. Things would start working again if they reset their router.

Over at PowerDNS, I had developed a lot of tooling to study PCAP traffic, since for some reason I had gotten quite good at dealing with recorded packets. This included the ‘dnswasher’ tool which allowed users to strip out customer IP addresses, so PCAPs could be shared with us.

Usually our set of PCAP tooling would quickly enough show the problem, but here, we got nothing. It was almost as if the affected customers didn’t have any internet at all. We did not see any DNS packets from those users at the PowerDNS nodes.

Eventually matters got bad enough that XS4ALL got permission to look at the actual customer packets, straight from the wire. And there we found that DNS traffic just stopped if a customer had received a single DNS response from PowerDNS.. But not if the DNS response came from somewhere else.

A bit-by-bit comparison quickly found the problem - DNS has a primitive form of compression built in, which is (mostly) optional. A DNS response starts by repeating the question, which is also found earlier in the packet, which offers a good chance for compression. Almost all DNS software therefore begins its response with ‘C0 0C’, which means: this name can be found at the start of the payload.

Turns out some customers were using a CPE router that thought ‘C0 0C’ was some kind of ‘answer starts here’ marker. And if it did not find that marker, its DNS component would crash. And at that time, PowerDNS did not compress that first response record, so there was no ‘C0 0C’.

I share this long winded story as an example of problems you only encounter while scaling to really big volumes, but also as an example of how optional things in DNS are no longer optional. And this too is part of the DNS Camel problem I alluded to in my ‘Goodbye DNS, Goodbye PowerDNS’ post.

Wikipedia

In part one we read that the origin of PowerDNS was the need to get DNS-based geographical load balancing. And oddly enough, we never implemented that in PowerDNS - only in the djbdns-based prototype. We’d meant to finish that one day, but that day never came. Until Mark Bergsma needed this functionality for the Wikipedia, coded it up and contributed it. He also performed a very useful cleanup of one of the more difficult parts of PowerDNS, the label compression code. In the end, PowerDNS did DNS for Wikipedia for many years, using code we should have written, but which Wikimedia contributed. Eventually Wikipedia moved on to dedicated load balancing software.

It took until 2018 before PowerDNS would get better geographical load balancing, in the form of the Lua records, which enable true traffic engineering across continents and clusters. Incidentally, the Lua records were almost my last contribution to PowerDNS.

The Nominum Squeeze

As noted, at one point, there really was only one DNS resolver you could safely run at scale: Nominum. And this point was not lost on Nominum! After aggressively conquering the market, the time had come for the VC-backed company to actually make money. And they picked a good time for it.

Pricing negotiations with telecommunication companies are never fun, at least, unless you ask for tiny amounts of money. And even then. Because of significant inertia in the telco sphere, many vendors will bet that their initial low pricing can be recouped three years down the road when the original contract expires, or when additional features are required. Because once a telecommunication company has migrated to your product, the effort of migrating to something else is huge.

In this way, attractive pricing in year Y often ends up as extortionate pricing in year Z.

It appears that somewhere around 2006, 2007, Nominum did a round of this and attempted to squeeze all their customers at the same time. Rumor has it that they delivered an ‘End of Life’ notice to one customer who hadn’t even yet gotten round to deploying the software yet - forcing them to restart negotiations immediately. This is another terrible trick to raise money as a software company if you have a captive audience.

Starting 2007, a small trickle of Nominum customers jumped ship and migrated to PowerDNS. The first to make the move was Liberty, a cable ISP with a large European footprint. They were joined by Shaw Cable from Canada.

I was at that point still asking for trivial amounts of money - I already had a job that paid the rent. In hindsight this was super scary and I can’t recommend against this highly enough. If you do business with billion dollar entities, do not under any circumstances sell anything to them cheaply.

This is not because you should price gauge them, but big companies can cause enormous amounts of hassle for you very easily. They have large legal departments that can make your life miserable, even if you have done nothing wrong. Simply getting the procurement department of a major company to accept your invoice is sheer hell. Getting them to pay out can also cost you whole days on the phone.

A specific risk is that a large company appears to buy something from you in good faith, but because of circumstances beyond your control, it simply doesn’t work out. All the nice people you work with are now gone, and the folks that are left decide that you have caused all their problems. Not only won’t you get paid, if you are lucky you end up not getting sued.

I truly lucked out there by never getting into serious trouble. But don’t do this. If you ever do business with huge companies, make sure you ask for enough money that you can survive only ever receiving half of it, 120 days after the end of their mysterious payment cycle.

2007 also saw business from companies like Register.com, EasyDNS, Locaweb, Mediatemple and the Dutch state broadcaster NPO. Oddly enough, such business from hosting companies would be rare in later years.

Staying alive

As I noted earlier, even though PowerDNS as a company ground to a halt in 2002, 2003, we kept up appearances. The website was still there, the phone was being answered (although no one ever called), regular software releases happened, bugs kept being fixed. All without a budget.

When I later asked our new raft of customers why they went into business with us, many of them noted that we were now a 7 year old software company with a track record. There is a harsh lesson in there - if you want to be successful, there are no shortcuts to becoming an “established” company. Just existing for a number of years adds a lot of credibility. But by definition, it takes years.

In hindsight, keeping the company alive optically, even with no revenues, was key to reemerging successfully a few years later.

Directi

As noted above, XS4ALL and later AOL were responsible for a significant push to get the PowerDNS Recursor ready for large scale production use. The PowerDNS Authoritative Server had of course already seen a lot of use, but the performance just wasn’t good enough.

In 2008, Directi, a famous hosting company and domain registrar from India, contacted us to request improvements. This turned into a wild ride. I had never done any serious business in India, but I quickly learned a lot. Directi was seeing large scale denial of service attacks on its own infrastructure and on services it hosted for third parties. And PowerDNS just wasn’t ready for that.

On a series of conference calls we discussed this, where the Directi CEO (Bhavin Turakhia) played a very active role. Why was PowerDNS using a binary tree? Why a red-black tree? There were other data structures that were far more favourable! This jolted me - in The Netherlands, in Europe, we are used to CEOs that have no idea what their company is doing. “Use shorter sentences and simpler words - this is for the board!”.

Not so at Directi. Based on their feedback and detailed requirements, PowerDNS Authoritative Server performance increased remarkably. But I think that only with the advent of dnsdist (to be discussed in part 3 of this history) we finally delivered what they needed.

Meanwhile CEO Bhavin went on to great business success, and he is now a billionaire investor. And most Dutch (and European) boards continue to be headed by collections of lawyers and other people very far removed from what their company is actually doing.

The Ursula von der Leyen breakthrough

In 2008 or so, odd messages started to arrive from Germany, from a somewhat anonymous email address. Could PowerDNS do X? Would we consider adding Y? Why does Z not work?

Initially I was not quite sure what to make of these emails, but through my other work (I had since left the government to start a joint-venture with security company Fox-IT), I had learned how to recognize good customer originated product management. This user was clearly expressing things they really needed in production. So over a period of many months, we went back and forth a bit, and I implemented most of their requirements (phrased as suggestions).

A key component appeared to be based on the anticipated need for providing dynamic answers. Germany’s then minister for family affairs Ursula von der Leyen pushed the idea that domains featuring child pornography should be blocked. In a very complicated arrangement, the German government would distribute a list of hashes of “verboten” domain names that needed to be blocked. But the details were all secret and could not be shared with vendors.

Although I had complicated feelings about such blocking, we found a way to implement sufficiently dynamic behaviour in PowerDNS so that the operator could develop such policies themselves, using the powerful Lua scripting language we integrated. Integrating Lua may have been one of the more innovative things PowerDNS has done.

Then for some time nothing happened and suddenly Deutsche Telekom procurement sent a detailed request for a quote for a DNS solution with specific abilities. They also wanted real 24/7 support with good response times.

For PowerDNS, this changed everything. It was clear to me, because I was the only one running the PowerDNS company that something had to change. I needed a 24/7 phone number, and I also needed to be sure I would be able to hire someone in case there ever were any real problems.

From the Fox-IT JV, I had also gotten over my fear of asking for real money. So I consulted some contacts, what would be a reasonable amount of money to ask for this 24/7 support from a gigantic telecommunication company? Eventually we came up with a number and I sent out a quote. Very quickly DT responded if they could get a 10% discount, so I gave them 5%.

In hindsight, the annual amount of money was still probably close to what they were paying their previous vendor per month. Anecdotally I later got an indication how much money Nominum wanted to have for ‘VdL-filtering’ in other places. I think DT got a pretty good deal in the end. And in an odd development, the filter was never deployed.

(Note that going over our notes, we’re not sure if the specific hash-based filtering was part of the early requirements, or if that only came later. Perhaps only the groundwork (Lua) was an early requirement).

After DT had signed, other large telcos followed & replaced Nominum by PowerDNS. This created a very healthy cash flow for the company - except that it was still just all me.

For quite a long time I had to make sure the software was good enough that none of my customers would ever need to call me for 24/7 support. But all that time, I also made sure I could actually be reached at all times in order to provide support if anyone needed it.

DNSSEC

When PowerDNS began in 1999, work on DNSSEC had already started. In short, DNS is an ancient protocol, with no encryption or authentication on it at all. It is fully plaintext, you just have to hope the data you are getting is correct. And even in 1999 this hurt, so attempts were being made to shoehorn cryptography into DNS.

This would prove to be an exceptionally painful process, taking over 15 years to complete. For much of that time, my opinion was that DNSSEC was too much trouble to be worth it. Implementers would not get it right, domain hosters would not get it right, administrators would mess things up etc. Turns out all of this was true.

However, DNSSEC had some magic support going on where despite all the setbacks, the push continued. And I have to admit, when the process was done, DNSSEC had become very complicated, but it did in fact meet all the requirements, and there were no significant mistakes in the protocol.

At some point I decided that we could no longer resist and would have to start supporting DNSSEC. But if we did it, we wanted to make sure it would be a success. SIDN, the maintainer of the .NL domain, also really wanted to support our efforts, and it was there that we had our breakthrough insight in 2010 or so.

Up till then, DNSSEC had been “a hobby for nerds”. The implementations kept adding more features and more algorithms. I had tried and failed to get DNSSEC going myself with the then available tools - these asked me too many questions and it was very likely you’d give impossible answers. What hash algorithm do you want to use? What asymmetric cryptography? How many bits? Which NSEC? How many rounds of salting for NSEC? Opt-in? Opt-out? Most combinations of answers would be nonsensical - you had to get it just right.

The key insight we developed during a meeting with Markus Travaille over at the Dutch registry SIDN was that PowerDNS should not add YET MORE features to the DNSSEC pile, we should add “it just works DNSSEC”. One switch you can flick, and boom, safe DNSSEC with a known good configuration.

I think this is one other teachable moment from the history of PowerDNS

  • when you enter a market, think real hard about what everyone else is doing, but don’t join in and start doing the same thing. Up to that point, DNSSEC really had been a glorious playground for people who like their cryptography strong and difficult. But that’s not where the market is. End-users did care about DNSSEC, but only a bit. And definitely not enough to invest in learning cryptography. This is reasonably classic Innovator’s Dilemma stuff.

PowerDNS made doing DNSSEC easy enough that, combined with the SIDN financial incentives, it was interesting enough for hosters to roll it out. SIDN and other registries had been offering an incentive already, but it clearly was not enough to get people ‘over the hump’. The combination of easy software AND incentive paid off however.

In 2012 SIDN facilitated this by (among other things) buying a PowerDNS support agreement “on behalf of the entire .NL hosting industry”. In return, for several years, PowerDNS privately helped out many prominent Dutch hosters with their DNSSEC migrations.

Later, this role was mostly taken over by Kees Monshouwer, one of the wonderful PowerDNS certified engineers, who also completed and perfected our DNSSEC implementation.

The end result of Kees' and our work was that over half of .NL domains became DNSSEC secure.

How do you make money as an open source company?

So to begin with, PowerDNS solved this problem. This proves that it can at least sometimes be solved. However, we solved it only for one specific company in one specific market.

Before I head into the details, I want to make the point that developing open source software is way way cheaper than writing closed source software. You get to benefit from so much free infrastructure, QA testing, feedback, suggestions, community product development, portability adjustments etc. Only if you develop closed source software after doing open source do you realise you now suddenly need a whole staff to do what you earlier achieved together with your community.

Anyhow, on to making money: the key insight to any business is that you have to sell something that other people value, and that they can’t get more cheaply somewhere else.

The other insight is that developing open source software is a lot of fun, to the point that finding staff is way easier than if you are a closed source company. You don’t have to bribe people to come work for you, developers sometimes literally show up automatically if you’ve shown yourself to be a real worthwhile open source place of work.

These two paragraphs set stringent limits on what companies can turn open source into a business success. This will not work (say) for payroll software - you can’t do that as a mission oriented company. PowerDNS has a clear mission that developers can get behind - keep the internet running smoothly and safely. You don’t get that for payroll.

What you also need is a split in interests. To be open source means you give away all or almost all of your intellectual property. The trick is that there must be something left for customers to buy.

What PowerDNS has been selling for a long time is reassurance - yes you can use this software, yes it will become a success, no one will blame you for this choice: the vendor ticked all the procurement boxes, there is solid 24/7 support. Note that I listed the support part last.

Many commercial closed source companies do great business selling support since you actually need so much of it. The software simply isn’t that good. Open source does not tend to have this problem - if it sucks that badly, no one would use the product. There is no sales force to trick you into using bad open source software!

Yet most attempts at commercializing open source revolve around selling support - support most people in fact don’t need. And for what it is worth, most open source developers also seriously hate providing support.

PowerDNS was in the lucky position that many of its paying users (large scale communications companies) did need support, at least on paper - but what they mostly needed was reassurance that the project would be a success.

And this is something you can do too: be a credible partner, ask for credible amounts of money, work with the project staff.

Until 2012 or so, this was all we needed to do to be successful as an open source business: sell reassurance. And up to that time we did not sell a single line of non-open source code.

Selling “total support”

One final important thing to note is that the support we did deliver was really very very good. Some commercial vendors actually have the gall to sell this as a separate “white glove” level of support. The expectation level of commercial software support is currently set at spending multiple hours on the phone with poorly paid people and having to do battle with them until they admit you might have an actual problem. This is good for no one.

Instead, we fervently believed our customers - don’t try to contradict someone who feels they are experiencing a bug! It is sure they have a problem - it might not be caused by your software, but the only way you are going to figure that out is by working together. And best get started on that immediately!

Over at PowerDNS at the time, what we provided was “total support”. Often we found the root cause of issues not in PowerDNS, but in firewalls, switches, storage platforms, operating systems or a long forgotten F5 load balancer somewhere over at Microsoft or eBay. And sometimes in PowerDNS too of course. But we’d make sure it would get fixed in either case.

One major example involved a slowness that on analysis revealed a glaring shortcoming in the scalability of IPv6 in Linux. We made the PowerDNS software tune a parameter to evade this problem but also worked with the Linux IPv6 people to get the issue fixed there.

Peter van Dijk

In 2011, I sold my Fox-IT joint-venture. The new owner rightfully demanded that I would now actually work for them, as is customary with M&A transactions. So I finally became a regular employee somewhere!

This of course would cause problems for PowerDNS. Even though I negotiated that I could do development in evening hours, I could not deliver 24/7 support for PowerDNS while working for another company.

Faced with this dilemma, I realised I need an all round entrepreneurial person to take over, someone who could do software development, deal with bugs, help customers, all with minimal to no supervision.

Way back in the mists of time, I had already done some work with Peter van Dijk, who knew a lot about DNS and even PowerDNS already.

And luckily enough, I was able to convince him to join PowerDNS as an employee with a revenue sharing arrangement.

This turned out to have been a very good development for PowerDNS, and I will forever be grateful for how he was able to take on so much of the work. During the time I was distracted, PowerDNS flourished like never before!

After I became a free agent again, Peter stayed on and together we grew the company further. Although we lost one of our original large customers (who temporarily defected back to Nominum), business continued to be good.

Open source

One of the first things Peter did was move us to GitHub. I was initially a bit fearful - I found GitHub to be somewhat too open, somewhat too much “cloud”. But Peter charged on and we first got a GitHub mirror, which was already useful. We (that is, Peter) then embarked on a huge project to migrate each and every commit and each and every ticket from our old Subversion & Trac setup to GitHub.

This was a pretty big project, but the payoff was huge. Suddenly interacting with our open source community was so much easier. This truly opened the floodgates to really working together on the project. The advent of continuous integration only strengthened that. For the first time, major components of PowerDNS (like RFC 2136 support) were contributed externally.

I can’t stress enough how wonderful it is to be part of an actual community, where people on and off the payroll review pull requests and author them. I almost feel sorry for developers of closed source software! But not really.

Growing pains

Yet not all was well. As the scale of our customers increased, some of them started to find out they were betting their entire business on a 2 person company. Now, this never had been a secret, and because we had been delivering stellar service, the people actually doing the work were very happy with the situation. Also, we used many different email addresses (like sales@, invoice-queries@, support@) to make ourselves look bigger.

Another problem was that we had no office. By now this is all the rage, but back then we sometimes had to tell customers there really was no need to visit us, we happen to be in Vienna anyhow on that date! What a coincidence. This was not very convincing. At one point, a potential new business partner insisted on dropping by. They too just happened to be in The Netherlands by accident, probably on their way somewhere from San Francisco. But, they wanted to come over, and we also wanted to make a good impression.

We turned to PowerDNS co-founder Hans-Poul’s company, Trilab, and asked if we could fake it in their office for a bit. The problem was that this office listed all kinds of company names at the entrance, but not ours. Through some miracle, the fine people there found a way to print a very nice transparent PowerDNS logo that they glued in place just in time. It looked super duper professional. The glue was barely dry when our guests showed up. This meeting eventually did not go anywhere, but the thought was nice.

Years later when that office was vacant, people asked me if PowerDNS was still there. Turns out that glue was spectacularly good and no one could get the logo off there again. I know because I went over and tried!

Meanwhile, far larger vendors were performing top-down sales on our customers. Middle and senior management at large telecommunication companies are frequently wined and dined by the likes of Cisco, Juniper, Infoblox etc. And Peter and I could do a lot of things, but we could not do anything like that. Nor did we want to.

We had also found out that the majority of our potential new customers needed far, far more from us than we were willing to provide them with. A modern telecommunication company does not install software or deploy any services - they expect the vendor to do all that. They are also willing to pay for this, but you still also have to be willing to supply such integration work.

Since Peter and I weren’t well equipped to do this kind of thing, we went looking around to see what could be done - because something had to be done - we were at risk of losing our existing customers to top-down sales efforts, and since telecommunications companies had changed, our old model of selling “really good support only” was becoming unviable.

Please do read on in part 3 to find out what we ended up doing to save the company!