PRE-DRAFT, PLEASE DO NOT SHARE FURTHER, BUT PERHAPS ALREADY HAS SOME USEFUL THINGS

This article is a followup to the earlier post Cloud Naïve: Europe and the ‘Bijenkorf’ Megascaler, where I covered how no European providers are able to supply “cloud native” services at scale.

That post also covered the main gaps between what cloud native users expect and what European providers can deliver, based on two surveys, plus feedback received after a previous article and a report from a Dutch government hosted meeting. A large cast of cloud users and providers have also provided insights based on early drafts of this post.

In what follows I want to add both breadth and depth to what cloud native is, and what it is contrasted against.

The generic cloud & Software as a Service

As a preview, you can develop software to run on computers (which we have done for over 70 years now), or you can develop software that can provision its own cloud services to run on. We call the latter way ‘cloud native’. In the end, both ways end up running software on computers of course (where else). But there is a huge difference in where those computers are, who pays for them, who owns them, and how the software ends up running on those computers.

Most (especially smaller) organizations do not directly procure technical cloud services. In other words, they do not rent servers, databases or deploy software builds. Most organizations these days consume a lot of Software as a Service (SaaS) though. The most famous example is likely Microsoft 365, which gives you email, word processing and general office productivity tooling, all hosted on the Microsoft cloud.

Most SaaS products are similarly hosted in the big hyperscaler clouds. Or they move there over time.

Larger companies will not only consume SaaS but will also do business directly with one of the hyperscaler clouds: Azure (Microsoft), AWS (Amazon) and GCP (Google). These three American hyperscalers cover almost the entire market for “cloud-native computing services”. Oracle is also active.

The rest of this article focuses on the difference between hyperscalers & existing hosting/service providers, and how the hyperscalers came to absolutely dominate the market.

And I also include a stern warning for not-quite cloud-native providers to get their act together.

The essence of the gap

There are more or less four places to get your computers or services from:

The big American ‘hyperscaler’ clouds provide fully automated access to hundreds of advanced services, plus servers, storage and bandwidth. These clouds enable cloud native development. Many of these services are very good, but pricing can be unpredictable, incredibly complex, opaque, and hugely expensive. Also, these clouds are (in practice) not interchangeable - developers tend to lock in to a specific one.
There are a few players outside “the big three” that provide fully automated access to a core set of essential cloud native services, and also provide servers, storage and bandwidth, often at lower prices. These are at least 100 times smaller than the larger players.
There are many places where you can get automated access to servers, storage and bandwidth, often at very attractive and predictable prices. Many of these “hosters” (some of them very large) will also call themselves cloud providers, and they will indeed often offer some of the essential cloud native services, but not enough of them, or not reliably enough.
And finally there are internal company departments and external providers where you have to fill out forms, get on the phone, send email, and then wait perhaps very long to get access to a server or a database service. However, such external places may be much more a “partner” than a service provider. It is not all bad.

To many readers, the last model may sound extremely outdated, but based on ample feedback on drafts of this post and previous post, the “infrastructure department” is still going strong in many places, and often as disappointing as ever. Many external IT service providers also still operate on this model. Call us, and we’ll make the right things happen.

The perception: our own infrastructure is/was so terrible, we must go hyperscaler

These days there is the perception that the hyperscaler clouds are the only way you can still run services. This is (mostly) true if you’ve decided that cloud native is the only game in town. And this is quite an understandable position if you juxtapose the autoprovisioning rapid world of hyperscaler clouds with (the memories of) an oldfashioned infrastructure or database department.

In many places, if you need a (virtual) server, this is (or was) quite the procedure. You have to convince your IT department you actually need a server, and they’ll haggle with you over how much resources you get (memory, storage). Furthermore, there are also internal “political” considerations. Why do you want to run your own database server, we have a database group that can do that for you. Also we already run a web server, why not use that.

When people sing the praises of cloud native, this is the background to their song. This old corporate world is just ridiculous to them. And especially for development purposes, they are completely right.

In contrast, with cloud native development, power has somewhat uneasily shifted to developers, who now get to script just how much computing resources will be (automatically) procured to run their software. This has not traditionally been the role of software developers. Interestingly enough, the software people may now emergently end up using a lot of expensive resources, but programmers do not have profit & loss responsibility.

This creates an odd dynamic. Where inefficient programming used to result in a non-working overloaded service, the service will now eat up ever more money until it performs well enough.

Nevertheless, if you have ever experienced servers and services that are available to your spec, within minutes and without political/administrative hassle, you never ever want to go back to the old way of doing things. The idea of having to wait four months for a virtual server is just impossible to ponder anymore. A joke. Yet this is often what the cloud is compared against.

The hyperscalers

A hyperscaler has a lot going for it. For example, it is never a career risk. You’re doing business with a 100 billion revenue company. Everyone does. And as noted above, almost anyone will do better at delivering servers than your old way of doing things.

Interestingly enough, while the big three get treated as a block, in practice, developers tend do develop against a specific hyperscaler provider, like AWS, Google or Azure (Microsoft). Trying to get them to shift to one of the other two is quite the battle.

For my earlier post, proof-readers and I came up with a set of essential cloud-native services:

S3/Google Cloud Storage/Azure Blob Storage
Solid and reliable managed Kubernetes
Infrastructure as Code, Terraform/OpenTofu/CloudFormation
Identity and Access Management (IAM)
Managed databases (“just a database” and to some extent Aurora, Spanner)

Now, if you offer these, you do not magically turn into a cloud native compatible service provider. You also need the scale and reputation, for example.

However, what is absolutely sure is that if you do NOT cover these essential services extremely well, you are not a player in the cloud native market.

Hosting providers

Although it might appear that the whole world has gone cloud native, and that everything now runs on cloud services, this is not true by a very large margin.

Tons of solutions, I’d even say a majority, run on (rented) servers. Many of these servers have been “lifted and shifted” from company datacenters to hosting providers. Hosting providers include the hyperscalers, who will also happily rent whole servers to you.

There are lots of places that also offer this service, and they are very good at it. So good that some American users flock to European providers, because not only is the service good, it is also vastly cheaper than renting computing power over at the big hyperscalers. This is particularly true for bandwidth and storage costs. You can now also pay in dollars over at very German Hetzner, for example.

Many hosting providers will also offer some cloud (native) services, and this leads to some confusion. If you come to Leaseweb expecting to find “an AWS”, you’ll be sorely disappointed. There is no IAM. There is no database as a service. There is limited S3-like object storage, but to get it, it says “Contact our Sales Team”.

Despite this, if you need big servers, big bandwidth, big storage, these are the places to go to.

Non-hyperscaler Cloud-native providers

There is a pretty small number of these (at least in Europe):

Exoscale, Swiss/Austrian, owned through intermediates by A1 Telekom
Upcloud, founded in Finland, 10,000+ customers, “The best cloud you’ve never heard of”.
Safespring, Nordic public Cloud Platform

It might be tempting to list OVHCloud and Scaleway here, but lots of people (including me) have had terrible experiences trying to use their cloud features (beyond renting servers).

Old IT typically took weeks or months to provision servers or generally capacity. There are lots of (bad) reasons for this, which we’ll get to. This problem with old IT is often one of the largest drivers of change. However, fixing this does not mean you have to go “full cloud native”.

Cloud operations represent a huge transfer of power from centralised corporate bureaucracy to developers and “devops” people, who can now (at least out of the box) consume resources without administrative overhead or management control.

Finally, cloud users do not just want any provider of services. They want one that has the old status of “No one ever got fired for buying IBM”. This rules out many promising startups that can promise a lot of things, except having existed for a decade already.

Cloud problems

Readers also came with a lot of feedback how not everything in cloud native land is good. Specifically, it appears:

Cloud native costs are a total mystery and lead to large budget problems in organizations not setup for surprise billing. It is difficult to reverse engineer where costs are coming from, and it is all too easy for developers to claim lots of cloud resources, and not release them
In general, cloud complexity is getting out of hand, there are so many ways to get things done, and there are huge numbers of parameters that can be set. Cloud is not “simpler” than the old IT systems.
Developers are very attached to specific cloud providers and even if it makes budgetary sense are not in a mood to switch to another cloud
There are wide misunderstandings on who is responsible for what in the cloud. Specifically, many people not actually operating things are convinced that the cloud comes with security services or even guarantees. This is mostly not the case.

Traditional server operations

Production use may be different. Corporate IT may have been slow in delivering stuff, but companies can still have (security) policies they wish to enforce. In addition, organizations have a justified interest in knowing how much money they spend, and on what. Current cloud practice is showing that costs quickly get out of hand, and companies often have tons no longer necessary cloud infrastructure around, that still costs a lot of money.

Any evaluation of cloud-native, under any useful definition, will have to include an evaluation of this massive shift of power away from corporate IT. This shift may be desirable if bureaucracy had gotten out of hand, but it should be an active decision to do away with all this control/management. Alternatively, there is now a nascent industry that tries to help you at least manage the cost of your cloud usage. This could enable you to make the cloud about as painful as corporate IT used to be.

As an aside, many companies may only be too happy to get rid of their physical server operations. Running any kind of datacenter is hard work, and full of things you can mess up easily enough. And when it breaks, stuff might actually be on fire. Running a DC also comes with ever more environmental/climate compliance work.

I can’t overstate how big a factor this is in the ‘cloud native’ discussion - it is not all about object storage and kubernetes, a ton of it is about instantly getting any kind of resources without spending days/weeks/months on internal procedures.

A cloud partner must not be a career risk

Based on lots of feedback, it appears there are some ambitious providers in Europe that do appear to offer a reasonably complete minimal cloud native experience, for example:

Exoscale, Swiss/Austrian, owned through intermediates by A1 Telekom
Upcloud, founded in Finland, 10,000+ customers, “The best cloud you’ve never heard of”.
Safespring, Nordic public Cloud Platform

It might be tempting to list OVHCloud and Scaleway here, but lots of people (including me) have had terrible experiences trying to use their cloud features (beyond renting servers).

The three modes

These days organizations provide services based roughly on three models:

The supermarket model. You buy or rent servers and storage somewhere, and you run your software on top of that. This entails running your own database, object store, logging, monitoring etc. You have to cook for yourself. Except for the absolute basics, certification is typically your problem. This is the traditional, pre-cloud, model.
The catering model. You get a company to assemble/operate the services you need, which they run on servers they buy or rent. Your software runs on top of their services. The company may be specialised in your niche (say, medical data), and offer the relevant certifications.
The restaurant model. You consume standard services (‘cloud native’) from a large cloud provider, and all your stuff runs on top of these services (which can end up being surprisingly expensive). You are a guest/customer in their environment. They have many certifications already, but perhaps not the one you need in which case you are out of luck.

Interestingly, the large cloud providers can help you with 1 and 3. But unless you are exceptionally large, they won’t do any special catering for you. You get the standard stuff or nothing (compare McDonald’s, where you won’t get very far asking for a medium rare burger).

Industry has meanwhile overwhelmingly decided that they want to run all their stuff based on the cloud services from 3. Governments are following suit. This means that in Europe we have a big problem - we aren’t delivering what people have decided they want to base everything on.

It is also good to note that in all these three modes there is software that runs on services that runs on hardware. The main difference is who maintains the software and the hardware and who is responsible for what. Almost all IT services could be run on all three models, and it is nonsense to say there are things that only large cloud providers can do.

It is however true that the large scale ‘restaurant’ cloud providers offer many services you can simply click on, which would require custom work under models 1 and 2.

Meanwhile, it is also true that running a complex service will always be complex and the cloud shared responsibility model means that even cloud native customers can’t simply make everything “the cloud’s problem”.

Only the supermarket model allows an operator to truly own their services, and sometimes this is necessary. Think of the infrastructure of intelligence agencies, or the systems that maintain the population registry or issue passports. Although industry has decided that everything should be ‘cloud native’ and run on big clouds, this is not going to be the entire future. It is therefore necessary to maintain independent skills too.

The choice of model

The big cloud providers (model number 3) offer a whole ecosystem that is very seductive. Things that run on this ecosystem sometimes go by the name ‘cloud native’, which is a marketing term for software that runs on top of (proprietary) clouds instead of on computers directly.

I can’t stress enough how important it is to realise that “the cloud” is not just a bunch of computers. Deeply technical people do look at the cloud like this, and will often proudly state that “there is no cloud, only other people’s computers”. And while this is true in a technical sense, to a whole generation of cloud users this is an inane statement.

These people have chosen to run their software on top of infrastructure and services, and they don’t then have to worry about that underlying layer of machinery and code. Their software starts based on infinite storage (S3 or workalikes, Amazon EFS) and managed databases (RDS, Aurora, Spanner). They combine services using fine-grained Identity and Access Management (IAM), services that can interoperate and be found on marketplaces. They provision computing power as a set of rules, and not as servers you have to buy and rack. Or they take a step further and make their cloud of choice auto-provision capacity.

You really can order anything from the AWS Solutions Library

In contrast to this, there are also still lots of people that run their software and services on top of actual hardware, often virtual, often rented, but sometimes also on their own “bare metal” (model 1). These people have to do more work themselves, but in return they get far lower prices which are also far more predictable. The best people in this category can achieve an order of magnitude cost reduction compared to “the cloud”. In addition, they can also claim to fully own their service, and not depend on companies continents away operating under weak privacy regimes. It is pretty cool (and sometimes necessary) to have your own infrastructure that is really yours.

As noted above - these two groups of people are very different. The “own infrastructure” folks heap scorn on the cloud native crowd, and meanwhile tries to get the cloud people to see the merits of their fully owned infrastructure. This does not go down well. The cloud native folk meanwhile often talk about the own infrastructure field as cavemen at best, people living in the previous century. ‘Cloud first!’.

As noted above, an apt comparison is expecting a restaurant but walking into a supermarket. Sure they have everything you need to make a meal, but in terms of stuff you can eat right now, you get to pick from a bunch of sandwiches or maybe some salads. It is a pretty disappointing experience if you were hoping to get a restaurant meal.

Meanwhile the reverse experience is just as bad - a restaurant only offers set meals at TREMENDOUS prices compared to the grocery store. If you wanted a supermarket, this is not what you came for and it feels pretty bad to be ripped off (from your perspective).

Trying to get cloud native people to run on your bare metal hardware is a nearly impossible struggle. They have made their choice, and they’ve become good at it. They know where the skeletons are hidden, they have the tools to make it all shine. The folks that run their own servers meanwhile can also run stuff on “cloud servers”, but as they do so they complain loudly about the cost and performance. They meanwhile often fail to factor in their own personnel cost (“I don’t cost any money”).

The alienation between these groups is a thing to behold, and it might be in large part why we’ve ended up with this continental divide where none of the big clouds come from Europe. Providers here have mostly focused on infrastructure, and have not really committed to becoming cloud native providers (although some put up a show to that effect).

The chasm

In Europe we have woken up to the fact that we have no big cloud providers at all, and this is a legitimate problem. Mutual dependencies are great, but total dependency is not. Ask how the rest of the world is feeling about all high-end chip / lithography equipment coming from a small area in Europe.

Where essential chip / lithography equipment is made (map from openstreetmap.org)

The first step in addressing this issue is to admit it. And this is where we run into a problem. As described above, the people that are ‘cloud native’ have well-formed and reasonable expectations of what a cloud provider should offer. Meanwhile, it almost appears as if European contenders aren’t even aware (in detail) of these expectations.

Many of them sound like supermarkets claiming to be restaurants because they also sell the ingredients that restaurants use. But this is missing the point. It is not enough to mount a ‘RESTAURANT’ sign on your supermarket.

There are also providers do offer some of the essential cloud services, but then miss out on some key ingredients that would complete the picture. Let’s take a look at what services we are talking about.

The stack of services

Both the hyperscaler clouds and smaller players offer virtual servers and storage. For these services there is a well functioning market where there are many players to choose from. If you want to rent a server, even a physical one, you can either go to a large cloud provider or one of the many hosters in Europe. Interestingly, this basic service appears to form a huge part of Amazon, Azure and Google revenues as companies “lift and shift” their former data centers into the “cloud”.

Once we look at even the most basic of cloud services (beyond servers), this picture starts to change dramatically. Based on two online surveys I held (with 200 responses), plus ample feedback on my previous cloud blog post, it appears these are universal cloud requirements:

“S3”

AWS/GCP/Azure all offer infinitely scaling object stores, going by the names of S3/Google Cloud Storage/Azure Blob Storage respectively.

In these stores you can store data, files, images, movies, at different levels of redundancy, availability, geographic spread, encryption and performance.

While it sounds deceptively simple, this kind of storage has proven to be absolutely vital infrastructure and also surprisingly difficult to implement well. A lot of work has gone into making the various S3 as good as they are.

Outside of the big US clouds, almost no one offers something comparable out of the box. There are some geographically limited options here and there, but nothing that comes close in terms of reliability/availability and features.

I’ve spoken to several providers who disagreed they don’t offer S3 because they could definitely set something up if a customer asked. This is the catering model. I’ve heard similar remarks about other services. But it turns out that the cloud customer does not want something special made for them. They want a service you are proudly advertising that already has many users do the same thing they want to be doing.

Managed Kubernetes

Invented by Google, Kubernetes promises “computing power as a resource”, where you create services that automatically organize as much capacity as is required to deliver your services to your current audience. So if you are a tax agency and filing season arrives, your Kubernetes-based setup will not be overwhelmed, it will simply recruit more resources. And once done, these resources (computers) will be released again. It is a very seductive concept, where you’d only pay for the computing power you use.

Kubernetes is freely available open source. It is however surprisingly hard to operate, and the consensus is that this is best left to professionals, such as very large cloud providers.

Software people can then focus on making sure their services integrate well with a managed Kubernetes environment.

Kubernetes is not all roses, but it is what the industry is standardizing upon, and Google, Amazon and Microsoft all offer it, and you must also deliver that to be taken seriously.

As far as I can tell there are no credible European players of any scale that offer a reliable equivalent, with (I am told) the possible exception of Upcloud.

Infrastructure as Code

Like Kubernetes this is a very seductive idea - instead of configuring servers manually and setting up their network connections, domain names and storage, why not have a piece of code that makes this all happen?

Software like Terraform/OpenTofu makes this very easy. Developers can script what their software requires in terms of servers. To deploy, the infrastructure code communicates with the cloud provider and there the right resources get spun up and configured.

Incidentally, this can lead to huge bills - whereas previously programmers had to make do with the hardware that was available, you can now simply add some more or larger servers.

But still, once you are used to getting infrastructure configured through code, having to email someone to get a new server (or service) starts to feel positively medieval.

From what I hear, there are providers in Europe that have support for being provisioned through Terraform/OpenTofu, but I also hear that the quality and availability of servers is so so.

Identity and Access Management (IAM)

When procuring many services, these need to talk to each other and get limited access to data. For example a component that checks addresses / post codes only needs to see these details of a user, and not all their other details. Users also need to only have access to their own data.

Such access not only needs to be limited in scope but also in time - we don’t want our partner provider to have infinite access to a service, for example.

IAM is often overlooked since it is not a visible feature, and it is no fun to work with. But only through very well working IAM is it possible to combine services together securely. If you take a look at this Google document you can see how many aspects IAM has to deal with.

Well-functioning IAM is therefore an absolute requirement for offering cloud native services, and it appears the European providers generally only have something very basic on the menu.

Managed databases (RDS, Aurora, Spanner..)

Databases are in one sense pretty commodity off the shelf technology these days. They can quickly become complicated though if you want to get worldwide replication, with the ability to accept updates on multiple database instances simultaneously, and be resilient to failures.

The large cloud providers all offer a variety of database solutions, ranging from simple (“a database server in the cloud”) to extremely advanced, like for example, Google’s Spanner which probably requires a hundred person team at least to operate even for small scale use. And this is after spending the thousand person years required to develop something like this.

Not everyone needs such a magical database though. It is entirely possible to procure a good hosted database in Europe, but you might have to shop around, and this is typically not what cloud native customers want to do. It needs to come with the solution.

Some further notes

Pricing, complexity

It is good to note that if we liken the big cloud providers to restaurants, their menus come with very confusing pricing, like $.02 per bite per day. “Surprise billing” is a big fact of life in the cloud. Even when pricing looks reasonable, it often turns out that your service requires enabling three other servers that somehow deliver a stunning invoice at the end of the month.

Much of this surprise billing is an artifact of the by now astounding complexity of the offerings of the big clouds. I have very frequently heard from cloud users that this is getting borderline ridiculous.

I recently tried to deploy a simple computer program on an Amazon serverless environment and it is indeed a three ring circus to set that up.

Quality

It pains me to say this, but there are providers here in Europe that on the face of it appear to offer Kubernetes, IAM etc, but where you find that if you try these services it is all not good. I’m not going to name names here, but know that some services simply aren’t that great. Do test. This also goes by the way for Microsoft Azure which has been plagued by security and availability concerns.

A preview of what to do

Since Europe has no large scale cloud providers that offer the ‘restaurant model’, we should meanwhile attempt to make good use of our remaining providers than can cater great services for us. These companies exist and operate tons of infrastructure for us. Companies like Hetzner, Leaseweb, OVH, Scaleway, team.blue etc have significant hardware capabilities and there are tons of IT suppliers that could do great things there. Let us not throw that away.

To be very clear, if someone claims that “no one in Europe” can support a specific IT system, this is almost never true. What they mean is, if I want to operate this in a specific model (“cloud native”), THEN I can’t find anyone.

However, the infrastructure providers where you can currently procure servers and storage should stop claiming to also be large scale cloud providers, and take a harsh look at what they are offering versus what ‘cloud native’ customers reasonably expect. They should either commit to being a cloud native provider, or specialize in hosting servers and storage.

If these providers plug the largest gaps, they can again start to compete for cloud native business. For reasons we’ll go over below this will still be an uphill battle, but as long as basic services aren’t being offered with sufficient reliability, the complaining about lack of business sounds a bit hollow.

Plugging the largest gaps will be work, but there is already (open source) software available for most “must have” services. Yet, this software needs to be scaled up and made ready for commercial, multi-tenant, use. The good news though is that a lot of the talent that writes this kind of software is actually here in Europe. We definitely have the people. And through industry cooperation, it should be possible to bring a lot of love to the software we need.

From a competition standpoint, it should not be a problem if lots of European hosting companies combine forces to create or enhance the software that is required to offer these services. All these companies together are still not very big compared to Microsoft.

A great analogy may also be the coordination that happens in the world of semiconductors where even competitors align their roadmaps so that the industry can build on a common set of future technologies.

Meanwhile, governments and the EU could commit to engaging with European providers, hosting and “catering”, to keep the industry viable. Given some of the special requirements that governments have with respect to owning their data, and not falling under American espionage regimes, they are in a wonderful position to put out tenders that favour local economic development even as they deliver sovereign services.

Even committing to moving a tiny percentage of government IT spending to newly formed local clouds would provide a huge impact, far larger than providing subsidies.

For more detail do read on.

Summarising

A stark choice facing Europe - industry and even our own governments are standardising on cloud native solutions, yet none of our own companies are delivering the services you need for that. Yet many of them act as if they do, and are not visibly working on addressing that. There is talk of large monetary investments, but there is little concrete action in terms of turning up new services and improving the reliability of existing ones.

Yet there is a reasonable set of services that would satisfy many cloud needs. If European operators combine forces to create software to plug these gaps, they could become new of still somewhat limited players in the cloud native world.

Industry will likely continue to pick the most popular cloud providers with the most features. But governments with sovereignty requirements (and people feeling very European) would be able to pick a European cloud native provider, if only this reasonable, core, set of services were reliably available.

And meanwhile, we have the thousands of IT companies that can cater IT services, so customers (like governments) don’t have to run all services themselves, and could still provide services on top of managed services.

XXX

Additionally you might find out after all the administrative stuff that there isn’t even any server capacity available, and that new servers are expected four months from now, depending on the Q3 financial figures (I’ve experienced this multiple times). And if you manage to get a server, it runs a corporate Linux version that is too ancient to run the things you want to run. Also you don’t get root (administrative rights), let alone IPv6.

Meanwhile, developers can within a few short minutes rent virtual servers on the outside, paid by the hour, with any operating system you want. When this is done to evade corporate IT departments or policies, this is part of what is called “shadow IT”.

As of 2024 virtual servers and services should be available without friction (at least for development and testing purposes).