tl;dr: Microsoft and other email security scanners will visit the links in email you transmit, and run the JavaScript in those links, including calls that lead to POSTs going out. This used to be unacceptable, since POSTs have side effects. Yet here we are. This breaks even somewhat sophisticated single-use sign-on / email confirmation messages. Read on for how to deal with this, and some thoughts on how we should treat gatekeepers like Microsoft that can randomly break things & get away with it.

On my Mastodon profile I have pinned this quote:

“Writing has been called the process by which you find out you don’t know what you are talking about. Actually doing stuff meanwhile is the process by which you find out you also did not know what you were writing about.” - from A 2024 Plea for Lean Software (with running code)

Recently I’ve been actually doing some things again, namely, building a site that monitors Dutch parliament where users can configure monitors that inform them of parliamentary happenings. This involves having people sign up, sending email, getting users to log in and out etc.

From this I learned yesterday that a user could no longer perform a password-less sign-on to my site, instead always being told that the single-use link had been used already. How come?

Cyber norms

“Cyber norms” on the Internet have always been shifting. When I started online in 1992, it was still the case that ‘port numbers below 1024’ were special. Only root (UNIX administrator) users could operate on such a port, and some software would only respond to packets coming from such low port numbers, since these were more trusted. These days, the administrative restrictions on privileged ports remain, but it is not considered a useful security barrier anymore. Anyone can run on port 123.

Similarly, and you may find this hard to believe, but back when the first spam filters were invented, there were arduous discussions among mail server operators if they should even be allowed to do that, even if users wanted it. Was it ethical to have algorithms study email & possibly block transmission? Blocking and inspecting email was considered highly controversial in the 1990s.

Also, software “phoning home” was regarded as reprehensible, even for such things as checking the availability of new versions with possible security updates. Over time, we found out that checking for security updates was a very good thing. However, these days commonly used software “phones home” to the tune of dozens of megabytes per day, which we call “telemetry” but it is actually spying.

The problem also is that no one announces it when they transgress norms and establish new de-facto boundaries of what is acceptable. We just find out about it “as we go along”.

HTTP

Seasoned developers know that HTTP GET requests need to be idempotent. Or in other words, a GET request should not alter the state of anything. Browsers and basically everyone in the chain should be free to do a GET request to a URL, and it should not matter.

If you want to actually change a state or make something happen that is not idempotent, you should use a POST request. There is therefore this norm that no software should be doing or repeating POSTs without making sure that this is what the user wanted.

Meanwhile, miscreants like to spread malware. We’re fine with folks scanning email attachments for badness. Over time, it also became OK for software to visit links in email to find out what was behind them. This involved GET requests, so this was acceptable.

This did already break a few workflows over the past few years. Single-use magic (sign-on) links got “consumed”. Once the user saw the email & they clicked on the sign-on link, the link would be dead already. Microsoft had already automatically eaten the single-use link for you, before you could use it.

This, by the way, also massively confuses some phishing tests - making it look like all users click on all links. However, recently, it has been Microsoft that is doing the “clicking”.

However, we could argue that these people had it coming - GET requests should not have side effects, and Microsoft was in their rights to perform these requests for security purposes. To prevent this, wise operators of single-use sign-on links make the actual sign-on be a POST, something scanners aren’t supposed to do, for reasons outlined above.

Lately however, operators of services (like me) have found that the links they send to users to sign in, or confirm their accounts, actually lead to POST requests coming from Microsoft. Microsoft’s security scanning will now not only visit the links you mail out, they will also run the JavaScript on your page, and will also send out any POSTs generated by that JavaScript:

The first line is the actual user. On the second line, 40.94.87.80 is a Microsoft IP address. Their POST receives a 55 byte response. The third line is the user again, getting only a 28 byte response (a failure, in this case). Note the fake user agent also on line 2.

Now, this is a strong violation of the norm we had: you can GET all you want, but POSTs have side-effects, and it is dangerous to do them “to see what happens”. Not only is Microsoft transgressing an important norm here, but as far as I can tell, they also didn’t tell anyone about this or warn us in any way.

A charitable read of this is that this is OK since Microsoft performed a GET request to get our link, and our link then caused a POST to be performed. So you could argue that it is us that was in the wrong, we caused side-effects based on a GET. There is maybe some truth to this.

However, the new ‘cyber norm’ now appears to be: we’ll visit your page, run your JavaScript, and perform any POSTs requested by it. It appears that for now, the security software will not virtually click on an actual “sign-up” button to see what happens then. But this is probably only a matter of time.

We may wonder also what fun a determined hacker could have with Microsoft running random bits of JavaScript on their servers and allowing these to talk to the world (or to Microsoft itself even).

What to do?

If you send any kind of sign-on or email confirmation link, you must assume that security scanners will eventually do anything they can to execute that sign-on or confirmation.

In other words, your software can no longer have links that work just once. You must deal with Microsoft (and other security scanners) finishing an account sign-on on behalf of your user.

So what your site must do is deal with a user finishing their sign-up or email confirmation a second (or third..) time. The earlier confirmations were fake. Now, some operators have attempted to put a captcha on the link to prevent such fake confirmations but that will likely 1) annoy your users and 2) get you on the Microsoft naughty list, from which there is no escaping.

Some might argue that these confirmation / single-use sign-on links are no good anyhow, but for now, they are what we have to get people to sign up for services.

Final thoughts

It would be nice if the big dominant operators were forced to be more transparent about the norms they transgress. I understand the security dilemma, trust me, I really do. You can’t tell the bad people of the world your exact security practices.

But you can’t also silently break all kinds of things.

The EU Digital Markets Act recognizes that some players are “designated gatekeepers”, with such market force that they must adhere to additional rules. This currently includes Alphabet, Amazon, Apple, ByteDance, Meta and Microsoft.

These gatekeepers should be more accountable about what they do online, and I hope we can make that happen.

Because who knows what they might break next, without giving us time to prepare.