The State of Galileo as seen by an outsider

By bert hubert / bert@hubertnet.nl

In this post I want to document some mostly non-technical aspects of the EU’s Galileo Satellite Navigation Network. I will also share some observations on how things are going, and perhaps how they could be improved.

NOTE: A shorter version of this article, focusing only on the July outage, can be found here.

Since the week-long outage in July I’ve been fascinated by Galileo and, together with a wonderful crew of developers, experts and receiver operators, have learned so much about what I now know are called ‘Global Navigation Satellite Systems’ or GNSS.

I wrote a technical summary in a previous post called GPS, Galileo & More: How do they work & what happened during the big outage? I also turned this into a one hour interactive school lesson for 10-11 year olds, which can be found here.

My interest in Galileo started during the July outage when I could find no information on what was going on. It then proved possible using affordable receivers to build a credible Galileo monitoring centre, and this quickly gained a lot of support world-wide, which has now brought 24/7 coverage of not only Galileo but also GPS, GLONASS and BeiDou.

This has all been a wonderful experience. From what I hear, and backed up by the visitor statistics, our galmon.eu open source project is considered a valuable resource, especially when combined with the Grafana dashboards and the Jupyter/Pandas/Matplotlib-based reports. In addition, several Galileo suppliers or partners now run their own copy of our software. If I compare our feature set to that sought by upcoming Galileo tenders, I know we already fulfill many of the requirements.

In finding out how galmon.eu’s open source code & data could be made useful for the Galileo project, I have also learned how Galileo gets run from day to day, or at least, the outline of how this is supposed to happen. This took outrageously long to figure out, and I think it would be helpful if it were more widely known.

Please note that this is a rather critical article. But I truly want Galileo to succeed and I know almost everyone involved shares this ideal. I wrote this article not to offend but to share how things are going, what is being promised and what I perceive current problems to be. And I do this in the hope that this might be helpful.

Here goes.

Note: This article is my personal opinion and not that of the large & wonderful cast of Galmon.eu receiver operators, open source contributors and supporters. In addition, I want to thank the many people that helped me understand GNSS and/or proofread & improved drafts of this article, but they are not to blame for the result!

Galileo

Galileo is a civilian programme managed by the European Commission on behalf of the European Union. This is unlike the American GPS, the Russian GLONASS and Chinese BeiDou projects which all at least have a military component, or are in fact completely military with civilians hitching a free ride.

Galileo is however still considered to have military/safety applications, and has modes intended for government-only use, so to work on Galileo rightfully requires security clearances and hassle.

The European Commission does not develop, order and launch satellites itself, it has left that to the European Space Agency (ESA). ESA is not an EU agency, as it has non-EU members (like Switzerland and Norway, and even Canada partially), and not all EU member states are actually part of ESA. There are however very close ties between the EU and ESA, expressed both in terms of a gigantic budget, framework agreements and a shared European Space Policy.

On 15 December 2016, Galileo started offering Initial Operational Capability (IOC) services, which is a very narrowly defined mode of operation which states for example that accurate positioning will be available ‘at least 77% of the time’. Note that this is not a typo of ‘99%’ (“two nines”), the goal is actually 77%.

This will change when ‘Full Operational Capability’ is achieved, likely somewhere beyond 2020 (but see below). But for now, Galileo is best seen as “something to use in conjunction with GPS to enhance coverage and accuracy”, a configuration used (for example) by over a billion phones.

ESA

ESA has developed and procured the satellites and their launches, the latter from Arianespace. The first four ‘validation’ satellites were built by Astrium GmbH and Thales Alenia Space. Afterwards, 22 production models were ordered from OHB System and Surrey Satellite Technology Limited (an Airbus company).

ESA has a ton of expertise in designing satellites, getting them built and launched. They are also ace operators of satellites.

It is not that everything they do is a success, but ESA is at least stocked with space experts.

EU GNSS Agency

ESA may develop and get things launched, but operational control of Galileo is vested in the European GNSS Agency, hosted in Prague. It is typically called ’the GSA’ after its old name, the European GNSS Supervisory Authority.

The GSA itself however does not operate Galileo. As far as I can tell, the GSA is predominantly a procurement and contractual powerhouse.

Galileo is actually run by the ‘GSOp’ or the Galileo Service Operator, for which a tender went out in early 2017. It was won by a special purpose company called Spaceopal GmbH, more about which later.

In addition, the GSA has launched a Galileo Reference Centre (GRC), which is designed to be an independent monitor of Galileo performance. This too was tendered, and this independent monitoring is supplied by the Spanish space, IT and defense company GMV, although the GRC is physically in The Netherlands.

The GSA publishes public quarterly performance reports which are very pleasing to the eye and where everything works well. No report has yet been released that covers the big July 2019 outage. The Q2 one however is available, and it is the first to align with the new May 2019 Service Definition Document.

GSOp

The GSOp contract was won by Spaceopal (’We make Galileo fly’). Spaceopal was already running Galileo before that as prime contractor, and in 2017 it extended this after a public tender. Spaceopal is a marriage of:

• the commercial spaceflight arm (?) of the German aeronautical agency, the “DLR Gesellschaft für Raumfahrtanwendungen (GfR) mbH”
• Telespazio S.p.A, which itself is a joint venture between:
  • Italian defense giant Leonardo (formerly Finmeccanica)
  • French defense giant Thales

Spaceopal also runs the GNSS Service Center (GSC) which runs a website with information for Galileo users. This is also where you can ask questions, read about service disruptions or observe the constellation status. The GSC is in Spain.

Part of the complexity of this diagram stems from the complicated history of the Galileo project, starting from the ‘Galileo Joint Undertaking’, which envisioned a model where private industry would fully own Galileo. This idea was found not to be workable, but some of the structures from that time have remained.

EGNOS

The GSA also coordinates the European Geostationary Navigation Overlay Service.

EGNOS is an ‘overlay’ on top of GPS (and Galileo a few years down the road). This overlay provides additional guarantees that make it possible for planes to land based on satellite based navigation, which is very cool. The US version of EGNOS is called WAAS and may be better known. EGNOS is operated by European Satellite Services Provider which is owned by several national air traffic control systems. Thales is one of the main suppliers of EGNOS satellite payloads and other infrastructure. Like with Galileo, the design and procurement of EGNOS runs through ESA.

Search and Rescue

A highlight of Galileo is the Search and Rescue component present on most of the Galileo satellites.

When in distress, users can activate a rescue beacon, and that beacon will then transmit a signal to any listening satellites. The Galileo constellation can rapidly pick up and localize such a signal, enabling rescue efforts to commence swiftly, often within minutes. This is a great improvement on earlier beacon systems.

Uniquely, Galileo can also send a return signal to a beacon to let the user know their call for help has been picked up successfully, although this is currently not yet enabled. The galmon project has however observed many test ‘Return Link Messages’, so it does appear work is ongoing.

The Galileo SAR component was successfully demonstrated recently in a great piece of public outreach called ‘Operation Shark Bait’, where using helicopters and ships an actual ‘person over board’ was found and recovered quickly. I was invited to join this demonstration but very sadly could not make it, but as outreach it was highly effective.

It should be noted that the SAR component did not suffer from the July outage and remained available continuously.

The actual day to day work

Running Galileo is 24/7 work. As explained in my earlier technical article, for a GNSS to function, the performance of the atomic clocks in space must be modeled and tracked closely. Simultaneously, the precision on earth will never be better than how precisely we know where the satellites in space are.

Extremely simplified, a GNSS satellite is a machine that beeps exactly once a second, while broadcasting its orbit and atomic clock to decimeter & nanosecond precision.

Achieving such precision is hard work - if the atomic clock of a satellite is behind by 3 nanoseconds, this can look like it being 1 meter further away from the receiver. So which is it? Is the orbit not known precisely enough or is the clock off? Or, heaven forbid, has the timing facility on earth lost the plot - making everything look wrong?

Note: this is what actually happened during the week long outage, more details below

Determining the interlocking behaviour of constellation orbits, space clocks and ground clocks requires iteration & convergence - after an extended bit of the orbit it becomes clear which is off more, the clock or the ’ephemeris’ that describes the orbit.

To achieve the very high accuracy Galileo has promised requires uploading a new ephemeris frequently, as often as a few times per hour.

Currently we often see Galileo satellites broadcast that their position is not known precisely. When we later compare the orbit and clock performance against what independent observers measured, we find that the satellites were mostly just fine - it was something on the ground that lost track of what was going on, leading the satellite to be declared ‘No Accuracy Prediction Available’ (NAPA), effectively disabling its use by receivers (like phones and cars).

Galileo Reference Centre

This is a pretty building right next to ESA’s ESTEC campus in Noordwijk, The Netherlands. It is however not part of ESA.

From a press release:

• The GRC is operated by the GSA: it provides the GSA with an independent means of evaluating the performance of the Galileo Service Operator (GSOp) and the quality of the signals in space.
• The GRC is fully independent of the system and the Galileo Service Operator with respect to both the technical solution and operations

The GRC has been contracted out to the Spanish space, IT and defense company GMV. The contract for operating the GRC is up for renewal and it will be tendered in 2020. The budget is 17.5 million euros for supporting the old software, developing new software & supporting it, plus providing coffee, catering, cleaning and gardening services. The tender can mercifully be split up in multiple parts.


The GRC also creates Galileo performance reports but these are not publicly available (this page is a screenshot from a GRC presentation).

Other components

Various other components of Galileo:

• The Geodetic Reference Service Provider (GRSP): This entity supports the GCC (Galileo Control Center) in realising the Galileo Terrestrial Reference Frame (GTRF), consistently with the international Terrestrial Reference Frame (ITRF).
• The Time Service Provider (TSP): This entity supports the GCC in the realisation of the Galileo System Time (GST) and its alignment to the Coordinated Universal Time (UTC).
• The Galileo Security Monitoring Centre (GSMC): This facility is in charge of monitoring the system security.
• Time and Geodetic Validation Facility (TGVF), operated by ESA
• Galileo Integrity Processing Facility (IPF), supplied by GMV
• Galileo Orbit Synchronization Processing Facility (OSPF), supplied by GMV
• Galileo Flight Dynamics Facility (FDF), supplied by GMV

Current status

The current status of Galileo is not that great. There are 26 satellites in space, 2 of them in botched (elliptical) orbits, 2 of them unavailable, 1 unavailable until further notice. This makes for 21 functional satellites, which is less than the “healthy” number of 24. Anything below 24 causes gaps in coverage, or at least, areas where the positioning accuracy will not be good enough.

As NASA, ESA and others are fond of saying, “space is hard”, and they are not wrong. Space is a harsh environment. GNSS vehicles consist of very delicate components that all have to work just right, and from time to time there are problems. This means that lately, on average, around 5% of the Galileo satellites are unavailable because of minor problems (as described above). Often these problems appear to be ground based and have to do with software on earth and not with things being broken in space.

It appears this software was originally procured by ESA and that any changes have to flow from the original developer, to ESA, to the GSA and thence to Spaceopal for deployment. This may mean that fixing software will take some time, but I am sure they’ll eventually get there.

But this does nothing for the fact that there are currently only 21 functional satellites in space, and that these are not always available. According to the current schedule, this situation will only improve in 2020 when two additional Galileo satellites will be launched, on only the second Ariane 6 flight.

Ariane 6 is the new flag-ship rocket from Arianespace, and being on the second launch almost guarantees there will be delays. There is no set date for the launch right now. Predicting a launch date slip to 2021 is therefore quite realistic.

Meanwhile, out of the 21 functional satellites, one is still from the original prototype batch, launched in 2011. The other one from that batch has already failed it appears.

It is therefore not unrealistic to predict further failures between now and the likely 2021 launch date. And even this launch will only provide two new satellites in a single orbital plane.

From GPS and BeiDou we learn that ‘30’ is a pretty good number of satellites to have in space as this provides for a lot of cushioning in case of failures. Quantity has a quality all of its own.

And what about the week long outage?

During and after the outage, the GSA and others referred to the Service Definition Document (SDD), which specifies the Minimum Performance Levels (MPLs) we can expect from Galileo during the period of initial services. As noted in the article “Lessons to be learned from Galileo Signal Outage” on the Inside GNSS website, this rings hollow:

Eventually, some days into the event, the GSA did post a statement on its website, taking responsibility and even apologizing for the failure. But then it attempted to minimize its import by arguing that, after all, Galileo is still in its “initial services” phase and therefore should not be expected to work without interruption. Among many Galileo’s supporters, that message fell flat.

A major problem during the outage was in fact the communication of what was (not) going on. As the Inside GNSS article correctly notes, no one within Galileo was allowed to communicate anything except the European Commission itself. And the EC is in effect in the worst position to communicate because it is so far removed from operations.

The defense that Galileo was operating as specified in the SDD may in fact be correct. Within any 30 day period, there can be a full 6.9 day outage and Minimum Performance Levels can still be met.

In September an Independent Inquiry Board was formed to study the week long Galileo outage. The plan was to release preliminary findings in October with final recommendations by the end of 2019.

From rumors, we know a report has been written and that it has reached some firm conclusions. It appears a lot was wrong, but the findings are currently EU classified.

Although October has come and gone, the independent board has not published any initial findings so far. As an aside, an industry insider has told me that the best way to tell how well Galileo is actually doing is to see how late the performance reports are.

Meanwhile, earlier this week Pierre Delsaux, European Commission Deputy Director General in charge of Galileo, reportedly blamed the outage on the mistakes of a single person. Mr Delsaux then went on to compound this terrible statement by adding that the EC had been transparent because the reasons for the outage had been discussed at a conference earlier this year.

Accident details

It is indeed true that a presentation was held in Florida where details were shared with that audience, and by paying $24 we can download the presentation that was held there. From the slides, we learn that the outage stemmed from a failure in the system that determines the satellite orbits and clock parameters, which are normally uploaded to the satellites many times per day.

The outage in the ephemeris provisioning happened because simultaneously:

• The backup system was not available
• New equipment was being deployed and mishandled during an upgrade exercise
• There was an anomaly in the Galileo system reference time system
• Which was then also in a non-normal configuration

In a way this is very good news - if a major outage needed many things to go wrong at the same time, that means it was not theoretically an accident waiting to happen. We can of course wonder why upgrade work was happening while the backup site was not available. It should be noted that in general, it often happens that rarely used backup facilities are not immediately able to take over in the face of failure.

After the incident started, it took a while to determine what was going one before operations could be restarted, but by that time, the constellation had already drifted too far from a known state that the status of the orbits and clocks could be converged upon quickly. If the backup site had been live, it would have been a great place to restart from, since it presumably would have been in a converged state already.

As noted earlier in this page, determining the orbits and clock configurations is a complex task, where errors in the clocks, or the individual clock drift rates, can look just like errors in the orbit details. Over time, by tracking orbits and clocks, such ambiguities can be resolved, and the system then converges on a known state. From that point on, that state can then be tracked closely.

After sufficient downtime however, such convergence is not a given, and it appears the Galileo constellation had to perform a ‘cold boot’ from first principles. The presentation at the ION conference also notes that due to quality concerns, no shortcuts are taken in such a cold boot.

GNSS ephemeris and clock convergence is a dark art only practiced by a few systems on the planet. Despite the fact that this should all never have happened, I am quite impressed that it was possible to restart the system all over again in only a few days.

Also please note that while this presentation does confirm an operator error was involved, the actual reality is that Pierre Delsaux’s reported claim that the incident was due to a single person making a mistake vastly understates the many reasons for the prolonged outage.

Galileo promotions

Despite Galileo officially being within an extremely limited performance guarantee (‘initial services’), which allows week long downtimes, promotion of Galileo is going strong. The EU is justifiably proud of having achieved a mostly functioning GNSS, and can rightfully celebrate having a billion receivers using Galileo. It is an achievement, especially getting all chipset manufacturers on board in such a short timeframe.

Beyond this however, Galileo is currently also being promoted as being more accurate than GPS, as outlined in the ‘When close isn’t enough, use Galileo’ campaign.

This may be true under some circumstances, especially when comparing “original GPS” performance to a modern dual band Galileo receiver. But given the spotty ongoing Galileo service, it will also very often not be true.

ESA and EU relations, governance

As it stands, the EU appears to be unhappy with its relation with ESA. ESA, as noted, has built Galileo and is working on Galileo’s next generation. ESA however is not operating the current satellites.

The EU GNSS Agency meanwhile is poised to become the ‘EUSA’, the actual EU Space Agency, unlike ESA, which is the “European” Space Agency. The plan is to rename the GSA to the European Union Agency for the Space Programme.

The EU runs another space programme, one which is very successful, called Copernicus which studies the environment and climate. This too was built by ESA but is actually owned by the European Commission, and would then move to the ‘EUASP’.

ESA is of course not happy with the creation of a competing space agency, and a small battle is ongoing. In addition, the new Von der Leyen European Commission will move Galileo from the Directorate General for Growth and Development to the new DG “Defence Industry and Space”, and ESA is not happy about this either. This last link is also a good explanation of ESA governance & the potential impact of Brexit.

These upheavals will likely not help with any of Galileo’s challenges in the short term.

My own feelings

During the big Galileo outage in July, I got all fired up about this exciting project. I learned that very smart people had built Galileo, and on meeting them, I found them to be highly involved and motivated.

Quickly the galmon project gathered volunteers and we gained some interest from the people that currently are responsible for Galileo.

I tried to contact the Galileo Reference Centre three times, but they did not get back to me. I thought that having an actually independent open source Galileo monitoring solution might be helpful for their stated mission.

Others higher within the Galileo governance did contact me and very kindly educated me that some of the problems I had spotted weren’t actually problems, but also encouraged me to continue the monitoring project, because it turns out that from time to time we do spot problems that have gone under-reported.

Both Galileo and ESA procure products and services almost exclusively through public procurement. ESA offers a relatively light-weight way of doing so that could possibly be navigated by smaller companies (for tiny projects). The GSA procurement is of the “only apply if you already operate satellites in space” variety.

This could be solved via partnerships, but other rules of the procurement process state that no contact is allowed with the GSA during the process, except through a legal gateway. This effectively rules out smaller companies that do not have a whole legal department to regulate communications, or that actually need a lot of communications to figure out what needs to happen.

Our website galmon.eu clearly shows how well, or not, Galileo is doing. It also keeps track of GPS, GLONASS and BeiDou, so comparisons can be made. Part of our output clearly shows when Galileo has an off-day (or an off-week). This has made our work both helpful and controversial.

Meanwhile the culture around Galileo is not a healthy one. I’ve spoken to quite a number of people and most of them really want to make the project a success, but almost all of them also only want to talk off the record. This goes for people within EU and ESA, but also for suppliers. One department went so far that they would only meet me in an off-site location & not at their offices. I’ve had the best and most productive contact with researchers not actually working (directly) for Galileo.

I’ve been told that Galileo has two main problems: being two satellites down (I would say 6), and the other one being terrible communications. In my somewhat applicable experience, (human) communication problems are generally a symptom of deeper problems and frequently reflect structural deficiencies.

If I had to make a guess, having all kinds of technical expertise spread around, but not concentrated on an empowered GSA would be one such structural deficiency.

It may not be easy to change this - GNSS experts currently have the choice of working for industry (which tends to pay well) or an actual space agency, which may be the preference for ambitious specialists. An organization (necessarily) burdened with a lot of legal overhead is not a very attractive place for engineers to work. Because of these reasons, GSA likely has a hard time attracting seasoned space talent.

In addition, I wonder about the wisdom of tendering almost every component of Galileo. A good contract is no substitute for local expertise, especially if things go wrong. I am also not sure how wise it was to hire one large defense contractor to monitor the performance of another one.

It also appears that GMV (which won the Galileo Reference Centre tender) is partially monitoring its own work, specifically the Orbit Synchronization Processing Facility (OSPF) that is responsible for generating ephemerides. Note that some of the currently observed ‘NAPA’ problems are specifically related to the work of the OSPF.

We may wonder if in hindsight it might have been better for Galileo to operate an independent monitoring capability itself instead of tendering it out.

In addition, the long chain from software vendor to ESA to GSA to Spaceopal might well be contributing to the slow resolution to the ground-segment software problems.

Finally, the secrecy surrounding the July outage is far from helpful. I understand how some things must stay confidential, but from my previous work experience, I know that classified documents take longer to disseminate which slows down remedial actions a lot.

Galileo as GPS Enhancement

It appears likely that Galileo will continue to struggle at least until 2021, more likely 2022. By that time sufficient satellites may be aloft to achieve ‘Full Operational Capability’. Given that things are likely going to get worse before they get better (because the satellites are aging), it may not be tenable to continue claiming that Galileo is more precise or even better than GPS.

It is however true that the working Galileo vehicles provide excellent service. “GPS enhanced by Galileo” is a superb combination, and almost all phones currently sold benefit from using both systems. And in fact, Galileo has been carefully engineered so it can interoperate with GPS very well.

GPS itself is also in need of a refresh, it may yet be that Galileo will prove its value by rescuing GPS performance when the American system might struggle in the future.

Some final notes

I wish the Galileo project all the best. There has been some speculation (and phone calls) if I secretly work for an out of luck Galileo supplier wanting to shake up the market - but no such thing is the case. I merely barged into Galileo at a time when I was open to doing a new project. My main work is actually another open source project.

I sincerely hope that galmon.eu, the software, our world-wide network of volunteer operators & our openly available dataset can contribute, if only a little, to the welfare of Galileo and GNSS in general.