The insoluble pancake

Let’s start with some philosophy. Where does new knowledge come from? Well, from doing experiments, and comparing the results of those experiments with ideas — hypotheses — concerning physical laws. This works: technology created from knowledge gained this way has transformed the world.

However, as our knowledge of the Universe increases, so each new experiment becomes more complicated, harder to do and more expensive. They have to because each new hypothesis must also explain all the previous experiments. In astronomy, technology enables new voyages to some unknown part of “parameter space”, which in turn lead to ever more stringent tests of our hypotheses concerning how the world works. These experiments allows us to take a good long look at something fainter, faster, further away. Something which was undetectable before but detectable now.

Space missions are really different from traditional science experiments. For one, the margin of error is minuscule and generally errors are catastrophic although there can be a few happy counter-examples. What this means is that a careful web of “requirements” must be written before launch. The influence of every aspect of the mission on the final scientific goal is estimated together with the likely margin of error.

So here’s the paradox: how do you build a vastly complicated experiment which is supposed to find out something new and be certain that it will work? How to make sure that you covered all the bases, that you thought of all things and still leave open the possibility for discovery? Even harder, how do you persuade someone to give you a big chunk of change to do it? The answer is a kind of weird mixture of psychology of and book-keeping. So first the (conceptually) straightforward bit: the book-keeping, which comes from trying to carefully chart all the tiny effects which will perturb your final measurement. This is actually notoriously difficult.

There are many celebrated examples of this kind of thing which didn’t quite work out from the annals of astronomy missions. After the launch of the Gaia satellite, astronomers were dismayed to find that there was an unknown source of background light. It turned out that this came from sunlight scattering off fibres sticking out from the sun-shield, which nobody had thought about before. Some changes to the instrument configuration and processing have helped mitigate this problem.

An even more epic example is Gravity Probe B experiment, designed to make a stringent test of General Relativity. This experiment featured the smoothest metal balls ever produced. Planning, launching and analysing data from this satellite took almost a half-century (work started in 1963 and the satellite survived multiple threatened cancellations). The objective was to measure how relativistic effects changed the rotation of these two metal balls. After an enormous amount of work analyzing data from the satellite featuring very smart people indeed, a result was announced, confirming General Relativity — but with errors around an order of magnitude larger than expected (Cliff Will has an excellent write-up here. Despite decades of work, three sources of error were missed, the most important of which being stray patches of static electricity on the balls’ surface, which exploded the final error budget. In the case of both Gaia and Gravity Probe B the missions were successful overall, but unknown sources of error were not entirely accounted for in mission planning.

Part of the Euclid challenge is to make the most accurate measurement of galaxy shapes of all time. Light rays passing through the Universe are very slightly perturbed by the presence of dark matter. If two rays pass next to the same bit of dark matter, they are perturbed in the same way. Euclid aims to measure this signal on the “cosmic wallpaper” of very faint distant background galaxies. These galaxies are effectively a big sheet of cosmic graph paper: by measuring how this correlated alignment depends on distance between the galaxies you can find out about the underlying cosmology of the Universe.

So how do you do this? The problem is that on an individual galaxy the effect is undetectable. Millions of sources must be measured and combined, and instrumental effects can completely submerge the very weak cosmological signal. We need to know what these effects are, and to correct for them. Some are conceptually straightforward: the path of light inside the telescope will also deform the galaxies. Or maybe as the camera is read out electric charge falls into holes in the detector silicon (drilled by passing charged particles) and gets trailed out. This, annoyingly, also changes galaxy shapes. Even worse: imagine that galaxies are not really randomly orientated on the sky, but line up because that’s how they were made back in the mists of cosmic time. You need to find some way to measure that signal and subtract it from the one coming from dark matter. In general, the smaller the effect you want to measure, the more care you need to take. This is all the more important today, where in general the limiting factor is not how many things you have to measure (as it was before) but how well you can measure them. In the end, your only hope is to try to list each effect and leave enough margin so that if anything goes wrong, if you miss anything, the mission is not compromised.

After accountancy, psychology, or more particularly cognitive biases. For example “strong gravitational lensing”– where background galaxies are visibly deformed by dark matter present in massive objects like galaxy clusters– had already been seen on photographic plates well before it was “discovered” in electronic images in the 1980s. Before that, people were not expecting it, and in any case those distorted galaxies on photographic plates looked too much like defects and were ignored.

So how do you plan an experiment to derive cosmological parameters without including some cosmological parameters in your analysis? After dodging all the bullets of unknown systematic errors, how to do you make sure you haven’t included an unknown bias which comes from people just having some preconceived ideas about how the universe should be? The answer must come from trying to design an analysis with as few unwarranted assumptions as possible, and if there are any to be made, hide them from the researchers doing the sums.

The recent story of the discovery of gravitational waves provides a fine example. Most scientists didn’t know until very late on that the signal they were dealing with was real and not a simulation. Such simulations had been routinely injected into the computers by colleagues wanting to make sure everything was working (this was how they had been testing everything). For Euclid, that would be the “Matrix” solution: most astronomers wouldn’t know if data under analysis was real or a simulation after some secret sleight-of-hand early on. But making a realistic simulation of whole Universe as seen by the satellite might be, to say the least, very challenging. More realistically this test might happen later on, with catalogues of objects being shuffled around so that only a few people would know which one corresponded to the real Universe. Like drug trials, but with galaxies.

In the end, you can’t plan for the unexpected because, well, it’s unexpected. But you can at least try to prepare for it. You have to, if you want your results to stand up to the scrutiny of peer-review and make that new discovery about how the universe really works.