Andreas Völker

Comparing JPEG, WebP and HEIC images after bit flips

2024-02-28T11:00:11+01:00

In the last few months, I have thought a little about long-term archiving files, the effects of random bit errors and how it interacts with compression. So today, I looked a little at the different image formats.

I am not an expert in modern image compression, so I ran a totally unscientific experiment for fun. I wrote a terrible a Python script that uses ImageMagick to convert an image into different formats and flip either a single or four random bits. This is repeated nine times and the images are converted to PNG and composed into a grid.

I look at JPEG, WebP and HEIC (Apple’s format based on H265). My assumption is that JPEG uses much older and less efficient compression algorithms that leave some redundancy in the file and might degrade less, while WebP and HEIC seem kind of similar to me.

1 bit flipped

Click images to enlarge.

JPEG

Eight of the nine images look fine, while in the middle image, something important seems to be destroyed. Maybe something is in the file header.

WebP

Here the horrible blocky artifacts start from some line and everything after is broken, but retains the basic color palette.

Two images look fine, and two others have major artifacts (to me, comparable to old photographs with major water damage), but if the motif was important to me, they seem kind of okay.

The other five images are basically destroyed. You might recognize them when you know the image, but they seem mostly useless to me.

HEIC

This one confuses me. The artifacts seem only to wander down and to the right, and they are incredible horrible to look at. Blocky geometric shapes in strange colors.

Again, two images look fine. For the other images, I am at a loss how to categorize them. There are more perfectly good pixels than in the WebP images, but the artifacts are horrible. Some might be okay as mementos if you blur and fade out the color of the artifacts. Five might be salvageable in this way.

One is a complete loss except for the bird head, but even that might be good in some cases.

4 bits flipped

Click images to enlarge.

JPEG

Four images are fine. The other ones have a line where the color shift or the motif is shifted a little. They are not great, but totally acceptable. This time, there was no header damage.

WebP

Three images are kind of okay with some major artifacts; five are for bird-head enthusiasts only, and the last one is a complete loss.

HEIC

This is a horror show.

One image was too damaged to be an image and shows up as a white spot
One image is almost only an artifact without any relevance to the source image
Five images are again for the bird-head enjoyers
Two images have the main motif kind of undamaged, but there are again absolutely horrible artifacts beside it.

BMP with 1000 bit flips

Now, for fun, let’s look at BMP as an uncompressed file format, but let’s introduce 1000 bit errors instead of four.

This looks perfectly fine. There are obviously some pixel errors, but they don’t destroy the image. I think it would be trivial to find the worst of them and simply replace them with an average of the surrounding pixels.

Of course, at this bit-flip rate, we were lucky not to destroy the header fields. Even if this happens, we might be able to fix the header with a text editor and calculator, because BMP is a really simple format.

Restrictions

Especially the comparison with the bitmap is kind of unfair. The uncompressed file is orders of magnitude bigger and thus will have a much higher rate of bit error.

If I had written less horrible Python code, I might have corrected the flip rate for different file sizes.

What did we learn at the end?

Use redundant storage with checksums, so that there are no bit flips.

JPEG is kind of an okayish compromise. It is still much more efficient to store five copies of every JPEG instead of a BMP.

Why are the artifacts so different between WebP and HEIC? This might be a question for an expert in image compression, instead of some dubious experiment.

Readjusting watch hands on UTS DCF clockworks

2023-06-08T19:00:11+02:00

UTS DCF is a high-quality automatic clockwork based on the German DCF77 standard. I have a clock using one of these I like very much, but due to unfortunate circumstances, the watch hands got into the wrong alignment¹. So I very much wanted to fix this. This is what I did follow my instruction at your own risk.

At first, you need to get the clockwork into the 0:00:00 position. To do this ensure that the clock has a battery and short the reset pins. I used a small slow screwdriver. The clockwork will now go into the position where all hands are supposed to point directly up². Then the clockwork stops and will do nothing.

The next step is to pull out the battery and put a pin into the pinhole to lock the mechanism. New clockworks tend to come with a nice pin that fits perfectly for this, but you can use any small enough pin.

Now you can adjust the hands with careful rotation or just pull them out and put them in anew. Adjust them that all of them point directly up.

Now plug out the pin and insert the battery. Don’t do this the other way around. Now the clock should start to move to 4:00:00.

Mine did not. So now comes a strange part that scares me, but worked perfectly for two of these clockworks. Put the battery in the wrong way wait a few seconds and put it in normally.

After this, my clock did a full rotation of the second hand and then moved to 4:00:00. This is a little strange and most clockworks are supposed to go a full round to 0:00:00, but this one goes to the next interval round 4 hours. So if you plug out the battery again and reinsert it, it will go to 8:00:00 and on the next try to 0:00:00.

Now you should wait a few minutes and the clock will show the correct time when it gets a signal. It will usually go faster if you put the clock next to a window.

I hope this will help you to make your clock work correctly in less than the two hours I needed.

You should think before you rip out clock hands. I didn’t. ↩
At least in Germany this is usually the 12. ↩

Using interval arithmetic for better error propagation

2023-03-07T06:00:11+01:00

Errors and their propagation

If an experiment measures a number, the number usually contains some uncertainty that is often called the value’s error. This might result from the limited resolution of the measurement device or resulting from statistical analysis of multiple measurements.

For example for a value $x=5$, we call its error $\Delta x=0.2$. This is sometimes written as $x = 5\pm 0.2$.

If we want to derive another value $y = f(x)$, the error in $x$ will result in an error $\Delta y$ for $y$. The process of calculating this new error is called error propagation.

The usual method to do this in natural science is Gauss error propagation law. I think it is a badly behaved approximation and should basically never be used. Interval arithmetic is conceptionally simpler and always exact.

Our example

We will look at a value $x=0$ with $\Delta x = 2\pi$ and $y=\sin(x)$ and $z=\cos(x)$ for demonstration. This is kind of an evil example because it uses a transcendental function with an error larger than the value.

A less extreme version of this problem might happen in the physics of resonance. I vaguely remember using an in-principle similar function for a spin wave experiment.

Gaussian error propagation

Gauss had some clever ideas¹ to approximate the error, but the resulting formulas are easy enough. In this article we will only need the case for one input variable which is very simple:

\[\begin{align*} \Delta y = |f'(x)|\Delta x \end{align*}\]

Our example

The derivative of $\sin$ and $\cos$ are easy enough to calculate so we get:

\[\begin{align*} \Delta y &= |\cos(x)|\Delta x = 2\pi \cos(0) = 2\pi \\ \Delta z &= |-\sin(x)|\Delta x = 2\pi \sin(0) = 0 \end{align*}\]

Both of these are insane. $y = 0 \pm 2\pi$ is bigger than the possible range of $sin(x)\in [0,1]$. $z = 1 \pm 0$ does not have any error no matter what.

The value with its error comes from the full range of $\sin$ and $\cos$ and so the result should be able to take every possible resulting value of the functions and their ranges must be $y = z = 0\pm 1$.

Interval arithmetic

With interval arithmetic, we stop calculating with numbers and only consider intervals in which a number must be. For example, we can get an interval from a value and error as $x_I = [x-\Delta x, x+\Delta x]$ For our initial example of $x = 5\pm 0.2$ we get $x_I = [4.8, 5.2]$.

We can always² recover a value and error from an interval $x_I=[x_1, x_2]$ trivially as $x = \frac{x_1+x_2}{2}$ and $\Delta x = \frac{x_2-x_1}{2}$.

The general formula for error propagation with intervals is to calculate the image of the function on the interval

\[\begin{align*} f[x_I] = \{f(x): x \in x_I\} \end{align*}\]

Then the resulting interval is $y_I = [\inf f[x_I], \sup f[x_I]]$.

This simplifies to something much simpler in most cases, where you only have to consider the interval bounds. For example, the multiplication for two intervals is

\[\begin{align*} [a, b] \cdot [c, d] = [\min \{ac, ad, bc, bd\}, \max \{ac, ad, bc, bd\}] \end{align*}\]

which is trivial to calculate.

For a function like $\cos(x)$ it is a little more tricky:

Shift $x$ by some multiple of $2\pi$, so it is in $[0, 2\pi]$.
Find the endpoint $x_{max}$ of the interval that is nearest to either $0$ or $2\pi$.
Find the point $x_{min}$ that is nearest to $\pi$. This might be either of the endpoints of the interval, or the interval might contain this $\pi$.
The resulting interval is $y_I = [\cos(x_{min}), \cos(x_{max})]$.

That is not so hard.

Our example

Considering the example, we get the interval $x_I = [-2\pi, 2\pi]$. This still is obviously the full range of a period of our functions, and we get:

\[\begin{align*} y_I = z_I = [-1, 1] \end{align*}\]

Practical considerations: Using python

No one should ever do error propagation by hand, except to torture junior students. The most popular and sane³ tool to analyze measurements is python. I will demonstrate how similar it is using appropriate packages.

For example, we will use some values $x \in [0, 2]$ with error $\Delta x = \frac{x}{2}$ and calculate $\sin x$.

Gaussian error propagation

Using the package uncertainties, the resulting code is:

xs = [x/25 for x in range(50)]

from uncertainties import ufloat
import uncertainties.umath as umath

xsu = [ufloat(x, x/2) for x in xs]
ysu = [umath.sin(x) for x in xsu]

Interval arithmetic

Using the package interval, the resulting code is very similar:

xs = [x/25 for x in range(50)]

from interval import interval, inf, imath

xsi = [interval[x-x/2, x+x/2] for x in xs]
ysi = [imath.sin(x) for x in xsi]

Plotting it

Now after we calculate if it is easy enough to plot it and compare them graphically. As before the Gaussian error propagation has error bars that are above $1$, and they vanish around $x=\frac{\pi}{2} \approx 1.57$.

Conclusion

While using computers, interval arithmetic is just as easy as conventional error propagation and it avoids many strange artifacts of Gaussian error propagation.

The basic idea of gauss was that errors are very small (which is absolutely not always true) and linearize the function that derives the new value and calculates a spheroid around the value based on the input errors. Nobody really explains this ever, because it is very confusing and clever. ↩
Except if the interval is infinite on at least one side. This is a good thing because this makes intervals fundamentally more powerful than value and error. ↩
Julia might be slightly more sane than python ↩

Release of my first iOS app

2023-02-08T06:00:11+01:00

I release my first app for iPhone/iPad on the app store. In principle, it is really simple and it helps you to figure out when to ventilate your home without messing up the humidity.

It currently only works in Germany, but you can get it here:

Here come some thoughts on how it went.

Programming language: Swift

These days swift is the language of choice for native iOS apps. In the end, I also used swift for the relatively simple server component to gather the data from the weather service.

I think it is a really nice language. It has all the features I want without being too complicated. The syntax is nice to look at for me and easy enough to read.

Of cause it is a little messy, due to its compatibility with some parts of Objective-C which was preferred by Apple before developing Swift. It is very much a multi-paradigm language with all the pros and cons of that.

I had to implement some basic data structures like a Pairing heap and a Bounding volume hierarchy for arbitrary dimensional data. From my experience, this can get messy fast in most languages, but it was unexpectedly simple.

UI Framework: SwiftUI

SwiftUI is Apple’s relatively new UI Framework. It kind of follows the idea of react and vue.js, but I like it much more than these two. A big part of this is that there are specific features built into Swift to make it nicer to use, which Javascript misses at the moment. I also think it took some ideas from Immediate mode GUI that work very well. I think it is very well designed.

Data source: Germany public weather service

I get my data from the DWD. It has an amazing amount of freely accessible data for Germany. There is also a huge amount of data with a worldwide application, but it is not as nicely preprocessed as the local data.

The documentation and data formats are a little old-school at first, but easy to parse when you figured out what you need. Tip: If nothing works check if that data uses latin1 encoding instead of UTF-8. As far as I figured out this is not documented anywhere…

A proof for the closed form of triangular numbers

2022-12-31T06:00:11+01:00

The triangular numbers

\[T_n = \sum_{i=1}^n i, \text{for }n>0\]

have the generally known closed form

\[T_n = \frac{1}{2}n(n+1)\]

This can be easily proven by induction or young Gauss’s regrouping of terms.

Calculating some sums, I found another way using the sum of the squares. Even though I haven’t found it on the web yet, it is probably well-known.

Proof

We look at the sum of squares

\[S = \sum_{i=0}^{n+1} i^2\]

On the one hand, we can remove the last term from the series:

\[S = (n+1)^2 + \sum_{i=0}^n i^2\]

On the other hand, we can remove the first term and shift indices:

\[\begin{aligned} S &= 0 + \sum_{i=1}^{n+1} i^2 = \sum_{i=0}^n (i+1)^2 \\ &= \sum_{i=0}^n (i^2+2i+1) = \sum_{i=0}^n i^2+\sum_{i=0}^n 2i+\sum_{i=0}^n 1 \\ &= \sum_{i=0}^n i^2 + 2\sum_{i=0}^n i + (n+1) \end{aligned}\]

Equating both sides we get

\[\begin{aligned} &(n+1)^2 + \sum_{i=0}^n i^2 = \sum_{i=0}^n i^2 + 2\sum_{i=0}^n i + (n+1) \\ \Leftrightarrow &(n+1)^2 = n+1+ 2\sum_{i=0}^n i \\ \Leftrightarrow &2\sum_{i=0}^n i = (n+1)^2-(n+1) = n(n+1) \\ \Leftrightarrow &\sum_{i=0}^n i = \sum_{i=1}^n i = \frac{1}{2}n(n+1) \end{aligned}\]

QED

Some thoughts

This is kind of similar to the proof by telescoping series. A similar idea to remove the first and last element of a series is often used to calculate the partial sums of the geometric series.

This proof does not give as much insight as Gauss’s proof, but it is relatively easy for a proof by manipulation of sums. It also generalizes to the sum of $i^k$ for $k \in \mathbb{N}$ by always using the sum of $i^{k+1}$ and the binomic expansion of $(i+1)^{k+1}$.

A simple script for cron Prometheus export

2022-11-30T06:00:11+01:00

I like Prometheus for collecting metrics from servers and programs. Sadly the cron deamon does not support it. So I wrote a simple python script to report some metrics:

Start time
Duration of time
Exit code

It works with the help of the node exporter that is most likely used in most Prometheus setups.

The script

#!/usr/bin/env python3

from prometheus_client import REGISTRY, write_to_textfile, Gauge
import prometheus_client
from time import time
import sys
import subprocess

# debian/ubuntu
#OUTPUT_DIR = "/var/lib/prometheus/node-exporter"
#freebsd
OUTPUT_DIR = "/var/tmp/node_exporter"


jobname = sys.argv[1]
command = sys.argv[2:]

output_file = OUTPUT_DIR+"/"+jobname+".prom"


prometheus_client.REGISTRY.unregister(prometheus_client.GC_COLLECTOR)
prometheus_client.REGISTRY.unregister(prometheus_client.PLATFORM_COLLECTOR)
prometheus_client.REGISTRY.unregister(prometheus_client.PROCESS_COLLECTOR)

start_time = time()
completed = subprocess.run(command)
end_time = time()
exit_code = completed.returncode

Gauge('cron_starttime', 'Start time of cron job', ['job']).labels(jobname).set(start_time)
Gauge('cron_duration', 'Duration of cron job in seconds', ['job']).labels(jobname).set(end_time-start_time)
Gauge('cron_exitcode', 'Exit code of cron job', ['job']).labels(jobname).set(exit_code)
write_to_textfile(output_file, registry=REGISTRY)

How to install

This assumes basic familiarity with Prometheus and node_exporter.

Install, setup and run the prometheus node_exporter
Install the prometheus_clientpackage for python ¹
Copy the script to /usr/bin/cron_exporter ²
Make it executable
Adjust the OUTPUT_DIR variable for your system. I already have examples for your

Usage

Edit your crontab as usual, but put the cron_exporter and the name of the job in front of every line. So for example

@daily /usr/bin/certbot renew

should become

@daily cron_exporter certbot /usr/bin/certbot renew

Access from prometheus with PromQL

Duration of certbot: cron_duration{exported_job="certbot"}
Time since the last run for all jobs (in minutes): (time()-cron_starttime)/60
Exit codes for a job named backup: cron_exitcode{exported_job="backup"}

You can simply to pip install prometheus-client as root. Most people seem to think this is a bad idea, but it works for me. ↩
I think in principle it would belong into /usr/local/bin, but that does not work out with $PATH in Debian and Ubuntu ↩

Solving a $x^2+\varepsilon x=1$ with perturbation theory for two regimes

2022-11-27T06:00:11+01:00

Solving the equation directly

Let’s look at the equation $x^2+\varepsilon x=1$ We can rewrite it as $x^2+\varepsilon x-1=0$ for convenience. As with all quadratic equations, this equation has two solutions. Luckily both solutions of our equation are real numbers for any $\varepsilon$.

In this article, we will assume $\varepsilon > 0$ and we will only look at the bigger solution. So we obviously get $x(\varepsilon)=\frac{1}{2}\big(\sqrt{\varepsilon^2+4}-\varepsilon\big)$.

Limits for $0$ and $+\infty$

Let’s look at some limits of the function. At $0$ we trivially get $\lim_{\varepsilon\rightarrow\infty}x(\varepsilon)=\frac{1}{2}\big(\sqrt{0^2+4}-0\big)=1$.

$\lim_{\varepsilon\rightarrow\infty}x(\varepsilon)=0$ is more tricky to proof. The definition for this is¹

\[(\forall C > 0) (\exists \varepsilon > 0)(\forall s>\varepsilon) x(s) < C\]

Because $x(\varepsilon)$ is strictly monotonically decreasing² and bigger then $0$ this all comes down to finding an $\varepsilon$ so that $x(\varepsilon) < C$ for any $C > 0$. So let’s check:

\[\begin{aligned} &\frac{1}{2}(\sqrt{\varepsilon^2+4}-\varepsilon) < C\\ \Leftrightarrow &\sqrt{\varepsilon^2+4} < 2C+\varepsilon\\ \Leftrightarrow &\varepsilon^2+4 < (2C+\varepsilon)^2 = 4C^2+4C\varepsilon+\varepsilon^2 \\ \Leftrightarrow & 4 < 4C^2+4C\varepsilon\\ \Leftrightarrow &\frac{1-C^2}{C} < \varepsilon \end{aligned}\]

QED

Pertrubation theory for small $\varepsilon$

So let’s assume $\varepsilon$ is small. If we assume $\varepsilon = 0$ we can solve the resulting equation $x^2 - 1 = 0$ instantly. This is nice and we can formalize this by using a power series for $x$ as

\[x = \sum_{n=0}^\infty x_n\varepsilon^n = x_0 + \varepsilon x_1 + \varepsilon^2 x_2 + \varepsilon^3 x_3 + \varepsilon^4 x_4+\dots\]

with unknown coefficients $x_n$.

So we can plug this in our equation and get $\begin{aligned}0&=x^2+\varepsilon x-1\\ &=(x_0 + \varepsilon x_1 + \varepsilon^2 x_2 + \varepsilon^3 x_3 + \varepsilon^4 x_4+\dots)(x_0 + \varepsilon x_1 + \varepsilon^2 x_2 + \varepsilon^3 x_3 + \varepsilon^4 x_4+\dots)\\ &\quad+\varepsilon(x_0 + \varepsilon x_1 + \varepsilon^2 x_2 + \varepsilon^3 x_3 + \varepsilon^4 x_4+\dots)-1\end{aligned}$

We can multiply this out $\begin{aligned}0=&x_1^2+2\varepsilon x_0x_1+2\varepsilon^2 x_0x_2+\varepsilon^2 x_1^2+2\varepsilon^3x_0x_3+2\varepsilon^3x_1x_2+\\ &2\varepsilon^4x_0x_4+2\varepsilon^4x_1x_3+\varepsilon^4x_2^2+\\ &\varepsilon x_0+\varepsilon^2 x_1+\varepsilon^3 x_2+\varepsilon^4 x_3-1+\mathcal{O}(\varepsilon^5) \end{aligned}$

Grouping together by powers of $\varepsilon$ we get $\begin{aligned} 0=&(x_1^2-1)+\varepsilon(2x_0x_1+x_0)+\varepsilon^2(2x_0x_2+x_1^2+x_1)\\ &\varepsilon^3(2x_0x_3+2x_1x_2+x_2)+\varepsilon^4(2x_0x_4+2x_1x_3+x_2^2+x_3)+\mathcal O(\varepsilon^5) \end{aligned}$

This equation needs to be zero for an arbitrary $\varepsilon$. This is only possible if each factor is equal to $0$. So we will look at each factor.

$\varepsilon^0=1$

We get $x_0^2-1=0$. Again we only want the larger solution so we take $x_0=1$. After this the higher orders are simpler (but mostly not easier), due to only being linear equations.

$\varepsilon$

We have $2x_0x_1+x_0=0$. Solving for $x_1$ we get $x_1 = -\frac{1}{2x_0}x_0=-\frac{1}{2}$.

$\varepsilon^2$

We have $2x_0x_2+x_1^2+x_1$ and solving for $x_2$ we get $x_2 = -\frac{1}{2x_0}(x_1^2+x_1)$.

Now we have a small problem. There are still variables in our formula for $x_2$. Luckily (and by construction) we already know $x_0$ and $x_1$ and can plug them in. So with easy calculation $x_2 = \frac{1}{8}$.

$\varepsilon^3$

$2x_0x_3+2x_1x_2+x_2 \Rightarrow x_3 = -\frac{1}{2x_0}(2x_1x_2+x_2)=-\frac{1}{32}$

Higher orders

We can continue this schema as long as we want. The equations get increasingly longer but stay in principle simple.

Summing up the series

So I calculated two more orders before I got bored. Let’s plug them into our definition of the power series again:

\[x = 1-\frac{1}{2}\varepsilon+\frac{1}{8}\varepsilon^2-\frac{1}{32}\varepsilon^3 +0\varepsilon^4+\frac{1}{512}\varepsilon^5+\mathcal O (\varepsilon^6)\]

To get an approximation we just ignore all orders we did not calculate:

\[x \approx 1-\frac{1}{2}\varepsilon+\frac{1}{8}\varepsilon^2-\frac{1}{32}\varepsilon^3 +0\varepsilon^4+\frac{1}{512}\varepsilon^5\]

So for $\varepsilon$ smaller $1$ our approximation works out well and gets worse for larger $\varepsilon$. We assumed $\varepsilon$ is small at the beginning so all this is as expected.

If we look at the limits, at $\varepsilon\rightarrow0$ everything fits, and at $\varepsilon\rightarrow\infty$ our series diverges.

This is all rather well-known. If you have some formal training in quantum physics or advanced ordinary differential equations you will have heard this before. In the next part, I will show you something quite similar, but it looks funny and it is a little surprising that it works.

Pertrubation theory for large $\varepsilon$

In the last step, we looked at a power series that works well for $\varepsilon$ small. Now let us construct a power series that works well when $\varepsilon$ becomes very large. In this case, $\varepsilon^{-1}=\frac{1}{\varepsilon}$ becomes very small, so let’s use it for a series: $x=\sum_{n=0}^{\infty}x_n\varepsilon^{-n}$

We can again plug this into our quadratic equation and look at the resulting powers of $\varepsilon$.

$\varepsilon$

Somewhat suprisingly we still have a positive power of $\varepsilon$ (from the term with $\varepsilon$ in the equation): $x_0=0$

Nothing to do here. Turns out we could have started our series at $n=1$.

$\varepsilon^0=1$

We get $x_0^2+x_1-1=0$, so $x_1=1$.

Another surprise is that we don’t get to choose between two solutions this time. I think this could be because the smaller solution diverges at infinity. If you work it out tell me what you find.

$\varepsilon^{-1}$

We get $2x_0x_1+x_2=0$ and so $x_2=0$. It turns out all even order terms in the series are zero. If you don’t believe me, try to prove it by induction. It is not that hard, but messy to write down correctly.

$\varepsilon^{-2}$

Now we get to the first order without a surprise. The equation is $2x_0x_2+x_1^2+x_3=0 \Rightarrow x_3 = -x_1^2 = -1$. Now we can go on for the higher(lower) orders.

Summing up

With some more orders, I got

$x = \varepsilon^{-1}-\varepsilon^{-3}+3\varepsilon^{-5}$.

In most cases, it is a very bad omen, when the coefficients in a power series increase, but it is not that bad in this case. For $\epsilon \rightarrow 0$ this diverges badly anyway.

Due to $x_0=0$, this converges at $\varepsilon\rightarrow\infty$ to $0$ as it should.

If we plot this function, as expected it works well for large $\varepsilon$ (in this case larger than $3$) and does not work for small values.

The strange case of $\varepsilon\approx 1$

So we found a good approximation for small and large $\varepsilon$ with simple calculations, but they both break down at $\varepsilon=1$. That seems strange to me. After some months of thought, I have not found something similar that works there.

Of cause, you can calculate more orders and the approximation at $1$ will become good enough for practical applications.

You might try a series of the form $x = \sum_{n=0}^\infty x_n(\varepsilon-1)^n$, but this looks so ugly to me, that I did not even want to work it out for fun.³

Is this useful?

Not really for quadratic (or other algebraic) equations.

The principle for small $\varepsilon$ is used in many calculations in high energy physics. You can think of this quadratic equation as one of the simplest possible model systems for the method.

If one could work out the the method for large $\varepsilon$ for quantum mechanics, it could possibly help for some quantum-systems with strong coupling. Maybe.

Don’t be confused: the names of the variables look kind of strange because we already used $x$ and $\varepsilon$ ↩
For $\varepsilon \ge 0$
\[\begin{aligned} &x'(\varepsilon) = \frac{1}{2}\big(\frac{\varepsilon}{\sqrt{\varepsilon^2+4}}-1\big) < 0\\ \Leftrightarrow & \frac{\varepsilon}{\sqrt{\varepsilon^2+4}} < 1\\ \Leftrightarrow & \varepsilon < \sqrt{\varepsilon^2+4} \end{aligned}\]
QED ↩
I think it would be beautiful if you can make the combined series $\sum_{n=-\infty}^{\infty}a_n\varepsilon^n$ work, but I just get an infinite number of terms for each order. ↩

Interpolation on earth surface (far from the dateline)

2022-10-27T07:00:11+02:00

Introduction

If you want to get the current temperature you could go outside, but you can also approximate it with interpolation. All over the world are public weather stations where you can get the current temperature there.

You could just take the nearest to your home and believe it, which often is fine, but let’s do better anyway. You can take the ones around you and use the average of them. Even better you weigh them and take give the values nearer you more relevance. In other words: You interpolate their temperatures at your current location.

In this article, I will discuss how you figure out which stations are around you and how you weigh them and how you can do that kind of fast with a computer.

“Around you” is surprisingly hard to define and figure out. The simplest way is to take the k nearest stations, but this does ignore a central problem: occlusion. If there are two information sources in a line from you, you always only want to look at the first of them and the farther behind and occluded ones should almost always be ignored.

Now the title of this post says we only do interpolation far from the date line. As you almost surely know a point on the earth is often addressed with a latitude and longitude which are basically angles. As always with angles they skip somewhere from 360 degrees back to 0 degrees. On the earth, this is defined to happen in the region of the pacific. This is messy to handle and you don’t always need it. For example, I work with the data from the Germanys public weather service which is only really good inside of Germany and Germany is very far from the pacific. Luckily Germany also does not have colonies anymore.

Why not do natural neighbor interpolation

(You can safely skip this paragraph if you don’t know what natural neighbor interpolation is or you know what it is and don’t like it at all)

If you have ever looked at multidimensional interpolation you heard of natural neighbor interpolation. This is a really clever algorithm based on Voronoi diagrams. It automatically handles occlusion and it gives a kind of natural way to get the weights without having to mess around.

I think it is a brilliant idea and I don’t like it at all. I collaborated on a based on this principle some time ago and I don’t like it very much anymore. This algorithm is horrible to implement fast and even then it is still slow. It is a mess of numerical problems if points have too much symmetry. And implementation is only the smallest problem.

Naturalness is very nice because you don’t have to tune any parameters. Until your and the algorithm’s definition of naturalness is not the same. Then you want to tune parameters and you can’t. For example, on earth, the distance on the surface is not the same as the distance between the coordinates as vectors. There is no direct way to make this work with natural neighbor interpolation.

Someone clever might figure out how to make this work for this special case. I’m sure that will make the algorithm even more horrible to implement efficiently and you can’t reuse it for the next modification. For example, you might want to include the different elevations of the weather stations, because it turns out the temperature on the next mountain is not that applicable in the valley below. Let us use something else.

Getting the weights

Getting the weights between the at a target location from the source stations is easy from their distance. For any station, you take the distance and apply a decreasing function to it.

Classically you use a weight[i] = 1/pow(distance[i], e) for some exponent e which is usually at least 2¹. Now the sum of all weights is not 1 as it should be so you have to calculate their sum and divide each weight by the sum.

In swift, this could look like the following²

func interpolate(datapoints: [Double], distances: [Double]) -> Double{
    let weights = distances.map{distance in 1.0/pow(distance, 2)}
    let weightSum = weights.sum()
    let normalizedWeights = weights.map{weight in weight/weightSum}
    let interpolated = zip(normalizedWeights, datapoints).map{ weight, data in
        return weight*data
    }.sum()
    return interpolated
}

You can use any decreasing function of the distance. I kind of like a shifted and mirrored sigmoid functions, but it is fun and easy to try different functions that come to your mind.

We can also put anything else into this formula. For example, we might add 100m more distance for every 1m of elevation difference.

To get the distance it might work fine to ignore the curvature of the earth and assume longitude and latitude are a flat grid³ and take their distance with the Pythagoras theorem. If your stations are near enough this works fine, but we can do better and use the real distance on the earth’s surface. This is easy to calculate by using the (Haversine formula)[https://rosettacode.org/wiki/Haversine_formula]⁴.

Making it faster with nearest neighbors

So until now, we look at every station in (potentially) the world and this sounds like a lot of useless work. So let us just get the nearest ones. In most cases k=5-10 stations are enough.

A simple way to do this is to loop through all stations and put them into a max-heap ordered by the distance. Whenever the heap contains more than k values you can remove the most distant one to keep the heap small and efficient. So in the end the heap will contain the k nearest stations.

This can be done much more efficiently by using bounding volume hierarchys⁵. If the heap contains at least k element its max is always the farthest a relevant point can ever be. We can use its distance to prune large parts of the bounding volume hierarchy.

We could use the correct distance, but this will get messy real fast when we want to use the bounding volume hierarchy. Just use the simple Pythagorean distance!

The simplest implementation of this might look something like this:

func KNN(_ k: Int, at: Vec2, from sources: [Station]) -> [Station]{
    var heap = BinaryHeap<Station>(withComparision: {a, b in Vec2.distance(at, a.position()) >= Vec2.distance(at, b.position())})
    for point in sources{
        heap.insert(point)
        if heap.count > k{
            heap.extract()
        }
    }
    var ret = [Station]()
    while heap.count > 0{
        ret.append(heap.extract()!)
    }
    return ret
}

Occlusion

Now we got a few nearest points left we can do occlusion. So let’s just do the simplest thing possible. For each point we define an area behind it, that is occluded from the view of our target location. So for every station, we check which other stations are in the occluded area and throw them away.

We same basic linear algebra and trigonometry you can work this out and you might find the following:

func isOccluded(from source: Vec2, occluder: Vec2, occluded: Vec2, openingAngle: Double = 45) -> Bool{
    let limit = cos(openingAngle*Double.pi/180)
    if occluded == occluded{
        return false
    }
    let normal = (occluder-source).normalize()
    let cos = normal.dot(occluded-occluder)/Vec2.distance(occluder, occluded)
    return cos > limit
}

Now we can check each pair of stations and reject all that is occluded in any way and we are done. Because we only have a few nearest stations this is fast enough for most practical applications and we don’t need to think of a more clever algorithm to do this.

I use an opening angle of 45°, but you might tune this to your liking and application.

How to incorporate some curvature

Until now we ignored the curvature of the earth for the most part, but this is simple enough to fix.

We can take something like 30% more nearest neighbors than we really want. Now we sort these by the correct surface distance and only take the nearest k of them. So after that, we have almost surely the nearest points considering curvature with very little more work.

Review

So let us review what we do

Think of a target point for our interpolation
Find a little more than k nearest stations with some approximations maybe with the help of a bounding volume hierarchy
Select the real nearest k from them
Check which ones are occluded by others and ignore them
Calculate the weights from the distance to the target point and make them sum up to 1
Get the weighted average of the temperature
Go for a walk with temperature adequate cloth

This obviously works for any other positions on earth than stations and any other quantity than temperature.

How to work near the dateline

So this only works when the latitude and longitude are near enough to normal rectangular coordinates. This definitely does not hold near the date line. So what to do there?

The trick is not to use latitude and longitude. We can simply use a 3D coordinate system for the earth which are always well behaved. It is simple to calculate the positions in real space and it turns out bounding volume hierarchies and our occlusion code generalizes without any problems to 3D. It only gets a little slower to calculate.

For the calculation of the weights, we might want to go back to the latitude and longitude because that formula works perfectly well with the date line.

This is called Shepard’s method. ↩

For this to work sum must be defined something like this

extension Sequence where Element: BinaryFloatingPoint{
    func sum() -> Element{
        return self.reduce(0.0, +)
    }
}

↩

This is basically a Mercator projection, which is often used for world maps. ↩
Vincenty’s formula is even more precise and considers that the earth is more an ellipsoid and not a perfect sphere. I’m pretty sure you don’t need this for interpolation. ↩
This article series is a really good overview for conventional bounding volume hierarchies. I like to do this rather differently and will try to write it down one day. ↩

SI model stability with unusual perturbation theory

2022-04-12T07:00:11+02:00

The SI model is one of the simplest models to describe the spread of infectious disease using differential equations. I figured out a funny way to look at its steady states. It reproduces the well-known and fairly obvious result.

The model

The model is defined by the equations

\[\begin{aligned} \dot I(t) &= \beta S(t) I(t) - \gamma I(t)\\ \dot S(t) &= -\beta S(t) I(t). \end{aligned}\]

where $S(t) \in [0,1]$ is the still suseptible part of the population and $I(t) \in [0, 1]$ the currently infected part. $\beta >0$ and $\gamma>0$ are constants for the rates of infection and resolution of infections.

The idea is that the rate of new infections is proportional to the chance of a susceptible individual meeting an infected individual. We assume the infected are only infectious for some time and shrink exponentially in absence of new infections. This is a really basic model that ignores reinfections, interventions, inhomogeneous populations, spatial dynamics…

As with almost any nonlinear equation, there is no practical general solution, so lets at least look at the steady states of the system.

Steady states and their stability

The obvious steady state for this model is $S=\bar S$ arbitrary and $I=0$. If there are no infections nobody else can get infected. The slightly unusual thing, in this case, is, that there are an infinite number of dense steady states.

Now we will try to see if these are stable, i.e. if a little perturbation shrinks back to zero or explodes. The usual tool for this is linear stability analysis. It looks at the eigenvalues of the jacobian so let’s calculate it for the steady states:

\[J = \begin{bmatrix}\beta \bar S & \gamma\\\\ 0 & 0\end{bmatrix}\]

The last row of this matrix is $0$ and this implies that $0$ is one of its eigenvalues. Sadly, in this case, linear stability analysis can’t make any conclusion about the stability so we need another tool: perturbation theory.

Perturbation theory

The idea of perturbation theory is to transform a hard problem into a infinite series of easy problems and get an approximation to arbitrary precision by solving these. As is usual we insert a small parameter $\varepsilon$ into our system and develop the solution as power series of it. So let’s do this:

\[\begin{aligned} I(t) &= \sum_{i=0}^{\infty} \varepsilon^n I_n(t)\\ S(t) &= \sum_{i=0}^{\infty} \varepsilon^n S_n(t) \end{aligned}\]

Now we can plug these into our equations:

\[\begin{aligned} \dot I_0 +\varepsilon \dot I_1+ \cdots &= \beta (S_0+\varepsilon S_1+\dots)(I_0+\varepsilon I_1+\dots)-\gamma(I_0+\varepsilon I_1+\dots) \\ &=\beta S_0I_0-\gamma I_0+\varepsilon(\beta I_0 S_1+\beta I_1 S_0-\gamma I_1)+\dots\\ \dot S_0 +\varepsilon \dot S_1+ \dots &= -\beta (S_0+\varepsilon S_1+\dots)(I_0+\varepsilon I_1+\dots) \\ &= -\beta S_0I_0+\varepsilon(\beta I_0 S_1+\beta I_1 S_0)+\dots \end{aligned}\]

If you have some experience with perturbation theory you might think we should have put an $\varepsilon$ into our equations. Even worse: if we look at all terms without an $\varepsilon$ you find we still have the old equations from before and we still don’t know how to solve them.

Now here comes my unusual trick: We put an $\varepsilon$ into the initial condition of the equations and set $S(0) = \bar S$ and $I(0) = \varepsilon \bar I$ with an arbitrary (and in the end boring) $\bar I$. Now let’s solve our system by order!

$\varepsilon^0$ Order

If we collect everything without an $\varepsilon$ we get:

\[\begin{aligned} \dot I_0 &= \beta S_0 I_0 - \gamma I_0 \\ \dot S_0 &= -\beta S I_0 \\ S_0(0) &= \bar S \text{ and } I_0(0) = 0 \\ \end{aligned}\]

Now if we look at the first equation we see an $I_0$ in every term and with the initial condition it turns out this equation can be trivially be solved as

\[I_0(t) = 0.\]

Plugging this in the second equation the right side becomes zero and so the solution of the equation is a constant. With the initial condition, we directly get

\[S_0(t) = \bar S.\]

By solving the 0th order we basically showed that the steady-state is steady.

$\varepsilon^1$ order

Now we come to the interesting terms. Collecting everything with $\varepsilon$ we get

\[\begin{aligned} \dot I_1 &= \beta S_0 I_1 + \beta S_1 I_0 -\gamma I_1= (\beta \bar S-\gamma) I_1 \\ \dot S_1 &= -\beta S_0 I_1 - \beta S_1 I_0 = \beta \bar S I_1 \\ S_1(0) &= 0 \text{ and } I_1(0) = \bar I \end{aligned}\]

So the first equation is linear, homogeneous with constant coefficients and the solution is well known and easy to check. When plugging in the initial condition we get

\[I_1(t) = \bar I e^{(\beta\bar S-\gamma) t}\]

If we plug this into the second equation it becomes an integrable equation:

\[\dot S_1 = -\beta \bar S \bar I e^{(\beta\bar S-\gamma) t} \Rightarrow S_1 = -\beta \bar I \bar S \int dt\, e^{(\beta\bar S-\gamma) t}\]

and with the initial condition, we get

\[S_1(t) = -\frac{\beta \bar I \bar S}{\beta\bar S-\gamma}\left(e^{(\beta\bar S-\gamma) t}-1\right).\]

Higher orders

With the same principle we always get the equations for each order and could in principle solve them fairly easily (using the substitution $I_n(t) = \tilde I(t) e^{(\beta \bar S-\gamma)t}$). This gets increasingly messy and I have not figured out the general form of them. But with some thought, we can see $I_N$ und $S_N$ have the form

\[\sum_{n=0}^N c_n e^{n(\beta \bar S-\gamma) t}\]

with constants $c_n$. This means these higher terms have similar but faster growth/shrink behavior as the first orders.

Now a mathematician should check the radius of convergence which is not obvious at all. Luckily I come from a physics background and can ignore such “details”.

What that means for stability

So let’s collect what we figured out:

\[\begin{aligned} I(t) &= \varepsilon \bar I e^{(\beta\bar S-\gamma) t} +\mathcal O (\varepsilon^2)\\ S(t) &= \bar S -\varepsilon\frac{\beta \bar I \bar S}{\beta\bar S-\gamma}\left(e^{(\beta\bar S-\gamma) t}-1\right)+\mathcal O (\varepsilon^2) \end{aligned}\]

Looking at the exponential function they shrink exactly if $\beta \bar S-\gamma < 0 \Leftrightarrow \bar S < \frac{\gamma}{\beta}=:\frac{1}{R_0}$. If the sign is flipped the number of infections explodes. The newly defined $R_0$ is these days fairly well known as the basic reproduction rate. Our condition reproduces exactly the condition for herd immunity, which is nice.

Going back to our equation we can see if this condition holds infection fall back to zero and it seems to be stable. But we also need to look at the stability of the susceptible population $S(t)$. Turns out it does not return to $\bar S$ and shrinks somewhat. This is not compatible with the definition of stability!

So in the classical sense there exist no stable steady states. But above the herd immunity threshold, small perturbations are fairly inconsequential and only move us a little down the line of steady states. For practical concerns, I would call this stable.

Checking with numerics

Now we got a solution we should check if this first-order works at all. The model is easy to solve with numerics (Sourcecode).

Overall it seems to work well, if the exponent $E = \beta \bar S - \gamma <0 $ and $\bar I$ is smaller. If this gets near $0$ there are significant deviations. The explanation for this is, that with a big $E$ or $\bar I$ the changes of $S(t)$ are bigger and the first order basically assumes $S(t) = \bar S$. This might improve in the higher orders.

$E=-0.02$ $\bar I=0.001$

Conclusion

In some cases, you can do perturbation theory by putting your perturbation constant $\varepsilon$ only into the initial conditions. To my astonishment, it solves the equation, does not produce contradictions, and is in this case the most straightforward method to show and analyze the stability I found.

The great thing with perturbation theory is you can put an $\varepsilon$ in all kinds of creative places and see what happens and sometimes it even works out.

Writing scientific software by yourself

2021-07-25T12:00:11+02:00

Today many scientists have problems that are solved by developing new programs. Sadly most of them are not software developers, so the quality of the resulting programs is mostly up to luck and individual interest. They mostly only have to take basic courses into programming by other scientists. Often the professors mostly teach from books because they only wrote basic programs 30 years ago in their own Ph.D. time.

I was in the situation to write some software for theoretical physics, except I studied computer science before and did some work as a programmer on a larger project. In this article I will write about some ideas and techniques I found along the way.

I will only look at programs written by a single researcher by himself, as it was done in my community. Projects with many researchers and distributed to end-users are a much more complex problem. I will only look at program development, not project management.

Why care about this?

Scientists might ask why they should care. They might have some Fortran files they copied from someone else and mess around in till it mostly works. They are here to do science, not software. They might even dismiss their program as mere “codes”. I will try to give some reasons you might want to care.

In this linked article, Freeman Dyson discusses if tools are as important as ideas to science. Even if you disagree, it makes clear that tools are important and programs are clearly tools for science.
The programs are a relevant result of the research. They often contain all the messy details and tricks left out in articles and I think they should even be published with the articles. Sadly not even computer science does that so it might take decades for natural sciences to get there.
You will be faster and more comfortable using and modifying your programs for your research.
You will be (a little) more confident in your programs if they are less messy.
These days there are many jobs involving software if you ever leave academia.
Making better programs is fun.

An example of a program

To make the kind of programs discussed here more clear I will explain my own research. I look at the time evolution of electrons in quantum wires described by differential equations. There were multiple numerical schemas to solve these equations and many ways to combine them with different models of electrodynamics. At first, I looked at some aspects only in 1D systems and later extended them to 2D. These simulations take hours of calculation time and produced up to a few gigabytes of raw data.

The next step was a program to analyze the raw data and figure out how to get the relevant information. I needed to do much experimentation to figure out what I could get out of the data. Here every run might take a few minutes and output less than a megabyte of data.

The last step is to take the results and make good-looking plots with clear labeling. This takes a few seconds and does not contain any ideas relevant to physics.

Infrastructure

Let’s begin with something seemingly boring, that everyone working in software knows anyway. There is some infrastructure around your programming. Let’s look at some stuff.

Programming language: Look at the programming language you use. Spend some hours to learn what it can do. There are usually good tutorials online on their official website. I use C++ for the computationally complex part and python for the rest because I know them well.
Text editor: You will spend much time programming in a text editor or integrated development environment. Spend an hour to test a few of them and stay with the one you like. Don’t blindly use what everyone else in your group uses; it might be worse than what was used in the 90s. I like Visual Studio Code because it works on most operating systems and has many cool features through plugins.
Version control systems: Every halfway reasonable software developer uses version control to put his software into and to have access to all previous changes. You should also do this. I think you should use git. It is the industry standard, it works and it is easy to get help with it. There are even multiple graphical interfaces to use git, but I don’t know much about them.

Git allows you to have a remote server to sync with. This is especially useful if you work from different computers on the same program. You can check if your university provides such a server or use a (free) commercial service like github. It might even work as (hopefully another) backup.
Build system: You probably need to compile your software. Coping the command to do that from the first line of your source file or your shell history is a bad idea. You should use a system for that as everyone else in software does for 40 years. At least use Make or some shell script.

I use CMake for my C++ programs. It is very flexible, works fine, and even has integration into my editor.
Compilers: If there are multiple good compilers for your programming language make your code work with more than one of them. This usually finds bugs in your programs and improves the chances that they conform to standards and will continue to work in a few years or on the cluster of your organization. I made sure my programs work with GCC and Clang on Linux and Visual C++ on Windows.

I know all of this sounds like tedious work, but it is all useful. You will see it in the end!

Modules

It is good practice to split the software into modules and reuse these. The idea is to have the components in a library with well-structured implementations and use these to build multiple programs. This prevents copying of code and enforces good interfaces.

In the diagram above you can see a library simcore, that is used by multiple executables in the row below in my project. This library contains

Multiple implementations of physical systems with different numerical implementation or dimensionality
Basic mathematical helper functions an
Many predefined laser pulses
Helpers for file output
…

The library has a few 1000 lines of code, while all the binaries are below 200 lines of code. If I am doing something in multiple binaries, I search for an abstraction and put it into the libraries. Trivial examples of this are writing a matrix to disk, showing the progress of the program, or compression of output data.

I solved this problem by building simcore as a static library that is linked to all the binaries, that there is no time wasted with compiling code multiple times. This is very easy with the build system CMake I use, but there are many more possibilities with different infrastructure.

The reason for having multiple binaries is to investigate your system in different environments. First, it is just the most basic thing possible and then you can add more interesting components. Of course, you can incrementally improve one program, but then you can’t go back to the basic program if you eventually don’t understand your complex problem and need to get back to the basics.

The only hard part is that you must sometimes make sure you don’t break your old basic programs while improving the advanced ones. You should build all of them regularly.

Automatic testing

These days it is usual to test software automatically. The classical approach is to test little parts of the software with so-called unit tests. These can be very helpful and Kent Beck has written an excellent book about these.

In my experience, there is another component to testing scientific software. The underlying models often have special cases with know (approximate) solutions, so you can run your general program to handle these cases and compare them to analytic results.

For these tests perfect is the enemy of the good. It is better to have an incomplete test case than none because it would be hard. Some example from my work are:

I looked at wave packets moving with some speed. So I initialized a gaussian wavepacket and look where its maximum value was after some time and checked if it had the expected velocity. There was no fit to get the exact center of the wave or ensure its shape. Just a check if a rough approximation of the velocity is in 10% of the expected value
There are complex formulas to predict the state of a two-level quantum system excited with different lasers. I just looked at the simplest perfectly tuned pulse with the simplest formula to see if it worked the same with my numerics

Now if you structured your program into modules, as shown above, this is all very easy to do. Your tests can just be another binary besides the other programs to test all the complicated stuff in the library.

Just run a simplified simulation, check the results in your program and print the deviation if something wrong and a green checkmark or smiley face if everything worked.

If at all possible make sure your tests can run in a few seconds. Otherwise, you might get frustrated to wait and stop using them regularly.

Configuration

Almost every scientific program needs many external constants to work. These might be physical constants, material parameters, setup constants, times, numerical parameters. Managing these in a well-structured manner can significantly help to keep an overview of everything at the time of development and even years later. For me, a good configuration system should fulfill the following requirements

Options should be named. Otherwise, you have to remember the 5. line is the speed of light and you will be confused eventually.
No information should be redundant/be copied. If you just want to run another simulation, you should not always have to reenter the speed of light.
Move some (simple) logic from your program into your configuration.

One solution to this is use a scripting language like Python, Lua or Scheme for configuration. In my experience, these are kind of complicated to integrate with a program, but otherwise I like this solution a lot and it is sometimes used. It might be possible to make it a little easier to expose your numerics to python by something like F2PY or Boost.Python.

I prefer a simpler approach to define my own simple configuration format. I base the format on .ini files, but variables can contain formulas and it is possible to include base files. A configuration might look like this:

Base.conf

c = 3e8

# space 
c = 300
Nx = 100
dx = x/Nx

#time
Nt = T/dt
dt = 0.98/c/sqrt(1/(dx^2)+1/(dy^2))

Sim.conf

{Base.conf}
T = 30
x = 500
pulse_width = x/5
pulse_energy = exp(5*pi*acos(T/4))

In my implementation (Sourcecode on github) I would use these file as

#define AFPCFG_IMPL
#include "avcfg.h"

typedef struct {
  double pulse_energy;
  double pulse_location;
  /* ... */
} config;

config load_config()
{
  config mycfg;
  avcfg_config *config = avcfg_load("Sim.conf", NULL);
  mycfg.pulse_energy = avcfg_getf(config, "pulse_energy");
  /* Give default value if the variable is not defined in the configfile */
  mycfg.pulse_location = avcfg_getf_op(config, "pulse_location", 0.0);
  /* ... */
  avcfg_free(config);
  return mycfg;
}

A nice trick is that you can now write the processed config (with only numerical values and no formulas in a single file) to disk with the output of your simulation. Here you can check if the values are as expected and you can rerun exactly the same configuration. Because it is basically a .ini file there is a parser for almost any programming language you could want to use for analysis.

Data storage

Simulations can create huge quantities of data. It is in most cases favored to save as much data as possible to do any imaginable analysis without having to rerun the slow simulations.

It took me a long time to figure out a nice structure to store that simulation output and derived analysis. I think in the end it would be optimal to have a versioned database that can run analysis in the background and dynamically move data between fast local storage to large network shares. But that would be a large system with many moving parts that would take many months of design and development from skilled programmers.

In the absence of that the best structure I found was something like:

/raw-data/ (Many gigabytes)
|- /Simulation run 1/
|- /Simulation run 2/
'- ...
/analysis-results/ (A few Megabytes)
|- /Simulation run 1/
|- /Simulation run 2/
'- ...
/plots/ (Some pictures)
|- /Simulation run 1/
|- /Simulation run 2/
'- ...

It might be counterintuitive to interleave the data of different simulation runs, but it has some nice properties.

The raw data can live on a slow (high latency) storage elsewhere, but it is trivial to just copy the analysis result onto your local computer to make plots faster.
You can also take the analysis data onto your laptop for a conference or home office where you might not fast (or any) internet.
If you want to write a paper you can simply copy all the plots folder without putting too much irrelevant stuff next to your Latex documents. If you redo your sim/analysis it is trivial to replace them without confusion and messing up.

This exactly replicates my description of time scales from above and also the different iteration numbers. You will presumably, do much more iteration on your analysis than your simulation and hopefully even more iteration on your plots!

Numerics and algorithms

This is the hard part you have to solve and I can’t help you. Sorry.

Neat tricks

At last, I will share a few little ideas you can integrate into your program without too much hassle to make life a little easier.

Floating point precision

Almost any computer you will use has 32bit (float, single) and 64bit (double, full) floating-point numbers. I science most people use 64 bit for more precision and fewer numerical problems. This is mostly a good idea, but in most cases these numbers are (kind of) twice as slow. This can be rather time-wasting while developing your program. Sadly most compilers don’t let you switch automatically, but in C and C++ you can build that yourself easily. For example, I use:

#if 0 /*set to 1 for full precision */
typedef float F;
typedef std::complex<F> C;
typedef Eigen::ArrayXf VecF;
typedef Eigen::ArrayXcf VecC;
#else
typedef double F;
typedef std::complex<F> C;
typedef Eigen::ArrayXd VecF;
typedef Eigen::ArrayXcd VecC;
#endif

And you get very short type names for free!

Also, see if -ffast-math or something similar can speed up your test runs in development.

Simulation progress

Sometimes you wait eagerly in front of your running simulation for results. I build myself a little progress reporter for the command line. It can show percentage progress and extrapolated time remaining, so you know if there is enough idle time for making tea or even lunch at the moment.

My systems need to know the number of the total simulation steps at the beginning of the program and has to be notified after every step. To not use too much computation time I print the information only if 5 seconds elapsed or 5% of total progress were made. There are many more possibilities how you can make this more awesome, but this was enough for me.

config.ini   0.1% done in 0:06. Expected remaining time: 75:51
config.ini   0.3% done in 0:12. Expected remaining time: 76:34
config.ini   0.4% done in 0:18. Expected remaining time: 76:32
config.ini   0.5% done in 0:24. Expected remaining time: 76:04

My (not that advanced) implementation of this idea is on GitHub.

Program version in data

At the beginning of this article, I told you about some boring infrastructure stuff and now it is time to build something cool with it!

If you use version control your code always has a revision ID (kind of version number). I found on the internet how to always put that ID into your code and resulting executable with my build system. Now I told my program to always put a file with this revision number next to the output files.

If you also saved your configuration file as I did you now know exactly what program and what configuration you used to generate a simulation output. With this, you can trivially and with confidence reproduce the output if you mess it up or investigate if it turns out to be strange a few months later.

This might be useless in most cases, but it also might save you from a few sleepless panicked nights before a deadline.