From this 1981 interview:

So when did the loss of certainty begin? Where did we take a wrong turn?

It began around 1800, and it began with geometry. I usually like to quote Mark Twain about this. He said that man is the only animal that has the one true religion—several of them. And that is just what happened with geometry.

The geometry that came from the Greeks is usually called Euclidean geometry, after Euclid. But suddenly at the beginning of the 19th century other geometries were developed—non-Euclidean geometries. Who gets the credit for this is sometimes disputed among historians, but I would say Carl Friedrich Gauss. He was the man who said flatly that we can no longer be sure that Euclidean geometry describes the physical world correctly. The various geometries conflict, although one of them, according to thousands of years of tradition, should describe the truth. You can see the problem.

Can you give me an example of an alternative geometry?

Well, one can cite as an example the theorem of Euclidean geometry that the sum of the angles of a triangle is one hundred eighty degrees. In one of the non-Euclidean geometries, called hyperbolic geometry, the sum is less than one hundred eighty degrees; in another, called double-elliptic non-Euclidean geometry, the sum is always larger than one hundred eighty degrees. Yet all of these geometries are equally accurate insofar as man can measure the sums of angles of triangles.

I actually disagree with Kline on that point! Draw a (virtual) triangle connecting the North Pole to two points on the Equator that are 90 degrees apart in latitude, and each angle of that triangle will be 90 degrees. 90 + 90 + 90 = 270: we can measure that.

The interview continues:

What did the mathematicians do when the bottom dropped out of geometry so to speak?

Many mathematicians tried to rescue and maintain as truths the portion of mathematics built on arithmetic, which by 1850 was far more extensive and vital for science than the several geometries. Unfortunately, other shattering events were to follow. Arithmetic and algebra were the next to go by the board.

The best example of this I could give in a semi-popular book was the creation of what are called quaternions, in 1843, by the great mathematical physicist William Rowan Hamilton. Now in the algebra of quaternions, a kind of number known as a hyper-number, multiplication is not commutative. In other words, if I were talking quaternions, I could not say that three times four is the same as four times three. Other strange algebras were created, and it made people start to worry about the laws of ordinary arithmetic. (The one I just stated is known as the commutative law of multiplication). And if we can have perfectly good algebras in which the old familiar laws don’t work, then how do we know they work in the case of the real numbers? That’s where a mathematician named Hermann von Helmholtz stepped in and told us we don’t know it at all. They work in some situations, but not in all.

Are there any elementary examples of these sorts of algebras, where 2 + 2 = 6, or where 5 x 7 = 35, but 7 x 5 is only 34?

I can think of several. Take a quart of water at 40 degrees and mix it with another quart of water at 50 degrees. Do you get two quarts at 90 degrees? You do not. It’s more like 45 degrees. So you can’t just say I’m going to add 40 and 50 and automatically get 90. It depends on the physical situation.

Consider music, a simple musical tone with a unique frequency and amplitude, say one hundred cycles per second. Now suppose on top of that you impose another note at two hundred cycles per second. Do you get a note at three hundred cycles? Again you do not. It is a note of two hundred cycles, the first harmonic above the one-hundred-cycle note. It is the highest harmonic that determines the pitch—two hundred cycles. This is an important factor in the design of musical instruments.

Those are excellent examples. And recall that the laws of probability do not apply in real life (that is, quantum mechanics; see section 2 of this article).

This is interesting:

If mathematics has no underlying truth–if it is filled with contradictions and uncertainties, why does it work?

There is no definitive answer to that. It just works. The only test we have that mathematics is reliable–not certain, but reliable–is that one can apply its laws to physical problems and make predictions. If the predictions come through, then we can say that mathematics has some substantial basis, but not certainty. I think people can’t help being impressed by what mathematics achieves. Consider the problem of sending a spaceship to the moon and bringing it back. It is entirely mathematical. Of course, there is a tremendous amount of engineering involved in the production of the ship, but the entire plan for it is mathematical. We have a theory about the sun, the planets, and more distant heavenly bodies. We say that what makes them behave as they do is the force of gravity. But nobody knows whether there is such a thing as gravity. We have no physical understanding of it. The theory is mathematical–gravity is a scientific fiction.

The same could be said about electricity and magnetism, couldn’t it?

That’s exactly right. Everybody today knows what a radio is, and what a TV is, but nobody knows what a radio wave or a TV wave is. You can’t smell one or hear one or taste one. But we do have a wonderful mathematical theory developed in the nineteenth century by the mathematical physicist James Clerk Maxwell. The evidence for this wonderful theory is the performance of our radio and TV sets. So we have to accept the fact that mathematics works, or else abandon our radios and our TV sets.

I pretty much agree but I’d put it slightly differently. There are various mathematical theories that don’t work, and because of that we don’t use them to design radios and TV sets. To put it another way, it’s not quite right to say that “mathematics” works; rather, some branches of mathematics work. For example, you could think of various goofy variants of logic and probability as mathematics, but nobody’s using them to build ships. Or, for another example, physicists don’t use classical (“Boltzmann”) probability in quantum problems, because . . . it doesn’t work. The mathematics that works, that’s the stuff that works. Any bit of mathematics works until it doesn’t, which is the point where people try to push it beyond its bounds of applicability.

I enjoyed this bit:

Are most mathematicians since the loss of certainty now working on these physical problems?

No, they aren’t. Most of the mathematics created today–maybe ninety percent of it–is a waste of time. That is an opinion, but one that authorities who are far more creative and far better known share with me.
Can you give us an example of mathematics you consider a waste of time?

Some problems now being considered in the theory of numbers, for example, are a waste of time. Take pairs of primes, called double primes. These are prime numbers in a sequence, eleven and thirteen, for example. No even numbers, of course, are primes. Are there an infinite number of these pairs? Are there triple primes? Endless papers are written about these subjects. Who cares?

That’s how I feel too! That said, I understand that the sorts of insights required to solve this sort of number theory problem are cognitively similar to the sorts of insights required to solve what I would consider to be interesting and important problems in mathematics and statistics. So, even though I agree on “Who cares?”, I don’t think that research in this area is “a waste of time,” any more than it’s a waste of time if you’re an athlete to do cross-training.

The interview continues:

It makes mathematics sound a lot like playing chess or bridge. Exciting, beautiful, challenging; the same words apply to all three kinds of activity.

That’s right. I’m glad you suggested it because it makes the point sharper. People enjoy playing chess. Some people even devote their lives to it. But no matter how ingenious a man is at playing chess or bridge, it isn’t going to change this world one iota. Now mathematicians may probe deeper problems, but it is the same thing.

Again, I kinda feel that Kline is missing the point. For one thing, the effort spent to build programs that can win at chess and go has led to general improvements in machine learning and AI. For better or worse, the study of chess has changed the world, and by more than one iota.

Overall, I’m a big fan of Kline and I like a lot of what’s in that interview, which is one reason it’s interesting to see where we disagree.

Using experiments on soil mites, we show that sexual selection associated with an armed and aggressive male phenotype can reduce population size and stability, which lowers their resilience against acute heat stress. These effects linked with armed and aggressive phenotypes underline the importance of sexual selection in mediating population dynamics and resilience to environmental change. ABSTRACT Population recovery following environmental stress is known to depend on demographic structure, life‐history and evolutionary dynamics. However, it is unclear how traits shaped by sexual selection affect population dynamics and recovery. We examined this by manipulating presence/absence of males expressing either a non‐aggressive ‘scrambler’ phenotype or an aggressive and lethally armed ‘fighter’ phenotype in soil mite populations of different size. We experimentally altered the male phenotype in populations, subjected them to heat stress, and analysed their population dynamics and recovery. We show that populations with fighter males exhibited (i) reduced population size and stability, (ii) greater decline in response to heat stress in larger populations, (iii) higher rate of growth and (iv) incomplete population recovery. Such reduced population stability and recovery linked with armed and aggressive phenotypes underlines the importance of sexual selection in mediating population dynamics and resilience to environmental change with implications for managing natural populations.

This post is from Bob.

I used to work in natural language semantics, and the following dialogue from Lewis Carroll’s Through the Looking Glass, and What Alice Found There was the most common pull-quote to see at the beginning of a thesis.

“When I use a word,: Humpty Dumpty said in rather a scornful tone, “it means just what I choose it to mean — neither more nor less.”

“The question is,” said Alice, “whether you can make words mean so many different things.”

“The question is,” said Humpty Dumpty, “which is to be master — that’s all.”

Humpty Dumpty came to mind recently after a spate of discussions with ML folks about inference (i.e., what they call “learning”).

What in the world does “empirical Bayes” mean?”

Empirical Bayes came up on Wednesday with some ML folks I was talking to, then I ran into Dave Blei this morning, who told me he’s giving a sequence of talks on empirical Bayes over the next few weeks (at Columbia and at University of Chicago). I asked him what “empirical Bayes” meant to him, because it seems to be used very fluidly in ML. He gave me a new usage, saying he used it for any model that uses data to fit parameters of a prior, including just plain old hierarchical modeling fit with sampling. Dave gave the example of ARD in Gaussian processes (aka, hierarchical models).

I would only use the term in the way described in the Wikipedia entry for “Empirical Bayes”, namely

… empirical Bayes may be viewed as an approximation to a fully Bayesian treatment of a hierarchical model wherein the parameters at the highest level of the hierarchy are set to their most likely values, instead of being integrated out.

I pinged Mark Goldstein, one of our top-notch ML postdocs, and he pretty much reeled off the Wikipedia definition. So there seems to be a lot of variation in how this is used.

Robbins on Empirical Bayes

Dave also pointed me to the following video by Herbert Robbins (yes, that Robbins, who was so far ahead of the computational statistics curve that he introduced stochastic gradient descent and multi-armed bandits in the early 1950s).

• Herbert Robbins. 1990. Origins of empirical Bayes. YouTube!

Terminology drift in ML

I’ve been a bit shocked at how many technical terms have drifted in meaning in ML. I’m not talking about people making honest mistakes or clueless mistakes, I’m talking about true drift in meaning where the ML folks will stand by their definitions.

Likelihood: I’ve seen “likelihood” used for what I’d call the data generating distribution and sometimes just as a synonym for density. Aki has this one covered in his recent post, A data model is not just a likelihood. In stats, the likelihood is defined as the function L(theta) = p(y_obs | theta)—that is, it’s a function of theta for some fixed observed data.

Causal: With the advent of LLMs, anything with an autoregressive structure is now being described as “causal” in a “past causes the future” sense. This is even being extended to arbitrary directed graphical models, which are now being called “causal” even when there’s no explicit causality being modeled. That is, you can now describe a simple regression from an observational experience as “causal” with no extra work.

Estimation: Estimation is almost always called “learning” in ML.

Parameters: These are usually called “weights” for neural networks, which I think now make up 99.9% of all work in ML. But if you tell ML folks that neural networks are parametric models, they’ll most often deny it. A statistician would confusingly call a neural network a “non-parametric model” and then tell you that means it has a lot of parameters.

Inference: In our diffusion model reading group, the ML postdocs tell me that “inference” means what I would call “posterior predictive sampling”. For example, generating output to a query from an LLM would be called “inference.”

Bias: In statistics, this usually means expected error. In ML, it’s heavily overloaded. It can be used to name just about any kind of error measure (e.g., errors in Matt Hoffman’s sampling papers). ML folks also use the term “bias” to mean the intercept in a regression (no, I’m not kidding).

Prior: This is often called an “inductive bias” in ML circles, which can include aspects of data generating distributions as well as priors.

In Bayesian statistics, a prior is the marginal distribution over parameters. In ML and informal presentations of “Bayesian statistics,” it’s just any marginal that gets plugged into Bayes’s rule. For instance, the prevalence of a disease p(disease+) is called the “prior” when I evaluate positive predictive accuracy p(disease=+ | test=+) given a testing sensitivity distribution p(test=+ | disease=+). In a k-way classification, “prior” means the marginal distribution over categories.

Bayesian: I think in ML this term is used very broadly for any situation in which there’s a prior not strictly stated as a penalty function for penalized learning (i.e., regression). I think Andrew’s down with this definition as he also uses Bayesian for anything that looks vaguely Bayesian no matter how inference is performed. For example, Empirical Bayes is just Bayes to Andrew, as is using a Laplace approximation or even a simple maximum likelihood estimate (just think of it as a one-point posterior summary!).

Regression: Note quite on topic, but I think of neural networks as just a GPU-friendly form of non-linear regression.

Uncertainty quantification: This is the primary subject of statistics, though I think the term “uncertainty quantification” is much more prevalent in engineering/signal processing than in ML. There are even journals of that title that look sort of like statistics journals.

I’d find reading ML papers easier if there was less meaning drift from well-established terminology. I’m not saying the ML folks should be up to date with Gelman’s idiomatic statistical lexicon (the concepts are fun, and it can be useful for talking to Andrew and people in his circle like the blog readers, but I wouldn’t recommend using these terms in papers without explanation.

I’m sure there are many more terms have been coined or have drifted one way or another that I’m forgetting about.

The latest annual account of greenhouse gas emissions shows Canada's emissions reductions slowed in 2024 to almost nothing.