Reading Advice: All sections up to the “My Billboard Phrase” are mostly my opinions-on/history-with statistics. The billboard is a bit of personal experimentation
Disclaimer: I only have a small sample size that I am drawing some conclusions in this post from.
Quality Rating: 8/10
All models are wrong
But some are useful
“All models are wrong, but some are useful” is an aphorism from the legendary statistician George Box. Those who know me know of my fondness for it, and how it basically defines my working life, providing a never-ending source of inspiration to me as the ultimate goal of scientific/statistical modelling. It evolved over time into its modern form in a fun way too (see wikiquote and this 2025 review for its rich history and origins), I think more quotes should have their history tracked in this way, perhaps even with a family tree with Levenshtein distances1 to see how many small/large “mutations” there have been in its history for being said/written (A clickbait headline for this type of analysis could be “are some memes more adaptable than others?”).
(Not) My Variant
I personally developed a fun variant: “all models are wrong, but some are more wrong than others”. I think it can be a pithy retort when I detect people (ab)using the original quote to attack the idea of statistical modelling in general. I long for an opportunite to use it to attack a rival’s dodgy statistical methodology (to do that I would need to attain a nemesis, which I have yet to do and doubt I ever will2).
However, upon my moment of synthesising it3 I discovered that the quote has previously appeared on the internet. My internet archiving skills have since dredged up this use in the philosophy-of-mind world in September 2016, but that does not have the exact same meaning. What I believe to be the first statistical use was in late 2016 from Dr. Anna Fergusson (then at the University of Auckland): https://new.censusatschool.org.nz/resource/all-models-are-wrong-but-some-are-more-wrong-than-others-informally-assessing-the-fit-of-probability-distribution-models-as91586/ . To the best of my knowledge I came up with it independently, but this is the first time I am revealing it in public print. There has also since been a published paper with the title “All models are wrong, but some are more wrong than others” too.
I think it is interesting to ponder whether these separate uses arose independently. Often in history (at least, history of mathematics, about which I am most familiar) there are examples where an idea appears to be “in the air” and you get seemingly independent discoveries of the same thing by different people. This appearing in 3 reasonably unrelated fields (and possibly twice within statistics) within a decade of each other appears to be coincidence as best I can tell, and given Box and Orwell are both famous enough and have been dead long enough it is curious that the fusion of their famous quotes has only started occurring recently. I mildly worry about selection bias of old internet sites being discontinued or not being easy to find nowadays and the true first-originator is unknown.
Other Aphorisms
What may not be more widely known is that there are other witty and insightful statistical aphorisms out there. Some I’ve assembled/found over the years include (but are not limited to):
- All models are wrong, some are wrong in ways we can measure
- All models are wrong, but hindsight can’t even be wrong
- All models are wrong, but some are profitably wrong
- Models don’t remove uncertainty, they repackage it
- A result without error bars is just a bedtime story
- Models are scaffolding for uncertainty, not blueprints for truth
- Every model result is guilty until proven useful
- Model code can be like a sausage - you don’t want to see how it’s made, but it can be tasty once cooked
- Models are like umbrellas: useless in a hurricane, but handy in the rain4
- A model can be like a compass, not a GPS - it points, but never pinpoints
- A model’s job is not to be right, but to be less wrong than guessing
- Models are like jokes: if you need to explain them, they’ve failed5
- All model assumptions are violated, but some violations are fatal
- Every dataset can tell a story, but most are fiction6
My Billboard Phrase
But my personal favourite is not so much an aphorism but a little fable, that I came across soon after becoming a professional statistician:
“Don’t use statistics like a drunk uses a lamppost: for support, rather than illumination.”7
I can do no better than link to the quote investigator to trace its history (it seemed to be all-the-rage in 1937). Personally, of the variants I most prefer the one above.
I love this quote.
I love it because the analogy of statistics shining a light in the darkness of the unknown feels visceral to me.
I love it because it implicitly suggests the importance of increasing the brightness/coverage of the isolated islands/lampposts of scientific knowledge, and building torches/flashlights to bridge between them8.
I love it because the inverse square law of light evoked has some deep mathematical principles and qualitatively reflects my experience in statistical research that bringing more light to a problem can bring realisation that there are more areas in near-darkness to explore, and that it is a superlinear task to fill all the space in the world with the amount of light I’d like to see.
And I love it because I also see the laughable literal meaning. Possibly the first thing that comes into people’s minds would be a politician, who might have a bad reputation for cherrypicking the facts/statistics that support their views. Personally, I see it as a problem shared across wider society, and perhaps all humanity. To quote this excellent book review, “Of the fifty-odd biases discovered by Kahneman, Tversky, and their successors, forty-nine are cute quirks, and one is destroying civilization. This last one is confirmation bias - our tendency to interpret evidence as confirming our pre-existing beliefs instead of changing our minds”. Confirmation bias is the heart of this quote, it is extremely easy to come to seek out things that confirm your prejudices when analysing a dataset. I understand that for plenty of members of the public the claims of you can say anything with statistics resonate, but it feels alien to my day-to-day life as a practicing statistician where I try to find out ways in which my or others’ preconceived ideas are wrong. To me, the statistical method is a way of discovering truths from data, and quantifying the limits of what we know, and I think this quote sums it up better than any other analogy could.
My Personal Statistics Billboard
A few months ago I lost a cotton tote bag that I had had for over 10 years. I realised that I was actually quite attached to it even though it was fraying, it was branded with a recycling message from a local government organisation so to think of it and its contents out there in the world going unrecycled gave a grim sort of irony.
It then reminded me that at university careers fairs and freshers fairs I always found the cotton tote bags useful, and remembered their messages - they were always branded. Branding objects you carry around with you is a bit like a personal billboard, and it rekindled a desire I’ve long harboured to have more humour in my day-to-day life. I had long wanted to get a doormat that says “garbage in, garbage out” (a colleague had once had the phrase put on a poster on a door to an office), and I once bought a friend a plain white t-shirt for the birthday with “maths puns, they’re the first sine of madness” printed on it.
So now was the time to combine my love for the billboard phrase with a practical bag replacement. One of the upsides of the current incineration of cash by the AI behemoths is that custom image generation is basically free, so I (with surprising difficulty) managed to get an image of the lampposts I am used to seeing along UK streets9 casting a small amount of light into the inky night. I then used a basic image altering tool to add the text I wanted. It was eerily easy to then attach it to a custom bag printing website (plenty offer discounts for a large bulk order, I don’t think many people have had my inspired idea of personalising their bags with words/images that mean a lot to them yet), and I paid a similar sum to getting someone else’s branded bag.
Since receiving it and using it roughly daily for approximately one month, I have received a few sly grins and compliments from statistically-minded colleagues. I have also noticed at least two people walk roughly in-step with me looking at it, as well as people glancing at it as they pass by on trains. Even at these relatively small samples I feel confident in saying it gets attention.
Who knows, maybe one day a stranger may even initiate a conversation about it, and if so it will have paid back the time/money investment by more than an order-of-magnitude: I place great value on finding fellow statistically-minded people.
Is It Worth It?
If you are regularly walking around in major urban centres or on major public transport routes, then despite the feeling of invisibility, you do actually occupy a non-negligible portion of the field of vision of lots of people. Hence we see billboard advertising in roads or inside trains, and occasionally companies even pay people a tiny wage to sit/stand with a small advertising sign at major pedestrian congregating points. In the UK, advertising on a train into London costs £300 per square metre for one month according to a basic LLM (actual data would be welcome), so a simple cotton tote bag covering about 0.1 square metre of message over the analogous period of time (assuming you spend 2 hours per day on trains/“in public” with your bag) is eminently worth it over 1 year compared to the extra marginal printing cost of the design on the bag. Indeed you being a moving object and an actual person might arguably mean people pay more attention to you than typical advertising boards. Criticisms of this overly simplistic calculation would be greatly appreciated!
Recommendation/Open Offer
If you think you have something meaningful to say to people, I heartily endorse designing a bag.
If you send an email to leptokurtains@gmail.com with the subject “bag” then I promise that out of my own pocket I will pay for the production of this particular bag, and maybe also even pay shipping to a location of your choice if you would like a different statistical aphorism that is new to me and I enjoy.
Post-Script
It would be remiss of me to have a post containing Box’s aphorism without linking the wise words of Prof. Andrew Gelman on the matter .
Footnotes
Or more broadly an appropriate edit distance for semantic meaning.↩︎
The closest I may ever get is how I imagine Nate Silver felt upon seeing his now-extinct 538 brand being taken over by someone with rival election modelling methodology, https://www.natesilver.net/p/why-i-dont-buy-538s-new-election and https://www.natesilver.net/p/polling-averages-shouldnt-be-political are I believe the two most relevant public posts. https://www.natesilver.net/p/sbsq-10-everything-you-always-wanted also contains some background but may be behind a paywall.↩︎
At 17:20 on the 18th February 2021 in an internal work chat to a colleague, to be precise. I do not have permission to share evidence.↩︎
This might be my favourite, a nudge towards a little-talked-about phenomenon known as context drift.↩︎
I don’t think I personally agree with this one, but also think it’s funny.↩︎
I think more quotes along these lines should be created - wariness to overfitting is a common theme in my professional life that I’d like to see more of in the world where humans are inclined to see pattern in noise like some of the examples on the pareidolia wikipedia page.↩︎
The precise wording that I was prompted with fresh out of university was “An unsophisticated forecaster uses statistics as a drunken man uses lamp-posts - for support rather than illumination”, attributed to Andrew Lang. I prefer not to use this particular wording and do not endorse the attribution really either.↩︎
I vaguely recall seeing this analogy somewhere online, and I think Start Under the Streetlight, then Push into the Shadows seems to be the most relevant that I can find, it might be the comments underneath this reply to the post that I remembered.↩︎
The classic inverted “L” shape, which was the key to getting the image generator to stop giving me vertical “old-school” lampposts from the Victorian era.↩︎