Underestimated analytics practices people hate
This is for everyone.
We’ve all seen great practices for writing clean code, formatting SQL and creating visualizations. They’re shared in blogs, discussed in forums, signed, sealed, and delivered. But at some point, technical mastery alone isn’t enough — especially when your work involves people. Also, let’s not forget: AI is supposedly coming for your job. And mine (so they say).
Humans, unlike code, are unpredictable and can be insanely wrong. We bring biases we can’t fully account for, turning every “truth” in analytics into something written in stone, and not in pencil. That’s why we need to think carefully before hitting the “act” button.
When we do, we come up with standards or “best practices.” Then someone else comes along and adds their two cents. So here I am, adding my current deposit to that collective bank.
Attention: these practices are for everyone.
Be the Sherlock of skepticism, but stay humble
Analytics, at its core, is about uncovering “somewhat truths”, but also acknowledging that those “somewhat truths” can shift. Every model, every hypothesis, every interpretation is prone to being wrong. Thus, the best analytical people aren’t the ones who claim to have it all figured out, but those who are humble enough to admit they might be wrong.
When one metric paints an overly positive picture, is there another metric painting the shadows? When an A/B test shows a 3% higher conversion rate for the treatment group, could there be an issue in the setup? And when the data looks off, do you press forward with conclusions anyway?
These scenarios create the risk of overconfidence in analytics. And that is not all. It isn’t just the initial blindness to errors, but the cumulative effect of that overconfidence over time. This becomes problematic in interconnected systems, such as organizations where teams depend on each other’s insights. A flawed conclusion in one department, such as marketing, can lead to taking wrong actions in product development. It spreads, and one cannot control it once it’s out. This is especially true in presentations with conclusions that seem too certain.
The main danger of overconfidence in analytics isn’t just the initial blindness to errors, but the cumulative effect of that overconfidence over time.
Stop and make a Goddamn decision
Data is overwhelming. That’s why we outsource it to a team of data people. The same goes for analysis. But the questions? Those don’t get outsourced. Stakeholders often fall into an endless loop of questions, trying to validate every possible scenario before moving forward. Eventually, they hand off the analysis. “Could the user’s country affect churn behavior?” “Do new users tend to churn more?” “What if…?”
What’s the harm in asking questions? Isn’t that the essence of analysis? Sure, but time is finite — and so is the money spent on analytics that doesn’t deliver business impact. At some point, someone has to stand up and say, “It’s time to make a decision.” Decisions don’t need perfect information; 60%–70% confidence sometimes can be enough. Businesses can’t afford to wait for perfection, or they’ll fall behind.
In a multi-dimensional world, there’s an infinite number of questions you could ask. But what makes you remarkable? Knowing when to stop asking and start deciding.
Get your duck and talk
The concept of “rubber duck debugging” process is probably as old as language, even though the term originated in the 1999 book “The Pragmatic Programmer” by Andrew Hunt and David Thomas. They described a programmer debugging their code by explaining it line-by-line to a rubber duck they carried around. The essence is simple: explain your problem in as much detail as possible — as though to a non-expert — to better understand it yourself. Your “duck” can be anything: a toy, your pet, or even a coworker. You choose your weapon.
The power lies in forcing yourself to be simple but also the listener to listen. This applies to analytical thinking as well — it’s not about having the answer right away, but about organizing your thoughts and testing your assumptions.
Let’s say, while “explaining to the duck,” you can start by sharing the assumptions behind your analysis. “I assumed that user engagement drops are only affected by site speed.” Hearing it aloud can help spot hidden biases or gaps in your logic. Next, you could test your methodology, by explaining each step of your analysis to the duck. For instance, “I grouped the data by month and calculated the average retention rate. Then I compared it year-over-year.” Doing this can reveal errors, like forgetting to account for outliers.
This is not only good for analysts, but again, for everyone. Are you a decision-maker? Is there a decision that needs to be made? What happens if you don’t make it? Why does it matter? Tell your rubber duck.
Have a couple of options open
When analyzing data, it’s easy to go with the first reasoning for why a certain behaviour happens. Let’s say we notice a drop in conversion rate on our platform. There could be a million reasons for it — slow loading times, marketing campaigns, incorrect tracking, and so on. As humans, we don’t have time to consider every single possibility. But what we do well is coming up with what I’d call “candidate reasons.” These are our quick guesses, and I’ve already mentioned a few of them:
Can it be that slow loading times caused the drop in conversion rate?
Can it be that marketing activities caused the drop in conversion rate?
Can it be that both contributed to the drop in conversion rate?
… and so on.
On paper (or on screen), these all seem pretty reasonable. But here’s where we run into trouble: too often, once we validate our first candidate reason, we stop there. “Here it is, the truth — we figured it out right from the start.” Sounds a bit too smooth, doesn’t it? Availability and anchoring can do wonders to our minds.
If you’re a chess fan, the concept of “candidate options” or “candidate moves” also appears in chess, and was shortly explained by Kotov in the book “Think like a Grandmaster”.
Test, not validate
If you read the previous point you could have noticed that I used the word “test” and “validate”. They seem close in meaning but they are not. These two views come from different lines of reasoning.
Testing allows for the possibility it could be proven false (falsifiability), whereas validation simply checks if the hypothesis meets specific conditions or criteria. Still too close? Let’s dive it a notch. Testing involves forming hypotheses and experimenting to prove or disprove them. Validation, on the other hand, is about confirming whether something is true. So validation can lead us down the dangerous path of confirmation bias, where we only seek to confirm what we already believe. That gets worse when sourcing your information via genAI that’s set to give you always an answer, even if it’s wrong. Humans are wired to prefer certainty over ambiguity. It’s why even analysts may prefer validating their initial hypothesis instead of testing it. Testing requires actively looking for ways a hypothesis might fail, which takes time but ultimately leads to more robust conclusions.
Take a classic example: drug development. Scientists don’t validate a drug by finding only the conditions where it works, they test it under various scenarios to see where it fails. So when analyzing a dataset, ask yourself, “What would it look like if this hypothesis were wrong?” or “What might I be missing?”
Avoid your own hallucinations. Test, please.
When in doubt, “suspect” not “conclude”
The way we use language has always been important, but with the rise of GenAI, it’s become even more critical. We’ve always relied on words, but now we’re more aware of how they shape decisions. Words can lead to bad or great decisions, so we all need to be a bit careful with that. Example?
When presented with data, the right approach is not to be sure of the answers but to remain open to a variety of interpretations. Use cautious language — avoid the certainty of “we conclude” or “we know.” Instead, frame your findings as hypotheses, ideas, or possibilities. The more interpretations you can generate, the better. Remember, data often tells us what might be true, not what is.
This rule applies to everyone, not just the messenger but especially the decision-maker. The decision-maker interprets the message and delivers the decision. What we want is to avoid too much interpretation. For that, it’s also the decision-maker's job not to “conclude” when the data consultant clearly says “We assume”. While we all aim to deliver certainty, it’s better to accept uncertainty and simplify, rather than trying to act like management consultants.
Until then, “suspect,” do not “conclude.”
