About thirty years ago there was much talk that geologists ought only to observe and not theorise; and I well remember some one saying that at this rate a man might as well go into a gravel-pit and count the pebbles and describe the colours. How odd it is that anyone should not see that all observation must be for or against some view if it is to be of any service! —Charles Darwin
Imagine you're contacted by the FBI and asked to help solve a difficult murder case. You have full access to the evidence they've collected so far. What approach would you take in trying to structure your investigation, locate a suspect, and build your case?
If you're anything like me, you'll begin by going over all the facts you have so far and freely imagining scenarios that would be consistent with all the available evidence. As your brain is busily generating scenarios, going through all the permutations of different possibilities, you'll begin ruling out all the scenarios that are patently ridiculous and prioritizing the rest by likelihood, maybe without even being consciously aware of it. Otherwise, you would be swamped in a heap of every possible scenario your mind could concoct.
Many people seem to believe that hunches are unscientific, and that science is hostile to or incompatible with hunches. Nothing could be further from the truth! A scientist's finished work, a published paper, should be as free as possible of hunches and speculation, but the scientific process would be starved and impotent without hunches to feed it.
Creativity plays a much bigger role than generating an array of hypotheses to test. It's imbued into every step of the process: it takes nothing short of genius to design a scientific study that properly controls for every possible way the evidence could be tainted. It takes an active imagination to visualize all the ways your swarm of preconceptions could sneak into the data and morph it into an unintentional deception (maybe even a profitable one). A scientist is like a werewolf waiting for the moon to transform it, trying to outsmart itself and keep the inner beast chained through the night. It takes cunning, not just a dry, emotionless commitment to scientific rigor, to be a good scientist. You might say science is an art, not a science.
But hunches alone will be just as worthless: pure emotion-laden preconceptions. The true value of hunches is when you give them enough slack to guide the scientific process, but not enough to compromise it. Intuitions are quick and powerful tricks the brain has developed to come to reasonably good answers. The scientific method is a system to check those reasonably good guesses and flesh them out in such a way that you (and others) can check your brain's work. I like how Robert Pirsig describes it:
When I think of formal scientific method, an image sometimes comes to mind of an enormous juggernaut, a huge bulldozer…slow, tedious, lumbering, laborious, but invincible.Put hunches and scientific controls together, and you have a recipe for optimal problem solving, steadily ratcheting your way from existing knowledge to new knowledge.
