I just finished reading Uncontrolled by Jim Manzi. He starts the book spending a surprising amount of time in the early chapters of the book trying to give the reader a crash course in the philosophy of science. The TL; DR is that I’m rather lucky to be a physicist. The longer answer is that in the hard sciences such as physics and chemistry, we often have basic models of how the world work which we constantly test against new observations. If we’re lucky, we find that our models don’t describe something in nature and that there is new science to discover. But, at a deep level, we have a few articles of faith. For example, we believe that the laws of physics don’t change from place to place. We also believe that the laws of nature don’t change with time (even if our understanding of them does). Other fields such as medicine are not so lucky. We still have a very rudimentary understanding of the human body. We don’t have strong enough models to say whether a given compound should cure a given disease. There are also many confounding factors. So, randomized trials are necessary to see whether the effects observed are intrinsic or accidental. Even if a randomized trial indicates that an effect is present, it can be difficult to generalize it to a different context or population. Sociology and economics are even more difficult. The author brings up a number of cases in economics where authors have overreached and made predictions that really should have been tested in randomized experiments—otherwise, they are far too overreaching with too little support from data.
After this introduction, Manzi delves into perhaps the strongest section of the book where he outlines how he used randomized experiments in business to determine strategy. When doing this, he looked into the literature of random trials in medicine, as well as random trials in social sciences (for example, leading up to welfare reform in the 90s, the federal government required a number of random trials for states which wanted to try different strategies.).
There is another section of political trials, which reminded me of Rick Perry and His Eggheads and the better Get Out the Vote, which deal with experimental tests in politics of what strategies work and which don’t in get out the vote efforts.
Perhaps the weakest section of the book is where the author attempts to make policy suggestions. But overall, I would recommend reading this book. If nothing else, it is helpful in increasing skepticism of predictions which lack experimental support.
