In baseball, of course, this means things like:
The Mariners, predicted to be terrible this season, are tied for first in the league!

(Also there are nineteen pitchers tied with an unbelievable 0.00 ERA. This season looks set to turn a lot of conventional wisdom on its head.)

We see Small Sample Size Theater in other domains as well; no surprise that most trend reporting is of this type. I wanted to post this today because I think the term is so apt. And of course, if my posting this year keeps up at this rate, I’ll post well over 300 entries, which would more than double my previous record. In year seven, anything is possible.

This superb book is a collection of fifteen interviews with well-known and highly-regarded programmers (Norvig, Armstrong, Knuth, etc). Siebel (author of Practical Common Lisp) is a professional programmer with a keen sense of the (brief) history of the profession. This gives the interviews a depth and a richness that even a clever journalist could never have matched. Siebel is a consistently thoughtful interviewer who asks just the right mix of questions. In any one interview, the questions range from practical ones concerned with how the subjects debug code to more general questions about whether the nature of programming has changed over time. Across interviews, Siebel asks enough of the same questions that we can start to view the answers in comparative perspective, while also allowing what is special about the careers and interests of the subjects to emerge.

In short, if you’re interested in programming, this book is wildly engrossing. A word of warning: If you don’t have any experience programming, and some background knowledge of the field, you’re probably not going to be able to get much out of the book. Some passages were certainly over my head, as I’ve only been a professional programmer since June, when I got my green card, and if I recall correctly, only really got started teaching myself Python about a year and a half ago. But most of it was accessible and inspiring to this junior programmer.

Amy Sohn. Prospect Park West

We lived briefly in (very South) Park Slope when we first moved to Brooklyn, and although we’ve since moved out to Flatbush, we’re back in the Slope all the time. We eat at Al Di La whenever we can afford to. We’ve been members of the infamous Park Slope Food Coop for several years now, and we’re set to have a baby in the Spring. So although my expectations weren’t all that high, I pretty much had to check Prospect Park West out of the Brooklyn Public library, after waiting patiently for my turn in a queue that was over 250 holds long. Prospect Park West is set against this familiar background. The plot follows the ill-considered affair of a Park Slope mother, whose life is connected to a few other characters by a string of coincidences that I would have found far-fetched ten years ago, before I started to notice equally striking coincidences in my own life. (Always remember that odds are that life will be filled with the improbable, since there are an enormous number of possible improbable events—so many that it would be highly improbable for us to go long without another improbable event occurring. This is one reason, among several, that life is filled with strangeness and magic, if you keep an eye out for it.)

Prospect Park West is not a great work of literature, but it’s readable enough. The book’s basic outlook is misanthropic without much in the way of compensating insight. I get that some Park Slope mothers can be a bit much, but so can the author when she (in the mouths of her characters) gets going about them. The author gets points, though, for her depiction of the strange, confusing, prickly racial tension you run across in Brooklyn all the time, and which I struggle to explain to my friends back in Canada. This too was perhaps also a bit overdone, but unfortunately not by much.

One correction: A check out line at the Coop that stretches back to the bread section does not count as long. I don’t know when Sohn shops, but that’s pretty routine in my experience. Long is when it goes all the way along the produce aisle as far back as the milk section.

Charles Dickens. Oliver Twist

This is only the second Dickens novel I’ve read, the other being A Tale of Two Cities. I found A Tale of Two Cities pretty silly, but against my better judgment found the ending weirdly sublime. I didn’t have as much luck with Oliver Twist, which I read for the sole reason that we’re naming our kid “Oliver” and I figured I should at least read the book that helped make his name famous. (On my to do list: Who the hell is Oliver Cromwell?) I found the social commentary in the first part of the book entertaining enough, if heavy-handed. But as the plot advanced, the melodrama and the general absurdity of it all started to suck the fun out of it. Also, I know the book is a product of the early nineteenth century, but the fact that one of the characters is usually referred to simply as “the Jew” and even gets to be the butt of a big nose joke was driving me nuts. What’s that? Dickens was a child of his era, so cut him some slack? Well, I’m a child of my era, so take your own advice and cut me some slack while you’re at it.

Vivant Denon. Introduction by Peter BrooksNo Tomorrow

Vivant Denon was, among other things, the first director of the Louvre Museum, in charge of sorting and cataloging all the goodies that Napoleon stole from the Egyptians. A wing of the Louvre bears his name to this day. Denon was also “maybe, probably,” in the words of Peter Brooks, the author of No Tomorrow a thirty odd page long erotic masterpiece. The New York Review of Books has recently published a fine bilingual edition of the story with an introduction by Peter Brooks. The intellectual imprimatur provided by the publisher and the scholarly introduction makes it totally not skeevy that I’m writing about erotica on my blog.

There’s a lot to admire in Denon’s story and the way he tells it. As for the tale, a woman seduces a man, for pleasure, without negative consequence for either. As for the telling, Denon is delicate without ever being prudish, erotic without ever being explicit. It’s good clean fun for the adults in the family.

Surendra Verma. The Little Book of Maths, Theorems, Theories, and Things

This book covers a very wide variety of mathematical and logical puzzles and problems and more. The author even throws in a discussion of the Body-Mass Index*, presumably because it’s . . . expressed in numbers? Because it’s a little book, and because it’s trying to get to so many subjects, and because the author also likes to throw in limericks and factoids and anecdotes willy-nilly, this book treats each of its subjects in an extremely superficial way. I like limericks and factoids and anecdotes as much as the next guy, but there really wasn’t room for a lot of math in this book, or much opportunity for the author to make the case that mathematics is intrinsically interesting.

Let me also take a moment to plead with the publisher to fix the typos in this book before reprinting, if the book ever gets another shot at life. You know you’re in bad hands when you read the sentence: “No one has ever found an even number that can be expressed as the sum of two prime numbers” (p. 76). Oh, really? Cause I think I might be about to make mathematical history!

* Verma tells us that knowing your BMI “can give you an idea of how healthy your weight is.” He doesn’t note that a lot of researchers think the BMI is misleading or useless.

I suppose it’s fitting since my last post was right before the 2004 election, but I never thought my return would be about politics. Well, kind of.

For various reasons, including the interesting Obama/Clinton delegate math, I’ve been following this year’s election in greater detail than any in the past. Which, unfortunately, means I’ve been reading a lot of political articles. In the course of my travails I came across a particularly egregious example of the mis-use of statistics that got me worked up enough that I had to write about it somewhere. So here you go.

I found this in a post yesterday by Peggy Noonan at WSJ.com:

I’m bumping into a lot of critics who do not buy the legitimacy of small town mayorship (Palin had two terms in Wasilla, Alaska, population 9,000 or so) and executive as opposed to legislative experience. But executives, even of small towns, run something. There are 262 cities in this country with a population of 100,000 or more. But there are close to a hundred thousand small towns with ten thousand people or less. “You do the math,” the conservative pollster Kellyanne Conway told me. “We are a nation of Wasillas, not Chicagos.”

The worst thing is that this even passes the sniff test; 262 times 100,000 is way, way less than 100,000 times 10,000, so it does seem like there are more people living in small cities than big cities.

Except: the first alarm should go off when you look at the numbers for small cities. 100,000 times 10,000 is 1 billion, and the population of the US is just over 0.3 billion. And of course, when you actually get to the facts, you find that over 58% of the US lives in cities with over 200,000 people.

The trick is using a floor for the number you want to minimize, and a ceiling for the number you want to maximize. The counting in the quote above counts New York City as a city of 100,000, and counts Eastport, Maine (my ancestral home, population 1640) and many other towns with population under 1000 as cities of 10,000.

Another example of this fun statistical manipulation: only 1% of the US population has a household income of over $400,000, but over 50% of the population has a household income of under $50,000. Clearly, the evidence show that most of the wealth in the US lies in the hands of working families.

One of the wonderful things about picking up even the slightest bit of programming is that you can test and play around with things like this. The Python programming language makes it especially easy for a beginner to muddle through to a test very quickly:

import random

remainingchoice = []
carcount = 0.0
trials = 100000

for i in range(1, trials):
    possibilities = ['goat', 'goat', 'car']
    random.shuffle(possibilities)
    if possibilities[1] == 'goat':
        remainingchoice.append(possibilities[2])
    else:
        remainingchoice.append(possibilities[1])

remainingchoice.sort()
for item in remainingchoice:
    if item == 'goat':
        pass
    else:
        carcount +=1    

print (carcount * 100) / trials

Somehow makes it all seem more solid. Except when I changed the trials variable to 100000000 and my computer was all like “What the fuuuu?” and then Python crashed hard.

This is a book about heuristic, the study “of the methods and rules of discovery and invention,” in which most (but not all) of the examples are drawn from mathematics. Polya is interested in the question of how we go about solving puzzles in general, and, having acquired a facility with problem-solving, how we then go on to teach others the same skill. There’s no straightforward algorithm for problem-solving, but there are general patterns. As Polya never tires of reminding us, we typically need to ask ourselves: What is the unknown? What are the data? Do we know a related problem? Can we use this problem in the solution of our current problem? And so on. These might sound obvious, but there’s a value in having them stated clearly, and significant value in some of Polya’s imaginary discussions with students who are walked through the solutions to puzzles by a teacher making intelligent use of Socratic questioning.

Since it’s about thinking in general, and not just mathematics, I think this book might be read with profit by most people. I imagine it would be especially useful for mathematics teachers, especially because the author clearly has a keen sense of pedagogy. Unfortunately, many of the mathematical examples were a bit over my head, since I’m awfully rusty these days. And the book is marred, in my opinion, by an unconscionable amount of repetition. Still, on the whole a decent book.

This book is not really about shopping at Walmart, but I’ll start there anyway. Suppose you’re grousing about Walmart. It wants to move into your town and you’re worried about the effects on the local economy. Or it already has, and you think the effects you warned about are already becoming apparent. There’s a good chance that the person you’re grousing to is going to point out that no one makes anyone shop at Walmart. Indeed, if your interlocutor impolite enough she may even raise the awkward point that she saw you just last week emerging from that very store. If everyone, including you, shops at Walmart, what better evidence, then, that the community has collectively decided to welcome Walmart by making a series of individual choices to support it?

The answer, according to Slee, is no, and his book is a very careful and methodical cataloging of some of the most important ways in which choice is more complicated than the view sketched above suggests, which Slee calls MarketThink. Very briefly, our choices are not made in a vacuum. We make the choices we do while responding to agents who are making choices of their own, which choices themselves are in part responses to our own choices or what they anticipate will be our choices. And in choice situations of this sort, it is often the case that every individual agent makes choices which are perfectly reasonable from where she is situated, but which lead to outcomes which no one involved would prefer all things considered.

Preference, then, turns out to be more complicated than it first appears. Sometimes “preference” refers to what an agent chooses from the options available to her, given the choices that other agents are making or intending to make. Sometimes, by contrast, it attaches to the outcome which the agent would prefer. A great deal of Slee’s book is taken up with explaining how and why these two notions of preference frequently come apart.

Readers who know even the most basic game theory will know that Slee is not just being modest when he claims in the preface that there is little original in his book. I’m not sure I actually learned anything new from this book, since I’ve already read my Rapaport and my Axelrod and my Sen and so on (and I really haven’t read much beyond that). Even so, I found it well worth my time. There’s real value in having the inadequacies of MarketThink detailed for two hundred pages of marvelously clear prose. I think that I was already predisposed to agree with virtually everything Slee said, but reading him made me aware that I do sometimes slip into versions of MarketThink when I clearly shouldn’t. So, the book was edifying and entertaining at the same time.

It’s worth noting, though I don’t intend this as a criticism, that the book is almost entirely negative. That is, it’s about the inadequacies of MarketThink, and not about how to correct for those inadequacies. That’s just fine with me, but I found myself wondering from time to time how an intelligent libertarian would respond to Slee’s main line of argument. I suspect that an intelligent libertarian would have to concede that MarketThink, as Slee depicts it, is crude and inadequate. But a libertarian version of Slee might just as easily write a whole book, also drawing on economics and game theory, to show how regulation and intervention in the market often leads to unintended and frequently unwanted results. The failure of MarketThink does not automatically establish the soundness of any alternative, of course. Now, I think the response to this point is to try to get more specific about exactly what interventions are warranted and how we propose to avoid unwanted consequences. But that just means that the argument goes on (and Slee would surely agree). But thanks to Slee (and the people whose work he draws on) the argument ought to go on without silly appeals to MarketThink. And that’s an advance worth celebrating.

Nassim Nicholas Taleb. The Black Swan: The Impact of the Highly Improbable

What a strange book! This book engages with an extensive literature on risk, probability and the psychology of risk and probability assessment, but it does so in a most unacademic way. Taleb tells stories, engages in autobiography, subverts our expectations about the relevance of the autobiographical passages, harangues, insults, scolds, relates his fantasies about humiliating rival thinkers, bullies, ends expository paragraphs with “Capiche?”, pleads, and repeats himself again and again. Taleb writes in a rough, informal, and highly idiosyncratic way. I cannot imagine that the editor of this book wanted the book as it is in its final form, and I sort of get a kick out of imagining Taleb forcing them to accept it anyway (I’m sure they made money on it nonetheless). He must be a serious pain in the ass to work with.

Anyway, Taleb’s basic idea is that we – human beings, that is – are incredibly bad at assessing risk. Our models for risk assessment tend underestimate the impact of the highly improbable. But marinate on this, brother: There are so many possible highly improbable events that it is highly probable that the highly improbable will intrude, and intrude very messily, into reality, and blow all our little models to bits. We just don’t know what they’ll be. We live in a world dominated by the highly improbable, and most of our risk models are worse than useless: because they encourage us to think we’ve got a handle on things, they make us even more vulnerable to extraordinarily improbable events when they do occur. Taleb doesn’t just confine himself to the world of finance, where he made his living dealing with risk, in order to illustrate this point, preferring to range over a much broader field of history in search of arguments and examples.

So why we do this? Much of Taleb’s book is a meditation on this very question, and, I think, a very useful one. When we look at past events, we have a tendency to slip into narratives that make events seem to follow one another in a natural and expected way. This encourages us to think that, going forward, events can be expected to follow one another in a natural and expected way. It isn’t so. And if you actually look at the track record of experts in various fields, you’ll find them regularly getting blindsided by events which were anything but predictable. And if you actually listen to the experts defending wrong predictions after the fact, you’ll regularly hear them defending their predictions in the following form, “But I was exactly right in my prediction, except that X,” where X is something highly improbable that unfortunately threw everything off. But if this happens again and again, we ought to stop and wonder about the value of such predictions in a world that serves up so many Xs.

This insight is not just valuable for people making their living predicting the future, in my opinion. I think this book should be required reading for anyone doing historical work, which, in my experience, frequently a) falls into the habit of ad hoc explanations which oversimplify reality, and b) often attempts conjectural reconstructions of the past on the basis of mere plausibility, again in a way that I think grossly oversimplifies matters. (I have a post in draft now illustrating (b).)

There’s more – much more – to Taleb’s story about the human propensity to underestimate the potential impact of the improbable, which I won’t go into. I will say that Taleb seems to me just a bit too enamored with stock evolutionary psychology explanations – which is funny in a book about how prone we are to manufacturing bogus ad hoc explanations, since the same vice is pretty common in evolutionary psychology, in my opinion. But whatever. There’s still a lot of great material here. And I found myself grateful for the repetition in the end. The first twenty times I read Taleb complaining about bad excuses for predictions gone wrong I nodded my head and thought “Yeah, sure.” But about the twenty-first time I thought “Holy fuck! That is so true. I do that too!”

Anyway, if you’re curious here’s a Malcolm Gladwell piece on the guy from the New Yorker. And here’s his (ugly!) website, where you can get a taste of how feisty and combative he is, since he appears to respond to all the reviews his books have gotten. I was amused to see his response to Gregg Easterbrook, who is seriously the dumbest fucking guy ever.

Henry Fielding. Joseph Andrews

Not nearly as good as Fielding’s Tom Jones, but then practically nothing is. I found the first fifty and the last fifty pages awesome, with some pretty plodding material in between them. Fielding’s theme is sexual desire in its various forms, its frustrations, its gratification, and so on. As I was reading it I thought I was entertaining myself with a fluffy, silly story about a man and a woman eager to get married so that they could get it on. But when I finished it and looked back I realized that for a fluffy, silly book it managed to sneak in quite a bit of interesting reflection about its theme while it was at it. Anyway, check out Chapter V, which is pretty damn funny:

At this time an accident happened which put a stop to those agreeable walks, which probably would have soon puffed up the cheeks of Fame, and caused her to blow her brazen trumpet through the town; and this was no other than the death of Sir Thomas Booby, who, departing this life, left his disconsolate lady confined to her house, as closely as if she herself had been attacked by some violent disease. During the first six days the poor lady admitted none but Mrs. Slipslop, and three female friends, who made a party at cards: but on the seventh she ordered Joey, whom, for a good reason, we shall hereafter call JOSEPH, to bring up her tea-kettle. The lady being in bed, called Joseph to her, bade him sit down, and, having accidentally laid her hand on his, she asked him if he had ever been in love. Joseph answered, with some confusion, it was time enough for one so young as himself to think on such things. “As young as you are,” replied the lady, “I am convinced you are no stranger to that passion. Come, Joey,” says she, “tell me truly, who is the happy girl whose eyes have made a conquest of you?” Joseph returned, that all the women he had ever seen were equally indifferent to him. “Oh then,” said the lady, “you are a general lover. Indeed, you handsome fellows, like handsome women, are very long and difficult in fixing; but yet you shall never persuade me that your heart is so insusceptible of affection; I rather impute what you say to your secrecy, a very commendable quality, and what I am far from being angry with you for. Nothing can be more unworthy in a young man, than to betray any intimacies with the ladies.” “Ladies! madam,” said Joseph, “I am sure I never had the impudence to think of any that deserve that name.” “Don’t pretend to too much modesty,” said she, “for that sometimes may be impertinent: but pray answer me this question. Suppose a lady should happen to like you; suppose she should prefer you to all your sex, and admit you to the same familiarities as you might have hoped for if you had been born her equal, are you certain that no vanity could tempt you to discover her? Answer me honestly, Joseph; have you so much more sense and so much more virtue than you handsome young fellows generally have, who make no scruple of sacrificing our dear reputation to your pride, without considering the great obligation we lay on you by our condescension and confidence? Can you keep a secret, my Joey?” “Madam,” says he, “I hope your ladyship can’t tax me with ever betraying the secrets of the family; and I hope, if you was to turn me away, I might have that character of you.” “I don’t intend to turn you away, Joey,” said she, and sighed; “I am afraid it is not in my power.” She then raised herself a little in her bed, and discovered one of the whitest necks that ever was seen; at which Joseph blushed. “La!” says she, in an affected surprize, “what am I doing? I have trusted myself with a man alone, naked in bed; suppose you should have any wicked intentions upon my honour, how should I defend myself?” Joseph protested that he never had the least evil design against her. “No,” says she, “perhaps you may not call your designs wicked; and perhaps they are not so.”–He swore they were not. “You misunderstand me,” says she; “I mean if they were against my honour, they may not be wicked; but the world calls them so. But then, say you, the world will never know anything of the matter; yet
would not that be trusting to your secrecy? Must not my reputation be then in your power? Would you not then be my master?” Joseph begged her ladyship to be comforted; for that he would never imagine the least wicked thing against her, and that he had rather die a thousand deaths than give her any reason to suspect him. “Yes,” said she, “I must have reason to suspect you. Are you not a man? and, without vanity, I may pretend to some charms. But perhaps you may fear I should prosecute you; indeed I hope you do; and yet Heaven knows I should never have the confidence to appear before a court of justice; and you know, Joey, I am of a forgiving temper. Tell me, Joey, don’t you think I should forgive you?”–”Indeed, madam,” says Joseph, “I will never do anything to disoblige your ladyship.”–”How,” says she, “do you think it would not disoblige me then? Do you think I would willingly suffer you?”–”I don’t understand you, madam,” says Joseph.–”Don’t you?” said she, “then you are either a fool, or pretend to be so; I find I was mistaken in you. So get you downstairs, and never let me see your face again; your pretended innocence cannot impose on me.”–”Madam,” said Joseph, “I would not have your ladyship think any evil of me. I have always endeavoured to be a dutiful servant both to you and my master.”–”O thou villain!” answered my lady; “why didst thou mention the name of that dear man, unless to torment me, to bring his precious memory to my mind?” (and then she burst into a fit of tears.) “Get thee from my sight! I shall never endure thee more.” At which words she turned away from him; and Joseph retreated from the room in a most disconsolate condition, and writ that letter which the reader will find in the next chapter.

Ha! She’s so horny.

Brecht explores the moral difficulties in Gallileo’s decision to recant. Not bad.

Paulos, John Allen. Innumeracy

A fun little book that provides a healthy dose of motivation to the non-mathematical to get their (our!) act together. Paulos provides lots of examples of fuzzy thinking that follow from a neglect of basic mathematics. At times Paulos seems to cast his net a bit more broadly than mathematics even, commenting on various fallacies in informal reasoning. But that’s ok – those mistakes matter too.

Frank. Falling Behind: How Rising Inequality Harms the Middle Class

Entertaining and reasonably well-written. Frank charts the rise of inequality in American society since WWII, and then explains why he thinks that inequality is so harmful. Some goods are absolute goods. These we care about regardless of how much other people have. Others are positional goods. These we value very differently depending on context, most importantly how others around us are doing with respect to that good. Frank argues that many more goods are positional than one might first think, and then ties this insight to his observations about rising inequality. The result is a decent critique of a lot of mainstream assumptions about inequality in American society, and more broadly of the social policies that have produced it.

Two quibbles. First, it’s ok to dumb down a bit for a popular book, but Frank’s remarks about evolutionary psychology were pretty silly at times. I’d have to read Frank’s other work on the subject to know whether I would find a more careful statement of his views silly. But anyway, I don’t really think Frank needed to introduce claims about evolutionary psychology in the first place. His motivation for doing so, if I understood it correctly, was just to point out that the psychological tendencies he’s attributing to us are fairly stubbornly entrenched. But a) you don’t need to point to evolutionary considerations to do that; and b) you shouldn’t point to evolutionary considerations to do that (just for starters, innateness and malleability are completely distinct issues).

Second quibble: Frank talks throughout about the middle class. He even put the middle class in the subtitle of his book. But the book really seems to be about how just about everyone gets screwed by rising inequality, even very well-off people. So perhaps the subtitle to his book ought to have been “How Rising Inequality Harms Us All.”

Tufte, Edward R.The Visual Display of Quantitative Information

Superb. Tufte wrote the book in the last seventies and early eighties; it changed the way many people think about how to display quantitative information in a clear, engaging and helpful way. Tufte’s book is part polemic against a dumbing down of statistical charts on the grounds that no one finds them interesting, and part analysis of what considerations go into getting it right. Good stuff.

My parents live in Maine, so I’m well aware that you have to pass through New Hampshire to get there. #%@*&! tolls and highway liquor stores.

This would be the largest known prime number, with roughly 7 million digits. There’s a $100,000 prize for the discovery of the first prime with 10 million digits, which GIMPS is likely to claim within a couple years.

The discovery of the 41st Mersenne prime would also mean the discovery of the 41st perfect number.

Update: After finding the message boards where the discovery is being discussed, it looks quite possible that there has been a new Mersenne prime found, but that it might be smaller that a previously known Mersenne prime.

There are two issues here: the first is that using a computer to check cases or run lengthy computations doesn’t affect the worthiness of the math; you’re not going to get shunted off to a computational journal for that. The second is that a proof that is actually found by a computer is not that interesting from a math standpoint. It’s nice to know a theorem is true, but mathematicians have been proving theorems based on unknown facts for thousands of years.

Stephen Goldman, possibly the best baseball writer of the last 25 years not from Kansas, wrote about similar issues in the baseball community. Computers in math, and the new (mainly statistical) methodologies used in baseball are just tools, and correspondingly can be used for good or evil. But dismissing them out of hand is foolish.