Skip to main content

Simpson's Paradox

 


My random reading in matters mathematical has brought me to Simpson's Paradox, the idea that a clear trend in a group of data can disappear, or even reverse, when different groups of data that taken severally exhibit the same trend are aggregated. 

It is named for Edward Simpson, who described the effect in "The Interpretation of Interaction in Contingency Tables" in the Journal of the Royal Statistical Society in 1951.

Colin Blyth rediscovered this paper and gave the phenomenon the name Simpson's Paradox in a 1972 paper.

It was once referenced in an episode of the prime-time cartoon The Simpsons because ... well, the name.

Wikipedia has a good discussion.

This is not really a "paradox" at all, though the name will probably stick.  It wouldn't impress Zeno, or whoever started the thing about the lying Cretan. But the phrase "Simpson's odd-seeming phenomenon" doesn't seem very resonant. 

Good brief discussion in pp. 70 - 72 of Titelbaum, Fundamentals of Bayesian epistemology.

The following example comes from Titelbaum's book. It involves two basketball pros active in the 2016-2017 season in the NBA: James Harden of the Houston Rockets and DeMar DeRozan of the Toronto Raptors. 

1. Harden made a higher percentage of his two-point shots than DeRozan.

2. Harden also made a higher percentage of his three-point shots than DeRozan.

Given both of those facts, you probably would not expect this:

3. Harden made a smaller percentage of his overall shots than DeRozan. 

Aggregation can kill a trend. The forest can be very different from the trees.  Insert some other cliche here. I know, right?  

Labels:

Comments

  1. Paul D. Van PeltOctober 24, 2024 at 8:01 AM

    Facts are pretty crucial to getting to the crux of anything. If one bases a conclusion on less than all of them, one's *conclusion* is no more than an educated inference. Inference does not=reality in such matters. I would hold that the "paradox" is not a paradox at all---only an incomplete assessment. Jack Webb, in *Dragnet*: gimme the facts, m'am, just the facts*.
    [I used to do some administrative law work]

    ReplyDelete
  2. Paul D. Van PeltOctober 24, 2024 at 10:44 AM

    This is, arguably, diissociated from paradox...maybe. Today, I read a post on "elite, social justice activism". That struck me as odd. At best. The terms, *elite*, and, *social justice * are mutually contradictory, if not outright antagonistic, seems to me. When did elitists become advocates of social justice? Answer: when phrases such as *effective altruism* and a more-or-less similar characterization of philanthropy surfaced, d'aurtrement (spelling?), they are not. I could not log a comment on the blog, without committing money. Ding! Lost my attention, tout de suite. Et, non, mon tablet, il ne parle pas Francais, o, Espanol. Love your posts, ami.
    Carry on.

    ReplyDelete

Post a Comment

Popular posts from this blog

A Story About Coleridge

This is a quote from a memoir by Dorothy Wordsworth, reflecting on a trip she took with two famous poets, her brother, William Wordsworth, and their similarly gifted companion, Samuel Taylor Coleridge.   We sat upon a bench, placed for the sake of one of these views, whence we looked down upon the waterfall, and over the open country ... A lady and gentleman, more expeditious tourists than ourselves, came to the spot; they left us at the seat, and we found them again at another station above the Falls. Coleridge, who is always good-natured enough to enter into conversation with anybody whom he meets in his way, began to talk with the gentleman, who observed that it was a majestic waterfall. Coleridge was delighted with the accuracy of the epithet, particularly as he had been settling in his own mind the precise meaning of the words grand, majestic, sublime, etc., and had discussed the subject with William at some length the day before. “Yes, sir,” says Coleridge, “it is a maj...
9 comments
Read more

Searle: The Chinese Room

John Searle has become the object of accusations of improper conduct. These accusations even have some people in the world of academic philosophy saying that instructors in that world should try to avoid teaching Searle's views. That is an odd contention, and has given rise to heated exchanges in certain corners of the blogosphere.  At Leiter Reports, I encountered a comment from someone describing himself as "grad student drop out." GSDO said: " This is a side question (and not at all an attempt to answer the question BL posed): How important is John Searle's work? Are people still working on speech act theory or is that just another dead end in the history of 20th century philosophy? My impression is that his reputation is somewhat inflated from all of his speaking engagements and NYRoB reviews. The Chinese room argument is a classic, but is there much more to his work than that?" I took it upon myself to answer that on LR. But here I'll tak...
17 comments
Read more

Recent Controversies Involving Nassim Taleb, Part I

I've written about Nassim Taleb on earlier occasions in this blog. I'll let you do the search yourself, dear reader, for the full background. The short answer to the question "who is Taleb?" is this: he is a 57 year old man born in Lebanon, educated in France, who has been both a hedge fund manager and a derivatives trader. He retired from active participation from the financial world sometime between 2004 and 2006, and has been a full-time writer and provocateur ever since. Taleb's writings for the general public began where one might expect -- in the field where he had made his money -- and he explained certain financial issues to a broad audiences in a very dramatic non-technical way. Since then, he has widened has fields of study, writing about just about everything, applying the intellectual tools he honed in that earlier work. As you might have gather from the above, I respect Taleb, though I have sometimes been critical of him when my own writing ab...
4 comments
Read more