Attention conservation notice: 1200+ words about a nearly sixty-year-old paper on the foundations of microeconomics, maliciously re-interpreting it in ways its author explicitly disclaimed, by someone with no qualifications in economics. I wrote this in 2012 or 2013, re-titled in 2014, and never got around to properly finishing it. I dust it off now because I'm in the middle of reading Jason Smith's A Random Physicist Takes on Economics, and see that he had basically the same idea, but said it better and in public. This might, therefore, serve to amplify his message.

I recently re-read Gary Becker's "Irrational Behavior in Economic Theory" paper (Journal of Political Economy 70 (1962): 1--13), and want to talk about it. (Like most of what I know about economics, I was introduced to this paper by my father, many years ago now.) Becker was, to my mind, the embodiment of much of what is wrong with the discipline and practice of economics (see, e.g., John Emerson on Becker's work on families [1, 2]), but this is a genuinely brilliant paper. In many ways, the paper was wiser than its author.

Let me start with the basic observation of the paper. Suppose you have a monthly grocery budget. (Or book budget, or drug budget, or...) This budget, together with the prices of grocery items, dictates what you could buy in the way of groceries. Suppose that the price of onions goes up. Then, just as a matter of arithmetic, the maximum number of onions that you can buy goes down, and for any given level of consuming other goods, you can buy fewer onions. Even if you pick a totally random collection of groceries each week, having a fixed amount of money to spend will ensure that when the price of something go up, you will (on average) buy less of it.

If all the consumers in a market also have budget constraints, then the same reasoning applies to them, too, and so the total quantity of onions bought will tend to fall as their price rises. This is, of course, a basic idea of standard economics, but what Becker shows is that this has nothing to do with satisfying preferences or maximizing utility, or any other assumption about how individuals make decisions. Consumers' could make decisions by lots of different mechanisms --- indeed different consumers could be driven by different mechanisms --- and it wouldn't matter. What does matter is that they all have budget constraints.

Becker's treatment of the producer side of the market is subtly different, and doesn't rely on budget constraints. (Becker indeed insists producers aren't subject to budget constraints, which not only would be news to many small business owners I know, but would also mess up applying his reasoning about consumption to markets for capital goods.) Rather, the the insight is that a wider range of productive techniques, and of scales of production, become profitable at higher prices. This matters, says Becker, because producers cannot keep running losses forever. If they're not running at a loss, though, they can stay in business. So, again without any story about preferences or maximization, as prices rise more firms could produce for the market and stay in it, and as prices fall more firms will be driven out, reducing supply. Again, nothing about individual preferences enters into the argument. Production processes which are physically perfectly feasible but un-profitable get suppressed, because capitalism has institutions to make them go way. If you think this account of producers and supply is all less tidy and satisfactory than the treatment of consumers and demand, I agree with you, but it is, after all, far better than the usual microeconomic account of these matters.

Becker doesn't go in this direction, but of course this completely undermines welfare economics. The whole grasp of that sub-field on reality comes from assuming that individuals' choices reflect their judgments of utility. If that's true, we could read off a lot about people's (subjective) welfare from aggregated market prices and quantities. By breaking the link between microeconomic relations for such aggregated variables and individual utility, Becker breaks the link between observations and welfare economics.

Going further, Becker also removes the link between positive microeconomics, at the level of the market, and individual behavior. Though Becker doesn't put it this way, and would have been horrified by the thought, his argument is it really doesn't matter how individuals make decisions, or even whether economic behavior is usefully thought of as "making decisions" at all. Rather, everything depends on the constraints put on behavior by institutions. That consumers have budgets and can't spend more than that, and that producers can't keep running at a loss, these are both institutional facts about the social organization of capitalism, not facts about how individuals act. (Of course, institutions have to be implemented through individual actions, institutions aren't some sort of cloud hovering over society telling people what to do, but that's another story.) This implies that there are all sorts of different possible micro-foundations for our positive economics, since anything that preserved the institutions would serve. It also suggests that a different set of institutions would lead to a different positive economics. Since institutions change over space and time, then, positive economics would have to be historically relative.

In short, I am inclined to say that Becker fully deserves his Nobel Prize on the strength of this paper alone. The citation, though, should have read "For exploding the foundations of welfare economics, and re-constituting positive microeconomics on an institutionalist basis".

Now, this is not what Becker thought he was doing. From the opening and closing of the paper, it's clear that what he thought he was defending business-as-usual for the Chicago School. Nowadays, we can say that the assumptions of that school about individual behavior are experimentally refuted; science knows that's no more how humans behave than phlogiston is how stuff burns. Even back then, though, there was a lot of criticism for the lack of realism of these assumptions. For instance, Herbert Simon (pbuh) kept pointing out that human rationality is computationally bounded...

Chicago Schoolers had a range of response to such criticism. One was to ignore it and go ahead with the work. (This is part of why I am leery of "shut up and calculate" responses to conceptual criticisms; they're only as good as the basis for the calculations.) The prize for chutzpah went to Milton Friedman, here as elsewhere a profoundly malign influence, who claimed that this lack of realism was in fact a virtue, and actually what made economics extra scientific. (I am not exaggerating, and outsource further commentary to Simon, 1963.)

Compared to that, Becker must have thought he was holding out an olive branch. Look, he (in effect) said, I'll grant for the sake of argument that we're totally wrong about how people make decisions. At the market level, it wouldn't matter, things would still come out the way we say they should. So why not let us hold to those assumptions, just to have something definite to work with?

But of course this won't do. Whatever you feel about Occam's Razor, "This assumption is superfluous, therefore it should be retained" is a very strange rule of method. (Why not assume that people maximize utility, and that utility is purple?) A more sensible reaction to Becker's own argument would surely be to avoid relying on those assumptions for anything, at least until he could provide some independent support for them.

On a more constructive note, I suspect there is a connection to be made here with John Sutton's "bounds approach" to studying industrial organization, where the aim isn't to solve for the equilibrium of the one true model, but instead to find relationships, generally inequalities, which hold across a broad range of models. There might even be some sort of analogy to thermodynamics, which is what you get for nearly-any system built out of molecules when its parts can only interact through a restricted set of collective degrees of freedom (viz., exchange of heat, pressures, a few chemical species, etc.). But both of those would call for real thought and original work on my part, rather than mildly-malicious exposition.

The Dismal Science

Attention conservation notice: I have no taste. I also have no qualifications to opine on the sociology of experts and anti-intellectualism, or on the history of Islamic civilization.

Andrea Camilleri, A Nest of Vipers and Pyramid of Mud

Mind candy mysteries, the nth and (n+1)st in the series (previous entries to be found here), with all the usual pleasures. But Nest was first one I can remember where I worked out whodunnit before Montalbano. Whether this means I am getting better, or Camilleri (or Montalbano?) is slipping, is not for me to say.

Ruth Downie, Memento Mori

Mind candy historical mystery: latest (previously) in her series about an ex-legionary doctor and his British (i.e., barbarian native) wife, and their misadventures in Roman Britain --- this time, Aquæ Sulis (i.e., Bath). It continues to be really good, though I am a bit worried about where the end of this book leaves our protagonists.

Ausma Zehanat Khan, The Bloodprint

Mind-candy epic fantasy. On those terms, it's enjoyable enough, though unremarkable. What makes it really notable is that it's entirely based on the recent history of Afghanistan and environs, with a magic system based on Quranic recitation. Many reviewers have noticed that "the Talisman" = the Taliban, but none I've seen have remarked on the other correspondences. (This does not pretend to be a complete list.)

• Khorasan = Afghanistan
  • West Khorasan = eastern Iran
  • North Khorasan = Trans-Oxiana, ex-Soviet Central Asia
• Candor = Kandahar, Hira = Herat (I did not notice counter-parts for Kabul, Mazar-i-Sharif, Jalalabad, or Ghazni)
• Sailing Pass = Salang Pass
• Zergil tribe = Gilzai tribe
• Valley of the Five Lions = Panjshir
• the Claim = the Qur'an (I'm not sure which translation Khan used, but I am pretty sure every verse of the Claim is in fact from the Qur'an)
• the Hazaras and Hazarajat = the Hazaras and Hazarajat
• Turquoise mountain = Turquoise mountain
• the Wall = the border with the former Soviet Union
• The Authoritan = Karimov
• basmachi = basmachi
• Jaslyk prison and boiling dissidents alive = Jaslyk prison and boiling dissidents alive

My only surprise is that Khan's family are evidently Pashtuns from Pakistan, rather than from Afghanistan.

Tom Nichols, The Death of Expertise: The Campaign Against Established Knowledge and Why It Matters

This is an expansion of a blog post that went viral. It's easy enough to find summaries (e.g.), so I will skip giving one. It's good, as far as it goes, but I don't think it really goes far enough, or probes deeply enough. I will offer some critical remarks, but I want to emphasize that I do think this is a good book, by a skilled and well-intentioned writer, and worth reading. (Also, I realize that some of my complaints amount to wishing he'd written an academic treatise rather than a popular essay.)

1. Nichols starts with a sensible definition (p. 29): "I will use the words 'professionals,' 'intellectuals,' and 'experts' interchangeably, in the broader sense of people who have mastered particular skills or bodies of knowledge and who practice those skills or use that knowledge as their main occupation in life". Unfortunately, he does not really wrestle with how this opens up a line of attack on expertise, which is that a profession is really a group of people who make their livings by claiming to have mastered what they say is a shared, unusual body of knowledge. But maybe all they have really mastered (whatever they might think) is a shared body of jargon, mistake, fallacy and delusion --- their purported knowledge might really be bogus. Haruspicy, or even homeopathy, would be too easy as targets. Take something more mainstream: there are many people who claim to be experts in personality testing, and sell their services to schools, corporations, law courts, etc. But the methods they use are well-established to be scientifically worthless, even by the not-exactly-demanding standards of social psychology. Or, perhaps more consequentially, we have created a profession of "forensics" ("forensic science", even), which we use to help make literally life-or-death decisions in the courts. It turns out that most of the methods employed (and taught, and certified) are unscientific and untrustworthy, and that's before we get to admissions of massive failure stretching over decades. (Beside things like this, my pet peeve of statsiness hardly matters, but it's in the mix, too.) The personality testers and the forensic scientists, while organized professionals, aren't really experts, just specialists. And if that's true of those professions, how many others might also be mere bogus specialisms? What about, say, the law?
   Yes, we live in a vastly elaborated division of labor and responsibility, where different professions claim jurisdiction over different specialized tasks, but how much of that is jargon and bullshit, rather than genuine knowledge? Maybe that's inevitable, maybe deferring to specialists who are just (collectively) making shit up is the price we pay for our prosperity, but can you really blame anyone for resenting that, or for refusing to acknowledge the authority of non-existent knowledge? (Proverbially: "Don't piss on my leg and tell me that it's raining.") Most people are, after all, much more likely to be told by a boss to take a personality test than to interact with, say, a rocket engineer.
   Or, worse: I have been writing as though some profession just are bogus, no question, because that's what I happen to think. But a more neutral description would be that these are conflicts between rival groups of professionals, where one group (personality testers, forensic technicians) claim expertise and the authority that goes with it, while another group (social psychologists, statisticians) attacks the first group's claims. If you are unable or unwilling to enter into those specific debates on their merits, you might well think that all the claims to expert knowledge here are bogus, perhaps merely rationalizing professional interests. At the very least, you should be willing to entertain it as a possibility.
   Now, I do not think this nightmare vision is correct. (I also do not believe in my neutral model of inquiry.) And I know that the overwhelming majority of the people driving Nichols to despair aren't making an argument this coherent. But this is kind of idea many of them are fumbling towards, and I wish this book had addressed it. Instead, the chapter on "when the experts are wrong" limits itself to specific mistakes (e.g., is it healthy to eat eggs?), rather than the possibility of global, albeit sincere quackery (do "dieticians" actually know anything?).
2. This brings me, indirectly, to a curious omission. The book never mentions the well-funded and well-organized campaigns by vested corporate interests to cast doubt on expert opinions when those are inconvenient. The paradigm case is tobacco companies trying to obfuscate the dangers of smoking, but of course the same tactics, and even the same personnel, got recycled for global warming. (Merchants of Doubt is the essential reading here, but good journalists following the public relations industry have been warning about this stuff for decades. [I don't think Nichols ever mentions PR; it's not in his index.]) Some of this has been, pretty explicitly, about attacking scientific expertise as such. Even more of it has been about creating confusion about who is an expert and what the expert consensus is. The goal is to get lay people to see dueling experts who can't agree on anything. To the extent this has been successful, sensible lay people should show less deference to self-described expert opinion. More precisely, once lay people are persuaded that experts are always in deep disagreement, and that there is even no consensus on who the experts are, they should show less deference to opinions which are presented to them as those of experts. [0]
   (Relatedly: if many people are now telling their doctors what drugs to prescribe them, might this have something to do with drug makers spending massive amounts on advertising at patients, aiming to achieve just this?)
3. Nichols has his political and ideological commitments, which show through in places. This is fine (at least, I hope it is, for my own sake), except that I think it may have led him in places to pass over facts and cases which would complicate (though perhaps not refute) his argument. For instance: as an example of journalists failing to abide by the principles of their calling, he mentions Sabrina Ederley and Rolling Stone (pp. 163--4). He does not mention, say, Stephen Glass and The New Republic; that comparison would have been instructive, since, after all, Glass's fabrications happened decades before what Nichols sees as the recent corruption of journalism.
   Or, again, Nichols is very unhappy about the way many college students these days seem to be very resistant to hearing opposing viewpoints (chapter 3). Well, so am I [1]. Here, though, are some things which are notable by their absence from this chapter:
   1. evidence that this failing of undergraduates is worse now than it has been in the past;
   2. evidence that contemporary undergraduates are worse about this than their age-mates who don't go to college;
   3. evidence that college makes this worse among those who attend;
   4. argument, and supporting evidence, connecting this failing to the way colleges and universities increasingly treat undergraduates as customers to be pleased, rather than students to be taught, something the book righteously, and rightly, denounces [2].
   None of this means that he's wrong, but it is a failing, by the standards of argument and inquiry which Nichols, very properly, espouses, and which I'm sure he'd adhere to in his more strictly-scholarly work.
   As it happens, since the book was published we have had some evidence about (a)--(c), which would at the very least complicate the argument of the chapter. Of course, no one study is definitive, and there are specific ways one might attack this one. (E.g., perhaps different cohorts, or different educational groups, use the word "racist" very differently, so that apparently small variations in tolerance for "racist" speech conceals huge actual differences [cf. Becker].) But this is the kind of thing I wish Nichols had engaged in, or at least marked out as a very clear area of relevant uncertainty, rather than simply editorializing.
4. That last point brings me to a larger failing of the book, which is its failing to make the necessary historical case [3]. The book wants to argue both that the relationship between experts and the lay public is bad now, and that it has gotten worse, that it is in a uniquely bad state. For the first part, examples of contemporary dysfunction are persuasive, though I would have liked those to have been more systematic (if not necessarily quantitative, though that would have been nice too).
   But for the second part, the claim that things are worse now than they've ever been, there needs to be comparative, historical evidence, and that's just not there. Hofstadter's Anti-Intellectualism in American Life gets mentioned (pp. 18--19; oddly calling Hofstadter a "political scientist", rather than a historian). But this book never really wrestles with Hofstadter's, or with the larger historical literature of which the latter is a part. Those histories at least seem to show that there has always been a very strong current in American life which despises intellectuals, experts, and education (even if it's still fascinated with technology). One could even make out a counter-scenario that Nichols's "death of expertise" is just America reverting to normal, after a transient, out-of-character Cold War episode, when competing with the USSR over the Space Race and the Third World lent value to all sorts of obscure skills. That wouldn't make our current plight any better, but it would change the diagnosis! N.B., I am not not claiming that's the right interpretation; I don't know, but I wish Nichols had considered such alternatives, or at least made his necessarily-historical case better.
   To put it more briefly: Chapter 3 ends (p. 104) with the heart-felt plaint that

   [I]f college graduates can no longer be counted on to lead reasoned debate and discussion in American life, and to know the difference between knowledge and feeling, then we're indeed in the kind of deep trouble no expert can fix.

   I'm a bit skeptical that we ever could count on college graduates to play that role in American life, not least because many generations of earlier writers lamented that they didn't [4].

Again, just to recap: I liked the book, and I share a lot of Nichols's exasperations and anxieties. I think the book is worth reading. But I wish, in the name of the standards it very rightly advocates, it had gone much deeper.

[0]: There is another, related issue which should have made it into the book, but didn't. This is the trend, over the last few decades, towards removing expertise from public bodies, in favor of relying on private industry and its lobbyists. The last chapter of this book is about the role of experts in democratic policy-making. It emphasizes, correctly, that in a modern, representative democracy, people delegate power and authority to elected representatives, who are expected to know more about issues than most citizens. At the same time, even those representatives cannot master all of the intricacies of every issue, and so must consult experts. (Such consultation isn't, or shouldn't be, just doing what experts say to do.) At most, some areas of policy might be further delegated to appointees, but even that is revocable. All this is basic and right as far as it goes, but the disturbing development over recent decades is the on-going erosion of the government's own capacity for expertise, whether in the legislative or the executive branch, in favor of relying on private industry and its lobbyists, sometimes laundred through think-tanks. To be fair, this continues to trade on the appearance of disinterested expertise, rather than de-legitimizing it. ^

[1]: For the record, as threats to academic freedom, I rank ranty undergraduates, however appalling, below administrative branding and fund-raising practices, interfering state legislatures and/or trustees and donors, and the increasing use of un-tenured teachers. The actions of administrators, legislators, and donors are, arguably, relevant to the topic of decreasing deference to expertise, but I also think it is defensible for the book to not generally explore the threats to academia. ^

[2]: The book doesn't say why schools have come to see undergraduates that way, though it leaves the impression that this was a sort of unforced error on the part of university faculty and administrators. It might be that, but it might also have something do to with a shift in how we support the bulk of higher education, away from state governments directly funding their universities and towards the federal government underwriting student loans. (This would seem to be worth investigating, if it hasn't been already.) ^

[3]: I realize that I made a similar complaint about Chris Hayes's Twilight of the Elites. I would pay good money to read a dialogue between Hayes and Nichols. ^

[4]: Another encounter I would pay good money to observe would be Nichols's grappling with the late Jacques Barzun's 1959 The House of Intellect. ^

Richard W. Bulliet, The Case for Islamo-Christian Civilization

This is in some-ways a period piece, the period being 2002--2003. It is nonetheless full of good sense, especially in the title essay.

(I cannot decide whether the definite description of Edward Said on p. 59 --- "Elegantly attired Palestinian professors at renowned universities write cutting-edge works that command worldwide respect" --- is affectionate or catty.)

Books to Read While the Algae Grow in Your Fur; Pleasures of Detection, Portraits of Crime; Writing for Antiquity; Afghanistan and Central Asia; Scientifiction and Fantastica; The Beloved Republic; Learned Folly; Islam; Tales of Our Ancestors

Attention conservation notice: I have no taste. I also have no qualifications to discuss folklore, structuralism, optics and painting in the early modern Netherlands, Aztec culture, or Cold War espionage.

Vladimir I. Propp, Morphology of the Folktale [as Morfologija skazki, Leningrad, 1928; translated by Svatava Pirkova-Jakobson, Indiana University Press, 1958; second edition, revised by Louis A. Wagner and with an introduction by Alan Dundes, Austin: University of Texas Press, 1968]

I'd known about this book for quite some time, and browsed in it long ago, but never actually read it until this year. It's a really incredible piece of work.

Propp set out to identify the basic elements of the plots of Russian fairy tales, working at a level of abstraction where "it does not matter whether a dragon kidnaps a princess or whether a devil makes off with either a priest's or a peasant's daughter". He came up with 31 such "functions". Just listing them (chapter 3) has a certain folkloric quality:

1. One of the members of a family absents himself from home ($\beta$)
2. An interdiction is addressed to the hero ($\gamma$)
3. The interdiction is violated ($\delta$)
4. The villain makes an attempt at reconnaissance ($\epsilon$)
5. The villain receives information about his victim ($\zeta$)
6. The villain attempts to deceive his victim in order to take possession of him or of his belongings ($\eta$)
7. The victim submits to deception and thereby unwittingly helps his enemy ($\theta$)
8. The villain causes harm or injury to a member of a family ($A$) or One member of a family either lacks something or desires to have something ($a$)
9. Misfortune or lack is made known; the hero is approached with a request or command; he is allowed to go or he is dispatched ($B$)
10. The seeker agrees to or decides upon counteraction ($C$)
11. The hero leaves home ($\uparrow$)
12. The hero is tested, interrogated, attacked, etc., which prepares the way for receiving either a magical agent or helper ($D$)
13. The hero reacts to the actions of the future donor ($E$)
14. The hero acquires the use of a magical agent ($F$)
15. The hero is transferred, delivered, or led to the whereabouts of an object of search ($G$)
16. The hero and the villain join in direct combat ($H$)
17. The hero is branded or marked ($J$)
18. The villain is defeated ($I$)
19. The initial misfortune or lack is liquidated ($K$)
20. The hero returns ($\downarrow$)
21. The hero is pursued ($Pr$)
22. Rescue of the hero from pursuit ($Rs$)

At this point, Propp observes, the tale can more or less begin over again, with the transition from the first "move" to the second being initiated by a new act of villainy, typically "Ivan's brothers steal his prize, and throw him into a chasm" ($\* A$). This leads to $C--G$ again.

23. The hero, unrecognized, arrives home or in another country ($o$)
24. A false hero presents unfounded claims ($L$)
25. A difficult task is proposed to the hero ($M$)
26. The task is resolved ($N$)
27. The hero is recognized ($Q$)
28. The false hero or villain is exposed ($Ex$)
29. The hero is given a new appearance ($T$)
30. The villain is punished ($U$)
31. The hero is married and ascends the throne ($W$)

Each abstract function has, naturally, a great many more concrete sub-types (e.g., seven distinct variants of pursuit, ranging from $Pr^1$, "the pursuer flies after the hero", to $Pr^7$, "He tries to gnaw through the tree in which the hero is taking refuge".

Based on extensive study of the corpus of Russian fairytales, Propp claims that the initial functions, designated by Greek letters, are less essential than the ones designated by Roman letters. In fact, in what I take to be the central finding of the book (ch. IX, sec. D, pp. 104--105), he claims that all the tales in the corpus belong to four, and only four, categories:

1. Tales with a struggle and victory, but no difficult tasks, following the scheme $ABC\uparrow DEFGHJIK\downarrow Pr Rs o LQ Ex TUW$.
2. Tales with difficult tasks, but no combat, following the scheme $ABC\uparrow DEFG o LMJNK\downarrow Pr Rs Q Ex TUW$.
3. Tales which include struggle-victory and difficult tasks, under the scheme $ABC\uparrow FH-IK\downarrow LM-NQ Ex U W$. (That is, the struggle always comes before the difficult tasks.)
4. Tales which include neither struggle nor difficult tasks, under the scheme $ABC\uparrow DEFGK\downarrow Pr Rs Q Ex T U W$.

As he remarks after making these claims,

To the variable scheme \[ ABC\uparrow DEFG \frac{HJIK\downarrow Pr-Rs o L}{LMJNK\downarrow Pr-Rs} Q Ex TUW \] are subject all the tales of our material: moves with $H-I$ develop according to the upper branch; moves with $M-N$ develop according to the lower branch; moves with both pairs first follow the upper part and then, without coming to an end, develop following the lower offshoot; moves without either $H-I$ or $M-N$ develop by bypassing the distinctive elements of each. [p. 105]

What I find so astonishing here is that this is a formal grammar, though propounded many years before that notion emerged in linguistics, logic and computer science. Specifically, it is a formal grammar which generates fairytale plots. Propp realized this, and used the schema to create new fairy tales [1] (unfortunately, not recorded). A basic principle of formal language theory is that a schema which generates all and only the valid strings of a language can also be used to recognize whether a string belongs to that language; Propp implicitly grasped this, and argued on this basis that some non-fairy-tales in his corpus were more properly classed with the fairy tales.

It's especially noteworthy to me that Propp's schema is a regular grammar, i.e., at the lowest level of the Chomsky hierarchy. These correspond to the regular expressions familiar to programmers, to finite-state machines, and to (functions of) Markov chains. The production rules would be something like \[ \begin{eqnarray*} Story & \rightarrow & ActI Act2 Act3\\ Act1 & \rightarrow & ABC\uparrow DEFG\\ Act2 &\rightarrow & (Struggle | 0) (Task | 0)\\ Act3 & \rightarrow & Q Ex TUW\\ Struggle & \rightarrow & HJIK\downarrow Pr-Rs o L\\ Task & \rightarrow & LMJNK\downarrow Pr-Rs\\ \end{eqnarray*} \] using $|$ as usual to represent alternatives, and $0$ to represent a null story element. There would, then, have to be further production rules where abstract villainy, pursuit, marking of the hero, etc., are differentiated into their more concrete types.

In a pure regular grammar, which choice gets made at each application of a production rule is totally independent of the choices made at every other application of a rule. (This is because regular languages are a sub-type of "context free" languages, and is what gives both kinds of language their madlibs flavor.) Propp is at some pains to argue (pp. 109--113) that this is very, very nearly true of fairytales. The exceptions are few enough that they could, I think, be handled within the finite-state, regular-grammar framework, by expanding the set of non-terminal symbols a little.

To sum up, Propp did grammatical induction on fairytales by hand, in 1928, and came up with a regular language.

Naturally, I have questions.

• How reliably can people identify his plot functions in the text of tales?
• Do all tales in his corpus in fact fit his grammar?
• How much interpretive violence is necessary to make an arbitrary story seem to fit his schema? If it's easy to warp anything so it seems to fit, this becomes much less interesting.
• Do other story corpora show the same set of functions? If so, do they follow the same story grammar? (My sense is that the overwhelming majority of fiction I read either conforms exactly, or is very close.)

I am sure that folklorists must have tackled questions like this, and I would very much appreciate pointers to the literature.

[1] Propp, pp. 111--112, on how his conclusions "may also be verified experimentally":

It is possible to artificially create new plots of an unlimited number. All of these plots will reflect the basic scheme, while they may not resemble one another. In order to create a tale artificially, one may take any $A$, then one of the possible $B$'s then a $C\uparrow$, followed by absolutely any $D$, than an $E$, then one of the possible $F$'s, then any $G$, and so on. In doing this, any element may be dropped (except possibly for $A$ or $a$), or repeated three times, or repeated in various forms. If one then distributes functions according to the dramatis personae of the tale's supply or by following one's own taste, these schemes come alive and become tales. Of course, one must also keep motivations, connections, and other auxiliary elements in mind.

Unfortunately, Propp provides no samples of tales generated in this manner. If they have survived, it would be very interesting to read them. (Dundes, in his introduction to the 2nd English edition, mentions "programm[ing] a computer" to do this, but I haven't tracked down that reference (Alan Dundes, "On Computers and Folklore", Western Folklore 24 (1965): 185--189).

He immediately goes on:

The application of these conclusions to folk creation naturally requires great caution. The psychology of the storyteller and the psychology of his creative work as a part of the over-all psychology of creation must be studied independently. But it is possible to assume that the basic, vivid moments of our essentially very simple scheme also play the psychological role of a kind of root.

Again, this cries out for follow-up study, which may well have been done. ^.

Indra Das, The Devourers

Mind candy, historical fantasy/horror division: European shapechangers (not werewolves, exactly) in Mughal India, and modern Calcutta. Angst ensues. (I was very disappointed that the narrator's deep dark secret, at the very end, proves to be something as mundane as cross-dressing.)

Laura J. Snyder, Eye of the Beholder: Johannes Vermeer, Antoni van Leeuwenhoek, and the Reinvention of Seeing

Parallel lives of two 17th century inhabitants of Delft --- not revelatory as either art history or history of science, but deftly done, even to the explanations of some fairly involved optics.

Garrett Birkhoff, Hydrodynamics: A Study in Logic, Fact, and Similitude

This is Birkhoff surveying the state of hydrodynamics in 1950, and in particular looking at why some theoretical results so conspicuously fail to match observations (the "logic" part), and when the uses of physical scale models can be justified (the "similtude" part). For the former, his diagnosis is not just mathematical sloppiness on the part of physicists, or making inappropriate approximations, but taking inconsistent assumptions. The latter part largely turns on the method of dimensional analysis, its limitations, and how it can be seen as a special case of more general group-theoretic approaches to finding similar solutions to partial differential equations.

These are, of course, more general morals about mathematical modeling, but nobody who isn't pretty familiar with hydrodynamics and group theory will get anything out of this book.

An incidental observation: It's striking to me that Birkhoff cites many much (relatively) older works than a contemporary writer would, and that he cites plenty of French- and German- language publications. (I can't remember if he cites any Italian or Russian works in the original, rather than translations.) There is a little lesson here about the transformation of post-war science...

Donald B. Rubin, Multiple Imputation for Nonresponse in Surveys

"Imputation" is the more dignified name statistics gives for "making stuff up to fill in missing observations". (To be fair, that's a mouthful.) Rubin was, back in the day, a very forceful and necessary advocate for multiple imputation, i.e., for making up a whole bunch of different things to fill in the missing observations, and trying them all out, to make sure that your results aren't just creatures of accidents in your imputation. While this is clearly almost always a better idea than single imputation, there are also, clearly, some details that will need to be pinned down. This short (258 pp.) book does a remarkably good job of pinning down those details in an easy-to-follow way. It also includes a summary of Rubin's equally-influential work on missing data, i.e., when exactly it's a problem for what kinds of inferences. Some of the computational advice is antiquated, and I could have wished it was less Bayesian, but it's still a very nice piece of work.

Ob. "Why Oh Why Can't We Have a Better Academic Publishing System": Wiley has kept the book in print, as a "classic", but the list price is \$ 158, or over sixty cents per page.

Inga Clendinnen, Aztecs: An Interpretation

This is a brilliant book, a learned, intelligent and sympathetic attempt to try understand something of how things must have felt to what must stirke most readers as a very strange and unsympathetic society. I also cannot help but feel that huge chunks of it are massive speculations, starting from the first substantive chapter on Aztec notions of "the sacred" and going on from there. The sources seem to me just too thin, and too peculiar*, to support the very elaborate interpretations Clendinnen erects upon them. (Which doesn't mean she was wrong.) But she was the expert who was immersed in the source material, not me.

[*]: Largely, they are accounts given many decades after the conquest, and it seems unclear how much of them was the sources recalling what happened, as opposed to giving their views about what ought to have happened, or what they wanted Spanish missionaries and their helpers to think happened. Even if they were doing their best to stick to their memories of the facts, they couldn't possible have experienced, say, the new fire ceremony which took place every 52 years more than once, and that, perhaps, when they were quite young. ^

David E. Hoffman, The Billion Dollar Spy: A True Story of Cold War Espionage and Betrayal

In which the poisonous legacy of the Stalinist purges inspires a Soviet engineer to volunteer to spy for the US, extremely successfully.

Books to Read While the Algae Grow in Your Fur; Writing for Antiquity; The Commonwealth of Letters; Physics; Scientifiction and Fantastica; Enigmas of Chance

Attention conservation notice: Over 14,000 words about an archaic monograph on political economy. Either you already care about it, in which case this adds nothing original, or you don't, in which it's vanishingly unlikely to spark an interest. Contains an extra 1920 words of quotations, a fair bit of linear algebra, and claims of being able to say what one of the great thinkers of western civilization meant better than he could. Finally, while I come from the sort of family where a great-uncle's middle name was, in fact, "Karl Marx", I am a mere squishy social democrat with liberal tendencies and a weakness for markets as tools for coordination and feedback, so I am at once too close for objectivity and too remote for involvement. Finally, I am, of course, neither an economist nor a historian of 19th century thought nor an expert on Marxism.

(R Markdown source file)

Corrupting the Young; Enigmas of Chance

In which we see how to use linear models without assuming that they are correct, or that anything at all is even remotely Gaussian.

(R Markdown source file)

Corrupting the Young; Enigmas of Chance

Attention conservation notice: An exhortation to the young to demonstrate a literally-academic virtue which I myself find hard to muster.

Written a few years ago, and excavated from the drafts folder because I was preaching the same sermon in e-mail.

Having found myself having to repeat the same advice with more than usual frequency lately, I thought I would write it down. This is the importance of grasping, or really of making part of one's academic self, two truths about peer review.

1. The quality of peer review is generally abysmal.
2. Peer reviewers are better readers of your work than almost anyone else.

The first truth will speak to itself for any academic — or, if you're just starting out, trust me, it will soon. Drawing a veil over reports which mere products of nepotism and intrigue *, referee reports are often horrible. The referees completely fail to understand ideas we've adapted to the meanest understanding, they display astonishing gaps in their knowledge, and lots of them can't (as my mother puts it) think their way out of a wet paper bag. Even if you discard these as mere dregs, far too many of the rest seem to miss the point, even points which we've especially labored to sharpen. Really good, valuable referee reports exist, but they are vanishingly rare.

The second truth is perhaps even more depressing. Even making all allowances for this, your referees have (probably) read your manuscript with more attention, care, sympathy and general clue that most other readers will muster. In the first place: most papers which get published receive almost no attention post-publication; hardly anyone cites them because hardly anyone reads them. In the second place: if one of your papers somehow does become popular, it will begin to be cited for a crude general idea of what it is about, with little reference to what it actually says.

I hope readers will forgive me for illustrating that last notion with a personal reference. My two most popular papers, by far, are both largely negative. (I wish this were otherwise.) One of them might as well have been titled "So, you think you have a power law, do you? Well, isn't that special?", and the other "A social network is a machine for producing endogenous selection bias". Naturally, a huge fraction of their citations come from people using them as authorities to say, respectively, "Power laws, hell yeah!" and "I can just see peer effects". It's actually not uncommon for those papers to be cited as positively endorsing techniques they specifically show are unreliable-to-worthless. This has put me in the odd position, as an anonymous referee myself, of arguing with authors about what is in my own papers.

None of this should be surprising. One of my favorite books is one of the very few thoroughly empirical contributions to literary criticism, I. A. Richard's Practical Criticism. In an experiment lasting over several years in the 1920s, Richards took a few dozen poems, typed up in a uniform format and with identifying information removed, and presented them to literature students at Cambridge University, collecting their "protocols" of reaction to the poems. It is really striking just how bad the students were at receiving even the literal text of the poems, never mind providing any sort of sensible interpretation or reaction. And these were, specifically, students of literature at one of the premier institutions of higher learning in the world. As Richards said (p. 310), anyone who thinks their alma mater could do better is invited to try it **. Poems are not, of course, scientific papers, and I don't know of anyone who's done a translation of Richards's protocols to academic peer review. But I know of no reason to think highly-educated people are systematically much better at reading papers than poems.

The moral I would draw from this is not to seek a world without referees. It is this: whatever your referees find difficult, confusing or objectionable, no matter how wrong they might be on the merits, will give many of your other readers at least as much trouble. Since science is not about intellectual self-gratification but the advancement of public knowledge, this means that we have to take deep breaths, count backwards from twenty and/or swear, and patiently attend to whatever the referees complain about. If they say you're unclear, you were, by that very token, unclear. If they say you're wrong, you have to patiently, politely, figure out why they think that, and re-express yourself in a way which they will understand. Anger or sarcasm, however momentarily gratifying (and wow are they momentarily gratifying) will not actually change anyone's mind, and so they do not actually serve your long-term goal of persuading your readers of your conclusions.

"When the referees have a problem, there's a problem" is, quite literally, one of the most ego-destroying lessons of a life in science, but I am afraid it is a lesson, and the sooner it's absorbed the better.

*: Vanishingly rare, in my experience, but I am here to tell you that it does happen, and that posting an arxiv version with an inarguable date-stamp before you submit is always a good idea. ^

**: Admittedly, this was before access to higher education exploded after WWII, thereby driving up the average intellectual level of university students, but replications in the 1970s were not noticeably more encouraging. (I would be extremely interested in more recent replications.) ^

Learned Folly

(R Markdown source file)

Corrupting the Young; Enigmas of Chance

Attention conservation notice: Last-minute notice of a technical talk in a city you don't live in. Only of interest if you (1) care actor/critic or co-training methods for fitting generative models, and (2) have free time in Pittsburgh this afternoon.

I have been remiss in blogging the statistics department's seminars for the new academic year. So let me try to rectify that:

Arthur Gretton, "The Maximum Mean Discrepancy for Training Generative Adversarial Networks"

Abstract: Generative adversarial networks (GANs) use neural networks as generative models, creating realistic samples that mimic real-life reference samples (for instance, images of faces, bedrooms, and more). These networks require an adaptive critic function while training, to teach the networks how to move improve their samples to better match the reference data. I will describe a kernel divergence measure, the maximum mean discrepancy, which represents one such critic function. With gradient regularisation, the MMD is used to obtain current state-of-the art performance on challenging image generation tasks, including 160 × 160 CelebA and 64 × 64 ImageNet. In addition to adversarial network training, I'll discuss issues of gradient bias for GANs based on integral probability metrics, and mechanisms for benchmarking GAN performance.

Time and place: 4:00--5:00 pm on Monday, 24 September 2018, in the Mellon Auditorium (room A35), Posner Hall, Carnegie Mellon University

As always, talks are free and open to the public.

Enigmas of Chance

(R Markdown source file)

Corrupting the Young; Enigmas of Chance

I have fallen behind on posting announcements for the lectures, and I don't feel like writing five of these at once (*). So I'll just list them:

9. Separating Signal and Noise with Linear Methods (a.k.a. the Wiener filter and seasonal adjustment; .Rmd)
10. Fourier Methods I (a.k.a. a child's primer of spectral analysis; .Rmd)
11. Midterm review
12. Guest lecture by Prof. Patrick Manning: "African Population and Migration: Statistical Estimates, 1650--1900" [PDF handout]
13. Linear Generative Models for Time Series (a.k.a. the eigendecomposition of the evolution operator is the source of all knowledge; .Rmd)
14. Linear Generative Models for Spatial and Spatio-Temporal Data (a.k.a. conditional and simultaneous autoregressions; .Rmd)

*: Yes, this is a sign that I need to change my workflow. Several readers have recommended Blogdown, which looks good, but which I haven't had a chance to try out yet.

Corrupting the Young; Enigmas of Chance

Attention conservation notice: 2700+ words elaborating a presentation from a non-technical conference about AI, where the conversation devolved to "blockchain" within an hour; includes unexplained econometric jargon. Life is short, and you should have more self-respect.

Inference I: How Inference with Independent Data Really Works (.Rmd)

Inference II: Ergodic Theory and Inference under Dependence (.Rmd)

Corrupting the Young; Enigmas of Chance

Lecture 16 was canceled.

(.Rmd)

Corrupting the Young; Enigmas of Chance

I met Joyce through her late husband Stephen, my admired and much-missed colleague. I won't pretend that she was a close friend, but she was a friend, and you could hardly hope to meet a kinder or more decent person. A massacre by a deluded bigot would be awful enough even if his victims had been prickly and unpleasant individuals. But that he murdered someone like Joyce --- five blocks from where I live --- makes it especially hard to take. I am too sad to have anything constructive to say, and too angry at living in a running morbid joke to remember her the way she deserves.

Lecture 18: The Bootstrap (.Rmd)

Lecture 19: The Method of Simulated Moments and Indirect Inference (.Rmd)

Corrupting the Young; Enigmas of Chance

Attention conservation notice: Announcement of an advanced undergraduate course at a school you don't attend in a subject you don't care about.

I will be teaching 36-402/36-608, Advanced Data Analysis, in the spring.

This will be the seventh time I'll have taught it, since I took it over and re-vamped it in 2011. The biggest change from previous iterations will be in how I'll be handling class-room time, by introducing in-class small-group exercises. I've been doing this in this semester's class, and it seems to at least not be hurting their understanding, so we'll see how well it scales to a class with four or five times as many students.

(The other change is that by the time the class begins in January, the textbook will, inshallah, be in the hands of the publisher. I've finished adding everything I'm going to add, and now it's a matter of cutting stuff, and fixing mistakes.)

Advanced Data Analysis from an Elementary Point of View

(.Rmd)

Corrupting the Young; Enigmas of Chance

Attention conservation notice: I have no taste. I also have no qualifications to discuss corporate fraud.

John Carreyrou, Bad Blood: Secrets and Lies in a Silicon Valley Startup

This is a deservedly-famous story, told meticulously. It says some very bad things about the culture around Silicon Valley which made this fraud (and waste) possible. (To be scurpulously fair, investment companies with experience in medical devices and the like don't seem to have bought in.) It also says some very bad things about our elites more broadly, since lots of influential people who were in no position to know anything useful about whether Theranos could fulfill its promises endorsed them, apparently on the basis of will-to-believe and their own arrogance. (I hereby include by reference Khurana's book on the charisma of corporate CEOs, and Xavier Marquez's great post on charisma.)

The real heroes here are, of course, the people who quietly kept following through on established procedures and regulations, and refused to bend to considerable pressure.

Luca D'Andrea, Beneath the Mountain

Mind candy: in which a stranger investigates the secrets of a small, isolated community's past, for multiple values of "past".

Walter Jon Williams, Quillifer

Misadventures of a rogue in a fantasy world whose technology level seems to be about the 1500s in our world. Quillifer has some genuinely horrible things happen to him, and brings others on himself, but keeps bouncing back, and keeps his eye on various main chances (befitting the only law clerk I can think of in fantasy literature who isn't just cannon-fodder). I didn't like him, exactly, but I was definitely entertained.

Books to Read While the Algae Grow in Your Fur; Pleasures of Detection, Portraits of Crime; Scientifiction and Fantastica

Attention conservation notice: I have no taste. I also have no qualifications to discuss the history of photography, or of black Pittsburgh.

Cheryl Finley, Laurence Glasco and Joe W. Trotter, with an introduction by Deborah Willis, Teenie Harris, Photographer: Image, Memory, History

A terrific collection of Harris's photos of (primarily) Pittsburgh's black community from the 1930s to the 1970s, with good biographical and historical-contextual essays.

Disclaimer: Prof. Trotter is also on the faculty at CMU, but I don't believe we've ever actually met.

Ben Aaronovitch, Lies Sleeping

Mind candy: the latest installment in the long-running supernatural-procedural mystery series, where the Folly gets tangled up with the Matter of Britain.

Charles Stross, The Labyrinth Index

Mind candy; Latest installment in Stross's long-running Lovecraftian spy-fiction series. I imagine a novel about the US Presidency being taken over by a malevolent occult force seemed a lot more amusing before 2016, when this must have been mostly written. It's a good installment, but only suitable for those already immersed in the story.

Anna Lee Huber, The Anatomist's Wife and A Brush with Shadows

Mind-candy, historical mystery flavor. These are the first and sixth books in the series, because I couldn't lay hands on 2--5, but I will. (Update: More.)

Books to Read While the Algae Grow in Your Fur; Scientifiction and Fantastica; Pleasures of Detection, Portraits of Crime; Tales of Our Ancestors; Cthulhiana; Heard About Pittsburgh, PA