Looking for 19th Century Authors

Summary

• Jockers, Witten, and Criddle attempted to address the question of whether two, 19th century authors combined might have produced the Book of Mormon, thus explaining why earlier studies found that the Book of Mormon styles didn't match single 19th century authors.
• They wrote a long paper that failed to demonstrate anything. At least 19.8% of unattributed Book of Mormon chapters are objectively mis-attributed. A significant percentage of others are plausibly mis-attributed after examining the results of Jockers's 2013 paper.
• Without even examining the papers produced by BYU and Maxwell Institute authors, only those committed for subjective reasons to the idea of purely known, 19th century authorship have any grounds for believing a purely 19th century authorship hypothesis, unless . . .
• . . . Joseph Smith created multiple fraudulent authorship styles. Of course, without evidence of Joseph attempting to make up different styles, this is pure speculation. It could be true, and if it is, Joseph, I can't love you for it--but I can respect your ever growing genius. . .

http://www.matthewjockers.net/publications/ (Preprints available)
Jockers, Matthew L. “Testing Authorship in the Personal Writings of Joseph Smith Using NSC Classification.” Literary and Linguistic Computing. 28.3, (2013): 371-381
Jockers, Matthew L., Daniela M. Witten, and Craig S. Criddle. “Reassessing Authorship of the Book of Mormon Using Delta and Nearest Shrunken Centroid Classification.” Literary and Linguistic Computing, 23.4 (2008): 465 – 492.

The 2008 Paper

If you take the time to read the background presented in any of the stylometry papers, you will find they are all very self-serving. That's normal for academic papers. They want to play up the importance of their contribution as much as possible. You have to be a salesperson if you want to keep getting paid. This paper from Jockers, Witten, and Criddle spends a lot of time pointing out the flaws of earlier studies. Many of the things they point to as flaws are, to my mind, not flaws at all, but are instead asking different questions. They criticize all three of the earlier studies for two things: 1. 'arbitrarily' dividing up the Book of Mormon text according to the authors indicated in the text, and 2. not considering that two 19th century authors might have written the text, together, thus giving a stylistic signature that is different from any individual author. I have three reactions to this. First, it is a valid question to ask if the styles of authors indicated by the text have distinct and self-consistent styles. The earlier studies did this in logical ways, and that is hardly a fault. It admittedly doesn't cover all possibilities for Book of Mormon authorship. Second, what does a combined or collaborative authorship signature look like? Third, the question of dual authorship is a valid question to ask. I won't go into the more detailed criticisms. Many of them are valid but have almost no consequence on the conclusions we can draw. I'll be happy to reexamine and discuss them on an individual basis upon request. Let's take a look at the results.

The authors use two methods, Delta and Nearest Shrunken Centroid. The first is an established method for authorship attribution. The second is borrowed from other statistical comparison problems. Both certainly have potential to illuminate authorship questions, so we will trust that the authors can do the math and programming right and skip straight to how the question is framed.

The samples of unknown authorship were the 239 chapters of the modern Book of Mormon (but with the 1830 text). Ultimately, 110 words (mostly noncontextual) were used to characterize the texts. These 239 texts were compared against 7 known authors: Oliver Cowdery, Parley Pratt, Sidney Rigdon, Solomon Spalding, Isaiah and Malachi, and Joel Barlow and Henry Wadsworth Longfellow as controls. The test was designed to rank which author's style was closest, 2nd closest, 3rd closest, etc. to each of the 239 unknown samples. Before running the test on the Book of Mormon, the authors did an important control.

There were 217 samples from the 7 known authors. Jockers et al. removed a small number of known samples from the 217 member training set, say 17 of them. They pretended like these were unknown, and classified them based on the remaining 200. In doing this a number of times, they only misclassified texts 8.8% with the NSC method. They got the author right over 90% of the time. Remember this number when we start looking at the Book of Mormon results. Jockers et al. were pretty thorough with these internal controls, and ran their comparisons a number of different ways to see if minor variations in how they formulated the questions resulted in big changes in the results. Bottom line was they didn't, so I will focus on just the main set of results.

	# of Book of Mormon Chapters Assigned to Author
Proposed Author	1st Choice	2nd Choice
Rigdon	93	104
Isaiah & Malachi	63	38
Spalding	52	58
Cowdery	20	17
Pratt	9	15
Barlow	0	1
Longfellow	2	6

This is an excerpt from the first table in the 2008 paper. We see that 38.9% of the chapters were assigned to Rigdon, 26.4% to Isaiah and Malachi, and 21.8% to Spalding. We know Isaiah and Malachi are quoted extensively in the Book of Mormon, and this powerful method (remember it got the authors right over 90% of the time on the test sets) identified Rigdon and Spalding as the authors of 60.7% of the text. That's most of the text that isn't Biblical quotes. With evidence like this, the Rigdon-Spalding theory of authorship looks pretty good, despite the tenuous historical support for the theory. Some problems do start to appear when we look closer, however. Isaiah and Malachi only wrote 36 chapters in the Book of Mormon. That means 13.3% of the chapters not written by Isaiah and Malachi are falsely attributed to them. Add to that the 2 assigned to Longfellow and the 9 assigned to Pratt, and 18.7% of the unattributed chapters are known false positives.

Now let's remember the initial test set and the results of the 2013 study. Only 8.8% of the test texts were assigned incorrectly by NSC. We know, based solely on chapters we know weren't penned by Isaiah, Malachi, Pratt, and Longfellow that the method is getting it wrong 18.7% of the time. From the 2013 study we saw that Rigdon and Spalding showed up as false positives 19.8% of the time, and that was in a data set where the actual author was nominally known. In addition, Cowdery showed up most often when he wasn't even scribe for the texts--31.3% of the total texts. We also saw that the NSC method got it 'right' only 15.6% of the time. No matter how you spin this, the chance that a large percentage of these Book of Mormon chapters are mis-attributed is far from zero. Even authorship pairing doesn't help. In the 2013 study, Rigdon and Spalding showed up paired 5 times for texts dictated to 5 different scribes. One of those was for Smith dictating to Cowdery, so maybe by a stretch you could say that the strong Rigdon-Spalding signal is actually evidence of Smith dictating to Cowdery--except no other previous group found a match between those dictated diaries and letters and the Book of Mormon. The only thing I can possibly see us learning from finding strong Rigdon, Spalding, and Cowdery signals in the Book of Mormon is that there are at least three distinct authorship styles in the Book of Mormon. And that's just from looking at the results produced by Jockers and his coworkers. If we look at the results from their Delta method, the picture looks even worse. Isaiah and Malachi wrote almost half of the Book of Mormon by that method. Then there's the trump card.

Jockers et al. asked their authorship question with closed-set methods. From the very start their method says: "One of these seven authors wrote each chapter of the Book of Mormon." This fails to answer the question of Book of Mormon authorship on two levels. The first is internal to their framing. Jockers et al. suggested that the Book of Mormon might have been written by two or more authors in collaboration. Besides being unable to demonstrate such collaboration historically, they didn't present a single control experiment illustrating what happens to stylometric signatures when authors collaborate. They only looked to see if chapters could be assigned to one author, not to collaborating authors. Finding authors ranked first and second only tells you that those authors both have similar styles to the passage, not that those authors collaborated to produce it. The second way it fails is best illustrated.

A closed-set method will always give a positive answer. Here's a simplified visual representation of what this closed-set study has definitively shown us:

I've collapsed the many dimensional stylometric measures of the candidate authors into 1 dimension. In reality, they will overlap in different ways in different dimensions, but the concept is easier to visualize in two. Each author has a distinct style with some overlap with one or more candidate authors. The blue numbers represent hypothetical styles of chapters of the Book of Mormon. What a closed-set study tells us is not whether the chapter has the same style as a candidate author, but whether it is closest to that author. For example, chapter 1 would be assigned to Cowdery, chapter 5 to Pratt, 6 to Spalding, etc., despite none of the chapters matching the styles of the candidate authors. Without some absolute measure of style, we just don't know how good the matches really are.

There is one important thing we can conclude from the study--Book of Mormon stylometric signatures span the range of at least 4-5 normal authors. This conclusion is 100% consistent with what we saw from Holmes's studies.

Conclusions

Anyone who thinks this study shows anything conclusive about the Book of Mormon, except confirming that it is likely the product of at least four different authors, is not looking at the data critically or objectively. Even if the person is rational in believing that the Book of Mormon is a purely 19th century product, this study gets them no closer to identifying who the 19th century author or authors were. Absolute measures of style, measures of what happens to style when authors collaborate, or an open set method, would be required to give us that knowledge.

Next Time

I'll start looking at what you learn when applying open-set methods to Book of Mormon authorship questions, where you allow the possibility that at least one unknown author might have written the text.