But forward citations are not without their warts. For example, both my longitudinal study of highly litigious NPEs and random patent litigants and Allison, Lemley & Scwhartz's cross-sectional study of all patent cases filed in 2008-2009 found that forward citations had no statistically significant impact on patent validity determinations. Additionally, Abrams, et al., found that actual licensing revenue followed an inverted "U" shape with respect to forward citations (Lisa writes about that paper here). That is, revenue grew as citations grew, but after a peak, revenues began to fall as forward citations grew even larger. This implies that the types of things we can measure with forward citations may be limited by just how many there are, and also by the particular thing we are trying to measure.
This is why it was so great to see a new NBER paper in my SSRN feed yesterday (it's not totally new - for those who can't get NBER papers, a draft was available about a year ago). The paper, by Petra Moser (NYU), Joerg Ohmstedt (Booz & Co.) & Paul W. Rhode (UNC) is called Patent Citations and the Size of the Inventive Step - Evidence from Hybrid Corn. The abstract follows:
Patents are the main source of data on innovation, but there are persistent concerns that patents may be a noisy and biased measure. An important challenge arises from unobservable variation in the size of the inventive step that is covered by a patent. The count of later patents that cite a patent as relevant prior art – so called forward citations – have become the standard measure to control for such variation. Citations may, however, also be a noisy and biased measure for the size of the inventive step. To address this issue, this paper examines field trial data for patented improvements in hybrid corn. Field trials report objective measures for improvements in hybrid corn, which we use to quantify the size of the inventive step. These data show a robust correlation between citations and improvements in yields, as the bottom line measure for improvements in hybrid corn. This correlation is robust to alternative measures for improvements in hybrid corn, and a broad range of other tests. We also investigate the process, by which patents generate citations. This analysis reveals that hybrids that serve as an input for genetically-related follow-on inventions are more likely to receive self-citations (by the same firm), which suggests that self-citations are a good predictor for follow-on invention.I love this study because it ties something not only measurable, but objective, to the forward citations. This is something that can't really be done with litigation and licensing studies, both of which have a variety selection effects that limit the random (shall we say, objective) nature of them. More on this after the jump.
The study makes an important point (with evidence) that most who are familiar with the patent system intuitively felt: raw patent counts may not be the best measure of technological value. In corn, at least, many of the improved varieties are not as productive as their forbears. This is consistent with other studies that find many patents to be of low value, and that many patents do not get renewed when maintenance fees are due.
If, however, we look at citations to patents, we can separate the wheat from the chaff (or the kernel from the husk, if you will). It appears that future patents don't cite to random corn patents, but instead to those patents that, on average, performed better. Interestingly, the authors find that highly cited patents are associated with even more increase - the cornerstone patents in the field. This contradicts the inverted U results a bit. But even if the highly cited patents are removed, an increase in corn yield still correlates with more citations.
I think this is also good news for the information value of patents - apparently enough is known and conveyed to tie the important patents to the important technology, rather than just a mish mosh of information.
Further, the analysis of self-citations is interesting. Though the economic literature sees this as an open question, I think there is a conventional wisdom that self-citations are valueless, and should not be counted. The authors, however, find that self-citation (at least in the studied field) is indicative of cumulative innovation. That is, you can trace growth in quality of invention because the better stuff is being cited and built on. I'm going to remember this as I rationalize citations to my own work in future articles....
There's a lot more to this article, and it's well worth a read. While there are surely technical questions (is it correlational or causal, and does it matter), I think the biggest question is whether the results are generalizable. Is there something about the corn (or other hybrid crop) industry that makes the citations track well? Can we trust that this type of correlation will translate to other industries, where inventive steps are more difficult to measure? I don't know the answers to these questions, but it inspires me to think of ways we might objectively measure quality in order to find out.
