Susan Nevelow Mart’s Results may vary in legal research databases published in ABA Journal

“Call me naive,” wrote Bob Ambrogi about the conclusions reached by Susan Nevelow Mart in her excellent research found in The Algorithm as a Human Artifact: Implications for Legal {Re}Search, “but I would have thought that entering the identical search query on, say, both Westlaw and Lexis Advance would return fairly similar results.” I was somewhat shocked that Bob hadn’t realized what law librarians have known for 40 years, an insight made even more important since WEXIS implemented “black box” searching because it won’t release proprietary information about the construction of their search algorithms.

Perhaps the editors of the ABA Journal thought their readers might also be naive because the Journal published a summary of Nevelow Mart’s research results in the March issue. See Results may vary in legal research databases by Susan Nevelow Mart. Here’s a snip:

At first glance, the various legal research databases seem similar. For instance, they all promote their natural language searching, so when the keywords go into the search box, researchers expect relevant results. The lawyer would also expect the results to be somewhat similar no matter which legal database a lawyer uses. After all, the algorithms are all trying to solve the same problem: translating a specific query into relevant results.

The reality is much different. In a comparison of six legal databases—Casetext, Fastcase, Google Scholar, Lexis Advance, Ravel and Westlaw—when researchers entered the identical search in the same jurisdictional database of reported cases, there was hardly any overlap in the top 10 cases returned in the results. Only 7 percent of the cases were in all six databases, and 40 percent of the cases each database returned in the results set were unique to that database. It turns out that when you give six groups of humans the same problem to solve, the results are a testament to the variability of human problem-solving. If your starting point for research is a keyword search, the divergent results in each of these six databases will frame the rest of your research in a very different way.

Highly recommended. This ABAJ article should be assigned reading for law school students — all law school students, not just 1Ls taking their LRW courses. — Joe

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s