Ride the Lightning

ROSS AI Plus Wexis Bests Westlaw and Lexis Nexis Alone in Study

January 19, 2017

This post has been updated to change the title, the original version of which made an inaccurate implication: See Bob Ambrogi's update here. The study did not compare ROSS as a standalone product against Westlaw and LexisNexis. Rather, it was always ROSS plus some part of either Westlaw or LexisNexis. Thanks Bob. I will share that crow pie with you. Pass the salt please 🙂

Yet another hat tip to my friend Bob Ambrogi. On January 17^th, Bob put out a fascinating post describing a study which found that ROSS Intelligence outperformed Westlaw and LexisNexis in finding relevant authorities, in user satisfaction and confidence, and in research efficiency. The study concluded that it is virtually certain to deliver a positive return on investment.

Bob's post is your best source of information about the study which will be made available for download at the ROSS Intelligence website.

The study, commissioned by ROSS and performed by Blue Hill Research, assigned a panel of 16 experienced legal research professionals to research seven questions modeling real-world issues in federal bankruptcy law – and yes, that is Ross' turf. Researchers were divided into four groups, with each group assigned to perform the research using a particular method:

Boolean search, in which researchers used only Boolean keyword search capabilities of major legal research platforms.

Natural language search, in which researchers used only the natural language search capabilities of major legal research platforms.

ROSS and Boolean Search, in which researchers were directed to use the ROSS platform and Boolean keyword search capabilities of major legal research platforms as they saw fit.

ROSS and natural language search, in which researchers were directed to use the ROSS platform and natural language search capabilities of major legal research platforms as they saw fit.

Of the Boolean and natural language search tools, researchers were assigned a mixture of Westlaw and LexisNexis tools. In addition, members of each research group were limited to individuals with no prior experience with the assigned tool and relatively minimal experience with bankruptcy law.

Research results were benchmarked against three assessment factors:

1. Information retrieval quality, including thoroughness, accuracy and ranking effectiveness.
2. User satisfaction, including ease of use and confidence in answer obtained.
3. Research efficiency, based on time to obtain a satisfactory answer.

The study concluded that ROSS generated better search results across all three benchmarks. "These findings indicate clear advantages resulting from the addition of the ROSS tool to electronic legal research involving traditional tools," the report concludes.

Perhaps most striking about the findings was what they showed about Boolean search. Boolean tools retrieved less than a third of relevant authorities within their first 20 results and only 25.8 percent of their results were relevant authorities. By comparison, ROSS retrieved 55.8 percent of the relevant authorities within its top 20 results and 37.9 percent of results were relevant authorities.

ROSS most significantly outperformed Boolean and natural language tools in its ranking of results, the report found. Using the NDCG standard described above, ROSS achieved a score of .61, which was 46.1 percent higher than natural language and 127.8 percent higher than Boolean.

Users took a survey which indicated user satisfaction of the participants using a ROSS-supported toolset with respect to the usability, presentation of search results, and inclusion of relevant authorities within the search results. Participants within these groups also indicated high levels of confidence in the ability of the tool to identify all authorities relevant to the matter. In nearly all cases, the responses indicated by participants using ROSS and another tool often exceeded those of organizations using only Boolean search or only Natural Language search by at least a full point.

In terms of research efficiency, the study found that the ROSS users reduced their research time by 30.3 percent over users of Boolean search and 22.3 percent over users of natural language search.

Based on these findings, Blue Hill went on to quantify how the research efficiency it found using ROSS would impact the bottom line for law firms and legal organizations. To do this, it compared the net gain that ROSS provides against the cost of acquiring it.

Blue Hill's analysis looked at how the reduction in research time from using ROSS could be converted to productive, revenue-generating work. It estimated that for an associate billing $320 an hour, the conversion could generate revenue of from $5,306 if 10 percent of the saved time is converted to $61,868 if 100 percent of the saved time is converted.

That translates to a return on investment of anywhere from a low of 10.5 percent to a high of nearly 2,500 percent, again depending on the extent of the conversion. The report says,

Notably, using Blue Hill's estimated billable rate of $320, a positive ROI is obtained at a 10% conversion rate, meaning that the investment drives a net gain to the firm with a minimal recovery of written-off hours. Similarly, a 176.4% to 544.5% ROI becomes possible with at least 25% conversion. Where exactly an organization will fall within these ranges will depend on a variety of factors related to its business, investment costs, and the scope of use adopted. In all cases, however, Blue Hill's model strongly suggests positive business gain is available from the investment in ROSS.

The report concludes:

Blue Hill finds that the ROSS tool provides significant, additive contributions to the effectiveness of legal researchers. … These results have the potential to unlock new gains in the efficient and profitable operation of legal organizations, as well as create opportunities for new revenue gain.

But it does offer these thoughts:

It should be noted that none of these findings indicate that AI-assisted legal research constitutes a dramatic transformation in the use of technology by legal organizations. Rather, the use cases and impact reviewed indicate that tools like ROSS Intelligence more closely represent a significant iteration in the continuing evolution of legal research tools that began with the launch of digital databases of authorities and have continued through developments in search technologies.

I agree with Bob that the study should not be discounted because Ross commissioned it. Blue Hill is a well-regarded independent research firm. I also agree that this is not a "robots replacing lawyers" study. ROSS was used as a tool to assist lawyers in research, not to perform research in their place.

And I further agree with Bob that this study is important in establishing the effectiveness and value of AI, which is often questioned by skeptics.

While ROSS was trained first on bankruptcy law, it is only a matter of time before ROSS is trained in many area of laws. Remember the vintage Chinese curse? "May you live in interesting times." We sure do – on many fronts . . .

E-mail: Phone: 703-359-0700
Digital Forensics/Information Security/Information Technology
http://www.senseient.com
http://tw
i
tter.com/sharonnelsonesq
http://www.linkedin.com/in/sharondnelson
https://amazon.com/author/sharonnelson