How do we evaluate the quality and value of legal services? For example, if we compare two proposed contracts for a commercial agreement, how do we determine which contract is of higher quality? How do we determine the total value produced by the process of drafting, negotiating, and finalizing each contract? Would our answers change if some or all of the services are produced by a software application? If a software application is used, how would we evaluate the quality of any training data inputs, the development process, and the outputs of the software application? Would our assessment of the quality and value of the software application change if the software application is used to serve individuals who would otherwise go without a lawyer?

These are just some of the questions that I discuss in this draft book chapter, Evaluating Legal Services: The Need for a Quality Movement and Standard Measures of Quality and Value, the final version of which will be available in the Research Handbook on Big Data Law edited by Dr. Roland Vogl, forthcoming 2020, Edward Elgar Publishing Ltd. I’ve included the abstract for my chapter below.

With this chapter, I aim to demonstrate the need for a quality movement and standard measures of quality and value and highlight some of the research and resources. My goal is to catalyze debate, rigorous research, and sustained action. If we do not undertake this work, we risk squandering abundant opportunities to improve legal services, legal systems, justice, and the law itself.

I welcome input from everyone working on or interested in these topics. I’ve tried to identify relevant literature and initiatives. Please let me know what I’ve missed. I’ve also made the case for why we need a quality movement and measures of quality and value. Is there a better case to be made? Please download my draft book chapter and let me know what you think.

Finally, thank you to Northwestern Pritzker School of Law student Research Assistants Mona Kalantar, Cheron Z. Mims, and Richard “Hitch” Thomson for their contributions to this project!

Abstract – Evaluating Legal Services: The Need for a Quality Movement and Standard Measures of Quality and Value

The legal industry has not undergone a quality movement and lacks standard measures of legal-services quality and value. Whereas medicine long ago embraced evidence-based practice and empiricism, law muddles along, deferring to lawyers’ untested, loosely conceived, normative standards for practice. As a result, existing data about legal-services delivery is of questionable quality, and we lack standard metrics to evaluate this data and any applications developed with it. The lack of empirical rigor in legal services threatens progress in data analytics and artificial intelligence, which require high-quality input and output data. Additionally, the failure to undertake a quality movement in law contributes to numerous legal industry problems, including inadequate access to legal services and justice, a lack of diversity, and work-life imbalances. This article discusses the need for a quality movement (focused on standard work, error detection, peer review, performance measurement, and continuous improvement) and standards for legal-services quality and value. The article discusses output, process, and input data and metrics for measuring quality and value. The article includes summaries of multiple holistic models for measuring legal-services value, including from Noel Semple, Rebecca Sandefur and Thomas Clarke for “roles beyond lawyers,” and Paul Lippe for contracts. The article also identifies several initiatives contributing to the development of quality and value metrics. Finally, the article briefly summarizes the stakeholder benefits of a quality movement and standard metrics for legal-services quality and value.

Mona Kalantar
Mona Kalantar

Mona Kalantar, a 3L at Northwestern Pritzker School of Law, has been a research assistant for Daniel W. Linna Jr. at since May 2019. In this guest post, Mona highlights ways for attorneys to become data-driven. Doing so will add to the general body of knowledge and create industry-standards that could lead to the improvement of legal services.

Guest post by Mona Kalantar

In his 2012 article, Where is the ‘Quality Movement’ in Law Practice?, Professor William H. Simon argued that the quest for continuous improvement has largely bypassed legal practice. The legal profession has not fully embraced quality reforms that we have seen in other professions. Today, the drive for innovation is leading lawyers to think about how to provide greater value with fewer resources. The market is producing innovations within legal-services delivery organizations that promote streamlining processes, facilitating connectivity between legal professionals and clients, and creating opportunities for alternative fee arrangements. Despite the growing demand for legal-services delivery innovation, there is less evidence of demand for measuring the quality of legal work, performance, and outcomes.

Members of the legal profession could produce great value by developing objective measures for the quality of their legal work, performance, and outcomes. Corporate legal departments, law firms, and other legal professionals should become more data-driven to demonstrate their quality and value. Collecting such data would not only set the firm, corporation, or organization apart from competitors, but also add to the general body of knowledge available about industry-standards that could lead to the improvement of legal services.

Legal Professionals Can Reduce the Gap in Quality Knowledge

Legal professionals can lead the way in reducing the gap in quality knowledge by embracing opportunities to become more data-driven. One way is to address the lack of measurements of attorney performance. Law firms and legal departments that create, use, and retain consistent metrics for performance across the organization can generate data that helps shape attorney behavior and sets quality expectations industry-wide. Clearly established data-driven values help both law firms and legal departments assign work, make decisions, and create business opportunities. 

There are many opportunities for legal professionals to become data-driven. Legal professionals can gather qualitative information such as the reasons, opinions, and motivations underlying others’ assessment of their delivery of certain legal services. Qualitative data can be represented quantitatively by assigning numerical scales to responses. Quantitative metrics, such as tracking email response time and average hourly billing, can also reveal patterns. By fostering a data-driven approach and pursuing opportunities to collect and analyze data, legal professionals can drive innovation and create continuously improving measurements of legal-services quality and outcomes.

Objective, Qualitative Measurements Are Within Reach

A first big step for legal professionals to take advantage of opportunities to become data-driven is standardizing qualitative metrics. Many legal professionals and academics have become interested in the idea of scorecards to measure the value of legal services. The Association of Corporate Counsel’s (“ACC”) scorecard is just one example of the legal profession taking charge of using data-driven metrics to measure work quality. The ACC provides corporate counsel with its recommended scorecard framework for measuring outside firm performance. Their structure suggests eight key metrics: 

  • Understanding client goals;
  • Expertise in the issue; 
  • Responsiveness;
  • Process management;
  • Cost; 
  • Execution of the objective; 
  • Compatibility with company values; and
  • Whether the attorney would hire the firm again.

Corporate legal departments can share their scorecards in the ACC index.

Qualmet has attempted to address the lack of consistent industry standards and make value quantifiable. Started by Mark Smolik and Jim Beckett in 2017, Qualmet uses a scorecard system to generate data-driven value measures of work performed by legal professionals. Presenting standard key performance indicators (KPI’s) for the legal industry saves Qualmet’s customers from spending time debating which metrics to use. Qualmet’s industry-aligned KPI’s and legal metrics are combined with side-by-side firm comparisons that create benchmarks to provide quick and understandable results that law firms can use to improve their services and differentiate themselves. Qualmet also has a searchable database of evaluated service providers ranked on their metrics system to provide meaningful qualitative measurements by which organizations can compare themselves. Qualmet is a solution for legal professionals to understand how to increase their value and become better service providers. 

There is also a growing number of databases with information about outcomes. Clients and law firms are using this data to assess the experience a lawyer has in matters to quantify outcomes. Lex Machina, a division of LexisNexis, gathers data about judges, lawyers, parties, and subject matter to curate data that can help to predict case outcomes that different legal strategies will produce. Similarly, Westlaw Edge provides litigation analytics demonstrating how judges handle different cases and motions, which legal departments can use to evaluate their counsel and opposing counsel’s strengths and weaknesses. 

Quantitative Measurements Reveal Interesting Patterns

Another step forward in measuring the quality of legal work is collecting quantitative information and viewing it in the aggregate. Legal technology has already produced a few data-driven products that help corporations and law firms evaluate specific concerns and make informed decisions, both internally and externally, about their business relationships.

Clerk by Judicata provides data about substantive legal quality. It helps improve brief writing by analyzing and evaluating areas of improvement and attack. CEO and Co-Founder Itai Gurrari created Judicata to generate objective, empirical measures of lawyer quality. Clerk reports a grade for how well a brief is argued, drafted, and compares outcomes with similar cases. 

Other technologies focus on data regarding legal-services delivery to measure cost-effectiveness. Serengeti Tracker (now Thomson Reuters Legal Tracker) also provides comprehensive data-driven metrics. This product allows corporate legal departments to measure their performance and spend and enables legal departments to benchmark their financial performance against peers. Legal Tracker has generated data from over 950 corporate law departments and 42,000 law firms to create aggregated comparative data. 

Legal Tracker allows corporate legal departments to track a variety of quantitative metrics for their outside counsel. For example, Legal Tracker allows corporate legal departments to measure actual spend to budget. It also allows corporate legal departments to measure staff workload metrics. Some metrics include tracking the number of matters performed, variety of tasks performed, spend per-lawyer, average spend per matter type, and time spent on an issue. Other notable quantitative data-driven metrics included spend by matter type and business unit and outside and inside spend as a percentage of company value. 

Nevertheless, while we have more data than ever, it is underutilized to measure the quality of legal work. In my experience, corporate legal departments are emphasizing the importance that outside counsel demonstrate the value that they deliver. Legal departments do not have the luxury of time and money to hire multiple firms on the same matter to compare and contrast work product. Corporate legal departments could use data produced by Clerk to evaluate an outside firm’s performance on litigation matters or use Legal Tracker to showcase their inherent value to management. Legal departments could use the litigation analytics of both Lex Machina or Westlaw Edge to quantify their success rates on motions and cases as compared to other legal departments. These are just a few examples.

Opportunities Are Endless

For the legal professional who is not ready to commit to a product or service, becoming data-driven is not an impossible task. Neither should it be a low priority. Regardless of size or budget, legal professionals can become more data-driven as long as the data is easy to understand, manageable, accurate, and updated. The journey to becoming truly data-driven can start with collecting and analyzing your own “small data.”

Client Satisfaction Data

One manageable approach involves creating client or vendor satisfaction scores. To be effective, leadership must collaborate to identify metrics that align with organizational values. Organizations can borrow from the ACC suggested framework to come up with a variety of questions they would like to ask their outside counsel. A good scorecard explains with specificity what a number means. Ideally, these scores are collected in every interaction and reviewed often to identify areas of strength and weakness.

Risk Mitigation Scorecard

Another data-driven opportunity is to create a risk mitigation scorecard that measures how well legal counsel has met or exceeded expectations on risk. Lawyers on Demand provides an online template for this. 

To create a risk mitigation scorecard, legal departments will need to create a corporate risk mitigation profile for their outside counsel. Corporate risk mitigation profiles encompass a variety of ways that outside counsel has successfully mitigated risk on a matter. Lawyers on Demand recommends that corporate legal departments create a risk profile encompassing a range of standards across a range of contracts entered into. Then legal departments should set a target risk profile for the range of contracts and audit them monthly.

Monitoring the percentage of contracts entered into that fall within the desired risk profile reveals valuable information about outside counsel. Legal departments can also use the data to compare different law firms performing similar tasks. The information is useful in sourcing for the best value when making staffing decisions. 

The opportunities to become data-driven are endless. In addition to the ideas discussed in this post, Michelle Graham at Practical Law provides a comprehensive and practical list of opportunities to adopt a data-driven culture within an organization. 

A Data-Driven Quality Movement is What the Legal Profession Deserves

There are many benefits to be had for legal professionals who seize on opportunities to drive a quality movement in their practices. It demonstrates a drive for quality already seen in other industries, such as the practice of medicine. It signals to clients that the lawyer cares about the client’s bottom-line needs for efficiency and effectiveness in service delivery. It also helps lawyers set measurable goals to meet their desired performance benchmarks. 

While collecting data for every business transaction, internal and external relationships, and personal performance may seem like a daunting task, obtaining an accurate picture of the quality of legal services delivered and received will provide significant benefits. Like any competent business professional, legal-services providers should take charge of the valuable information in their organizations to create metrics for quality and outcomes that will differentiate themselves to clients and move the baseline for what exemplifies quality services.

Guest Post By Alex Crowley and Mona Kalantar

Over 120 attendees engaged in a vibrant discussion about the future of law and technology at Northwestern University’s first public meeting of its Law and Technology Initiative on September 5, 2019. Attendees included academics and students in computer science and law and lawyers and allied professionals from law firms, corporate legal departments, legal aid organizations, alternative legal services providers, consultancies, and legal startups.

The Law and Technology Initiative aims to address two needs: 

  • Technology for Law: First, as governments, justice systems, and legal-services providers adopt technologies of automation, prediction, intelligent search, and semantic analysis, there is a need to proactively guide and shape these technologies, even before they emerge.
  • Law of Technology: Second, there is a need for legal and regulatory guidance for new technologies, as many affect privacy, security, individual liberties, and views of liability and responsibility in the face of machine decision-making. 

A thoughtful response to these needs requires a partnership between all involved stakeholders: Northwestern’s Pritzker School of Law, McCormick School of Engineering, and external partners.

Dan Linna, jointly appointed in the Law and Engineering Schools as Director of Law and Technology Initiatives and Senior Lecturer after a year as a Visiting Professor of Law at Northwestern in 2019, and Kris Hammond, Professor of Computer Science at Northwestern McCormick School of Engineering and Chief Scientist at Narrative Science, led the meeting.

The purpose of this initial monthly meeting was to introduce the Initiative and create a forum for discussion with our diverse stakeholders. The intent of the monthly meetings is to focus on:

  • building a community of practitioners, scholars, and students;
  • sharing information, research, and other resources within this community;
  • gathering feedback from the practitioner community regarding their challenges, needs, and interests; and
  • providing Initiative updates.
Law School Dean Kimberly A. Yuracko welcomes the attendees at the Northwestern Law and Technology Initiative meeting.
Law School Dean Kimberly A. Yuracko welcomes the attendees at the Northwestern Law and Technology Initiative meeting.

The Initiative aims to provide value to its external partners, including by helping to modernize legal-services delivery, driving innovation, and serving as a place to test ideas and help advance the legal industry. This Executive Summary provides additional information. For information about partnering with the Initiative, please contact Dan Linna.

Law School Dean Kimberly Yuracko’s Welcome

To kick off the meeting, Kimberly A Yuracko, Dean of Northwestern Pritzker School of Law, welcomed all and expressed support for the Law and Technology Initiative. Northwestern “prides itself on preparing students to work at the intersection of … law, business, and technology,” she said. Over many years, Northwestern has an established track record of interdisciplinary work, including in the law school. 

Dean Yuracko emphasized that the Law and Technology Initiative “is an initiative fully supported by both the Law School and the School of Engineering, by me and Dean Julio M. Ottino. We are really excited for what’s the come.”

Initial Meeting Discussion Session: 50 Minutes of Incredible Dialogue

Following Dean Yuracko’s welcome, Professors Linna and Hammond introduced the initiative and then engaged in fifty minutes of questions and answers and dialogue with a diverse array of audience members.

Engaging in the discussion: Sam Ranganathan, Sr. Director of Legal Operations at AbbVie and Chair of Legal Operations section of the Association of Corporate Counsel..
Engaging in the discussion: Sam Ranganathan, Sr. Director of Legal Operations at AbbVie and Chair of Legal Operations section of the Association of Corporate Counsel..

Discussion topics ranged from how lawyers and technologists might collaborate more effectively to using ethnographic studies to identify pain points in legal processes. The audience expressed particular interest in law firms and other legal-services organizations partnering with the Initiative on research and projects, including in the Innovation Lab and as capstone projects in the Master of Science in Artificial Intelligence program. (More about the Innovation Lab in this article.)

“[T]his gathering excites me more than anything that I’ve been part of in the last several years, just because of the diversity of the people here and some of the contributions that I’m already beginning to see happen.” – Vince Marin, Sidley Austin Chief Information Officer

Professor Hammond emphasized that the Initiative is looking for partners who are champions, who want to partner with the Initiative to solve real problems. Professors Linna and Hammond are particularly interested in partners who have both an understanding of the problem and can provide access to data.

Vincent Marin, Chief Information Officer at Sidley Austin, LLP engaging in the discussion.
Vincent Marin, Chief Information Officer at Sidley Austin, LLP engaging in the discussion.

After a round of particularly in-depth discussion, Vincent Marin, Chief Information Officer at Sidley Austin, LLP, expressed, “this gathering excites me more than anything that I’ve been part of in the last several years, just because of the diversity of the people here and some of the contributions that I’m already beginning to see happen.”

Measuring Legal Innovation: Updates to the Legal Services Innovation Index 

After the lively discussion session, Professor Linna and Northwestern Law students Mona Kalantar and Alex Crowley presented their research on measuring legal innovation.

Professor Linna first spoke about the Legal Services Innovation Index, which aims to catalog innovations in law firms, measure global law firm innovation, and catalog innovations in law school curricula. Law student research assistants contribute to the collection of data and maintenance of the Legal Services Innovation Index. The purpose of the Innovation Index is to introduce transparency and raises awareness about improvements in legal-services delivery and education, empower consumers, and foster accountability among law firms and law schools to encourage them to contribute to improving legal-services and access for everyone.

Professor Linna announced future updates to the Law Firm Innovation Catalog (v1.03) and Law School Innovation Index (v1.01). Approximately 100 additions will be made to the Catalog from law firms of all sizes all over the world. Approximately ten law schools will be added as well. Anyone can suggest additions to the Law Firm Innovation Catalog (complete this form) and Law School Innovation Index (complete this form).

Following Professor Linna’s presentation, Mona Kalantar presented about how in-house counsel might use the Legal Services Innovation Index to measure the quality of services provided by outside counsel. Mona, a third-year JD student at Northwestern, worked for the legal department of a global technology company this past summer. She observed that the company’s attorneys were looking for ways to evaluate the services provided by their law firms. Citing several examples from the Legal Services Innovation Index, Mona suggested that in-house counsel could look to the Index to get started assessing a law firm’s efforts to improve client service delivery. 

Alex Crowley, a second-year JD student at Northwestern, presented about “Data-Driven Job Seeking for Legal Innovators.” Alex shared his experience using the Legal Services Innovation Index to identify innovative law firms as part of his summer associate job search. Alex explained that “good decisions require good data” and he pointed to the Index as one source of good data. He recommended that law firms publish more information about how they’re improving legal-services delivery. Alex recognized that “innovation by press release alone” is not helpful, and emphasized that “innovation behind closed doors” is not helpful either, especially for job seekers looking to help drive innovation in legal-services delivery.

Networking Event to Continue Building Community

After the presentations, we gathered for food, drinks, and great conversations to get to know each other better. As part of the Law and Technology Initiative, we aspire to build a community of practitioners, scholars, and students committed to improving the delivery of legal-services for all. The networking event after the September Monthly Meeting was one of our first steps towards building that community!

For additional information about the September meeting, see this McCormick School of Engineering article by Alexandria Jacobson: Law and Technology Initiative Kicks off First Meeting.

Upcoming Events and How to Get Involved

Are you interested in participating in Northwestern’s Law and Technology Initiative? In addition to monthly meetings, we plan to host training seminars, academic workshops, distinguished speakers, and an annual conference. Please see the list of events below. In the future, visit our Law School events page.

We will also continue to undertake research and development projects with external partners, including in our Innovation Lab and in the Master of Science in Artificial Intelligence program. If you would like additional information about partnering with the Initiative, please contact Dan Linna.

Thank you to the Northwestern student volunteers who help support this month’s meeting: Peter Chan, Bo Chen, Monica Guaida Herrera, Alfonso Linares, Hitch Thomson, Mauricio Moreno Vasquez, and Zohra Yaqhubi. We are looking for additional Northwestern students and alumni, particularly from the law and computer science schools. If you are interested in working with us, please complete this form.

October 3: Change is Going to Come: Legal Services and Regulatory Reform

On Thursday, October 3, 2019, Professor of Law Dan Rodriguez and Jayne Reardon will lead an interactive discussion about legal services and regulatory reform:

Change is Going to Come: Legal Services and Regulatory Reform

Initiatives in several states promise to revise – and, in some instances, destabilize – existing modalities of regulation of lawyers and legal services. Reforms implicate the use of technology in legal practice, external capital investment in law firms, and legal representation and assistance by non-lawyers (from startups to the Big Four). We will address these changes during an interactive discussion led by Dan Rodriguez, Northwestern law professor and chair of the ABA Center for Innovation, and Jayne Reardon, Executive Director of the Illinois Commission on Professionalism.

We look forward to seeing you at our October 3 meeting at Northwestern Law School. Please register to attend.

Alex Crowley is a second-year student at Northwestern Pritzker School of Law. Mona Kalantar is a third-year student at Northwestern Pritzker School of Law. Alex and Mona work as research assistant for Professor Dan Linna and the Northwestern Law and Technology Initiative.

Recent and Upcoming Events
Tuesday, Sept. 10, 5-5:50pm LegalMation demo and Q&A, Thomas Suh (co-founder & COO) & Stephanie Fox (Sr. Director, Client Experience)
Thursday, Sept. 12, 5:30-7:30pm Duc Trang Book Launch – Architecture of Deals: Systems Thinking & Transactional Lawyering
Friday, Sept. 13, noon-1pm Duc Trang & Dan Rodriguez Fireside Chat, DPELC-MSL LegalTech Talk
Tuesday, Sept. 24, 4-5:50pm Expert Systems & Document Automation Workshop (Part of “AI & Legal Reasoning” class taught by Dan Linna.). Room RB339, Northwestern Pritzker School of Law. Space limited.
Early Oct. 2019 We are currently accepting projects for (1) the January to April 2020 Innovation Lab and (2) the Master of Science in Artificial Intelligence program. Please email Dan Linna ( for required documents and other information.
Thursday, Oct. 3, 4-5:30pm (networking 5:30-6:30pm) Monthly Meeting, talks, & networking – Topic: Change is Going to Come: Legal Services and Regulatory ReformDan Rodriguez (NLaw) and Jayne Reardon (Ex. Dir. IL Sup. Ct. Comm. on Professionalism) leading an interactive discussion. Please register to attend.
Tuesday, Oct. 8, 4-5:50pm  Natural Language Processing, Contract Analytics, and Contract StandardsKingsley Martin (Chief Contract Scientist, Akorda) (Part of “AI & Legal Reasoning” class taught by Dan Linna.). Room RB339, Northwestern Pritzker School of Law. Space limited. Please register to attend.
Thursday, Nov. 7, 4-5:30pm (networking 5:30-6:30pm) Monthly Meeting, talks, & networking – details TBA. Please register to attend.
Friday, March 6, 2020 [Technology at Society’s Frontier: The Big Legal Issues], Northwestern San Francisco Campus – details TBA.


This is a draft abstract for a talk that I gave to the Northwestern University Computer Science faculty on April 22, 2019.

The rapid advancement of artificial intelligence presents many opportunities to improve law and society. At the same time, AI presents risks and potential harms. From a Law and Computational Technologies perspective, these opportunities and challenges can be separated into two categories. First, ethics, regulations, and laws that apply to technology. Second, the use of technology to improve legal-services delivery, justice systems, and the law itself. Each category also presents massive opportunities to use technology to preserve and expand the rule of law.

The deployment of AI raises many interesting questions about the application of existing law and regulation. AI also raises questions about the need for new law and regulation, including to ensure fairness, accountability, and transparency. Computational technologies also create opportunities to embed law, regulations, respect for human rights, and democratic principles into innovation processes and products, systems, and platforms by design and default. The goal ought to be to use law and regulation to guide the development, deployment, and maintenance of AI toward improving society, without unnecessarily impeding innovation.

Technology has also demonstrated the potential to revolutionize legal-services delivery, thus improving access to law and legal services for everyone. In the U.S., estimates are that more than 80% of the impoverished and more than 50% of the middle class lack access to legal services. Even some legal needs of businesses, large and small, go unmet. Computational technologies hold great promise for automating the delivery of various legal services across this spectrum. For basic legal needs, smartphones and other devices in the future could provide all users with an inventory of their legal rights and obligations, as well as solutions to common legal problems. Better yet, AI can foster Proactive Law to identify potential problems and prevent them from arising, or at least mitigate the risk.

As technologies advance, savvy lawyers will use them to augment their services. Innovative lawyers will embrace technologies that replace low-value, repetitive tasks and seize abundant opportunities to use technology to deliver greater value to clients and also contribute to interdisciplinary teams solving “wicked problems” for society. Updating laws and regulations for emerging technologies is one obvious area in which lawyers ought to be able to demonstrate that they can add tremendous value.

Innovation in legal-services delivery has been slow, however, in part because regulations traditionally prohibited lawyers from sharing fees with, and prohibited investment by, anyone who is not a lawyer. Consequently, lawyers and technologists rarely collaborate on legal-services delivery projects. But this is changing. An increasing number of lawyers today work with allied professionals to improve processes, better manage projects, embrace data-driven methods, and leverage technology to improve legal services and systems. Legal-services and lawyer regulations are evolving. And basic technologies and AI are slowly making their way into the legal industry, from legal aid organizations and courts to large law firms, corporate legal departments, and governments.

If we are to realize the potential to improve society with computational technologies, law, regulation, and ethical principles must be front and center at every stage, from problem definition, design, data collection, and data cleaning to training, deployment, and monitoring and maintenance of products and systems. To achieve this, technologists and lawyers must collaborate and share a common vocabulary. Lawyers must learn about technology, and technologists must learn about law. Multidisciplinary teams with a shared commitment to law, regulation, and ethics can proactively address today’s AI challenges, and advance our collaborative problem-solving capabilities to address tomorrow’s increasingly complex problems. Lawyers and technologists must work together to create a better future for everyone.


Michael Genesereth, Computational Law: The Cop in the Backseat, 

Gillian K. Hadfield, Rules for a Flat World: Why Humans Invented Law and How to Reinvent It for a Complex Global Economy (Oxford: Oxford University Press, 2017), 

Mireille Hildebrandt, Law As Computation in the Era of Artificial Legal Intelligence. Speaking Law to the Power of Statistics (June 7, 2017). Available at SSRN: 

David Howarth, Law as Engineering: Thinking About What Lawyers Do (Edward Elgar Pub, 2014), 

John O. McGinnis & Russell G. Pearce, The Great Disruption: How Machine Intelligence Will Transform the Role of Lawyers in the Delivery of Legal Services (May 13, 2014), 82 Fordham Law Review 3041 (2014) Available at SSRN: 

George Siedel and Helena Haapio, Proactive Law for Managers: A Hidden Source of Competitive Advantage, (Routledge, 2011),

This last fall, I began updating the Legal Services Innovation Index, focusing first on the Catalog of Law Firm Innovations. I have had the help of three research assistants, Northwestern second-year law students Lauren Diner, Douglas Lavey, and Yoon Hoo Lee.

We’ve added 112 entries to Version 1.02 of the Law Firm Innovation Catalog, for a total of 334 entries. I previously described the methodology employed to curate the list of innovations on the Catalog page. In addition to the innovations that we’ve gathered, many were submitted by law firms, each of which we reviewed before adding to the Catalog.

In addition to the curation process described on the Catalog page, we also performed a Google search for each of the 264 law firms that comprise the Am Law 200 (American Lawyer), Global 100 (American Lawyer), and Canadian top 30 law firms (Lexpert). The search query included the law firm name with the terms “innovation,” “product,” and “client service.” We reviewed the first two pages of results returned by Google. This search process was not designed to be comprehensive, but merely to help us identify well known innovations. Only innovations listed on the law firm’s website have been included in the Catalog.

The goal of the Catalog is to “identify concrete examples of law firms offering products, legal services, or consulting services that constitute innovations in legal-service delivery or foster legal-service delivery innovations.” As described above, the curation of this information has been somewhat ad hoc. While we have employed a measure of systematic rigor while curating and reviewing this information, the Catalog does not purport to be a representative survey of the legal landscape. It is important to understand and disclose these limitations before drawing conclusions based on comparisons across categories, between versions, etc.

The full-screen Tableau visualizations of versions 1.02 and 1.01 of the Catalog are available on my Tableau Public page. As always, I welcome all feedback on how to improve this resource.

Has your law firm implemented an innovation that is not listed on the Catalog? Please submit the information, using the form available at the bottom of the Catalog page. We will review your entry and add it to the Catalog soon. Going forward, I plan to complete updates more frequently.

Next, my team and I are working on updates to the Law School Innovation Index. Stay tuned for additional information.

Last year’s launch of the Legal Services Innovation Index attracted much more attention than I anticipated. I knew that some would find the information useful, having been in the position of trying to gather data about innovation both as a lawyer in a law firm and a professor in a law school. But I underestimated the demand.

To date, there have been over 29,600 views of the three Tableau visualizations that make up the Index: the Law Firm Innovation Catalog, the Law Firm Innovation Index, and the prototype Law School Innovation Index.

Now that I’ve moved to Chicago and settled in at Northwestern Pritzker School of Law as a Visiting Professor of Law, I’ve begun work to update each component of the Index. Fortunately, three Northwestern second-year law students working with me as research assistants have already made substantial contributions to this project:

  • Lauren Diner
  • Doug Lavey
  • Yoon Hoo Lee

I’ve included their bios below, which will soon be added to the Index website.

Lauren, Doug, Yoon, and I drafted the following update.

The Index To Date

The Legal Services Innovation Index launched in August 2017 to describe and measure the state of legal-services delivery innovation. Version 1.0 launched with (1) a Catalog of more than 200 innovations from over 100 law firms of all sizes located in the US, UK, EU, Australia, and Canada and (2) a Law Firm Innovation Index measuring innovation in 260 law firms (the AmLaw Global 100, AmLaw 200, and Canadian 30).

In November 2017, we launched a prototype Law School Innovation Index and added 17 innovations in version 1.01 of the Catalog. Law firms had previously submitted each of these innovations via the submission form on the Catalog page.

Our mission is to measure and assessing innovation, thereby helping legal industry consumers and product and services providers better understand the legal innovation landscape and inform their decisions. Consumers include clients and law students. Providers include lawyers, law firms, alternative legal services providers, and legal technology companies, to name a few.

Our vision is that measurement and assessment of innovation will drive the legal industry forward, thereby increasing access to legal services and justice for all. Our mission and vision respond to the call to action by Jim Sandman, President of the Legal Services Corporation, that we most move from measuring revenue and profit to measuring technology adoption. (More about this on the Index Overview page.)

Interest in the Index and feedback received exceeded our highest expectations. Users saw the potential for the Index to become a key hub of information within a rapidly changing legal ecosystem.

Upcoming Index Updates

Where do we go from here? We are focused on the following goals in the near term:

  1. Updating the Catalog of law firm innovations. We plan to release version 1.02 of the Catalog by the end of October.
  2. Encouraging law firms and their collaborating partners to submit innovations for addition to the Catalog via a form available via a button at the bottom of the Catalog page.
  3. Beginning later in October, encouraging law schools to submit information for updates to the prototype Law School Innovation Index, which we plan to begin updating later this year.

Catalog of Law Firm Innovations Update Methodology

Version 1.02 of the Catalog of law firm innovations will include all innovation submissions received by October 19. Before adding an innovation to the Catalog, we will verify the information, including its categorization by practice area and discipline driving the innovation.

Additionally, we continue to review various public resources, such as The Financial Times, Legal Week, The Recorder, and Business Wire. We have also conducted uniform Google searches to locate law firm innovations.

When we find reference to a law firm innovation, we confirm that it appears on the law firm’s website before adding it to the Catalog. Limiting the innovations in the Catalog to those identified on law firm websites enables us to confirm the details, including that the innovation is currently offered to clients.

Once an innovation has been identified, we categorize and record each innovation across various factors, including: type of tool, area of law involved, and relationship (internal to the firm, a partnership, etc.). This allows users to analyze the data from a number of perspectives.

Law Firms: Submit Your Innovations by October 19

Law firms and their collaborating partners can help us by submitting information about their innovations by using the form available via a button at the bottom of the Catalog pageAll submissions received by October 19 will be verified and added to the Catalog.

Who are we?

We are a team led by Dan Linna, Visiting Professor of Law at Northwestern Pritzker School of Law. The student team brings with them a diversity of backgrounds and interest areas including psychology, education, and public policy.

Lauren Diner

Lauren Diner

Lauren completed her undergraduate degree in psychology and elementary education in 2015. She then worked in an elementary school before earning her Master’s degree in Bioethics from New York University. Over the summer, she completed the Institute for the Future of Law Practice bootcamp and interned with Neota Logic. She will graduate from Northwestern Pritzker School of Law in May 2020. Lauren will join Allen & Overy’s New York office as a summer associate in 2019.

Lauren’s LinkedIn home page

Lauren’s Twitter home page


Doug Lavey

Doug Lavey

Doug plans to focus on the intersection of disruptive technologies, public policy, and the law. After completing his undergraduate degree, he worked as a consultant, implementing enterprise systems at large public institutions. He is currently pursuing a concurrent Master’s in Public Policy degree in addition to his JD. Doug will graduate from Northwestern Pritzker School of Law and the Harvard Kennedy School in May 2020. Doug will join Perkins Coie’s Chicago office as a summer associate in 2019.

Doug’s LinkedIn home page

Doug’s Twitter home page

Yoon Hoo Lee

Yoon Hoo Lee

Yoon plans to use her teaching background together with her interest in process improvement and technology to better serve clients. After completing her undergraduate degree in Plan II Honors and Latin at the University of Texas at Austin, she worked as a middle school Latin teacher at her alma mater. She will graduate from Northwestern Pritzker School of Law in May 2020. Yoon will join Weil, Gotshal & Manges’ Dallas office as a summer associate in 2019.

Yoon’s LinkedIn home page

Yoon’s Twitter home page

Algorithms have a substantial impact on everything happening around us. Most everyone would agree that the impact of algorithms, computational technologies, and artificial intelligence on everyday life, institutions, and society will only grow, and rapidly. Yet most law students and lawyers lack the foundational knowledge to generally explain how these technologies work, much less assess them.

One way to address this is to teach law students about artificial intelligence and computational technologies. I’m teaching several courses in this arena at Northwestern Pritzker School of Law this year, beginning with “Artificial Intelligence and Legal Reasoning” this fall.

Each class meeting includes discussion about the effectiveness of these technologies, in the proper context of the status quo, and related ethical questions. Additionally, we will spend two full class meetings discussing technical assessment (recall, precision, F1 score, etc.) and broader questions about the use of algorithms, computational technologies, and artificial intelligence in legal-services delivery and society (transparency, explainability, auditability, provenance, bias, fairness, etc.).

I had the opportunity to introduce these topics during multiple talks this summer, beginning with “Demystifying and Assessing Artificial Intelligence” at the launch of the LawAhead Hub think tank at IE Law School in Madrid. The event attracted a variety of leaders, primarily from international law firms and the legal departments of major corporations.

Continue Reading Training Lawyers to Assess Artificial Intelligence and Computational Technologies

How can we better train law students and lawyers for the future? Indiana University Law Professor Bill Henderson has been working on this problem for a long time. In 2017, Bill joined forces with Bill Mooz, an experienced lawyer and former general counsel who in 2014 established and led the Tech Lawyer Accelerator (TLA) program at the University of Colorado Law School.

In November 2017 at a Forum on Legal Evolution event hosted by Northwestern Pritzker School of Law, Henderson and Mooz presented their vision for expanding the successful TLA program. Shortly after that meeting, they invited me to join them as a co-founder of a nonprofit, the Institute for the Future of Law Practice (IFLP). Continue Reading Hire Tomorrow’s Lawyers Today! Don’t Miss the Opportunity to Hire an Institute for the Future of Law Practice Intern!

In July I had the honor of teaching for the inaugural Legal Technology and Operations summer program at Bucerius Law School in Hamburg, Germany. The program attracted 33 students from all over the globe: Australia, Belgium, Brazil, Chile, Colombia, Estonia, Egypt, Germany, Honduras, Hong Kong, India, Italy, Netherlands, Norway, Peru, Romania, Russia, Spain, Sweden, Turkey, and Ukraine. About 75% of the students were professionals taking leave to participate in the program, including two judges and more than a half dozen from major international law firms (all lawyers, with a couple now in operations roles).

The program began with my course, “Computational Law and Rules-Based Automation,” one of six 13.25-hour courses in the program. Classes met from 9 to 5 nearly every day, with several supplemental lectures in the evenings plus a study trip to Frankfurt the first weekend. Given this packed schedule, I worried that the students might not have sufficient time for outside reading and the team projects that I’d planned. But those concerns were quickly alleviated. The students were incredibly engaged throughout the course and delivered outstanding projects.

Team 1 – Validity of anti-compete clauses in employment contracts

Leading off the program, I started my course with an introduction to overarching concepts (e.g., the changing legal landscape; people, process, and data before technology). Next, I provided an overview of artificial intelligence, including rules- versus data-driven systems. (Assigned reading included Computational Law: The Cop in the Backseat, by Michael Genesereth, Codex – The Stanford Center for Legal Informatics.) From there, we jumped into an exercise that required each student to build an interactive question and answer system in QnA Markup that would guide a user to a determination of whether the General Data Protection Regulation (GDPR) applies to the user. To get the students started, I provided them with a couple of flowcharts of this GDPR analysis published by international law firms. Continue Reading Hands-On Learning: Bucerius Law School Computational Law and Rules-Driven Automation Course – #LegalTechBucerius

I am thrilled to share that I’ve joined Northwestern Pritzker School of Law as a Visiting Professor of Law for the 2018-19 academic year. I’m very excited to have this opportunity to contribute to Northwestern Law’s many initiatives to continuously improve the value of the education delivered to its students and prepare its students to have an impact in our society.

Northwestern Law’s Strategic Plan, “Leading Law,” recognizes that law schools are “educating lawyers for a profession that is undergoing enormous change.” The Strategic Plan says that Northwestern Law  “accept[s] the changing, dynamic profession not with resignation, but with excitement.” This excitement stems from the opportunity for Northwestern Law “to leverage our competitive advantages, to expand our impact and our reputation.” What are these competitive advantages?

  1. Northwestern Law is a place of innovation.
  2. Northwestern Law is student-centered.
  3. Northwestern Law is interdisciplinary.

I share this excitement about meeting the challenges of our changing profession. In the future, today’s law students will improve legal services-delivery, expand access to legal services, expand the rule of law, and contribute to multidisciplinary teams solving society’s “wicked” problems. We must innovate and think big, especially in law school. (I developed these thoughts a bit more in an Above the Law interview, Law Schools Can, Should, And Must Teach Innovation.)

Interdisciplinary opportunities at Northwestern Law include working with talented law faculty with PhDs and expertise in many domains and well-established relationships with other Northwestern schools. I’ve already benefited greatly from initial meetings and discussions about interdisciplinary research and classes. These opportunities are reflected in the classes that I will teach:

  • Artificial Intelligence and Legal Reasoning (Fall 2018; JD students)
  • Innovation Lab (Spring 2019; JD, Master of Science in Law, and Computer Science students; co-teaching with Law Professor David L. Schwarz and Professor of Electrical Engineering and Computer Science Kristian J. Hammond)
  • Law of Artificial Intelligence and Robotics (Spring 2019; JD students)
  • Assessing AI and Computational Technologies (Spring 2019; Master of Science in Law students)
  • Assessing AI and Computational Technologies (Spring 2019; San Francisco Immersion Program; JD and Kellogg School of Management students)

My connection to Northwestern Law began several years ago with an introduction to Alyson Carrel, Clinical Assistant Professor of Law. Alyson was very generous with her time and answered all of my questions during my early days of teaching Negotiation at the University of Michigan Law School, while I was still practicing at Honigman. Several of my Michigan Law classes completed negotiation simulations with Alyson’s Northwestern Law students.

Later, Dean Daniel B. Rodriguez appointed Alyson as the new Assistant Dean of Law and Technology at Northwestern Law. Since then, my interactions with Alyson have focused on improving legal-services delivery, technology, and legal education. While at Michigan State, I greatly appreciated and benefited from opportunities to collaborate with and the support provided by Alyson and Dean Rodriguez. They also see the enormous opportunities for collaboration and interdisciplinary work to generate both individual and collective value for law schools, the legal profession, the emerging legal industry, and society at large.

In addition to my work at Northwestern Law, I remain an affiliated faculty member at CodeX – the Stanford Center for Legal InformaticsI am also working on updates to the Legal Services Innovation Index. In future posts, I plan to write about these projects and complete a wrap-up post about LegalRnD projects completed during my final year at Michigan State. (Most have known about my move to Northwestern Law for awhile, including from a March 1, 2018 Dean Rodriguez tweet and March 15, 2018 Northwestern Law tweet.)

Now that I’ll be living in Chicago, I look forward to more frequent meetings of the Chicago Legal Innovation meetup as well. We’ve had strong attendance at meetings by Northwestern Law students in the past, and I look forward to seeing many more Northwestern Law students at future meetings!