From the abstract for Ashley Deeks, The Judicial Demand for Explainable Artificial Intelligence, 119 Colum.L.Rev. ____ (Forthcoming 2019):

A recurrent concern about machine learning algorithms is that they operate as “black boxes,” making it difficult to identify how and why the algorithms reach particular decisions, recommendations, or predictions. Yet judges will confront machine learning algorithms with increasing frequency, including in criminal, administrative, and tort cases. This Essay argues that judges should demand explanations for these algorithmic outcomes. One way to address the “black box” problem is to design systems that explain how the algorithms reach their conclusions or predictions. If and as judges demand these explanations, they will play a seminal role in shaping the nature and form of “explainable artificial intelligence” (or “xAI”). Using the tools of the common law, courts can develop what xAI should mean in different legal contexts.

There are advantages to having courts to play this role: Judicial reasoning that builds from the bottom up, using case-by-case consideration of the facts to produce nuanced decisions, is a pragmatic way to develop rules for xAI. Further, courts are likely to stimulate the production of different forms of xAI that are responsive to distinct legal settings and audiences. More generally, we should favor the greater involvement of public actors in shaping xAI, which to date has largely been left in private hands.

From the abstract for Jake Goldenfein, Algorithmic Transparency and Decision-Making Accountability: Thoughts for Buying Machine Learning Algorithms (Sept. 9, 2019):

There has been a great deal of research on how to achieve algorithmic accountability and transparency in automated decision-making systems – especially for those used in public governance. However, good accountability in the implementation and use of automated decision-making systems is far from simple. It involves multiple overlapping institutional, technical, and political considerations, and becomes all the more complex in the context of machine learning based, rather than rule based, decision systems. This chapter argues that relying on human oversight of automated systems, so called ‘human-in-the-loop’ approaches, is entirely deficient, and suggests addressing transparency and accountability during the procurement phase of machine learning systems – during their specification and parameterisation – is absolutely critical. In a machine learning based automated decision system, the accountability typically associated with a public official making a decision has already been displaced into the actions and decisions of those creating the system – the bureaucrats and engineers involved in building the relevant models, curating the datasets, and implementing a system institutionally. But what should those system designers be thinking about and asking for when specifying those systems?

There are a lot of accountability mechanisms available for system designers to consider, including new computational transparency mechanisms, ‘fairness’ and non-discrimination, and ‘explainability’ of decisions. If an official specifies for a system to be transparent, fair, or explainable, however, it is important that they understand the limitations of such a specification in the context of machine learning. Each of these approaches is fraught with risks, limitations, and the challenging political economy of technology platforms in government. Without understand the complexities and limitations of those accountability and transparency ideas, they risk disempowering public officials in the face of private industry technology vendors, who use trade secrets and market power in deeply problematic ways, as well as producing deficient accountability outcomes. This chapter therefore outlines the risks associated with corporate cooption of those transparency and accountability mechanisms, and suggests that significant resources must be invested in developing the necessary skills in the public sector for deciding whether a machine learning system is useful and desirable, and how it might be made as accountable and transparent as possible.

“Managing the uncertainty that is inherent in machine learning for predictive modeling can be achieved via the tools and techniques from probability, a field specifically designed to handle uncertainty,” writes Jason Brownlee in A Gentle Introduction to Uncertainty in Machine Learning. In his post, one will learn:

  • Uncertainty is the biggest source of difficulty for beginners in machine learning, especially developers.
  • Noise in data, incomplete coverage of the domain, and imperfect models provide the three main sources of uncertainty in machine learning.
  • Probability provides the foundation and tools for quantifying, handling, and harnessing uncertainty in applied machine learning.

From Chapter One in Evaluating Machine Learning Models by Alice Zheng:

One of the core tasks in building a machine learning model is to evaluate its performance. It’s fundamental, and it’s also really hard. My mentors in machine learning research taught me to ask these questions at the outset of any project: “How can I measure success for this project?” and “How would I know when I’ve succeeded?” These questions allow me to set my goals realistically, so that I know when to stop. Sometimes they prevent me from working on ill-formulated projects where good measurement is vague or infeasible. It’s important to think about evaluation up front.

From the abstract for John Nay, Natural Language Processing and Machine Learning for Law and Policy Texts (Aug. 23, 2019):

Almost all law is expressed in natural language; therefore, natural language processing (NLP) is a key component of understanding and predicting law at scale. NLP converts unstructured text into a formal representation that computers can understand and analyze. The intersection of NLP and law is poised for innovation because there are (i.) a growing number of repositories of digitized machine-readable legal text data, (ii.) advances in NLP methods driven by algorithmic and hardware improvements, and (iii.) the potential to improve the effectiveness of legal services due to inefficiencies in its current practice.

NLP is a large field and like many research areas related to computer science, it is rapidly evolving. Within NLP, this paper focuses primarily on statistical machine learning techniques because they demonstrate significant promise for advancing text informatics systems and will likely be relevant in the foreseeable future.

First, we provide a brief overview of the different types of legal texts and the different types of machine learning methods to process those texts. We introduce the core idea of representing words and documents as numbers. Then we describe NLP tools for leveraging legal text data to accomplish tasks. Along the way, we define important NLP terms in italics and offer examples to illustrate the utility of these tools. We describe methods for automatically summarizing content (sentiment analyses, text summaries, topic models, extracting attributes and relations, document relevance scoring), predicting outcomes, and answering questions.

The Law Society’s Technology and Law Public Policy Commission was created to explore the role of, and concerns about, the use of algorithms in the justice system. Among the recommendations, the UK needs to create a ‘national register of algorithms’ used in the criminal justice system that would include a record of the datasets that were used in training. Interesting. Read the report.

To get AI systems off the ground, training data must be voluminous and accurately labeled and annotated. With AI becoming a growing enterprise priority, data science teams are under tremendous pressure to deliver projects but frequently are challenged to produce training data at the required scale and quality. Nearly eight out of 10 organizations engaged in AI and machine learning said that projects have stalled, according to a Dimensional Research’s  Artificial Intelligence and Machine Learning Projects Obstructed by Data Issues. The majority (96%) of these organizations said they have run into problems with data quality, data labeling necessary to train AI, and building model confidence.

Aryan Pegwar asks and answers the post title’s question. “Today Modern technologies like artificial intelligence, machine learning, data science have become the buzzwords. Everybody talks about but no one fully understands. They seem very complex to a layman. People often get confused by words like AI, ML and data science. In this article, we explain these technologies in simple words so that you can easily understand the difference between them.” Details here.

Related:

TechRepublic reports that Microsoft has announced that Word Online will incorporate a feature known as Ideas this fall. Backed by artificial intelligence and machine learning courtesy of Microsoft Graph, Ideas will suggest ways to help you enhance your writing and create better documents. Ideas will also show you how to better organize and structure your documents by suggesting tables, styles, and other features already available in Word.

Terms of service of on-line platforms too often contain clauses that are potentially unfair to the consumer. The developers of “CLAUDETTE” present an experimental study where machine learning is employed to automatically detect such potentially unfair clauses. Results show that the proposed system could provide a valuable tool for lawyers and consumers alike. Details here.

From the abstract for Daniel L. Chen, Machine Learning and the Rule of Law, Computational Analysis of Law, Santa Fe Institute Press, ed. M. Livermore and D. Rockmore, Forthcoming:

“Predictive judicial analytics holds the promise of increasing the fairness of law. Much empirical work observes inconsistencies in judicial behavior. By predicting judicial decisions—with more or less accuracy depending on judicial attributes or case characteristics—machine learning offers an approach to detecting when judges most likely to allow extra legal biases to influence their decision making. In particular, low predictive accuracy may identify cases of judicial “indifference,” where case characteristics (interacting with judicial attributes) do no strongly dispose a judge in favor of one or another outcome. In such cases, biases may hold greater sway, implicating the fairness of the legal system.”