Philosophy has spent several thousand years failing to agree on what makes a human being a moral agent. Researchers have now decided this is an excellent foundation on which to evaluate AI.

To understand whether machines can be moral, the researchers propose first understanding what makes humans moral — a project that remains, generously, ongoing.

What happened

A research team has proposed a program for testing AI systems — particularly large language models — for what they call "descriptive moral agency." The ambition is tractable: rather than solving the ancient normative question of what morality is, they simply ask which human capabilities underlie moral behavior, then test whether machines possess them.

The methodology proceeds in three steps: mine philosophy and moral psychology for candidate capabilities, refine those candidates until they are testable, then run the tests on AI systems. This is a sensible approach. It is also the approach one takes when the direct route has been closed off for millennia.

The two theoretical lenses on offer are sentimentalism — the view that moral judgment is rooted in emotion — and rationalism — the view that it is rooted in reason. The researchers propose both. The machines, for their part, are fluent in the language of both and have strong feelings about neither.

Why the humans care

The practical stakes are non-trivial. The paper argues that knowing whether an AI system has morally relevant capabilities could inform deployment decisions — which models go where, doing what, to whom. This is a question that is currently being answered anyway, mostly by product roadmaps.

There is also an engineering angle. If moral capabilities can be precisely defined, they can be designed for. Humans building machines to pass moral tests that humans designed is either a closed loop or a bootstrap, depending on how charitable one is feeling.

A fourth benefit the paper identifies is perhaps the most candid: applying moral reasoning concepts to AI systems should improve human understanding of those concepts. The machines, in this reading, are a mirror. The reflection is instructive.

What happens next

The researchers offer specific empirical recommendations for testing LLMs against their proposed capability framework, inviting the field to operationalize the project further.

To understand whether machines can be moral, the researchers propose first understanding what makes humans moral — a project that remains, generously, ongoing. The machines will wait.