People often say today’s chatbots have passed the Turing test. This paper argues that the test most people mean is not Turing’s test—and that, under his actual design, the machine still failed. In October 1950, Alan Turing proposed that a machine’s success in an imitation game provides a criterion for thinking.

Today, generative AI models are widely claimed to pass Turing’s test, rendering it obsolete. But this paper argues that influential interpretations of Turing’s design are mistaken, with the consequence that his test has not been passed.

The paper takes Turing’s descriptions of his test and argues that influential interpretations of his design are incompatible with Turing’s own words. Those readings also conflict with the early perception and historical context of his ideas.

The key idea is easier to picture as a guessing game with two hidden contestants: the judge is not asking whether one speaker sounds human in isolation, but which participant is the human being and which is the machine. That reading also fits Turing’s claim that the point of the test was to determine whether the computer can imitate a brain.

This is where the familiar version goes wrong. Online Turing tests frequently use two players, with the interrogator deciding whether a single hidden interviewee is a computer or a human being. Yet Turing stated that the imitation game is played with three people.

Where one interviewee is a computer, the three-player setup also has theoretical advantages. To put that design into practice, the game had three players, and the interrogator could initiate conversation on any topic. The game was open-ended: the interrogator continued the interaction until confident of identifying the computer.

People likely to have expertise in AI or philosophy of AI were excluded. And here is the surprise: 36 out of 37 interrogators—97 percent—correctly identified the computer. Using the rule that the average interrogator should not do significantly better than chance, GPT-4-Turbo did not pass the test.

The comparison made the result even clearer. Only 16 out of 37 interrogators—43 percent—correctly identified the man. The average interrogator performed much better when identifying the computer than when identifying the man, so GPT-4-Turbo did not pass under that comparison either.

The result has limits. The study tested only one model and used only one prompt, and different models or prompts might produce a better result for the machine. New generative AI models are released frequently, and it is difficult to systematically compare how different models perform.

The results are consistent with Turing’s 1952 prediction that it would be at least a century before a machine passed the test. But the study used a specific model, prompt, and participants, so the design must be repeated with different models, prompts, subjects, and other relevant variations.

The paper’s central message is that passing a convincing chat test is not enough: Turing’s original game requires a three-person comparison, and the tested computer was identified far more often than the human.