A group of researchers from Hao AI Lab at the University of California San Diego has suggested that Super Mario Bros. might actually be a tougher challenge for AI than Pokemon. In a recent experiment, AI models were tasked with playing the game, and while Anthropic's Claude 3.7 performed the best, models like Google's Gemini 1.5 Pro and OpenAI's GPT-4o struggled. The game was not the original 1985 version but instead ran in an emulator integrated with GamingAgent, a framework that provided basic instructions and screenshots for the AI to control Mario.

The AI had to generate inputs, such as Python code, based on the given instructions to navigate Mario through the game's challenges. The researchers found that while the game required models to plan complex manoeuvres and strategies, reasoning models like OpenAI's o1 performed worse than non-reasoning models. This is because reasoning models typically take longer to decide on actions, and in a real-time game like Super Mario Bros., timing is critical.

While games have long been used to benchmark AI, some experts question the relevance of gaming skills as a measure of technological advancement. Andrej Karpathy, a research scientist at OpenAI, has expressed concerns over the current AI evaluation process, calling it an 'evaluation crisis.' Despite these concerns, watching AI take on Super Mario Bros. provides an interesting glimpse into how far AI has come, even if the benchmarks remain unclear.