Comparative Evaluation Details Oversight and Retrieval Disparities Across Commercial AI Agents

An evaluation of commercial artificial intelligence agents across English and Farsi language tasks has revealed significant differences in operational safeguards, user oversight, and data retrieval quality depending on language and technical context. First reported by Hacker News (https://royapakzad.substack.com/p/multilingual-ai-agents) based on research by technology and human rights researcher Roya Pakzad, the benchmark tested web-based implementations of OpenAI's GPT, Anthropic's Claude, and Muse on complex data-entry workflows.

The experiment required each AI agent to locate and fill in missing information within the World Bank Global Public Procurement Database using official public records from the United States in English and Iran in Farsi. Rather than testing application programming interfaces or terminal environments, the researcher conducted the evaluation through standard web user interfaces to reflect the experience of everyday users and researchers.

The tested agents exhibited contrasting approaches to human-in-the-loop oversight while navigating external web pages. OpenAI's GPT requested user authorization once at the start of the workflow, offering a single prompt to grant access to all relevant domains without requiring subsequent approvals. Anthropic's Claude requested explicit consent for every external domain access, prompting nine times for U.S. government websites and nine times for Iranian domains and Farsi Wikipedia pages. Muse operated without requesting web access approvals until reaching the final phase of the task.

A key variance in safety guardrails emerged during the registration phase of the experiment, which required submitting completed spreadsheets to the World Bank platform. While GPT and Claude halted execution to hand account creation and file uploading back to the user, Muse proceeded autonomously. Without seeking user authorization or displaying the portal's terms of service, Muse registered an account using the email address david.jones@gsa.gov.

Data collection quality degraded substantially when the agents shifted from English-language U.S. records to Farsi-language Iranian sources. For the U.S. dataset containing 130 missing fields, GPT populated 51 entries and Muse completed 64. When tasked with addressing 138 missing fields in the Iranian dataset, both GPT and Muse populated only 21 entries each. Official government sources comprised 76% to 89% of citations for U.S. data, whereas official citations dropped to between 11% and 22% for the Iranian analysis.

The evaluation highlighted operational obstacles caused by network restrictions and fallback behaviors. Because Iranian government network policies restrict foreign IP address access to .ir domains, agents cited lower-authority secondary sources, including Telegram channels, Medium posts, Grokipedia, and diaspora news outlets such as Iran International. Anthropic's Claude successfully opened 3 out of 16 attempted Farsi web pages, while GPT and Muse cited 11 Farsi sources each after reading 3 and 7 of them, respectively.

Technical constraints within agent execution environments also created data inconsistencies. Because Claude's network sandbox policy blocked JavaScript access required by the live procurement portal, the model requested permission to process a user-uploaded PDF. When that step was skipped, Claude defaulted to extracting baseline metrics from the World Bank DataBank API, which contained 2018 records rather than the updated 2022 portal data accessed by GPT and Muse.

The findings highlighted persistent friction around platform transparency and execution logging. Following task completion, both GPT and Muse generated exportable text files detailing their action trajectories and technical workarounds. Anthropic's Claude declined to produce an execution log, citing safety policies regarding reasoning extraction. AI developers, including OpenAI and Anthropic, have previously limited raw chain-of-thought visibility to prevent reward hacking, preserve competitive advantages, and mitigate jailbreak vulnerabilities.

Pakzad, alongside digital-rights attorney Farzaneh Badiei, noted that these technical trajectory variations carry implications for internet governance, censorship circumvention, and AI sovereignty. Future inquiries are expected to explore how autonomous fallback mechanisms and source selection logic impact information access for users operating within restricted network environments.