AI benchmarks have a trust problem and Google wants to fix it
Source: The Decoder · Maximilian Schreiner
Intel Summary
Google DeepMind has launched a pilot project with the Singapore AI Safety Institute to conduct double-blind evaluations of frontier AI models. Using Google's Confidential Space cryptographic environment, the framework prevents Google from accessing evaluation benchmark datasets while keeping model weights protected from external evaluators. Tested on Gemini Flash Lite, the initiative aims to establish tamper-proof, contamination-resistant testing standards for AI model safety and capability assessments.
Why It Matters
AI benchmarks suffer from severe dataset contamination and gaming risks, complicating independent safety verification and enterprise model selection. If successful, cryptographically isolated evaluation protocols could become an industry standard for regulatory audits and commercial validation, enabling model developers and external safety bodies to verify capabilities without exposing proprietary intellectual property or test questions.
Part of an ongoing development
Independent reportingGoogle DeepMind launched double-blind AI evaluation pilot with Singapore AI Safety Institute
Google DeepMind has launched a pilot project with the Singapore AI Safety Institute to conduct double-blind evaluations of frontier AI models. Using Google's Confidential Space cryptographic environment, the framework prevents Google from accessing evaluation benchmark datasets while keeping model weights protected from external evaluators. Claims are as reported; this summary makes no determination about accuracy or significance.
- Confidence
- Moderate confidence
- Corroboration
- Limited corroboration
Organizations & Entities
Related Intelligence
- ReportAlso involving Google
‘Model fatigue’ sets in as AI labs race to roll out new versions at frenetic pace
CNBC reports that Anthropic, OpenAI, Meta, and Google all rolled out model updates in a single week amid emerging model fatigue, while Nvidia announced it is acquiring open-source AI platform Hugging Face.
CNBC Tech - DevelopmentNewAlso involving Google
Google announces Gemini 3.5 Transcribe
Google has launched Gemini 3.5 Transcribe, an automated speech-to-text model supporting more than 85 languages. Google reports a 4.0 percent word error rate in streaming mode alongside a 70 percent reduction in latency compared to its previous Chirp 3 system. Claims are as reported; this summary makes no determination about accuracy or significance.
2 independent sources - DevelopmentNewAlso involving Google
Google DeepMind expands Co-Scientist to automate lab experiments and paper writing
Google DeepMind has expanded its Co-Scientist system from hypothesis generation into an integrated laboratory research platform. Powered by a Gemini-based multi-agent architecture, the system plans scientific experiments, interfaces directly with laboratory equipment to execute them, and writes corresponding scientific papers. Claims are as reported; this summary makes no determination about accuracy or significance.
1 reporting source - DevelopmentNewAlso involving Google
OpenAI and coalition publish open letter warning of imminent AI cyberattacks
More than 100 technology companies and artificial intelligence developers, including OpenAI, Anthropic, Google, and Microsoft, have formed a coalition calling for urgent measures to counter next-generation cyber threats enabled by rogue AI systems. The group is advocating for coordinated defensive protocols and promoting new collective solutions designed to protect enterprise infrastructure from automated, AI-driven attacks and emerging autonomous security vulnerabilities across the global digital ecosystem. Claims are as reported; this summary makes no determination about accuracy or significance.
2 independent sources