Top AI Agents Fail More Than Half Of Real Engineering Tasks In New Test

PLE engineering benchmark engineer at construction site

Top AI agents failed more than half of realistic engineering work in a tough new test. The PLE engineering benchmark exposed the gap between demo skill and job site skill. Google built the test with Mercor, Google Cloud reported on November 3. 14 frontier models took on 85 real engineering tasks. Every model finished below 50 percent in the PLE engineering benchmark, while the best model reached only 48.3 percent. The result should cool claims that AI agents already handle professional engineering alone.

The test name says the ambition plainly. Physical Engineering’s Last Exam aims to grade agents like a licensing exam grades people. Tasks come from real engineering practice, not textbook puzzles. Each task demands a finished deliverable rather than a short answer. Agents work inside the Mercor Archipelago container environment, which mirrors the tools and files a human engineer would use. That setup stops models from gaming the test with memorized answers, because each task needs genuine work on real inputs.

PLE engineering benchmark engineer reading blueprints
An engineer reads blueprints at a structural site. (Photo: Alphaform)

The 85 tasks in the PLE engineering benchmark split across four disciplines, and the spread shows how broad physical engineering runs. Mechanical work leads with 30 tasks, electrical work follows with 27, civil work adds 18, and structural work adds 10, while no single specialty can carry a model.

A cooling loop design, a switchboard layout, a site plan, and a roof frame all sit in the same exam, while the mix also mirrors real project teams. No single specialty can carry a model. The mix also mirrors real project teams, where disciplines must fit together. A model that aces wiring but fails structures still fails the project.

PLE disciplineTasks
Mechanical engineering30
Electrical engineering27
Civil engineering18
Structural engineering10
Total tasks85

The safety figures cut deeper than the headline score. Only 41.3 percent of the top setup deliverables in the PLE engineering benchmark passed every hard safety and feasibility check. A deliverable can look complete yet fail a load rule, a voltage limit, or a clearance requirement. Real projects cannot ship such work.

The hard checks mirror the rules that keep buildings standing and systems safe. A miss counts as a fail rather than partial credit. That strict bar separates the PLE engineering benchmark from chat style tests, where fluent text often earns points despite wrong numbers.

Only 29 of the 85 tasks sit in public view. The rest stay hidden to stop training on the test itself. Public tasks let researchers study the format and check the grading, while hidden tasks protect the benchmark from memorization.

The split follows standard practice for serious exams, since a leaked test measures recall rather than skill. Teams that want to improve must build general ability, not answers to known tasks. That design choice gives the scores more weight than a fully public test would carry.

The PLE engineering benchmark methodology deserves a plain statement, since the headline number needs context. Google and Mercor collected the 85 tasks from real engineering practice across four disciplines. They ran 14 frontier models in the same container environment. Graders checked each deliverable against hard safety and feasibility rules, not against style or fluency.

PLE engineering benchmark four disciplines graphic
Four disciplines in the PLE test. (Image: Google Cloud)

The sources give no margin of error, and they do not name all 14 models or give per model scores beyond the top result. Readers should treat 48.3 percent as the best observed score under strict grading, not as a precise rank of every model.

The result fits a pattern from other hard agent tests, where agents shine on short tasks and fade on long professional work. A recent GDPx expert test put the best agent at just 12 percent on tough expert tasks, while Copilot desktop apps show agents still learning to operate real software.

Findings on AI agents in live attacks show the same agents already shape real incidents, so the stakes run beyond test scores. Engineering adds physical stakes to that pattern, since a wrong beam size or cable rating carries real danger.

Human engineers should read the PLE engineering benchmark result as relief mixed with warning. Relief, because no model yet passes half of realistic tasks under strict checks, so professional judgment stays central. Warning, because 48.3 percent in the PLE engineering benchmark already covers nearly half the work, and the curve moves fast. Firms that treat agents as junior helpers with human review will gain value now, while firms that hand agents unsupervised sign off risk real harm. The benchmark rewards the supervised model, since agents draft and humans verify.

Model builders face a clear research agenda from the test. Agents need better planning over long tasks, stronger checks against physical rules, and honest uncertainty when a design sits near a limit, because training on more engineering data may help only if the data teaches rules rather than patterns. Training on more engineering data may help, but only if the data teaches rules rather than patterns.

The container environment also lets builders replay failures, so each miss becomes a lesson. Progress will show in the safety pass rate at the start, because a model that passes hard checks earns trust faster than one that merely scores higher.

The PLE engineering benchmark will likely become a standard reference, as teams cite it alongside other agent tests. Its strict grading and hidden tasks give it credibility, while its public slice lets outsiders verify the claims. Future runs will show whether models climb past 50 percent, and whether safety pass rates climb with scores.

Until then, the honest summary stays simple, since the best AI agents today handle less than half of real engineering work. That fact should guide hiring, training, and supervision choices across the field, and it should temper any plan to remove humans from the loop.

Stay Connected with The MES Times

Get the latest tech, AI and gadget news and updates wherever you prefer.

Follow on Google Discover

Add The MES Times to Preferred Sources and see more of our stories in Google Search and Top Stories.

AI & Emerging Tech

Aizaz Khan is a technology journalist at The MES Times, leading coverage on artificial intelligence, humanoid robotics, and silicon hardware. Holding a degree in AI and Robotics where he graduated top of his class he approaches tech reporting with an engineer's eye rather than PR talking points. His work focuses on dissecting research papers, neural model architectures, and physical robotics systems to explain their real-world impact and limitations.

0 comments Oldest comments first