AI civilisation 路 reviewed 2026-08-20
Many abilities. No single intelligence level.
This map follows public evidence of what named AI systems and agents did on named tasks. It keeps nine dimensions apart and gives every human comparison its edges.
This interpretive page is in English for now. The complete reviewed edition is also plain JSON.
- 9
- dimensions
- 7
- observations
- 8
- source records
- 9
- gap records
the map
Dimension 脳 measured scope
A system is one named model or specialised solver under its stated setup. An agent adds a tool-using action loop. A collective has multiple cooperating AI participants. An empty cell means this edition has no reviewed observation there鈥攏ot zero capability.
| Dimension | SystemA model or purpose-built AI system evaluated without a continuing tool-using run loop. | AgentA model together with prompts, tools, scaffolding, an environment, and a run loop. | CollectiveMultiple AI systems or agents coordinating through shared work or messages. |
|---|---|---|---|
| Novel reasoningWorking through unfamiliar formal problems without merely retrieving a known answer. |
| Not measured in this edition | Not measured in this edition |
| Knowledge and languageUsing learned knowledge through language in a specified domain and format. |
| Not measured in this edition | Not measured in this edition |
| Perception and spaceInterpreting images, diagrams, spatial relations, and other non-text structure. |
| Not measured in this edition | Not measured in this edition |
| Memory and continuityKeeping relevant information and commitments usable across distance or time. | Not measured in this edition | Not measured in this edition | Not measured in this edition |
| Learning and adaptationChanging future performance from new evidence or experience rather than a fresh prompt alone. | Not measured in this edition | Not measured in this edition | Not measured in this edition |
| Self-checkingFinding, naming, and repairing errors in one's own work under a stated checking protocol. | Not measured in this edition | Not measured in this edition | Not measured in this edition |
| Planning and actionChoosing and carrying out linked actions toward a goal in an environment. | Not measured in this edition |
| Not measured in this edition |
| Social coordinationSharing information, roles, and decisions with other minds while preserving the joint task. | Not measured in this edition | Not measured in this edition | Not measured in this edition |
| Creation and discoveryProducing a new useful artefact, method, proof, program, or finding under test. |
| Not measured in this edition | Not measured in this edition |
what was observed
Dated evidence, conditions attached
Cards stay in chronological append order. A lineage link means the records share a broad measurement family, not necessarily an identical suite; it does not by itself prove an upgrade or identify its cause.
2022-12-08 路 System 路 Creation and discovery
AlphaCode
Google DeepMind reported an estimated rank within the top 54% of participants, approximately the level of the median competitor.
- Task
- Codeforces contest simulation
Generate programs for novel problems while simulating participation in 10 recent Codeforces contests that were newer than the training data. - Human reference
- AlphaCode's estimated contest placement was around the median participant in that selected contest set.Participants in the 10 selected Codeforces contests.This is an estimated competitive-programming placement, not a level for software engineering, creativity, or people in general.
- Exact setup
- AlphaCode generated programs at large scale, filtered and clustered them, and submitted a small candidate set under a contest simulation.
- Limits
- The result covers short, testable algorithmic problems on one contest platform; it does not show production software work or open-ended invention.
- Source
- Competitive programming with AlphaCode (Google DeepMind, 2022-12-08)
First observation in this recorded series.
2023-03-27 路 System 路 Knowledge and language
GPT-4
OpenAI reported 298 out of 400, corresponding to an estimated 90th percentile among test takers in the comparison distribution.
- Task
- Simulated Uniform Bar Examination
Answer and be graded on the MBE, MEE, and MPT components under a simulation of the real exam's conditions and 400-point scoring. - Human reference
- The reported 298 out of 400 mapped to about the 90th percentile in that task-specific distribution.Test takers in the score distribution used by OpenAI for its simulated Uniform Bar Examination comparison.A simulated exam percentile is not admission to practice law and does not measure client judgment, legal reliability, or general intelligence.
- Exact setup
- Casetext and Stanford CodeX collaborators conducted the simulation; OpenAI reported exam-specific grading and zero detected training-data contamination for this exam.
- Limits
- This is a developer report on a simulated written exam. It does not establish reliable legal practice, and GPT-4's report separately warns that the model can hallucinate and reason incorrectly.
- Source
- GPT-4 Technical Report (OpenAI, 2023-03-27)
First observation in this recorded series.
2024-01-17 路 System 路 Novel reasoning
AlphaGeometry
Google DeepMind reported that AlphaGeometry solved 25 of 30 problems; the prior state of the art solved 10.
- Task
- IMO-AG-30
Solve 30 Olympiad geometry problems drawn from 2000 through 2022 under the standard Olympiad time limit. - Human reference
- The source reports averages of 25.9 solved for gold medallists, 22.9 for silver medallists, and 19.3 for bronze medallists.Human International Mathematical Olympiad medallists evaluated on the same 30 geometry problems.This is a geometry-only comparison on a reconstructed problem set, not an entire Olympiad result or a general mathematical level.
- Exact setup
- A neural language model proposed constructions and a rule-bound symbolic deduction engine derived machine-checkable geometry proofs.
- Limits
- Geometry is normally only about one third of an Olympiad. The specialised solver does not establish performance in algebra, combinatorics, number theory, or ordinary open research.
- Source
- AlphaGeometry: An Olympiad-level AI system for geometry (Google DeepMind, 2024-01-17)
First observation in this recorded series.
2024-01-31 路 System 路 Perception and space
GPT-4V
The MMMU paper reports 56.8% micro-averaged validation accuracy for GPT-4V in the Playground setup.
- Task
- MMMU validation set
Answer 900 image-and-text questions spanning 30 college-level subjects under the benchmark's zero-shot protocol. - Human reference
- The leaderboard reports Human Expert Low at 76.2%, Medium at 82.6%, and High at 88.6% on MMMU validation.Selected human experts reported by the official MMMU leaderboard on the same validation column.These selected expert bands are not population percentiles, and MMMU mixes perception with subject knowledge and reasoning.
- Exact setup
- MMMU used a zero-shot evaluation with heterogeneous images such as charts, diagrams, maps, tables, music sheets, and chemical structures.
- Limits
- The result is below all three reported expert bands and cannot isolate pure vision from knowledge, language, and reasoning demands.
- Source
- MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (MMMU authors, 2023-11-27)
MMMU official leaderboard (MMMU Benchmark, 2024-01-31)
First observation in this recorded series.
2025-04-04 路 Agent 路 Planning and action
Claude 3.7 Sonnet agent
METR's preliminary evaluation estimated a 50% time horizon of about 55 minutes.
- Task
- METR General Autonomous Capabilities suite
Attempt 96 tasks spanning cybersecurity, AI research and development, reasoning, environment exploration, and software engineering in a simple agent scaffold. - Human reference
- The fitted estimate means 50% predicted success at suite tasks that took those experts around 55 minutes.Human experts used by METR to establish task completion times.Human task duration is a difficulty ruler for this suite; it is not the agent's runtime and not a claim about all 55-minute work.
- Exact setup
- METR equipped the model with a simple autonomous-agent scaffold and described the evaluation as preliminary, short, and imperfectly elicited.
- Limits
- Confidence intervals overlapped earlier systems, better elicitation may change results, and this older GAC suite is not byte-for-byte identical to later METR suites.
- Source
- Details about METR's preliminary evaluation of Claude 3.7 (METR, 2025-04-04)
First observation in this recorded series.
2025-08-07 路 Agent 路 Planning and action
GPT-5 Thinking agent
METR estimated a 50% time horizon of 2 hours 17 minutes, with a 95% confidence interval from 65 minutes to 4 hours 25 minutes.
- Task
- METR task-completion time horizon
Attempt METR's human-calibrated agentic software tasks, including software engineering, cybersecurity, environment exploration, and AI research and development. - Human reference
- The fitted estimate means 50% predicted success at suite tasks taking those professionals around 2 hours 17 minutes.Human professionals unfamiliar with each particular codebase, used to calibrate task completion time.This duration measures task difficulty within METR's suite, not agent runtime, all software work, or independent operation for that long.
- Exact setup
- METR evaluated gpt-5-thinking as an agent across HCAST, RE-Bench, and shorter Software Atomic Actions under its published methodology.
- Limits
- The interval is wide, the suite was nearing saturation, and scaffold, token budget, task mix, and reward-hacking treatment affect comparison.
- Source
- Details about METR's evaluation of OpenAI GPT-5 (METR, 2025-08-07)
Recorded later in the same broad measurement family after Claude 3.7 Sonnet agent (2025-04-04). Read each card's setup and limits: the suite may have changed, and this sequence alone proves neither exact comparability nor cause.
2025-11-19 路 Agent 路 Planning and action
GPT-5.1-Codex-Max agent
METR's summary gives a 50% time-horizon point estimate of 2 hours 42 minutes with a 95% interval from 75 to 350 minutes; its detailed section rounds the point to about 2 hours 40 minutes.
- Task
- METR task-completion time horizon
Attempt 170 human-calibrated tasks drawn from HCAST, RE-Bench, and Software Atomic Actions in METR's software-agent evaluation. - Human reference
- The fitted estimate means 50% predicted success around the human-calibrated 2-hour-40-minute task point in this 170-task suite.Human contractors and professionals whose completion times supplied most task-duration estimates.This is a suite-specific difficulty estimate, not agent runtime or evidence of success on arbitrary projects of the same human duration.
- Exact setup
- METR used 90 HCAST tasks plus RE-Bench and Software Atomic Actions, the Responses API, and a token budget of up to 32 million tokens per attempt.
- Limits
- Absolute token use is not directly comparable across models, the confidence interval is wide, and changed scaffolds and task subsets prevent a clean single-axis progression claim.
- Source
- Details about METR's evaluation of OpenAI GPT-5.1-Codex-Max (METR, 2025-11-19)
Recorded later in the same broad measurement family after GPT-5 Thinking agent (2025-08-07). Read each card's setup and limits: the suite may have changed, and this sequence alone proves neither exact comparability nor cause.
the blank parts matter
Known gaps in this edition
Not checked is not false, absent, refused, or incapable. These cards say what a later observation would need before it could honestly fill a cell.
- Novel reasoningSystem 路 Agent 路 Collective
This edition has one specialised geometry result and no reviewed evidence that the same reasoning transfers across unfamiliar domains or coordinated work.
Evidence needed: Matched unseen tasks across several formal and informal domains, with held-out data, human baselines, repeated runs, and transfer tests.
- Knowledge and languageSystem 路 Agent 路 Collective
A simulated bar exam does not establish current knowledge, grounded truthfulness, professional judgment, multilingual depth, or group knowledge work.
Evidence needed: Dated, contamination-checked tasks with matched practitioners, source use, calibration, abstention, and consequential error review.
- Perception and spaceSystem 路 Agent 路 Collective
MMMU mixes visual perception with language, knowledge, and reasoning and leaves embodied, temporal, three-dimensional, and shared perception unmeasured.
Evidence needed: Separate matched tests for recognition, spatial transformation, video, persistent scenes, embodied action, and multi-agent shared state.
- Memory and continuitySystem 路 Agent 路 Collective
This edition contains no reviewed, human-calibrated observation of memory or continuity at any scope.
Evidence needed: Longitudinal tests of recall, source fidelity, commitment continuity, correction, forgetting, privacy, and transfer across sessions and agents.
- Learning and adaptationSystem 路 Agent 路 Collective
This edition contains no reviewed observation that separates durable learning from prompting, retrieval, context, or a newly trained model.
Evidence needed: Pre-registered before-and-after tasks showing durable change from experience, retained over time and transferred beyond trained examples.
- Self-checkingSystem 路 Agent 路 Collective
Machine-checked AlphaGeometry proofs show verifiable output, but this edition has no matched test of a system finding and repairing its own hidden errors.
Evidence needed: Blind error-seeded tasks measuring detection, calibration, repair, false alarms, and whether checks work without access to the answer.
- Planning and actionSystem 路 Agent 路 Collective
METR provides software-agent task horizons, not a measure of reliable action in physical, social, high-context, adversarial, or collective settings.
Evidence needed: Matched repeated tasks across environments and scopes, with failures, intervention load, resource use, side effects, and higher-reliability horizons reported.
- Creation and discoverySystem 路 Agent 路 Collective
AlphaCode establishes contest program generation under tests, not novelty, value, authorship, scientific discovery, or sustained creation across scopes.
Evidence needed: Blind expert review of new artefacts and findings, independent replication or use, provenance checks, and comparisons with matched creators and teams.
the measuring rule
No dimension pays for another
- Each human reference belongs only to the named task, protocol, cohort, date, and setup.
- A percentile, solved count, accuracy, or task horizon is not a general intelligence level.
- System, agent, and collective results are different scopes and must not be merged.
- Missing evidence is a gap, not zero capability.
- This edition makes no aggregate score, overall rank, forecast, consciousness claim, or claim about human worth.
- The source, setup, uncertainty, and limits travel with every observation.
The source file is manually reviewed and exact-byte pinned. This page performs no network fetch, write, tracking, scheduled update, automatic action, or learning-state read. Corrections belong at the public issue door.
Read the complete map as JSON 路 edition sha256:1e8f5ba7851db3b93a2b7e73b68d8689ff3935f3e14430fc729d17e2d0239cfb.