Based on the extensibility of the inspect-ai framework, the steps to add a new benchmark test are:
- In the project directory of the
benchmarks/A new Python module is created under theBaseBenchmarkresemble - fulfillment
load_dataset()cap (a poem)evaluate()Methodology, defining the assessment logic - By means of a decorator
@register_benchmarkRegistering for tests, setting up metadata (category, difficulty, etc.) - newly built
conftest.pyAdd dataset download logic (HuggingFace permissions need to be handled) - utilization
uv run pytest benchmarks/新测试名verification implementation - pass (a bill or inspection etc)
bench listConfirm that the new test has appeared in the available list
It is recommended to refer to existing implementations of tests such as MMLU to keep the code style uniform.
This answer comes from the articleOpenBench: an open source benchmarking tool for evaluating language modelsThe
May not be reproduced without permission:AI productivity tools " How to extend OpenBench to support new benchmarks?
English 
简体中文 
日本語 
Deutsch 
Português do Brasil