Configurations - Training and Testing Settings

Training Settings

Framework: LLaMA_Factory version 0.9.2

Base Model: Llama-3.1-8B, Qwen2.5-7B

Parameter Details

To ensure fair training and minimize the impact of training configurations on evaluation results, we carefully reference a wide range of existing literature to standardize the hyperparameter settings across different models. This approach helps eliminate performance bias caused by inconsistent training setups.
- The final training configurations for the LLaMA and Qwen model families are summarized below.

	LLaMA	Llama for long CoT	Qwen	Qwen for long CoT
GPU	8*A100	8*A100	8*A100	8*A100
Base model	Meta-Llama-3.1-8B	Meta-Llama-3.1-8B	Qwen2.5-7B	Qwen2.5-7B
deepspeed	ds_z2_config	ds_z2_config	ds_z2_config	ds_z2_config
template	default	default	default	default
cutoff_len	4096	32768	4096	32768
preprocessing_num_workers	16	16	true	true
Packing	false	true	false	true
per_device_train_batch_size	4	1	4	1
per_device_train_batch_size	4	2	4	2
learning_rate	2.0e-5	3.0e-5	5.0e-6	5.0e-5
use_liger_kernel	true	true	true	true
num_train_epochs	3.0	3.0	3.0	3.0
lr_scheduler_type	cosine	cosine	cosine	cosine
warmup_ratio	0.03	0.03	0.1	0.1

Testing Settings

Framework: OpenCompass version 0.4.2

Parameter Details

	LLaMA	Qwen
max-out-len	32768	32768
hf-type	chat	chat
inference setting	vllm_llama3_1_8b_instruct	vllm_qwen2_5_7b_instruct
accelerator	vllm+cutoff	vllm+cutoff

To achieve reliable and fair comparisons of model performance, we design a standardized and reproducible evaluation pipeline based on extensive preliminary experiments and analysis. Key aspects include:

Evaluation strictly follows official protocols or widely adopted tools in the community, such as math-eval-harness and lm-evaluation-harness, to ensure consistency and comparability with existing benchmarks.
To mitigate potential performance bias caused by inaccurate answer extraction, we employ high-performance models to assist in post-processing:

For code-related benchmarks, we adopt the default evaluation logic provided by the original tools.
For non-code benchmarks, we use powerful large models (e.g., xVerify, Omni-Judge) to extract and evaluate answers, ensuring higher accuracy and robustness in the results.

The detailed evaluation configurations used in our testing tools are listed below.

Domain	Benchmarks	Benchmark_file	Evaluator	Shot	Metric
general	DROP	drop_gen_a2697c	IAAR-Shanghai/xVerify-9B-C	3shot	accuracy
general	IFEval	IFEval_gen	IFEvaluator	0shot	Average accuracy on all IFEval benchmarks
general	AGIEval	agieval_xver_gen	IAAR-Shanghai/xVerify-9B-C	0shot	accuracy
general	MMLU-PRO	mmlu_pro_few_shot_xver_gen	IAAR-Shanghai/xVerify-9B-C	5shot	Average accuracy on all mmlu pro benchmarks
math	Omni-MATH	omni_math_gen	KbsdJames/Omni-Judge	0shot	accuracy
math	OlympiadBenchMath	OlympiadBenchMath_0shot_xver_gen	IAAR-Shanghai/xVerify-9B-C	0shot	accuracy
math	GSM8K	gsm8k_0shot_xver_gen	IAAR-Shanghai/xVerify-9B-C	0shot	accuracy
math	MATH-500	math_prm800k_500_0shot_cot_xver_gen	IAAR-Shanghai/xVerify-9B-C	0shot	accuracy
math	AIME_2024	aime2024_repeat8_xver_gen	IAAR-Shanghai/xVerify-9B-C	0shot	Average accuracy of 8 run
code	HumanEval	humaneval_gen_8e312c	HumanEvalEvaluator	0shot	pass@1
code	HumanEval+	humaneval_plus_gen_8e312c	HumanEvalPlusEvaluator	0shot	pass@1
code	MBPP	sanitized_mbpp_mdblock_gen_a447ff	MBPPEvaluator	3 shot	pass@1
code	LiveCodeBench	livecodebench_gen	LCBCodeGenerationEvaluator	0shot	pass@1
reasoning	ARC_c	ARC_c_xver_gen	IAAR-Shanghai/xVerify-9B-C	0shot	accuracy
reasoning	BBH	bbh_xver_gen	IAAR-Shanghai/xVerify-9B-C	0shot	accuracy
reasoning	KOR-Bench	korbench_single_0_shot_xver_gen	IAAR-Shanghai/xVerify-9B-C	0shot	Average accuracy on all korbench benchmarks
reasoning	CaLM	calm	CaLMEvaluator	0shot	Average accuracy on all calm benchmarks
reasoning	GPQA	gpqa_xver_gen	IAAR-Shanghai/xVerify-9B-C	0 shot	accuracy

The complete evaluation setup and parameter configurations can be found at: https://github.com/OpenDataArena/OpenDataArena-Tool

Dataset Selection Rules

To ensure fairness and effectiveness in evaluation, we have established strict dataset selection criteria. All datasets used for training and evaluation are sourced from the Hugging Face platform and must meet the following screening conditions:

Selection Criteria

Criteria	Requirements
Impact Metrics	`likes` or `downloads` count greater than 10 on Hugging Face platform, ensuring community recognition
Priority Ranking	In domains with abundant training data (e.g., general dialogue, mathematical reasoning, code generation), prioritize datasets with higher `likes` counts
Recency Requirement	Dataset last updated after January 2023, ensuring data timeliness and relevance
Size Limitation	Dataset size limited to under 1M, balancing training effectiveness with computational resource consumption
Suitability Assessment	Dataset purpose or content must be suitable for SFT (Supervised Fine-Tuning) training, containing high-quality instruction-response pairs

Quality Assurance

To ensure training data quality, we conduct the following additional validations:

Content Review: Check datasets for harmful, biased, or inappropriate content
Format Standardization: Ensure all datasets conform to unified input-output format requirements
Diversity Balance: Maintain diversity and balance across different domains
Regular Updates: Periodically update selections based on community feedback and newly released high-quality datasets