1. Training & Evaluation Tools

Our training and testing tools provide a consistent, controllable, and reproducible experimental platform, enabling fair comparisons across different datasets and model configurations.

🔧 Train Tool: model_train

• Built upon the widely used LLaMA Factory training framework.
• Standardized training pipeline supporting major open-source models such as LLaMA, Qwen, and others.
• Configurable components including data loading, fine-tuning strategies, logging, and more.
• Training configurations are derived from extensive literature review to ensure fairness and comparability.

🔧 Test Tool: model_eval

• Built on the popular OpenCompass evaluation framework.
• Unified evaluation pipeline compatible with a variety of tasks (e.g., code generation, mathematical reasoning, open-domain QA).
• Aligned with mainstream tools such as math-eval-harness and lm-evaluation-harness.
• Supports automated result extraction and formatting, enhancing evaluation efficiency and consistency.

2. Data Scoring Tool (Data Scorer)

We provide a multi-dimensional data scoring framework designed to assess the quality and value of data from various perspectives. It supports LLM-based evaluation, statistical analysis, and task-related performance metrics.

🔧 Test Tool: data_scorer

• A versatile and extensible framework for multi-type, multi-dimensional data scoring.

• Supports LLM-based subjective scoring, including tools like LLM-as-a-Judge.
• Supports model-based metrics such as diversity, correctness, and complexity.
• Includes statistical data features such as data length.

• Seamlessly integrates with the training and testing tools to evaluate data effectiveness based on real downstream performance.

• Applicable for tasks like selecting high-quality data, building dataset subsets, and analyzing the impact of data on model performance.