[Fix] computation error in SemScore

[penguinwang96825]

Fix computation error for SemScore in this PR.

Example usage:

evalscope eval \ --model qwen3-omni-flash \ --api-url https://dashscope-intl.aliyuncs.com/compatible-mode/v1 \ --api-key <API_KEY> \ --eval-type server \ --datasets torgo \ --dataset-hub huggingface \ --limit 10 \ --use-cache /evalscope_cache/qwen3-omni-flash/torgo

Overall report table:

+------------------+-----------+-----------+----------+-------+---------+---------+ | Model | Dataset | Metric | Subset | Num | Score | Cat.0 | +==================+===========+===========+==========+=======+=========+=========+ | qwen3-omni-flash | torgo | cer | mild | 10 | 0.245 | default | +------------------+-----------+-----------+----------+-------+---------+---------+ | qwen3-omni-flash | torgo | cer | moderate | 10 | 0.3746 | default | +------------------+-----------+-----------+----------+-------+---------+---------+ | qwen3-omni-flash | torgo | cer | severe | 10 | 0.7456 | default | +------------------+-----------+-----------+----------+-------+---------+---------+ | qwen3-omni-flash | torgo | cer | OVERALL | 30 | 0.4551 | - | +------------------+-----------+-----------+----------+-------+---------+---------+ | qwen3-omni-flash | torgo | wer | mild | 10 | 0.4 | default | +------------------+-----------+-----------+----------+-------+---------+---------+ | qwen3-omni-flash | torgo | wer | moderate | 10 | 0.5143 | default | +------------------+-----------+-----------+----------+-------+---------+---------+ | qwen3-omni-flash | torgo | wer | severe | 10 | 0.925 | default | +------------------+-----------+-----------+----------+-------+---------+---------+ | qwen3-omni-flash | torgo | sem_score | mild | 10 | 0.6381 | default | +------------------+-----------+-----------+----------+-------+---------+---------+ | qwen3-omni-flash | torgo | sem_score | moderate | 10 | 0.5761 | default | +------------------+-----------+-----------+----------+-------+---------+---------+ | qwen3-omni-flash | torgo | sem_score | severe | 10 | 0.3659 | default | +------------------+-----------+-----------+----------+-------+---------+---------+

[gemini-code-assist]

Summary of Changes

Hello @penguinwang96825, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses and fixes a computation error within the SemScore metric. The changes involve a significant refactoring of how semantic and other related metrics are handled, moving towards a more modular and maintainable architecture. The core SemScore implementation now manages its NLI model dependencies internally, simplifying its external usage, while common metric calculations like CER have been abstracted into dedicated classes. These updates aim to enhance the accuracy and reliability of metric computations within the evaluation framework.

Highlights

• SemScore Initialization Simplification: The initialization of the SemScore metric has been simplified across the codebase. The model_id_or_path parameter is no longer explicitly passed during SemScore instantiation in torgo_adapter.py, as the SemScorer class now internally manages the loading of its NLI model with a hardcoded path.
• Metric Refactoring and Centralization: The logic for calculating CER (Character Error Rate) has been extracted from torgo_adapter.py into a new, dedicated CER metric class in metric.py. This centralizes metric calculation and improves modularity. A new match_score method was also introduced in torgo_adapter.py for single-sample metric computation.
• Code Clarity and Maintainability: Significant improvements have been made to sem_score/scorer.py, including the addition of comprehensive type hints, detailed docstrings, and the definition of score bounds as constants. The NLI model loading now uses specific Roberta classes instead of generic Auto classes, and the NLI scoring and normalization methods have been refactored for better readability and robustness.
• Minor Adapter Configuration Changes: The TorgoAdapter now explicitly sets add_overall_metric and add_aggregation_name to False during initialization, and the transcript field is now included in the sample metadata.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request fixes a computation error in SemScore by refactoring the scorer and centralizing metric calculations. The changes significantly improve code quality in evalscope/metrics/sem_score/scorer.py by adding type hints, improving documentation, and making the code more robust. I've identified a few issues: a bug in the new CER metric where the language parameter is ignored, a potential crash in SemScorer if an unsupported model is used, an unsafe tensor creation pattern, and an unused parameter. Addressing these will further improve the code.

[Yunnglin]

Thank you for your PR, could you fix the lint issues? Run the following command:

pip install pre-commit pre-commit install pre-commit run --all-files