add mteb evaluation #8538
Merged
add mteb evaluation #8538
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Thanks for your contribution! |
w5688414 reviewed Jun 4, 2024
| @@ -0,0 +1,97 @@ | |||
| # MTEB基准评估 | |||
Contributor
There was a problem hiding this comment.
把meteb的代码和原始的evaluation目录进行合并，不单独开一个目录，结构参考：
Contributor Author
There was a problem hiding this comment.
在原evaluation下加了mteb目录
| ## 模型评估 | ||
| 使用评估脚本`eval_mteb.py`： | ||
| | ||
| - `base_model_name_or_path`: 模型名称或路径 |
Contributor
There was a problem hiding this comment.
把这个README.md和主readme进行融合，维护主readme即可
Contributor Author
There was a problem hiding this comment.
已与主README融合，并已测试过
Contributor
| 代码融合后，需要检查一下，原来的readme里面的内容是否还能跑通 |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@ ## develop #8538 +/- ## ======================================== Coverage 53.97% 53.97% ======================================== Files 618 618 Lines 96827 96827 ======================================== + Hits 52258 52259 +1 + Misses 44569 44568 -1 ☔ View full report in Codecov by Sentry. |
w5688414 reviewed Jun 5, 2024
| 是一个大规模文本嵌入评测基准，包含了丰富的向量检索评估任务和数据集。 | ||
| 本仓库主要面向其中的中英文检索任务（Retrieval），并以SciFact数据集作为主要示例。 | ||
| | ||
| 使用评估脚本`evaluation/mteb/eval_mteb.py`： |
Contributor
There was a problem hiding this comment.
参数的解释说明写到执行命令的后面，跟上面的写法保持一致。
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
PR types
New features
PR changes
Others
Description
Add the evaluation scripts for MTEB benchmark in the pipeline example "contrastive_training"