Add new mistral #7425
Add new mistral #7425
Conversation
| Thanks for your contribution! |
…nto add-new-mistral
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@ ## develop #7425 +/- ## =========================================== + Coverage 55.61% 55.74% +0.12% =========================================== Files 620 623 +3 Lines 96965 97450 +485 =========================================== + Hits 53930 54322 +392 - Misses 43035 43128 +93 ☔ View full report in Codecov by Sentry. |
| def set_input_embeddings(self, value): | ||
| self.embed_tokens = value | ||
| | ||
| def _prepare_decoder_attention_mask( |
There was a problem hiding this comment.
确认一下目前attention_mask符合我们的标准，支持2d和4d， 1和0的含义一致，可以支持intokens策略
| @wtmlon 不需要了吗？ |
| This Pull Request is stale because it has been open for 60 days with no activity. 当前Pull Request 60天内无活动，被标记为stale。 |
…nto add-new-mistral
…nto add-new-mistral
llm/data.py Outdated
| # Copyright (c) 2023 PaddlePaddle Authors. All Rights Reserved. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. |
llm/data.py Outdated
| from paddlenlp.peft import LoRAModel, PrefixModelForCausalLM | ||
| | ||
| | ||
| def get_convert_example(model): |
There was a problem hiding this comment.
mistral有chat_template吗。确认支持吗？
There was a problem hiding this comment.
llm/data.py Outdated
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at |
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| from .configuration import MistralConfig |
There was a problem hiding this comment.
官方的 mistral 直接用的 LlamaTokenizer：https://huggingface.co/mistralai/Mistral-7B-v0.3/blob/b67d6a03ca097c5122fa65904fce0413500bf8c8/tokenizer_config.json#L6183
| @@ -0,0 +1,30 @@ | |||
| { | |||
There was a problem hiding this comment.
放到config文件里面，config目录里新增mistral包含json和readme。顺带更新llm目录下的readme
| 支持zero_padding策略吗，同时新增支持dpo |
都已支持 |
3 participants
PR types
PR changes
Description
新增mistral模型