[GCU] Support llama for GCU #8445
Conversation
| Thanks for your contribution! |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@ ## develop #8445 +/- ## =========================================== - Coverage 55.42% 54.29% -1.14% =========================================== Files 617 617 Lines 96286 96340 +54 =========================================== - Hits 53367 52303 -1064 - Misses 42919 44037 +1118 ☔ View full report in Codecov by Sentry. |
| ) # [bs, 1, seq_len, seq_len] | ||
| is_casual = False | ||
| if self.config.use_flash_attention: | ||
| if self.config.use_flash_attention and get_env_device() != "gcu": |
There was a problem hiding this comment.
这里在attention mask的处理上，GCU不一样的地方是什么？
There was a problem hiding this comment.
基于 use_flash_attention kernel 的实现，is_casual 情况下也是需要当前与输入相同dtype的attention_mask，而不是None或者bool类型的mask。
| parser = get_eval_parser() | ||
| args = parser.parse_args() | ||
| paddle.set_default_dtype(args.dtype) | ||
| paddle.set_device(args.device) |
There was a problem hiding this comment.
在训练初始的位置设置set_device，这里再重新设置的原因是什么？
There was a problem hiding this comment.
单独使用的eval.py测试，没有训练初始位置？ 当然不设置默认应该也是当前的device，可以去除
| ) | ||
| else: | ||
| cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len) | ||
| cos, sin, _ = self.rotary_emb(value_states, seq_len=kv_seq_len) |
There was a problem hiding this comment.
一定要加 cos_sin 的优化吗？代码改动很大，而且会导致其他设备性能下降，凭空多了很多开销。
或者你们需要的时候再自己去造一个 cos_sin
There was a problem hiding this comment.
这里主要是因为算子的实现与paper或者vllm一致，使用了与这里不同的sin/cos。关于其他设备性能开销，一方面，应该大多table的计算只在初始化阶段，另一方面，我们将按照第一个issue的建议，在特定设备进行计算，这里仅仅只会多返回一个None。
EnflameGCU force-pushed the paddle_llm_upstream branch from 1ce5c16 to 0afe3fa Compare EnflameGCU force-pushed the paddle_llm_upstream branch from 0afe3fa to 32d66ef Compare 
| https://xly.bce.baidu.com/paddlepaddle/Paddle-NLP/newipipe/detail/10720664/job/26276076 这个PR的 rope 接口改动，貌似导致自动并行代码挂了 |
4 participants
PR types
New features
PR changes
Models
Description
Support llama for GCU