[Tokenizer] Add Fast Tokenizer

[DrownFish19]

PR types

New features

PR changes

APIs

Description

Add Fast Tokenizer.

• Take the tokenizers as the backend of new fast tokenziers.
• Compatible with the current tokenizers and new fast tokenizers.
• LLaMA3.1 and LLaMA3 can use PretrainedTokenizerFast to achieve better performance. LLaMA 1 and LLaMA 2 also can use LlamaTokenizerFast to improve tokenization performance.

[paddle-bot]

Thanks for your contribution!

[codecov]

Codecov Report

Attention: Patch coverage is 49.03537% with 317 lines in your changes missing coverage. Please review.

Project coverage is 54.81%. Comparing base (e0d2809) to head (e63092e).
Report is 225 commits behind head on develop.

Files with missing lines	Patch %	Lines
paddlenlp/transformers/convert_slow_tokenizer.py	21.73%	126 Missing ⚠️
paddlenlp/transformers/tokenizer_utils_fast.py	60.93%	125 Missing ⚠️
paddlenlp/transformers/llama/tokenizer_fast.py	43.58%	44 Missing ⚠️
paddlenlp/transformers/tokenizer_utils_base.py	58.69%	19 Missing ⚠️
paddlenlp/transformers/tokenizer_utils.py	70.00%	3 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@ ## develop #8832 +/- ## =========================================== + Coverage 54.79% 54.81% +0.01%  =========================================== Files 636 639 +3 Lines 99876 100475 +599 =========================================== + Hits 54732 55079 +347  - Misses 45144 45396 +252 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.