[WebLLM][WASM] Add v0_2_48 models for web-llm #129
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
CharlieFRuan added a commit to mlc-ai/web-llm that referenced this pull request Jul 12, 2024
Before this PR, we may run into issues like the following when using json mode with Phi3: ``` index.js:1913 [FATAL] /Users/cfruan/Documents/mlc-llm/cpp/grammar/grammar_state_matcher.cc:202: Check failed: (token_id >= 0 && token_id < init_ctx_->vocab_size) is false: Invalid token id 32042 for GrammarStateMatcher ``` This PR fixes this issue for models like Phi3, where the `vocab_size` in `config.json` may be larger than vocabs recorded in `tokenizer.json`. For more see: - mlc-ai/mlc-llm#2651 As a result, we had to recompile all WASMs, hence updating the wasm version to v0_2_48: - mlc-ai/binary-mlc-llm-libs#129 To support this in runtime, `ChatConfig` adds a new field `vocab_size` read from `mlc-chat-config.json` and update the signature of internal grammar function `findNextTokenBitmask()`.
CharlieFRuan added a commit to mlc-ai/web-llm that referenced this pull request Jul 12, 2024
### Changes - Fixed potential error in json mode for some models like Phi3 and Qwen2; for more see - #501 ### Model WASM Version Due to the fix, updated WASMs to `v0_2_48` - mlc-ai/binary-mlc-llm-libs#129 ### TVMjs No change, compiled at apache/tvm@32e9a48
jingyi-zhao-01 pushed a commit to jingyi-zhao-01/web-llm that referenced this pull request Dec 8, 2024
Before this PR, we may run into issues like the following when using json mode with Phi3: ``` index.js:1913 [FATAL] /Users/cfruan/Documents/mlc-llm/cpp/grammar/grammar_state_matcher.cc:202: Check failed: (token_id >= 0 && token_id < init_ctx_->vocab_size) is false: Invalid token id 32042 for GrammarStateMatcher ``` This PR fixes this issue for models like Phi3, where the `vocab_size` in `config.json` may be larger than vocabs recorded in `tokenizer.json`. For more see: - mlc-ai/mlc-llm#2651 As a result, we had to recompile all WASMs, hence updating the wasm version to v0_2_48: - mlc-ai/binary-mlc-llm-libs#129 To support this in runtime, `ChatConfig` adds a new field `vocab_size` read from `mlc-chat-config.json` and update the signature of internal grammar function `findNextTokenBitmask()`.
jingyi-zhao-01 pushed a commit to jingyi-zhao-01/web-llm that referenced this pull request Dec 8, 2024
### Changes - Fixed potential error in json mode for some models like Phi3 and Qwen2; for more see - mlc-ai#501 ### Model WASM Version Due to the fix, updated WASMs to `v0_2_48` - mlc-ai/binary-mlc-llm-libs#129 ### TVMjs No change, compiled at apache/tvm@32e9a48
atebites-hub pushed a commit to atebites-hub/web-llm that referenced this pull request Oct 4, 2025
Before this PR, we may run into issues like the following when using json mode with Phi3: ``` index.js:1913 [FATAL] /Users/cfruan/Documents/mlc-llm/cpp/grammar/grammar_state_matcher.cc:202: Check failed: (token_id >= 0 && token_id < init_ctx_->vocab_size) is false: Invalid token id 32042 for GrammarStateMatcher ``` This PR fixes this issue for models like Phi3, where the `vocab_size` in `config.json` may be larger than vocabs recorded in `tokenizer.json`. For more see: - mlc-ai/mlc-llm#2651 As a result, we had to recompile all WASMs, hence updating the wasm version to v0_2_48: - mlc-ai/binary-mlc-llm-libs#129 To support this in runtime, `ChatConfig` adds a new field `vocab_size` read from `mlc-chat-config.json` and update the signature of internal grammar function `findNextTokenBitmask()`.
atebites-hub pushed a commit to atebites-hub/web-llm that referenced this pull request Oct 4, 2025
### Changes - Fixed potential error in json mode for some models like Phi3 and Qwen2; for more see - mlc-ai#501 ### Model WASM Version Due to the fix, updated WASMs to `v0_2_48` - mlc-ai/binary-mlc-llm-libs#129 ### TVMjs No change, compiled at apache/tvm@32e9a48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Compiled at:
The only expected change in the WASM is the creation of a bitmask in grammar, fixing grammar correctness for models like Phi-3 and QWen2. For more see: