Skip to content

Conversation

@JianlongCao
Copy link

1.4.0 chonkie changed the api and parameter which will cause the memory init fail by "SentenceChunker.init() got an unexpected keyword argument 'tokenizer_or_token_counter'"

Description

Summary:
Fix compatibility issue introduced by chonkie 1.4.0, where the API and parameter changes cause memory initialization to fail with the following error:
SentenceChunker.init() got an unexpected keyword argument 'tokenizer_or_token_counter'.

This PR updates the memory initialization module to align with the new chonkie API and corrects parameter handling during the chunking process.

Fix: #578

Docs Issue/PR: NA

Reviewer: TBA

Checklist:

  • [ x] I have performed a self-review of my own code | 我已自行检查了自己的代码
  • [ x] I have commented my code in hard-to-understand areas | 我已在难以理解的地方对代码进行了注释
  • [x ] I have added tests that prove my fix is effective or that my feature works | 我已添加测试以证明我的修复有效或功能正常
  • [x ] I have created related documentation issue/PR in MemOS-Docs (if applicable) | 我已在 MemOS-Docs 中创建了相关的文档 issue/PR(如果适用)
  • [x ] I have linked the issue to this PR (if applicable) | 我已将 issue 链接到此 PR(如果适用)
  • [x ] I have mentioned the person who will review this PR | 我已提及将审查此 PR 的人
1.4.0 chonkie changed the api and parameter which will cause the memory init fail by "SentenceChunker.__init__() got an unexpected keyword argument 'tokenizer_or_token_counter'"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant