Commit 5369ab6

authored

Tokenization notebook (#173)

* Tokenization notebook * remove comment * exclude special tokens before decoding * improve descriptions * add overlap; remove text file * run nbtest through Makefile * reformulate recommendation * remove paragraph I wanted to omit * move to `document-chunking` folder

1 parent 5264994 commit 5369ab6Copy full SHA for 5369ab6

2 files changed

+330

-0

lines changed

notebooks/document-chunking
- Makefile
- tokenization.ipynb

2 files changed

+330

-0

lines changed

`‎notebooks/document-chunking/Makefile‎`

-Original file line number
+Diff line change
@@ @@ -1,5 +1,6 @@ @@
 NBTEST = ../../bin/nbtest
 NOTEBOOKS = \
 +tokenization.ipynb \
 with-index-pipelines.ipynb \
 with-langchain-splitters.ipynb

Comments

(0)