kuromoji analyzer

The kuromoji analyzer uses the following analysis chain:

• CJKWidthCharFilter from Lucene
• kuromoji_tokenizer
• kuromoji_baseform token filter
• kuromoji_part_of_speech token filter
• ja_stop token filter
• kuromoji_stemmer token filter
• lowercase token filter

It supports the mode and user_dictionary settings from kuromoji_tokenizer.

The kuromoji_tokenizer tokenizer uses characters from the MeCab-IPADIC dictionary to split text into tokens. The dictionary includes some full-width characters, such as ｏ and ｆ. If a text contains full-width characters, the tokenizer can produce unexpected tokens.

For example, the kuromoji_tokenizer tokenizer converts the text Ｃｕｌｔｕｒｅ ｏｆ Ｊａｐａｎ to the tokens [ culture, o, f, japan ] instead of [ culture, of, japan ].

To avoid this, add the icu_normalizer character filter to a custom analyzer based on the kuromoji analyzer. The icu_normalizer character filter converts full-width characters to their normal equivalents.

First, duplicate the kuromoji analyzer to create the basis for a custom analyzer. Then add the icu_normalizer character filter to the custom analyzer. For example:

 PUT index-00001 { "settings": { "index": { "analysis": { "analyzer": { "kuromoji_normalize": { "char_filter": [ "icu_normalizer" ], "tokenizer": "kuromoji_tokenizer", "filter": [ "kuromoji_baseform", "kuromoji_part_of_speech", "cjk_width", "ja_stop", "kuromoji_stemmer", "lowercase" ] } } } } } } 

1. Creates a new custom analyzer, kuromoji_normalize, based on the kuromoji analyzer.
2. Adds the icu_normalizer character filter to the analyzer.