groonga-normalizer-mysql

Groonga-normalizer-mysql is a Groonga plugin. It provides MySQL compatible normalizers and a custom normalizers to Groonga.

Here are MySQL compatible normalizers:

• NormalizerMySQLGeneralCI for utf8mb4_general_ci
• NormalizerMySQLUnicodeCI for utf8mb4_unicode_ci
• NormalizerMySQLUnicode520CI for utf8mb4_unicode_520_ci
• NormalizerMySQLUnicode900 (deprecated by NormalizerMySQLUnicode) for:
  • utf8mb4_0900_ai_ci (NormalizerMySQLUnicode900)
  • utf8mb4_0900_as_ci (NormalizerMySQLUnicode900("weight_level", 2))
  • utf8mb4_0900_as_cs (NormalizerMySQLUnicode900("weight_level", 3))
  • utf8mb4_ja_0900_as_cs (NormalizerMySQLUnicode900("locale", "ja", "weight_level", 3))
  • utf8mb4_ja_0900_as_cs_ks (NormalizerMySQLUnicode900("locale", "ja", "weight_level", 4))
• NormalizerMySQLUnicode for:
  • utf8mb4_0900_ai_ci (NormalizerMySQLUnicode("version", "9.0.0"))
  • utf8mb4_0900_as_ci (NormalizerMySQLUnicode("version", "9.0.0", "accent_sensitive", true))
  • utf8mb4_0900_as_cs (NormalizerMySQLUnicode("version", "9.0.0", "accent_sensitive", true, "case_sensitive", true))
  • utf8mb4_ja_0900_as_cs (NormalizerMySQLUnicode("version", "9.0.0", "accent_sensitive", true, "case_sensitive", true, "locale", "ja"))
  • utf8mb4_ja_0900_as_cs_ks (NormalizerMySQLUnicode("version", "9.0.0", "accent_sensitive", true, "case_sensitive", true, "locale", "ja", "kana_sensitive", true))
  • utf8mb4_uca1400_ai_ci (NormalizerMySQLUnicode("version", "14.0.0"))
  • utf8mb4_uca1400_ai_cs (NormalizerMySQLUnicode("version", "14.0.0", "case_sensitive", true))
  • utf8mb4_uca1400_as_ci (NormalizerMySQLUnicode("version", "14.0.0", "accent_sensitive", true))
  • utf8mb4_uca1400_as_cs (NormalizerMySQLUnicode("version", "14.0.0", "accent_sensitive", true, "case_sensistive", true))

Here are custom normalizers:

• NormalizerMySQLUnicodeCIExceptKanaCIKanaWithVoicedSoundMark
  • It's based on NormalizerMySQLUnicodeCI
• NormalizerMySQLUnicode520CIExceptKanaCIKanaWithVoicedSoundMark
  • It's based on NormalizerMySQLUnicode520CI

They are self-descriptive name but long. They are variant normalizers of NormalizerMySQLUnicodeCI and NormalizerMySQLUnicode520CI. They have different behaviors. The followings are the different behaviors. They describes with NormalizerMySQLUnicodeCIExceptKanaCIKanaWithVoicedSoundMark but they are true for NormalizerMySQLUnicode520CIExceptKanaCIKanaWithVoicedSoundMark.

• NormalizerMySQLUnicodeCI normalizes all small Hiragana such as ぁ, っ to Hiragana such as あ, つ. NormalizerMySQLUnicodeCIExceptKanaCIKanaWithVoicedSoundMark doesn't normalize ぁ to あ nor っ to つ. ぁ and あ are different characters. っ and つ are also different characters. This behavior is described by ExceptKanaCI in the long name. This following behaviors ared described by ExceptKanaWithVoicedSoundMark in the long name.
• NormalizerMySQLUnicode normalizes all Hiragana with voiced sound mark such as が to Hiragana without voiced sound mark such as か. NormalizerMySQLUnicodeCIExceptKanaCIKanaWithVoicedSoundMark doesn't normalize が to か. が and か are different characters.
• NormalizerMySQLUnicode normalizes all Hiragana with semi-voiced sound mark such as ぱ to Hiragana without semi-voiced sound mark such as は. NormalizerMySQLUnicodeCIExceptKanaCIKanaWithVoicedSoundMark doesn't normalize ぱ to は. ぱ and は are different characters.
• NormalizerMySQLUnicode normalizes all Katakana with voiced sound mark such as ガ to Katakana without voiced sound mark such as カ. NormalizerMySQLUnicodeCIExceptKanaCIKanaWithVoicedSoundMark doesn't normalize ガ to カ. ガ and カ are different characters.
• NormalizerMySQLUnicode normalizes all Katakana with semi-voiced sound mark such as パ to Hiragana without semi-voiced sound mark such as ハ. NormalizerMySQLUnicodeCIExceptKanaCIKanaWithVoicedSoundMark doesn't normalize パ to ハ. パ and ハ are different characters.
• NormalizerMySQLUnicode normalizes all halfwidth Katakana with voiced sound mark such as ｶﾞ to halfwidth Katakana without voiced sound mark such as ｶ. NormalizerMySQLUnicodeCIExceptKanaCIKanaWithVoicedSoundMark normalizes all halfwidth Katakana with voided sound mark such as ｶﾞ to fullwidth Katakana with voiced sound mark such as ガ.

NormalizerMySQLUnicodeCIExceptKanaCIKanaWithVoicedSoundMark and NormalizerMySQLUnicode520CIExceptKanaCIKanaWithVoicedSoundMark are MySQL incompatible normalizers but they are useful for Japanese text. For example, ふらつく and ブラック has different means. NormalizerMySQLUnicodeCI identifies ふらつく with ブラック but NormalizerMySQLUnicodeCIExceptKanaCIKanaWithVoicedSoundMark doesn't identify them.

Add apt-line for the Groonga deb package repository and install groonga-normalizer-mysql package:

% sudo apt-get -y install groonga-normalizer-mysql 

Add apt-line for the Groonga deb package repository and install groonga-normalizer-mysql package:

% sudo apt-get -y install groonga-normalizer-mysql 

Install groonga-repository package:

% sudo dnf install -y https://packages.groonga.org/almalinux/8/groonga-release-latest.noarch.rpm 

Then install groonga-normalizer-mysql package:

% sudo dnf install -y --enablerepo=epel groonga-normalizer-mysql 

Install groonga-repository package:

% sudo dnf install -y https://packages.groonga.org/almalinux/9/groonga-release-latest.noarch.rpm 

Then install groonga-normalizer-mysql package:

% sudo dnf install -y --enablerepo=epel groonga-normalizer-mysql 

Install groonga-repository package:

% sudo dnf install -y https://packages.groonga.org/amazon-linux/2023/groonga-release-latest.noarch.rpm 

Then install groonga-normalizer-mysql package:

% sudo dnf install -y --enablerepo=epel groonga-normalizer-mysql 

Install groonga package (which includes groonga-normalizer-mysql):

% brew install groonga 

You need to build from source. Here are build instructions.

Install the following build tools:

• Microsoft Visual C++
• CMake

Download the latest Groonga source from GitHub releases. Source file name is formatted as groonga-X.Y.Z.zip.

Extract the source and move to the source folder:

> cd ...\groonga-X.Y.Z groonga-X.Y.Z> 

Run CMake. Here is a command line to install Groonga to C:\groonga folder:

groonga-X.Y.Z> cmake . -G "Visual Studio 14 Win64" -DCMAKE_INSTALL_PREFIX=C:\groonga 

Build:

groonga-X.Y.Z> cmake --build . --config Release 

Install:

groonga-X.Y.Z> cmake --build . --config Release --target Install 

Download the latest groonga-normalizer-mysql source from GitHub releases. Source file name is formatted as groonga-normalizer-X.Y.Z.zip.

Extract the source and move to the source folder:

> cd ...\groonga-normalizer-mysql-X.Y.Z groonga-normalizer-mysql-X.Y.Z> 

IMPORTANT!!!: Set PKG_CONFIG_PATH environment variable:

groonga-normalizer-mysql-X.Y.Z> set PKG_CONFIG_PATH=C:\groonga\local\lib\pkgconfig 

Run CMake. Here is a command line to install Groonga to C:\groonga folder:

groonga-normalizer-mysql-X.Y.Z> cmake . -G "Visual Studio 14 Win64" -DCMAKE_INSTALL_PREFIX=C:\groonga 

Build:

groonga-normalizer-mysql-X.Y.Z> cmake --build . --config Release 

Install:

groonga-normalizer-mysql-X.Y.Z> cmake --build . --config Release --target Install 

First, you need to register normalizers/mysql plugin:

groonga> register normalizers/mysql 

Then, you can use NormalizerMySQLGeneralCI and NormalizerMySQLUnicodeCI as normalizers:

groonga> table_create Lexicon TABLE_PAT_KEY --default_tokenizer TokenBigram --normalizer NormalizerMySQLGeneralCI 

• Groonga >= 8.0.4

• English: groonga-talk@lists.sourceforge.net
• Japanese: groonga-dev@lists.sourceforge.jp

• Alexander Barkov <bar@udm.net>: The author of MYSQL_SOURCE/strings/ctype-utf8.c.
• ...

• Kouhei Sutou <kou@clear-code.com>

LGPLv2 only. See doc/text/lgpl-2.0.txt for details.

This program uses normalization table defined in MySQL source code. So this program is derived work of MYSQL_SOURCE/strings/ctype-utf8.c, MYSQL_SOURCE/strings/uca900_data.h, MYSQL_SOURCE/strings/uca900_ja_data.h. This program is the same license as them and they are licensed under LGPLv2 only.

This program also uses normalization table defined in MariaDB source code. The table is generated from https://www.unicode.org/Public/UCA/14.0.0/allkeys.txt . So the normalization table is licensed under https://www.unicode.org/copyright.html . It's compatible with LGPLv2 only. So the program can use LGPLv2 only.

editor doc/text/news.md

rake release

release task execute the following tasks:

• rake release:version:update
• rake release:tag
• rake dev:version:bump

A package for Ubuntu build and publish automatically on GitHub Actions. So, we only confirm result of build and publish on LaunchPad.