Unsupervised Segmentation

July 25, 2018

Boundary prediction

Branching Entropy（BE）를 이용한 방법

• Accessor Variety & Branching Entropy / lovit
• “Branching Entropy” / ratsgo’s blog에 잘 설명되어 있다.

library

• Unsupervised tokenizers in soynlp project

WordRank

KR-WordRank, 토크나이저를 이용하지 않는 한국어 키워드 추출기

Articles

• A Simple and Effective Unsupervised Word Segmentation Approach
• KR-WordRank : WordRank를 개선한 비지도학습 기반 한국어 단어 추출 방법

Library

• wordseg_wordrank in python

기타

• Mutual Information (MI) / Sun, Shen, and Tsou (1998)
• WordEnds / Fleck (2008)
• Minimum Description Length (MDL) criterion and local statistics BE / Zhikov, Takamura, and Okumura (2010)

Word recognition

Accessor Variety

library

• Unsupervised tokenizers in soynlp project

Cohesion Probability를 이용한 방법.

• Cohesion score + L-Tokenizer / lovit
• “Cohesion Probability” / ratsgo’s blog에 잘 설명되어 있다.

library

• Unsupervised tokenizers in soynlp project

기타

• Description Length Gain (DLG) / Kit and Wilks (1999)
• Dirichlet Process (DP) and Hierarchical Dirichlet Process (HDP) / Goldwater, Griffiths, and Johnson (2006)

Word Embeadding

Deeplearning을 이용한 방법

tutorial

library

• Deepnlp package에