TText representation

Turn writing into computable values with analysis pipelines, tokenization, morphological analysis, and TF-IDF. — 7 entries

A sequence of transformations from collected articles to words, vectors, and clusters.

Splitting text into a sequence of units called tokens.

Breaking words into morphemes and assigning parts of speech.

Representing each document by term counts while discarding word order.

Counting runs of N adjacent tokens to preserve short-range order.

Giving more weight to terms common in one document but rare across the corpus.

Comparing how strongly a term favors one group over another.