First, each token in the vocabulary is mapped to an integer in a lookup dictionary. Next, documents are converted to sequences of integers where each integer is an index of the token from the dictionary.
seq_builder(
data,
text,
doc_id = NULL,
vocab = NULL,
maxlen = NULL,
matrix = TRUE
)Data.frame with column of texts and column of document ids
Name of the column with documents' text
Name of the column with documents' unique ids.
Default is NULL, if a list of terms is provided,
the function will return a DTM with terms restricted
to this vocabulary. Columns will also be in the same
order as the list of terms.
Integer indicating the maximum document length. If NULL (default), the length of the longest document is used.
Logical, TRUE (default) returns a matrix, FALSE a list
returns a matrix or list
Function will return a matrix of integer sequences by default.
The columns will be the length of the longest document or
maxlen, with shorter documents padded with zeros. The
dictionary will be an attribute of the matrix accessed with
attr(seq, "dic"). If matrix = FALSE, the function will
return a list of integer sequences. The vocabulary will either
be each unique token in the corpus, or a the list of words
provided to the vocab argument. This kind of text
representation is used in tensorflow
and keras.