Skip to contents

text2map.pretrained

Pretrained Models for Text Analysis

This is an R package to load and download pretrained text analysis models. Some models are bundled with the package; others are larger and must be separately downloaded. See also text2map.

Please check out our book Mapping Texts: Computational Text Analysis for the Social Sciences

Installation 

library(remotes)
install_gitlab("culturalcartography/text2map.pretrained")

library(text2map.pretrained)

Usage

Bundled Models

A few smaller topic models are included when the package is installed. Use load_pretrained() to load them:

stm <- load_pretrained("stm_envsoc")
stm <- load_pretrained("stm_fiction_cohort")

Downloaded Models

Word embedding models are larger and must be downloaded once per machine, then loaded with load_pretrained().

# download the model once per machine
download_pretrained("vecs_fasttext300_wiki_news")

# load the model each session
wv <- load_pretrained("vecs_fasttext300_wiki_news")
dim(wv)

Available Models

Structural Topic Models (Bundled)

MODEL N Docs Description
stm_envsoc 817 Environmental sociology abstracts
stm_fiction_cohort 1,000 Fiction-author cohort study

Static Word Embedding Models (Download)

MODEL Language N Terms Dims Method
vecs_fasttext300_wiki_news English 1,000,000 300 fastText
vecs_fasttext300_wiki_news_subword English 1,000,000 300 fastText
vecs_fasttext300_commoncrawl English 2,000,000 300 fastText
vecs_glove300_wiki_gigaword English 400,000 300 GloVe
vecs_cbow300_googlenews English 3,000,000 300 CBOW
vecs_sgns300_bnc_pos English 163,473 300 SGNS
vecs_sgns300_googlengrams_kte_en English 928,250 300 SGNS
vecs_glove300_metal_lyrics English 52,885 300 GloVe
vecs_svd20_metal_type English 54,187 20 SVD
vecs_svd20_metal_position English 74 20 SVD
vecs_svd20_metal_bpe English 3,299 20 SVD

Note: vecs_svd20_metal_bpe rownames are integer BPE token IDs encoded as character strings (e.g. "3847", "14825"), not human-readable words. Decode them on the caller side with rtiktoken::decode_tokens(as.integer(rownames(mat)), "r50k_base"). rtiktoken is not a dependency of text2map.pretrained.

Diachronic (Temporal) Word Embedding Models (Download)

MODEL Language N Terms Dims Method Years
vecs_sgns300_coha_histwords English 50,000 300 SGNS 1810-2000
vecs_sgns300_googlengrams_histwords English 100,000 300 SGNS 1800-1990
vecs_sgns300_googlengrams_fic_histwords English 100,000 300 SGNS 1800-1990
vecs_sgns300_googlengrams_histwords_fr French 100,000 300 SGNS 1800-1990
vecs_sgns300_googlengrams_histwords_de German 100,000 300 SGNS 1800-1990
vecs_sgns300_googlengrams_histwords_zh Chinese 29,701 300 SGNS 1950-1990
vecs_svd300_googlengrams_histwords English 75,682 300 SVD 1800-1990
vecs_sgns200_british_news English 78,879 200 SGNS 1800-1910

BERT Contextual Word-Token Embeddings (Download)

MODEL Language N Tokens Dims Quantization Size
vecs_bert768_metal_lyrics_fp32 English 867,305 768 none (baseline) 2,970 MB
vecs_bert768_metal_lyrics_fp16 English 867,305 768 half-precision (lossless) 1,170 MB
vecs_bert768_metal_lyrics_int8 English 867,305 768 8-bit 765 MB
vecs_bert768_metal_lyrics_int4 English 867,305 768 4-bit (recommended default) 382 MB
vecs_bert768_metal_lyrics_binary English 867,305 768 1-bit 123 MB
vecs_bert768_scifi_pulp_fp32 English 2,084,199 768 none (baseline) 6,700 MB
vecs_bert768_scifi_pulp_fp16 English 2,084,199 768 half-precision (lossless) 2,700 MB
vecs_bert768_scifi_pulp_int8 English 2,084,199 768 8-bit 1,700 MB
vecs_bert768_scifi_pulp_int4 English 2,084,199 768 4-bit (recommended default) 835 MB
vecs_bert768_scifi_pulp_binary English 2,084,199 768 1-bit 293 MB
vecs_bert768_scifi_pulp_sub English 8,758 768 all 5 levels (list) 90 MB

Note: Unlike the other vecs_* models, these are contextual embeddings — one vector per word occurrence (rowname word_position, e.g. "chorus_1"), not per word type, from bert-base-uncased run over a stratified subset of the source corpus (4,707 songs for metal lyrics; 4,793 magazine passages for scifi pulp). Each row carries a document-id attribute (song_id or passage_id). load_pretrained() dequantizes the int8/int4/binary levels to a double matrix by default; pass dequantize = FALSE to get the raw compact integer matrix instead. vecs_bert768_scifi_pulp_sub is a small already-dequantized list (5 quantization levels + metadata) for comparing quantization quality on a focal-word subset, not a plain matrix.

Transformer / LLM Models (Download)

MODEL Language Architecture Layers Hidden Dim N Params (approx)
llm_bert_tiny4l312d_general English TinyBERT (distilled BERT) 4 312 14.5M

Note: Unlike the vecs_* matrix models, these ship as a full model directory (config, weights, tokenizer files) in HuggingFace transformers format. load_pretrained() does not read the model into an R object — it returns list(path, config). Load the model itself on the caller side, e.g. with reticulate + Python transformers.

LoRA Adapter Models (Download)

MODEL Language Base Model Task Rank Target Modules Training Corpus
lora_qwen1.5b_aita_cls English Qwen/Qwen2-1.5B Sequence classification (AITA verdict) 8 q_proj, v_proj corpus_reddit_aita10k
lora_qwen1.5b_aita_gen English Qwen/Qwen2-1.5B Causal LM (AITA-style generation) 16 all attention + MLP projections corpus_reddit_aita10k

Note: These are PEFT LoRA adapters, not full models — the directory contains adapter_config.json, adapter_model.safetensors, and tokenizer files, but no base-model weights. load_pretrained() returns list(path, config), where config is the parsed adapter_config.json (LoRA hyperparameters). Fetch the base model separately (e.g. from Hugging Face using config$base_model_name_or_path) and apply the adapter on top of it with peft/transformers, e.g. via reticulate. Both adapters were fine-tuned on corpus_reddit_aita10k, a corpus of ~10k Reddit “Am I the Asshole” posts.

File Formats

Models are stored in multiple formats, loaded in this priority order:

  1. .qs2 — Fastest loading (~10x faster than .rda)
  2. .fst — Fast loading (~3x faster than .rda)
  3. .rda — Standard R format (fallback)

The download_pretrained() function downloads the best available format from the repository. The load_pretrained() function handles both bundled models (via data()) and downloaded models (auto-detecting format).

Helper Functions

The package provides several helper functions for managing models:

Function Description
list_models() List all available models with metadata
model_info() Get detailed info about a specific model
model_exists() Check if a model is available (bundled or downloaded)
model_path() Get file path to downloaded model (NA for bundled)
download_pretrained() Download an on-demand model
delete_model() Remove a downloaded model 
# List all available models
list_models()

# List only available (bundled + downloaded) models
list_models(downloaded_only = TRUE)

# Get info about a specific model
model_info("stm_fiction_cohort")

# Check if model is available
model_exists("vecs_sgns300_bnc_pos")

# Get path to downloaded model
model_path("vecs_sgns300_bnc_pos")

# Delete a downloaded model (cannot delete bundled models)
delete_model("vecs_sgns300_bnc_pos")

There are several related packages hosted on GitLab:

install.packages("text2map")

library(remotes)
install_gitlab("culturalcartography/text2map.theme")
install_gitlab("culturalcartography/text2map.corpora")
install_gitlab("culturalcartography/text2map.dictionaries")

Contributions and Support

We welcome new models. If you have an embedding model or topic model you would like to be easily available to other researchers in R, send us an email (maintainers [at] textmapping.com) or submit pull requests.

Please report any issues or bugs here: https://gitlab.com/culturalcartography/text2map.pretrained/-/issues