867k BERT contextual word-token embeddings from metal lyrics

A matrix of 867,305 word-token vectors (rows) and 768 dimensions (columns). Unlike the `vecs_*` static embeddings (one vector per word type), these are **contextual**: `bert-base-uncased` last-hidden-state vectors for each word position in a stratified 4,707-song subset of the metal lyrics corpus, with WordPiece subwords merged back into whole words. Rownames are `word_position` (e.g. `"chorus_1"`, `"return_2"`), so the same word type has a different vector at each occurrence. A `song_id` attribute (integer, 1-4707) maps each row to its source song.

Format

A matrix of 867,305 rows and 768 columns

Source

`bert-base-uncased` embeddings of a stratified subset of the metal lyrics corpus in text2map.corpora

Details

Five quantization levels are available, trading file size for precision: `_fp32` (baseline), `_fp16` (lossless half-precision), `_int8`, `_int4` (near-identical quality to int8 at half the size), and `_binary` (1 bit/value, coarsest). By default `load_pretrained()` dequantizes `int8`/`int4`/`binary` back to a double matrix; pass `dequantize = FALSE` to get the raw compact integer matrix with its `scale`/`offset`/`quantized` attributes.

Examples

if (FALSE) { # \dontrun{

## download the model (once per machine) -- int4 is the recommended default
download_pretrained("vecs_bert768_metal_lyrics_int4")

## load the model each session (dequantized to double by default)
wv <- load_pretrained("vecs_bert768_metal_lyrics_int4")
dim(wv) == c(867305, 768)

## get the raw quantized integer matrix instead
wv_raw <- load_pretrained("vecs_bert768_metal_lyrics_int4", dequantize = FALSE)
attr(wv_raw, "quantized") == "int4"

} # }