2.08M BERT contextual word-token embeddings from scifi pulp magazines

A matrix of 2,093,785 word-token vectors (rows) and 768 dimensions (columns). Unlike the `vecs_*` static embeddings (one vector per word type), these are **contextual**: `bert-base-uncased` last-hidden-state vectors for each word position in a stratified 4,793-passage subset of the scifi pulp magazine corpus (Amazing Stories, Astounding Stories, Galaxy Magazine, and others, 1926-2017), with WordPiece subwords merged back into whole words. Rownames are `word_position` (e.g. `"scientist_12"`, `"the_211"`), so the same word type has a different vector at each occurrence. A `passage_id` attribute (integer, 1-4800) maps each row to its source passage.

Format

A matrix of 2,093,785 rows and 768 columns (`_sub` is a list and `_cls` is 4,793 x 768; see Details)

Source

`bert-base-uncased` embeddings of a stratified subset of the scifi pulp magazine corpus in text2map.corpora

Details

Four quantization levels are available, trading file size for precision: `_fp16` (lossless half-precision, the baseline), `_int8`, `_int4` (near-identical quality to int8 at half the size), and `_binary` (1 bit/value, coarsest). By default `load_pretrained()` dequantizes `int8`/`int4`/`binary` back to a double matrix; pass `dequantize = FALSE` to get the raw compact integer matrix with its `scale`/`offset`/`quantized` attributes.

`vecs_bert768_scifi_pulp_sub` is a separate, much smaller model: a list (not a matrix) of the 4 quantization levels already dequantized to double, for a focal-word subset (`scientist`, `professor`, `discovery`, `experiment`, plus a human/artificial semantic-direction word set and a sampled `"the"` baseline) — useful for directly comparing quantization quality without downloading the full-size models. `load_pretrained()` returns it as-is (`dequantize` has no effect, since it's already double); it has elements `$fp16`, `$int8`, `$int4`, `$binary` (each an 8,758 x 768 matrix) and `$meta` (rownames, passage_id, word labels, and reference cosine-similarity statistics).

`vecs_bert768_scifi_pulp_cls` is a **passage-level** companion model: the `[CLS]` token's contextual vector for each of the 4,793 passages (one row per passage, 4,793 x 768, stored as double). Unlike the word-level matrices above, this gives a single summary embedding per passage, suitable for passage-level similarity/clustering/classification. A `passage_id` attribute (integer, 1-4793) maps each row to its source passage. `load_pretrained()` returns it as-is (`dequantize` has no effect, since it's already double).

Examples

if (FALSE) { # \dontrun{

## download the model (once per machine) -- int4 is the recommended default
download_pretrained("vecs_bert768_scifi_pulp_int4")

## load the model each session (dequantized to double by default)
wv <- load_pretrained("vecs_bert768_scifi_pulp_int4")
dim(wv) == c(2093785, 768)

## get the raw quantized integer matrix instead
wv_raw <- load_pretrained("vecs_bert768_scifi_pulp_int4", dequantize = FALSE)
attr(wv_raw, "quantized") == "int4"

## quantization-comparison subset (list of 4 levels + meta)
download_pretrained("vecs_bert768_scifi_pulp_sub")
sub <- load_pretrained("vecs_bert768_scifi_pulp_sub")
names(sub)

## passage-level [CLS] embeddings (one vector per passage)
download_pretrained("vecs_bert768_scifi_pulp_cls")
cls <- load_pretrained("vecs_bert768_scifi_pulp_cls")
dim(cls) == c(4793, 768)

} # }