4-layer, 312-dim general-purpose TinyBERT

A distilled BERT encoder (4 transformer layers, hidden size 312, 12 attention heads, 30,522-token WordPiece vocabulary) for general-domain English. Unlike the vecs_* matrix models, this ships as a full model directory (config.json, pytorch_model.bin, vocab.txt, tokenizer.json, tokenizer_config.json) in HuggingFace `transformers` format, not a single R object.

Format

A directory of HuggingFace `transformers`-format model files

Source

General TinyBERT (2nd generation), 4 layers / 312 dim

Details

`load_pretrained()` does not read this into an R object. It returns a list with `$path` (the extracted model directory) and `$config` (parsed `config.json`). Load the model itself on the caller side, e.g. via `reticulate` + Python `transformers`.

Examples

if (FALSE) { # \dontrun{

## download the model (once per machine)
download_pretrained("llm_bert_tiny4l312d_general")

## load the model each session
bert <- load_pretrained("llm_bert_tiny4l312d_general")

bert$path
bert$config$hidden_size == 312

## load the actual model on the caller side, e.g.:
# reticulate::import("transformers")$AutoModel$from_pretrained(bert$path)

} # }