Evaluate anchor sets in defining semantic relations
Source:R/utils-embedding-vectors.R
test_anchors.RdThis function evaluates how well an anchor set defines a semantic relations using one of two methods: pairdir (which only evaluates semantic directions) or relco which evaluates semantic directions, semantic centroids and compound concepts). See details.
Arguments
- anchors
A data frame or list of 'anchor' terms
- wv
Matrix of word embedding vectors (a.k.a embedding model) with rows as terms.
- non_anchors
For 'relco', terms that are not anchors (random, unrelated, or distinctive terms).
- method
Which metric used to evaluate, 'pairdir' or 'relco'
- all
Logical (default
FALSE). Whether to evaluate all possible pairwise combinations of two sets of anchors. IfFALSEonly the input pairs are used in evaluation and anchor sets must be of equal lengths.- type
For 'relco', indicate which kind of relation, "direction", "centroid", "compound"
- conf
For 'relco', confidence interval
- dir_method
For 'relco' and
type = "direction", indicate the method for calculating direction ("paired", "pooled", "L2", "PCA"), Seeget_direction()for details.- n_runs
For 'relco', number of runs
- null
For 'relco', null hypothesis, default is 0.
- alpha
For 'relco', term selection weighting (default is 0.5)
- seed
For 'relco', set sampling seed
- order_non_anchors
Logical (default
FALSE). For 'relco', if TRUE the subset of non-anchor terms is fixed across runs.- summarize
Logical (default
TRUE). Returns a dataframe with AVERAGE scores for input pairs along with each pairs' contribution. Ifsummarize = FALSE, returns a list with each offset matrix, each contribution, and the average score.
Details
PairDir evaluates how parallel two anchor sets are when used to define a semantic direction. According to Boutyline and Johnston (2023):
"We find that PairDir – a measure of parallelism between the offset vectors (and thus of the internal reliability of the estimated relation) – consistently outperforms other reliability metrics in explaining axis accuracy."
Boutyline and Johnston only consider analyst specified pairs. However,
if all = TRUE, all pairwise combinations of terms between each set
are evaluated. This can allow for unequal sets of anchors, however this
increases computational complexity considerably.
Relco (anchor reliability coefficient) evaluates how well individual anchors
index a given semantic relation in comparison to a set of non-anchor words.
This can be used on semantic directions, semantic relations, or compound concepts.
See Taylor et al (2025) for details; see also the CMDist() function.
References
Boutyline, Andrei, and Ethan Johnston. 2023. “Forging Better Axes: Evaluating and Improving the Measurement of Semantic Dimensions in Word Embeddings.” doi:10.31235/osf.io/576h3
Taylor, Marshall, et al. 2025. "A Simulation-Based Slope Metric for Anchor List Reliability in Word Embedding Spaces." doi:10.31235/osf.io/sc2ub_v3
Examples
# load example word embeddings
data(ft_wv_sample)
df_anchors <- data.frame(
a = c("rest", "rested", "stay", "stand"),
z = c("coming", "embarked", "fast", "move")
)
# test pairdir
test_anchors(df_anchors, ft_wv_sample, method = "pairdir")
#> anchor_pair pair_dir
#> 1 AVERAGE 0.13890810
#> 2 rest-coming 0.18960552
#> 3 rested-embarked 0.18302837
#> 4 stay-fast 0.10699562
#> 5 stand-move 0.07600288
test_anchors(df_anchors, ft_wv_sample, method = "pairdir", all = TRUE)
#> anchor_pair pair_dir
#> 1 AVERAGE 0.2748587
#> 2 rest-coming 0.3153744
#> 3 rested-coming 0.2752213
#> 4 stay-coming 0.2356302
#> 5 stand-coming 0.2242636
#> 6 rest-embarked 0.3004799
#> 7 rested-embarked 0.3048728
#> 8 stay-embarked 0.2208549
#> 9 stand-embarked 0.2094862
#> 10 rest-fast 0.3272416
#> 11 rested-fast 0.3054702
#> 12 stay-fast 0.3019808
#> 13 stand-fast 0.2737485
#> 14 rest-move 0.3153754
#> 15 rested-move 0.2671968
#> 16 stay-move 0.2791464
#> 17 stand-move 0.2413955
# test relco
non_anchors <- c("writ", "alloys", "ills", "atlas", "saturn", "cape", "unfolds")
## centroid
test_anchors(df_anchors[, 1], ft_wv_sample, method = "relco",
type = "centroid", non_anchors = non_anchors)
#> # A data frame: 4 × 4
#> term mean lower_ci upper_ci
#> <chr> <dbl> <dbl> <dbl>
#> 1 rest 0.319 0.314 0.323
#> 2 rested 0.308 0.306 0.311
#> 3 stay 0.270 0.263 0.276
#> 4 stand 0.276 0.271 0.281
## compound
test_anchors(df_anchors$a, ft_wv_sample, method = "relco",
type = "compound", non_anchors = non_anchors)
#> # A data frame: 4 × 4
#> term mean lower_ci upper_ci
#> <chr> <dbl> <dbl> <dbl>
#> 1 rest 0.280 0.274 0.285
#> 2 rested 0.290 0.279 0.301
#> 3 stay 0.265 0.260 0.271
#> 4 stand 0.255 0.248 0.262
## direction
test_anchors(df_anchors, ft_wv_sample, method = "relco",
type = "direction", dir_method = "paired",
non_anchors = non_anchors)
#> # A data frame: 8 × 5
#> term mean lower_ci upper_ci pole
#> <chr> <dbl> <dbl> <dbl> <chr>
#> 1 rest 0.130 0.113 0.147 pole1
#> 2 rested 0.133 0.126 0.140 pole1
#> 3 stay 0.0566 0.0409 0.0723 pole1
#> 4 stand 0.0856 0.0694 0.102 pole1
#> 5 coming -0.0301 -0.0345 -0.0258 pole2
#> 6 embarked -0.0180 -0.0252 -0.0108 pole2
#> 7 fast -0.0327 -0.0365 -0.0289 pole2
#> 8 move -0.0308 -0.0338 -0.0277 pole2
test_anchors(df_anchors, ft_wv_sample, method = "relco",
type = "direction", dir_method = "pooled",
non_anchors = non_anchors)
#> # A data frame: 8 × 5
#> term mean lower_ci upper_ci pole
#> <chr> <dbl> <dbl> <dbl> <chr>
#> 1 rest 0.129 0.115 0.143 pole1
#> 2 rested 0.136 0.127 0.145 pole1
#> 3 stay 0.0642 0.0490 0.0794 pole1
#> 4 stand 0.0997 0.0862 0.113 pole1
#> 5 coming -0.0296 -0.0330 -0.0262 pole2
#> 6 embarked -0.0129 -0.0210 -0.00494 pole2
#> 7 fast -0.0321 -0.0360 -0.0282 pole2
#> 8 move -0.0267 -0.0311 -0.0223 pole2