Evaluate anchor sets in defining semantic relations

This function evaluates how well an anchor set defines a semantic relations using one of two methods: pairdir (which only evaluates semantic directions) or relco which evaluates semantic directions, semantic centroids and compound concepts). See details.

Usage

test_anchors(
  anchors,
  wv,
  non_anchors = NULL,
  method,
  all = FALSE,
  type = c("direction", "centroid", "compound"),
  conf = 0.95,
  dir_method = c("paired", "pooled", "L2", "PCA"),
  n_runs = 100,
  null = 0,
  alpha = 0.5,
  seed = NULL,
  order_non_anchors = FALSE,
  summarize = TRUE
)

Arguments

anchors: A data frame or list of 'anchor' terms
wv: Matrix of word embedding vectors (a.k.a embedding model) with rows as terms.
non_anchors: For 'relco', terms that are not anchors (random, unrelated, or distinctive terms).
method: Which metric used to evaluate, 'pairdir' or 'relco'
all: Logical (default FALSE). Whether to evaluate all possible pairwise combinations of two sets of anchors. If FALSE only the input pairs are used in evaluation and anchor sets must be of equal lengths.
type: For 'relco', indicate which kind of relation, "direction", "centroid", "compound"
conf: For 'relco', confidence interval
dir_method: For 'relco' and type = "direction", indicate the method for calculating direction ("paired", "pooled", "L2", "PCA"), See get_direction() for details.
n_runs: For 'relco', number of runs
null: For 'relco', null hypothesis, default is 0.
alpha: For 'relco', term selection weighting (default is 0.5)
seed: For 'relco', set sampling seed
order_non_anchors: Logical (default FALSE). For 'relco', if TRUE the subset of non-anchor terms is fixed across runs.
summarize: Logical (default TRUE). Returns a dataframe with AVERAGE scores for input pairs along with each pairs' contribution. If summarize = FALSE, returns a list with each offset matrix, each contribution, and the average score.

Value

dataframe or list

Details

PairDir evaluates how parallel two anchor sets are when used to define a semantic direction. According to Boutyline and Johnston (2023):

"We find that PairDir – a measure of parallelism between the offset vectors (and thus of the internal reliability of the estimated relation) – consistently outperforms other reliability metrics in explaining axis accuracy."

Boutyline and Johnston only consider analyst specified pairs. However, if all = TRUE, all pairwise combinations of terms between each set are evaluated. This can allow for unequal sets of anchors, however this increases computational complexity considerably.

Relco (anchor reliability coefficient) evaluates how well individual anchors index a given semantic relation in comparison to a set of non-anchor words. This can be used on semantic directions, semantic relations, or compound concepts. See Taylor et al (2025) for details; see also the CMDist() function.

References

Boutyline, Andrei, and Ethan Johnston. 2023. “Forging Better Axes: Evaluating and Improving the Measurement of Semantic Dimensions in Word Embeddings.” doi:10.31235/osf.io/576h3

Taylor, Marshall, et al. 2025. "A Simulation-Based Slope Metric for Anchor List Reliability in Word Embedding Spaces." doi:10.31235/osf.io/sc2ub_v3

Examples



# load example word embeddings
data(ft_wv_sample)

df_anchors <- data.frame(
  a = c("rest", "rested", "stay", "stand"),
  z = c("coming", "embarked", "fast", "move")
)

# test pairdir
test_anchors(df_anchors, ft_wv_sample, method = "pairdir")
#>       anchor_pair   pair_dir
#> 1         AVERAGE 0.13890810
#> 2     rest-coming 0.18960552
#> 3 rested-embarked 0.18302837
#> 4       stay-fast 0.10699562
#> 5      stand-move 0.07600288
test_anchors(df_anchors, ft_wv_sample, method = "pairdir", all = TRUE)
#>        anchor_pair  pair_dir
#> 1          AVERAGE 0.2748587
#> 2      rest-coming 0.3153744
#> 3    rested-coming 0.2752213
#> 4      stay-coming 0.2356302
#> 5     stand-coming 0.2242636
#> 6    rest-embarked 0.3004799
#> 7  rested-embarked 0.3048728
#> 8    stay-embarked 0.2208549
#> 9   stand-embarked 0.2094862
#> 10       rest-fast 0.3272416
#> 11     rested-fast 0.3054702
#> 12       stay-fast 0.3019808
#> 13      stand-fast 0.2737485
#> 14       rest-move 0.3153754
#> 15     rested-move 0.2671968
#> 16       stay-move 0.2791464
#> 17      stand-move 0.2413955

# test relco
non_anchors <- c("writ", "alloys", "ills", "atlas", "saturn", "cape", "unfolds")
## centroid
test_anchors(df_anchors[, 1], ft_wv_sample, method = "relco", 
             type = "centroid", non_anchors = non_anchors)
#> # Relation Type:                    centroid
#> # Global Reliability Coefficient:   0.3931
#> # 4 Highest Contributors:           rest, rested, stay, stand
#> # Confidence Interval (two-tailed): 0.3796 to 0.4066 at 95%
#> # t-test:                           t = 57.8954, df = 99, p-value = 0
#> # Alternative Hypothesis:           True global reliability coefficient > 0
#> # Term-Level Contributions:        
#>   term    mean lower_ci upper_ci
#>   <chr>  <dbl>    <dbl>    <dbl>
#> 1 rest   0.252    0.243    0.262
#> 2 rested 0.202    0.191    0.212
#> 3 stay   0.177    0.168    0.186
#> 4 stand  0.159    0.148    0.169
#> 
## compound
test_anchors(df_anchors$a, ft_wv_sample, method = "relco", 
             type = "compound", non_anchors = non_anchors)
#> # Relation Type:                    compound
#> # Global Reliability Coefficient:   0.4438
#> # 4 Highest Contributors:           rested, rest, stay, stand
#> # Confidence Interval (two-tailed): 0.4341 to 0.4534 at 95%
#> # t-test:                           t = 91.1538, df = 99, p-value = 0
#> # Alternative Hypothesis:           True global reliability coefficient > 0
#> # Term-Level Contributions:        
#>   term    mean lower_ci upper_ci
#>   <chr>  <dbl>    <dbl>    <dbl>
#> 1 rest   0.218    0.211    0.224
#> 2 rested 0.238    0.224    0.252
#> 3 stay   0.213    0.203    0.222
#> 4 stand  0.204    0.192    0.216
#> 
## direction
test_anchors(df_anchors, ft_wv_sample, method = "relco", 
             type = "direction", dir_method = "paired", 
             non_anchors = non_anchors)
#> # Relation Type:                    direction
#> # Global Reliability Coefficient:   0.2108
#> # 4 Highest Contributors (Pole 1):  rest, rested, stand, stay
#> # 4 Highest Contributors (Pole 2):  fast, move, coming, embarked
#> # Confidence Interval (two-tailed): 0.1981 to 0.2234 at 95%
#> # t-test:                           t = 33.0637, df = 99, p-value = 0
#> # Alternative Hypothesis:           True global reliability coefficient > 0
#> # Term-Level Contributions:        
#>   term        mean lower_ci upper_ci pole 
#>   <chr>      <dbl>    <dbl>    <dbl> <chr>
#> 1 rest      0.136    0.127   0.145   pole1
#> 2 rested    0.128    0.119   0.137   pole1
#> 3 stay      0.0566   0.0383  0.0749  pole1
#> 4 stand     0.0928   0.0749  0.111   pole1
#> 5 coming   -0.0295  -0.0335 -0.0256  pole2
#> 6 embarked -0.0147  -0.0221 -0.00735 pole2
#> 7 fast     -0.0327  -0.0358 -0.0296  pole2
#> 8 move     -0.0300  -0.0338 -0.0262  pole2
#> 
test_anchors(df_anchors, ft_wv_sample, method = "relco", 
             type = "direction", dir_method = "pooled", 
             non_anchors = non_anchors)
#> # Relation Type:                    direction
#> # Global Reliability Coefficient:   0.2158
#> # 4 Highest Contributors (Pole 1):  rest, rested, stand, stay
#> # 4 Highest Contributors (Pole 2):  fast, move, coming, embarked
#> # Confidence Interval (two-tailed): 0.2023 to 0.2293 at 95%
#> # t-test:                           t = 31.6549, df = 99, p-value = 0
#> # Alternative Hypothesis:           True global reliability coefficient > 0
#> # Term-Level Contributions:        
#>   term        mean lower_ci upper_ci pole 
#>   <chr>      <dbl>    <dbl>    <dbl> <chr>
#> 1 rest      0.130    0.115   0.144   pole1
#> 2 rested    0.101    0.0896  0.111   pole1
#> 3 stay      0.0409   0.0227  0.0590  pole1
#> 4 stand     0.0629   0.0449  0.0809  pole1
#> 5 coming   -0.0249  -0.0321 -0.0176  pole2
#> 6 embarked -0.0131  -0.0211 -0.00507 pole2
#> 7 fast     -0.0361  -0.0406 -0.0315  pole2
#> 8 move     -0.0264  -0.0314 -0.0215  pole2
#>