Similarity metrics

The score of a pair compares the two Cell Ontology terms that the labels matched, not the labels themselves. Two labels that matched the same term score 1 under every metric. For different terms, metric decides how they are compared: by the embeddings of their descriptions, by their places in the ontology’s hierarchy, or by both.

df = await cyto.compare(
    author_labels=author_labels,
    algorithms=algorithms,
    run_id="lung_v1_path",
    metric="path",
)

In the hosted API, set the request fields metric and metricParams.

Metrics¶

The embedding metrics compare the terms’ descriptions, as written by the LLM of the model pair. The hierarchy metrics use cl.owl and depend only on the ontology. Depth is the longest chain of is_a parents from a term up to cell (CL:0000000). The cosine_kernel and cosine_direct scores depend on the model pair, so compare them only between runs with the same models.

Reference scores¶

These scores of some term pairs come from the precomputed embeddings of the default model pair, Kimi K2.6 with Qwen3-Embedding-8B, and from cl.owl:

The metrics answer different questions:

• cosine_direct varies little: across all 4.2 million pairs of the 2,902 terms, half of the cosine similarities lie between 0.38 and 0.46, and 95 % are below 0.56. cosine_kernel spreads them out: 76 % of all pairs score below 0.1, and fewer than 1 % score 0.5 or more, so related terms stand out from unrelated ones.

• The embedding metrics follow what the descriptions say. A broad term and its specific child, such as macrophage and alveolar macrophage, can score lower than two sibling terms.

• The hierarchy metrics follow the ontology’s structure. path gives siblings the score of identical terms, and weighted:num_ancestors scores T cell and B cell almost as high as identical terms.

Tune cosine_kernel¶

cosine_kernel maps the cosine similarity c of the two terms’ embeddings to a score with a Gaussian:

score = amplitude * exp(-((c - center) ** 2) / (2 * width**2))

center is 1.0 by default, width 0.25 and amplitude 1.0. Pass them in metric_params; other metrics ignore it:

df = await cyto.compare(
    author_labels=author_labels,
    algorithms=algorithms,
    run_id="lung_v1_wide",
    metric="cosine_kernel",
    metric_params={"width": 0.5},
)

The width sets how fast the score falls as the terms’ cosine similarity drops:

A narrow width rewards only near-identical terms; a wide one gives partial credit to more distant terms. Keep center and amplitude at 1: identical terms always score 1, whatever the amplitude.

The match threshold¶

min_match_similarity is the lowest cosine similarity between the embedding of a label’s description and the embedding of its closest term at which the label counts as matched. It is 0.1 by default, which is low enough for nearly every label to match some term. Raise it to leave weak matches unmatched. A pair of single-part labels with an unmatched label scores 0, and its similarity_method is partial_match or no_matches. In a compound pair, an unmatched part scores 0 against every part of the other label.

df = await cyto.compare(
    author_labels=author_labels,
    algorithms=algorithms,
    run_id="lung_v1_strict",
    min_match_similarity=0.5,
)

To choose a threshold, look at the author_embedding_similarity and algorithm_embedding_similarity columns of a run, which hold the similarity of each label with its closest term, whether or not it was matched.

Unknown metrics and missing terms¶

A metric name that CyteOnto does not know falls back to simple, and the error is logged for each pair. The hierarchy metrics also fall back to simple for a term that is not in cl.owl. If the scores look wrong, check the log for these messages.