Metrics¶

Every metric used, what it measures, and which direction is good.

Memorise the directions before reading any plot

Lower is better: FOSCTTM, MRRE, Stress. Higher is better: FOSKNN, Accuracy, ASW-F1, ARI-F1, Steadiness, Cohesiveness.

Supervised metrics¶

FOSCTTM — Fraction Of Samples Closer Than True Match¶

The proportion of cells that sit closer to a given cell than its true partner does. Lower is better; 0 is perfect.

\[ \begin{split} &\texttt{FOSCTTM}(X, Y, N, (x_{\texttt{match}}, y_{\texttt{match}})) = \\ &\frac{1}{2N}\left(\sum_{x \in X}\frac{\sum_{y \in Y}\mathbb{I}(\|x - y\|_2 < \|x - y_{\texttt{match}}\|_2)}{N}\right. + \\ &\left. \sum_{y \in Y}\frac{\sum_{x \in X}\mathbb{I}(\|y - x\|_2 < \|y - x_{\texttt{match}}\|_2)}{N}\right) \end{split} \]

It counts intruders: how many wrong cells lie between you and your true partner, in both directions. Zero intruders either way gives 0.

Requires ground-truth 1–1 correspondences, so it is available for the Antibodies benchmark only.

FOSKNN — Fraction Of Samples in k-Nearest Neighbours¶

Is the true counterpart within each cell's \(k\) nearest neighbours, bidirectionally? Higher is better.

\[ \texttt{FOSKNN}(Y, Z, N, k) = \frac{1}{2N} \left(\sum_{i=1}^{N} \mathbb{I}(z_i \in \text{kNN}(y_i, Z, k)) + \sum_{i=1}^{N} \mathbb{I}(y_i \in \text{kNN}(z_i, Y, k))\right) \]

\(k\) is conventionally 5% of dataset size; the results plot it sweeping from 1% to 5%.

Accuracy¶

The percentage of labels that match after alignment: how well cell-type annotations from different modalities land on the same cell type. Available at multiple granularities: Levels 1–3 (8, 31, 58 labels) for Antibodies, Cluster Info/Term (6 labels) for Tonsils.

ASW-F1 and ARI-F1¶

Clustering-quality F1 scores that balance batch mixing against cell-type separation:

\[ \text{Label-F1} = \frac{2 \cdot (1 - B_{\texttt{norm}}) \cdot C_{\texttt{norm}}}{(1 - B_{\texttt{norm}}) + C_{\texttt{norm}}} \]

where \(B_{\texttt{norm}}\) scores clustering against modality labels and \(C_{\texttt{norm}}\) against cell-type labels, both normalised to \([0,1]\) and median-aggregated over subsamples.

The harmonic mean forces both objectives simultaneously:

\(B_{\texttt{norm}} \to 0\) means you cannot tell which modality a cell came from: batch effect removed. Hence the \((1 - B_{\texttt{norm}})\) term.
\(C_{\texttt{norm}} \to 1\) means cell types form clean, recoverable clusters.

A degenerate embedding that collapses everything to one point scores perfectly on batch mixing and zero on cell types, giving F1 → 0. One that keeps modalities completely separate fails the other way. Only genuine integration scores well, which is exactly the objective.

Components: ASW (Average Silhouette Width, range \([-1,1]\); ~0 means cluster overlap) and ARI (Adjusted Rand Index, range \([-0.5,1]\); 0 = random labelling, 1 = perfect).

Unsupervised metrics¶

MRRE — Mean Relative Rank Error (local)¶

\[ \text{MRRE} =\frac{1}{n} \sum_{i=1}^{n} \left( \frac{1}{c} \sum_{j \in N_i^{\text{target}}} \frac{|R_i^{\text{base}}(j) - R_i^{\text{target}}(j)|}{R_i^{\text{target}}(j)} \right) \]

with normalising constant \(c = \sum_{r=1}^{k} \frac{|n - 2r + 1|}{r}\).

Two directional variants:

Variant	Base → Target	Detects
MRRE false	original → embedding	Neighbours wrongly introduced (false positives)
MRRE missing	embedding → original	True neighbours lost (false negatives)

MRRE is rank-based, not top-\(k\) membership. It catches a neighbour moving from 5th to 15th position, a change a simple overlap metric would miss entirely. That sensitivity is why it is the local-scale metric of choice.

Steadiness & Cohesiveness (cluster)¶

Read these as a quadrant, never as two independent numbers.

	High Cohesiveness	Low Cohesiveness
High Steadiness	✅ Reliable — both spaces preserve each other	⚠️ Missing structures — real clusters not captured in the embedding
Low Steadiness	⚠️ False-positive structures — the embedding invented clusters	❌ Severe fidelity loss in both directions

Steadiness — inter-cluster reliability inside the embedding: how well embedding clusters reflect the original data.
Cohesiveness — inter-cluster reliability in the original data: how well original clusters reflect the embedding.

Both range \([0,1]\) with 1 optimal, and use HDBSCAN for clustering, chosen because it handles noise, produces clusters of varying densities, and its hierarchical structure permits examination at multiple levels.

The off-diagonal cells are genuinely different diagnoses with different fixes, which is why reporting them separately loses information. In the results, the meta-modality shows low cohesiveness: MaxFuse invented cluster structure absent from the original data, traceable to Leiden summarisation.

Kruskal's Stress (global)¶

\[ \text{Stress} = \sqrt{ \frac{ \sum_{i<j} \left(d_{i,j}^{\text{high}} - \alpha \cdot d_{i,j}^{\text{low}} \right)^2 }{ \sum_{i<j} \left(d_{i,j}^{\text{high}} \right)^2 } }, \qquad \alpha = \frac{ \sum_{i<j} d_{i,j}^{\text{high}} \cdot d_{i,j}^{\text{low}} }{ \sum_{i<j} \left( d_{i,j}^{\text{low}} \right)^2 } \]

Measures discrepancy between pairwise distances in the original and embedded spaces. 0 is perfect. The scaling factor \(\alpha\) makes it scale-invariant: an embedding uniformly three times smaller is not penalised, only genuine distortion is.

Visualisation¶

CheckViz and Reliability Maps visualise Steadiness & Cohesiveness distortions spatially. CheckViz highlights tears and false neighbourhoods; the Reliability Map encodes pointwise distortion through edge-based colouring. Both are produced from local_evaluation_metrics.json.

Summary table¶

Metric	Type	Scale	Direction	Needs ground-truth matching?
FOSCTTM	Supervised	local	↓	Yes
FOSKNN	Supervised	local	↑	Yes
Accuracy	Supervised	—	↑	No (labels only)
ASW-F1	Supervised	cluster	↑	No (labels only)
ARI-F1	Supervised	cluster	↑	No (labels only)
MRRE (false/missing)	Unsupervised	local	↓	No
Steadiness	Unsupervised	cluster	↑	No
Cohesiveness	Unsupervised	cluster	↑	No
Stress	Unsupervised	global	↓	No