Evaluation Framework¶
Diagonal integration has no anchors to validate against, so quality is measured from two independent directions.
Why two tracks¶
| Track | Uses | Strength | Weakness |
|---|---|---|---|
| Supervised | Cell labels and known 1–1 correspondences | Direct, interpretable | Vulnerable to erroneous or inconsistently-applied labels across datasets |
| Unsupervised | Embedding structure vs the original data | No labels needed; the only option in real deployments | Harder to interpret |
Supervised metrics answer "did it match the right cells?"; unsupervised metrics answer "did the embedding preserve the data's structure?" A method can pass one and fail the other, so both are required.
The local / cluster / global triad¶
For the unsupervised track, pick one metric per structural scale:
| Scale | Question | Metric |
|---|---|---|
| Local | Are point-to-point neighbourhood relationships preserved? | MRRE |
| Cluster | Do same-cluster points stay together? | Steadiness & Cohesiveness |
| Global | Is overall pairwise distance structure retained? | Kruskal's Stress |
Covering one tier is not evaluation. A method can ace local structure and destroy global structure, and only the triad exposes it.
Experimental design¶
Four experiment types, each isolating one variable:
| Experiment | Configurations × repetitions | Question |
|---|---|---|
| Centralized vs Federated | 1 × 250 each | Does federation cost accuracy? |
| Centralized batching | 10 × 25 | Can MaxFuse extend beyond two participants? |
| Shared feature selection | 11 × 25 | How thin can the linked-feature set get? |
| CCA random initialisation | 11 × 25 | Does the privacy noise cost accuracy? |
2,600 runs total, 1,300 per dataset.
Statistical testing¶
Differences are assessed with the Mann–Whitney U test at \(p < 0.05\).
The choices are deliberate:
- Non-parametric. No distributional assumption is made about metric values.
- \(p < 0.05\). Chosen to reflect the exploratory nature of the work, the applied model-comparison context, and MaxFuse's intrinsic stochasticity. A stricter threshold would be inappropriate here.
- 25–250 repetitions per configuration. These exist specifically to average over that stochasticity. Leiden clustering, k-NN graph construction, random batch assignment, and the CCA noise initialisation all vary between runs.
One-knob sweep discipline
Vary exactly one configuration key and freeze everything else. Include the undesirable
baseline (for example randomness: 0.0) so the sweep quantifies what avoiding it costs. This
is what makes an effect attributable rather than merely observed.
The evaluation problem in federated settings¶
This is the deepest limitation of the work, and worth stating plainly.
Most validation metrics require complete embeddings and explicit cell-level correspondences in one place, precisely what federation exists to avoid. Benchmark datasets like CITE-seq/PBMC permit supervised evaluation because ground truth exists and can be pooled for scoring. Practical federated deployments cannot do that.
Transmitting full embeddings to a central node for evaluation would compromise the privacy the method is designed to provide. Developing metrics that assess alignment quality across silos without full data sharing is an open research problem, and arguably the strongest direction arising from this work.
Memory constraints¶
The unsupervised metrics are implemented via the ZADU package and rely on full distance matrices, distance-rank matrices, or zero-dominated sparse matrices.
| Condition | Requirement |
|---|---|
| >100,000 cells, 32-bit floats | ≥ 37 GB |
| Scaling | \(O(n^2)\) with dataset size |
| Affected | MRRE, Steadiness, Cohesiveness |
This capped what could be evaluated on the Tonsils dataset and is why training and evaluation run as separate jobs. More memory-efficient implementations of these metrics would be a valuable contribution independent of MaxFuse.
Reproducibility practices¶
| Practice | Implementation |
|---|---|
| Configuration generation | Master JSON with parameter lists → auto-expanded per-run configs |
| Run identity | UUID + scheduler-generated ID |
| Incremental writes | Six artifacts per run, written as produced so a crash still leaves usable data |
| Environment | Exactly pinned dependencies; containerised via Docker → GHCR → Apptainer |
| Job separation | Training and evaluation submitted as distinct jobs (different memory profiles) |
Every published number traces back to an exact parameter set and an exact run ID.