Evaluation Module¶
src/fed_maxfuse/evaluation/
Implements the two-track evaluation framework. Definitions and formulae: Metrics.
Layout¶
| Path | Role |
|---|---|
model_evaluation.py |
Evaluation orchestration |
metrics/supervised/classification.py |
Accuracy, label-based scores |
metrics/supervised/cluster.py |
ASW-F1, ARI-F1 |
metrics/supervised/local.py |
FOSCTTM, FOSKNN |
metrics/unsupervised/local.py |
MRRE |
metrics/unsupervised/cluster.py |
Steadiness & Cohesiveness |
metrics/unsupervised/global_distances.py |
Kruskal's Stress |
metrics/utils.py |
Shared helpers |
Metric selection¶
ZaduAbbreviationEnum maps to ZADU measurement names:
| Value | Metric |
|---|---|
TNC |
Trustworthiness & Continuity |
MRRE |
Mean Relative Rank Error |
LCMC |
Local Continuity Meta-Criteria |
NH |
Neighborhood Hit |
CA_TNC |
Class-aware Trustworthiness & Continuity |
L_TNC |
Label Trustworthiness & Continuity |
ND |
Neighbor Dissimilarity |
DTM |
Distance to Measure |
KL_DIV |
KL Divergence |
DSC |
Distance Consistency |
PR |
Pearson r |
Not all are used in the reported results. The thesis reports MRRE, Steadiness & Cohesiveness, and Stress alongside the supervised metrics.
Running evaluation¶
cli run evaluation DATASET_ID OUT-DATA-PATH -p ./data/preprocessed -at federated -v
cli run mrre DATASET_ID OUT-DATA-PATH -p ./data/preprocessed -at federated
Or inline with training via cli run train ... --evaluate.
Run evaluation as a separate job for large datasets
Unsupervised metrics need full distance and rank matrices, scaling \(O(n^2)\). Above 100,000 cells at 32-bit precision this required ≥ 37 GB. Training and evaluation have very different memory profiles and should be submitted separately.
Outputs¶
| File | Contents |
|---|---|
supervised_evaluation_metrics.json |
Accuracy, ASW-F1, ARI-F1, FOSCTTM, FOSKNN |
unsupervised_evaluation_metrics.json |
MRRE, Steadiness, Cohesiveness, Stress |
local_evaluation_metrics.json |
Pointwise values for CheckViz and Reliability Maps |
Interpreting results¶
- Directions: FOSCTTM ↓, FOSKNN ↑, MRRE ↓, Stress ↓, Steadiness ↑, Cohesiveness ↑, F1s ↑.
- Read Steadiness × Cohesiveness as a quadrant, not as two independent numbers: the off-diagonals are distinct diagnoses.
- Compare distributions, not single runs. MaxFuse is stochastic; the reported experiments use 25–250 repetitions per configuration and a Mann–Whitney U test at \(p<0.05\).