Preprocessing Module¶
src/fed_maxfuse/data/ and src/fed_maxfuse/processors/
Produces the four matrices MaxFuse requires: active and shared, per modality. See Local Preprocessing for the conceptual pipeline.
Modules¶
| Module | Responsibility |
|---|---|
data/pre_processor.py |
The preprocessing pipeline |
data/pre_processed_data_loader.py |
Load prepared data from disk |
data/graph.py |
k-NN graph construction, Leiden clustering |
data/singular_value_decomposition.py |
SVD denoising and embedding |
data/synth_generation.py |
Synthetic data for testing |
data/utils.py, data/constants.py |
Helpers and defaults |
processors/anndata_loader.py |
AnnData ingestion |
processors/data_prepare.py |
Preparation orchestration |
utils/rna_protein_constant.py |
RNA ↔ protein name mapping |
Pipeline order¶
- Build shared feature sets from biological correspondences
- Variability filtering (shared)
- Normalise by target total
log(X+1)- HVG selection (RNA-type active data only)
- Scale to zero mean, unit variance
- Variability filtering (active)
Feature-name conversion¶
utils/rna_protein_constant.py maps protein marker names to RNA gene names. The mapping is
one-to-many: CD8 → CD8A, CD8B; CD3 → CD3D, CD3E. This is why 46 CODEX proteins expand
to 53 gene names before variability filtering reduces them to 32.
Mappings are tissue-dependent
The shipped mapping is immune/PBMC-oriented. Other tissues will need additions. Broad-spectrum antibodies (for example PanCK) have no clean single-gene correspondence and are best excluded rather than mapped approximately: random or wrong correspondences degrade integration severely. See Performance Highlights.
CLI¶
cli preprocess data DATASET-ID --out PATH
cli preprocess info DATASET-ID
cli preprocess labels DATASET-ID
cli preprocess distances DATASET-ID --verbose
Data contracts¶
| Requirement | Detail |
|---|---|
| Orientation | cells × features |
| Shared column counts | Identical across modalities, positionally paired |
| Row order | Shared and active matrices must describe the same cells in the same order |
| Density | Dense arrays for the matching path |
| Normalisation | Applied by the pipeline; do not double-normalise pre-normalised inputs (CODEX arrives already scaled) |