GPTomics/bio-spatial-transcriptomics-spatial-domains

Version Compatibility

Reference examples tested with: squidpy 1.4+, scanpy 1.10+, anndata 0.10+, scikit-learn 1.4+; BANKSY (banksy_py / R Banksy), BayesSpace 1.12+ (R), STAGATE/GraphST optional

Before using code patterns, verify installed versions match. If versions differ:

• Python: pip show <package> then help(module.function) to check signatures
• R: packageVersion('<pkg>') then ?function_name to verify parameters

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Spatial Domain Detection

"Identify tissue domains in my section" -> Partition the tissue into spatially contiguous regions of homogeneous expression/composition, using BOTH transcriptional similarity AND spatial proximity, so the output is a region label per spot/cell that is spatially coherent.

• Python: squidpy.gr.spatial_neighbors for the graph, then a neighbor-augmented or graph-based domain method (BANKSY, STAGATE, GraphST)
• R: BayesSpace (spatialCluster), BASS, or Banksy

Governing Principle

A spatial domain is a REGION, not a cell type, and not a niche -- conflating the three is the central conceptual error of this analysis. A CELL TYPE is one cell's transcriptional identity. A NICHE (cellular neighborhood) is the local cell-type COMPOSITION around a cell -- which types co-occur. A SPATIAL DOMAIN is a contiguous tissue region (a cortical layer, tumor core, stromal band) that contains MANY cell types and several niches. If the question is "which cell types co-occur," that is a niche question and belongs to neighborhood-enrichment analysis (spatial-statistics), NOT domain segmentation; running the wrong one answers the wrong question.

Domain methods exist because plain Leiden/Louvain on expression alone ignores coordinates and produces salt-and-pepper, spatially-incoherent labels. Every domain method deliberately adds a spatial term so neighbors tend to share a label. The load-bearing decision is therefore NOT the clustering algorithm -- it is the SPATIAL-WEIGHT knob (BANKSY lambda, BayesSpace smoothing, SpaGCN histology weight, GNN graph radius). Too much spatial weight is the #1 trap, over-smoothing: it erases real boundaries and merges biologically distinct regions into blobs. Too little reverts to salt-and-pepper. The number of domains k is the second decision, and it is a BIOLOGICAL choice (how fine a regionalization the question needs), not a statistic to optimize against a silhouette score -- report k, justify it, and show sensitivity to k+-1.

Domain vs Niche vs Cell Type

| Concept | What it is | Unit | Right tool | |---------|-----------|------|-----------| | Cell type | One cell's transcriptional identity (lineage/state) | a cell | clustering + markers (single-cell/clustering) | | Niche / cellular neighborhood | Local cell-type composition -- which types co-occur around a cell | a neighborhood | neighborhood enrichment / co-occurrence (spatial-statistics) | | Spatial domain | Contiguous tissue region of homogeneous expression, containing many types | a region | domain segmentation (this skill) |

A domain can contain several niches; a niche can span domain boundaries. BANKSY and BASS switch between cell-typing and domain detection via a single parameter, which underscores that these are different outputs of related machinery, not the same task.

Domain Methods by Mechanism

The mechanism for "using the neighbors" is the axis that separates the methods. No method is a universal winner (Yuan 2024 Nat Methods); selection is scenario-specific.

| Method | Spatial mechanism | Needs k? | Best when | Fails when | |--------|-------------------|----------|-----------|------------| | BayesSpace | MRF (Potts) smoothing prior on a t-mixture in PCA space; also enhances sub-spot resolution | yes (q) | Visium laminar/continuous tissue; want sub-spot enhancement | non-contiguous regions; MCMC is slow; smoothing strength is a fixed prior | | BASS | Bayesian hierarchical: cell-type AND domain jointly, Potts prior, multi-sample | yes (C, D) | joint cell-type + domain, multi-sample, low-continuity tissue | heavier to run; needs both counts set | | STAGATE | Graph attention autoencoder; learns per-edge weights, reconstructs from a spatially-smoothed latent | embedding free; downstream k for mclust | continuous tissue; attention down-weights cross-boundary edges (fights over-smoothing); scales to Slide-seq/Stereo-seq | black-box embedding; sensitive to graph radius | | GraphST | Graph self-supervised contrastive learning | yes (mclust) | low-res Visium; also does integration + deconvolution | sensitive to graph construction | | SpaGCN | GCN fusing expression + coordinates + histology RGB | searches resolution to hit target count | H&E histology is informative | needs registered histology | | SEDR | Masked autoencoder + variational graph autoencoder | yes (mclust) | Visium/Slide-seq/Stereo-seq, robust on DLPFC | graph-construction sensitivity | | BANKSY | Neighbor-AUGMENTED features: own + neighborhood-mean + azimuthal Gabor; then Leiden/k-means | no fixed k | high-res imaging AND sequencing; unifies cell typing (lambda~0.2) and domains (lambda~0.8); very scalable; transparent | lambda mis-set collapses the task it solves | | UTAG | Message passing: multiply features by normalized adjacency (one-hop average), then Leiden | no fixed k | multiplexed imaging/proteomics (IMC, CODEX, MIBI); fast | one-hop smoothing only | | stLearn (SME) | Histology-CNN-weighted smoothing of each spot's expression, then clustering | downstream Louvain/k-means | H&E available and well-registered | only as good as image registration |

Mechanism families: MRF/Bayesian smoothing (BayesSpace, BASS, PRECAST) vs graph neural net (STAGATE, GraphST, SpaGCN, SEDR) vs neighbor-augmentation (BANKSY, UTAG) vs histology-guided (stLearn, SpaGCN). Benchmarks evolve fast -- verify the current verdict for the tissue geometry before committing.

Read the Benchmark With the DLPFC Caveat

The standard ground truth is DLPFC (Maynard 2021 Nat Neurosci 24:425-436): 12 Visium sections of human dorsolateral prefrontal cortex, manually annotated into 6 cortical layers + white matter, scored by ARI. The Yuan 2024 (Nat Methods 21:712-722) benchmark of 13 methods x 34 datasets concludes there is NO single winner -- methods are complementary across accuracy, spatial continuity, marker detection, scalability, and robustness. On DLPFC, GNN/graph methods (STAGATE, SEDR, DeepST) and BayesSpace are among the most robust, and a technology-stratified benchmark (Chen 2025 iMeta 4:e70084) finds STAGATE/GraphST best on low-resolution Visium while BASS/stLearn/BANKSY lead on high-resolution platforms.

The caveat: DLPFC is a CONTINUOUS, LAMINAR tissue, which flatters smoothing-friendly methods. Do not over-generalize these rankings to non-laminar tissue. ALL methods struggle with NON-CONTIGUOUS domains (the same region appearing in separated patches -- scattered tumor nests, immune aggregates) because the spatial prior assumes contiguity; for non-contiguous biology, lower the spatial weight or switch to a niche/neighborhood analysis rather than domain segmentation.

Build the Spatial Graph

Goal: Construct the spatial neighbor graph that every domain method inherits.

Approach: Use a hex lattice for Visium (6 neighbors) and a generic kNN/Delaunay graph for imaging point clouds; the graph radius IS a spatial-weight knob (too dense over-smooths, too sparse fragments).

import squidpy as sq
import scanpy as sc

adata = sc.read_h5ad('preprocessed.h5ad')

# Visium hex lattice: 6 immediate neighbors. For imaging point clouds use
# coord_type='generic' with KNN or Delaunay. Build in microns where possible --
# a graph built in pixels has a different radius than one built in microns.
sq.gr.spatial_neighbors(adata, coord_type='grid', n_neighs=6)

Expression-Only Clustering Is the Salt-and-Pepper Baseline

Goal: Show why a domain method is needed at all.

Approach: Cluster on expression PCA with no spatial term; the result is spatially incoherent and demonstrates the failure domain methods correct.

sc.pp.neighbors(adata, n_neighbors=15, n_pcs=30)
sc.tl.leiden(adata, resolution=0.5, key_added='expr_leiden',
             flavor='igraph', n_iterations=2, directed=False)

# Plot on tissue: speckled, no contiguous regions -- this is the baseline a
# domain method is built to beat, not a domain result.
sq.pl.spatial_scatter(adata, color='expr_leiden')

Neighbor-Augmented Domains (BANKSY-style) and the Spatial-Weight Knob

Goal: Produce spatially coherent domains and make the over-smoothing knob explicit.

Approach: BANKSY concatenates each cell's own expression with its neighborhood-mean expression, mixed by lambda; lambda~0.2 yields cell typing, lambda~0.8 yields domains. A transparent neighbor-augmented matrix reproduces the idea with Squidpy when the BANKSY package is unavailable -- the lambda here is the load-bearing decision, not the clustering call.

import numpy as np
from sklearn.preprocessing import normalize

# Mean expression over each spot's spatial neighbors (the neighborhood signal).
W = normalize(adata.obsp['spatial_connectivities'], norm='l1', axis=1)
neighbor_mean = W @ adata.obsm['X_pca']

# lambda is the spatial-weight knob: ~0.8 for domains, ~0.2 for cell typing.
# Too high -> blobs (boundaries erased); too low -> salt-and-pepper.
lam = 0.8
augmented = np.concatenate(
    [np.sqrt(1 - lam) * adata.obsm['X_pca'], np.sqrt(lam) * neighbor_mean], axis=1)
adata.obsm['X_banksy'] = augmented

sc.pp.neighbors(adata, use_rep='X_banksy', key_added='banksy')
sc.tl.leiden(adata, resolution=0.5, key_added='domains', neighbors_key='banksy',
             flavor='igraph', n_iterations=2, directed=False)
sq.pl.spatial_scatter(adata, color='domains')

If the BANKSY package is installed, prefer it: import banksy_py (Python) or the R Banksy Bioconductor package compute the own + neighborhood-mean + azimuthal Gabor (AGF) features and expose lambda directly.

Sweep the Spatial Weight to Find the Over-Smoothing Edge

Goal: Locate the lambda where boundaries are sharp but regions stay contiguous.

Approach: Recompute domains across a lambda grid and inspect boundaries against histology; the right lambda is the largest value before distinct regions merge into blobs, judged visually, not by an internal score.

for lam in [0.2, 0.5, 0.8]:
    aug = np.concatenate(
        [np.sqrt(1 - lam) * adata.obsm['X_pca'],
         np.sqrt(lam) * (W @ adata.obsm['X_pca'])], axis=1)
    adata.obsm['X_banksy'] = aug
    sc.pp.neighbors(adata, use_rep='X_banksy', key_added='banksy')
    sc.tl.leiden(adata, resolution=0.5, key_added=f'domains_l{lam}',
                 neighbors_key='banksy', flavor='igraph', n_iterations=2, directed=False)

# Low lambda -> speckled; high lambda -> over-smoothed blobs. Pick by eye
# against known tissue architecture, not by silhouette.
sq.pl.spatial_scatter(adata, color=['domains_l0.2', 'domains_l0.5', 'domains_l0.8'])

Choose k as a Biological Decision, With k+-1 Sensitivity

Goal: Set the number of domains to the regionalization the question needs and show the answer is not fragile to it.

Approach: When a method takes k directly (BayesSpace q, mclust on a GNN embedding), run k, k-1, k+1 and report all three; do not silently optimize k against a clustering score, which has no biological ground truth.

from sklearn.mixture import GaussianMixture

for k in [6, 7, 8]:  # DLPFC has 6 layers + white matter -> k near 7
    gm = GaussianMixture(n_components=k, covariance_type='full', random_state=0)
    adata.obs[f'domains_k{k}'] = gm.fit_predict(adata.obsm['X_banksy']).astype(str)

# Report k+-1 side by side; a domain that only appears at one k is a weak claim.
sq.pl.spatial_scatter(adata, color=['domains_k6', 'domains_k7', 'domains_k8'])

BayesSpace for Laminar Tissue (R)

Goal: Apply an MRF smoothing prior, the robust choice on continuous Visium tissue.

Approach: Preprocess, then spatialCluster with q domains; the smoothing prior couples neighboring spots. Run in R and import the labels.

library(BayesSpace)

sce <- readRDS('sce.rds')
sce <- spatialPreprocess(sce, platform = 'Visium', n.PCs = 15)
# q is the biological k; nrep is MCMC iterations. spatialEnhance() can split
# spots into subspots for higher resolution after spatialCluster().
sce <- spatialCluster(sce, q = 7, nrep = 10000)
write.csv(data.frame(barcode = colnames(sce), domain = sce$spatial.cluster),
          'bayesspace_domains.csv', row.names = FALSE)

STAGATE for High-Resolution or Large Sections (optional)

Goal: Learn a spatially-aware embedding whose attention down-weights cross-boundary edges.

Approach: Build the STAGATE radius graph, train the graph-attention autoencoder, then cluster the embedding; set the radius to the over-smoothing knob.

import STAGATE  # optional dependency; pip install STAGATE_pyG or use the TF build

STAGATE.Cal_Spatial_Net(adata, rad_cutoff=150)  # rad_cutoff is the graph radius knob
STAGATE.Stats_Spatial_Net(adata)
adata = STAGATE.train_STAGATE(adata)

sc.pp.neighbors(adata, use_rep='STAGATE')
sc.tl.leiden(adata, resolution=0.5, key_added='stagate_domains',
             flavor='igraph', n_iterations=2, directed=False)

Name Domains From Markers

Goal: Attach anatomical labels to spatially coherent clusters.

Approach: Rank per-domain markers, then map cluster IDs to region names; marker p-values from the same data that defined the domains are for ranking and labeling only, not inference (the double-dipping caveat from single-cell/clustering).

sc.tl.rank_genes_groups(adata, groupby='domains', method='wilcoxon')
markers = sc.get.rank_genes_groups_df(adata, group=None)
print(markers.groupby('group').head(5))

domain_names = {'0': 'White matter', '1': 'Layer 1', '2': 'Layer 2/3'}
adata.obs['region'] = adata.obs['domains'].map(domain_names)
sq.pl.spatial_scatter(adata, color='region')

Common Errors

| Symptom | Cause | Fix | |---------|-------|-----| | Speckled, spatially incoherent labels | No spatial term (plain Leiden on expression) or spatial weight too low | Use a domain method; raise lambda / smoothing / graph density | | Regions merged into smooth blobs, boundaries gone | Over-smoothing -- spatial weight too high or graph radius too large | Lower lambda / smoothing strength / rad_cutoff; check boundaries against histology | | Scattered tumor nests collapse into one domain or vanish | Non-contiguous domain; spatial prior assumes contiguity (Yuan 2024) | Lower the spatial weight, or switch to niche/neighborhood analysis (spatial-statistics) | | Domain count feels arbitrary / reviewer questions k | k optimized against a silhouette score instead of chosen biologically | Set k from the biology; report k+-1 sensitivity; justify the regionalization | | Domains track a single sample/section | Batch confounded with biology in multi-sample data | Use a multi-sample method (BASS, PRECAST, GraphST integration) or integrate first | | Answer changes with no parameter change | Spatial graph rebuilt with different units (pixels vs microns) or different kNN/Delaunay | Pin the graph construction and coordinate units; build once, reuse | | "Domain" answer to a "which types co-occur" question | Domain segmentation used for a niche question | Use neighborhood enrichment / co-occurrence (spatial-statistics) instead | | Marker p-values quoted as proof a domain is real | Double-dipping (testing the clustering that defined the groups) | Use markers for ranking/labeling only; validate regions against known architecture |

Related Skills

• spatial-neighbors - Build and tune the spatial graph every domain method inherits
• spatial-statistics - Niche/neighborhood enrichment and co-occurrence when the question is which cell types co-occur, not which region this is
• spatial-deconvolution - Per-spot cell-type composition; domains are regions, deconvolution is composition within a spot
• single-cell/clustering - Non-spatial clustering, resolution sweeps, and the double-dipping caveat on post-clustering marker tests

References

• Zhao et al. (2021). Spatial transcriptomics at subspot resolution with BayesSpace. Nat Biotechnol 39:1375-1384.
• Hu et al. (2021). SpaGCN: integrating gene expression, spatial location and histology to identify spatial domains and SVGs by graph convolutional network. Nat Methods 18:1342-1351.
• Dong & Zhang (2022). Deciphering spatial domains from spatially resolved transcriptomics with an adaptive graph attention auto-encoder (STAGATE). Nat Commun 13:1739.
• Long et al. (2023). Spatially informed clustering, integration, and deconvolution of spatial transcriptomics with GraphST. Nat Commun 14:1155.
• Singhal et al. (2024). BANKSY unifies cell typing and tissue domain segmentation for scalable spatial omics data analysis. Nat Genet 56:431-441.
• Kim et al. (2022). Unsupervised discovery of tissue architecture in multiplexed imaging (UTAG). Nat Methods 19:1653-1661.
• Xu et al. (2024). Unsupervised spatially embedded deep representation of spatial transcriptomics (SEDR). Genome Med 16:12.
• Li & Zhou (2022). BASS: multi-scale and multi-sample analysis enables accurate cell type clustering and spatial domain detection in spatial transcriptomic studies. Genome Biol 23:168.
• Yuan et al. (2024). Benchmarking spatial clustering methods with spatially resolved transcriptomics data. Nat Methods 21:712-722.
• Chen et al. (2025). A comprehensive benchmarking for spatially resolved transcriptomics clustering methods across variable technologies, organs, and replicates. iMeta 4:e70084.
• Maynard et al. (2021). Transcriptome-scale spatial gene expression in the human dorsolateral prefrontal cortex. Nat Neurosci 24:425-436.
• Palla et al. (2022). Squidpy: a scalable framework for spatial omics analysis. Nat Methods 19:171-178.