Working at scale

Two independent answers to a dataset that will not fit: analyse a representative subset, or keep the matrix on disk.

Sketching draws a leverage-weighted subset — rare states kept rather than sampled away — analyses that, then extends the result back to every cell. leverage_score gets the per-cell scores via a CountSketch, without a full SVD.

LazyMatrix is the on-disk path, memory-mapped compressed-sparse-column arrays in BPCells' spirit but with no new dependency. A slice reads only the cells it touches, col_blocks streams a million cells at bounded RAM, and it drops straight into an Assay5 layer. Against BPCells on PBMC 3k, truecell's on-disk and in-memory paths are bit-identical to each other; Seurat's differ by 1.0e-06. The comparison.

Sketching

leverage_score

leverage_score(obj, nsketch: int = 5000, ndims: Optional[int] = None, features: Optional[list[str]] = None, assay: Optional[str] = None, layer: str = 'data', var_name: Optional[str] = 'leverage.score', eps: float = 0.5, seed: int = 123) -> ndarray

Per-cell statistical leverage (Seurat's LeverageScore).

A cell's leverage is how much it influences the column space of the data — low in a dense, redundant cloud, high in a sparse, distinctive corner — so sampling proportional to it keeps the rare states a uniform draw would lose.

Which of the two regimes runs is decided exactly as Seurat decides it. With fewer than nsketch * 1.5 cells the scores come from a rank-50 truncated SVD and sum to 50; above that a CountSketch embedding, a QR and a Johnson–Lindenstrauss projection stand in for the SVD, and the scores are on the projection's scale rather than summing to 50. Compare scores within one call, never across the two regimes.

Parameters:

• obj –

  a Truecell object (normalized).

• nsketch (int, default: 5000 ) –

  rows of the random sketch, and the threshold that picks the regime.

• ndims (Optional[int], default: None ) –

  dimension the JL projection targets before eps shrinks it (default: the cell count, as in Seurat). Sketched regime only.

• features (Optional[list[str]], default: None ) –

  features to score on (default: the assay's variable features).

• assay (Optional[str], default: None ) –

  assay to use (default: active assay).

• layer (str, default: 'data' ) –

  layer to draw the data from. Defaults to "data" — the log-normalized values, which is what Seurat scores — not "scale.data".

• var_name (Optional[str], default: 'leverage.score' ) –

  if given, the scores are also written to obj.meta_data[var_name].

• eps (float, default: 0.5 ) –

  Johnson–Lindenstrauss distortion, 0 < eps <= 1 (Seurat's 0.5). Smaller keeps more projected dimensions. Sketched regime only.

• seed (int, default: 123 ) –

  random seed for the sketch and the projection.

Returns:

• ndarray –

  One leverage score per cell, in obj.cell_names() order.

Source code in truecell/sketch.py

def leverage_score(
    obj,
    nsketch: int = 5000,
    ndims: Optional[int] = None,
    features: Optional[list[str]] = None,
    assay: Optional[str] = None,
    layer: str = "data",
    var_name: Optional[str] = "leverage.score",
    eps: float = 0.5,
    seed: int = 123,
) -> np.ndarray:
    """Per-cell statistical leverage (Seurat's ``LeverageScore``).

    A cell's leverage is how much it influences the column space of the data — low
    in a dense, redundant cloud, high in a sparse, distinctive corner — so sampling
    proportional to it keeps the rare states a uniform draw would lose.

    Which of the two regimes runs is decided exactly as Seurat decides it. With
    fewer than ``nsketch * 1.5`` cells the scores come from a rank-``50`` truncated
    SVD and sum to 50; above that a ``CountSketch`` embedding, a ``QR`` and a
    Johnson–Lindenstrauss projection stand in for the SVD, and the scores are on
    the projection's scale rather than summing to 50. Compare scores *within* one
    call, never across the two regimes.

    Parameters
    ----------
    obj      : a :class:`~truecell.Truecell` object (normalized).
    nsketch  : rows of the random sketch, and the threshold that picks the regime.
    ndims    : dimension the JL projection targets before ``eps`` shrinks it
               (default: the cell count, as in Seurat). Sketched regime only.
    features : features to score on (default: the assay's variable features).
    assay    : assay to use (default: active assay).
    layer    : layer to draw the data from. Defaults to ``"data"`` — the
               log-normalized values, which is what Seurat scores — *not*
               ``"scale.data"``.
    var_name : if given, the scores are also written to ``obj.meta_data[var_name]``.
    eps      : Johnson–Lindenstrauss distortion, ``0 < eps <= 1`` (Seurat's 0.5).
               Smaller keeps more projected dimensions. Sketched regime only.
    seed     : random seed for the sketch and the projection.

    Returns
    -------
    numpy.ndarray
        One leverage score per cell, in ``obj.cell_names()`` order.
    """
    assay_name = assay or obj.active_assay
    assay_obj = obj.assays[assay_name]
    feats = _default_features(assay_obj, features)

    # (n_features × n_cells), left sparse when it already is: sketching exists to
    # keep large data cheap, and densifying here would give that away up front.
    A = _leverage_matrix(assay_obj, feats, layer)
    n_cells = A.shape[1]

    if n_cells < nsketch * 1.5:
        scores = _leverage_exact(A)
    else:
        rng = np.random.default_rng(seed)
        scores = _leverage_sketched(A, nsketch, ndims or n_cells, eps, rng)

    if var_name is not None:
        obj.meta_data[var_name] = scores
    return scores

sketch_data

sketch_data(obj, ncells: int = 5000, method: str = 'LeverageScore', features: Optional[list[str]] = None, assay: Optional[str] = None, layer: str = 'data', nsketch: int = 5000, sketched_assay: str = 'sketch', var_name: Optional[str] = 'leverage.score', seed: int = 123)

Draw a leverage-weighted subset of cells (Seurat's SketchData).

Mirrors SketchData(object, ncells = 5000, method = "LeverageScore"). Each cell is sampled without replacement with probability proportional to its leverage_score, so the rare states a uniform sample would drop are kept (indeed over-represented). The leverage scores are written back onto obj's metadata, and the drawn subset is returned as a standalone Truecell object — run the expensive analysis (PCA, clustering, UMAP) on it and use project_data to extend the results to every cell.

This departs from Seurat, which stores the sketch as an extra assay on the same object; here it is a separate object, matching the roadmap and truecell's subset model. Its active assay is renamed to sketched_assay so the provenance is visible, and obj.misc["sketch"] records how it was drawn.

Parameters:

• obj –

  a Truecell object (normalized).

• ncells (int, default: 5000 ) –

  cells to keep (capped at the number available).

• method (str, default: 'LeverageScore' ) –

  "LeverageScore" (leverage-weighted) or "Uniform" (equal weights), as in Seurat. "Uniform" is the control that shows what leverage weighting is buying.

• features (Optional[list[str]], default: None ) –

  features to score on (default: variable features).

• assay (Optional[str], default: None ) –

  assay to use (default: active assay).

• layer (str, default: 'data' ) –

  layer to draw the data from (default "data").

• nsketch (int, default: 5000 ) –

  sketch size passed to leverage_score.

• sketched_assay (str, default: 'sketch' ) –

  name the returned object's active assay is renamed to.

• var_name (Optional[str], default: 'leverage.score' ) –

  metadata column the scores are written to on obj.

• seed (int, default: 123 ) –

  random seed for scoring and sampling.

Returns:

• Truecell –

  The sketched subset (a new object).

Source code in truecell/sketch.py

def sketch_data(
    obj,
    ncells: int = 5000,
    method: str = "LeverageScore",
    features: Optional[list[str]] = None,
    assay: Optional[str] = None,
    layer: str = "data",
    nsketch: int = 5000,
    sketched_assay: str = "sketch",
    var_name: Optional[str] = "leverage.score",
    seed: int = 123,
):
    """Draw a leverage-weighted subset of cells (Seurat's ``SketchData``).

    Mirrors ``SketchData(object, ncells = 5000, method = "LeverageScore")``. Each
    cell is sampled without replacement with probability proportional to its
    :func:`leverage_score`, so the rare states a uniform sample would drop are kept
    (indeed over-represented). The leverage scores are written back onto ``obj``'s
    metadata, and the drawn subset is returned as a **standalone**
    :class:`~truecell.Truecell` object — run the expensive analysis (PCA, clustering,
    UMAP) on it and use :func:`project_data` to extend the results to every cell.

    This departs from Seurat, which stores the sketch as an extra assay on the same
    object; here it is a separate object, matching the roadmap and truecell's
    ``subset`` model. Its active assay is renamed to ``sketched_assay`` so the
    provenance is visible, and ``obj.misc["sketch"]`` records how it was drawn.

    Parameters
    ----------
    obj            : a :class:`~truecell.Truecell` object (normalized).
    ncells         : cells to keep (capped at the number available).
    method         : ``"LeverageScore"`` (leverage-weighted) or ``"Uniform"``
                     (equal weights), as in Seurat. ``"Uniform"`` is the control
                     that shows what leverage weighting is buying.
    features       : features to score on (default: variable features).
    assay          : assay to use (default: active assay).
    layer          : layer to draw the data from (default ``"data"``).
    nsketch        : sketch size passed to :func:`leverage_score`.
    sketched_assay : name the returned object's active assay is renamed to.
    var_name       : metadata column the scores are written to on ``obj``.
    seed           : random seed for scoring and sampling.

    Returns
    -------
    Truecell
        The sketched subset (a new object).
    """
    if method not in ("LeverageScore", "Uniform"):
        raise ValueError(
            f"Unknown sketch method {method!r}; expected 'LeverageScore' or 'Uniform'."
        )

    n = len(obj)
    if method == "Uniform":
        scores = np.ones(n)
        if var_name is not None:
            obj.meta_data[var_name] = scores
    else:
        scores = leverage_score(
            obj, nsketch=nsketch, features=features, assay=assay, layer=layer,
            var_name=var_name, seed=seed,
        )

    k = min(ncells, n)
    total = float(scores.sum())
    probs = scores / total if total > 0 else np.full(n, 1.0 / n)

    rng = np.random.default_rng(seed + 1)
    idx = np.sort(rng.choice(n, size=k, replace=False, p=probs))
    cell_names = obj.cell_names()
    cells = [cell_names[i] for i in idx]

    sketched = obj.subset(cells=cells)
    if sketched_assay and sketched_assay != sketched.active_assay:
        src = sketched.active_assay
        sketched.assays[sketched_assay] = sketched.assays.pop(src)
        sketched.active_assay = sketched_assay
    sketched.misc["sketch"] = {
        "method": method,
        "ncells": k,
        "from_cells": n,
        "source_assay": obj.active_assay,
        "leverage_var": var_name,
    }
    return sketched

project_data

project_data(full, sketch, reduction: str = 'pca', full_reduction: str = 'pca.full', umap_reduction: str = 'umap', full_umap_reduction: str = 'ref.umap', refdata: Optional[Union[str, dict]] = None, project_umap: bool = True, dims: Optional[Union[list[int], range]] = None, k_weight: int = 50, sd_weight: float = 1.0, layer: str = 'scale.data')

Extend a sketch's analysis to the full dataset (Seurat's ProjectData).

The inverse of sketch_data: once the sketch has been reduced and (optionally) clustered, every full-dataset cell is placed into the sketch's coordinate system and, if asked, given the sketch's labels.

1. PCA. Each full cell is pushed through the sketch's PCA loadings — the same "project into a space this cell never helped define" linear map that truecell.project_umap uses — and stored as full.reductions[full_reduction].
2. UMAP (when project_umap and the sketch carries a fitted UMAP model): the projected cells are run through the sketch's UMAP via truecell.project_umap, stored as full.reductions[full_umap_reduction].
3. Labels (when refdata is given): a weighted k-nearest-neighbour vote inside the projected reduction, where the sketch's own rows are the reference — Seurat's TransferSketchLabels. Written onto full.meta_data.

Step 3 is deliberately not the truecell.transfer anchor path, which is what an earlier version of this function used. Seurat does not use anchors here, and the difference is not academic: finding anchors between the sketch and the full dataset costs exactly what sketching exists to avoid, so on the million-cell objects this is written for the anchor route is unusable rather than merely different. On ifnb the two now agree per-cell 98.1 % of the time, at matching accuracy.

full is mutated in place and returned.

Parameters:

• full –

  the full Truecell object (normalized + scaled on the sketch's PCA features).

• sketch –

  the sketched object from sketch_data, already carrying a PCA (and optionally a fitted UMAP).

• reduction (str, default: 'pca' ) –

  sketch reduction whose loadings project the full data.

• full_reduction (str, default: 'pca.full' ) –

  storage key for the projected PCA on full.

• umap_reduction (str, default: 'umap' ) –

  sketch reduction holding the fitted UMAP model.

• full_umap_reduction (str, default: 'ref.umap' ) –

  storage key for the projected UMAP on full.

• refdata (Optional[Union[str, dict]], default: None ) –

  sketch metadata to transfer, as in Seurat: a dict {new_col: sketch_col} writes each label under new_col plus new_col.score, and a bare str is shorthand for {col: col}. Must name a column on the sketch — like R, raw label arrays are not taken. None skips transfer.

• project_umap (bool, default: True ) –

  also project the sketch's UMAP when a fitted model exists.

• dims (Optional[Union[list[int], range]], default: None ) –

  reduction dimensions used for the UMAP model and the label vote (default: all).

• k_weight (int, default: 50 ) –

  neighbours each cell votes over (Seurat's k.weight).

• sd_weight (float, default: 1.0 ) –

  bandwidth of the distance kernel (Seurat fixes this at 1).

• layer (str, default: 'scale.data' ) –

  layer to draw the full data's expression from.

Returns:

• Truecell –

  full, now carrying the projected reduction(s) and any transferred labels.

Source code in truecell/sketch.py

def project_data(
    full,
    sketch,
    reduction: str = "pca",
    full_reduction: str = "pca.full",
    umap_reduction: str = "umap",
    full_umap_reduction: str = "ref.umap",
    refdata: Optional[Union[str, dict]] = None,
    project_umap: bool = True,
    dims: Optional[Union[list[int], range]] = None,
    k_weight: int = 50,
    sd_weight: float = 1.0,
    layer: str = "scale.data",
):
    """Extend a sketch's analysis to the full dataset (Seurat's ``ProjectData``).

    The inverse of :func:`sketch_data`: once the sketch has been reduced and
    (optionally) clustered, every full-dataset cell is placed into the sketch's
    coordinate system and, if asked, given the sketch's labels.

    1. **PCA.** Each full cell is pushed through the sketch's PCA loadings — the
       same "project into a space this cell never helped define" linear map that
       :func:`truecell.project_umap` uses — and stored as
       ``full.reductions[full_reduction]``.
    2. **UMAP** (when ``project_umap`` and the sketch carries a fitted UMAP model):
       the projected cells are run through the sketch's UMAP via
       :func:`truecell.project_umap`, stored as ``full.reductions[full_umap_reduction]``.
    3. **Labels** (when ``refdata`` is given): a weighted k-nearest-neighbour vote
       *inside the projected reduction*, where the sketch's own rows are the
       reference — Seurat's ``TransferSketchLabels``. Written onto
       ``full.meta_data``.

    Step 3 is deliberately **not** the :mod:`truecell.transfer` anchor path, which
    is what an earlier version of this function used. Seurat does not use anchors
    here, and the difference is not academic: finding anchors between the sketch
    and the full dataset costs exactly what sketching exists to avoid, so on the
    million-cell objects this is written for the anchor route is unusable rather
    than merely different. On ifnb the two now agree per-cell **98.1 %** of the
    time, at matching accuracy.

    ``full`` is mutated in place and returned.

    Parameters
    ----------
    full                : the full :class:`~truecell.Truecell` object (normalized +
                          scaled on the sketch's PCA features).
    sketch              : the sketched object from :func:`sketch_data`, already
                          carrying a PCA (and optionally a fitted UMAP).
    reduction           : sketch reduction whose loadings project the full data.
    full_reduction      : storage key for the projected PCA on ``full``.
    umap_reduction      : sketch reduction holding the fitted UMAP model.
    full_umap_reduction : storage key for the projected UMAP on ``full``.
    refdata             : sketch metadata to transfer, as in Seurat: a ``dict``
                          ``{new_col: sketch_col}`` writes each label under
                          ``new_col`` plus ``new_col.score``, and a bare ``str``
                          is shorthand for ``{col: col}``. Must name a column on
                          the *sketch* — like R, raw label arrays are not taken.
                          ``None`` skips transfer.
    project_umap        : also project the sketch's UMAP when a fitted model exists.
    dims                : reduction dimensions used for the UMAP model and the
                          label vote (default: all).
    k_weight            : neighbours each cell votes over (Seurat's ``k.weight``).
    sd_weight           : bandwidth of the distance kernel (Seurat fixes this at 1).
    layer               : layer to draw the full data's expression from.

    Returns
    -------
    Truecell
        ``full``, now carrying the projected reduction(s) and any transferred labels.
    """
    from .mapping import _project_into_reference_pca, project_umap as _project_umap

    if reduction not in sketch.reductions:
        raise KeyError(
            f"Sketch reduction {reduction!r} not found; run run_pca(sketch) first."
        )

    sketch_pca = sketch.reductions[reduction]
    full_pca = _project_into_reference_pca(full, sketch, reduction, layer)

    key = getattr(sketch_pca, "_key", "PC_") or "PC_"
    dim_names = [f"{key}{i + 1}" for i in range(full_pca.shape[1])]
    full.reductions[full_reduction] = DimReduc(
        cell_embeddings=full_pca,
        cell_names=full.cell_names(),
        feature_loadings=np.asarray(sketch_pca.feature_loadings),
        feature_names=dim_names if not sketch_pca.features() else list(sketch_pca.features()),
        assay_used=full.active_assay,
        key=key,
        misc={"projected_from": reduction, "sketch_reduction": reduction},
    )

    if (
        project_umap
        and umap_reduction in sketch.reductions
        and sketch.reductions[umap_reduction].misc.get("umap_model") is not None
    ):
        _project_umap(
            full,
            sketch,
            reduction=reduction,
            umap_reduction=umap_reduction,
            dims=dims,
            reduction_name=full_umap_reduction,
            layer=layer,
        )

    if refdata is not None:
        _transfer_from_sketch(
            full, sketch, full_reduction, refdata, k_weight, sd_weight, dims
        )

    return full

Out-of-core matrices

LazyMatrix

LazyMatrix(path: Union[str, Path], shape: Tuple[int, int], data: ndarray, indices: ndarray, indptr: ndarray)

A memory-mapped, on-disk compressed-sparse-column matrix.

Instances are created by write_lazy_matrix (persist an in-memory matrix) or open_lazy_matrix (map an existing store); they are not constructed directly. The three CSC arrays are memory-mapped read-only, so the object is cheap to hold and its footprint is the slices you touch — not the whole matrix.

Supports the slicing idioms the assay layer accessors use — m[rows, cols], m[np.ix_(rows, cols)], m[idx, :], m[:, idx], contiguous slices — returning a scipy.sparse.csc_matrix block. Tuple indexing is always an outer (cross-product) selection, matching np.ix_ and how layers are block-subset throughout truecell; element-wise pair indexing is not supported.

Source code in truecell/lazy.py

def __init__(
    self,
    path: Union[str, Path],
    shape: Tuple[int, int],
    data: np.ndarray,
    indices: np.ndarray,
    indptr: np.ndarray,
) -> None:
    self._path = Path(path)
    self._shape = (int(shape[0]), int(shape[1]))
    self._data = data
    self._indices = indices
    self._indptr = indptr

col_blocks

col_blocks(block_size: int = 10000) -> Iterator[Tuple[int, int, csc_matrix]]

Stream the matrix in blocks of block_size columns (cells).

Yields (start, stop, block) where block is an in-memory csc_matrix of columns [start:stop). This is the primitive for an out-of-core reduction: process a million cells at bounded peak memory.

Source code in truecell/lazy.py

def col_blocks(
    self, block_size: int = 10_000
) -> Iterator[Tuple[int, int, sp.csc_matrix]]:
    """Stream the matrix in blocks of ``block_size`` columns (cells).

    Yields ``(start, stop, block)`` where ``block`` is an in-memory
    ``csc_matrix`` of columns ``[start:stop)``. This is the primitive for an
    out-of-core reduction: process a million cells at bounded peak memory.
    """
    if block_size <= 0:
        raise ValueError("block_size must be positive.")
    for start in range(0, self.ncol, block_size):
        stop = min(start + block_size, self.ncol)
        yield start, stop, self._read_columns(slice(start, stop))

sum

sum(axis: Optional[int] = None)

Sum over axis (0 → per-cell, 1 → per-feature, None → scalar).

Source code in truecell/lazy.py

def sum(self, axis: Optional[int] = None):
    """Sum over ``axis`` (0 → per-cell, 1 → per-feature, None → scalar)."""
    data = np.asarray(self._data)
    if axis is None:
        return float(data.sum())
    if data.size == 0:
        return np.zeros(self.ncol if axis == 0 else self.nrow, dtype=float)
    if axis == 0:
        counts = np.diff(np.asarray(self._indptr))
        col_ids = np.repeat(np.arange(self.ncol), counts)
        return np.bincount(col_ids, weights=data, minlength=self.ncol).astype(float)
    if axis == 1:
        return np.bincount(
            np.asarray(self._indices), weights=data, minlength=self.nrow
        ).astype(float)
    raise ValueError("axis must be 0, 1, or None.")

mean

mean(axis: Optional[int] = None)

Mean over axis, dividing the streamed sums by the matrix extent.

Source code in truecell/lazy.py

def mean(self, axis: Optional[int] = None):
    """Mean over ``axis``, dividing the streamed sums by the matrix extent."""
    total = self.sum(axis)
    if axis is None:
        return total / (self.nrow * self.ncol)
    denom = self.nrow if axis == 0 else self.ncol
    return total / denom

nnz_per_col

nnz_per_col() -> ndarray

Non-zeros per column (cell) — the nFeature count, read from indptr.

Source code in truecell/lazy.py

def nnz_per_col(self) -> np.ndarray:
    """Non-zeros per column (cell) — the ``nFeature`` count, read from indptr."""
    return np.diff(np.asarray(self._indptr)).astype(np.int64)

nnz_per_row

nnz_per_row(block_size: int = 10000) -> ndarray

Non-zeros per row (feature) — the min_cells count.

The column counterpart is free from indptr; this one has to look at every non-zero's row index, so it streams in cell-blocks rather than mapping the whole indices array at once.

Source code in truecell/lazy.py

def nnz_per_row(self, block_size: int = 10_000) -> np.ndarray:
    """Non-zeros per row (feature) — the ``min_cells`` count.

    The column counterpart is free from ``indptr``; this one has to look at
    every non-zero's row index, so it streams in cell-blocks rather than
    mapping the whole ``indices`` array at once.
    """
    counts = np.zeros(self.nrow, dtype=np.int64)
    for _, _, block in self.col_blocks(block_size):
        counts += np.bincount(block.indices, minlength=self.nrow).astype(np.int64)
    return counts

to_scipy

to_scipy() -> csc_matrix

Read the whole matrix into an in-memory csc_matrix.

Source code in truecell/lazy.py

def to_scipy(self) -> sp.csc_matrix:
    """Read the whole matrix into an in-memory ``csc_matrix``."""
    return sp.csc_matrix(
        (np.array(self._data), np.array(self._indices), np.array(self._indptr)),
        shape=self._shape,
    )

toarray

toarray() -> ndarray

Read the whole matrix into a dense ndarray.

Source code in truecell/lazy.py

def toarray(self) -> np.ndarray:
    """Read the whole matrix into a dense ``ndarray``."""
    return self.to_scipy().toarray()

close

close() -> None

Release the memory-mapped arrays.

Source code in truecell/lazy.py

def close(self) -> None:
    """Release the memory-mapped arrays."""
    for name in ("_data", "_indices", "_indptr"):
        arr = getattr(self, name, None)
        mm = getattr(arr, "_mmap", None) if arr is not None else None
        if mm is not None:
            mm.close()
        setattr(self, name, None)

write_lazy_matrix

write_lazy_matrix(matrix, path: Union[str, Path], *, overwrite: bool = False) -> LazyMatrix

Write matrix to path as an on-disk CSC store and open it lazily.

matrix may be a scipy sparse matrix, a dense array-like, or another LazyMatrix; it is canonicalised to sorted, duplicate-summed CSC before being saved as three .npy arrays plus a JSON header. Returns a LazyMatrix mapping the freshly written store.

Source code in truecell/lazy.py

def write_lazy_matrix(
    matrix, path: Union[str, Path], *, overwrite: bool = False
) -> LazyMatrix:
    """Write ``matrix`` to ``path`` as an on-disk CSC store and open it lazily.

    ``matrix`` may be a scipy sparse matrix, a dense array-like, or another
    :class:`LazyMatrix`; it is canonicalised to sorted, duplicate-summed CSC
    before being saved as three ``.npy`` arrays plus a JSON header. Returns a
    :class:`LazyMatrix` mapping the freshly written store.
    """
    path = Path(path)
    if isinstance(matrix, LazyMatrix):
        matrix = matrix.to_scipy()
    m = as_sparse(matrix, "csc")
    m.sum_duplicates()
    m.sort_indices()

    if path.exists():
        looks_like_store = (path / _META).exists()
        is_empty_dir = path.is_dir() and not any(path.iterdir())
        if not overwrite:
            raise FileExistsError(
                f"'{path}' already exists; pass overwrite=True to replace it."
            )
        if not (looks_like_store or is_empty_dir):
            raise ValueError(
                f"Refusing to overwrite '{path}': it is not a lazy-matrix store."
            )
        shutil.rmtree(path)
    path.mkdir(parents=True)

    np.save(path / _DATA, m.data)
    np.save(path / _INDICES, m.indices)
    np.save(path / _INDPTR, m.indptr)
    meta = {
        "format": _FORMAT,
        "shape": [int(m.shape[0]), int(m.shape[1])],
        "dtype": str(m.data.dtype),
        "nnz": int(m.nnz),
    }
    (path / _META).write_text(json.dumps(meta, indent=2))
    return open_lazy_matrix(path)

open_lazy_matrix

open_lazy_matrix(path: Union[str, Path]) -> LazyMatrix

Memory-map an on-disk CSC store written by write_lazy_matrix.

Source code in truecell/lazy.py

def open_lazy_matrix(path: Union[str, Path]) -> LazyMatrix:
    """Memory-map an on-disk CSC store written by :func:`write_lazy_matrix`."""
    path = Path(path)
    meta_path = path / _META
    if not meta_path.exists():
        raise FileNotFoundError(f"No lazy-matrix store at '{path}'.")
    meta = json.loads(meta_path.read_text())
    if meta.get("format") != _FORMAT:
        raise ValueError(f"Unknown lazy-matrix format: {meta.get('format')!r}.")

    data = np.load(path / _DATA, mmap_mode="r")
    indices = np.load(path / _INDICES, mmap_mode="r")
    indptr = np.load(path / _INDPTR, mmap_mode="r")
    return LazyMatrix(path, tuple(meta["shape"]), data, indices, indptr)

is_lazy

is_lazy(x) -> bool

True if x is a LazyMatrix (an on-disk, memory-mapped layer).

Source code in truecell/lazy.py

def is_lazy(x) -> bool:
    """True if ``x`` is a :class:`LazyMatrix` (an on-disk, memory-mapped layer)."""
    return isinstance(x, LazyMatrix)