Metadata version bumps: hca_2024 now points to
hca2024_v2.4.0.parquet (previously v2.3.2) and
hca_2025 to hca2025_v0.2.0.parquet (previously
v0.1.1). The sample database URL (SAMPLE_DATABASE_URL) has
been updated accordingly.
keep_quality_cells() gains a feature-count filter.
Two new parameters: nfeature_col (default
"feature_count") and min_features (default
5000L).
Cells with fewer than min_features detected features are
now excluded in addition to empty droplets, dead cells, and doublets.
Removed tissue_groups column: the harmonised metadata column
tissue_groups has been retired due to mapping fragility.
Column rename: inverse_transform is now documented as
inversed_inferred_distribution to reflect its actual name in the
metadata.
feature_count is added as a column to the metadata.
Vignettes and README updated throughout to reflect the new metadata
version, the removal of tissue_groups, and revised usage examples.
Batched parallel downloads: sync_remote_files() now splits large
download queues into batches (default 500 files per batch) before passing
them to curl::multi_download(). Previously, all URLs were submitted
in a single call; when requesting multiple assays simultaneously (e.g.\
assays = c("counts", "cpm", "rank") over 1\,000 files = 3\,000
requests), the Nectar OpenStack Swift server would reject the excess
connections with HTTP 4xx/5xx responses, silently failing thousands of
downloads. Batching keeps each parallel burst below the server's
per-account connection threshold.
The batch size defaults to 500 and can be adjusted via
options(cellNexus.download_batch_size = N).
When more than one batch is required, a progress message is printed for each batch (e.g.\ Batch 2/6: 500 files...).
Failure counting is now accumulated across all batches so the final warning correctly reports the total number of files that could not be downloaded.
Cache integrity: sync_remote_file() and
sync_remote_files() now verify that a locally cached file has not
been partially downloaded or corrupted by comparing its MD5 digest against
the remote object's ETag (OpenStack Swift stores the MD5 as the ETag for
every object). When a mismatch is detected the stale file is deleted and
the download restarts cleanly, preventing silent failures on the second
query after an interrupted first download.
New internal helper url_file_head_info() performs a single
parallel batch of HEAD requests and returns both the ETag (MD5)
and Content-Length for each URL.
New internal helper check_file_integrity() uses a
two-stage approach to avoid reading large files unnecessarily:
Size check (no file I/O): compare
file.size() with Content-Length. A mismatch means
the file was partially downloaded and is caught immediately.
MD5 check (reads the file): compare
tools::md5sum() with the ETag. Only reached when sizes
agree, guarding against silent disk corruption without penalising
correctly cached files where the size is wrong.
sync_remote_files() retrieves both Content-Length
and ETag in one parallel HEAD batch, then applies the two-stage
check to all already-cached files before deciding which files need to
be (re-)downloaded.
get_metadata() and sync_all_assay_files() now pass
the full file list to sync_remote_files() instead of
pre-filtering to only missing files, so the integrity check fires for
already-cached parquet and H5AD files too, not just freshly
downloaded ones.
Set options(cellNexus.check_cache_integrity = FALSE) to
skip the integrity check only when the cache is known to be intact.
Added download_only argument (default FALSE) to
get_single_cell_experiment(), get_seurat(), and
get_pseudobulk().
When TRUE, remote files are synchronised to
cache_directory but are not read or assembled into an
in-memory object.
Quality-control filtering (keep_quality_cells()) is
applied automatically in both modes, so only the files required for
quality-controlled cells are downloaded. A warning is emitted when
cells are excluded so users are informed even if they omit
keep_quality_cells() explicitly.
For the sct assay, the post-sync missing-file check
(.skip_groups_missing_sct_file()) now runs before the
download_only early return, so warnings about samples whose
SCT normalisation failed are emitted regardless of mode.
When assays = "sct" only and every sample is missing
its SCT file, the function emits a warning and returns
invisible(NULL) instead of aborting.
Added donor_id to harmonised metadata, enabling direct donor-level
queries across datasets.
Updated metadata parquet versions to hca2024_v2.3.2
Expanded and corrected documentation for metadata columns on website.
Removed internal columns (sample_,
sample_heuristic, sample_chunk, cell_chunk,
sample_pseudobulk_chunk) from documentation.
get_seurat() now correctly handles multiple assays and non-default
first assays.
When assays = c("counts", "cpm") (or any multi-assay request),
all assays are transferred to the returned Seurat object: the first assay
occupies the "originalexp" slot and every subsequent assay is added
under its own name (e.g. "cpm").
When the first requested assay is not "counts" (e.g.
assays = "cpm"), the conversion no longer errors with
"No data in provided assay - counts". The first assay in the SCE is
now passed explicitly to SeuratObject::as.Seurat().
SCT assay: robust handling of samples for which SCT normalisation failed.
HTTP 404 responses for SCT h5ad files are now treated as
"file not available" rather than a fatal download error. The SCT
sync path uses ignore_not_found = TRUE while all other assay syncs
remain strict.
A new internal helper .skip_groups_missing_sct_file() detects
cached-file absence after sync, emits a cellNexus says: warning with
the affected cell/sample counts and up to three example sample_ids,
and instructs users to identify skipped samples post-retrieval via
dplyr::setdiff(unique(your_metadata\$sample_id),
unique(SummarizedExperiment::colData(result)\$sample_id)).
When assays = "sct" only and every sample is missing its
SCT file, get_single_cell_experiment() aborts with a clear message.
When assays = c("counts", "sct") (or any mixed-assay request)
and every sample is missing its SCT file, the function falls back
gracefully to returning the non-SCT assays for all cells with a warning,
rather than aborting.
Tests. Added coverage for get_seurat() multi-assay and
non-default-assay paths.
get_pseudobulk() now retains metadata columns that are constant within each
sample_id x cell_type_unified_ensemble combination, including
user-added annotations, when building colData. Cell-level columns are dropped.
get_metacell() uses the same once-per-query column selection for
sample_id x metacell-level annotations, so cbind() across
files shares an identical colData schema.
Added internal helpers get_specific_annotation_columns() and
keep_specific_annotation_columns() to identify and keep columns functionally
determined by selected key column(s). Works on local data frames and lazy DuckDB
tables from get_metadata().
Tests. Added coverage for annotation-column detection and for preserving
sample-grain user columns through get_pseudobulk(), using
SAMPLE_DATABASE_URL.
get_pseudobulk() gained as_SummarizedExperiment to optionally return a SummarizedExperiment. Coercion restores feature rownames, which are otherwise dropped by as(x, "SummarizedExperiment").
Tests. Added a test ensuring feature rownames are preserved when as_SummarizedExperiment = TRUE.
Consistent features handling across retrieval functions. get_pseudobulk(), get_single_cell_experiment(), and get_metacell() now drop experiments/samples that do not contain all requested features and align rows to the requested feature set before merging. A warning indicates how many experiments were dropped.
Stabilised feature subsetting during merge to avoid index/subscript errors and ensure identical row order across experiments prior to cbind().
Qualified functional calls (e.g., purrr::map_lgl) to avoid missing import issues.
Documentation updates. Clarified the features parameter docs for all three functions to describe the drop-and-align behavior; regenerated help files.
Tests. Added a test verifying pseudobulk subsetting to the specific feature ENSG00000065485.
Added a NEWS file to track changes to the package.