The purpose of this vignette is to explore the file manifests available from the Human Cell Atlas project.
These files provide a metadata summary for a collection of files in a tabular format, including but not limited to information about process and workflow used to generate the file, information about the specimens the file data were derived from, and identifiers connect specific projects, files, and specimens.
The WARP (WDL Analysis Research Pipelines) repository contains information on a variety of pipelines, and can be used alongside a manifest to better understand the metadata.
Evaluate the following code chunk to install packages required for this vignette.
## install from Bioconductor if you haven't already
pkgs <- c("LoomExperiment", "hca")
pkgs_needed <- pkgs[!pkgs %in% rownames(installed.packages())]
BiocManager::install(pkgs_needed)Load the packages into your R session.
The manifest for all files available can be obtained with (this can takes several minutes to complete)
This is seldom useful; instead, create a filter identifying the files of interest.
manifest_filter <- hca::filters(
projectId = list(is = "4a95101c-9ffc-4f30-a809-f04518a23803"),
fileFormat = list(is = "loom"),
workflow = list(is = c("optimus_v4.2.2", "optimus_v4.2.3"))
)Retrieve the manifest
manifest_tibble <- hca::manifest(filters = manifest_filter)
## requesting manifest
## waiting 1 more seconds (1 seconds total)...
## waiting 1 more seconds (2 seconds total)...
## waiting 1 more seconds (3 seconds total)...
## waiting 4 more seconds (7 seconds total)...
manifest_tibble
## # A tibble: 20 × 56
## source_id source_spec bundle_uuid bundle_version file_document_id
## <chr> <chr> <chr> <dttm> <chr>
## 1 2bf54baa-538e-4… tdr:bigque… b593b66a-d… 2020-02-03 01:00:00 131ea511-25f7-5…
## 2 2bf54baa-538e-4… tdr:bigque… 5a63dd0b-5… 2021-02-02 23:50:00 1bb375a5-d22b-5…
## 3 2bf54baa-538e-4… tdr:bigque… 40733888-3… 2021-02-02 23:55:00 1f8ff0fa-6892-5…
## 4 2bf54baa-538e-4… tdr:bigque… 1a41ebe6-e… 2021-02-02 23:50:00 2fffe225-ba6c-5…
## 5 2bf54baa-538e-4… tdr:bigque… c12a6ca2-3… 2021-02-02 23:50:00 31aa5a18-2a4e-5…
## 6 2bf54baa-538e-4… tdr:bigque… f58d690c-b… 2021-02-02 23:50:00 48eea299-8823-5…
## 7 2bf54baa-538e-4… tdr:bigque… 21c4e2de-e… 2021-02-02 23:50:00 51458973-404c-5…
## 8 2bf54baa-538e-4… tdr:bigque… 50620c50-2… 2021-02-02 23:50:00 5bbebef4-9b14-5…
## 9 2bf54baa-538e-4… tdr:bigque… e3ecdfc2-4… 2021-02-02 23:55:00 5bc232f2-b77c-5…
## 10 2bf54baa-538e-4… tdr:bigque… ae338c4e-6… 2021-02-02 23:50:00 6326b602-0f63-5…
## 11 2bf54baa-538e-4… tdr:bigque… d62c4599-4… 2020-02-03 01:00:00 7848d80b-6b1d-5…
## 12 2bf54baa-538e-4… tdr:bigque… 81df106e-e… 2021-02-02 23:55:00 9f8bc032-6276-5…
## 13 2bf54baa-538e-4… tdr:bigque… 2838323c-c… 2020-02-03 01:00:00 b98cfaac-64f5-5…
## 14 2bf54baa-538e-4… tdr:bigque… c3f672ad-e… 2021-02-02 23:50:00 bf7751ae-ac9d-5…
## 15 2bf54baa-538e-4… tdr:bigque… a9c90392-c… 2020-02-03 01:00:00 c7b6470c-e2f0-5…
## 16 2bf54baa-538e-4… tdr:bigque… 9d0f5cd1-0… 2020-02-03 01:00:00 d0b95f2c-98ae-5…
## 17 2bf54baa-538e-4… tdr:bigque… 59de15e1-f… 2021-02-02 23:50:00 d18759a6-2a95-5…
## 18 2bf54baa-538e-4… tdr:bigque… 54fb0e25-5… 2021-02-02 23:55:00 dfd9905b-d6c9-5…
## 19 2bf54baa-538e-4… tdr:bigque… 7516565a-e… 2021-02-02 23:55:00 e07ca731-b20a-5…
## 20 2bf54baa-538e-4… tdr:bigque… 8e850d2d-0… 2021-02-02 23:50:00 fd41f3d6-7664-5…
## # ℹ 51 more variables: file_type <chr>, file_name <chr>, file_format <chr>,
## # read_index <lgl>, file_size <dbl>, file_uuid <chr>, file_version <dttm>,
## # file_crc32c <chr>, file_sha256 <chr>, file_content_type <chr>,
## # file_drs_uri <chr>, file_url <chr>,
## # cell_suspension.provenance.document_id <chr>,
## # cell_suspension.biomaterial_core.biomaterial_id <chr>,
## # cell_suspension.estimated_cell_count <lgl>, …And perform additional filtering, e.g., identifying the specimen organs represented in the files.
manifest_tibble and select
one for downloadmanifest_tibble
## # A tibble: 20 × 56
## source_id source_spec bundle_uuid bundle_version file_document_id
## <chr> <chr> <chr> <dttm> <chr>
## 1 2bf54baa-538e-4… tdr:bigque… b593b66a-d… 2020-02-03 01:00:00 131ea511-25f7-5…
## 2 2bf54baa-538e-4… tdr:bigque… 5a63dd0b-5… 2021-02-02 23:50:00 1bb375a5-d22b-5…
## 3 2bf54baa-538e-4… tdr:bigque… 40733888-3… 2021-02-02 23:55:00 1f8ff0fa-6892-5…
## 4 2bf54baa-538e-4… tdr:bigque… 1a41ebe6-e… 2021-02-02 23:50:00 2fffe225-ba6c-5…
## 5 2bf54baa-538e-4… tdr:bigque… c12a6ca2-3… 2021-02-02 23:50:00 31aa5a18-2a4e-5…
## 6 2bf54baa-538e-4… tdr:bigque… f58d690c-b… 2021-02-02 23:50:00 48eea299-8823-5…
## 7 2bf54baa-538e-4… tdr:bigque… 21c4e2de-e… 2021-02-02 23:50:00 51458973-404c-5…
## 8 2bf54baa-538e-4… tdr:bigque… 50620c50-2… 2021-02-02 23:50:00 5bbebef4-9b14-5…
## 9 2bf54baa-538e-4… tdr:bigque… e3ecdfc2-4… 2021-02-02 23:55:00 5bc232f2-b77c-5…
## 10 2bf54baa-538e-4… tdr:bigque… ae338c4e-6… 2021-02-02 23:50:00 6326b602-0f63-5…
## 11 2bf54baa-538e-4… tdr:bigque… d62c4599-4… 2020-02-03 01:00:00 7848d80b-6b1d-5…
## 12 2bf54baa-538e-4… tdr:bigque… 81df106e-e… 2021-02-02 23:55:00 9f8bc032-6276-5…
## 13 2bf54baa-538e-4… tdr:bigque… 2838323c-c… 2020-02-03 01:00:00 b98cfaac-64f5-5…
## 14 2bf54baa-538e-4… tdr:bigque… c3f672ad-e… 2021-02-02 23:50:00 bf7751ae-ac9d-5…
## 15 2bf54baa-538e-4… tdr:bigque… a9c90392-c… 2020-02-03 01:00:00 c7b6470c-e2f0-5…
## 16 2bf54baa-538e-4… tdr:bigque… 9d0f5cd1-0… 2020-02-03 01:00:00 d0b95f2c-98ae-5…
## 17 2bf54baa-538e-4… tdr:bigque… 59de15e1-f… 2021-02-02 23:50:00 d18759a6-2a95-5…
## 18 2bf54baa-538e-4… tdr:bigque… 54fb0e25-5… 2021-02-02 23:55:00 dfd9905b-d6c9-5…
## 19 2bf54baa-538e-4… tdr:bigque… 7516565a-e… 2021-02-02 23:55:00 e07ca731-b20a-5…
## 20 2bf54baa-538e-4… tdr:bigque… 8e850d2d-0… 2021-02-02 23:50:00 fd41f3d6-7664-5…
## # ℹ 51 more variables: file_type <chr>, file_name <chr>, file_format <chr>,
## # read_index <lgl>, file_size <dbl>, file_uuid <chr>, file_version <dttm>,
## # file_crc32c <chr>, file_sha256 <chr>, file_content_type <chr>,
## # file_drs_uri <chr>, file_url <chr>,
## # cell_suspension.provenance.document_id <chr>,
## # cell_suspension.biomaterial_core.biomaterial_id <chr>,
## # cell_suspension.estimated_cell_count <lgl>, …file_hca_tbl for the file based on it’s
uuidfile_filter <- hca::filters(
fileId = list(is = file_uuid)
)
file_tbl <- hca::files(filters = file_filter)
file_tbl
## # A tibble: 1 × 8
## fileId name fileFormat size version projectTitle projectId url
## <chr> <chr> <chr> <int> <chr> <chr> <chr> <chr>
## 1 24a8a323-7ecd-50… t-ce… loom 3.90e8 2021-0… Single-cell… 4a95101c… http…file_location <-
file_tbl |>
hca::files_download()
file_location
## 24a8a323-7ecd-504e-a253-b0e0892dd730-2021-02-11T19:00:05.000000Z
## "/github/home/.cache/R/hca/175e74e671ab_175e74e671ab.loom"LoomExperiment objectloom <- LoomExperiment::import(file_location)
metadata(loom) |>
dplyr::glimpse()
## List of 15
## $ last_modified : chr "20210211T185949.186062Z"
## $ CreationDate : chr "20210211T185658.758915Z"
## $ LOOM_SPEC_VERSION : chr "3.0.0"
## $ donor_organism.genus_species : chr "Homo sapiens"
## $ expression_data_type : chr "exonic"
## $ input_id : chr "58a18a4c-5423-4c59-9b3c-50b7f30b1ca5, c763f679-e13d-4f81-844f-c2c80fc90f46, c76d90b8-c190-4c58-b9bc-b31f586ec7f"| __truncated__
## $ input_id_metadata_field : chr "sequencing_process.provenance.document_id"
## $ input_name : chr "PP012_suspension, PP003_suspension, PP004_suspension, PP011_suspension"
## $ input_name_metadata_field : chr "sequencing_input.biomaterial_core.biomaterial_id"
## $ library_preparation_protocol.library_construction_approach: chr "10X v2 sequencing"
## $ optimus_output_schema_version : chr "1.0.0"
## $ pipeline_version : chr "Optimus_v4.2.2"
## $ project.project_core.project_name : chr "HumanTissueTcellActivation"
## $ project.provenance.document_id : chr "4a95101c-9ffc-4f30-a809-f04518a23803"
## $ specimen_from_organism.organ : chr "hematopoietic system"
colData(loom) |>
dplyr::as_tibble() |>
dplyr::glimpse()
## Rows: 91,713
## Columns: 43
## $ CellID <chr> "GCTTCCATCACCGT…
## $ antisense_reads <int> 0, 0, 0, 0, 0, …
## $ cell_barcode_fraction_bases_above_30_mean <dbl> 0.9846281, 0.98…
## $ cell_barcode_fraction_bases_above_30_variance <dbl> 0.003249023, 0.…
## $ cell_names <chr> "GCTTCCATCACCGT…
## $ duplicate_reads <int> 0, 0, 0, 0, 0, …
## $ emptydrops_FDR <dbl> 1.000000000, 0.…
## $ emptydrops_IsCell <raw> 00, 01, 00, 00,…
## $ emptydrops_Limited <raw> 00, 01, 00, 00,…
## $ emptydrops_LogProb <dbl> -689.6831, -120…
## $ emptydrops_PValue <dbl> 0.91840816, 0.0…
## $ emptydrops_Total <int> 255, 16705, 681…
## $ fragments_per_molecule <dbl> 1.693252, 8.453…
## $ fragments_with_single_read_evidence <int> 504, 139828, 58…
## $ genes_detected_multiple_observations <int> 82, 2873, 1552,…
## $ genomic_read_quality_mean <dbl> 36.62988, 36.87…
## $ genomic_read_quality_variance <dbl> 25.99015, 20.19…
## $ genomic_reads_fraction_bases_quality_above_30_mean <dbl> 0.8584288, 0.86…
## $ genomic_reads_fraction_bases_quality_above_30_variance <dbl> 0.03981779, 0.0…
## $ input_id <chr> "58a18a4c-5423-…
## $ molecule_barcode_fraction_bases_above_30_mean <dbl> 0.9820324, 0.98…
## $ molecule_barcode_fraction_bases_above_30_variance <dbl> 0.005782884, 0.…
## $ molecules_with_single_read_evidence <int> 276, 5028, 2041…
## $ n_fragments <int> 552, 202060, 84…
## $ n_genes <int> 227, 3381, 1826…
## $ n_mitochondrial_genes <int> 5, 22, 17, 5, 2…
## $ n_mitochondrial_molecules <int> 8, 3528, 2928, …
## $ n_molecules <int> 326, 23902, 998…
## $ n_reads <int> 679, 341669, 13…
## $ noise_reads <int> 0, 0, 0, 0, 0, …
## $ pct_mitochondrial_molecules <dbl> 1.1782032, 1.03…
## $ perfect_cell_barcodes <int> 667, 336674, 13…
## $ perfect_molecule_barcodes <int> 384, 227854, 89…
## $ reads_mapped_exonic <int> 343, 210716, 84…
## $ reads_mapped_intergenic <int> 39, 19439, 8042…
## $ reads_mapped_intronic <int> 175, 58833, 276…
## $ reads_mapped_multiple <int> 162, 90968, 365…
## $ reads_mapped_too_many_loci <int> 0, 0, 0, 0, 0, …
## $ reads_mapped_uniquely <int> 450, 227309, 93…
## $ reads_mapped_utr <int> 55, 29289, 1039…
## $ reads_per_fragment <dbl> 1.230072, 1.690…
## $ reads_unmapped <int> 67, 23392, 9340…
## $ spliced_reads <int> 99, 73817, 2987….loom
fileThe function optimus_loom_annotation() takes in the file
path of a .loom file generated by the Optimus
pipeline and returns a LoomExperiment object whose
colData has been annotated with additional specimen data
extracted from a manifest.
annotated_loom <- optimus_loom_annotation(file_location)
## requesting manifest
## waiting 1 more seconds (1 seconds total)...
annotated_loom
## class: SingleCellLoomExperiment
## dim: 58347 91713
## metadata(16): last_modified CreationDate ...
## specimen_from_organism.organ manifest
## assays(1): matrix
## rownames: NULL
## rowData names(29): Gene antisense_reads ... reads_per_molecule
## spliced_reads
## colnames: NULL
## colData names(98): input_id CellID ...
## sequencing_input.biomaterial_core.biomaterial_id
## sequencing_input_type
## reducedDimNames(0):
## mainExpName: NULL
## altExpNames(0):
## rowGraphs(0): NULL
## colGraphs(0): NULL
## new metadata
setdiff(
names(metadata(annotated_loom)),
names(metadata(loom))
)
## [1] "manifest"
metadata(annotated_loom)$manifest
## # A tibble: 4 × 56
## source_id source_spec bundle_uuid bundle_version file_document_id
## <chr> <chr> <chr> <dttm> <chr>
## 1 2bf54baa-538e-47… tdr:bigque… e3ecdfc2-4… 2021-02-02 23:55:00 5bc232f2-b77c-5…
## 2 2bf54baa-538e-47… tdr:bigque… 81df106e-e… 2021-02-02 23:55:00 9f8bc032-6276-5…
## 3 2bf54baa-538e-47… tdr:bigque… 54fb0e25-5… 2021-02-02 23:55:00 dfd9905b-d6c9-5…
## 4 2bf54baa-538e-47… tdr:bigque… 7516565a-e… 2021-02-02 23:55:00 e07ca731-b20a-5…
## # ℹ 51 more variables: file_type <chr>, file_name <chr>, file_format <chr>,
## # read_index <chr>, file_size <dbl>, file_uuid <chr>, file_version <dttm>,
## # file_crc32c <chr>, file_sha256 <chr>, file_content_type <chr>,
## # file_drs_uri <chr>, file_url <chr>,
## # cell_suspension.provenance.document_id <chr>,
## # cell_suspension.biomaterial_core.biomaterial_id <chr>,
## # cell_suspension.estimated_cell_count <lgl>, …
## new colData columns
setdiff(
names(colData(annotated_loom)),
names(colData(loom))
)
## [1] "source_id"
## [2] "source_spec"
## [3] "bundle_uuid"
## [4] "bundle_version"
## [5] "file_document_id"
## [6] "file_type"
## [7] "file_name"
## [8] "file_format"
## [9] "read_index"
## [10] "file_size"
## [11] "file_uuid"
## [12] "file_version"
## [13] "file_crc32c"
## [14] "file_sha256"
## [15] "file_content_type"
## [16] "file_drs_uri"
## [17] "file_url"
## [18] "cell_suspension.provenance.document_id"
## [19] "cell_suspension.biomaterial_core.biomaterial_id"
## [20] "cell_suspension.estimated_cell_count"
## [21] "cell_suspension.selected_cell_type"
## [22] "sequencing_protocol.instrument_manufacturer_model"
## [23] "sequencing_protocol.paired_end"
## [24] "library_preparation_protocol.library_construction_approach"
## [25] "library_preparation_protocol.nucleic_acid_source"
## [26] "project.provenance.document_id"
## [27] "project.contributors.institution"
## [28] "project.contributors.laboratory"
## [29] "project.project_core.project_short_name"
## [30] "project.project_core.project_title"
## [31] "project.estimated_cell_count"
## [32] "specimen_from_organism.provenance.document_id"
## [33] "specimen_from_organism.diseases"
## [34] "specimen_from_organism.organ"
## [35] "specimen_from_organism.organ_part"
## [36] "specimen_from_organism.preservation_storage.preservation_method"
## [37] "donor_organism.sex"
## [38] "donor_organism.biomaterial_core.biomaterial_id"
## [39] "donor_organism.provenance.document_id"
## [40] "donor_organism.genus_species"
## [41] "donor_organism.development_stage"
## [42] "donor_organism.diseases"
## [43] "donor_organism.organism_age"
## [44] "cell_line.provenance.document_id"
## [45] "cell_line.biomaterial_core.biomaterial_id"
## [46] "organoid.provenance.document_id"
## [47] "organoid.biomaterial_core.biomaterial_id"
## [48] "organoid.model_organ"
## [49] "organoid.model_organ_part"
## [50] "_entity_type"
## [51] "sample.provenance.document_id"
## [52] "sample.biomaterial_core.biomaterial_id"
## [53] "sequencing_input.provenance.document_id"
## [54] "sequencing_input.biomaterial_core.biomaterial_id"
## [55] "sequencing_input_type"sessionInfo()
## R version 4.4.2 (2024-10-31)
## Platform: x86_64-pc-linux-gnu
## Running under: Ubuntu 24.04.1 LTS
##
## Matrix products: default
## BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
## LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so; LAPACK version 3.12.0
##
## locale:
## [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
## [3] LC_TIME=en_US.UTF-8 LC_COLLATE=C
## [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
## [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
## [9] LC_ADDRESS=C LC_TELEPHONE=C
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
##
## time zone: Etc/UTC
## tzcode source: system (glibc)
##
## attached base packages:
## [1] stats4 stats graphics grDevices utils datasets methods
## [8] base
##
## other attached packages:
## [1] hca_1.15.0 LoomExperiment_1.25.0
## [3] BiocIO_1.17.1 rhdf5_2.51.0
## [5] SingleCellExperiment_1.29.1 SummarizedExperiment_1.37.0
## [7] Biobase_2.67.0 GenomicRanges_1.59.1
## [9] GenomeInfoDb_1.43.2 IRanges_2.41.1
## [11] S4Vectors_0.45.2 BiocGenerics_0.53.3
## [13] generics_0.1.3 MatrixGenerics_1.19.0
## [15] matrixStats_1.4.1 dplyr_1.1.4
## [17] BiocStyle_2.35.0
##
## loaded via a namespace (and not attached):
## [1] tidyselect_1.2.1 blob_1.2.4 filelock_1.0.3
## [4] fastmap_1.2.0 BiocFileCache_2.15.0 promises_1.3.2
## [7] digest_0.6.37 mime_0.12 lifecycle_1.0.4
## [10] RSQLite_2.3.8 magrittr_2.0.3 compiler_4.4.2
## [13] rlang_1.1.4 sass_0.4.9 tools_4.4.2
## [16] utf8_1.2.4 yaml_2.3.10 knitr_1.49
## [19] S4Arrays_1.7.1 htmlwidgets_1.6.4 bit_4.5.0
## [22] curl_6.0.1 DelayedArray_0.33.2 abind_1.4-8
## [25] miniUI_0.1.1.1 HDF5Array_1.35.1 withr_3.0.2
## [28] purrr_1.0.2 sys_3.4.3 grid_4.4.2
## [31] fansi_1.0.6 xtable_1.8-4 Rhdf5lib_1.29.0
## [34] cli_3.6.3 rmarkdown_2.29 crayon_1.5.3
## [37] httr_1.4.7 tzdb_0.4.0 DBI_1.2.3
## [40] cachem_1.1.0 stringr_1.5.1 zlibbioc_1.52.0
## [43] parallel_4.4.2 BiocManager_1.30.25 XVector_0.47.0
## [46] vctrs_0.6.5 Matrix_1.7-1 jsonlite_1.8.9
## [49] hms_1.1.3 bit64_4.5.2 maketools_1.3.1
## [52] jquerylib_0.1.4 tidyr_1.3.1 glue_1.8.0
## [55] DT_0.33 stringi_1.8.4 later_1.4.1
## [58] UCSC.utils_1.3.0 tibble_3.2.1 pillar_1.9.0
## [61] htmltools_0.5.8.1 rhdf5filters_1.19.0 GenomeInfoDbData_1.2.13
## [64] R6_2.5.1 dbplyr_2.5.0 vroom_1.6.5
## [67] evaluate_1.0.1 shiny_1.9.1 lattice_0.22-6
## [70] readr_2.1.5 memoise_2.0.1 httpuv_1.6.15
## [73] bslib_0.8.0 Rcpp_1.0.13-1 SparseArray_1.7.2
## [76] xfun_0.49 buildtools_1.0.0 pkgconfig_2.0.3