Introduction | Install and load the required packages | The test datasets | Sparse vs dense representation | TENxMatrix vs HDF5Matrix objects | Bring the sparse dataset in R | Bring the dense dataset in R | Create the test datasets | Block-processed normalization and PCA | Code used for normalization and PCA | Block processing and block size | Monitoring memory usage | Normalization and PCA of the 27,998 x 12,500 test dataset | Normalization | TENxMatrix (sparse) | HDF5Matrix (dense) | HDF5Matrix as sparse | On-disk realization of the normalized datasets | PCA | Comprehensive timings obtained on various machines | Timings for DELL XPS 15 laptop | Timings for Supermicro SuperServer 1029GQ-TRT | Timings for Apple Silicon Mac Pro | Timings for Intel Mac Pro | Discussion | TENxMatrix (sparse) vs HDF5Matrix (dense) | Hybrid approach | A note about memory usage | Main takeaways | Session information
Package: HDF5Array 1.41.0
Hervé Pagès
HDF5Array: HDF5 datasets as array-like objects in R
The HDF5Array package is an HDF5 backend for DelayedArray objects. It implements the HDF5Array, H5SparseMatrix, H5ADMatrix, and TENxMatrix classes, 4 convenient and memory-efficient array-like containers for representing and manipulating either: (1) a conventional (a.k.a. dense) HDF5 dataset, (2) an HDF5 sparse matrix (stored in CSR/CSC/Yale format), (3) the central matrix of an h5ad file (or any matrix in the /layers group), or (4) a 10x Genomics sparse matrix. All these containers are DelayedArray extensions and thus support all operations (delayed or block-processed) supported by DelayedArray objects.
Authors:
HDF5Array_1.41.0.tar.gz
HDF5Array_1.41.0.zip(r-4.7)HDF5Array_1.41.0.zip(r-4.6)HDF5Array_1.41.0.zip(r-4.5)
HDF5Array_1.41.0.tgz(r-4.6-any)HDF5Array_1.41.0.tgz(r-4.5-any)
HDF5Array_1.41.0.tar.gz(r-4.7-any)HDF5Array_1.41.0.tar.gz(r-4.6-any)
HDF5Array_1.41.0.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
DESCRIPTION |NEWS
card.svg |card.png
HDF5Array/json (API)
| # Install 'HDF5Array' in R: |
| install.packages('HDF5Array', repos = c('https://bioc.r-universe.dev', 'https://cloud.r-project.org')) |
Bug tracker:https://github.com/bioconductor/hdf5array/issues
On BioConductor:HDF5Array-1.41.0(bioc 3.24)HDF5Array-1.40.0(bioc 3.23)
infrastructuredatarepresentationdataimportsequencingrnaseqcoverageannotationgenomeannotationsinglecellimmunooncologybioconductor-packagecore-packageu24ca289073
Last updated from:e6c2cb3b95. Checks:8 NOTE, 2 OK. Indexed: yes.
| Target | Result | Time | Files | Syslog |
|---|---|---|---|---|
| bioc-checks | NOTE | 221 | ||
| linux-devel-x86_64 | NOTE | 1294 | ||
| source / vignettes | OK | 823 | ||
| linux-release-x86_64 | NOTE | 1272 | ||
| macos-release-arm64 | NOTE | 818 | ||
| macos-oldrel-arm64 | NOTE | 1130 | ||
| windows-devel | NOTE | 514 | ||
| windows-release | NOTE | 522 | ||
| windows-oldrel | NOTE | 3245 | ||
| wasm-release | OK | 187 |
Exports:appendDatasetCreationToHDF5DumpLogcheck_and_delete_fileschunkdimclosecoercecreate_dirDelayedArrayextract_arrayextract_sparse_arrayextractNonzeroDataByColextractNonzeroDataByRowgetHDF5DumpChunkDimgetHDF5DumpChunkLengthgetHDF5DumpChunkShapegetHDF5DumpCompressionLevelgetHDF5DumpDirgetHDF5DumpFilegetHDF5DumpNameH5ADMatrixH5ADMatrixSeedH5SparseMatrixH5SparseMatrixSeedHDF5ArrayHDF5ArraySeedHDF5RealizationSinkis_sparseis_sparse<-loadHDF5SummarizedExperimentlsHDF5DumpFilematrixClassnzcountpathpath<-quickResaveHDF5SummarizedExperimentreplace_dirReshapedHDF5ArrayReshapedHDF5ArraySeedrestore_absolute_assay2h5_linkssaveHDF5SummarizedExperimentsetHDF5DumpChunkLengthsetHDF5DumpChunkShapesetHDF5DumpCompressionLevelsetHDF5DumpDirsetHDF5DumpFilesetHDF5DumpNameshorten_assay2h5_linksshowshowHDF5DumpLogstop_if_bad_dirtt.CSC_H5ADMatrixSeedt.CSC_H5SparseMatrixSeedt.CSR_H5ADMatrixSeedt.CSR_H5SparseMatrixSeedTENxMatrixTENxMatrixSeedTENxRealizationSinktypeupdateObjectvalidate_HDF5ArraySeed_dataset_geometrywrite_blockwrite_h5_assayswriteHDF5ArraywriteTENxMatrix
Dependencies:abindBiocGenericsbiocmakeDelayedArraydir.expiryfilelockgenericsh5mreadIRangeslatticeMatrixMatrixGenericsmatrixStatsrhdf5rhdf5filtersRhdf5libS4ArraysS4VectorsSparseArrayXVector
Readme and manuals
Help Manual
| Help page | Topics |
|---|---|
| h5ad central matrices (or matrices in the /layers group) as DelayedMatrix objects | class:H5ADMatrix DelayedArray,H5ADMatrixSeed-method extractNonzeroDataByCol,H5ADMatrix-method extractNonzeroDataByRow,H5ADMatrix-method H5ADMatrix H5ADMatrix-class nzcount,H5ADMatrix-method |
| H5ADMatrixSeed objects | class:CSC_H5ADMatrixSeed class:CSR_H5ADMatrixSeed class:Dense_H5ADMatrixSeed class:H5ADMatrixSeed CSC_H5ADMatrixSeed CSC_H5ADMatrixSeed-class CSR_H5ADMatrixSeed CSR_H5ADMatrixSeed-class Dense_H5ADMatrixSeed Dense_H5ADMatrixSeed-class dimnames,Dense_H5ADMatrixSeed-method H5ADMatrixSeed H5ADMatrixSeed-class t,CSC_H5ADMatrixSeed-method t,CSR_H5ADMatrixSeed-method t.CSC_H5ADMatrixSeed t.CSR_H5ADMatrixSeed |
| HDF5 sparse matrices as DelayedMatrix objects | class:H5SparseMatrix DelayedArray,H5SparseMatrixSeed-method extractNonzeroDataByCol,H5SparseMatrix-method extractNonzeroDataByRow,H5SparseMatrix-method H5SparseMatrix H5SparseMatrix-class nzcount,H5SparseMatrix-method |
| H5SparseMatrixSeed objects | chunkdim,CSC_H5SparseMatrixSeed-method chunkdim,CSR_H5SparseMatrixSeed-method class:CSC_H5SparseMatrixSeed class:CSR_H5SparseMatrixSeed class:H5SparseMatrixSeed CSC_H5SparseMatrixSeed CSC_H5SparseMatrixSeed-class CSR_H5SparseMatrixSeed CSR_H5SparseMatrixSeed-class dim,H5SparseMatrixSeed-method dimnames,H5SparseMatrixSeed-method extractNonzeroDataByCol extractNonzeroDataByCol,CSC_H5SparseMatrixSeed-method extractNonzeroDataByRow extractNonzeroDataByRow,CSR_H5SparseMatrixSeed-method extract_array,H5SparseMatrixSeed-method extract_sparse_array,CSC_H5SparseMatrixSeed-method extract_sparse_array,CSR_H5SparseMatrixSeed-method H5SparseMatrixSeed H5SparseMatrixSeed-class is_sparse,H5SparseMatrixSeed-method nzcount,H5SparseMatrixSeed-method path,H5SparseMatrixSeed-method path<-,H5SparseMatrixSeed-method show,H5SparseMatrixSeed-method t,CSC_H5SparseMatrixSeed-method t,CSR_H5SparseMatrixSeed-method t.CSC_H5SparseMatrixSeed t.CSR_H5SparseMatrixSeed |
| HDF5 dump management | appendDatasetCreationToHDF5DumpLog dump-management getHDF5DumpChunkDim getHDF5DumpChunkLength getHDF5DumpChunkShape getHDF5DumpCompressionLevel getHDF5DumpDir getHDF5DumpFile getHDF5DumpName HDF5-dump-management lsHDF5DumpFile setHDF5DumpChunkLength setHDF5DumpChunkShape setHDF5DumpCompressionLevel setHDF5DumpDir setHDF5DumpFile setHDF5DumpName showHDF5DumpLog |
| HDF5 datasets as DelayedArray objects | class:HDF5Array class:HDF5Matrix coerce,ANY,HDF5Matrix-method coerce,HDF5Array,HDF5Matrix-method coerce,HDF5Matrix,HDF5Array-method DelayedArray,HDF5ArraySeed-method HDF5Array HDF5Array-class HDF5Matrix HDF5Matrix-class is_sparse<-,HDF5Array-method matrixClass,HDF5Array-method |
| HDF5ArraySeed objects | chunkdim,HDF5ArraySeed-method class:HDF5ArraySeed dim,HDF5ArraySeed-method dimnames,HDF5ArraySeed-method extract_array,HDF5ArraySeed-method extract_sparse_array,HDF5ArraySeed-method HDF5ArraySeed HDF5ArraySeed-class is_sparse,HDF5ArraySeed-method is_sparse<-,HDF5ArraySeed-method path,HDF5ArraySeed-method path<-,HDF5ArraySeed-method type,HDF5ArraySeed-method updateObject,HDF5ArraySeed-method |
| Virtually reshaped HDF5 datasets as DelayedArray objects | class:ReshapedHDF5Array class:ReshapedHDF5Matrix coerce,ANY,ReshapedHDF5Matrix-method coerce,ReshapedHDF5Array,ReshapedHDF5Matrix-method coerce,ReshapedHDF5Matrix,ReshapedHDF5Array-method DelayedArray,ReshapedHDF5ArraySeed-method matrixClass,ReshapedHDF5Array-method ReshapedHDF5Array ReshapedHDF5Array-class ReshapedHDF5Matrix ReshapedHDF5Matrix-class |
| ReshapedHDF5ArraySeed objects | chunkdim,ReshapedHDF5ArraySeed-method class:ReshapedHDF5ArraySeed dim,ReshapedHDF5ArraySeed-method extract_array,ReshapedHDF5ArraySeed-method ReshapedHDF5ArraySeed ReshapedHDF5ArraySeed-class |
| Save/load an HDF5-based SummarizedExperiment object | loadHDF5SummarizedExperiment quickResaveHDF5SummarizedExperiment saveHDF5SummarizedExperiment |
| 10x Genomics datasets as DelayedMatrix objects | class:TENxMatrix DelayedArray,TENxMatrixSeed-method extractNonzeroDataByCol,TENxMatrix-method nzcount,TENxMatrix-method TENxMatrix TENxMatrix-class |
| TENxMatrixSeed objects | class:TENxMatrixSeed TENxMatrixSeed TENxMatrixSeed-class |
| Write an array-like object to an HDF5 file | chunkdim,HDF5RealizationSink-method class:HDF5RealizationSink coerce,ANY,HDF5Array-method coerce,DelayedArray,HDF5Array-method coerce,DelayedMatrix,HDF5Matrix-method coerce,HDF5RealizationSink,DelayedArray-method coerce,HDF5RealizationSink,HDF5Array-method coerce,HDF5RealizationSink,HDF5ArraySeed-method dimnames,HDF5RealizationSink-method HDF5RealizationSink HDF5RealizationSink-class is_sparse,HDF5RealizationSink-method type,HDF5RealizationSink-method writeHDF5Array write_block,HDF5RealizationSink-method |
| Write a matrix-like object as an HDF5-based sparse matrix | chunkdim,TENxRealizationSink-method class:TENxRealizationSink close,TENxRealizationSink-method coerce,ANY,TENxMatrix-method coerce,DelayedArray,TENxMatrix-method coerce,DelayedMatrix,TENxMatrix-method coerce,TENxRealizationSink,DelayedArray-method coerce,TENxRealizationSink,TENxMatrix-method coerce,TENxRealizationSink,TENxMatrixSeed-method dimnames,TENxRealizationSink-method TENxRealizationSink TENxRealizationSink-class type,TENxRealizationSink-method writeTENxMatrix write_block,TENxRealizationSink-method |
