How to use iSEE with big data
Compiled date: 2024-10-30
Last edited: 2018-03-08
License: MIT + file LICENSE
Overview
Some tweaks can be performed to enable iSEE to run efficiently on large datasets. This includes datasets with many features (methylation, SNPs) or many columns (cytometry, single-cell RNA-seq). To demonstrate some of this functionality, we will use a dataset from the TENxPBMCData dataset:
library(TENxPBMCData)
sce.pbmc <- TENxPBMCData("pbmc68k")
sce.pbmc$Library <- factor(sce.pbmc$Library)
sce.pbmc
#> class: SingleCellExperiment
#> dim: 32738 68579
#> metadata(0):
#> assays(1): counts
#> rownames(32738): ENSG00000243485 ENSG00000237613 ... ENSG00000215616
#> ENSG00000215611
#> rowData names(3): ENSEMBL_ID Symbol_TENx Symbol
#> colnames: NULL
#> colData names(11): Sample Barcode ... Individual Date_published
#> reducedDimNames(0):
#> mainExpName: NULL
#> altExpNames(0):Using out-of-memory matrices
Many SummarizedExperiment objects store assay matrices
as in-memory matrix-like objects, be they ordinary matrices or
alternative representations such as sparse matrices from the Matrix
package. For example, if we looked at the Allen data, we would see that
the counts are stored as an ordinary matrix.
library(scRNAseq)
sce.allen <- ReprocessedAllenData("tophat_counts")
class(assay(sce.allen, "tophat_counts"))
#> [1] "matrix" "array"In situations involving large datasets and limited computational
resources, storing the entire assay in memory may not be feasible.
Rather, we can represent the data as a file-backed matrix where contents
are stored on disk and retrieved on demand. Within the Bioconductor
ecosystem, the easiest way of doing this is to create a
HDF5Matrix, which uses a HDF5 file
to store all the assay data. We see that this has already been done for
use in the 68K PBMC dataset:
counts(sce.pbmc, withDimnames=FALSE)
#> <32738 x 68579> sparse HDF5Matrix object of type "integer":
#> [,1] [,2] [,3] [,4] ... [,68576] [,68577] [,68578]
#> [1,] 0 0 0 0 . 0 0 0
#> [2,] 0 0 0 0 . 0 0 0
#> [3,] 0 0 0 0 . 0 0 0
#> [4,] 0 0 0 0 . 0 0 0
#> [5,] 0 0 0 0 . 0 0 0
#> ... . . . . . . . .
#> [32734,] 0 0 0 0 . 0 0 0
#> [32735,] 0 0 0 0 . 0 0 0
#> [32736,] 0 0 0 0 . 0 0 0
#> [32737,] 0 0 0 0 . 0 0 0
#> [32738,] 0 0 0 0 . 0 0 0
#> [,68579]
#> [1,] 0
#> [2,] 0
#> [3,] 0
#> [4,] 0
#> [5,] 0
#> ... .
#> [32734,] 0
#> [32735,] 0
#> [32736,] 0
#> [32737,] 0
#> [32738,] 0Despite the dimensions of this matrix, the HDF5Matrix
object occupies very little space in memory.
However, parts of the data can still be read in on demand. For all
intents and purposes, the HDF5Matrix appears to be an
ordinary matrix to downstream applications and can be used as such.
This means that we can use the 68K PBMC
SingleCellExperiment object in iSEE() without
any extra work. The app below shows the distribution of counts for
everyone’s favorite gene MALAT1 across libraries. Here,
iSEE() is simply retrieving data on demand from the
HDF5Matrix without ever loading the entire assay matrix
into memory. This enables it to run efficiently on arbitrary large
datasets with limited resources.
library(iSEE)
app <- iSEE(sce.pbmc, initial=
list(RowDataTable(Selected="ENSG00000251562", Search="MALAT1"),
FeatureAssayPlot(XAxis="Column data", XAxisColumnData="Library",
YAxisFeatureSource="RowDataTable1")
)
)
Generally speaking, these HDF5 files are written once by a process
with sufficient computational resources (i.e., memory and time). We
typically create HDF5Matrix objects using the
writeHDF5Array() function from the HDF5Array
package. After the file is created, the objects can be read many times
in more deprived environments.
sce.h5 <- sce.allen
library(HDF5Array)
assay(sce.h5, "tophat_counts", withDimnames=FALSE) <-
writeHDF5Array(assay(sce.h5, "tophat_counts"), file="assay.h5", name="counts")
class(assay(sce.h5, "tophat_counts", withDimnames=FALSE))
#> [1] "HDF5Matrix"
#> attr(,"package")
#> [1] "HDF5Array"
list.files("assay.h5")
#> character(0)It is worth noting that iSEE() does not know or care
that the data is stored in a HDF5 file. The app is fully compatible with
any matrix-like representation of the assay data that supports
dim() and [,. As such, iSEE() can
be directly used with other memory-efficient objects like the
DeferredMatrix and LowRankMatrix from the
BiocSingular
package, or perhaps the ResidualMatrix from the batchelor
package.
Downsampling points
It is also possible to downsample points to reduce the time required to generate the plot. This involves subsetting the dataset so that only the most recently plotted point for an overlapping set of points is shown. In this manner, we avoid wasting time in plotting many points that would not be visible anyway. To demonstrate, we will re-use the 68K PBMC example and perform downsampling on the feature assay plot; we can see that its aesthetics are largely similar to the non-downsampled counterpart above.
library(iSEE)
app <- iSEE(sce.pbmc, initial=
list(RowDataTable(Selected="ENSG00000251562", Search="MALAT1"),
FeatureAssayPlot(XAxis="Column data", XAxisColumnData="Library",
YAxisFeatureSource="RowDataTable1",
VisualChoices="Point", Downsample=TRUE,
VisualBoxOpen=TRUE
)
)
)
Downsampling is possible in all iSEE() plotting panels
that represent features or samples as points. We can turn on
downsampling for all such panels using the relevant field in
panelDefaults(), which spares us the hassle of setting
Downsample= individually in each panel constructor.
The downsampling only affects the visualization and the speed of the plot rendering. Any interactions with other panels occur as if all of the points were still there. For example, if one makes a brush, all of the points therein will be selected regardless of whether they were downsampled.
The downsampling resolution determines the degree to which points are considered to be overlapping. Decreasing the resolution will downsample more aggressively, improving plotting speed but potentially affecting the fidelity of the visualization. This may compromise the aesthetics of the plot when the size of the points is small, in which case an increase in resolution may be required at the cost of speed.
Obviously, downsampling will not preserve overlays for partially transparent points, but any reliance on partial transparency is probably not a good idea in the first place when there are many points.
Changing the interface
One can generally improve the speed of the iSEE()
interface by only initializing the app with the desired panels. For
example, it makes little sense to spend time rendering a
RowDataPlot when only the ReducedDimensionPlot
is of interest. Specification of the initial state is straightforward
with the initial= argument, as described in a previous
vignette.
On occasion, there may be alternative panels with more efficient
visualizations for the same data. The prime example is the
ReducedDimensionHexPlot class from the iSEEu
package; this will create a hexplot rather than a scatter plot, thus
avoiding the need to render each point in the latter.
Session Info
sessionInfo()
#> R version 4.4.1 (2024-06-14)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 24.04.1 LTS
#>
#> Matrix products: default
#> BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so; LAPACK version 3.12.0
#>
#> locale:
#> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
#> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=C
#> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
#> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
#> [9] LC_ADDRESS=C LC_TELEPHONE=C
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
#>
#> time zone: Etc/UTC
#> tzcode source: system (glibc)
#>
#> attached base packages:
#> [1] stats4 stats graphics grDevices utils datasets methods
#> [8] base
#>
#> other attached packages:
#> [1] TENxPBMCData_1.23.0 HDF5Array_1.33.8
#> [3] rhdf5_2.49.0 DelayedArray_0.33.1
#> [5] SparseArray_1.5.45 S4Arrays_1.5.11
#> [7] abind_1.4-8 Matrix_1.7-1
#> [9] scater_1.33.4 ggplot2_3.5.1
#> [11] scuttle_1.15.5 scRNAseq_2.19.1
#> [13] iSEE_2.19.0 SingleCellExperiment_1.27.2
#> [15] SummarizedExperiment_1.35.5 Biobase_2.67.0
#> [17] GenomicRanges_1.57.2 GenomeInfoDb_1.41.2
#> [19] IRanges_2.39.2 S4Vectors_0.43.2
#> [21] BiocGenerics_0.53.0 MatrixGenerics_1.17.1
#> [23] matrixStats_1.4.1 BiocStyle_2.35.0
#>
#> loaded via a namespace (and not attached):
#> [1] splines_4.4.1 later_1.3.2 BiocIO_1.17.0
#> [4] bitops_1.0-9 filelock_1.0.3 tibble_3.2.1
#> [7] XML_3.99-0.17 lifecycle_1.0.4 httr2_1.0.5
#> [10] doParallel_1.0.17 lattice_0.22-6 ensembldb_2.29.1
#> [13] alabaster.base_1.7.0 magrittr_2.0.3 sass_0.4.9
#> [16] rmarkdown_2.28 jquerylib_0.1.4 yaml_2.3.10
#> [19] httpuv_1.6.15 DBI_1.2.3 buildtools_1.0.0
#> [22] RColorBrewer_1.1-3 zlibbioc_1.51.2 Rtsne_0.17
#> [25] purrr_1.0.2 AnnotationFilter_1.31.0 RCurl_1.98-1.16
#> [28] rappdirs_0.3.3 circlize_0.4.16 GenomeInfoDbData_1.2.13
#> [31] ggrepel_0.9.6 irlba_2.3.5.1 alabaster.sce_1.7.0
#> [34] maketools_1.3.1 codetools_0.2-20 DT_0.33
#> [37] tidyselect_1.2.1 shape_1.4.6.1 UCSC.utils_1.1.0
#> [40] ScaledMatrix_1.13.0 viridis_0.6.5 shinyWidgets_0.8.7
#> [43] BiocFileCache_2.15.0 GenomicAlignments_1.41.0 jsonlite_1.8.9
#> [46] GetoptLong_1.0.5 BiocNeighbors_2.1.0 iterators_1.0.14
#> [49] foreach_1.5.2 tools_4.4.1 Rcpp_1.0.13
#> [52] glue_1.8.0 gridExtra_2.3 xfun_0.48
#> [55] mgcv_1.9-1 dplyr_1.1.4 gypsum_1.1.6
#> [58] shinydashboard_0.7.2 withr_3.0.2 BiocManager_1.30.25
#> [61] fastmap_1.2.0 rhdf5filters_1.17.0 fansi_1.0.6
#> [64] shinyjs_2.1.0 digest_0.6.37 rsvd_1.0.5
#> [67] R6_2.5.1 mime_0.12 colorspace_2.1-1
#> [70] listviewer_4.0.0 RSQLite_2.3.7 utf8_1.2.4
#> [73] generics_0.1.3 rtracklayer_1.65.0 httr_1.4.7
#> [76] htmlwidgets_1.6.4 pkgconfig_2.0.3 gtable_0.3.6
#> [79] blob_1.2.4 ComplexHeatmap_2.23.0 XVector_0.45.0
#> [82] sys_3.4.3 htmltools_0.5.8.1 ProtGenerics_1.37.1
#> [85] rintrojs_0.3.4 clue_0.3-65 scales_1.3.0
#> [88] alabaster.matrix_1.7.0 png_0.1-8 knitr_1.48
#> [91] rjson_0.2.23 nlme_3.1-166 curl_5.2.3
#> [94] shinyAce_0.4.3 cachem_1.1.0 GlobalOptions_0.1.2
#> [97] BiocVersion_3.21.1 parallel_4.4.1 miniUI_0.1.1.1
#> [100] vipor_0.4.7 AnnotationDbi_1.69.0 restfulr_0.0.15
#> [103] pillar_1.9.0 grid_4.4.1 alabaster.schemas_1.7.0
#> [106] vctrs_0.6.5 promises_1.3.0 BiocSingular_1.23.0
#> [109] dbplyr_2.5.0 beachmat_2.23.0 xtable_1.8-4
#> [112] cluster_2.1.6 beeswarm_0.4.0 evaluate_1.0.1
#> [115] GenomicFeatures_1.57.1 cli_3.6.3 compiler_4.4.1
#> [118] Rsamtools_2.21.2 rlang_1.1.4 crayon_1.5.3
#> [121] ggbeeswarm_0.7.2 viridisLite_0.4.2 alabaster.se_1.7.0
#> [124] BiocParallel_1.41.0 munsell_0.5.1 Biostrings_2.75.0
#> [127] lazyeval_0.2.2 colourpicker_1.3.0 ExperimentHub_2.13.1
#> [130] bit64_4.5.2 Rhdf5lib_1.27.0 KEGGREST_1.45.1
#> [133] shiny_1.9.1 highr_0.11 alabaster.ranges_1.7.0
#> [136] AnnotationHub_3.15.0 fontawesome_0.5.2 igraph_2.1.1
#> [139] memoise_2.0.1 bslib_0.8.0 bit_4.5.0
# devtools::session_info()
Comments on deployment
It is straightforward to host iSEE applications on hosting platforms like Shiny Server or Rstudio Connect. All one needs to do is to create an
app.Rfile that callsiSEE()with the desired parameters, and then follow the instructions for the target platform. For a better user experience, we suggest setting a minimum number of processes to avoid the initial delay from R start-up.It is also possible to deploy and host Shiny app on shinyapps.io, a platform as a service (PaaS) provided by RStudio. In many cases, users will need to configure the settings of their deployed apps, in particular selecting larger instances to provide sufficient memory for the app. The maximum amount of 1GB available to free accounts may not be sufficient to deploy large datasets; in which case you may consider using out-of-memory matrices, filtering your dataset (e.g., removing lowly detected features), or going for a paid account. Detailed instructions to get started are available at https://shiny.rstudio.com/articles/shinyapps.html. For example, see the isee-shiny-contest app, winner of the 1st Shiny Contest.