Compiled date: 2024-12-11
Last edited: 2018-11-29
License: MIT + file LICENSE
Using JavaScript, iSEE applications can leverage
lightweight speech recognition libraries that react to specific vocal
commands (think “OK Google”, “Hey Siri”) and trigger updates of the UI
equivalent to one or more mouse or keyboard interaction with the UI
components (Rue-Albrecht et al. 2018).
Note: As we value privacy, this feature is disabled
by default: iSEE(..., voice=FALSE).
To keep the spoken commands reasonably short, only one panel may be under voice command at any one time. All spoken commands will affect the currently active panel, until a new panel is selected for voice command. See section Vocal commands available.
We use the annyang lightweight JavaScript library to handle speech recognition and update Shiny reactive values in the same way as mouse and keyboard UI elements trigger panel updates.
Note that annyang requires an active internet connection, as it relies on the browser’s own speech recognition engine (see the annyang FAQ). For instance, in Google Chrome, this engine performs the recognition in the cloud.
Note that the speech recognition library that we use does not work with every web browser. We currently only validated this feature in Google Chrome. Please refer to the annyang FAQ for details.
Using the sce object that we generated earlier,
enabling speech recognition is as simple as setting
voice=TRUE below:
With voice=TRUE, the lightweight JavaScript speech
recognition library annyang is loaded and activated in any web
browser tab that runs app.
If your default browser is not compatible with the feature, or if you
work in RStudio, you can prevent the application from opening
in the default browser by setting launch.browser=FALSE as
follows:
At that point, your R console should be displaying the address and
port where app is running. In the example above, that would
be:
Listening on http://127.0.0.1:1234Using a compatible browser, navigate to the indicated address and port. Note that when the web page opens, you may be prompted to allow the web browser to use your microphone, which you must accept to enable the functionality.
As a proof of concept, only a subset of spoken commands are currently implemented, compared to the full range of interactions possible using the mouse and keyboard.
Note that in the commands below, words in brackets are optional.
colData column name) of the
panel under vocal control.sessionInfo()
#> R version 4.4.2 (2024-10-31)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 24.04.1 LTS
#>
#> Matrix products: default
#> BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so; LAPACK version 3.12.0
#>
#> locale:
#> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
#> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=C
#> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
#> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
#> [9] LC_ADDRESS=C LC_TELEPHONE=C
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
#>
#> time zone: Etc/UTC
#> tzcode source: system (glibc)
#>
#> attached base packages:
#> [1] stats4 stats graphics grDevices utils datasets methods
#> [8] base
#>
#> other attached packages:
#> [1] TENxPBMCData_1.24.0 HDF5Array_1.35.2
#> [3] rhdf5_2.51.1 DelayedArray_0.33.3
#> [5] SparseArray_1.7.2 S4Arrays_1.7.1
#> [7] abind_1.4-8 Matrix_1.7-1
#> [9] scater_1.35.0 ggplot2_3.5.1
#> [11] scuttle_1.17.0 scRNAseq_2.20.0
#> [13] iSEE_2.19.2 SingleCellExperiment_1.29.1
#> [15] SummarizedExperiment_1.37.0 Biobase_2.67.0
#> [17] GenomicRanges_1.59.1 GenomeInfoDb_1.43.2
#> [19] IRanges_2.41.2 S4Vectors_0.45.2
#> [21] BiocGenerics_0.53.3 generics_0.1.3
#> [23] MatrixGenerics_1.19.0 matrixStats_1.4.1
#> [25] BiocStyle_2.35.0
#>
#> loaded via a namespace (and not attached):
#> [1] splines_4.4.2 later_1.4.1 BiocIO_1.17.1
#> [4] bitops_1.0-9 filelock_1.0.3 tibble_3.2.1
#> [7] XML_3.99-0.17 lifecycle_1.0.4 httr2_1.0.7
#> [10] doParallel_1.0.17 lattice_0.22-6 ensembldb_2.31.0
#> [13] alabaster.base_1.7.2 magrittr_2.0.3 sass_0.4.9
#> [16] rmarkdown_2.29 jquerylib_0.1.4 yaml_2.3.10
#> [19] httpuv_1.6.15 DBI_1.2.3 buildtools_1.0.0
#> [22] RColorBrewer_1.1-3 zlibbioc_1.52.0 Rtsne_0.17
#> [25] purrr_1.0.2 AnnotationFilter_1.31.0 RCurl_1.98-1.16
#> [28] rappdirs_0.3.3 circlize_0.4.16 GenomeInfoDbData_1.2.13
#> [31] ggrepel_0.9.6 irlba_2.3.5.1 alabaster.sce_1.7.0
#> [34] maketools_1.3.1 codetools_0.2-20 DT_0.33
#> [37] tidyselect_1.2.1 shape_1.4.6.1 UCSC.utils_1.3.0
#> [40] ScaledMatrix_1.15.0 viridis_0.6.5 shinyWidgets_0.8.7
#> [43] BiocFileCache_2.15.0 GenomicAlignments_1.43.0 jsonlite_1.8.9
#> [46] GetoptLong_1.0.5 BiocNeighbors_2.1.2 iterators_1.0.14
#> [49] foreach_1.5.2 tools_4.4.2 Rcpp_1.0.13-1
#> [52] glue_1.8.0 gridExtra_2.3 xfun_0.49
#> [55] mgcv_1.9-1 dplyr_1.1.4 gypsum_1.3.0
#> [58] shinydashboard_0.7.2 withr_3.0.2 BiocManager_1.30.25
#> [61] fastmap_1.2.0 rhdf5filters_1.19.0 fansi_1.0.6
#> [64] shinyjs_2.1.0 digest_0.6.37 rsvd_1.0.5
#> [67] R6_2.5.1 mime_0.12 colorspace_2.1-1
#> [70] listviewer_4.0.0 RSQLite_2.3.9 utf8_1.2.4
#> [73] rtracklayer_1.67.0 httr_1.4.7 htmlwidgets_1.6.4
#> [76] pkgconfig_2.0.3 gtable_0.3.6 blob_1.2.4
#> [79] ComplexHeatmap_2.23.0 XVector_0.47.0 sys_3.4.3
#> [82] htmltools_0.5.8.1 ProtGenerics_1.39.0 rintrojs_0.3.4
#> [85] clue_0.3-66 scales_1.3.0 alabaster.matrix_1.7.4
#> [88] png_0.1-8 knitr_1.49 rjson_0.2.23
#> [91] nlme_3.1-166 curl_6.0.1 shinyAce_0.4.3
#> [94] cachem_1.1.0 GlobalOptions_0.1.2 BiocVersion_3.21.1
#> [97] parallel_4.4.2 miniUI_0.1.1.1 vipor_0.4.7
#> [100] AnnotationDbi_1.69.0 restfulr_0.0.15 pillar_1.9.0
#> [103] grid_4.4.2 alabaster.schemas_1.7.0 vctrs_0.6.5
#> [106] promises_1.3.2 BiocSingular_1.23.0 dbplyr_2.5.0
#> [109] beachmat_2.23.4 xtable_1.8-4 cluster_2.1.7
#> [112] beeswarm_0.4.0 evaluate_1.0.1 GenomicFeatures_1.59.1
#> [115] cli_3.6.3 compiler_4.4.2 Rsamtools_2.23.1
#> [118] rlang_1.1.4 crayon_1.5.3 ggbeeswarm_0.7.2
#> [121] viridisLite_0.4.2 alabaster.se_1.7.0 BiocParallel_1.41.0
#> [124] munsell_0.5.1 Biostrings_2.75.2 lazyeval_0.2.2
#> [127] colourpicker_1.3.0 ExperimentHub_2.15.0 bit64_4.5.2
#> [130] Rhdf5lib_1.29.0 KEGGREST_1.47.0 shiny_1.9.1
#> [133] alabaster.ranges_1.7.0 AnnotationHub_3.15.0 fontawesome_0.5.3
#> [136] igraph_2.1.2 memoise_2.0.1 bslib_0.8.0
#> [139] bit_4.5.0.1
# devtools::session_info()