Overview

The son.of.basilisk package (provided in the inst/example directory of this package) is provided as an example of how one might write a client package that depends on basilisk. This is a fully operational example package that can be installed and run, so prospective developers should use it as a template for their own packages. We will assume that readers are familiar with general R package development practices and will limit our discussion to the basilisk-specific elements.

Setting up the package

StagedInstall: no should be set, to ensure that Python packages are installed with the correct hard-coded paths within the R package installation directory.

Imports: basilisk should be set along with appropriate directives in the NAMESPACE for all basilisk functions that are used.

.BBSoptions should contain UnsupportedPlatforms: win32, as builds on Windows 32-bit are not supported.

Specifying environments

library(basilisk)
my_env <- BasiliskEnvironment(envname="my_env_name",
    pkgname="ClientPackage",
    packages=c("pandas==1.4.3", "scikit-learn==1.1.1")
)

second_env <- BasiliskEnvironment(envname="second_env_name",
    pkgname="ClientPackage",
    packages=c("scipy=1.9.1", "numpy==1.22.1") 
)

#!/bin/sh

${R_HOME}/bin/Rscript -e "basilisk::configureBasiliskEnv()"

#!/bin/sh

${R_HOME}/bin${R_ARCH_BIN}/Rscript.exe -e "basilisk::configureBasiliskEnv()"

Using the environments

my_example_function <- function(ARG_VALUE_1, ARG_VALUE_2) { 
    proc <- basiliskStart(my_env)
    on.exit(basiliskStop(proc))

    some_useful_thing <- basiliskRun(proc, fun=function(arg1, arg2) {
        mod <- reticulate::import("scikit-learn")
        output <- mod$some_calculation(arg1, arg2)

        # The return value MUST be a pure R object, i.e., no reticulate
        # Python objects, no pointers to shared memory. 
        output 
    }, arg1=ARG_VALUE_1, arg2=ARG_VALUE_2)

    some_useful_thing
}

my_example_function <- function() { 
    proc <- basiliskStart(my_env)
    on.exit(basiliskStop(proc))

    basiliskRun(proc, fun=function(store) {
        store$something <- rand(1)
        invisible(NULL) 
    }, persist=TRUE)

    basiliskRun(proc, fun=function(store) {
        store$something
    }, persist=TRUE)
}

Troubleshooting known issues

In most cases, end users should not have to read this document. Properly configured basilisk clients should handle all aspects of Python environment creation and loading without requiring user intervention. That said, some system configurations are less cooperative than others: this section contains a list of known issues and possible fixes.

Historically, the Miniforge installers did not work if the installation directory contained spaces. Recent Conda versions have improved support for spaces but mileage may vary in the packages. If spaces are causing problems in the default installation path (check basilisk.utils::getExternalDir()), consider setting the BASILISK_EXTERNAL_DIR environment variable to a location without spaces.

Windows has a limit of 260 characters for its file paths. This is occasionally exceeded due to deeply nested directories for some packages, causing the conda installation to be incomplete or fail outright. Furthermore, the Windows Miniforge installer has a limit of 46 characters for the top-level Conda installation directory. If either of these constraints are causing problems, it may be possible to circumvent them by setting BASILISK_EXTERNAL_DIR to a shorter path.

Builds for 32-bit Windows are not supported due to a lack of demand relative to the difficulty of setting it up.

Older versions of Rstudio on MacOSX have some difficulties with the generation of separate processes (see here). As a workaround in such cases, users should set:

parallel:::setDefaultClusterOptions(setup_strategy = "sequential")

Conda installations and environments tend to use a lot of disk space, so basilisk will automatically attempt to remove its old conda installations as well as unused environments for each client package. However, this removal may not be fast enough on systems with low disk usage quotas, resulting in incomplete or failed installations. In such cases, users can forcibly clear the external directory themselves to free up some space:

# Remove obsolete environments for specific package:
basilisk.utils::clearExternalDir(package = "pkg_name", obsolete.only = TRUE)

# Remove all environments for a specific package:
basilisk.utils::clearExternalDir(package = "pkg_name")

# Remove all basilisk-managed conda installations and environments:
basilisk.utils::clearExternalDir()

IMPORTANT! the automatic installation brokered by basilisk assumes that the user accepts the Anaconda terms of service. This mostly boils down to “don’t hammer the Anaconda repository with requests”, see the commentary here.

Fine-tuning basilisk’s behavior

Administrators of an R installation can modify the behavior of basilisk by setting a few environment variables. All environment variables described here must be set at both installation time and run time to have any effect. If any value is changed, it is generally safest to reinstall basilisk and all of its clients.

Setting the BASILISK_EXTERNAL_DIR environment variable will change where the conda instance and environments are placed by basiliskStart() during lazy installation. This is usually unnecessary unless the default path contains spaces or the combination of the default location and conda’s directory structure exceeds the file path length limit on Windows.

Setting BASILISK_USE_SYSTEM_DIR to 1 will instruct basilisk to install the conda instance in the R system directory during R package installation. Similarly, all (correctly configured) client packages will install their environments in the corresponding system directory when they themselves are being installed. This is very useful for enterprise-level deployments as the conda instances and environments are (i) not duplicated in each user’s home directory, and (ii) always available to any user with access to the R installation. However, it requires installation from source and thus is not set by default.

It is possible to direct basilisk to use an existing Conda installation, by setting the BASILISK_EXTERNAL_CONDA environment variable to an absolute path to the conda installation directory. This may be desirable to avoid redundant copies of the same conda installation. Any replacement conda instance should be using a similar version of Python as basilisk’s default (3.10.14) to satisfy client packages that rely on the implict pinning.

Setting BASILISK_MINIFORGE_VERSION will change the Miniforge version installed by basilisk. This should be set to the name of the tagged releases on the Miniforce installation page, .e.g, "24.3.0-0".

Setting BASILISK_NO_DESTROY to 1 will instruct basilisk to not destroy previous conda instances and environments upon installation of a new version of basilisk. This destruction is done by default to avoid accumulating many large obsolete conda instances. However, it is not desirable if there are multiple R instances running different versions of basilisk from the same Bioconductor release, as installation by one R instance would delete the installed content for the other. (Multiple R instances running different Bioconductor releases are not affected.) This option has no effect if BASILISK_USE_SYSTEM_DIR is set.

Setting BASILISK_NO_FALLBACK_R to 1 will instruct basilisk to not create an conda-based R installation as a last-resort fallback (see comments in ?basiliskStart). This avoids the overhead of an internal R installation if it is known that shared library version conflicts will not occur. It is most useful for streamlining applications where developers can (i) test that no conflicts occur in the installed subset of basilisk clients and/or (ii) control the versions of system libraries used by R. This option only has an effect if BASILISK_USE_SYSTEM_DIR is set, as otherwise the fallback R installation is only created upon encountering version conflicts.

Using basilisk directly for analyses

While basilisk is primarily intended for package developers, end users can also take advantage of its graceful handling of multiple Python environments in complex workflows. For example, we can easily instantiate a conda environment in our working directory with createLocalBasiliskEnv():

if (.Platform$OS.type != "windows") {
tmp <- createLocalBasiliskEnv("basilisk-vignette-test",
    packages=c("scikit-learn=1.1.1", "numpy=1.22.1"))
}

We can then supply this environment’s path to basiliskRun() to execute Python-based calculations. To demonstrate, we’ll apply scikit-learn’s truncated PCA on a random matrix. Note that the restrictions mentioned above for fun are still applicable here.

if (.Platform$OS.type != "windows") {
x <- matrix(rnorm(1000), ncol=10)
basiliskRun(env=tmp, fun=function(mat) {
    module <- reticulate::import("sklearn.decomposition")
    runner <- module$TruncatedSVD()
    output <- runner$fit(mat) 
    output$singular_values_
}, mat = x, testload="scipy.optimize")
}

## [1] 13.02368 11.98517

basiliskRun() can also be used with conda environments constructed outside of basilisk. Of course, in this case, it is the user’s responsibility to ensure that the environment is correctly provisioned. Some caution is also required as basilisk is not guaranteed to work with environments created by a different version of conda, though problems seem to be rare in practice.

if (.Platform$OS.type != "windows") {
library(reticulate)

# In this case, we'll use reticulate directly to construct our conda
# environment; though we'll cheat a little and use basilisk's conda
# installation, otherwise reticulate will try to install its own miniconda.
tmp2 <- file.path(getwd(), "basilisk-vignette-test2")
if (!file.exists(tmp2)) {
    conda.bin <- file.path(
        basilisk.utils::getCondaDir(), 
        basilisk.utils::getCondaBinary()
    )
    conda_install(tmp2, 
        packages=c("scipy==1.9.1"), 
        python_version="3.10", 
        channels="conda-forge",
        additional_create_args="--override-channels",
        additional_install_args="--override-channels",
        conda=conda.bin
    )
}

basiliskRun(env=tmp2, fun=function(mat) {
    module <- reticulate::import("scipy.stats")
    norm <- module$norm
    norm$cdf(c(-1, 0, 1))
}, mat = x, testload="scipy.optimize")
}

## [1] 0.1586553 0.5000000 0.8413447

It is worth highlighting the fact that we were able to call basiliskRun() successfully on two different environments within the same R session. This enables the construction of complex analysis workflows that span across R and multiple Python environments.