Introduction

The alabaster.base package (and its family) implements methods to save common Bioconductor objects to file artifacts and load them back into R. This aims to provide a functional equivalent to RDS-based serialization that is:

• More stable to changes in the class definition. Such changes would typically require costly updateObject() operations at best, or invalidate RDS files at worst.
• More interoperable with other analysis frameworks. All artifacts are saved in standard formats (e.g., JSON, HDF5) and can be easily parsed by applications in other languages by following the relevant specifications.
• More modular, with each object split into multiple artifacts. This enables parts of the object to be loaded into memory according to each application’s needs. Parts can also be updated cheaply on disk without rewriting all files.

Quick start

To demonstrate, let’s mock up a DataFrame object from the S4Vectors package.

library(S4Vectors)
df <- DataFrame(X=1:10, Y=letters[1:10])
df

## DataFrame with 10 rows and 2 columns
##            X           Y
##    <integer> <character>
## 1          1           a
## 2          2           b
## 3          3           c
## 4          4           d
## 5          5           e
## 6          6           f
## 7          7           g
## 8          8           h
## 9          9           i
## 10        10           j

We’ll save this DataFrame to a directory:

tmp <- tempfile()
library(alabaster.base)
saveObject(df, tmp)

And read it back in:

readObject(tmp)

## DataFrame with 10 rows and 2 columns
##            X           Y
##    <integer> <character>
## 1          1           a
## 2          2           b
## 3          3           c
## 4          4           d
## 5          5           e
## 6          6           f
## 7          7           g
## 8          8           h
## 9          9           i
## 10        10           j

Class-specific methods

Each class implements a saving and reading method for use by the alabaster framework. The saving method (for the saveObject() generic) will save the object to one or more files inside a user-specified directory:

tmp <- tempfile()
saveObject(df, tmp)
list.files(tmp, recursive=TRUE)

## [1] "OBJECT"           "basic_columns.h5"

Conversely, the reading function will - as the name suggests - load the object back into memory, given the path to its directory. The correct loading function for each class is automatically called by the readObject() function:

readObject(tmp)

## DataFrame with 10 rows and 2 columns
##            X           Y
##    <integer> <character>
## 1          1           a
## 2          2           b
## 3          3           c
## 4          4           d
## 5          5           e
## 6          6           f
## 7          7           g
## 8          8           h
## 9          9           i
## 10        10           j

alabaster.base also provides a validateObject() function, which checks that each object’s on-disk representation follows its associated specification. For all alabaster-supported objects, this validates the file contents against the takane specifications - successful validation provides guarantees for readers like readObject() and dolomite-base (for Python). In fact, saveObject() will automatically run validateObject() on the directory to ensure compliance.

validateObject(tmp)

alabaster.base itself supports a small set of classes from the S4Vectors packages; support for additional classes can be found in other packages like alabaster.ranges and alabaster.se. Third-party developers can also add support for their own classes by defining new methods, see the Extensions vignette for details.

Operating on directories

Users can move freely rename or relocate directories and readObject() function will still work. For example, we can easily copy the entire directory to a new file system and everything will still be correctly referenced within the directory. The simplest way to share objects is to just zip or tar the staging directory for ad hoc distribution. For more serious applications, alabaster.base can be used in conjunction with storage systems like AWS S3 for large-scale distribution.

tmp <- tempfile()
saveObject(df, tmp)

tmp2 <- tempfile()
file.rename(tmp, tmp2)

## [1] TRUE

readObject(tmp2)

## DataFrame with 10 rows and 2 columns
##            X           Y
##    <integer> <character>
## 1          1           a
## 2          2           b
## 3          3           c
## 4          4           d
## 5          5           e
## 6          6           f
## 7          7           g
## 8          8           h
## 9          9           i
## 10        10           j

That said, it is unwise to manipulate the files inside the directory created by saveObject(). Reading functions will usually depend on specific file names or subdirectory structures within the directory, and fiddling with them may cause unexpected results. Advanced users can exploit this by loading components from subdirectories if only the full object is not required:

# Creating a nested DF to be a little spicy:
df2 <- DataFrame(Z=factor(1:5), AA=I(DataFrame(B=runif(5), C=rnorm(5))))
tmp <- tempfile()
meta2 <- saveObject(df2, tmp)

# Now reading in the nested DF:
list.files(tmp, recursive=TRUE)

## [1] "OBJECT"                           "basic_columns.h5"                
## [3] "other_columns/1/OBJECT"           "other_columns/1/basic_columns.h5"

readObject(file.path(tmp, "other_columns/1"))

## DataFrame with 5 rows and 2 columns
##           B          C
##   <numeric>  <numeric>
## 1 0.0196919  0.1261901
## 2 0.7982964 -0.8294884
## 3 0.7786083 -0.5711057
## 4 0.6334135 -0.0311062
## 5 0.0557657  0.2537396

Extending to new classes

The alabaster framework is easily extended to new classes by:

1. Writing a method for saveObject(). This should accept an instance of the object and a path to a directory, and save the contents of the object inside the directory. It should also produce an OBJECT file that specifies the type of the object, e.g., data_frame, hdf5_sparse_matrix.
2. Writing a function for readObject() and registering it with registerReadObjectFunction() (or, for core Bioconductor classes, by requesting a change to the default registry in alabaster.base). This should accept a path to a directory and read its contents to reconstruct the object. The registered type should be the same as that used in the OBJECT file.
3. Writing a function for validateObject() and registering it with registerValidateObjectFunction(). This should accept a path to a directory and read its contents to determine if it is a valid on-disk representation. The registered type should be the same as that used in the OBJECT file.
   • (optional) Devleopers can alternatively formalize the on-disk representation by adding a specification to the takane repository. This aims to provide C++-based validators for each representation, allowing us to enforce consistency across multiple languages (e.g., Python). Any takane validator is automatically used by validateObject() so no registration is required.

To illustrate, let’s extend alabaster to the dgTMatrix from the Matrix package. First, the saving method:

library(Matrix)
setMethod("saveObject", "dgTMatrix", function(x, path, ...) {
    # Create a directory to stash our contents.
    dir.create(path)

    # Saving a DataFrame with the triplet data.
    df <- DataFrame(i = x@i, j = x@j, x = x@x)
    write.csv(df, file.path(path, "matrix.csv"), row.names=FALSE)

    # Adding some more information.
    write(dim(x), file=file.path(path, "dimensions.txt"), ncol=1)

    # Creating an object file.
    saveObjectFile(path, "triplet_sparse_matrix")
})

And now the reading and validation methods. The registration is usually done in the extension package’s onLoad() function.

readSparseTripletMatrix <- function(path, metadata, ...) {
    df <- read.table(file.path(path, "matrix.csv"), header=TRUE, sep=",")
    dims <- readLines(file.path(path, "dimensions.txt"))
    sparseMatrix(
         i=df$i + 1L, 
         j=df$j + 1L, 
         x=df$x, 
         dims=as.integer(dims),
         repr="T"
    )
}
registerReadObjectFunction("triplet_sparse_matrix", readSparseTripletMatrix)

validateSparseTripletMatrix <- function(path, metadata) {
    df <- read.table(file.path(path, "matrix.csv"), header=TRUE, sep=",")
    dims <- as.integer(readLines(file.path(path, "dimensions.txt")))
    stopifnot(is.integer(df$i), all(df$i >= 0 & df$i < dims[1]))
    stopifnot(is.integer(df$j), all(df$j >= 0 & df$j < dims[2]))
    stopifnot(is.numeric(df$x))
}
registerValidateObjectFunction("triplet_sparse_matrix", validateSparseTripletMatrix)

## NULL

Let’s run them and see how it works:

x <- sparseMatrix(
    i=c(1,2,3,5,6), 
    j=c(3,6,1,3,8), 
    x=runif(5), 
    dims=c(10, 10), 
    repr="T"
)
x

## 10 x 10 sparse Matrix of class "dgTMatrix"
##                                                          
##  [1,] .         . 0.32918510 . . .         . .        . .
##  [2,] .         . .          . . 0.8271621 . .        . .
##  [3,] 0.2751764 . .          . . .         . .        . .
##  [4,] .         . .          . . .         . .        . .
##  [5,] .         . 0.02819651 . . .         . .        . .
##  [6,] .         . .          . . .         . 0.542808 . .
##  [7,] .         . .          . . .         . .        . .
##  [8,] .         . .          . . .         . .        . .
##  [9,] .         . .          . . .         . .        . .
## [10,] .         . .          . . .         . .        . .

tmp <- tempfile()
saveObject(x, tmp)
list.files(tmp, recursive=TRUE)

## [1] "OBJECT"         "dimensions.txt" "matrix.csv"

readObject(tmp)

## 10 x 10 sparse Matrix of class "dgTMatrix"
##                                                          
##  [1,] .         . 0.32918510 . . .         . .        . .
##  [2,] .         . .          . . 0.8271621 . .        . .
##  [3,] 0.2751764 . .          . . .         . .        . .
##  [4,] .         . .          . . .         . .        . .
##  [5,] .         . 0.02819651 . . .         . .        . .
##  [6,] .         . .          . . .         . 0.542808 . .
##  [7,] .         . .          . . .         . .        . .
##  [8,] .         . .          . . .         . .        . .
##  [9,] .         . .          . . .         . .        . .
## [10,] .         . .          . . .         . .        . .

For more complex objects that are composed of multiple smaller “child” objects, developers should consider saving each of their children in subdirectories of path. This can be achieved by calling altSaveObject() and altReadObject() in the saving and reading functions, respectively. (We use the alt* versions of these functions to respect application overrides, see below.)

Creating applications

Developers can also create applications that customize the machinery of the alabaster framework for specific needs. In most cases, this involves storing more metadata to describe the object in more detail. For example, we might want to remember the identity of the author for each object. This is achieved by creating an application-specific saving generic with the same signature as saveObject():

setGeneric("appSaveObject", function(x, path, ...) {
    ans <- standardGeneric("appSaveObject")

    # File names with leading underscores are reserved for application-specific
    # use, so they won't clash with anything produced by saveObject.
    metapath <- file.path(path, "_metadata.json")
    write(jsonlite::toJSON(ans, auto_unbox=TRUE), file=metapath)
})

## [1] "appSaveObject"

setMethod("appSaveObject", "ANY", function(x, path, ...) {
    saveObject(x, path, ...) # does the real work
    list(authors=I(Sys.info()[["user"]])) # adds the desired metadata
})

# We can specialize the behavior for specific classes like DataFrames.
setMethod("appSaveObject", "DFrame", function(x, path, ...) {
    ans <- callNextMethod()
    ans$columns <- I(colnames(x))
    ans
})

Applications should call altSaveObjectFunction() to instruct altSaveObject() to use this new generic. This ensures that the customizations are applied to all child objects, such as the nested DataFrame below.

# Create a friendly user-visible function to handle the generic override; this
# is reversed on function exit to avoid interfering with other applications. 
saveForApplication <- function(x, path, ...) { 
    old <- altSaveObjectFunction(appSaveObject)
    on.exit(altSaveObjectFunction(old)) 
    altSaveObject(x, path, ...)
}

# Saving our mocked up DataFrame with our overrides active.
df2 <- DataFrame(Z=factor(1:5), AA=I(DataFrame(B=runif(5), C=rnorm(5))))
tmp <- tempfile()
saveForApplication(df2, tmp)

# Both the parent and child DataFrames have new metadata.
cat(readLines(file.path(tmp, "_metadata.json")), sep="\n")

## {"authors":["root"],"columns":["Z","AA"]}

cat(readLines(file.path(tmp, "other_columns/1/_metadata.json")), sep="\n")

## {"authors":["root"],"columns":["B","C"]}

The reading function can be similarly overridden by setting altReadObjectFunction() to instruct all altReadObject() calls to use the override. This allows applications to, e.g., do something with the metadata that we just added.

# Defining the override for altReadObject().
appReadObject <- function(path, metadata=NULL, ...) {
    if (is.null(metadata)) {
        metadata <- readObjectFile(path)
    }

    # Print custom message based on the type and application-specific metadata.
    appmeta <- jsonlite::fromJSON(file.path(path, "_metadata.json"))
    cat("I am a ", metadata$type, " created by ", appmeta$authors[1], ".\n", sep="")
    if (metadata$type == "data_frame") {
        all.cols <- paste(appmeta$columns, collapse=", ")
        cat("I have the following columns: ", all.cols, ".\n", sep="")
    }

    readObject(path, metadata=metadata, ...)
}

# Creating a user-friendly function to set the override before the read.
readForApplication <- function(path, metadata=NULL, ...) {
    old <- altReadObjectFunction(appReadObject)
    on.exit(altReadObjectFunction(old))
    altReadObject(path, metadata, ...)
}

# This diverts to the override with printing of custom messages.
readForApplication(tmp)

## I am a data_frame created by root.
## I have the following columns: Z, AA.
## I am a data_frame created by root.
## I have the following columns: B, C.

## DataFrame with 5 rows and 2 columns
##          Z                 AA
##   <factor>        <DataFrame>
## 1        1 0.246322:-1.210172
## 2        2 0.154906:-0.429744
## 3        3 0.915335:-0.633189
## 4        4 0.239906: 0.734313
## 5        5 0.705503: 1.126401

By overriding the saving and reading process for one or more classes, each application can customize the behavior of alabaster to their own needs.