---
title: "Storage Modes of MS Data Objects"
output:
    BiocStyle::html_document:
        toc_float: true
vignette: >
    %\VignetteIndexEntry{Storage Modes of MS Data Objects}
    %\VignetteEngine{knitr::rmarkdown}
    %\VignetteEncoding{UTF-8}
    %\VignettePackage{MsStash}
    %\VignetteDepends{MsStash,BiocStyle,alabaster.base,fs}
---

```{r, echo = FALSE, message = FALSE}
knitr::opts_chunk$set(echo = TRUE, message = FALSE)
library(BiocStyle)
```

# Introduction

Data objects in R can be serialized to disk in R's *Rds* format using the base R
`save()` function and re-imported using the `load()` function. This R-specific
binary data format can however not be used or read by other programming
languages preventing thus the exchange of R data objects between software or
programming languages. The *MsStash* package defines basic classes and generic
methods to export and import mass spectrometry (MS) data objects in various
storage formats aiming to facilitate data exchange between software. This
includes, among other formats, also storage of data objects using Bioconductor's
`r Biocpkg("alabaster.base")` package.

For export or import of MS data objects, the `saveMsObject()` and
`readMsObject()` functions can be used. For `saveMsObject()`, the first
parameter is the MS data object that should be stored, for `readMsObject()` it
defines type of MS object that should be restored (returned). The second
parameter `param` defines and configures the storage format of the MS data. The
currently supported formats and the respective parameter objects are:

- `PlainTextParam`: storage of data in (a custom) plain text file format.
- `AlabasterParam`: storage of MS data using Bioconductor's
  `r Biocpkg("alabaster.base")` framework based files in HDF5 and JSON format.

These storage formats are described in more details in the following sections.

An example use of these functions and parameters: `saveMsObject(x, param =
PlainTextParam(storage_path))` to store an MS data object assigned to a variable
`x` to a directory `storage_path` using the plain text file format. To restore
the data (assuming `x` was an instance of a `MsExperiment` class):
`readMsObject(MsExperiment(), param = PlainTextParam(storage_path))`.

# Installation

The package can be installed with the *BiocManager* package. To install
*BiocManager* use `install.packages("BiocManager")` and, after that,
`BiocManager::install("MsStash")` to install this package.

# Example implementations

To illustrate how the save/read functionality can be implemented for a specific
data class, we first define a simple toy R S4 object to represent the data from
a single mass spectrum. This `MySpectrum` class contains slots to hold the
spectrum's *m/z* and intensity values as well as some (limited) metadata.

```{r}
#' Class definition
setClass("MySpectrum",
         slots = c(mz = "numeric",
                   intensity = "numeric",
                   rtime = "numeric",
                   msl = "integer"),
         prototype = prototype(
             mz = numeric(),
             intensity = numeric(),
             rtime = numeric(),
             msl = integer()))

#' Default constructor function
MySpectrum <- function(mz = numeric(), intensity = numeric(),
                       rtime = numeric(), msl = integer()) {
    stopifnot(length(mz) == length(intensity))
    if (length(mz) && !length(rtime)) rtime <- NA_real_
    if (length(mz) && !length(msl)) msl <- NA_integer_
    new("MySpectrum", mz = mz, intensity = intensity, rtime = rtime,
        msl = as.integer(msl))
}
```

We can now create an example `MySpectrum` object.

```{r}
s <- MySpectrum(c(1.4, 1.6, 1.9, 2.56), c(123.1, 1235.3, 12.45, 51.5))
s
```

## Suggested properties of implemented methods

To ensure consistency, the `saveMsObject()` should:

- first create the directory to which the data should be exported (defined by
  param `path`).
- throw an error if the directory exists or contains already an exported object
  (avoiding thus accidental overwriting and eventual data
  corruption/inconsistencies).

Both methods support also `...`, hence, if needed, additional parameters can be
added to an implementation of the generic method if needed.

```{r}
library(MsStash)
```

## `PlainTextParam`

Storage of MS data objects in *plain* text format aims to support an easy
exchange of data, and in particular analysis results, with external software,
such as [MS-DIAL](https://systemsomicslab.github.io/compms/msdial/main.html) or
[mzmine3](http://mzmine.github.io/download.html). In most cases, the data is
stored as tabulator delimited text files simplifying the use of the data and
results across multiple programming languages, or their import into spreadsheet
applications. MS data objects stored in plain text format can also be fully
re-imported into R providing thus an alternative, and more flexible, object
serialization approach than the R internal *Rds*/*RData* format.

We implement a `saveMsObject()` method for our `MySpectrum` class and the
`PlainTextParam`. This function first creates the required directory and throws
an error if an result file is already stored there. Then it exports the data:
for our example we store the data of the object into a single text file in a
custom format we define: the metadata if first written to the file, one line per
metadata item followed by the *m/z* and intensity values, each *m/z*-intensity
pair in one line separated by a tabulator.

```{r}
#' Write example class to a plain text file
setMethod("saveMsObject", signature(object = "MySpectrum",
                                    param = "PlainTextParam"),
          function(object, param) {
              dir.create(path = param@path, recursive = TRUE,
                         showWarnings = FALSE)
              fl <- file.path(param@path, "my_spectrum.txt")
              if (file.exists(fl))
                  stop("Overwriting an existing result object is not ",
                       "supported.")
              ## Write the type of object as a comment followed by the
              ## metadata.
              writeLines(c(paste0("# ", class(object)[1L]),
                           paste0("rtime:", object@rtime),
                           paste0("msl:", object@msl)), con = fl)
              ## Write the peak data, i.e. m/z and intensity values
              write.table(cbind(object@mz, object@intensity), file = fl,
                          sep = "\t", append = TRUE, col.names = FALSE,
                          row.names = FALSE)
          })
```

We next export our example object `s` with the `saveMsData()` method to a
temporary folder.

```{r}
p <- PlainTextParam(path = file.path(tempdir(), "text_format"))
saveMsObject(s, p)
```

The data was thus exported to this text file. The individual lines are:

```{r}
readLines(file.path(p@path, "my_spectrum.txt"))
```

We next implement the `readMsObject()` method for this class. This function will
read the text file content and assign the imported values to the different slots
of the `MySpectrum` class.

```{r}
#' Read example object from plain text file storage format
setMethod("readMsObject", signature(object = "MySpectrum",
                                    param = "PlainTextParam"),
          function(object, param) {
              fl <- file.path(param@path, "my_spectrum.txt")
              if (!file.exists(fl))
                  stop("my_spectrum.txt not found in the provided path")
              l <- readLines(fl, n = 3) # read the comment and the metadata
              p <- read.table(fl, sep = "\t", skip = 3)
              MySpectrum(
                  mz = p[, 1L], intensity = p[, 2L],
                  rtime = suppressWarnings(
                      as.numeric(sub("rtime:", "", l[2], fixed = TRUE))),
                  msl = suppressWarnings(
                      as.integer(sub("msl:", "", l[3], fixed = TRUE))))
          })
```

We can now restore our `MySpectrum` object with the `readMsObject()` method from
the exported text file:

```{r}
p <- PlainTextParam(path = file.path(tempdir(), "text_format"))
b <- readMsObject(MySpectrum(), p)
b
```

## `AlabasterParam`

The [alabaster framework](https://github.com/ArtifactDB/alabaster.base) and
related Bioconductor package `r Biocpkg("alabaster.base")` implements methods to
save a variety of R/Bioconductor objects to on-disk representations based on
standard file formats like HDF5 and JSON. This ensures that Bioconductor objects
can be easily read from other languages like Python and Javascript. With
`AlabasterParam`, *MsStash* provides a parameter class to configure saving MS
data objects in the *alabaster* storage format.

To enable writing in this format a `saveMsObject()` method should be implemented
for the MS data object and `AlabasterParam`. To enable full *alabaster* support
it is also suggested to implement the `alabaster.base::saveObject` method, a
validation method and a function to read from an alabaster format. For more
details refer also to the package vignette of the `r Biocpkg("alabaster.base")`
package, in particular chapter 5 *Extending to new classes*.

We below define a `saveObject()` method. The generic for this method is defined
in the *alabaster.base* package. While it would be possible to simply save the
data as simple text files as we did above, we use *alabaster*'s strategy to
allow storage of more complex objects (such as S4 objects in the individual
slots). This uses `altSaveObject()` and `altReadObject()` to save individual
slots or parent/child classes in sub-directories of `path`. For each of these
classes, a `saveObject()` needs to be defined.

```{r}
library(alabaster.base)

setMethod("saveObject", "MySpectrum", function(x, path, ...) {
    ## Create the directory where to save the data
    dir.create(path = path, recursive = TRUE, showWarnings = FALSE)
    ## Create an "object" file; this defines the type of object stored in path
    saveObjectFile(path, "my_spectrum")
    ## save each slot into it's own directory
    altSaveObject(x@mz, path = file.path(path, "mz"))
    altSaveObject(x@intensity, path = file.path(path, "intensity"))
    altSaveObject(x@rtime, path = file.path(path, "retention_time"))
    altSaveObject(x@msl, path = file.path(path, "ms_level"))
})
```

We next need to implement a *validation function* for the stash (directory). For
our example we simply check that the `path` contains the expected
sub-directories with the object's content. This function needs then to be
registered with the `registerValidateObjectFunction()` method for our class.

```{r}
#' Define a helper function to check that the folder contains all
#' expected sub-directories.
validateMySpectrum <- function(path, metadata) {
    if (!dir.exists(path))
        stop("Directory ", path, " does not exist")
    req_dir <- c("mz", "intensity", "retention_time", "ms_level")
    if (any(miss <- !dir.exists(file.path(path, req_dir))))
        stop("Required directories ",
             paste0("\"", req_dir[miss], "\"", collapse = ", "),
             " not found in ", path)
}

#' Register the validation function
registerValidateObjectFunction("my_spectrum", validateMySpectrum)
```

Finally we define the function to read the data back from the stash. We then
register this function with *alabaster*'s `registerReadObjectFunction()`
function.

```{r}
#' Define a function that can read from an alabaster-based serialization
#' of `MySpectrum` objects
readMySpectrum <- function(path, metadata, ...) {
    validateMySpectrum(path)
    ## Read the data from individual sub-directories
    mz <- altReadObject(file.path(path, "mz"))
    int <- altReadObject(file.path(path, "intensity"))
    rtime <- altReadObject(file.path(path, "retention_time"))
    msl <- altReadObject(file.path(path, "ms_level"))
    MySpectrum(mz = mz, intensity = int, rtime = rtime, msl = msl)
}

#' Register the read function
registerReadObjectFunction("my_spectrum", readMySpectrum)
```

Registration of the validation and read functions is generally done in the
extension package's `onLoad()` function.

With these functions defined and registered, we can store an instance of
`MySpectrum` directly with *alabaster*'s `saveObject()` method:

```{r}
#' Define the path where we want to export out data
p <- file.path(tempdir(), "alabaster_export")

#' Save the object
saveObject(s, path = p)
```

This saved the object's content to the directory specified with `path`. The
content of this folder is:

```{r}
library(fs)
dir_tree(p)
```

We can read the serialized object again as a `MySpectrum` object:

```{r}
b <- readObject(p)
b
```

We next implement the `saveMsObject()` and `readMsObject()` methods for
`MySpectrum` and `AlabasterParam`. These can simply re-use the functions
implemented above.

```{r}
#' Write example class to a plain text file
setMethod("saveMsObject", signature(object = "MySpectrum",
                                    param = "AlabasterParam"),
          function(object, param) {
              if (file.exists(file.path(param@path, "OBJECT")))
                  stop("'path' contains already an MS data stash. Overwriting",
                       " is not supported. Please remove 'path' first.")
              saveObject(object, param@path)
          })

#' Read example object from plain text file storage format
setMethod("readMsObject", signature(object = "MySpectrum",
                                    param = "AlabasterParam"),
          function(object, param) {
              readMySpectrum(param@path)
          })
```

We can now stash our MS object in either the text file-based format
(`PlainTextParam`) or the alabaster-based format (`AlabasterParam`). Below we
write it using the alabaster approach.

```{r}
p <- file.path(tempdir(), "alabaster_format_2")
ap <- AlabasterParam(p)

saveMsObject(s, ap)
```

To read the data back we can then use `readMsObject()` specifying in addition
the type of object we want to read.

```{r}
b <- readMsObject(MySpectrum(), ap)
b
```

# Session information

```{r}
sessionInfo()
```
