Introduction

The gypsum package implements an R client for the API of the same name. This allows Bioconductor packages to easily store and retrieve data from the gypsum backend. It also provides mechanisms to allow package maintainers to easily manage upload authorizations and third-party contributions. Readers are referred to API’s documentation for a description of the concepts; this guide will strictly focus on the usage of the gypsum package.

Reading files

gypsum provides several convenience methods for reading from the gypsum bucket:

library(gypsum)
listAssets("test-R")

## [1] "basic"      "upload-dir"

listVersions("test-R", "basic")

## [1] "v1" "v2" "v3"

listFiles("test-R", "basic", "v1")

## [1] "..manifest"  "..summary"   "blah.txt"    "foo/bar.txt"

out <- saveFile("test-R", "basic", "v1", "blah.txt")
readLines(out)

##  [1] "A" "B" "C" "D" "E" "F" "G" "H" "I" "J" "K" "L" "M" "N" "O" "P" "Q" "R" "S"
## [20] "T" "U" "V" "W" "X" "Y" "Z"

dir <- saveVersion("test-R", "basic", "v1")
list.files(dir, all.files=TRUE, recursive=TRUE)

## [1] "..manifest"  "..summary"   "blah.txt"    "foo/bar.txt"

We can fetch the summaries and manifests for each version of a project’s assets.

fetchManifest("test-R", "basic", "v1")

## $blah.txt
## $blah.txt$size
## [1] 52
## 
## $blah.txt$md5sum
## [1] "0eb827652a5c272e1c82002f1c972018"
## 
## 
## $`foo/bar.txt`
## $`foo/bar.txt`$size
## [1] 21
## 
## $`foo/bar.txt`$md5sum
## [1] "c34bcea9f2263deb3379103c9f10c130"

fetchSummary("test-R", "basic", "v1")

## $upload_user_id
## [1] "LTLA"
## 
## $upload_start
## [1] "2024-02-23 07:17:24 UTC"
## 
## $upload_finish
## [1] "2024-02-23 07:17:26 UTC"

We can get the latest version of an asset:

fetchLatest("test-R", "basic")

## [1] "v3"

All read operations involve a publicly accessible bucket so no authentication is required.

Uploading files

tmp <- tempfile()
dir.create(tmp)
write(file=file.path(tmp, "foo"), letters)
write(file=file.path(tmp, "bar"), LETTERS)
write(file=file.path(tmp, "whee"), 1:10)

init <- startUpload(
    project=project_name,
    asset=asset_name,
    version=version_name,
    files=list.files(tmp, recursive=TRUE),
    directory=tmp
)

tryCatch({
    uploadFiles(init, directory=tmp)
    completeUpload(init)
}, error=function(e) {
    abortUpload(init) # clean up if the upload fails.
    stop(e)
})

# Creates links from lun/aaron.txt ==> test-R/basic/v1/foo/bar.txt
# and kancherla/jayaram ==> test-R/basic/v1/blah.txt
init <- startUpload(
    project=project_name,
    asset=asset_name,
    version=version_name,
    files=character(0),
    links=data.frame(
        from.path=c("lun/aaron.txt", "kancherla/jayaram.txt"),
        to.project=c("test-R", "test-R"),
        to.asset=c("basic", "basic"),
        to.version=c("v1", "v1"),
        to.path=c("foo/bar.txt", "blah.txt")
    ),
    directory=tmp
)

dest <- tempfile()
cloneVersion("test-R", "basic", "v1", destination=dest)

# Do some modifications in 'dest' to create a new version, e.g., add a file.
# However, users should treat symlinks as read-only - so if you want to modify
# a file, instead delete the symlink and replace it with a new file.
write(file=file.path(dest, "BFFs"), c("Aaron", "Jayaram"))

to.upload <- prepareDirectoryUpload(dest)
to.upload

## $files
## [1] "BFFs"
## 
## $links
##     from.path to.project to.asset to.version     to.path
## 1    blah.txt     test-R    basic         v1    blah.txt
## 2 foo/bar.txt     test-R    basic         v1 foo/bar.txt

init <- startUpload(
    project=project_name,
    asset=asset_name,
    version=version_name,
    files=to.upload$files,
    links=to.upload$links,
    directory=dest
)

Changing permissions

Upload authorization is determined by each project’s permissions, which are controlled by project owners. Both uploaders and owners are identified based on their GitHub logins:

fetchPermissions("test-R")

## $owners
## $owners[[1]]
## [1] "LTLA"
## 
## 
## $uploaders
## $uploaders[[1]]
## $uploaders[[1]]$id
## [1] "ArtifactDB-bot"

Owners can add more uploaders (or owners) via the setPermissions() function. Uploaders can be scoped to individual assets or versions, and an expiry date may be attached to each authorization:

setPermissions("test-R", 
    uploaders=list(
        list(
            id="jkanche", 
            until=Sys.time() + 24 * 60 * 60,
            asset="jays-happy-fun-time",
            version="1"
        )
    )
)

Organizations may also be added in the permissions, in which case the authorization extends to all members of that organization.

Probational uploads

Unless specified otherwise, all uploaders are considered to be “untrusted”. Any uploads from such untrusted users are considered “probational” and must be approved by the project owners before they are considered complete. Alternatively, an owner may reject an upload, which deletes all the uploaded files from the backend.

approveProbation("test-R", "third-party-upload", "good")
rejectProbation("test-R", "third-party-upload", "bad")

An uploader can be trusted by setting trusted=TRUE in setPermissions(). Owners and trusted uploaders may still perform probational uploads (e.g., for testing) by setting probation=TRUE in startUpload().

Inspecting the quota

Each project has a quota that specifies how much storage space is available for uploaders. The quota is computed as a linear function of baseline + growth_rate * (NOW - year), which provides some baseline storage that grows over time.

fetchQuota("test-R")

## $baseline
## [1] 1e+10
## 
## $growth_rate
## [1] 2e+10
## 
## $year
## [1] 2024

Once the project’s contents exceed this quota, all uploads are blocked. The current usage of the project can be easily inspected:

fetchUsage("test-R")

## [1] 146

Changes to the quota must be performed by administrators, see below.

Validating metadata

Databases can operate downstream of the gypsum backend to create performant search indices, usually based on special metadata files. gypsum provides some utilities to check that metadata follows the JSON schema of some known downstream databases.

schema <- fetchMetadataSchema()
cat(head(readLines(schema)), sep="\n")

## {
##     "$schema": "http://json-schema.org/draft-07/schema",
##     "$id": "bioconductor.json",
##     "type": "object",
##     "title": "Bioconductor metadata standards",
##     "properties": {

Uploaders can verify that their metadata respects this schema via the validateMetadata() function. This ensures that the uploaded files can be successfully indexed by the database, given that the gypsum backend itself applies no such checks.

metadata <- list(                           
    title="Fatherhood",
    description="Luke ich bin dein Vater.",
    sources=list(
       list(provider="GEO", id="GSE12345")
    ),
    taxonomy_id=list("9606"),
    genome=list("GRCm38"),
    maintainer_name="Darth Vader",
    maintainer_email="[email protected]",
    bioconductor_version="3.10"
)

validateMetadata(metadata, schema)

Administration

Administrators of the gypsum instance can create projects with new permissions:

createProject("my-new-project", 
    owners="jkanche", 
    uploaders=list(
        list(
            id="LTLA", 
            asset="aarons-stuff"
        )
    )
)

They can alter the quota parameters for a project:

setQuota("my-new-project",
    baseline=10 * 2^30,
    growth=5 * 2^30,
    year=2019
)

They can manually refresh the latest version for an asset and the quota usage for a project. This is only required on very rare occasions where there are simultaneous uploads to the same project.

refreshLatest("test-R", "basic")
refreshUsage("test-R")

Administrators may also delete projects, assets or versions, though this should be done sparingly as it violates gypsum’s expectations of immutability.

removeProject("my-new_project")