Overview

First, one must decide if an ExperimentHub or AnnotationHub package is appropriate.

The AnnotationHubData package provides tools to acquire, annotate, convert and store data for use in Bioconductor’s AnnotationHub. BED files from the Encode project, gtf files from Ensembl, or annotation tracks from UCSC, are examples of data that can be downloaded, described with metadata, transformed to standard Bioconductor data types, and stored so that they may be conveniently served up on demand to users via the AnnotationHub client. While data are often manipulated into a more R-friendly form, the data themselves retain their raw content and are not normally filtered or curated like those in ExperimentHub. Each resource has associated metadata that can be searched through the AnnotationHub client interface.

ExperimentHubData provides tools to add or modify resources in Bioconductor’s ExperimentHub. This ‘hub’ houses curated data from courses, publications, or experiments. It is often convenient to store data to be used in package examples, testings, or vignettes in the ExperimentHub. The resources can be files of raw data or more often are R / Bioconductor objects such as GRanges, SummarizedExperiment, data.frame etc. Each resource has associated metadata that can be searched through the ExperimentHub client interface.

It is advisable to create a separate package for annotations or experiment data rather than an all encompassing package of data and code. However, it is sometimes understandable to have a Software package that also serves as the package front end for the hubs. Although this is generally not recommended; if you think you have a use case please reach out to [email protected] to confirm before proceeding with a single package rather than the accompanied package approach.

Setting up a package to use a Hub

``` r
.onLoad <- function(libname, pkgname) {
   fl <- system.file("extdata", "metadata.csv", package=pkgname)
   titles <- read.csv(fl, stringsAsFactors=FALSE)$Title
   createHubAccessors(pkgname, titles)
}
```

`ExperimentHub::createHubAccessors()` and
`ExperimentHub:::.hubAccessorFactory()` provide internal
detail. The resource-named function has a single 'metadata'
argument. When metadata=TRUE, the metadata are loaded (equivalent
to single-bracket method on an ExperimentHub object) and when
FALSE the full resource is loaded (equivalent to double-bracket
method).

Bug fixes

A bug fix may involve a change to the metadata, data resource or both.

Remove resources

Removing resources should be done with caution. The intent is that resources in the Hubs be ‘reproducible’ research by providing a stable snapshot of the data. Data made available in Bioconductor version x.y.z should be available for all versions greater than x.y.z. Unfortunately this is not always possible. If you find it necessary to remove data from AnnotationHub/ExperimentHub please contact [email protected] or [email protected] for assistance.

When a resource is removed from ExperimentHub or AnnotationHub two things happen: the ‘rdatadateremoved’ field is populated with a date and the ‘status’ field is populated with a reason why the resource is no longer available. Once these changes are made, the ExperimentHub() or AnnotationHub() constructor will not list the resource among the available ids. An attempt to extract the resource with ‘[[’ and the EH/AH id will return an error along with the status message. The function getInfoOnIds() will display metadata information for any resource including resources still in the database but no longer available.

In general, resources are only removed when they are no longer available (e.g., moved from web location, no longer provided etc.).

To remove a resource from AnnotationHub contact [email protected] or [email protected].

Versioning

Versioning of resources is handled by the maintainer. If you plan to provide incremental updates to a file for the same organism / genome build, we recommend including a version in the title of the resource so it is easy to distinguish which is most current. We also would recommend when uploading the data to genomic data lake or your publicly accessible site to have a directory structure accounting for versioning.

If you do not include a version, or make the title unique in some way, multiple files with the same title will be listed in the ExperimentHub or AnnotationHub object. The user will have to use the ‘rdatadateadded’ metadata field to determine which file is the most current or try an infer from ids which can lead to confusion.

Visibility

Several metadata fields control which resources are visible when a user invokes ExperimentHub()/AnnotationHub(). Records are filtered based on these criteria:

• ‘snapshotdate’ >= the date of the Bioconductor release being used
• ‘rdatadateadded’ >= today’s date
• ‘rdatadateremoved’ is NULL / NA
• ‘biocVersion’ is <= to the Bioconductor version being used

Once a record is added to ExperimentHub/AnnotationHub it is visible from that point forward until stamped with ‘rdatadateremoved’. For example, a record added on May 1, 2017 with ‘biocVersion’ 3.6 will be visible in all snapshots >= May 1, 2017 and in all Bioconductor versions >= 3.6.

A special filter for OrgDb is utilized in AnnotationHub. Only one OrgDb is available per release/devel cycle. Therefore contributed OrgDb added to a devel cycle are masked until the following release. There are options for debugging these masked resources. See ?setAnnotationHubOption

Storage of Data Files

The data should not be included in the package. This keeps the package light weight and quick for a user to install. This allows the user to investigate functions and documentation without downloading large data files and only proceeding with the download when necessary. There are two options for storing data: Bioconductor Microsoft Azure Genomic Data Lake or hosting the data elsewhere on a publicly accessible site. See information below and choose the option that fits best for your situation.

if (!requireNamespace("AzureStor", quietly = TRUE))
    install.packages("AzureStor")

library("AzureStor")

sas <- <sas token>
url <- "https://bioconductorhubs.blob.core.windows.net"
ep <- storage_endpoint(url, sas = sas)
container <- storage_container(ep, "staginghub")

files <- dir("/home/user/mypackage/data", recursive=TRUE)
src <- dir("/home/user/mypackage/data", recursive=TRUE, full.names=TRUE)
dest <- paste0("mypackage/", files)
storage_multiupload(container, src=src, dest=dest)

library(httr)

# get the list of files for the repository
response <- GET("https://api.github.com/repos/MyGithub/MyPackage/git/trees/master?recursive=1")

# get the blob urls and file names
src <- sapply(content(response)$tree, function(elt) elt$url)
names <- sapply(content(response)$tree, function(elt) elt$path)

# filter for the files in the data directory
# if you are uploaded subdirectories filter out the github blob for the
# directory name, subdirectories will be created automatically
keep <- grepl("^data/", names)
src <- src[keep]
names <- names[keep]

# we want the data in a directory with the package name
# the data should only have relevant subdirectories
dest = paste0("MyPackage/", gsub("data/","",names))

# upload to azure
multicopy_url_to_storage(container, src=src, dest=dest)

token = <github PAT>
auth_header = paste("token", token)
multicopy_url_to_storage(container, src=src, dest=dest, auth_header = auth_header)

azcopy copy --recursive MyPackageData <sas url>

Validating

The best way to validate record metadata is to read inst/extdata/metadata.csv (or aptly named csv file in inst/extdata) using the AnnotationHubData::makeAnnotationHubMetadata() or ExperimentHubData::makeExperimentHubMetadata(). If that is successful the metadata should be valid and able to be entered into the database.

Example metadata.csv file and more information

As described above the metadata.csv file (or multiple metadata.csv files) will need to be created before the data can be added to the database. To ensure proper formatting one should run AnnotationHubData::makeAnnotationHubMetadata or ExperimentHubData::makeExperimentHubMetadata on the package with any/all metadata files, and address any ERRORs that occur. Each data object uploaded to data server should have an entry (row) in the metadata file. Briefly, a description of the metadata columns required:

• Title: ‘character(1)’. Name of the resource. This can be the exact file name (if self-describing) or a more complete description. For ExperimentHub resources, the title can be used to generate the function name to load the resource, so spaces should be avoided in this case.
• Description: ‘character(1)’. Brief description of the resource, similar to the ‘Description’ field in a package DESCRIPTION file.
• BiocVersion: ‘character(1)’. The first Bioconductor version the resource was made available for. Unless removed from the hub, the resource will be available for all versions greater than or equal to this field. This generally is the current devel version of Bioconductor.
• Genome: ‘character(1)’. Genome. Can be NA.
• SourceType: ‘character(1)’. Format of original data, e.g., FASTA, BAM, BigWig, etc. ‘AnnotationHubData::getValidSourceTypes()’ list currently acceptable values. If nothing seems appropiate for your data reach out to [email protected]
• SourceUrl: ‘character(1)’. Location of original data files. Multiple urls should be provided as a comma separated string. If the data is simulated we recommend putting either a lab url or the url of the Bioconductor package.
• SourceVersion: ‘character(1)’. Version of original data.
• Species: ‘character(1)’. Species. For help on valid species see ‘getSpeciesList, validSpecies, or suggestSpecies.’ Can be NA.
• TaxonomyId: ‘character(1)’. Taxonomy ID. There are checks for valid taxonomyId given the Species which produce warnings. See GenomeInfoDb::loadTaxonomyDb() for full validation table. Can be NA.
• Coordinate_1_based: ‘logical’. TRUE if data are 1-based. Can be NA.
• DataProvider: ‘character(1)’. Name of company or institution that supplied the original (raw) data.
• Maintainer: ‘character(1)’. Maintainer name and email in the following format: Maintainer Name username@address.
• RDataClass: ‘character(1)’. R / Bioconductor class the data are stored in, e.g., GRanges, SummarizedExperiment, ExpressionSet etc. If the file is loaded or read into R what is the class of the object.
• DispatchClass: ‘character(1)’. Determines how data are loaded into R. The value for this field should be ‘Rda’ if the data were serialized with ‘save()’ and ‘Rds’ if serialized with ‘saveRDS’. The filename should have the appropriate ‘rda’ or ‘rds’ extension. There are other available DispathClass types and the function ‘AnnotationHub::DispatchClassList()’ will output a matrix of currently implemented DispatchClass and brief description of utility. If a predefine class does not seem appropriate contact [email protected]. An all purpose DispathClass is FilePath that instead of trying to load the file into R, will only return the path to the locally downloaded file.
• Location_Prefix: ‘character(1)’. Do not include this field if data are stored in the Bioconductor Data Lake; it will be generated automatically. If data will be accessed from a location other than Bioconductor’s Microsoft Data Lake location, this field should be the base url.
• RDataPath: ‘character(1)’.This field should be the remainder of the path to the resource. The ‘Location_Prefix’ will be prepended to ‘RDataPath’ for the full path to the resource. If the resource is stored in Bioconductor’s AWS S3 buckets, it should start with the name of the package associated with the metadata and should not start with a leading slash. It should include the resource file name.
• Tags: ‘character() vector’. ‘Tags’ are search terms used to define a subset of resources in a ‘Hub’ object, e.g, in a call to ‘query’. Multiple ‘Tags’ are specified as a colon separated string, e.g., tags for two resources would look “tag1:tag2”. The tags in the database are a combination of any individually listed ‘Tag’ provided here in the metadata file as well as any biocViews terms listed in the DESCRIPTION of the package

Any additional columns in the metadata.csv file will be ignored but could be included for internal reference.

More on Location_Prefix and RDataPath. These two fields make up the complete file path url for downloading the data file. If using the Bioconductor Microsoft Azure Genomic Data Lake the Location_Prefix should not be included in the metadata file[s] as this field will be populated automatically. The RDataPath will be the directory structure you uploaded to the Data Lake. If you uploaded a directory MyAnnotation/, and that directory had a subdirectory v1/ that contained two files counts.rds and coldata.rds, your metadata file will contain two rows and the RDataPaths would be MyAnnotation/v1/counts.rds and MyAnnotation/v1/coldata.rds. If you host your data on a publicly accessible site you must include a base url as the Location_Prefix. If your data file was at ftp://myinstiututeserver/biostats/project2/counts.rds, your metadata file will have one row and the Location_Prefix would be ftp://myinstiututeserver/ and the RDataPath would be biostats/project2/counts.rds.

This is a bad example because these annotations are already in the hubs but it should give you an idea of the format for AnnotationHub. Let’s say I have a package myAnnotations and I upload two annotation files for dog and cow with information extracted from ensembl to Bioconductor’s Data Lake location. You would want the following saved as a csv (comma seperated output) but for easier view we show in a table:

This is a dummy example but hopefully it will give you an idea of the format for ExperimentHub. Let’s say I have a package myExperimentPackage and I upload two files one a SummarizedExperiments of expression data saved as a .rda and the other a sqlite database both considered simulated data. You would want the following saved as a csv (comma seperated output) but for easier view we show in a table: