GDS (Genomic Data Structure) is a high-performance file format for storing and accessing large-scale genomic data, implemented by the gdsfmt package. It supports hierarchical data organization with efficient random access and data compression.
The gdscloud package extends gdsfmt to provide transparent read-only access
to GDS files stored on cloud storage services or any HTTP/HTTPS URL:
http:// or https:// URLs)s3://bucket/key), and S3-compatible services such as
MinIO, Ceph, Cloudflare R2, Wasabi or Backblaze B2 via a custom endpointgs://bucket/key)az://container/blob)# Install from Bioconductor
if (!requireNamespace("BiocManager", quietly=TRUE))
install.packages("BiocManager")
BiocManager::install("gdscloud")
LIB_CURL to point to the curl
toolchain path used by src/Makevars.win.Xiuwen Zheng
Once gdscloud is loaded, openfn.gds() from the gdsfmt package automatically recognizes cloud URLs and opens them transparently. Since gdscloud works transparently through openfn.gds(), packages built on gdsfmt, such as SeqArray, can open cloud-hosted files directly.
library(gdscloud)
# Open a GDS file from S3 — transparent via openfn.gds()
gds <- openfn.gds("s3://gds-stat/download/hapmap/hapmap_r23a.gds")
gds
## File: s3://gds-stat/download/hapmap/hapmap_r23a.gds (86.5M)
## + [ ] *
## |--+ description [ ] *
## |--+ sample.id { Str8 270 LZMA_ra(17.3%), 381B } *
## |--+ variant.id { Int32 4098136 LZMA_ra(3.19%), 511.4K } *
## |--+ position { Int32 4098136 LZMA_ra(48.2%), 7.5M } *
## |--+ chromosome { Str8 4098136 LZMA_ra(0.02%), 1.7K } *
## |--+ allele { Str8 4098136 LZMA_ra(12.8%), 2.0M } *
## |--+ genotype [ ] *
## | |--+ data { Bit2 2x270x4098136 LZMA_ra(12.4%), 65.4M } *
## | |--+ extra.index { Int32 3x0 LZMA_ra, 18B } *
## | \--+ extra { Int16 0 LZMA_ra, 18B }
## |--+ phase [ ]
## | |--+ data { Bit1 270x4098136 LZMA_ra(0.01%), 19.8K } *
## | |--+ extra.index { Int32 3x0 LZMA_ra, 18B } *
## | \--+ extra { Bit1 0 LZMA_ra, 18B }
## |--+ annotation [ ]
## | |--+ id { Str8 4098136 LZMA_ra(27.7%), 11.1M } *
## | |--+ qual { Float32 4098136 LZMA_ra(0.02%), 2.5K } *
## | |--+ filter { Int32,factor 4098136 LZMA_ra(0.02%), 2.5K } *
## | |--+ info [ ]
## | \--+ format [ ]
## \--+ sample.annotation [ ]
## |--+ family { Str8 270 LZMA_ra(17.3%), 381B } *
## |--+ father { Str8 270 LZMA_ra(28.2%), 261B } *
## |--+ mother { Str8 270 LZMA_ra(28.2%), 261B } *
## |--+ sex { Str8 270 LZMA_ra(27.0%), 153B } *
## \--+ phenotype { Int32 270 LZMA_ra(8.70%), 101B } *
read.gdsn(index.gdsn(gds, "chromosome"))
## "1", ...
read.gdsn(index.gdsn(gds, "position"))
## 45162 45257 45413 46844 72434 72515 ...
closefn.gds(gds)
Note:
seqParallel() to automatically load cloud-related packages on worker processes, so parallel operations on cloud-hosted GDS files work seamlessly.For public URLs, no authentication is needed:
gds <- gdsCloudOpen("https://example.com/path/to/file.gds")
closefn.gds(gds)
For authenticated endpoints, set a Bearer token via environment variable
(in R, use Sys.setenv(), or add GDSCLOUD_HTTP_TOKEN=your_token to your
~/.Renviron file to persist it across sessions):
Sys.setenv(GDSCLOUD_HTTP_TOKEN = "your_token")
Or configure in R:
gdsCloudConfigHTTP(bearer_token = "your_token")
# URL-specific token
gdsCloudConfigHTTP(bearer_token = "other_token", url = "https://private.example.com/")
# only http:// and https:// prefixes are accepted for `url`
Token arguments (access_token for GCS and Azure, bearer_token for HTTP)
can also be a function returning the token; it is called each time a file
is opened, so expiring tokens are refreshed automatically (e.g.
gdsCloudConfigGCS(access_token = function() system2("gcloud", c("auth", "print-access-token"), stdout = TRUE))). For AWS
temporary credentials, gdsCloudConfigS3(credentials = function() list(...))
returns the access key, secret and session token together.
Set environment variables (via Sys.setenv() in R, or the corresponding
NAME=value lines in ~/.Renviron):
Sys.setenv(
AWS_ACCESS_KEY_ID = "your_key",
AWS_SECRET_ACCESS_KEY = "your_secret",
AWS_DEFAULT_REGION = "us-east-1"
)
# Optional temporary-credential token:
Sys.setenv(AWS_SESSION_TOKEN = "your_token")
Or configure in R:
gdsCloudConfigS3(
aws_access_key_id = "your_key",
aws_secret_access_key = "your_secret",
region = "us-east-1"
)
For an S3-compatible service, add its endpoint (path-style addressing is used by default for custom endpoints):
gdsCloudConfigS3(
aws_access_key_id = "minioadmin",
aws_secret_access_key = "minioadmin",
endpoint = "http://minio.lab.internal:9000"
)
gds <- gdsCloudOpen("s3://genomics/hapmap.gds")
The endpoint can also come from the AWS_ENDPOINT_URL environment variable.
Sys.setenv(GCS_ACCESS_TOKEN = "your_token")
# or add GCS_ACCESS_TOKEN=your_token to ~/.Renviron
Or:
gdsCloudConfigGCS(access_token = "your_token")
Sys.setenv(
AZURE_STORAGE_ACCOUNT = "your_account",
AZURE_STORAGE_KEY = "your_key"
)
# Or use a SAS token instead:
Sys.setenv(AZURE_STORAGE_SAS_TOKEN = "your_sas")
# or add the corresponding NAME=value lines to ~/.Renviron
Or:
gdsCloudConfigAzure(account_name = "your_account", account_key = "your_key")
# Entra ID (OAuth2) token instead of a key, e.g. from
# az account get-access-token --resource https://storage.azure.com/
gdsCloudConfigAzure(account_name = "your_account", access_token = "eyJ0eXAi...")
# sovereign clouds and local emulators
gdsCloudConfigAzure(endpoint_suffix = "blob.core.chinacloudapi.cn")
gdsCloudConfigAzure(endpoint = "http://127.0.0.1:10000/devstoreaccount1")
# Set cache size (default: 64MB)
gdsCloudCacheSize(128)
# Clear all caches
gdsCloudCacheClear()
# Show cache statistics
gdsCloudCacheInfo()