Package: BenchHub 0.99.15

SOMS Maintainer

BenchHub: Comprehensive Collection of Curated Benchmarking Datasets and their Evaluation

The trio is the combination of a data set, a metric and supporting evidence which provides some best case scenario, if not the ground truth itself. BenchHub has data downloaders for FigShare, G.E.O., and ExperimentHub. Caching is used to avoid lengthy downloads after the first time a data set is accessed. The user may also specify their own data set and supporting evidence. The Benchmark Insights module provides functionality for comparing and contrasting the performance of alternative algorithms.

Authors:Cabiria Liang [aut], Sanghyun Kim [aut], Nick Robertson [aut], Marni Torkel [aut], Yue Cao [aut], Dario Strbenac [aut], Jean Yang [aut], SOMS Maintainer [aut, cre]

BenchHub_0.99.15.tar.gz
BenchHub_0.99.15.zip(r-4.7)BenchHub_0.99.15.zip(r-4.6)BenchHub_0.99.15.zip(r-4.5)
BenchHub_0.99.15.tgz(r-4.6-any)BenchHub_0.99.15.tgz(r-4.5-any)
BenchHub_0.99.15.tar.gz(r-4.7-any)BenchHub_0.99.15.tar.gz(r-4.6-any)
BenchHub_0.99.15.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
DESCRIPTION |NEWS
card.svg |card.png
BenchHub/json (API)

# Install 'BenchHub' in R:
install.packages('BenchHub', repos = c('https://bioc.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/sydneybiox/benchhub/issues

Pkgdown/docs site:https://sydneybiox.github.io

Datasets:
  • lubomPD - Human Gut Microbiome Abundance and Patient Classes
  • x - Human Gut Microbiome Abundance and Patient Classes

On BioConductor:BenchHub-0.99.15(bioc 3.24)

infrastructuredataimportdatarepresentationsoftwareworkflowstepvisualization

6.59 score 12 scripts 59 exports 112 dependencies

Last updated from:7f702ca89f. Checks:8 NOTE, 2 OK. Indexed: yes.

TargetResultTimeFilesSyslog
bioc-checksNOTE336
linux-devel-x86_64NOTE408
source / vignettesOK445
linux-release-x86_64NOTE373
macos-release-arm64NOTE250
macos-oldrel-arm64NOTE167
windows-develNOTE546
windows-releaseNOTE498
windows-oldrelNOTE576
wasm-releaseOK279

Exports:ARImetricbalAccMetricbalErrMetricbeggCIndexMetricBenchmarkInsightsBenchmarkStudybrierScoreMetricbuildDatasetEvidenceSubmissionbuildDatasetSubmissionbuildDatasetTaskMetricSubmissionbuildDatasetTaskSubmissionbuildMetricSubmissionbuildStudySubmissionbuildStudySubmissionPayloadbuildTrioSubmissionbuildTrioSubmissionPayloadcollectDatasetSubmissionInfocollectDatasetTaskMetricSubmissioncollectEvidenceSubmissionInfocollectMetricSubmissionInfocollectStudySubmissionInfocollectTaskSubmissionInfodownloadSubmissionStudydownloadSubmissionTriogetSubmissionStudygetSubmissionStudyDatasetsghCIndexMetricharrelCIndexMetricinteractivePrepareStudySubmissioninteractivePrepareStudyUpdateSubmissionJSDmetrickdeMetriclistCuratedTrioDatasetslistCuratedTrioStudieslistSubmissionStudieslistSubmissionStudyDatasetsmacroF1MetricmacroPrecMetricmacroRecMetricMCCmetricmicroF1MetricmicroPrecMetricmicroRecMetricMSEmetricNMImetricprepareStudySubmissionprepareStudyUpdateSubmissionprepareTrioSubmissionBundleprepareTrioSubmissionFilesprepareTrioSubmissionMetricsRMSEmetricstudySubmissionToJSONsubmitStudySubmissionsubmitTrioSubmissiontimeDependentAUCMetricTriotrioSubmissionToJSONunoCIndexMetricwriteSubmission

Dependencies:askpassbackportsbase64encbayestestRbroombslibcachemcellrangercheckmatecliclustercodetoolscolorspacecpp11curldata.tabledatawizarddigestdotwhiskerdplyrevaluatefarverfastmapfontawesomeforeignFormulafsgarglegenericsggcorrplotggplot2ggrepelggsciggstancegluegoogledrivegooglesheets4gridExtragtablehighrHmischtmlTablehtmltoolshtmlwidgetshttrhttr2idsinsightisobandjquerylibjsonliteknitrlabelinglatticelifecyclemagrittrmarginaleffectsMASSMatrixMatrixModelsmemoisemimemultcompmvtnormnlmennetopensslparameterspatchworkperformancepillarpkgconfigplyrpolsplinepurrrquantregR6rappdirsRColorBrewerRcpprematchrematch2reshape2rlangrmarkdownrmsrpartrstudioapiS7sandwichsassscalesSparseMsplitToolsstringistringrsurvAUCsurvivalsysTH.datatibbletidyrtidyselecttinytexutf8uuidvctrsviridisLitewithrxfunyamlzoo

3 Introduction of BenchmarkInsights class
Motivation | Creating BenchmarkInsights class | Visualization | Available plot | Interpretation benchmark result | Case Study: What is the overview of summary? | Case Study: What is the correlation between evidence/metric/method? | Case Study: What is the time and memory trend? | Case Study: Which metric is most effective on the method? | Case Study: How does method variability differ across datasets for a specific metric? | Cheatsheet | Session Info
Rendered fromv03_intro_bmi.Rmdusingknitr::rmarkdown

Last update: 2026-04-23
Started: 2025-03-11

5 BenchmarkStudy
Overview | For Benchmark Developer | Initialising the Study | Define mapping function and protocol function | Upload Study | For Method Developer | Loading the Study | Preparing for evaluation | Evaluate | Summary | Session Info
Rendered fromv05_benchmarkstudy.Rmdusingknitr::rmarkdown

Last update: 2026-04-23
Started: 2025-12-09

4 Preparing and Submitting a Trio
Preparing and Submitting a Trio | Step 1: GitHub Personal Access Token (PAT) | Set the PAT in R | Step 2: Build the submission bundle | Step 3: Submit the Trio | Step 4: Verify the submission | Step 5: Downloading an existing Trio | Notes | Session Info
Rendered fromv04_uploading_a_Trio.rmdusingknitr::rmarkdown

Last update: 2026-04-23
Started: 2025-07-02

2 Evaluation using Trio
Import microbiome data | Initialise a Trio object | Cross validation - using the trio split function | Without Cross-validation | Importing result to BenchmarkInsights | Session Info
Rendered fromv02_Evaluation_using_Trio.Rmdusingknitr::rmarkdown

Last update: 2026-04-23
Started: 2025-03-07

1 Introduction to the Trio Class
Creating Trio object | Method 1: BenchHub datasets | Method 2: Source and ID | Method 3: Load an object directly | Bonus: Using a custom loader | Adding Components to Trio Sequentially | Adding metrics | Other Features | Caching | Data Splitting | Conclusion | Session Info
Rendered fromv01_intro_trio.Rmdusingknitr::rmarkdown

Last update: 2026-04-23
Started: 2025-03-06

Readme and manuals

Help Manual

Help pageTopics
Adjusted Rand Index (ARI) MetricARImetric
Balanced Accuracy MetricbalAccMetric
Balanced Error MetricbalErrMetric
Begg's C-Index MetricbeggCIndexMetric
A BenchmarkInsights objectBenchmarkInsights
BenchmarkStudy ClassBenchmarkStudy
Brier Score MetricbrierScoreMetric
Build DatasetEvidence submission rows from a Trio objectbuildDatasetEvidenceSubmission
Build a Dataset submission row from a Trio objectbuildDatasetSubmission
Build DatasetTaskMetric submission rows from a Trio objectbuildDatasetTaskMetricSubmission
Build DatasetTask submission rows from a Trio objectbuildDatasetTaskSubmission
Build Metric submission rows from a Trio objectbuildMetricSubmission
Build Study and StudyDataset submission tablesbuildStudySubmission
Convert a Study submission to payload structurebuildStudySubmissionPayload
Build a combined Trio submission objectbuildTrioSubmission
Convert a combined Trio submission to payload structurebuildTrioSubmissionPayload
Collect Dataset submission metadata interactivelycollectDatasetSubmissionInfo
Collect DatasetTaskMetric submission rows from Trio assignmentscollectDatasetTaskMetricSubmission
Collect DatasetEvidence submission metadata interactivelycollectEvidenceSubmissionInfo
Collect Metric submission metadata interactivelycollectMetricSubmissionInfo
Collect Study submission metadatacollectStudySubmissionInfo
Collect DatasetTask submission metadata interactivelycollectTaskSubmissionInfo
Download a BenchmarkStudy from the submission databasedownloadSubmissionStudy
Download a Trio from the five-table submission databasedownloadSubmissionTrio
Get one existing Study row by studyIDgetSubmissionStudy
Get linked StudyDataset rows by studyIDgetSubmissionStudyDatasets
GH C-Index MetricghCIndexMetric
Harrel's C-Index MetricharrelCIndexMetric
Interactively prepare a Study submissioninteractivePrepareStudySubmission
Interactively prepare a Study update submissioninteractivePrepareStudyUpdateSubmission
Jensen-Shannon Divergence (JSD) MetricJSDmetric
Kernel Density Estimation (KDE) MetrickdeMetric
List the curated Trio datasetslistCuratedTrioDatasets
List the curated Trio studieslistCuratedTrioStudies
List existing Study rowslistSubmissionStudies
List available datasets for Study submissionlistSubmissionStudyDatasets
Human Gut Microbiome Abundance and Patient ClasseslubomPD lubomski_microbiome_data x
Macro F1 Score MetricmacroF1Metric
Macro Precision MetricmacroPrecMetric
Macro Recall MetricmacroRecMetric
Matthews Correlation Coefficient (MCC) MetricMCCmetric
Micro F1 Score MetricmicroF1Metric
Micro Precision MetricmicroPrecMetric
Micro Recall MetricmicroRecMetric
Mean Squared Error (MSE) MetricMSEmetric
Normalized Mutual Information (NMI) MetricNMImetric
Prepare a Study submission bundleprepareStudySubmission
Prepare an update submission from an existing Study versionprepareStudyUpdateSubmission
Build a pre-submit Trio submission bundleprepareTrioSubmissionBundle
Prepare Trio files and download metadata for submissionprepareTrioSubmissionFiles
Prepare Trio metric metadata for submissionprepareTrioSubmissionMetrics
Root Mean Squared Error (RMSE) MetricRMSEmetric
Convert a Study submission to JSONstudySubmissionToJSON
Submit a Study submission payload to Google Apps ScriptsubmitStudySubmission
Submit a Trio submission payload to Google Apps ScriptsubmitTrioSubmission
Time-Dependent AUC MetrictimeDependentAUCMetric
A Trio objectTrio
Convert a combined Trio submission to JSONtrioSubmissionToJSON
Uno's C-Index MetricunoCIndexMetric
Interactively prepare a Trio submission bundlewriteSubmission