Package 'onlineFDR'

ADDIS: Adaptive discarding algorithm for online FDR control

Description

Implements the ADDIS algorithm for online FDR control, where ADDIS stands for an ADaptive algorithm that DIScards conservative nulls, as presented by Tian and Ramdas (2019). The algorithm compensates for the power loss of SAFFRON with conservative nulls, by including both adaptivity in the fraction of null hypotheses (like SAFFRON) and the conservativeness of nulls (unlike SAFFRON).

Usage

ADDIS(
  d,
  alpha = 0.05,
  gammai,
  w0,
  lambda = 0.25,
  tau = 0.5,
  async = FALSE,
  random = TRUE,
  display_progress = FALSE,
  date.format = "%Y-%m-%d"
)

Arguments

d	Either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), date (‘date’) and p-value (‘pval’). For the asynchronous version, the date column is replaced by decision times (‘decision.times’).
alpha	Overall significance level of the procedure, the default is 0.05.
gammai	Optional vector of $\gamma_i$. A default is provided with $\gamma_j$ proportional to $1/j^{1.6}$.
w0	Initial ‘wealth’ of the procedure, defaults to $\alpha/2$.
lambda	Optional parameter that sets the threshold for ‘candidate’ hypotheses. Must be between 0 and tau, defaults to 0.25.
tau	Optional threshold for hypotheses to be selected for testing. Must be between 0 and 1, defaults to 0.5.
async	Logical. If TRUE runs the version for an asynchronous testing process. Defaults to FALSE.
random	Logical. If TRUE (the default), then the order of the p-values in each batch (i.e. those that have exactly the same date) is randomised. Only needed if async=FALSE.
display_progress	Logical. If TRUE prints out a progress bar for the algorithm runtime.
date.format	Optional string giving the format that is used for dates.

Details

The function takes as its input either a vector of p-values, or a dataframe with three columns. The dataframe requires an identifier (‘id’), date (‘date’) and p-value (‘pval’). If the asynchronous version is specified (see below), then the column date should be replaced by the decision times.

Given an overall significance level $\alpha$, ADDIS depends on constants $w_0$, $\lambda$ and $\tau$. Here $w_0$ represents the initial ‘wealth’ of the procedure and satisfies $0 \le w_0 \le \alpha$. $\tau \in (0,1]$ represents the threshold for a hypothesis to be selected for testing: p-values greater than $\tau$ are implicitly ‘discarded’ by the procedure. Finally, $\lambda \in (0,\tau)$ sets the threshold for a p-value to be a candidate for rejection: ADDIS will never reject a p-value larger than $\lambda$. The algorithm also requires a sequence of non-negative non-increasing numbers $\gamma_i$ that sum to 1.

The ADDIS procedure provably controls the FDR for independent p-values. Tian and Ramdas (2019) also presented a version for an asynchronous testing process, consisting of tests that start and finish at (potentially) random times. The discretised finish times of the tests correspond to the decision times. These decision times are given as the input decision.times. Note that this asynchronous version controls the modified FDR (mFDR) and, under the conditions given in Tian and Ramdas (2019), Appendix C, also the FDR.

Further details of the ADDIS algorithms can be found in Tian and Ramdas (2019).

Value

out

A dataframe with the p-values pval (reordered if there are batches, async = FALSE and random = TRUE), the adjusted significance thresholds $\alpha_i$, the indicator function of discoveries R and, if d is a dataframe with a column id, the identifiers id. Hypothesis $i$ is rejected if the $i$-th p-value is less than or equal to $\alpha_i$, in which case R[i] = 1 (otherwise R[i] = 0).

References

Tian, J. and Ramdas, A. (2019). ADDIS: an adaptive discarding algorithm for online FDR control with conservative nulls. Advances in Neural Information Processing Systems 32, 9388-9396.

See Also

SAFFRON is the special case of ADDIS with tau = 1.

SAFFRONstar provides asynchronous versions of SAFFRON.

Examples

sample.df1 <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
date = as.Date(c(rep('2014-12-01',3),
                rep('2015-09-21',5),
                rep('2016-05-19',2),
                '2016-11-12',
                rep('2017-03-27',4))),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757))

ADDIS(sample.df1, random=FALSE)


sample.df2 <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757),
decision.times = seq_len(15) + 1)

ADDIS(sample.df2, async = TRUE) # Asynchronous

---

ADDIS-exhaustive: Exhaustive ADDIS-spending procedure for online FWER control

Description

Implements an exhaustive variant of the ADDIS-spending algorithm for online FWER control, as presented by Fischer et al. (2024). The procedure is a uniform improvement of ADDIS-spending, and, under the conditions of their Theorem 3.5, no other FWER controlling procedure can enlarge the event of rejecting any hypothesis.

Usage

ADDIS_exhaustive(d, alpha = 0.05, tau = 0.5, lambda = 0.25, gamma = NULL)

Arguments

d	Either a vector of p-values, or a dataframe with at least a pval column (and optionally id).
alpha	Overall significance level of the procedure, default 0.05.
tau	Optional threshold for hypotheses to be selected for testing. Must be between 0 and 1, defaults to 0.5.
lambda	Optional parameter that sets the threshold for ‘candidate’ hypotheses. Must be at least alpha * tau and less than tau, defaults to 0.25.
gamma	Optional vector of initial weights, with at least one more element than the number of p-values and summing to at most 1. If NULL (the default), a decreasing sequence proportional to $j^{-1.6}$ is used, as in ADDIS.

Details

The function takes as its input either a vector of p-values, or a dataframe with a column of p-values (‘pval’) and optionally an identifier (‘id’). Given an overall significance level $\alpha$, ADDIS-exhaustive depends on constants $\lambda$ and $\tau$, where $\alpha\tau \le \lambda < \tau$. Here $\tau \in (0,1]$ represents the threshold for a hypothesis to be selected for testing: p-values greater than $\tau$ are implicitly ‘discarded’ by the procedure, while $\lambda \in (0,1)$ sets the threshold for a p-value to be a candidate for rejection: ADDIS-exhaustive will never reject a p-value larger than $\lambda$. The algorithm also requires a sequence of non-negative numbers $\gamma_i$ that sum to at most 1.

The ADDIS-exhaustive procedure provably controls the FWER in the strong sense for independent p-values.

Value

A dataframe with the original p-values pval, the per-hypothesis testing levels alphai, and the indicator of discoveries R.

Author(s)

Lasse Fischer

References

Fischer, L., Bofill Roig, M. and Brannath, W. (2024). An exhaustive ADDIS principle for online FWER control. Biometrical Journal, 66(3):2300237.

---

ADDIS-spending: Adaptive discarding algorithm for online FWER control

Description

Implements the ADDIS algorithm for online FWER control, where ADDIS stands for an ADaptive algorithm that DIScards conservative nulls, as presented by Tian and Ramdas (2021). The procedure compensates for the power loss of Alpha-spending, by including both adaptivity in the fraction of null hypotheses and the conservativeness of nulls.

Usage

ADDIS_spending(
  d,
  alpha = 0.05,
  gammai,
  lambda = 0.25,
  tau = 0.5,
  dep = FALSE,
  display_progress = FALSE
)

Arguments

d	Either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), p-value (‘pval’), and lags (‘lags’).
alpha	Overall significance level of the procedure, the default is 0.05.
gammai	Optional vector of $\gamma_i$. A default is provided with $\gamma_j$ proportional to $1/j^{1.6}$.
lambda	Optional parameter that sets the threshold for ‘candidate’ hypotheses. Must be between 0 and tau, defaults to 0.25.
tau	Optional threshold for hypotheses to be selected for testing. Must be between 0 and 1, defaults to 0.5.
dep	Logical. If TRUE runs the version for locally dependent p-values.
display_progress	Logical. If TRUE prints out a progress bar for the algorithm runtime.

Details

The function takes as its input either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), p-value (‘pval’), and lags, if the dependent version is specified (see below). Given an overall significance level $\alpha$, ADDIS-spending depends on constants $\lambda$ and $\tau$, where $\lambda < \tau$. Here $\tau \in (0,1]$ represents the threshold for a hypothesis to be selected for testing: p-values greater than $\tau$ are implicitly ‘discarded’ by the procedure, while $\lambda \in (0,1)$ sets the threshold for a p-value to be a candidate for rejection: ADDIS-spending will never reject a p-value larger than $\lambda$. The algorithms also require a sequence of non-negative numbers $\gamma_i$ that sum to 1.

The ADDIS-spending procedure provably controls the FWER in the strong sense for independent p-values. Note that the procedure also controls the generalised familywise error rate (k-FWER) for $k > 1$ if $\alpha$ is replaced by min($1, k\alpha$).

Tian and Ramdas (2021) also presented a version for handling local dependence. More precisely, for any $t>0$ we allow the p-value $p_t$ to have arbitrary dependence on the previous $L_t$ p-values. The fixed sequence $L_t$ is referred to as ‘lags’, and is given as the input lags for this version of the ADDIS-spending algorithm. The lags must satisfy $L_{t+1} \le L_t + 1$.

Further details of the ADDIS-spending algorithms can be found in Tian and Ramdas (2021).

Value

out	A dataframe with the original p-values pval, the adjusted testing levels $\alpha_i$ and the indicator function of discoveries R. Hypothesis $i$ is rejected if the $i$-th p-value is less than or equal to $\alpha_i$, in which case R[i] = 1 (otherwise R[i] = 0).

References

Tian, J. and Ramdas, A. (2021). Online control of the familywise error rate. Statistical Methods in Medical Research, 30(4):976-993.

See Also

ADDIS provides online control of the FDR.

Examples

sample.df <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757),
lags = rep(1,15))

ADDIS_spending(sample.df) #independent

ADDIS_spending(sample.df, dep = TRUE) #Locally dependent

---

Alpha-investing for online FDR control

Description

Implements a variant of the Alpha-investing algorithm of Foster and Stine (2008) that guarantees FDR control, as proposed by Ramdas et al. (2018). This procedure uses SAFFRON's update rule with the constant $\lambda$ replaced by a sequence $\lambda_i = \alpha_i$. This is also equivalent to using the ADDIS algorithm with $\tau = 1$ and $\lambda_i = \alpha_i$.

Usage

Alpha_investing(
  d,
  alpha = 0.05,
  gammai,
  w0,
  random = TRUE,
  display_progress = FALSE,
  date.format = "%Y-%m-%d"
)

Arguments

d	Either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), date (‘date’) and p-value (‘pval’). If no column of dates is provided, then the p-values are treated as being ordered in sequence.
alpha	Overall significance level of the FDR procedure, the default is 0.05.
gammai	Optional vector of $\gamma_i$. A default is provided with $\gamma_j$ proportional to $1/j^{1.6}$.
w0	Initial ‘wealth’ of the procedure, defaults to $\alpha/2$. Must be at least 0 and less than $\alpha$.
random	Logical. If TRUE (the default), then the order of the p-values in each batch (i.e. those that have exactly the same date) is randomised.
display_progress	Logical. If TRUE prints out a progress bar for the algorithm runtime.
date.format	Optional string giving the format that is used for dates.

Details

The function takes as its input either a vector of p-values or a dataframe with three columns: an identifier (‘id’), date (‘date’) and p-value (‘pval’). The case where p-values arrive in batches corresponds to multiple instances of the same date. If no column of dates is provided, then the p-values are treated as being ordered in sequence.

The Alpha-investing procedure provably controls FDR for independent p-values. Given an overall significance level $\alpha$, we choose a sequence of non-negative non-increasing numbers $\gamma_i$ that sum to 1. Alpha-investing depends on a constant $w_0$, which satisfies $0 \le w_0 < \alpha$ and represents the initial ‘wealth’ of the procedure.

Further details of the Alpha-investing procedure and its modification can be found in Foster and Stine (2008) and Ramdas et al. (2018).

Value

out

A dataframe with the p-values pval (reordered if there are batches and random = TRUE), the adjusted significance thresholds $\alpha_i$, the indicator function of discoveries R and, if d is a dataframe with a column id, the identifiers id. Hypothesis $i$ is rejected if the $i$-th p-value is less than or equal to $\alpha_i$, in which case R[i] = 1 (otherwise R[i] = 0).

References

Foster, D. and Stine, R. (2008). $\alpha$-investing: a procedure for sequential control of expected false discoveries. Journal of the Royal Statistical Society (Series B), 70(2):429-444.

Ramdas, A., Zrnic, T., Wainwright, M.J. and Jordan, M.I. (2018). SAFFRON: an adaptive algorithm for online control of the false discovery rate. Proceedings of the 35th International Conference on Machine Learning, 80:4286-4294.

See Also

SAFFRON uses the update rule of Alpha-investing but with constant $\lambda$.

Examples

sample.df <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
date = as.Date(c(rep('2014-12-01',3),
               rep('2015-09-21',5),
                rep('2016-05-19',2),
                '2016-11-12',
               rep('2017-03-27',4))),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757))

Alpha_investing(sample.df, random=FALSE)

set.seed(1); Alpha_investing(sample.df)

set.seed(1); Alpha_investing(sample.df, alpha=0.1, w0=0.025)

---

Alpha-spending for online FWER control

Description

Implements online FWER control using a Bonferroni-like test.

Usage

Alpha_spending(
  d,
  alpha = 0.05,
  gammai,
  random = TRUE,
  date.format = "%Y-%m-%d"
)

Arguments

d	Either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), date (‘date’) and p-value (‘pval’). If no column of dates is provided, then the p-values are treated as being ordered in sequence, arriving one at a time.
alpha	Overall significance level of the FWER procedure, the default is 0.05.
gammai	Optional vector of $\gamma_i$, where hypothesis $i$ is rejected if the $i$-th p-value is less than or equal to $\alpha \gamma_i$. A default is provided as proposed by Javanmard and Montanari (2018), equation 31.
random	Logical. If TRUE (the default), then the order of the p-values in each batch (i.e. those that have exactly the same date) is randomised.
date.format	Optional string giving the format that is used for dates.

Details

The function takes as its input either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), date (‘date’) and p-value (‘pval’). The case where p-values arrive in batches corresponds to multiple instances of the same date. If no column of dates is provided, then the p-values are treated as being ordered in sequence, arriving one at a time.

Alpha-spending provides strong FWER control for a potentially infinite stream of p-values by using a Bonferroni-like test. Given an overall significance level $\alpha$, we choose a (potentially infinite) sequence of non-negative numbers $\gamma_i$ such that they sum to 1. Hypothesis $i$ is rejected if the $i$-th p-value is less than or equal to $\alpha \gamma_i$.

Note that the procedure controls the generalised familywise error rate (k-FWER) for $k > 1$ if $\alpha$ is replaced by $\min(1, k\alpha)$.

Value

out	A dataframe with the original data d (which will be reordered if there are batches and random = TRUE), the adjusted significance thresholds alphai and the indicator function of discoveries R, where R[i] = 1 corresponds to hypothesis $i$ being rejected (otherwise R[i] = 0).

References

Javanmard, A. and Montanari, A. (2018). Online Rules for Control of False Discovery Rate and False Discovery Exceedance. Annals of Statistics, 46(2):526-554.

Tian, J. and Ramdas, A. (2021). Online control of the familywise error rate. Statistical Methods in Medical Research, 30(4):976-993.

Examples

sample.df <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
date = as.Date(c(rep('2014-12-01',3),
                rep('2015-09-21',5),
                rep('2016-05-19',2),
                '2016-11-12',
                rep('2017-03-27',4))),
pval = c(2.90e-17, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757))

set.seed(1); Alpha_spending(sample.df)

Alpha_spending(sample.df, random=FALSE)

set.seed(1); Alpha_spending(sample.df, alpha=0.1)

---

BatchBH: Online batch FDR control using the BH procedure

Description

Implements the BatchBH algorithm for online FDR control, as presented by Zrnic et al. (2020).

Usage

BatchBH(d, alpha = 0.05, gammai, display_progress = FALSE)

Arguments

d	A dataframe with three columns: identifiers (‘id’), batch numbers (‘batch’) and p-values (‘pval’).
alpha	Overall significance level of the FDR procedure, the default is 0.05.
gammai	Optional vector of $\gamma_i$. A default is provided with $\gamma_j$ proportional to $1/j^{1.6}$.
display_progress	Logical. If TRUE prints out a progress bar for the algorithm runtime.

Details

The function takes as its input a dataframe with three columns: identifiers (‘id’), batch numbers (‘batch’) and p-values (‘pval’).

The BatchBH algorithm controls the FDR when the p-values in a batch are independent, and independent across batches. Given an overall significance level $\alpha$, we choose a sequence of non-negative numbers $\gamma_i$ such that they sum to 1. The algorithm runs the Benjamini-Hochberg procedure on each batch, where the values of the adjusted significance thresholds $\alpha_{t+1}$ depend on the number of previous discoveries.

Further details of the BatchBH algorithm can be found in Zrnic et al. (2020).

Value

out

A dataframe with the original data d, the indicator function of discoveries R and the test level alphai ($\alpha_t$) of each hypothesis's batch. Hypothesis $i$ in the $t$-th batch is rejected if its p-value is less than or equal to $(k/n)\alpha_t$, where $n$ is the number of hypotheses in the batch and $k$ is the largest $r$ such that the $r$-th smallest p-value in the batch is at most $(r/n)\alpha_t$. If hypothesis $i$ is rejected, R[i] = 1 (otherwise R[i] = 0).

References

Zrnic, T., Jiang, D., Ramdas, A. and Jordan, M.I. (2020). The Power of Batching in Multiple Hypothesis Testing. International Conference on Artificial Intelligence and Statistics, PMLR, 108:3806-3815.

Examples

sample.df <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757),
batch = c(rep(1,5), rep(2,6), rep(3,4)))

BatchBH(sample.df)

---

BatchPRDS: Online batch FDR control under positive dependence

Description

Implements the BatchPRDS algorithm for online FDR control, where PRDS stands for positive regression dependency on a subset, as presented by Zrnic et al. (2020).

Usage

BatchPRDS(d, alpha = 0.05, gammai, display_progress = FALSE)

Arguments

d	A dataframe with three columns: identifiers (‘id’), batch numbers (‘batch’) and p-values (‘pval’).
alpha	Overall significance level of the FDR procedure, the default is 0.05.
gammai	Optional vector of $\gamma_i$. A default is provided with $\gamma_j$ proportional to $1/j^{1.6}$.
display_progress	Logical. If TRUE prints out a progress bar for the algorithm runtime.

Details

The function takes as its input a dataframe with three columns: identifiers (‘id’), batch numbers (‘batch’) and p-values (‘pval’).

The BatchPRDS algorithm controls the FDR when the p-values in one batch are positively dependent, and independent across batches. Given an overall significance level $\alpha$, we choose a sequence of non-negative numbers $\gamma_i$ such that they sum to 1. The algorithm runs the Benjamini-Hochberg procedure on each batch, where the values of the adjusted significance thresholds $\alpha_{t+1}$ depend on the number of previous discoveries.

Further details of the BatchPRDS algorithm can be found in Zrnic et al. (2020).

Value

out

A dataframe with the original data d, the indicator function of discoveries R and the test level alphai ($\alpha_t$) of each hypothesis's batch. Hypothesis $i$ in the $t$-th batch is rejected if its p-value is less than or equal to $(k/n)\alpha_t$, where $n$ is the number of hypotheses in the batch and $k$ is the largest $r$ such that the $r$-th smallest p-value in the batch is at most $(r/n)\alpha_t$. If hypothesis $i$ is rejected, R[i] = 1 (otherwise R[i] = 0).

References

Zrnic, T., Jiang, D., Ramdas, A. and Jordan, M.I. (2020). The Power of Batching in Multiple Hypothesis Testing. International Conference on Artificial Intelligence and Statistics, PMLR, 108:3806-3815.

Examples

sample.df <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757),
batch = c(rep(1,5), rep(2,6), rep(3,4)))

BatchPRDS(sample.df)

---

BatchStBH: Online batch FDR control using the St-BH procedure

Description

Implements the BatchSt-BH algorithm for online FDR control, as presented by Zrnic et al. (2020). Within each batch it uses the Storey-BH procedure (Storey 2002) with the null proportion estimate of Storey, Taylor and Siegmund (2004), which adds 1 to the numerator for more stable results.

Usage

BatchStBH(d, alpha = 0.05, gammai, lambda = 0.5, display_progress = FALSE)

Arguments

d	A dataframe with three columns: identifiers (‘id’), batch numbers (‘batch’) and p-values (‘pval’).
alpha	Overall significance level of the FDR procedure, the default is 0.05.
gammai	Optional vector of $\gamma_i$. A default is provided with $\gamma_j$ proportional to $1/j^{1.6}$.
lambda	Threshold for Storey-BH, must be between 0 and 1. Defaults to 0.5.
display_progress	Logical. If TRUE prints out a progress bar for the algorithm runtime.

Details

The function takes as its input a dataframe with three columns: identifiers (‘id’), batch numbers (‘batch’) and p-values (‘pval’).

The BatchSt-BH algorithm controls the FDR when the p-values in a batch are independent, and independent across batches. Given an overall significance level $\alpha$, we choose a sequence of non-negative numbers $\gamma_i$ such that they sum to 1. The algorithm runs the Storey Benjamini-Hochberg procedure on each batch, where the values of the adjusted significance thresholds $\alpha_{t+1}$ depend on the number of previous discoveries.

Further details of the BatchSt-BH algorithm can be found in Zrnic et al. (2020).

Value

out

A dataframe with the original data d, the indicator function of discoveries R and the test level alphai ($\alpha_t$) of each hypothesis's batch. Hypothesis $i$ in the $t$-th batch is rejected if its p-value is less than or equal to $(k/n)\alpha_t / \hat{\pi}_0$, where $n$ is the number of hypotheses in the batch, $k$ is the largest $r$ such that the $r$-th smallest p-value in the batch is at most $(r/n)\alpha_t / \hat{\pi}_0$, and $\hat{\pi}_0 = (1 + n_\lambda)/(n(1 - \lambda))$, with $n_\lambda$ the number of p-values in the batch greater than $\lambda$. If hypothesis $i$ is rejected, R[i] = 1 (otherwise R[i] = 0).

References

Storey, J.D. (2002). A direct approach to false discovery rates. Journal of the Royal Statistical Society (Series B), 64(3):479-498.

Storey, J.D., Taylor, J.E. and Siegmund, D. (2004). Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach. Journal of the Royal Statistical Society (Series B), 66(1):187-205.

Zrnic, T., Jiang, D., Ramdas, A. and Jordan, M.I. (2020). The Power of Batching in Multiple Hypothesis Testing. International Conference on Artificial Intelligence and Statistics, PMLR, 108:3806-3815.

Examples

sample.df <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757),
batch = c(rep(1,5), rep(2,6), rep(3,4)))

BatchStBH(sample.df)

---

Online FDR control based on a Bonferroni-like test

Description

This function is deprecated, please use Alpha_spending instead.

Usage

bonfInfinite(
  d,
  alpha = 0.05,
  alphai,
  random = TRUE,
  date.format = "%Y-%m-%d"
)

Arguments

d	Either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), date (‘date’) and p-value (‘pval’). If no column of dates is provided, then the p-values are treated as being ordered in sequence, arriving one at a time.
alpha	Overall significance level of the FDR procedure, the default is 0.05.
alphai	Optional vector of $\alpha_i$, where hypothesis $i$ is rejected if the $i$-th p-value is less than or equal to $\alpha_i$. A default is provided as proposed by Javanmard and Montanari (2018), equation 31.
random	Logical. If TRUE (the default), then the order of the p-values in each batch (i.e. those that have exactly the same date) is randomised.
date.format	Optional string giving the format that is used for dates.

Details

Implements online FDR control using a Bonferroni-like test.

The function takes as its input either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), date (‘date’) and p-value (‘pval’). The case where p-values arrive in batches corresponds to multiple instances of the same date. If no column of dates is provided, then the p-values are treated as being ordered in sequence, arriving one at a time.

The procedure controls FDR for a potentially infinite stream of p-values by using a Bonferroni-like test. Given an overall significance level $\alpha$, we choose a (potentially infinite) sequence of non-negative numbers $\alpha_i$ such that they sum to $\alpha$. Hypothesis $i$ is rejected if the $i$-th p-value is less than or equal to $\alpha_i$.

Value

d.out

A dataframe with the original data d (which will be reordered if there are batches and random = TRUE), the adjusted significance thresholds alphai and the indicator function of discoveries R, where R[i] = 1 corresponds to hypothesis $i$ being rejected (otherwise R[i] = 0).

References

Javanmard, A. and Montanari, A. (2018). Online Rules for Control of False Discovery Rate and False Discovery Exceedance. Annals of Statistics, 46(2):526-554.

---

LOND: Online FDR control based on number of discoveries

Description

Implements the LOND algorithm for online FDR control, where LOND stands for (significance) Levels based On Number of Discoveries, as presented by Javanmard and Montanari (2015).

Usage

LOND(
  d,
  alpha = 0.05,
  betai,
  dep = FALSE,
  random = TRUE,
  display_progress = FALSE,
  date.format = "%Y-%m-%d",
  original = TRUE
)

Arguments

d	Either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), date (‘date’) and p-value (‘pval’). If no column of dates is provided, then the p-values are treated as being ordered in sequence, arriving one at a time.
alpha	Overall significance level of the FDR procedure, the default is 0.05.
betai	Optional vector of $\beta_i$. A default is provided as proposed by Javanmard and Montanari (2018), equation 31.
dep	Logical. If TRUE, runs the modified LOND algorithm which guarantees FDR control for dependent p-values. Defaults to FALSE.
random	Logical. If TRUE (the default), then the order of the p-values in each batch (i.e. those that have exactly the same date) is randomised.
display_progress	Logical. If TRUE prints out a progress bar for the algorithm runtime.
date.format	Optional string giving the format that is used for dates.
original	Logical. If TRUE, runs the original LOND algorithm of Javanmard and Montanari (2015), otherwise runs the modified algorithm with $\alpha_i = \max(D(i-1), 1)\beta_i$ (Javanmard and Montanari, 2015, Remark 2.3), which also controls the FDR under positive dependence (Zrnic et al., 2021). Defaults to TRUE.

Details

The function takes as its input either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), date (‘date’) and p-value (‘pval’). The case where p-values arrive in batches corresponds to multiple instances of the same date. If no column of dates is provided, then the p-values are treated as being ordered in sequence, arriving one at a time.

The LOND algorithm controls the FDR for independent p-values (see below for the modification for dependent p-values). Given an overall significance level $\alpha$, we choose a sequence of non-negative numbers $\beta_i$ such that they sum to $\alpha$. The values of the adjusted significance thresholds $\alpha_i$ are chosen as follows:

$\alpha_i = (D(i-1) + 1)\beta_i$

where $D(n)$ denotes the number of discoveries in the first $n$ hypotheses.

A slightly modified version of LOND with thresholds $\alpha_i = \max(D(i-1), 1)\beta_i$ (Javanmard and Montanari, 2015, Remark 2.3) provably controls the FDR under positive dependence (PRDS condition), see Zrnic et al. (2021).

For arbitrarily dependent p-values, LOND controls the FDR if it is modified with $\beta_i / H(i)$ in place of $\beta_i$, where $H(i)$ is the i-th harmonic number.

Further details of the LOND algorithm can be found in Javanmard and Montanari (2015).

Value

out

A dataframe with the p-values pval (reordered if there are batches and random = TRUE), the LOND-adjusted significance thresholds $\alpha_i$, the indicator function of discoveries R and, if d is a dataframe with a column id, the identifiers id. Hypothesis $i$ is rejected if the $i$-th p-value is less than or equal to $\alpha_i$, in which case R[i] = 1 (otherwise R[i] = 0).

References

Javanmard, A. and Montanari, A. (2015). On Online Control of False Discovery Rate. arXiv preprint, https://arxiv.org/abs/1502.06197.

Javanmard, A. and Montanari, A. (2018). Online Rules for Control of False Discovery Rate and False Discovery Exceedance. Annals of Statistics, 46(2):526-554.

Zrnic, T., Ramdas, A. and Jordan, M.I. (2021). Asynchronous Online Testing of Multiple Hypotheses. Journal of Machine Learning Research, 22(33):1-39.

See Also

LONDstar presents versions of LOND for asynchronous testing, i.e. where each hypothesis test can itself be a sequential process and the tests can overlap in time.

Examples

sample.df <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
date = as.Date(c(rep('2014-12-01',3),
                rep('2015-09-21',5),
                rep('2016-05-19',2),
                '2016-11-12',
                rep('2017-03-27',4))),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757))

set.seed(1); LOND(sample.df)

LOND(sample.df, random=FALSE)

set.seed(1); LOND(sample.df, alpha=0.1)

---

LONDstar: Asynchronous online mFDR control based on number of discoveries

Description

Implements the LOND algorithm for asynchronous online testing, as presented by Zrnic et al. (2021).

Usage

LONDstar(
  d,
  alpha = 0.05,
  version,
  betai,
  batch.sizes,
  display_progress = FALSE
)

Arguments

d	Either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), p-value (‘pval’), and either ‘decision.times’, or ‘lags’, depending on which version you're using. See version for more details.
alpha	Overall significance level of the procedure, the default is 0.05.
version	Takes values 'async', 'dep' or 'batch'. This specifies the version of LONDstar to use. version='async' requires a column of decision times (‘decision.times’). version='dep' requires a column of lags (‘lags’). version='batch' requires a vector of batch sizes (‘batch.sizes’).
betai	Optional vector of $\beta_i$. A default is provided as proposed by Javanmard and Montanari (2018), equation 31.
batch.sizes	A vector of batch sizes, this is required for version='batch'.
display_progress	Logical. If TRUE prints out a progress bar for the algorithm runtime.

Details

The function takes as its input either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), p-value (‘pval’), and a column describing the conflict sets for the hypotheses. This takes the form of a vector of decision times or lags. Batch sizes can be specified as a separate argument (see below).

Zrnic et al. (2021) present three explicit versions of LONDstar:

1) version='async' is for an asynchronous testing process, consisting of tests that start and finish at (potentially) random times. The discretised finish times of the tests correspond to the decision times. These decision times are given as the input decision.times for this version of the LONDstar algorithm.

2) version='dep' is for online testing under local dependence of the p-values. More precisely, for any $t>0$ we allow the p-value $p_t$ to have arbitrary dependence on the previous $L_t$ p-values. The fixed sequence $L_t$ is referred to as ‘lags’, and is given as the input lags for this version of the LONDstar algorithm.

3) version='batch' is for controlling the mFDR in mini-batch testing, where a mini-batch represents a grouping of tests run asynchronously which result in dependent p-values. Once a mini-batch of tests is fully completed, a new one can start, testing hypotheses independent of the previous batch. The batch sizes are given as the input batch.sizes for this version of the LONDstar algorithm.

Given an overall significance level $\alpha$, LONDstar requires a sequence of non-negative non-increasing numbers $\beta_i$ that sum to $\alpha$.

Note that these LONDstar algorithms control the modified FDR (mFDR).

Further details of the LONDstar algorithms can be found in Zrnic et al. (2021).

Value

out	A dataframe with the original p-values pval, the adjusted testing levels $\alpha_i$ and the indicator function of discoveries R. Hypothesis $i$ is rejected if the $i$-th p-value is less than or equal to $\alpha_i$, in which case R[i] = 1 (otherwise R[i] = 0).

References

Javanmard, A. and Montanari, A. (2018). Online Rules for Control of False Discovery Rate and False Discovery Exceedance. Annals of Statistics, 46(2):526-554.

Zrnic, T., Ramdas, A. and Jordan, M.I. (2021). Asynchronous Online Testing of Multiple Hypotheses. Journal of Machine Learning Research, 22(33):1-39.

See Also

LOND presents versions of LOND for synchronous p-values, i.e. where each test can only start when the previous test has finished.

Examples

sample.df <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757),
decision.times = seq_len(15) + 1)

LONDstar(sample.df, version='async')

sample.df2 <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757),
lags = rep(1,15))

LONDstar(sample.df2, version='dep')

sample.df3 <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757))

LONDstar(sample.df3, version='batch', batch.sizes = c(4,6,5))

---

LORD: Online FDR control based on recent discovery

Description

Implements the LORD procedure for online FDR control, where LORD stands for (significance) Levels based On Recent Discovery, as presented by Javanmard and Montanari (2018) and Ramdas et al. (2017).

Usage

LORD(
  d,
  alpha = 0.05,
  gammai,
  version = "++",
  w0,
  b0,
  tau.discard = 0.5,
  random = TRUE,
  display_progress = FALSE,
  date.format = "%Y-%m-%d"
)

Arguments

d	Either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), date (‘date’) and p-value (‘pval’). If no column of dates is provided, then the p-values are treated as being ordered in sequence, arriving one at a time.
alpha	Overall significance level of the FDR procedure, the default is 0.05.
gammai	Optional vector of $\gamma_i$. A default is provided as proposed by Javanmard and Montanari (2018), equation 31 for all versions of LORD except 'dep'. The latter is provided a default to satisfy a condition given in Javanmard and Montanari (2018), example 3.7.
version	Takes values '++', 3, 'discard', or 'dep'. This specifies the version of LORD to use, and defaults to '++'.
w0	Initial ‘wealth’ of the procedure, defaults to $\alpha/10$ ($\min(\alpha/10, \tau\alpha/2)$ for version='discard'). Version 'discard' requires $w_0 \le \tau\alpha$.
b0	The ‘payout’ for rejecting a hypothesis in versions 3 and 'dep' of LORD. Defaults to $\alpha - w_0$.
tau.discard	Optional threshold for hypotheses to be selected for testing. Must be in (0, 1], defaults to 0.5. This is only used if version='discard'.
random	Logical. If TRUE (the default), then the order of the p-values in each batch (i.e. those that have exactly the same date) is randomised.
display_progress	Logical. If TRUE prints out a progress bar for the algorithm runtime.
date.format	Optional string giving the format that is used for dates.

Details

The function takes as its input either a vector of p-values or a dataframe with three columns: an identifier (‘id’), date (‘date’) and p-value (‘pval’). The case where p-values arrive in batches corresponds to multiple instances of the same date. If no column of dates is provided, then the p-values are treated as being ordered in sequence, arriving one at a time.

LORD++ and the discarding version of LORD provably control the FDR for independent p-values; for LORD 3 this has only been shown empirically (see below for dependent p-values). Given an overall significance level $\alpha$, we choose a sequence of non-negative non-increasing numbers $\gamma_i$ that sum to 1.

Javanmard and Montanari (2018) presented versions of LORD which differ in the way the adjusted significance thresholds $\alpha_i$ are calculated. The significance thresholds for LORD 2 are based on all previous discovery times. LORD 2 has been superseded by the algorithm given in Ramdas et al. (2017), LORD++ (version='++'), which is the default version. The significance thresholds for LORD 3 (version=3) are based on the time of the last discovery as well as the 'wealth' accumulated at that time. Finally, Tian and Ramdas (2019) presented a version of LORD (version='discard') that can improve the power of the procedure in the presence of conservative nulls by adaptively ‘discarding’ these p-values.

LORD depends on constants $w_0$ and (for versions 3 and 'dep') $b_0$, where $0 \le w_0 \le \alpha$ represents the initial ‘wealth’ of the procedure and $b_0 > 0$ represents the ‘payout’ for rejecting a hypothesis. We require $w_0+b_0 \le \alpha$ for FDR control to hold. Version 'discard' also depends on a constant $\tau$, where $\tau \in (0,1]$ represents the threshold for a hypothesis to be selected for testing: p-values greater than $\tau$ are implicitly ‘discarded’ by the procedure.

Note that FDR control also holds for the LORD procedure if only the p-values corresponding to true nulls are mutually independent, and independent from the non-null p-values.

For dependent p-values, a modified LORD procedure was proposed in Javanmard and Montanari (2018), which is called by setting version='dep'. Given an overall significance level $\alpha$, we choose a sequence of non-negative numbers $\xi_i$ (supplied through the argument gammai) such that $\sum_i \xi_i (1 + \log i) \le \alpha/b_0$ if $w_0 \le b_0$, which is the condition in Javanmard and Montanari (2018), example 3.7. If $w_0 > b_0$, the condition is instead $\sum_i (w_0 + b_0 \log i) \xi_i \le \alpha$, which follows from their Theorem 3.6.

Further details of the LORD procedures can be found in Javanmard and Montanari (2018), Ramdas et al. (2017) and Tian and Ramdas (2019).

Value

d.out

A dataframe with the p-values pval (reordered if there are batches and random = TRUE), the LORD-adjusted significance thresholds $\alpha_i$, the indicator function of discoveries R and, if d is a dataframe with a column id, the identifiers id. Hypothesis $i$ is rejected if the $i$-th p-value is less than or equal to $\alpha_i$, in which case R[i] = 1 (otherwise R[i] = 0).

References

Javanmard, A. and Montanari, A. (2018). Online Rules for Control of False Discovery Rate and False Discovery Exceedance. Annals of Statistics, 46(2):526-554.

Ramdas, A., Yang, F., Wainwright, M.J. and Jordan, M.I. (2017). Online control of the false discovery rate with decaying memory. Advances in Neural Information Processing Systems 30, 5650-5659.

Tian, J. and Ramdas, A. (2019). ADDIS: an adaptive discarding algorithm for online FDR control with conservative nulls. Advances in Neural Information Processing Systems 32, 9388-9396.

See Also

LORDstar presents versions of LORD for asynchronous testing, i.e. where each hypothesis test can itself be a sequential process and the tests can overlap in time.

Examples

sample.df <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
date = as.Date(c(rep('2014-12-01',3),
                rep('2015-09-21',5),
                rep('2016-05-19',2),
                '2016-11-12',
                rep('2017-03-27',4))),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757))

LORD(sample.df, random=FALSE)

set.seed(1); LORD(sample.df, version='dep')

set.seed(1); LORD(sample.df, version='discard')

set.seed(1); LORD(sample.df, alpha=0.1, w0=0.05)

---

LORD (dep): Online FDR control based on recent discovery for dependent p-values

Description

This function is deprecated, please use LORD instead with version = 'dep'.

Usage

LORDdep(
  d,
  alpha = 0.05,
  xi,
  w0 = alpha/10,
  b0 = alpha - w0,
  random = TRUE,
  date.format = "%Y-%m-%d"
)

Arguments

d	Either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), date (‘date’) and p-value (‘pval’). If no column of dates is provided, then the p-values are treated as being ordered in sequence, arriving one at a time.
alpha	Overall significance level of the FDR procedure, the default is 0.05.
xi	Optional vector of $\xi_i$. A default is provided to satisfy the condition given in Javanmard and Montanari (2018), example 3.7.
w0	Initial ‘wealth’ of the procedure. Defaults to $\alpha/10$.
b0	The ‘payout’ for rejecting a hypothesis. Defaults to $\alpha - w_0$.
random	Logical. If TRUE (the default), then the order of the p-values in each batch (i.e. those that have exactly the same date) is randomised.
date.format	Optional string giving the format that is used for dates.

Details

LORDdep implements the LORD procedure for online FDR control for dependent p-values, where LORD stands for (significance) Levels based On Recent Discovery, as presented by Javanmard and Montanari (2018).

The function takes as its input either a vector of p-values or a dataframe with three columns: an identifier (‘id’), date (‘date’) and p-value (‘pval’). The case where p-values arrive in batches corresponds to multiple instances of the same date. If no column of dates is provided, then the p-values are treated as being ordered in sequence, arriving one at a time.

This modified LORD procedure controls FDR for dependent p-values. Given an overall significance level $\alpha$, we choose a sequence of non-negative numbers $\xi_i$ such that they satisfy a condition given in Javanmard and Montanari (2018), example 3.7.

The procedure depends on constants $w_0$ and $b_0$, where $w_0 \ge 0$ represents the initial ‘wealth’ and $b_0 > 0$ represents the ‘payout’ for rejecting a hypothesis. We require $w_0+b_0 \le \alpha$ and $w_0 \le b_0$ for FDR control to hold.

Further details of the modified LORD procedure can be found in Javanmard and Montanari (2018).

Value

d.out

A dataframe with the original data d (which will be reordered if there are batches and random = TRUE), the LORD-adjusted significance thresholds $\alpha_i$ and the indicator function of discoveries R. Hypothesis $i$ is rejected if the $i$-th p-value is less than or equal to $\alpha_i$, in which case R[i] = 1 (otherwise R[i] = 0).

References

Javanmard, A. and Montanari, A. (2018). Online Rules for Control of False Discovery Rate and False Discovery Exceedance. Annals of Statistics, 46(2):526-554.

---

LORDstar: Asynchronous online mFDR control based on recent discovery

Description

Implements LORD algorithms for asynchronous online testing, as presented by Zrnic et al. (2021).

Usage

LORDstar(
  d,
  alpha = 0.05,
  version,
  gammai,
  w0,
  batch.sizes,
  display_progress = FALSE
)

Arguments

d	Either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), p-value (‘pval’), and either ‘decision.times’, or ‘lags’, depending on which version you're using. See version for more details.
alpha	Overall significance level of the procedure, the default is 0.05.
version	Takes values 'async', 'dep' or 'batch'. This specifies the version of LORDstar to use. version='async' requires a column of decision times (‘decision.times’). version='dep' requires a column of lags (‘lags’). version='batch' requires a vector of batch sizes (‘batch.sizes’).
gammai	Optional vector of $\gamma_i$. A default is provided as proposed by Javanmard and Montanari (2018), equation 31.
w0	Initial ‘wealth’ of the procedure, defaults to $\alpha/10$.
batch.sizes	A vector of batch sizes, this is required for version='batch'.
display_progress	Logical. If TRUE prints out a progress bar for the algorithm runtime.

Details

The function takes as its input either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), p-value (‘pval’), and a column describing the conflict sets for the hypotheses. This takes the form of a vector of decision times or lags. Batch sizes can be specified as a separate argument (see below).

Zrnic et al. (2021) present three explicit versions of LORDstar:

1) version='async' is for an asynchronous testing process, consisting of tests that start and finish at (potentially) random times. The discretised finish times of the tests correspond to the decision times. These decision times are given as the input decision.times for this version of the LORDstar algorithm.

2) version='dep' is for online testing under local dependence of the p-values. More precisely, for any $t>0$ we allow the p-value $p_t$ to have arbitrary dependence on the previous $L_t$ p-values. The fixed sequence $L_t$ is referred to as ‘lags’, and is given as the input lags for this version of the LORDstar algorithm.

3) version='batch' is for controlling the mFDR in mini-batch testing, where a mini-batch represents a grouping of tests run asynchronously which result in dependent p-values. Once a mini-batch of tests is fully completed, a new one can start, testing hypotheses independent of the previous batch. The batch sizes are given as the input batch.sizes for this version of the LORDstar algorithm.

Given an overall significance level $\alpha$, LORDstar depends on $w_0$ (where $0 \le w_0 \le \alpha$), which represents the initial ‘wealth’ of the procedure. The algorithms also require a sequence of non-negative non-increasing numbers $\gamma_i$ that sum to 1.

Note that these LORDstar algorithms control the modified FDR (mFDR). The ‘async’ version also controls the usual FDR if the p-values are assumed to be independent.

Further details of the LORDstar algorithms can be found in Zrnic et al. (2021).

Value

out	A dataframe with the original p-values pval, the adjusted testing levels $\alpha_i$ and the indicator function of discoveries R. Hypothesis $i$ is rejected if the $i$-th p-value is less than or equal to $\alpha_i$, in which case R[i] = 1 (otherwise R[i] = 0).

References

Javanmard, A. and Montanari, A. (2018). Online Rules for Control of False Discovery Rate and False Discovery Exceedance. Annals of Statistics, 46(2):526-554.

Zrnic, T., Ramdas, A. and Jordan, M.I. (2021). Asynchronous Online Testing of Multiple Hypotheses. Journal of Machine Learning Research, 22(33):1-39.

See Also

LORD presents versions of LORD for synchronous p-values, i.e. where each test can only start when the previous test has finished.

Examples

sample.df <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757),
decision.times = seq_len(15) + 1)

LORDstar(sample.df, version='async')

sample.df2 <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757),
lags = rep(1,15))

LORDstar(sample.df2, version='dep')

---

Online fallback procedure for FWER control

Description

Implements the online fallback procedure of Tian and Ramdas (2021), which guarantees strong FWER control under arbitrary dependence of the p-values.

Usage

online_fallback(
  d,
  alpha = 0.05,
  gammai,
  random = TRUE,
  display_progress = FALSE,
  date.format = "%Y-%m-%d"
)

Arguments

d	Either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), date (‘date’) and p-value (‘pval’). If no column of dates is provided, then the p-values are treated as being ordered in sequence, arriving one at a time.
alpha	Overall significance level of the FWER procedure, the default is 0.05.
gammai	Optional vector of $\gamma_i$. A default is provided as proposed by Javanmard and Montanari (2018), equation 31.
random	Logical. If TRUE (the default), then the order of the p-values in each batch (i.e. those that have exactly the same date) is randomised.
display_progress	Logical. If TRUE prints out a progress bar for the algorithm runtime.
date.format	Optional string giving the format that is used for dates.

Details

The function takes as its input either a vector of p-values or a dataframe with three columns: an identifier (‘id’), date (‘date’) and p-value (‘pval’). The case where p-values arrive in batches corresponds to multiple instances of the same date. If no column of dates is provided, then the p-values are treated as being ordered in sequence, arriving one at a time. Given an overall significance level $\alpha$, we choose a sequence of non-negative non-increasing numbers $\gamma_i$ that sum to 1.

The online fallback procedure provides a uniformly more powerful method than Alpha-spending, by saving the significance level of a previous rejection. More specifically, the procedure tests hypothesis $H_i$ at level

$\alpha_i = \alpha \gamma_i + R_{i-1} \alpha_{i-1}$

for $i \ge 2$, with $\alpha_1 = \alpha\gamma_1$, where $R_i = 1\{p_i \leq \alpha_i\}$ denotes a rejected hypothesis.

Further details of the online fallback procedure can be found in Tian and Ramdas (2021).

Value

out

A dataframe with the p-values pval (reordered if there are batches and random = TRUE), the adjusted significance thresholds $\alpha_i$, the indicator function of discoveries R and, if d is a dataframe with a column id, the identifiers id. Hypothesis $i$ is rejected if the $i$-th p-value is less than or equal to $\alpha_i$, in which case R[i] = 1 (otherwise R[i] = 0).

References

Javanmard, A. and Montanari, A. (2018). Online Rules for Control of False Discovery Rate and False Discovery Exceedance. Annals of Statistics, 46(2):526-554.

Tian, J. and Ramdas, A. (2021). Online control of the familywise error rate. Statistical Methods in Medical Research, 30(4):976-993.

Examples

sample.df <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
date = as.Date(c(rep('2014-12-01',3),
               rep('2015-09-21',5),
                rep('2016-05-19',2),
                '2016-11-12',
               rep('2017-03-27',4))),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757))

online_fallback(sample.df, random=FALSE)

set.seed(1); online_fallback(sample.df)

set.seed(1); online_fallback(sample.df, alpha=0.1)

---

Deprecated functions in package ‘onlineFDR’

Description

These functions are provided for compatibility with older versions of ‘onlineFDR’ only, and will be defunct at the next release.

Details

The following functions are deprecated and will be made defunct; use the replacement indicated below:

• LORDdep: LORD with version='dep'

• bonfInfinite: Alpha_spending

Value

A dataframe with the p-values, the adjusted significance thresholds alphai and the indicator function of discoveries R, as for the replacement functions.

---

SAFFRON: Adaptive online FDR control

Description

Implements the SAFFRON procedure for online FDR control, where SAFFRON stands for Serial estimate of the Alpha Fraction that is Futilely Rationed On true Null hypotheses, as presented by Ramdas et al. (2018). The algorithm is based on an estimate of the proportion of true null hypotheses. More precisely, SAFFRON sets the adjusted test levels based on an estimate of the amount of alpha-wealth that is allocated to testing the true null hypotheses.

Usage

SAFFRON(
  d,
  alpha = 0.05,
  gammai,
  w0,
  lambda = 0.5,
  random = TRUE,
  display_progress = FALSE,
  date.format = "%Y-%m-%d"
)

Arguments

d	Either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), date (‘date’) and p-value (‘pval’). If no column of dates is provided, then the p-values are treated as being ordered in sequence, arriving one at a time.
alpha	Overall significance level of the FDR procedure, the default is 0.05.
gammai	Optional vector of $\gamma_i$. A default is provided with $\gamma_j$ proportional to $1/j^{1.6}$.
w0	Initial ‘wealth’ of the procedure, defaults to $\alpha/2$. Must be between 0 and $\alpha$.
lambda	Optional threshold for a ‘candidate’ hypothesis, must be between 0 and 1. Defaults to 0.5.
random	Logical. If TRUE (the default), then the order of the p-values in each batch (i.e. those that have exactly the same date) is randomised.
display_progress	Logical. If TRUE prints out a progress bar for the algorithm runtime.
date.format	Optional string giving the format that is used for dates.

Details

The function takes as its input either a vector of p-values or a dataframe with three columns: an identifier (‘id’), date (‘date’) and p-value (‘pval’). The case where p-values arrive in batches corresponds to multiple instances of the same date. If no column of dates is provided, then the p-values are treated as being ordered in sequence, arriving one at a time.

The SAFFRON procedure provably controls the FDR for independent p-values. Given an overall significance level $\alpha$, we choose a sequence of non-negative non-increasing numbers $\gamma_i$ that sum to 1.

SAFFRON depends on constants $w_0$ and $\lambda$, where $w_0$ satisfies $0 \le w_0 \le \alpha$ and represents the initial ‘wealth’ of the procedure, and $0 < \lambda < 1$ represents the threshold for a ‘candidate’ hypothesis. A ‘candidate’ refers to p-values no larger than $\lambda$, since SAFFRON will never reject a p-value larger than $\lambda$.

Note that FDR control also holds for the SAFFRON procedure if only the p-values corresponding to true nulls are mutually independent, and independent from the non-null p-values.

The SAFFRON procedure can lose power in the presence of conservative nulls, which can be compensated for by adaptively ‘discarding’ these p-values, as in the ADDIS algorithm (see ADDIS).

Further details of the SAFFRON procedure can be found in Ramdas et al. (2018).

Value

out

A dataframe with the p-values pval (reordered if there are batches and random = TRUE), the adjusted significance thresholds $\alpha_i$, the indicator function of discoveries R and, if d is a dataframe with a column id, the identifiers id. Hypothesis $i$ is rejected if the $i$-th p-value is less than or equal to $\alpha_i$, in which case R[i] = 1 (otherwise R[i] = 0).

References

Ramdas, A., Zrnic, T., Wainwright, M.J. and Jordan, M.I. (2018). SAFFRON: an adaptive algorithm for online control of the false discovery rate. Proceedings of the 35th International Conference on Machine Learning, 80:4286-4294.

See Also

SAFFRONstar presents versions of SAFFRON for asynchronous testing, i.e. where each hypothesis test can itself be a sequential process and the tests can overlap in time.

Examples

sample.df <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
date = as.Date(c(rep('2014-12-01',3),
               rep('2015-09-21',5),
                rep('2016-05-19',2),
                '2016-11-12',
               rep('2017-03-27',4))),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757))

SAFFRON(sample.df, random=FALSE)

set.seed(1); SAFFRON(sample.df)

set.seed(1); SAFFRON(sample.df, alpha=0.1, w0=0.025)

---

SAFFRONstar: Adaptive online mFDR control for asynchronous testing

Description

Implements the SAFFRON algorithm for asynchronous online testing, as presented by Zrnic et al. (2021).

Usage

SAFFRONstar(
  d,
  alpha = 0.05,
  version,
  gammai,
  w0,
  lambda = 0.5,
  batch.sizes,
  display_progress = FALSE
)

Arguments

d	Either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), p-value (‘pval’), and either ‘decision.times’, or ‘lags’, depending on which version you're using. See version for more details.
alpha	Overall significance level of the procedure, the default is 0.05.
version	Takes values 'async', 'dep' or 'batch'. This specifies the version of SAFFRONstar to use. version='async' requires a column of decision times (‘decision.times’). version='dep' requires a column of lags (‘lags’). version='batch' requires a vector of batch sizes (‘batch.sizes’).
gammai	Optional vector of $\gamma_i$. A default is provided with $\gamma_j$ proportional to $1/j^{1.6}$.
w0	Initial ‘wealth’ of the procedure, defaults to $\alpha/2$.
lambda	Optional threshold for a ‘candidate’ hypothesis, must be between 0 and 1. Defaults to 0.5.
batch.sizes	A vector of batch sizes, this is required for version='batch'.
display_progress	Logical. If TRUE prints out a progress bar for the algorithm runtime.

Details

The function takes as its input either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), p-value (‘pval’), and a column describing the conflict sets for the hypotheses. This takes the form of a vector of decision times or lags. Batch sizes can be specified as a separate argument (see below).

Zrnic et al. (2021) present three explicit versions of SAFFRONstar:

1) version='async' is for an asynchronous testing process, consisting of tests that start and finish at (potentially) random times. The discretised finish times of the tests correspond to the decision times. These decision times are given as the input decision.times for this version of the SAFFRONstar algorithm. For this version of SAFFRONstar, Tian and Ramdas (2019) presented an algorithm that can improve the power of the procedure in the presence of conservative nulls by adaptively ‘discarding’ these p-values. This is implemented by ADDIS with async=TRUE.

2) version='dep' is for online testing under local dependence of the p-values. More precisely, for any $t>0$ we allow the p-value $p_t$ to have arbitrary dependence on the previous $L_t$ p-values. The fixed sequence $L_t$ is referred to as ‘lags’, and is given as the input lags for this version of the SAFFRONstar algorithm.

3) version='batch' is for controlling the mFDR in mini-batch testing, where a mini-batch represents a grouping of tests run asynchronously which result in dependent p-values. Once a mini-batch of tests is fully completed, a new one can start, testing hypotheses independent of the previous batch. The batch sizes are given as the input batch.sizes for this version of the SAFFRONstar algorithm.

Given an overall significance level $\alpha$, SAFFRONstar depends on constants $w_0$ and $\lambda$, where $w_0$ satisfies $0 \le w_0 \le \alpha$ and represents the initial ‘wealth’ of the procedure, and $0 < \lambda < 1$ represents the threshold for a ‘candidate’ hypothesis. A ‘candidate’ refers to p-values no larger than $\lambda$, since SAFFRONstar will never reject a p-value larger than $\lambda$. The algorithms also require a sequence of non-negative non-increasing numbers $\gamma_i$ that sum to 1.

Note that these SAFFRONstar algorithms control the modified FDR (mFDR). The ‘async’ version also controls the usual FDR if the p-values are assumed to be independent.

Further details of the SAFFRONstar algorithms can be found in Zrnic et al. (2021).

Value

out	A dataframe with the original p-values pval, the adjusted testing levels $\alpha_i$ and the indicator function of discoveries R. Hypothesis $i$ is rejected if the $i$-th p-value is less than or equal to $\alpha_i$, in which case R[i] = 1 (otherwise R[i] = 0).

References

Tian, J. and Ramdas, A. (2019). ADDIS: an adaptive discarding algorithm for online FDR control with conservative nulls. Advances in Neural Information Processing Systems 32, 9388-9396.

Zrnic, T., Ramdas, A. and Jordan, M.I. (2021). Asynchronous Online Testing of Multiple Hypotheses. Journal of Machine Learning Research, 22(33):1-39.

See Also

SAFFRON presents versions of SAFFRON for synchronous p-values, i.e. where each test can only start when the previous test has finished.

Examples

sample.df <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757),
decision.times = seq_len(15) + 1)

SAFFRONstar(sample.df, version='async')

sample.df2 <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
    'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
    'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
pval = c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757),
lags = rep(1,15))

SAFFRONstar(sample.df2, version='dep')

---

setBound

Description

Calculates a default sequence of non-negative numbers $\gamma_i$ that sum to 1 (for LOND and LONDstar, a sequence $\beta_i$ that sums to $\alpha$), given an upper bound $N$ on the number of hypotheses to be tested. For LORDdep the sequence $\xi_i$ instead satisfies $\sum_{i=1}^N \xi_i (1 + \log i) = 1$, the condition in Javanmard and Montanari (2018), example 3.7, for LORD(version='dep').

Usage

setBound(alg, alpha = 0.05, N)

Arguments

alg	A string that takes the value of one of the following: LOND, LORD, LORDdep, SAFFRON, ADDIS, LONDstar, LORDstar, SAFFRONstar, Alpha_investing, Alpha_spending, ADDIS_spending or online_fallback
alpha	Overall significance level of the FDR procedure, the default is 0.05. The bounds for LOND and LONDstar depend on alpha.
N	An upper bound on the number of hypotheses to be tested

Value

bound

A vector giving the values of a default sequence $\gamma_i$ of non-negative numbers.

---

StoreyBH: Offline FDR control using the St-BH procedure

Description

Implements the Storey-BH algorithm for offline FDR control, as presented by Storey (2002), with the null proportion estimate of Storey, Taylor and Siegmund (2004).

Usage

StoreyBH(d, alpha = 0.05, lambda = 0.5)

Arguments

d	Either a vector of p-values, or a dataframe with the column: p-value (‘pval’).
alpha	Overall significance level of the FDR procedure, the default is 0.05.
lambda	Threshold for Storey-BH, must be in (0, 1). Defaults to 0.5.

Details

The function takes as its input either a vector of p-values, or a dataframe with a column of p-values (‘pval’).

Value

out

A dataframe with the original data d and the indicator function of discoveries R. Hypothesis $i$ is rejected if the $i$-th p-value is less than or equal to $(k/n)\alpha / \hat{\pi}_0$, where $k$ is the largest $r$ such that the $r$-th smallest p-value is at most $(r/n)\alpha / \hat{\pi}_0$, $n$ is the total number of hypotheses and $\hat{\pi}_0 = (1 + n_\lambda) / (n(1 - \lambda))$ is Storey's estimate of the proportion of true null hypotheses, with $n_\lambda$ the number of p-values greater than $\lambda$. If hypothesis $i$ is rejected, R[i] = 1 (otherwise R[i] = 0).

References

Storey, J.D. (2002). A direct approach to false discovery rates. Journal of the Royal Statistical Society (Series B), 64(3):479-498.

Storey, J.D., Taylor, J.E. and Siegmund, D. (2004). Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach. Journal of the Royal Statistical Society (Series B), 66(1):187-205.

Examples

pvals <- c(2.90e-08, 0.06743, 0.01514, 0.08174, 0.00171,
        3.60e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
        0.69274, 0.30443, 0.00136, 0.72342, 0.54757)

StoreyBH(pvals)

---

supLORD: Online control of the false discovery exceedance (FDX) and the FDR at stopping times

Description

Implements the supLORD procedure, which controls both FDX and FDR, including the FDR at stopping times, as presented by Xu and Ramdas (2021).

Usage

supLORD(
  d,
  delta = 0.05,
  eps,
  r,
  eta,
  rho,
  gammai,
  random = TRUE,
  display_progress = FALSE,
  date.format = "%Y-%m-%d"
)

Arguments

d	Either a vector of p-values, or a dataframe with three columns: an identifier (‘id’), date (‘date’) and p-value (‘pval’). If no column of dates is provided, then the p-values are treated as being ordered in sequence, arriving one at a time.
delta	Upper bound on the probability that the FDP exceeds eps (at any time step after making r rejections). Must be between 0 and 1, defaults to 0.05.
eps	The upper bound on the FDP. Must be between 0 and 1.
r	The threshold of rejections after which the error control begins to apply. Must be a positive integer.
eta	Controls the pace at which wealth is spent as a function of the algorithm's current wealth. Must be a positive real number.
rho	Controls the length of time before the spending sequence exhausts the wealth earned from a rejection. Must be a positive integer, no greater than the length of a user-supplied gammai.
gammai	Optional vector of $\gamma_i$. A default is provided as proposed by Javanmard and Montanari (2018), equation 31.
random	Logical. If TRUE (the default), then the order of the p-values in each batch (i.e. those that have exactly the same date) is randomised.
display_progress	Logical. If TRUE prints out a progress bar for the algorithm runtime.
date.format	Optional string giving the format that is used for dates.

Details

The function takes as its input either a vector of p-values or a dataframe with three columns: an identifier (‘id’), date (‘date’) and p-value (‘pval’). The case where p-values arrive in batches corresponds to multiple instances of the same date. If no column of dates is provided, then the p-values are treated as being ordered in sequence, arriving one at a time.

The supLORD procedure provably controls the FDX for p-values that are conditionally superuniform under the null. supLORD also controls the supFDR and hence the FDR (even at stopping times). Given an overall significance level $\alpha$, we choose a sequence of non-negative non-increasing numbers $\gamma_i$ that sum to 1.

supLORD requires the user to specify r, a threshold of rejections after which the error control begins to apply, eps, the upper bound on the false discovery proportion (FDP), and delta, an upper bound on the probability that the FDP exceeds eps at any time step after making r rejections. As well, the user should specify the variables eta, which controls the pace at which wealth is spent (as a function of the algorithm's current wealth), and rho, which controls the length of time before the spending sequence exhausts the wealth earned from a rejection.

The test levels follow the dynamic alpha schedule of Xu and Ramdas (2021), equations 16 and 17, with the default boost sequence of their Appendix A, equation 21: the initial wealth and each of the first r - 1 rejections earn $\beta_0 = b - a/r$, and the r-th and later rejections earn $b = \epsilon a \log(1 + L/a)/L$, where $L = \log(1/\delta)$. The offset $a$ is the canonical solution of their equation 20.

Further details of the supLORD procedure can be found in Xu and Ramdas (2021).

Value

d.out

A dataframe with the original data d (which will be reordered if there are batches and random = TRUE), the supLORD-adjusted significance thresholds $\alpha_i$ and the indicator function of discoveries R. Hypothesis $i$ is rejected if the $i$-th p-value is less than or equal to $\alpha_i$, in which case R[i] = 1 (otherwise R[i] = 0).

References

Javanmard, A. and Montanari, A. (2018). Online Rules for Control of False Discovery Rate and False Discovery Exceedance. Annals of Statistics, 46(2):526-554.

Xu, Z. and Ramdas, A. (2021). Dynamic Algorithms for Online Multiple Testing. Annual Conference on Mathematical and Scientific Machine Learning, PMLR, 145:955-986.

Examples

set.seed(1)
N <- 1000
B <- rbinom(N, 1, 0.5)
Z <- rnorm(N, mean = 3*B)
pval <- pnorm(-Z)

out <- supLORD(pval, eps=0.15, r=30, eta=0.05, rho=30, random=FALSE)
head(out)
sum(out$R)

---