---
title: "Using crumblr in practice"
subtitle: ''
author: "Developed by [Gabriel Hoffman](http://gabrielhoffman.github.io/)"
date: "Run on `r Sys.time()`"
documentclass: article
vignette: >
  %\VignetteIndexEntry{crumblr}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
  %\usepackage[utf8]{inputenc}
output:
  BiocStyle::html_document:
    toc: true
    toc_float: true
---

```{r setup, echo=FALSE, results="hide"}
knitr::opts_chunk$set(
  tidy = FALSE,
  cache = TRUE,
  echo = TRUE,
  dev = c("png", "pdf"),
  package.startup.message = FALSE,
  message = FALSE,
  error = FALSE,
  warning = FALSE
)

options(width = 100)
```	



<style>
body {
text-align: justify}
</style>

## Introduction
Changes in cell type composition play an important role in health and disease.  Recent advances in single cell technology have enabled measurement of cell type composition at increasing cell lineage resolution across large cohorts of individuals.  Yet this raises new challenges for statistical analysis of these compositional data to identify changes associated with a phenotype.  We introduce `crumblr`, a scalable statistical method for analyzing count ratio data using precision-weighted linear models incorporating random effects for complex study designs.  Uniquely, `crumblr` performs tests of association at multiple levels of the cell lineage hierarchy using multivariate regression to increase power over tests of a single cell component.  In simulations, `crumblr` increases power compared to existing methods, while controlling the false positive rate.

The `crumblr` package integrates with the [`variancePartition`](https://www.bioconductor.org/packages/variancePartition/) and [`dreamlet`](https://www.bioconductor.org/packages/dreamlet/) packages in the Bioconductor ecosystem.

Here we consider counts for 8 cell types from quantified using single cell RNA-seq data from unstimulated and interferon β stimulated PBMCs from 8 subjects [(Kang, et al., 2018)](https://www.nature.com/articles/nbt.4042).    

The functions here incorporate the precision weights:

- `variancePartition::fitExtractVarPartModel()`
- `variancePartition::dream()`
- `limma::lmFit()`


# Installation
To install this package, start R and enter:

```{r install, eval=FALSE}
# 1) Make sure Bioconductor is installed
if (!require("BiocManager", quietly = TRUE)) {
  install.packages("BiocManager")
}

# 2) Install crumblr and dependencies:
# From Bioconductor
BiocManager::install("crumblr")
```


# Analysis workflow
## Process data

Here we evaluate whether the observed cell proportions change in response to interferon β.  Given the results here, we cannot reject the null hypothesis that interferon β does not affect the cell type proportions.
 
```{r crumblr}
library(crumblr)

# Load cell counts, clustering and metadata
# from Kang, et al. (2018) https://doi.org/10.1038/nbt.4042
data(IFNCellCounts)

# Apply crumblr transformation
# cobj is an EList object compatable with limma workflow
# cobj$E stores transformed values
# cobj$weights stores precision weights
#    corresponding to the regularized inverse variance
cobj <- crumblr(df_cellCounts)
```

## Variance partitioning
Decomposing the variance illustrates that more variation is explained by subject than stimulation status.
```{r vp, fig.height=4, fig.width=6}
library(variancePartition)

# Partition variance into components for Subject (i.e. ind)
#   and stimulation status, and residual variation
form <- ~ (1 | ind) + (1 | StimStatus)
vp <- fitExtractVarPartModel(cobj, form, info)

# Plot variance fractions
fig.vp <- plotPercentBars(vp)
fig.vp
```


## PCA
Performing PCA on the transformed cell counts indicates that the samples cluster based on subject rather than stimulation status.  Here, we use `standardize()` so that each observation has approximately equal variance (i.e. homoscedasticity) by dividing the CLR transformed frequencies by their estimated sampling standard deviation.  Transforming the data to be approximately homoscedastic has been [shown](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-014-0550-8) to improve performance of PCA.

```{r pca, fig.height=5, fig.width=5}
library(ggplot2)

# Perform PCA
# use crumblr::standardize() to get values with
# approximately equal sampling variance,
# which is a key property for downstream PCA and clustering analysis.
pca <- prcomp(t(standardize(cobj)))

# merge with metadata
df_pca <- merge(pca$x, info, by = "row.names")

# Plot PCA
#   color by Subject
# 	shape by Stimulated vs unstimulated
ggplot(df_pca, aes(PC1, PC2, color = as.character(ind), shape = StimStatus)) +
  geom_point(size = 3) +
  theme_classic() +
  theme(aspect.ratio = 1) +
  scale_color_discrete(name = "Subject") +
  xlab("PC1") +
  ylab("PC2")
```

## Hierarchical clustering
The samples from the same subject also cluster together.
```{r hclust}
heatmap(cobj$E)
```


## Differential testing
```{r dream}
# Use variancePartition workflow to analyze each cell type
# Perform regression on each cell type separately
#  then use eBayes to shrink residual variance
# Also compatible with limma::lmFit()
fit <- dream(cobj, ~ StimStatus + ind, info)
fit <- eBayes(fit)

# Extract results for each cell type
topTable(fit, coef = "StimStatusstim", number = Inf)
```


### Multivariate testing along a tree
We can gain power by jointly testing multiple cell types using a multivariate statistical model, instead of testing one cell type at a time.  Here we construct a hierarchical clustering between cell types based on gene expression from pseudobulk, and perform a multivariate test for each internal node of the tree based on its leaf nodes.  The results for the leaves are the same as from `topTable()` above. At each internal node `treeTest()` performs a fixed effects meta-analysis of the coefficients of the leaves while modeling the covariance between coefficient estimates.  In the backend, this is implemented using `variancePartition::mvTest()` and [remaCor](https://cran.r-project.org/package=remaCor) package.

Here the hierarchical clustering, `hcl`, is precomputed from pseudobulk gene expression using `buildClusterTreeFromPB()`.

```{r treeTest}
# Perform multivariate test across the hierarchy
res <- treeTest(fit, cobj, hcl, coef = "StimStatusstim")

# Plot hierarchy and testing results
plotTreeTest(res)

# Plot hierarchy and regression coefficients
plotTreeTestBeta(res)
```     

#### Combined plotting
```{r combined, fig.width=12}
plotTreeTestBeta(res) +
  theme(legend.position = "bottom", legend.box = "vertical") |
  plotForest(res, hide = FALSE) |
  fig.vp
```

<br>

## Hierarchical clustering
The hierarchical clustering used by `treeTest()` can be computed a number of ways, depending on the available data and biological question.  For example, see [Article](integration.html) for details about how hierarchical clustering was run in this dataset.

In general, hierarchical clustering can be computed from 

 - pseudobulked single cell  gene expression 

```{r buildClusterTreeFromPB, eval=FALSE}
hcl <- buildClusterTreeFromPB(pb)
```

 - cell type frequencies

```{r dm, eval=FALSE}
# correlation matrix between all cell types
C <- cor(t(standardize(cobj)))

# convert to distance
dm <- as.dist(1 - abs(C))

# eval hierarchical clustering
hcl <- hclust(dm)
```

 
 - [Newick formated](https://en.wikipedia.org/wiki/Newick_format) tree computed from external data

```{r extern, eval=FALSE}
# Make sure packages are installed
# BiocManager::install(c("ctc", "ape", "phylogram"))
library(ape)
library(ctc)
library(phylogram)
library(tidyverse)

# Write tree to file, 
# edit manually
# then read back into R
# 

# Specify tree as text in Newick format
txt = "((CD14+ Monocytes,(B cells,(Dendritic cells,Megakaryocytes))),(CD8 T cells,(NK cells,(CD4 T cells,FCGR3A+ Monocytes))));"

# read from text
hcl_from_txt <- read.tree(text = txt) %>%
                  as.dendrogram.phylo %>%
                  as.hclust

# Alternatively, 
# write existing tree to file
# and edit manaully
write(hc2Newick(hcl),file='hcl.nwk')

hcl_from_txt2 <- read.tree(file = 'hcl.nwk') %>%
                  as.dendrogram.phylo %>%
                  as.hclust
```

# Considerations


## Computational scaling
The `crumblr()` workflow is very fast, especially compared to more demanding differential expression analysies. Applying `crumblr()` requires <1 sec, even for very large datsets.  Differential testing with `dream()` takes < 10 seconds for fixed effects models and < 30 seconds for mixed models with typical sample sizes and number of cell types.  Running `treeTest()` can be a little more demanding, but should finish in < 30 seconds with 20 cell types.

## Complex study designs
The `crumblr()` workflow can handle complex study designs with repeated measures or multiple random effects.  `dream()` uses `lme4::lmer()` in the backend to fit weighted linear mixed models.  Considerations about defining the regression model are described in documentation to [`variancePartition`](https://diseaseneurogenomics.github.io/variancePartition) or this [book](https://people.math.ethz.ch/~maechler/MEMo-pages/lMMwR.pdf) by the author of `lme4`.


## Tuning parameters
`crumblr()` uses two tuning parameters accessable to the user.  These are fixed to default values in all simulations and data analysis.  <u>The user is strongly recommended to keep these dfault values</u>.

- In order to deal with zero counts, `crumblr()` uses a pseudocount (default: 0.5) added to all observed counts. 

- For real data, the asymptotic variance formula can give weights that vary substantially across samples and give very high weights for a subset of samples.  In order to address this, we regularize the weights to reduce the variation in the weights to have a maximum ratio (default of 5) between the maximum and specified quantile value (default of 5%).








# Session Info
<details>
```{r session, echo=FALSE}
sessionInfo()
```
</details>