--- title: "Input formats: reports and SummarizedExperiment objects" author: "Sergio Ciordia" package: NADIA output: BiocStyle::html_document: toc: true toc_float: true number_sections: true vignette: > %\VignetteIndexEntry{Input formats: Spectronaut, DIA-NN, TMT and LFQ} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include = FALSE} knitr::opts_chunk$set(collapse = TRUE, comment = "#>", crop = NULL) ``` # Introduction NADIA accepts five types of input: Spectronaut reports, DIA-NN protein-group matrices, Proteome Discoverer TMT and LFQ reports, and `SummarizedExperiment` objects prepared with other tools. Each has a dedicated importer: `preprocess_spectronaut()`, `preprocess_diann()`, `preprocess_tmt()`, `preprocess_lfq()` or `preprocess_se()`. All return the same `proteomics_data` structure, so the imported data can follow a common processing and analysis workflow. This vignette applies each preprocessor to an example report and explains the main distinction among them: **where NADIA obtains the experimental design**. Correctly linking every measured column or run to its condition and replicate is essential, because that mapping determines the groups compared later in the analysis. For example, `preprocess_spectronaut()` converts a long-format report—one row for each reported combination of LC-MS run and protein group—into an object of class `proteomics_data`. The `proteomics_data` object separates the information contained in the source report into three coordinated tables: - `metadata` contains one row per sample or LC-MS run, including the sample identifier, experimental condition, replicate and any available run-level information. - `protein_id` contains identification-level information for each protein group, including protein and gene annotations and, when available, run-specific evidence such as peptide and precursor counts, sequence coverage and identification scores. - `protein_quant` contains the protein-level quantification data. It includes the abundance measured for each protein group in each sample, together with the protein annotations and quantitative evidence required by the downstream workflow. ::: {.lightbox-figure} ```{r input-format-overview, echo = FALSE, fig.align = "center", out.width = "100%", fig.cap = "Five input sources converge on the common `proteomics_data` structure used by the NADIA workflow.", fig.alt = "Workflow diagram showing four report formats and a SummarizedExperiment entering five dedicated importers. Spectronaut supplies the design in its report; DIA-NN, TMT and LFQ use column suffixes or annot_path; SummarizedExperiment uses colData. All five produce a proteomics_data object containing metadata, protein_id and protein_quant for process_proteomics."} knitr::include_graphics("figures/Input_Formats_Workflow.svg") ``` ::: In the Spectronaut example used below, `metadata` contains 12 samples, whereas `protein_id` and `protein_quant` each contain 2,000 protein groups. The three tables have distinct but complementary roles. `metadata` defines the experimental design; `protein_quant` provides the abundance matrix used for normalization, imputation and differential abundance analysis; and `protein_id` retains the more detailed identification evidence used for inspection and reporting. # Installation ```{r install, eval = FALSE} if (!requireNamespace("BiocManager", quietly = TRUE)) install.packages("BiocManager") BiocManager::install("NADIA") ``` ```{r library, message = FALSE} library(NADIA) ``` # DIA: Spectronaut ::: {.spectronaut-schema-note}

Spectronaut export schema included with NADIA

NADIA provides a ready-to-use schema for generating a compatible Spectronaut report. Locate the installed file from R with:

system.file("extdata", "NADIA_Report.rs", package = "NADIA")

`NADIA_Report.rs` is a binary file interpreted by Spectronaut, not a text file that lists the exported columns. Import it directly into Spectronaut and select it when generating the report. The resulting table will contain the structure and columns required by `preprocess_spectronaut()`. ::: Spectronaut exports a **long** report: one row per protein group *per run*, with the condition and replicate written in every row. Among the four input formats supported by NADIA, Spectronaut is the only one for which the experimental design does not need to be supplied separately. `preprocess_spectronaut()` extracts it directly from the `R.Condition` and `R.Replicate` columns in the report. ::: {.lightbox-figure} ```{r spectronaut-input-structure, echo = FALSE, fig.align = "center", out.width = "100%", fig.cap = "Structure of the long-format Spectronaut report accepted by `preprocess_spectronaut()`.", fig.alt = "Diagram of a long-format Spectronaut report. Each row contains one LC-MS run and one protein group. R.FileName, R.Condition and R.Replicate identify the sample and experimental design; PG.ProteinGroups identifies the protein; PG.Quantity contains its abundance; and further columns contain run-level annotations."} knitr::include_graphics("figures/Spectronaut_Input_Structure.svg") ``` ::: The arguments accepted by `preprocess_spectronaut()` are summarized below. The input report and the order of the conditions are required; the remaining arguments control optional export and the aggregation of run-level annotations.

Show the complete preprocess_spectronaut() call and default arguments

```{r spectronaut-defaults, eval = FALSE} dia <- preprocess_spectronaut( # Required input and experimental conditions file_path = "spectronaut_report.tsv", condition_order = c("Control", "Treatment"), # Optional export; NULL keeps the result in memory export_dir = NULL, # Aggregation of annotations reported for multiple runs agg_coverage_run = "max", agg_coverage_global = "max", agg_mw = "max", agg_cscore_runwise = "mean", # Exported file names and progress messages timestamp_suffix = TRUE, verbose = TRUE ) ```

```{r dia} dia_file <- system.file("extdata", "nadia_dia_report.tsv.gz", package = "NADIA") dia <- preprocess_spectronaut(dia_file, condition_order = c("A", "B", "D"), verbose = FALSE) dia ``` Because the design already comes from the report, `condition_order` does not infer or redefine the condition assigned to each run. Instead, it selects the conditions to retain—excluding any other conditions present in the report—and sets their order in the subsequent stages of the analysis, including downstream comparisons. It should therefore list every condition that is to be analyzed. The following output shows an example of the metadata table extracted and constructed from the imported Spectronaut report: ```{r dia-md} dia$metadata ``` Because the report is long and run-level, the same protein group may have multiple values for annotations such as sequence coverage, molecular weight and C-score. The `agg_*` arguments determine how those values are reduced when a single annotation value is required: ```{r dia-agg, eval = FALSE} dia_median <- preprocess_spectronaut( dia_file, condition_order = c("A", "B", "D"), agg_coverage_run = "median", # default "max" agg_cscore_runwise = "median", # default "mean" verbose = FALSE) ``` These arguments affect only the annotations stored alongside the quantitative data; they do not modify protein intensities. # DIA: DIA-NN A DIA-NN protein-group matrix (`report.pg_matrix.tsv`) is **wide**: it contains one row per protein group and one intensity column per run. This differs from the long Spectronaut report and resembles the wide Proteome Discoverer exports described below, which is why it requires a dedicated importer. ::: {.lightbox-figure} ```{r diann-input-structure, echo = FALSE, fig.align = "center", out.width = "100%", fig.cap = "Structure of the wide DIA-NN protein-group matrix accepted by `preprocess_diann()`.", fig.alt = "Diagram of a wide DIA-NN protein-group matrix. Each row represents one protein group. Protein annotations occupy the first columns, followed by one Abundance column for each run. The experimental design is obtained from Condition_Replicate suffixes or from annot_path."} knitr::include_graphics("figures/DIANN_Input_Structure.svg") ``` ::: The arguments accepted by `preprocess_diann()` are shown below. The report and the condition order are required. `annot_path` is optional and is used when the run columns do not follow the `Abundance: _` convention.

Show the complete preprocess_diann() call and default arguments

```{r diann-defaults, eval = FALSE} diann <- preprocess_diann( # Required input and conditions to retain file_path = "report.pg_matrix.tsv", condition_order = c("Control", "Treatment"), # Optional sample annotation file; NULL uses the column suffixes annot_path = NULL, # Optional export; NULL keeps the result in memory export_dir = NULL, timestamp_suffix = TRUE, # Progress messages verbose = TRUE ) ```

```{r diann} diann_file <- system.file("extdata", "nadia_diann_report.tsv.gz", package = "NADIA") diann <- preprocess_diann(diann_file, condition_order = c("A", "B", "D"), verbose = FALSE) diann ``` ## Declaring the experimental design DIA-NN names each intensity column after the raw file it came from: ``` D:\DIA-NN\Projects\20241014_XII_CursoProtQ_DIA_500ng_sample_A1.raw ``` The text `sample_A1` may suggest condition A and replicate 1 to a reader, but that interpretation depends on a laboratory-specific naming convention. A regular expression that worked for this report could silently misclassify the next one. NADIA therefore does not infer the design from raw-file names. Instead, it can be declared in either of two ways: 1. **Rename the run columns** using the `Abundance: _` convention also used by Proteome Discoverer. The example distributed with NADIA follows this convention, so the call above needs no additional argument. 2. **Supply an annotation file through `annot_path`** and leave the report exactly as DIA-NN exported it. The `Column` field must contain the complete run headers, including the raw-file paths: ```{r diann-annot, eval = FALSE} preprocess_diann( "report.pg_matrix.tsv", condition_order = c("A", "B", "D"), annot_path = "design.tsv") ``` `design.tsv` must be a tab-separated file with one row for each run that should be imported and two columns named exactly `Column` and `Condition`. `Column` contains the complete header of the corresponding intensity column in the DIA-NN matrix, while `Condition` assigns that run to an experimental group. For example: ``` Column Condition D:\DIA-NN\Projects\experiment\sample_A1.raw A D:\DIA-NN\Projects\experiment\sample_A2.raw A D:\DIA-NN\Projects\experiment\sample_B1.raw B ``` The values in `Column` must match the report headers exactly, including the complete raw-file path when DIA-NN uses paths as column names. Runs omitted from the annotation file are not imported. `condition_order` then selects which of the annotated conditions are retained and sets their order in the subsequent analysis. If neither valid `Abundance: _` headers nor an annotation file are supplied, the error message lists the headers that must be entered in `Column`. Whichever route is used, NADIA constructs one row of `metadata` per retained run. The columns have the following meaning: - `R.FileName` preserves the complete intensity-column header read from the report. - `R.Condition` contains the experimental condition, obtained either from the renamed header or from `Condition` in the annotation file. - `R.Replicate` is read from a `_` suffix when that naming convention is used. For original DIA-NN headers without such a suffix, runs are numbered sequentially within each condition. - `Coding` is the sample identifier used to name the quantitative columns. It is the `_` suffix for renamed columns and the value of `Column` when `annot_path` is used. - `R.ProteinGroupsIdentified` reports how many protein groups have a non-missing intensity in that run. The following output shows an example of the metadata table extracted and constructed from the imported DIA-NN report: ```{r diann-md} diann$metadata ``` **Note on identification information.** The DIA-NN protein-group matrix contains little identification evidence. The two available fields are `N.Sequences`, which gives the total peptide-sequence count, and `N.Proteotypic.Sequences`, which gives the proteotypic peptide count. To preserve the common structure of the `proteomics_data` object, NADIA repeats these protein-level values across runs in `protein_id`, as it does for the global Proteome Discoverer metrics in TMT data. NADIA also calculates the number of protein groups with a reported intensity in each run and stores it as `R.ProteinGroupsIdentified` in the metadata table shown above. # TMT: Proteome Discoverer A TMT report from Proteome Discoverer is **wide**: it contains one row per protein group and one abundance column per sample. As with DIA-NN, the experimental design must be declared either through column names that follow the `Abundance: _` convention or through an annotation file supplied with `annot_path`. ::: {.lightbox-figure} ```{r tmt-input-structure, echo = FALSE, fig.align = "center", out.width = "100%", fig.cap = "Structure of the wide Proteome Discoverer TMT report accepted by `preprocess_tmt()`.", fig.alt = "Diagram of a wide TMT report from Proteome Discoverer. Each row represents one protein group. Global protein and identification annotations occupy the first columns, followed by one Abundance column for each TMT channel. The design is obtained from Condition_Replicate suffixes or from annot_path."} knitr::include_graphics("figures/TMT_Input_Structure.svg") ``` ::: The arguments accepted by `preprocess_tmt()` are listed below. The report and the conditions to retain are required. `annot_path` provides an alternative way to declare the design when the abundance suffixes do not follow the naming convention.

Show the complete preprocess_tmt() call and default arguments

```{r tmt-defaults, eval = FALSE} tmt <- preprocess_tmt( # Required input and conditions to retain file_path = "tmt_report.tsv", condition_order = c("Control", "Treatment"), # Optional channel annotation file; NULL uses the column suffixes annot_path = NULL, # Optional export; NULL keeps the result in memory export_dir = NULL, timestamp_suffix = TRUE, # Progress messages verbose = TRUE ) ```

Show the complete preprocess_lfq() call and default arguments

```{r lfq-defaults, eval = FALSE} lfq <- preprocess_lfq( # Required input file_path = "lfq_report.tsv", # Optional design file and condition filtering/order annot_path = NULL, condition_order = NULL, # Optional export; NULL keeps the result in memory export_dir = NULL, timestamp_suffix = TRUE, # Progress messages verbose = TRUE ) ```

## Declaring the experimental design The design can be supplied in either of two ways: 1. **Use informative abundance-column suffixes.** Columns named `Abundance: A_1`, `Abundance: A_2` and `Abundance: B_1`, for example, identify the condition and replicate directly. This is the convention used by the example report distributed with NADIA. 2. **Supply an annotation file.** This route is used when the text after `Abundance: ` identifies a sample but does not encode its experimental condition and replicate. The report itself does not need to be renamed. `design.tsv` must be a tab-separated file with the columns `Column` and `Condition`. `Column` contains the exact sample identifier that follows the `Abundance: ` prefix in the report. For example: ``` Column Condition Sample01 Control Sample02 Control Sample03 Treatment Sample04 Treatment ``` Thus, the abundance column `Abundance: Sample01` is identified by `Sample01` in `Column`. Samples omitted from the annotation file are not imported. If `condition_order` is supplied, it additionally selects the conditions to retain and sets their order; otherwise, NADIA uses their order of appearance. The annotation file is passed to `preprocess_lfq()` as follows: ```{r lfq-annot, eval = FALSE} preprocess_lfq( "lfq_report.tsv", condition_order = c("Control", "Treatment"), annot_path = "design.tsv") ``` The example report distributed with NADIA contains three conditions with four replicates each. Its abundance-column suffixes already encode the complete design, so neither `annot_path` nor `condition_order` needs to be supplied: ```{r lfq-run} lfq_file <- system.file("extdata", "nadia_lfq_report.tsv.gz", package = "NADIA") lfq <- preprocess_lfq(lfq_file, verbose = FALSE) lfq ``` The following output shows an example of the metadata table extracted and constructed from the imported LFQ report: ```{r lfq-md} lfq$metadata ``` Any additional columns in `design.tsv` are ignored. Batch, TMT/LFQ experiment and other covariate information should instead be supplied through `covariate_df` in `process_proteomics()`, as explained in `vignette("batch-correction")`. # Bioconductor: SummarizedExperiment Protein intensities prepared with other Bioconductor tools can be supplied in a `SummarizedExperiment`, for instance a protein-level assay of a `QFeatures` object. `preprocess_se()` converts one into the same `proteomics_data` object, and `process_proteomics()` can call this importer automatically when given a `SummarizedExperiment`. NADIA needs three things from the object: - the intensities in an assay, proteins in rows and samples in columns, linear by default or on the log2 scale with `log2_input = TRUE`; - unique row names, which become the protein identifiers; - the condition of each sample in `colData()$Condition`, with the replicate in `colData()$Replicate` or, without that column, numbered within each condition. Use `assay_name` to select an assay when the object contains several; otherwise, the first assay is used. Gene names and descriptions are read from `rowData()$Genes` and `rowData()$Description` when present. The samples are renamed `_`, as the other readers do, and the original names are kept in `metadata$R.FileName`. ```{r from-se} quant_cols <- grep("^PG.Quantity_", names(dia$protein_quant)) log2_intensities <- log2(as.matrix(dia$protein_quant[, quant_cols])) rownames(log2_intensities) <- dia$protein_quant$PG.ProteinGroups se <- SummarizedExperiment::SummarizedExperiment( assays = list(log2 = log2_intensities), colData = data.frame(Condition = dia$metadata$R.Condition)) from_se <- preprocess_se(se, log2_input = TRUE, verbose = FALSE) from_se head(from_se$metadata[, c("R.FileName", "R.Condition", "R.Replicate", "Coding")]) ``` This importer does not transfer identification metrics from `rowData()`: the precursor counts of `protein_id` are left empty, and no peptide counts or scores are created, so the interactive tables show only the quantities. The quantitative analysis is unaffected. # Where the design comes from All five importers produce a common object, but the location of the experimental design depends on the input. NADIA must know which condition and replicate correspond to each sample: | Input | Where the design is stored | |---|---| | Spectronaut | `R.Condition` and `R.Replicate` in the long report | | DIA-NN | Run-column suffixes, or an annotation file supplied through `annot_path` | | TMT (PD) | Abundance-column suffixes, or `annot_path` | | LFQ (PD) | Abundance-column suffixes, or `annot_path` | | `SummarizedExperiment` | `Condition` and, optionally, `Replicate` in `colData()` | Spectronaut carries the design within its report. For the three wide report formats, use the `Abundance: _` naming convention or a two-column annotation file. DIA-NN matches the file's `Column` values against complete run headers; the Proteome Discoverer importers match them against the text after `Abundance: `. For a `SummarizedExperiment`, conditions come from `colData()$Condition`. Replicates come from `colData()$Replicate` when available; otherwise they are numbered within each condition. With `preprocess_se()`, `condition_column` and `replicate_column` allow these fields to have other names. No annotation file is needed. `condition_order` selects the conditions to retain and determines their order in the analysis. It is required by the Spectronaut, DIA-NN and TMT importers, and optional for LFQ and `preprocess_se()`. LFQ defaults to order of appearance; `preprocess_se()` uses the condition factor's levels, or order of appearance when the condition is not a factor. # Naming the conditions Regardless of the input format, condition names become **column names in the design matrix**. The differential abundance model evaluates a contrast such as `"B-A"` as the expression `B - A`; it is not only a display label. This determines which condition names are safe. For example, conditions named `Tr-1` and `Ct-1` would produce `Tr-1-Ct-1`. R would interpret the hyphens as several subtraction operators rather than as characters within two condition names, so the intended contrast could not be constructed. The rule is simple: **each condition must have a syntactically valid R name.** In practice: | In the name | Verdict | Why | |---|---|---| | letters, digits (not first) | fine | `Ctrl`, `Treat24h` | | `.` full stop | **fine** | `Ct.1` gives the contrast `Tr.1-Ct.1` | | `_` underscore | **fine** | `Ct_a` gives `Tr_a-Ct_a` | | `-` hyphen | **rejected** | collides with the contrast separator | | space | **rejected** | not a syntactic name | | leading digit | **rejected** | `1Ct` is not a name; R would rewrite it `X1Ct` | Names that follow the rows marked **fine** can be used safely and allow the pipeline to complete without naming-related errors. If there is any doubt, consult the table above or check the proposed names with `make.names()`: ```{r names-test} candidates <- c("Ctrl", "Ct.1", "Ct_a", "Ct-1", "Ct 1", "1Ct") data.frame(name = candidates, valid = make.names(candidates) == candidates, what_r_would_do = make.names(candidates)) ``` The pipeline validates condition names before fitting the model and stops with an informative error if any are invalid. Sample identifiers must also remain consistent. A `Column` value in `design.tsv` that does not match the report produces an error, whereas samples omitted from the annotation file are excluded. If `metadata$Coding` is subsequently edited and no longer matches the corresponding `PG.Quantity_` column, the unmatched samples are removed from the analysis and the result contains fewer samples than expected. In all three **wide formats**, the shared suffix parser reads the replicate from the trailing digits of `_`. An underscore within the condition remains valid: `Ct_a_1` is parsed as condition `Ct_a`, replicate 1. When `annot_path` is used, the labels in its `Condition` field define the experimental conditions. A condition label can therefore be corrected in `design.tsv` without renaming any abundance column in the quantitative report; the values in `Column` must still match the corresponding report identifiers. # Five sources, one common object The five importers convert their inputs into a `proteomics_data` object. Each retains a source-specific class, while sharing the same three components: `metadata`, `protein_id` and `protein_quant`. The following comparison includes the `SummarizedExperiment` imported above: ```{r classes} inputs <- list(Spectronaut = dia, DIA_NN = diann, TMT = tmt, LFQ = lfq, SummarizedExperiment = from_se) vapply(inputs, function(x) class(x)[1], character(1)) vapply(inputs, function(x) inherits(x, "proteomics_data"), logical(1)) ``` ```{r elements} lapply(inputs, names) ``` ## Sample information `metadata` links each sample to its condition, replicate and quantitative columns. Additional fields depend on the source: Spectronaut includes detailed run-level identification counts, whereas the other inputs may provide fewer annotations. The shared structure does not imply that every source supplies the same evidence: ```{r md-cols} lapply(inputs, function(x) colnames(x$metadata)) ``` ## Identification evidence NADIA uses common column names for identification metrics. Most follow the Spectronaut convention, while scores use neutral names. The suffix `*` below identifies a sample; the global score is stored in `protein_quant`. | Metric | NADIA column | |---|---| | Precursor or PSM count | `PG.NrOfPrecursorsIdentified_*` | | Sequence or peptide count | `PG.NrOfStrippedSequencesIdentified_*` | | Sequence coverage | `PG.Coverage_*` | | Sample-level score | `PG.Score.RunWise_*` | | Global score | `PG.Score` | The meaning and availability of these fields depend on the input: - **Spectronaut:** precursor and stripped-sequence counts, coverage and `PG.Cscore (Run-Wise)` provide the sample-level metrics; `PG.Cscore` supplies the global score. - **DIA-NN:** `N.Sequences` supplies the sequence count. Precursor counts are empty; coverage and score columns are not created. - **TMT (Proteome Discoverer):** `# PSMs`, `# Peptides`, `Coverage [%]` and `Sum PEP Score` supply the metrics. These are protein-level totals repeated across channels, not independent identifications for each channel. `Sum PEP Score` also supplies the global score. - **LFQ (Proteome Discoverer):** the sample-specific `# PSMs (by Search Engine)`, `# Peptides (by Search Engine)` and `Score Mascot` columns provide counts and scores. Protein-level `Coverage [%]` is repeated across samples, and `Sum PEP Score` supplies the global score. - **SummarizedExperiment:** the importer does not transfer identification metrics from `rowData()`. Precursor counts remain empty; peptide counts, coverage and scores are not created. Gene names and descriptions are read when the corresponding annotation fields are present. A common column name does not make these metrics equivalent: PSM counts are not precursor counts, and scores from different engines do not share a scale. NADIA records the original score name in `nadia_labels`, which interactive tables use for their column labels: ```{r score-labels} lapply(inputs, function(x) attr(x$protein_quant, "nadia_labels")) ``` ## Quantities and downstream analysis For every input, each sample's abundances are stored in a `PG.Quantity_` column linked to `metadata$Coding`. This common mapping allows all five objects to enter the same processing workflow, even when their identification annotations differ: ```{r pipeline} processed <- lapply(inputs, process_proteomics, verbose = FALSE) lapply(processed, function(x) x$comparisons) ``` The same downstream functions perform normalization, imputation, differential abundance analysis and visualization. Their results depend on the supplied quantities and analysis settings, rather than on the availability of optional identification metrics. See `vignette("results-and-export")` for presentation and export of the results. # Session information ```{r session} sessionInfo() ```