Fields definition

We just have one new field to define: chebi.ex.id. This is the accession field for our new connector. All connector accession fields are in the form <connector_class_id>.id. This accession field is mainly used inside other databases, when they make references to other databases. The field accession, which is used in all entries of biodb connectors, contains the same value as the connector accession field (chebi.ex.id in our case) and is preferable when accessing an entry. The definition of the new field is quite simple, See @ref(tab:fielddecl) for explanations of the different parameters.

Database definition

The main part is the declaration of the new connector. This is done in the databases section, under the key chebi.id, which is the database identifier. See @ref(tab:conndecl) for explanations of the different parameters.

Final version of the YAML file

After setting some parsing expressions, the URLs and the searchable fields, we get a complete definition file, that you can find at:

defFile <- system.file("extdata", "chebi_ex.yml", package='biodb')

Its content is as follow:

databases:

  chebi.ex:
    name: ChEBI example connector
    description: An example connector for ChEBI.
    compound.db: true
    entry.content.encoding: UTF-8
    entry.content.type: xml
    parsing.expr:
      accession: substring-after(//chebi:return/chebi:chebiId,'CHEBI:')
      formula:
        - //chebi:Formulae/chebi:source[text()='ChEBI']/../chebi:data
        - (//chebi:Formulae/chebi:data)[1]
      inchi: //chebi:return/chebi:inchi
      inchikey: //chebi:return/chebi:inchiKey
      mass: //chebi:mass
      monoisotopic.mass: //chebi:monoisotopicMass
      name:
        - //chebi:chebiAsciiName
      smiles: //chebi:return/chebi:smiles
    searchable.fields:
      - name
      - monoisotopic.mass
      - molecular.mass
    remote: true
    scheduler.t: 1
    scheduler.n: 3
    urls:
      base.url: https://www.ebi.ac.uk/chebi/
      ws.url: https://www.ebi.ac.uk/webservices/chebi/2.0/
    xml.ns:
      chebi: https://www.ebi.ac.uk/webservices/chebi
      xsd: http://www.w3.org/2001/XMLSchema

fields:

  chebi.ex.id:
    description: ChEBI ID
    type: id
    card: many
    forbids.duplicates: true
    case.insensitive: true

Inheritance

The connector class is responsible for the connection to the database. In our case, the database is a compound database.

Methods to implement

See the help inside R about BiodbConn for details about the parameters of those functions.

Remote connection methods

The remote methods are used for three different goals. First to build URLs that access the web site, to get the URL of an entry page (doGetEntryPageUrl()) or to get the URL of an entry picture (doGetEntryImageUrl()) like a molecule representation. Second to get a list of database entry identifiers (doGetEntryIds()). Third to Get the content of an entry (doGetEntryContentRequest()).

In our implementations of doGetEntryPageUrl(), doGetEntryImageUrl() and doGetEntryContentRequest() (see below), you may notice the use of the getPropValSlot() method to get some base URLs ("base.url", "ws.url"). These values are defined inside the connector YAML definition file that we will detail below. Also, in those methods, we use the BiobdUrl class to build the URLs. BiodbUrl handles the building of the URL parameters, as well as the encoding of special characters.

Method for searching for entries

The implemented method (doSearchForEntries()) is a generic method used to search for entries inside the database by name, mass, or any other field. For our example we have decided to implement only the search by name in order to keep the code as simple and short as possible. To see a full implementation of this method, look at the official biodb ChEBI connector at biodbChebi. Inside the method’s code you will see that the implementation of the call to the ChEBI web service API has been left to the dedicated method wsGetLiteEntity().

Prototype to respect for web service methods

In biodb official implementations of remote connectors, the implementations of calls to web services are done in separate dedicated methods having in common some principles.

These principles are important, because they assure a uniformity between biodb extension packages, allowing users to identify immediately a web service method and recognize the biodb generic parameters inside it.

Example of a web service method, taken from official biodb ChEBI extension package:

wsGetLiteEntity=function(search=NULL, search.category='ALL', stars='ALL',
                         max.results=10,
                         retfmt=c('plain', 'parsed', 'request', 'ids')) {
}

A web service method name must start with the prefix ws, which stands for web service, and be followed by the database API name of the web service written in Java style (i.e.: an uppercase letter for the start of each word and lowercase letters for the rest).

The first parameters of the method are the database web service parameters.

The last parameters (max.results and retfmt) are biodb specific.

max.results controls the maximum number of results wanted, and must have a default value (usually 10).

retfmt, which stands for return format, controls the format of the method’s returned value. The default value of retfmt is set to a vector and then processed inside the method with the match.arg() method. Thus the “real” default value is the first value of the vector, which must always be "plain". The set of possible values for retfmt is variable from one web service method to another. However some of the values are compulsory. See @ref(tab:retfmtValues) for a full list of retfmt possible values officially accepted by biodb.

You may want to look into some of biodb implementations of connectors to official remote databases, and see how the calls to web services have been implemented in dedicated web service methods. See @ref(tab:biodbOfficialRemoteConns).

Implementation

Here is our implementation of the connector class:

ChebiExConn <- R6::R6Class("ChebiExConn",
inherit=biodb::BiodbConn,

public=list(

initialize=function(...) {
    super$initialize(...)
},

wsGetLiteEntity=function(search=NULL, search.category='ALL', stars='ALL',
                         max.results=10,
                         retfmt=c('plain', 'parsed', 'request', 'ids')) {

    # Check parameters
    chk::chk_string(search)
    chk::chk_in(search.category, self$getSearchCategories())
    chk::chk_number(max.results)
    chk::chk_gte(max.results, 0)
    chk::chk_in(stars, self$getStarsCategories())
    retfmt <- match.arg(retfmt)

    # Build request
    params <- c(search=search,
                searchCategory=search.category,
                maximumResults=max.results,
                starsCategory=stars)
    url <- c(self$getPropValSlot('urls', 'ws.url'), 'test/getLiteEntity')
    request <- self$makeRequest(method='get', url=BiodbUrl$new(url=url,
                                                                params=params),
                                 encoding='UTF-8')
    if (retfmt == 'request')
        return(request)

    # Send request
    results <- self$getBiodb()$getRequestScheduler()$sendRequest(request)

    # Parse
    if (retfmt != 'plain') {

        # Parse XML
        results <-  XML::xmlInternalTreeParse(results, asText=TRUE)

        if (retfmt == 'ids') {
            ns <- self$getPropertyValue('xml.ns')
            results <- XML::xpathSApply(results, "//chebi:chebiId",
                                        XML::xmlValue, namespaces=ns)
            results <- sub('CHEBI:', '', results)
            if (length(grep("^[0-9]+$", results)) != length(results))
                self$error("Impossible to parse XML to get entry IDs.")
        }
    }

    return(results)
}
),

private=list(
doSearchForEntries=function(fields=NULL, max.results=0) {

    ids <- character()

    if ( ! is.null(fields)) {

        # Search by name
        if ('name' %in% names(fields))
            ids <- self$wsGetLiteEntity(search=fields$name,
                                         search.category="ALL NAMES",
                                         max.results=0, retfmt='ids')
    }

    # Cut
    if (max.results > 0 && max.results < length(ids))
        ids <- ids[seq_len(max.results)]

    return(ids)
},

doGetEntryContentRequest=function(id, concatenate=TRUE) {

    url <- c(self$getPropValSlot('urls', 'ws.url'), 'test',
             'getCompleteEntity')

    urls <- vapply(id, function(x) BiodbUrl$new(url=url,
                                            params=list(chebiId=x))$toString(),
                   FUN.VALUE='')

    return(urls)
},

doGetEntryIds=function(max.results=NA_integer_) {
    return(NULL)
},

doGetEntryPageUrl=function(id) {
    # Overrides super class' method

    url <- c(self$getPropValSlot('urls', 'base.url'), 'searchId.do')

    fct <- function(x) {
        BiodbUrl$new(url=url, params=list(chebiId=x))$toString()
    }
    
    urls <- vapply(id, fct, FUN.VALUE='')

    return(urls)
},

doGetEntryImageUrl=function(id) {
    # Overrides super class' method

    url <- c(self$getPropValSlot('urls', 'base.url'), 'displayImage.do')

    fct <- function(x) {
        BiodbUrl$new(url=url, params=list(defaultImage='true', imageIndex=0,
                                      chebiId=x, dimensions=400))$toString()
    }
    
    urls <- vapply(id, fct, FUN.VALUE='')

    return(urls)
}
))

Here is our implementation of the entry class:

ChebiExEntry <- R6::R6Class("ChebiExEntry",
inherit=BiodbXmlEntry,

public=list(

initialize=function(...) {
    super$initialize(...)
}
),

private=list(
doCheck=function(content) {
    
    # You can do some more checks of the content here.
    
    return(TRUE)
}

,doParseFieldsStep2=function(parsed.content) {
    
    # TODO Implement your custom parsing processing here.
}

))

Connector for a local database

With biodb we can also write a connector for a local database. As a matter of fact, all the connectors included in biodb base package are local connectors only: mass.csv.file, comp.csv.file and mass.sqlite. See @ref(tab:connMethods) for a list of methods to implement when writing a local connector.

Connector for a mass spectra database

In the example above, we have implemented a compound database. Another type of database is a mass spectra database. The following connectors included in biodb package are mass spectra database connectors: mass.csv.file and mass.sqlite. See @ref(tab:massdbConnMethods) for a list of methods to implement when writing a mass spectra database connector.

Connector for a downloadable database

Some database servers do not propose web services, or other connection to the database, but propose to download the whole database for local processing.

biodb offers the possibility to handle the connection to such database servers, by setting downloadable to TRUE inside the definition of the database connector.

See @ref(tab:downloadableMethods) for a list of methods to implement inside your connector when writing a downloadable database connector.

How to implement other types of entry classes

We have seen in the example how to parse XML entries by writing an entry class that inherits from the BiodbXmlEntry class. As stated before, biodb provides other types of abstract entry classes, that facilitate the parsing of diverse entry content formats. Here is a review of those formats.

path: //input[@id='DATA']

chrom.col.id:
- liquidChromatography
- columnCode

accession: id
compound.id: comp_id
formula: chem_form

MyEntryClass <- R6::R6Class("MyEntryClass", inherit=biodb::BiodbCsvEntry,
    public=list(
        initialize=function() {
            super$initialize(sep=';', na.strings=c('', 'NA'))
        }
))

accession: id
name: fullname

accession: ^ENTRY\s+(\S+)\s+Compound
exact.mass: ^EXACT_MASS\s+(\S+)$
formula: ^FORMULA\s+(\S+)$

doParseContent=function(content) {

    # Get lines of content
    lines <- strsplit(content, "\r?\n")[[1]]

    return(lines)
},

doParseFieldsStep1=function(parsed.content) {

    # Get parsing expressions
    parsing.expr <- .self$getParent()$getPropertyValue('parsing.expr')

    .self$.assertNotNull(parsed.content)
    .self$.assertNotNa(parsed.content)
    .self$.assertNotNull(parsing.expr)
    .self$.assertNotNa(parsing.expr)
    .self$.assertNotNull(names(parsing.expr))

    # Loop on all parsing expressions
    for (field in names(parsing.expr)) {

        # Match whole content 
        g <- stringr::str_match(parsed.content, parsing.expr[[field]])

        # Get positive results
        results <- g[ ! is.na(g[, 1]), , drop=FALSE]

        # Any match ?
        if (nrow(results) > 0)
            .self$setFieldValue(field, results[, 2])
    }
}

doParseFieldsStep2=function(parsed.content) {

    # Remove fields with empty string
    for (f in .self$getFieldNames()) {
        v <- .self$getFieldValue(f)
        if (is.character(v) && ! is.na(v) && v == '')
            .self$removeField(f)
    }

    # Correct InChIKey
    if (.self$hasField('INCHIKEY')) {
        v <- sub('^InChIKey=', '', .self$getFieldValue('INCHIKEY'), perl=TRUE)
        .self$setFieldValue('INCHIKEY', v)
    }

    # Synonyms
    synonyms <- XML::xpathSApply(parsed.content, "//synonym", XML::xmlValue)
    if (length(synonyms) > 0)
        .self$appendFieldValue('name', synonyms)
}