The main aim of Epistemonikos is to identify all of the systematic reviews relevant for health decision-making.
The identification of other types of evidence has not been completed yet, so Epistemonikos database should not be considered a comprehensive database of broad synthesis, primary studies or structured summaries (see definitions below).
In order to identify systematic reviews multiple electronic databases and other sources are regularly screened without any restriction, including:
All of the systematic reviews being summarized in structured summaries, broad syntheses, and methodological studies (which is an exclusion criteria from Epistemonikos) are also included in the database.
Additionally, Epistemonikos is a living project. So, hundreds of professionals and researchers are using it to identify the best available evidence for real-life questions. Any time they identify relevant reviews these are assessed and included if eligible.
We select articles in three main categories plus one secondary category:
We group in this category different types of articles aiming to make a synthesis of systematic reviews and, sometimes, primary studies.
We consider a broad synthesis for inclusion if,
We consider a systematic review for inclusion if it fulfills the following criteria:
We also include synthesis of primary studies that do not fulfill the above definition but are judged to add valuable information. Some examples are individual patient meta-analysis or meta-analysis of unpublished data.
Primary study is an umbrella term that includes any study design, qualitative or quantitative, where data is collected from individuals or groups of people.
We read the full text of the articles in the above categories to extract the list of included studies that the authors reported, and we incorporate them to the database. We do not add any additional criteria to those imposed by the researchers conducting the systematic reviews.
Apart from the 3 main categories we also include a secondary category of structured summaries. Structured summaries are summaries of an article developed for non-research audiences which follow a pre-established format which include at least a critical appraisal of the article.
We exclude articles from any of the above categories if they:
Articles coming from different databases are deduplicated using our own software followed by a manual revision of borderline cases.
Potentially eligible articles are initially classified by a machine-learning algorithm. Articles with a high probability of eligibility enter the database as soon as available. So, all of the articles are checked by the network of collaborators, but those with a higher score are assessed first.
Articles excluded by the machine are checked by human screeners, starting by those with a higher machine learning score.
Articles excluded by one screener are classified by the machine learning algorithm too, and those with a high score are checked by a second screener.
Any discrepancy between machine or human, or between different humans is resolved by a human arbiter.
All members of the network of screeners go through a brief training and follow a calibration exercise.
Collaborators read the selected systematic reviews in full text and extract the list of references of included studies. If a list of included studies is not clearly provided by a systematic review, we extract it from tables, annexes or from the text.
If it is not possible to identify a reliable list, we contact authors by email when possible.
The studies included in systematic reviews that fulfill our inclusion criteria are entered into the database independent of the source where the reviewers found them, language or publication status.
Starting from any systematic review for which the information of included studies has been entered in the database, Epistemonikos software generates a table displaying the cluster of systematic reviews that share at least one included study, and all of the studies included in those reviews.
This table is trimmed by collaborators so only systematic reviews and primary studies answering the question of interest are retained.
After removal of irrelevant articles, the software retrieves any other systematic review sharing included studies with any of the primary studies in the matrix. Relevant articles are added.
A second collaborator reviews the included/excluded articles of the matrix.
After completion of the matrix with the information in Epistemonikos database, cross-citation searches for all of the included primary studies are conducted in Google Scholar, Scopus and Web of Science. Relevant systematic reviews (and their included studies) are entered into Epistemonikos and then in the matrix of evidence.
Epistemonikos software automatically detect any new systematic review sharing primary studies with the matrix. Collaborators are notified and they include/exclude the new evidence as soon as it is assessed.
