The Task

Given a treatment A, a comparator B, and an outcome, infer the reported relationship between A and B with respect to outcome, and provide evidence supporting this from the text.


Approach

Rather than turn to large crowdsourcing platforms, like Amazon Mechanical Turk, we decided to hire a group of doctors via Upwork, as we felt that the quality of annotations would be of a higher standard. We split the group of doctors into three groups: Prompt Generation, Annotation, and Verification.

  1. Prompt Generation. Prompt generators are tasked with scanning through medical research papers and finding sentences that parallel the outcome, intervention, and comparator structure. The generator is then asked to identify how the intervention relates to the outcome with respect to a specific comparator.
  2. Annotation. The annotator is given the outcome, intervention, and comparator developed by the prompt generator, along with the corresponding article. Like the prompt generator, the annotator is asked to determine the relationship between the outcome, intervention, and comparator, while also citing a quote from the text that supports his or her claim.
  3. Verification. Prompt generation and annotation creates two independent set of answers and reasonings for a given prompt. The verifier is given both of these, and asked to determine which, if not both, are accurate.

Dataset

The immense growth of published randomized control trials impedes a doctor’s ability to confidently determine what interventions are most suited to a given problem, as it is not feasible for medical professionals to parse through enormous amounts existing literature. The goal of our research project is to generate avante-garde models and a new corpora, in order to automatically extract results for pairs of interventions, comparators, and outcomes with respect to a specific paper describing a research trial.

Citation and Reference

    Code for working with the data and training the models proposed thus far is available on GitHub: https://github.com/jayded/evidence-inference.
    • The first dataset iteration paper can be found at this link.
    • The second dataset iteration of this paper can be find at this link.
    • If you use this dataset, please download the following citation file.

This work is supported by NSF CAREER Award 1750978.

* Special thank you for the Newsies website for layout inspiration (https://summari.es/).