Download

Click here to download the most up-to-date data (evidence-inference 2.0).

If you would like to download the original dataset, click here.


File Distinction

There are 4 different files contained in this zip file:

  1. The file annotations_merged.csv is all annotator answer information.
  2. The file prompts_merged.csv is all prompt information.
  3. The folder "Split files" has a list of which prompts belong to which split (train/dev/test).
  4. The folder "Text files" has all plain-text versions of each XML.

Annotation Format

  1. Annotations: The annotation files consist of the following headings: UserID, PromptID, PMCID, Valid Label, Valid Reasoning, Label, Annotations, Label Code, In Abstract, Start Evidence, End Evidence.
  2. UserID: UserID is correlated to an ID of which doctor found the 'Label' and 'Annotations' column.
  3. PromptID: PromptID determines which prompt the doctor is answering. The PromptID is also given in the prompt-csv files, in which a lookup can be used to find the corresponding outcome/intervention/comparator.
  4. PMCID: This is the ID that we use to identify the articles. In order to find the correct article used, simply attach "PMC" + PMCID + '.nxml' and search within the xml_files folder.
  5. Valid Label: This value will be either a 0/1. This will determine if the verifier certfies the multiple-choice response of the annotator. '0' correlates to a rejection, which '1' indicates acception.
  6. Valid Reasoning: This value will be either a 0/1. This will determine if the verifier certfies the multiple-choice response of the annotator. '0' correlates to a rejection, which '1' indicates acception.
  7. Label: This value will have a string value of 'significantly increased', 'significantly decreased', 'no significant difference' or 'invalid prompt.' This corresponds to the response that the annotator answered for the given PromptID.
  8. Annotations: This value is a segment of strings, delimited by ",". This section consists of portions of the text that the annotator cited as to why they selected the label that they did.
  9. Label Code: This is simply an integer version of the label. '0' corresponds to 'no significant difference', '1' corresponds to 'significantly increased', and '-1' corresponds to 'significantly decreased'.
  10. In Abstract: This column consists of responses of '0' and '1'. This column reads '1' if the annotator got the answer from the abstract, and '0' if he or she used more than the abstract in order to answer the question.
  11. Start Evidence: This column represents what index in the text that the “reasoning” from this row starts at.
  12. End Evidence: This column represents what index in the text that the “reasoning” from this row ends at.

Prompt Format

  1. PromptID: Like previously stated, this is an ID given to this specific row, including the PMCID, outcome, intervention and comparator.
  2. PMCID: This is the ID that we use to identify the articles. In order to find the correct article used, simply attach "PMC" + PMCID + '.nxml' and search within the xml_files folder.
  3. Outcome/Intervention/Comparator: The outcome/intervention/comparator columns represent the fill-in-the-blank inputs for the following prompt formed: With respect to outcome, characterize the reported difference between intervention and those receiving comparator.