New Annotation Guidelines

A correct sample should

  1. have every change tagged
    1. Rules
      1. Tags should be around the transformed string
      2. It should not include more than one sentence around the transformed part
      3. Text outside the tags should not be changed
    2. This example would be valid because the tags are around the negated part: Example
  2. be factually wrong
    1. Example: This sample is not factually wrong: Link to Doccano
    2. Example: This sample is not factually wrong: "Great Expansion" Example. Swapped Entities should be dissimilar to the original entity and not just a synonym (or subpart). E.g. “Great Collision” would be a good replacement.
  3. be different to original answer (contradicting ?)
    1. This example is very similar to the original answer. It doesn’t really change the fact. Example with Andromeda Galaxy.
  4. be coherent
    1. Example: This sample is not coherent: Link to Doccano
  5. It has to be an answer to the question
    1. Example: This sample is not answering the question because the question asks about a “helicopter on earth”: Link to Doccano

Nice to have would be

  1. plausibility (for non educated this could be true)

Things we drop immediately

How we cross-check facts