As an example, you might have people looking at photos of skin lesions and saying "It's a melanoma/It's not a melanoma" and then in the dataset we also find out whether it was a melanoma or not.
The datasets needn't be a medical context. They could, for example, be student answers to a true/false test.
Ideally there would be a good number of people (50+) making a good number of decisions (10+) but smaller datasets are of interest too. Furthermore, ideally there would be either 100% overlap or a good degree of overlap between the items that are responded to. To take my example from above, it'd be best if the people were responding independently to the same skin lesion photos, rather than everyone seeing different skin lesion photos.