Creating Your First Small Labeled Dataset

Creating a dataset from scratch teaches you more than using someone else's. Here is how to build a high-quality small labeled dataset for a real task.

45 min · Reviewed 2026

Small, Careful, and Documented

A 500-example dataset you built yourself often teaches more than a 50,000-example dataset someone else assembled. Because you struggled with every label, you will notice bias, ambiguity, and quality issues that get papered over at scale. Here is how to do one well.

Project: classify tweets as complaint vs. praise

Define the task precisely
Collect 300-500 raw examples
Write a labeling guideline document
Label the examples (ideally with at least 2 annotators)
Measure inter-annotator agreement
Clean and release as a versioned dataset

Step 1: precise definition

Step 2: collect raw examples

import pandas as pd # Option A: use an existing Twitter-style dataset from Hugging Face from datasets import load_dataset raw = load_dataset('cardiffnlp/tweet_eval', 'sentiment', split='train[:500]') df = raw.to_pandas()[['text']] # Option B: collect your own via API (with the platform's consent) # tweepy, praw for reddit, etc. df.to_csv('to_label.csv', index=False)Sourcing 500 raw examples

Step 3: the labeling guideline

LABELING GUIDE v1.0 ============================================= Label one of: complaint | praise | neither COMPLAINT: The post expresses negative feelings about a product, service, or experience. e.g., My phone just died for the third time today. PRAISE: The post expresses positive feelings about a product, service, or experience. e.g., This new update is amazing, everything is so much faster now! NEITHER: Anything else — questions, statements of fact, off-topic, jokes without sentiment. e.g., Does anyone know if iOS 17 supports this? EDGE CASES: - Sarcastic praise that is really complaint -> complaint - Complaint about a product phrased politely -> complaint - Mixed (some good some bad) -> choose dominant; break tie with neither - Non-English -> neither (we are only labeling English for now)A short but complete labeling guide

Step 4 & 5: label with two annotators

# After labeling, measure agreement import pandas as pd from sklearn.metrics import cohen_kappa_score df = pd.read_csv('labeled.csv') # has columns: text, anno_a, anno_b kappa = cohen_kappa_score(df['anno_a'], df['anno_b']) print(f'Cohen kappa: {kappa:.3f}') # Resolve disagreements by discussion, not by averaging disagreements = df[df['anno_a'] != df['anno_b']] print(f'{len(disagreements)} disagreements to resolve')Two annotators + agreement check

Step 6: release with a data card

Write a README with purpose, size, limitations
Choose a license (CC-BY-4.0 is a safe default for most)
Include labeling guidelines as a file in the repo
Report inter-annotator agreement
Version the dataset (v1.0 is better than undated)

The big idea: the dataset you build teaches you the problem. Every edge case surfaces. Every definition gets stress-tested. Shipping a small, careful, documented dataset is a credential in its own right.

End-of-lesson check

8 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-data-create-labeled-dataset

What is the main idea of "Creating Your First Small Labeled Dataset"?
1. Creating a dataset from scratch teaches you more than using someone else's. Here is how to build a high-quality small labeled dataset for a real task.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "Creating Your First Small Labeled Dataset"?
1. annotation
2. labeling
3. guidelines
4. labeling guide
Which use of AI fits this topic best?
1. Let the AI decide what matters without your review
2. Use the answer before checking whether it fits the situation
3. Write a README with purpose, size, limitations
4. Treat the AI output as automatically correct
What should a careful learner remember about "Write the guidelines first"?
1. Use AI to draft or organize ideas about labeling, then verify before acting.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. Use AI for drafting and comparison, but verify before publishing or relying on it.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about labeling be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about labeling.
Which action would help you apply "Creating Your First Small Labeled Dataset" responsibly?
1. Use the tool to avoid thinking through the tradeoff
2. Keep going even if the output conflicts with a trusted source
3. Treat the AI output as automatically correct
4. Choose a license (CC-BY-4.0 is a safe default for most)

← Back to interactive lesson

Tendril · Creators · AI Foundations