AI / ML Fake News Dataset Annotator – Multimodal Annotation Tool
A standalone cross-platform GUI tool for collecting multimodal fake news detection datasets. Supports text, image, and video annotation with built-in duplicate auditing, inter-rater reliability testing, and real-time team sync.
OVERVIEW
A production-grade desktop application designed for academic research teams to collaboratively build multimodal fake news detection datasets. Features three operational modes (Annotate, Review, Re-label), comprehensive statistics dashboards, duplicate auditing, inter-rater reliability (Cohen & Fleiss Kappa), and automated GitHub Actions CI/CD for cross-platform releases.
Key features
Three operational modes: Annotate, Review, and Re-label for complete annotation workflow
Multimodal data entry supporting text, images, and video with drag-and-drop
Interactive statistics dashboard with dynamic multi-modal distribution analysis
Background duplicate auditing engine with side-by-side comparison modal
Inter-rater reliability testing with automatic balanced sample generation and Kappa scoring
Real-time team metrics sync via GitHub Gist integration
Integrated Telegram bot for distributing news links to assigned annotators
Cross-platform automated releases via GitHub Actions CI/CD (Windows, macOS, Linux)
Engineering
- Architected a modular MVC codebase with separated modes, UI builders, and data managers
- Built a background duplicate auditing thread with Jaccard/containment similarity checks
- Implemented automated CI/CD pipeline that builds standalone executables for 3 platforms on tag push
- Designed a dataset aggregation system that merges multi-annotator CSVs and non-duplicate lists seamlessly
Impact
- Used by a research team to collect a large-scale multimodal Bengali fake news dataset
- Enabled non-technical annotators to contribute without any coding knowledge
- Streamlined inter-rater reliability workflows that would otherwise require manual scripting
Screens
