Model & research
How SafeType detects harmful content and what we use today. Part of our Duke University Deep Learning project.
Our custom model (production)
SafeType uses our custom content-safety model developed for the Duke Deep Learning project. It is deployed and live at:
SafeType AI Keyboard Safety Monitor — Live API
This app (parent dashboard and Android capture) is linked to that API. Messages are sent to the custom model for classification; results (label and confidence) are used to flag threats on the dashboard.
- Live API: https://dl-project-2-second-version.onrender.com
- Source repo: DL_Project_2-Second_version- (code, training, data)
- Input: Message text (and optional batch).
- Output: Predicted label (one of 10 classes), confidence, and top-3 class probabilities.
Threat categories detected (10 classes)
Our custom model classifies messages into these 10 categories:
- clean — Safe, non-harmful content
- racism — Racial hate or discrimination
- sexism — Sexist or misogynistic content
- profanity — Obscene or swear words
- cyberbullying — Insults, personal attacks
- toxicity — Generally toxic language
- hate_speech — Generic hate speech
- implicit_hate — Implicit or indirect hate
- threat — Threats, aggression
- sarcasm — Sarcastic or ironic (context-dependent)
Duke Deep Learning project — custom model repo
The custom SafeType content-safety models are developed in a separate repository: DL_Project_2-Second_version-. It implements an NLP system that detects 10 categories of harmful or sensitive content for children in online chat.
Models (10-way classification)
| Type | Model | Artifacts | Accuracy | Weighted F1 |
|---|---|---|---|---|
| Naive baseline | Keyword-matching + majority class | naive_baseline.pkl |
~0.21 | ~0.17 |
| Classical ML | TF-IDF + Logistic Regression | logistic_regression.pkl, tfidf_vectorizer.pkl |
~0.63 | ~0.62 |
| Neural | Fine-tuned DistilBERT | distilbert-toxicity/ |
~0.72 | ~0.72 |
Label taxonomy (10 classes)
clean, racism, sexism, profanity, cyberbullying, toxicity, hate_speech, implicit_hate, threat, sarcasm.
Training data (8 sources, balanced to 50k)
- Davidson et al. — 24,783 tweets (
labeled_data.csv) - Jigsaw Toxic Comment v1 — 159,571 comments (multi-label)
- Jigsaw Unintended Bias v2 — 100,000 sampled
- UC Berkeley D-Lab Measuring Hate Speech — 39,565 comments (HuggingFace)
- EDOS — Explainable Detection of Online Sexism, 20,000 posts (Gab & Reddit)
- HateXplain — 20,148 posts with target-community annotations
- ImplicitHate — 6,346 implicit hate tweets (HuggingFace)
- TweetEval irony — ~4,601 tweets for sarcasm/irony (HuggingFace)
Combined ~375k samples → balanced to 50,000 (5,000 per class). Pipeline: make_dataset.py → build_features.py (TF-IDF, handcrafted features) → model.py (train all three) → experiment.py (training-size sensitivity, noise robustness). Inference: main.py (interactive / single / batch); app.py for API.
Repository summary
| Component | Details |
|---|---|
| Custom model (production) | Deployed at dl-project-2-second-version.onrender.com. Linked to this app for message classification. |
| Model code & training | DL_Project_2-Second_version- — 3 models (Naive Baseline, Logistic Regression, DistilBERT), 10-class, 8 data sources, 50k balanced dataset. |
| This app (DLPR2-APP) | Android capture, Supabase, parent dashboard. Calls the custom model API for threat detection. |