Skip to content
TARUN S / PORTFOLIOTAP OR PRESS ANY KEY TO SKIP

Developer

000

HalluDetect

TYPE
AI evaluation system
STACK
17 COMPONENTS
STATUS
SOURCE AVAILABLE
FEATURED
—
LINKS
SOURCE ↗

01 / OVERVIEW

A hallucination-detection system that fuses NLI, semantic similarity, and LLM-judge signals into a single calibrated faithfulness score — with an optional RAG pipeline that fetches its own reference context.

Built for the UE23AM343BA2 (Large Language Models and Their Applications) course, HalluDetect automatically flags when an LLM generates confident but unsupported or contradictory content. Its core contribution is HalluScore, a composite metric combining a DeBERTa-v3 NLI entailment scorer, a Sentence-Transformers semantic similarity/BERTScore module, and a Gemini-based LLM judge, weighted together with a rule-based factuality bonus and bootstrapped into a 95% confidence interval. On top of manual-context scoring, a RAG mode retrieves relevant chunks from a FAISS-backed knowledge base so no reference text needs to be supplied by hand. The system ships with a 15-sample hand-curated benchmark across five domains and four hallucination types, a full evaluation pipeline producing accuracy/F1/ROC-AUC/correlation metrics and six publication-quality plots, a Flask REST API with seven endpoints, and a dark-themed single-page dashboard for live scoring, weight tuning, and knowledge-base management.

GitHub ↗

02/ STACK & ARCHITECTURE

FRONTEND

  • HTML
  • CSS
  • JavaScript

BACKEND

  • Python
  • Flask
  • Flask-CORS
  • Transformers (DeBERTa-v3 NLI)
  • Sentence-Transformers
  • FAISS
  • Gemini API (google-genai)
  • scikit-learn
  • pandas
  • NumPy
  • SciPy
  • matplotlib
  • seaborn
  • NLTK

03 / WHAT WAS ACTUALLY HARD

  1. 01

    HalluScore — a calibrated composite faithfulness metric

    Combines NLI entailment (35%), semantic similarity (20%), LLM-judge reasoning (30%), and a rule-based factuality bonus (15%) into a single [0,1] score, with bootstrap 95% confidence intervals and FAITHFUL / UNCERTAIN / HALLUCINATED verdicts thresholded at 0.55 / 0.45.

  2. 02

    Three independent detection signals

    A DeBERTa-v3 cross-encoder scores sentence-level entailment/contradiction/neutral against the context; a MiniLM sentence-transformer computes BERTScore-style precision/recall/F1 for topic drift; a Gemini LLM judge reasons about fabricated facts and subtle errors via structured JSON verdicts, with robust parsing that handles markdown fences, thinking blocks, and truncated responses.

  3. 03

    RAG mode with automatic context retrieval

    A FAISS (with numpy fallback) vector store chunks and embeds a 12-document knowledge base spanning science, history, medicine, technology, and geography. Queries retrieve the top-k chunks, filter out anything below a 0.30 cosine-similarity threshold, and feed the result straight into the same NLI/semantic/LLM pipeline — with runtime document ingestion and index rebuilding exposed via the API.

  4. 04

    Full benchmark & analysis pipeline

    A 15-sample curated benchmark across 5 domains, 3 difficulty levels, and 4 hallucination types (contradictory, fabricated, unsupported) feeds an evaluator that computes accuracy, precision, recall, F1, ROC-AUC, Spearman/Pearson correlation, and per-domain accuracy, then generates six matplotlib/seaborn figures (score distributions, radar chart, domain accuracy, correlation heatmap, confusion matrix, hallucination-type breakdown) plus a text report.

  5. 05

    REST API and interactive dashboard

    A Flask backend exposes manual-context scoring, RAG scoring, knowledge-base search/add/stats, and cached benchmark results across 7 endpoints. A single-file dark-themed dashboard lets users run live scoring with sentence-level NLI breakdowns, tune HalluScore weights with sliders, browse a metric-comparison table against HHEM/SelfCheckGPT/G-Eval/FActScore, and manage the RAG knowledge base.