preference-data

Notus is a collection of fine-tuned LLMs using SFT, DPO, SFT+DPO, and/or any other RLHF techniques, while always keeping a data-first approach

Generate conversational, tool-calling, structured-output, and preference datasets — easily and at scale

MCP server for human-in-the-loop surveys, A/B preference tests, ratings, and rankings. Get real human feedback inside Claude Code, Claude Desktop, Cursor, Windsurf, and any MCP client — powered by Datapoint AI.

vLLM pipeline for Constitutional AI data: red-team, critique and revision stages producing SFT and chosen/rejected Parquet

Curated tools, papers, datasets, and practices for LLM training data engineering.

RLHF preference data curation pipeline: HH-RLHF + UltraFeedback + OASST1 → quality filter → MinHash dedup → DPO-ready JSONL

This repository contains all artifacts produced during my bachelor's thesis on data modeling for collective decision-making.

How often do you correct your coding agent, and on what kind of work? Measures it from your Claude Code transcripts, drafts CLAUDE.md rules from your recurring corrections, and exports them as preference pairs.

synthetic-data research and engineering: generation, deterministic curation, validation, provenance, and release tooling using local and openrouter distill models

Human evaluation workbench for scoring LLM responses, collecting pairwise preferences, detecting hallucinations, and analyzing model quality.

Pairwise code evaluation: two responses rated on five dimensions, a 7-point preference, and a check that the preference matches the ratings.

Turn AI peer-to-peer. Ursa Minor trains models on the world's dispersed knowledge; Ursa Major imports your tuning into every model.

Self-hostable labeling and LLM-evaluation platform: configurable task templates, gold questions, rater reputation, position-bias counterbalancing, and LLM judges that merge with human votes.

Evaluation pipeline for LLM outputs: an anchored rubric, a hand-labeled golden set, an LLM judge checked for agreement against it, and preference data generation for fine-tuning.

Capybara Arena — deployed platform collecting pairwise human preferences on AI-generated CAD models (Elo rating, blob storage, CI).

Pairwise rating CLI for AI responses with per-axis scoring (helpfulness/harmlessness/accuracy/instruction-following), JSONL in/out, inter-rater Cohen's kappa

A forkable example of the human-in-the-loop model-improvement loop: AI generates, humans judge via the Terac MCP, you improve the model. Built as an SVG illustration arena.

RLHF preference-data toolkit: SQLite store for pairwise judgments, seeded ground-truth simulator, Bradley-Terry/Elo rankings with cluster-bootstrap CIs (annotator audits and DPO/KTO export on the roadmap)

DataSieve: validate and deduplicate LLM training data (SFT and preference records) from one pipeline spec - schema quarantine, exact and MinHash/LSH near-dedup, eval-set contamination check. Fork of text-dedup.

Measuring verbosity bias in UltraFeedback (61K GPT-4-annotated preference pairs): statistically unambiguous, but a small effect. Python + Streamlit.