Rhysbot · an open model experiment
A mind,
in the making.
What can a small language model learn from years of writing, conversations, and collected knowledge?
Explore the corpus ↓ Read the methods ↗ Corpus released · both models trained
The corpus, training code, and measured results are released. Nanochat was trained from scratch; its answers are incoherent. Qwen was adapted with continued pretraining and instruction tuning. Try both saved checkpoints below. Qwen answers can still be wrong.
—released excerpts
—words in released text
—retained Anki cards
$200total authorized compute budget
The whole pipeline.
From source material to a model someone can use. Each stage has inspectable artifacts and measured results.
01 / RELEASED
Collect & review
Reviewed notes, writing, original posts, and public transcripts.
02 / TRAINED
Pretrain
Tiny nanochat trained from scratch. An experiment; current samples are incoherent.
03 / TRAINED
Post-train
Qwen: 1,000 continued-pretraining and 400 instruction-tuning updates.
04 / MEASURED
Evaluate
Saved checkpoints compared with untouched base and instruction baselines.
05 / LIVE
Serve & learn
Verified saved checkpoints, short answers, and inference that scales to zero.
Talk to AI Rhys.
Chat, hear an AI version of Rhys’s voice, or compare answers with Qwen. Open chat full screen ↗ · Train on your own corpus ↗
Saved training checkpoints · research demo
Try the saved checkpoints.
Explore a tiny model trained from scratch and a Qwen adaptation. Nanochat is a toy experiment, with incoherent samples so far. Each answer identifies its actual checkpoint. Answers are limited to 64 tokens; a cold start can take a minute.
Connecting to the hosted checkpoint demo… The corpus explorer is ready below.
What changed?
On a small frozen held-out sample, the adaptation predicts study material better. Lower loss does not establish better factual answers or a faithful model of Rhys.
Nanochat post-training lowered held-out bits per byte from 2.1565 to 1.8188, but its completions remain incoherent. Read the model card and actual samples ↗
| Checkpoint | Text NLL | Answer NLL |
|---|---|---|
| Untouched base | 4.0368 | 2.9791 |
| Instruction baseline | 3.8480 | 3.7246 |
| After CPT | 1.5905 | 2.4637 |
| After CPT + SFT | 1.5913 | 2.2501 |
32 held-out documents; 10 study questions. Answer loss uses shared ChatML, not each model’s best native prompt. Instruction-baseline samples use its official template with thinking disabled. This is a small diagnostic, not a broad benchmark. Open the all-Anki comparison and review actual answers ↗
What training looked like.
Actual optimizer logs and held-out measurements. Objectives have separate panels. Nanochat logged NaNs from SFT step 44. Lower loss does not establish factual accuracy.
Explore the material.
Choose a source below or search the whole release. These are source excerpts, including podcast and HEE transcripts, not model answers.
Speaker-separated transcripts cover 214 original recordings. Existing model checkpoints still use the original frozen corpus.
Recording coverage: 214 of 216 inventoried recordings have screened speaker transcripts. The Meta community-call clip could not be acquired; a video titled solo piano was excluded from speech transcription. See recording-level coverage ↗
Measuring released text…
| Collection | Documents | Excerpts | Text bytes | Text size |
|---|
Loads public text from the GitHub repository. Your searches stay in your browser.
Collected ≠ authored.
Anki notes can quote other people. Podcast guests have their own views. Transcripts can contain errors. Every excerpt keeps its source and authorship label. HEE narration is attributed to Rhys based on his confirmation. Speaker turns were inferred from original audio for 214 recordings. Rhys labels use a voice reference cross-checked against his confirmed Digicene narration; other voices keep local guest labels. Anonymous or ambiguous identities remain uncertain. The unavailable Meta clip retains its earlier captions with unverified attribution. All podcast, YouTube and HEE transcripts are in this research corpus, but were excluded from both models’ pretraining and fine-tuning inputs. Speaker-aware training remains unfinished.
The initial instruction examples use actual study-note answers and verbatim writing continuations. They are a starting dataset, not proof that the model thinks like Rhys.
Privacy comes first.
Raw local archives are kept out of Git. Anki cards require individual semantic review before publication; unreviewed cards stay private. Actual personal-life material, private contact details, and credentials are withheld. We retained 20,980 of 21,301 cards; 321 were withheld.
Public-source excerpts receive conservative automated screening. This is a partial release with known gaps and screening limitations. Read the dataset card or report a record.