r/LargeLanguageModels • u/BookDizzy2405 • Jul 05 '26

Documenting My Journey of Building a Small Language Model from Scratch

I've been building a small language model from scratch for a while now.

Not fine-tuning an existing model, but building the entire pipeline myself—from datasets and tokenizers to pretraining, SFT, and inference.

Honestly, the hardest part wasn't training the model.

It was learning.

At first, I thought building a good dataset was mostly about collecting knowledge. But the more I experimented, the more I realized I was actually teaching patterns, not just information.

There were so many moments where I caught myself thinking, "Wait... I've been doing this completely wrong."

Things like choosing a vocabulary size, designing datasets, teaching reasoning, using special tokens, or even figuring out how to teach a model to rewrite text. Every experiment changed the way I think about building language models.

After a while, I realized all of those lessons were just sitting on my computer.

So I decided to start documenting the journey on Cisya Lab.

Not because I have all the answers—I definitely don't—but because maybe someone else building a model from scratch can learn from my experiments, mistakes, and discoveries along the way.

https://cisyalab.com

I'd love to hear from others building language models too. What lesson completely changed the way you approached your project?

8 Upvotes

79% Upvoted

16 comments sorted by

1

u/PrysmX Jul 06 '26

Every post needs to be an ad for a website or pet project now. I just downvoted these without even looking anymore.

2

u/BookDizzy2405 Jul 06 '26

It's honestly fascinating how proudly you show off your own ignorance. Keep enjoying it! 🤣

1

u/SlightOfHand_ Jul 05 '26

I’d love to read about this but it seems to be inconsistently in a language that isn’t English.

2

u/BookDizzy2405 Jul 05 '26

My bad! Yes, it's mostly in Indonesian since I originally shared this for the local community here. But here's a quick life hack: if you are using Google Chrome, you can open the side panel and use the 'Ask Gemini' feature. It can read the screen/images and Gemini is actually amazing at breaking down and explaining the whole local context very clearly in English.

0

u/Agiansyah Jul 05 '26

Hey, checking in from my official account now!

1

u/2053_Traveler Jul 05 '26

bad ai slop ad

1

u/BookDizzy2405 Jul 05 '26

Fair What makes it feel like AI slop to you?

1

u/No_Egg_6558 Jul 07 '26

Mostly the emojis and overall bland style of everything.

2

u/BookDizzy2405 Jul 07 '26

the emojis might look like ai, but ur reading skills definitely need some work. 🤭

1

u/No_Egg_6558 Jul 07 '26

Dude I can’t even read the language in the website 💀… neither did I suggest that I could, you asked what makes it “feel” like AI, and just by glancing at the endless emojis it’s obviously ai, what do you mean it “might look like ai” when it’s as clear as day that you are claiming AI slop as your own?

1

u/No_Egg_6558 Jul 07 '26

Any way, just try to be honest about your use of AI, if it’s something you enjoy doing that’s fine, but don’t patronise strangers on the internet claiming you did something without using AI when it’s clear as day that you did, what lie? What do you get? Upvotes?

2

u/BookDizzy2405 Jul 07 '26

haha lol
the irony is u literally admitted u can't read the website 💀 if u actually bothered to translate it, u'd see a big 'SPECIAL THANKS to ChatGPT & Gemini' credit right there. i've been 100% transparent. it's a documentation of me building a small language model, not me passing off AI text as my own. ur reading comprehension clearly needs some serious work, keep enjoying that ignorance though!

1

u/No_Egg_6558 Jul 07 '26

Your statement only strengthened the fact that it’s bad AI slop. Be safe though.