hackertube.io

The spelled-out intro to language modeling: building makemore

Andrej Karpathy · tutorial · Watch on YouTube

Machine learning & deep learningLLMs & generative AI

What it is

Andrej Karpathy builds a bigram character-level language model from scratch, introducing torch.Tensor and the framework of training, sampling and loss evaluation. It is the first step toward a GPT-style Transformer in his makemore series.

Why Hacker News recommends it

HN recommends it as a long but worthwhile way to demystify how language models work technically.

  • 2 people recommended it
  • 2 different months
  • 2 strong recommendations
  • first 2024, last 2024

What HN commenters said

  1. “It's long (2h), but well worth the time if anyone is curious about the technical workings of LLMs.”

    Builds a basic language model from scratch, demystifying LLMs.

    islewis, 2024-02 · Strong recommendation · in “Markov Chains are the Original Language Models”

  2. “Highly recommend this one”

    Highly recommended pointer for next-token prediction.

    georgehill, 2024-04 · Strong recommendation · in “Show HN: Next-token prediction in JavaScript”

Excerpts from Hacker News comments, linked to the originals. Summaries and labels were written by a language model from those comments.

Topics: makemore, bigram, pytorch, neural networks, karpathy, next-token prediction

Also recommended by the same people