Essential Resources in learning AI

Legendary papers and blogs

• Richard Sutton’s Bitter Lesson: General methods leveraging computation and learning, rather than human-designed knowledge or rules, are the most effective path to achieving significant progress in AI.

• Attention Is All You Need: The landmark Google paper introducing transformers based on attention mechanism.

• Many-Shot In-Context Learning:
  • With enough examples, many-shot ICL (In-Context Learning) can match or even surpass the performance of fine-tuned models on several tasks.
  • Many-shot ICL can override pretraining biases and learn high-dimensional functions (e.g., numerical tasks) where few-shot ICL struggles.

Videos

• Andrej Karpathy’s LLM deep dive

Technical books

• Deep Learning by Aaron Courville, Ian Goodfellow, and Yoshua Bengio: The Deep Learning Bible
• Alice in a Differentiable Wonderland
• Artificial Intelligence: A Modern Approach by Peter Norvig and Stuart J. Russell