Dive into Deep Learning

Dive into Deep Learning

An interactive deep learning book with code, math, and discussions.

Announcements

• [Apr, 2020] We have re-organized Chapter: NLP pretraining and Chapter: NLP applications, and added sections of BERT (model, data, pretraining, fine-tuning, application) and natural language inference (data, model). To keep track of the latest updates, please follow D2L's open-source project.
• [Dec, 2019] All the code has been rewritten with the NumPy API. Check out new instructions to run this book on Amazon SageMaker and Google Colab.
• [Oct, 2019] We have improved Chapter: Preliminaries, added Chapter: Recommender Systems and Appendix: Mathematics for Deep Learning.
• [Jul, 2019] The Chinese version is the No. 1 best seller of new books in "Computers and Internet" at the largest Chinese online bookstore.
• [May, 2019] Slides, Jupyter notebooks, assignments, and videos of the Berkeley course can be found at the syllabus page.

Authors

Aston Zhang

Amazon Senior Scientist

Zack C. Lipton

Amazon Scientist
CMU Assistant Professor

Mu Li

Amazon Principal Scientist

Alex J. Smola

Amazon VP/Distinguished Scientist

Chapter Authors

Brent Werness

Amazon Research Scientist
Mathematics for Deep Learning

Rachel Hu

Amazon Applied Scientist
Mathematics for Deep Learning

Shuai Zhang

Postdoctoral Researcher at ETH Zürich
Recommender Systems

Yi Tay

Google Research Scientist
Recommender Systems

We thank all the community contributors
for making this open source book better for everyone.

Contribute to the book.

Star Fork

Each section is an executable Jupyter notebook

You can modify the code and tune hyperparameters to get instant feedback to accumulate practical experiences in deep learning.

Run
locally

Amazon
SageMaker

Google
Colab

Mathematics + Figures + Code

We offer an interactive learning experience with mathematics, figures, code, text, and discussions, where concepts and techniques are illustrated and implemented with experiments on real data sets.

Active community support

You can discuss and learn with thousands of peers in the community through the link provided in each section.

D2L as a textbook or a reference book

[+] Click here to show the full list.

Carnegie Mellon University
Duke University
Emory University
Fudan University
Gazi Üniversitesi
Georgia Institute of Technology
Habib University
Harbin Institute of Technology
Hasso-Plattner-Institut
Hangzhou Dianzi University
Hiroshima University
Hong Kong University of Science and Technology
Huazhong University of Science and Technology
Imperial College London
Indian Institute of Technology Bombay
Indian Institute of Technology Kanpur
Indian Institute of Technology Kharagpur
Indian Institute of Technology Mandi
Indian Institute of Technology Ropar
Indraprastha Institute of Information Technology, Delhi
Institut Supérieur De L'electronique Et Du Numérique
İstanbul Teknik Üniversitesi
King Abdullah University of Science and Technology
Kyungpook National University
Lancaster University
Massachusetts Institute of Technology
McGill University
Monash University
National Chung Hsing University
National Institute of Technical Teachers Training&Research
National Institute of Technology, Warangal
National Taiwan University
National United University
National University of Singapore
Nazarbayev University
Northeastern University
Ohio University
Peking University
Pontificia Universidad Católica de Chile
Portland State University
Rutgers, The State University of New Jersey
Sapienza Università di Roma
Shanghai Jiao Tong University
Shanghai University of Finance and Economics
Sogang University
Stanford University
Stevens Institute of Technology
Technische Universiteit Delft
Texas A&M University
The University of Texas at Austin
Tsinghua University
Universidad Carlos III de Madrid
Universidad Nacional de Colombia Sede Manizales
Universidade Federal de Minas Gerais
Università degli Studi di Brescia
Università degli Studi di Catania
Universität Heidelberg
Universitatea de Vest din Timișoara
University of Arkansas
University of California, Berkeley
University of California, Los Angeles
University of California, San Diego
University of California, Santa Barbara
University of Illinois at Urbana-Champaign
University of Maryland
University of Minnesota, Twin Cities
University of New Hampshire
University of North Carolina at Chapel Hill
University of Pennsylvania
University of Science and Technology of China
University of Technology Sydney
University of Washington
University of Waterloo
Univerzita Komenského v Bratislave
Vietnamese-German University
Yunnan University
Universitat Politècnica de Catalunya
Zhejiang University

If you use D2L to teach (or plan to) and would like to receive a hardcopy, please contact us at d2lbook.en@gmail.com.

BibTeX entry for citing the book

@book{zhang2020dive,
    title={Dive into Deep Learning},
    author={Aston Zhang and Zachary C. Lipton and Mu Li and Alexander J. Smola},
    note={\url{https://d2l.ai}},
    year={2020}
}

Table of contents

• Preface
• Installation
• Notation

• 1. Introduction
  • 1.1. A Motivating Example
  • 1.2. The Key Components: Data, Models, and Algorithms
  • 1.3. Kinds of Machine Learning
  • 1.4. Roots
  • 1.5. The Road to Deep Learning
  • 1.6. Success Stories
  • 1.7. Summary
  • 1.8. Exercises
• 2. Preliminaries
  • 2.1. Data Manipulation
  • 2.2. Data Preprocessing
  • 2.3. Linear Algebra
  • 2.4. Calculus
  • 2.5. Automatic Differentiation
  • 2.6. Probability
  • 2.7. Documentation
• 3. Linear Neural Networks
  • 3.1. Linear Regression
  • 3.2. Linear Regression Implementation from Scratch
  • 3.3. Concise Implementation of Linear Regression
  • 3.4. Softmax Regression
  • 3.5. The Image Classification Dataset
  • 3.6. Implementation of Softmax Regression from Scratch
  • 3.7. Concise Implementation of Softmax Regression
• 4. Multilayer Perceptrons
  • 4.1. Multilayer Perceptrons
  • 4.2. Implementation of Multilayer Perceptron from Scratch
  • 4.3. Concise Implementation of Multilayer Perceptron
  • 4.4. Model Selection, Underfitting and Overfitting
  • 4.5. Weight Decay
  • 4.6. Dropout
  • 4.7. Forward Propagation, Backward Propagation, and Computational Graphs
  • 4.8. Numerical Stability and Initialization
  • 4.9. Considering the Environment
  • 4.10. Predicting House Prices on Kaggle
• 5. Deep Learning Computation
  • 5.1. Layers and Blocks
  • 5.2. Parameter Management
  • 5.3. Deferred Initialization
  • 5.4. Custom Layers
  • 5.5. File I/O
  • 5.6. GPUs
• 6. Convolutional Neural Networks
  • 6.1. From Dense Layers to Convolutions
  • 6.2. Convolutions for Images
  • 6.3. Padding and Stride
  • 6.4. Multiple Input and Output Channels
  • 6.5. Pooling
  • 6.6. Convolutional Neural Networks (LeNet)
• 7. Modern Convolutional Neural Networks
  • 7.1. Deep Convolutional Neural Networks (AlexNet)
  • 7.2. Networks Using Blocks (VGG)
  • 7.3. Network in Network (NiN)
  • 7.4. Networks with Parallel Concatenations (GoogLeNet)
  • 7.5. Batch Normalization
  • 7.6. Residual Networks (ResNet)
  • 7.7. Densely Connected Networks (DenseNet)
• 8. Recurrent Neural Networks
  • 8.1. Sequence Models
  • 8.2. Text Preprocessing
  • 8.3. Language Models and the Dataset
  • 8.4. Recurrent Neural Networks
  • 8.5. Implementation of Recurrent Neural Networks from Scratch
  • 8.6. Concise Implementation of Recurrent Neural Networks
  • 8.7. Backpropagation Through Time
• 9. Modern Recurrent Neural Networks
  • 9.1. Gated Recurrent Units (GRU)
  • 9.2. Long Short Term Memory (LSTM)
  • 9.3. Deep Recurrent Neural Networks
  • 9.4. Bidirectional Recurrent Neural Networks
  • 9.5. Machine Translation and the Dataset
  • 9.6. Encoder-Decoder Architecture
  • 9.7. Sequence to Sequence
  • 9.8. Beam Search
• 10. Attention Mechanisms
  • 10.1. Attention Mechanisms
  • 10.2. Sequence to Sequence with Attention Mechanisms
  • 10.3. Transformer
• 11. Optimization Algorithms
  • 11.1. Optimization and Deep Learning
  • 11.2. Convexity
  • 11.3. Gradient Descent
  • 11.4. Stochastic Gradient Descent
  • 11.5. Minibatch Stochastic Gradient Descent
  • 11.6. Momentum
  • 11.7. Adagrad
  • 11.8. RMSProp
  • 11.9. Adadelta
  • 11.10. Adam
  • 11.11. Learning Rate Scheduling
• 12. Computational Performance
  • 12.1. Compilers and Interpreters
  • 12.2. Asynchronous Computation
  • 12.3. Automatic Parallelism
  • 12.4. Hardware
  • 12.5. Training on Multiple GPUs
  • 12.6. Concise Implementation for Multiple GPUs
  • 12.7. Parameter Servers
• 13. Computer Vision
  • 13.1. Image Augmentation
  • 13.2. Fine-Tuning
  • 13.3. Object Detection and Bounding Boxes
  • 13.4. Anchor Boxes
  • 13.5. Multiscale Object Detection
  • 13.6. The Object Detection Dataset
  • 13.7. Single Shot Multibox Detection (SSD)
  • 13.8. Region-based CNNs (R-CNNs)
  • 13.9. Semantic Segmentation and the Dataset
  • 13.10. Transposed Convolution
  • 13.11. Fully Convolutional Networks (FCN)
  • 13.12. Neural Style Transfer
  • 13.13. Image Classification (CIFAR-10) on Kaggle
  • 13.14. Dog Breed Identification (ImageNet Dogs) on Kaggle
• 14. Natural Language Processing: Pretraining
  • 14.1. Word Embedding (word2vec)
  • 14.2. Approximate Training
  • 14.3. The Dataset for Pretraining Word Embedding
  • 14.4. Pretraining word2vec
  • 14.5. Word Embedding with Global Vectors (GloVe)
  • 14.6. Subword Embedding
  • 14.7. Finding Synonyms and Analogies
  • 14.8. Bidirectional Encoder Representations from Transformers (BERT)
  • 14.9. The Dataset for Pretraining BERT
  • 14.10. Pretraining BERT
• 15. Natural Language Processing: Applications
  • 15.1. Sentiment Analysis and the Dataset
  • 15.2. Sentiment Analysis: Using Recurrent Neural Networks
  • 15.3. Sentiment Analysis: Using Convolutional Neural Networks
  • 15.4. Natural Language Inference and the Dataset
  • 15.5. Natural Language Inference: Using Attention
  • 15.6. Fine-Tuning BERT for Sequence-Level and Token-Level Applications
  • 15.7. Natural Language Inference: Fine-Tuning BERT
• 16. Recommender Systems
  • 16.1. Overview of Recommender Systems
  • 16.2. The MovieLens Dataset
  • 16.3. Matrix Factorization
  • 16.4. AutoRec: Rating Prediction with Autoencoders
  • 16.5. Personalized Ranking for Recommender Systems
  • 16.6. Neural Collaborative Filtering for Personalized Ranking
  • 16.7. Sequence-Aware Recommender Systems
  • 16.8. Feature-Rich Recommender Systems
  • 16.9. Factorization Machines
  • 16.10. Deep Factorization Machines
• 17. Generative Adversarial Networks
  • 17.1. Generative Adversarial Networks
  • 17.2. Deep Convolutional Generative Adversarial Networks
• 18. Appendix: Mathematics for Deep Learning
  • 18.1. Geometry and Linear Algebraic Operations
  • 18.2. Eigendecompositions
  • 18.3. Single Variable Calculus
  • 18.4. Multivariable Calculus
  • 18.5. Integral Calculus
  • 18.6. Random Variables
  • 18.7. Maximum Likelihood
  • 18.8. Distributions
  • 18.9. Naive Bayes
  • 18.10. Statistics
  • 18.11. Information Theory
• 19. Appendix: Tools for Deep Learning
  • 19.1. Using Jupyter
  • 19.2. Using Amazon SageMaker
  • 19.3. Using AWS EC2 Instances
  • 19.4. Using Google Colab
  • 19.5. Selecting Servers and GPUs
  • 19.6. Contributing to This Book
  • 19.7. d2l API Document

• References