Dive into Deep Learning

Table Of Contents

Dive into Deep Learning

Dive into Deep Learning

An interactive deep learning book with code, math, and discussions

The contents are under revision

Aston Zhang

Amazon Applied Scientist
UIUC Ph.D.

Zack C. Lipton

Amazon Applied Scientist
CMU Assistant Professor
UCSD Ph.D.

Mu Li

Amazon Principal Scientist
CMU Ph.D.

Alex J. Smola

Amazon Director
TU Berlin Ph.D.

We thank all the community contributors
for making this open source book better for everyone.

Contribute to the book.

Each section is an executable Jupyter notebook

You can modify the code and tune hyperparameters to get instant feedback to accumulate practical experiences in deep learning.

Mathematics + Figures + Code

We offer an interactive learning experience with mathematics, figures, code, text, and discussions, where concepts and techniques are illustrated and implemented with experiments on real data sets.

Active community support

You can discuss and learn with thousands of peers in the community through the link provided in each section.

Table of contents

• Preface
• Installation

• 1. Introduction
  • 1.1. A Motivating Example
  • 1.2. The Key Components: Data, Models, and Algorithms
  • 1.3. Kinds of Machine Learning
  • 1.4. Roots
  • 1.5. The Road to Deep Learning
  • 1.6. Success Stories
• 2. Crashcourse in the Preliminaries
  • 2.1. Data Manipulation
  • 2.2. Linear Algebra
  • 2.3. Automatic Differentiation
  • 2.4. Probability and Statistics
  • 2.5. Naive Bayes Classification
  • 2.6. Sampling
  • 2.7. Documentation
• 3. Linear Neural Networks
  • 3.1. Linear Regression
  • 3.2. Linear Regression Implementation from Scratch
  • 3.3. Concise Implementation of Linear Regression
  • 3.4. Softmax Regression
  • 3.5. Image Classification Data (Fashion-MNIST)
  • 3.6. Implementation of Softmax Regression from Scratch
  • 3.7. Concise Implementation of Softmax Regression
• 4. Multilayer Perceptrons
  • 4.1. Multilayer Perceptron
  • 4.2. Implementation of Multilayer Perceptron from Scratch
  • 4.3. Concise Implementation of Multilayer Perceptron
  • 4.4. Model Selection, Underfitting and Overfitting
  • 4.5. Weight Decay
  • 4.6. Dropout
  • 4.7. Forward Propagation, Back Propagation, and Computational Graphs
  • 4.8. Numerical Stability and Initialization
  • 4.9. Environment
  • 4.10. Predicting House Prices on Kaggle
• 5. Deep Learning Computation
  • 5.1. Layers and Blocks
  • 5.2. Parameter Management
  • 5.3. Deferred Initialization
  • 5.4. Custom Layers
  • 5.5. File I/O
  • 5.6. GPUs
• 6. Convolutional Neural Networks
  • 6.1. From Dense Layers to Convolutions
  • 6.2. Convolutions for Images
  • 6.3. Padding and Stride
  • 6.4. Multiple Input and Output Channels
  • 6.5. Pooling
  • 6.6. Convolutional Neural Networks (LeNet)
  • 6.7. Deep Convolutional Neural Networks (AlexNet)
  • 6.8. Networks Using Blocks (VGG)
  • 6.9. Network in Network (NiN)
  • 6.10. Networks with Parallel Concatenations (GoogLeNet)
  • 6.11. Batch Normalization
  • 6.12. Residual Networks (ResNet)
  • 6.13. Densely Connected Networks (DenseNet)
• 7. Recurrent Neural Networks
  • 7.1. Sequence Models
  • 7.2. Language Models
  • 7.3. Recurrent Neural Networks
  • 7.4. Text Preprocessing
  • 7.5. Implementation of Recurrent Neural Networks from Scratch
  • 7.6. Concise Implementation of Recurrent Neural Networks
  • 7.7. Back-propagation Through Time
  • 7.8. Gated Recurrent Unit (GRU)
  • 7.9. Long Short-term Memory (LSTM)
  • 7.10. Deep Recurrent Neural Networks
  • 7.11. Bidirectional Recurrent Neural Networks
• 8. Optimization Algorithms
  • 8.1. Optimization and Deep Learning
  • 8.2. Gradient Descent and Stochastic Gradient Descent
  • 8.3. Mini-batch Stochastic Gradient Descent
  • 8.4. Momentum
  • 8.5. Adagrad
  • 8.6. RMSProp
  • 8.7. Adadelta
  • 8.8. Adam
• 9. Computational Performance
  • 9.1. A Hybrid of Imperative and Symbolic Programming
  • 9.2. Asynchronous Computing
  • 9.3. Automatic Parallelism
  • 9.4. Multi-GPU Computation Implementation from Scratch
  • 9.5. Concise Implementation of Multi-GPU Computation
• 10. Computer Vision
  • 10.1. Image Augmentation
  • 10.2. Fine Tuning
  • 10.3. Object Detection and Bounding Boxes
  • 10.4. Anchor Boxes
  • 10.5. Multiscale Object Detection
  • 10.6. Object Detection Data Set (Pikachu)
  • 10.7. Single Shot Multibox Detection (SSD)
  • 10.8. Region-based CNNs (R-CNNs)
  • 10.9. Semantic Segmentation and Data Sets
  • 10.10. Fully Convolutional Networks (FCN)
  • 10.11. Neural Style Transfer
  • 10.12. Image Classification (CIFAR-10) on Kaggle
  • 10.13. Dog Breed Identification (ImageNet Dogs) on Kaggle
• 11. Natural Language Processing
  • 11.1. Word Embedding (word2vec)
  • 11.2. Approximate Training
  • 11.3. Implementation of Word2vec
  • 11.4. Subword Embedding (fastText)
  • 11.5. Word Embedding with Global Vectors (GloVe)
  • 11.6. Finding Synonyms and Analogies
  • 11.7. Text Sentiment Classification: Using Recurrent Neural Networks
  • 11.8. Text Sentiment Classification: Using Convolutional Neural Networks (textCNN)
  • 11.9. Encoder-Decoder (seq2seq)
  • 11.10. Beam Search
  • 11.11. Attention Mechanism
  • 11.12. Machine Translation
• 12. Appendix
  • 12.1. List of Main Symbols
  • 12.2. Mathematical Basis
  • 12.3. Using Jupyter Notebook
  • 12.4. Using AWS to Run Code
  • 12.5. GPU Purchase Guide
  • 12.6. How to Contribute to This Book
  • 12.7. d2l Package Index

Next

Preface