Dive into Deep Learning

Table Of Contents

Dive into Deep Learning

Dive into Deep Learning

An interactive deep learning book with code, math, and discussions

The contents are under revision

Aston Zhang

Amazon Applied Scientist
UIUC Ph.D.

Zack C. Lipton

Amazon Applied Scientist
CMU Assistant Professor
UCSD Ph.D.

Mu Li

Amazon Principal Scientist
CMU Ph.D.

Alex J. Smola

Amazon Director
TU Berlin Ph.D.

We thank all the community contributors
for making this open source book better for everyone.

Contribute to the book.

Each section is an executable Jupyter notebook

You can modify the code and tune hyperparameters to get instant feedback to accumulate practical experiences in deep learning.

Mathematics + Figures + Code

We offer an interactive learning experience with mathematics, figures, code, text, and discussions, where concepts and techniques are illustrated and implemented with experiments on real data sets.

Active community support

You can discuss and learn with thousands of peers in the community through the link provided in each section.

Table of contents

• Preface

• 1. Introduction
  • 1.1. Introduction to Deep Learning
  • 1.2. Using this Book
• 2. A Taste of Deep Learning
  • 2.1. Introduction
  • 2.2. Getting started with Gluon
  • 2.3. Manipulating Data with ndarray
  • 2.4. Linear algebra
  • 2.5. Automatic Differentiation
  • 2.6. Probability and statistics
  • 2.7. Naive Bayes Classification
  • 2.8. Sampling
  • 2.9. Documentation
• 3. Deep Learning Basics
  • 3.1. Linear Regression
  • 3.2. Linear regression implementation from scratch
  • 3.3. Gluon Implementation of Linear Regression
  • 3.4. Softmax Regression
  • 3.5. Image Classification Data (Fashion-MNIST)
  • 3.6. Softmax Regression from Scratch
  • 3.7. Softmax Regression in Gluon
  • 3.8. Multilayer Perceptron
  • 3.9. Implementing a Multilayer Perceptron from Scratch
  • 3.10. Multilayer Perceptron in Gluon
  • 3.11. Model Selection, Underfitting and Overfitting
  • 3.12. Weight Decay
  • 3.13. Dropout
  • 3.14. Forward Propagation, Back Propagation and Computational Graphs
  • 3.15. Numerical Stability and Initialization
  • 3.16. Environment
  • 3.17. Predicting House Prices on Kaggle
• 4. Deep Learning Computation
  • 4.1. Layers and Blocks
  • 4.2. Parameter Management
  • 4.3. Deferred Initialization
  • 4.4. Custom Layers
  • 4.5. File I/O
  • 4.6. GPUs
• 5. Convolutional Neural Networks
  • 5.1. From Dense Layers to Convolutions
  • 5.2. Convolutions for Images
  • 5.3. Padding and Stride
  • 5.4. Multiple Input and Output Channels
  • 5.5. Pooling
  • 5.6. Convolutional Neural Networks (LeNet)
  • 5.7. Deep Convolutional Neural Networks (AlexNet)
  • 5.8. Networks Using Blocks (VGG)
  • 5.9. Network in Network (NiN)
  • 5.10. Networks with Parallel Concatenations (GoogLeNet)
  • 5.11. Batch Normalization
  • 5.12. Residual Networks (ResNet)
  • 5.13. Densely Connected Networks (DenseNet)
• 6. Recurrent Neural Networks
  • 6.1. Sequence Models
  • 6.2. Language Models
  • 6.3. Recurrent Neural Networks
  • 6.4. Text Preprocessing
  • 6.5. Building a Recurrent Neural Network from Scratch
  • 6.6. Gluon Implementation in Recurrent Neural Networks
  • 6.7. Back-propagation Through Time
  • 6.8. Gated Recurrent Unit (GRU)
  • 6.9. Long Short-term Memory (LSTM)
  • 6.10. Deep Recurrent Neural Networks
  • 6.11. Bidirectional Recurrent Neural Networks
• 7. Optimization Algorithms
  • 7.1. Optimization and Deep Learning
  • 7.2. Gradient Descent and Stochastic Gradient Descent
  • 7.3. Mini-Batch Stochastic Gradient Descent
  • 7.4. Momentum
  • 7.5. Adagrad
  • 7.6. RMSProp
  • 7.7. Adadelta
  • 7.8. Adam
• 8. Computational Performance
  • 8.1. A Hybrid of Imperative and Symbolic Programming
  • 8.2. Asynchronous Computing
  • 8.3. Automatic Parallelism
  • 8.4. Multi-GPU Computation
  • 8.5. Gluon Implementation for Multi-GPU Computation
• 9. Computer Vision
  • 9.1. Image Augmentation
  • 9.2. Fine Tuning
  • 9.3. Object Detection and Bounding Box
  • 9.4. Anchor Boxes
  • 9.5. Multiscale Object Detection
  • 9.6. Object Detection Data Set (Pikachu)
  • 9.7. Single Shot Multibox Detection (SSD)
  • 9.8. Region-based CNNs (R-CNNs)
  • 9.9. Semantic Segmentation and Data Sets
  • 9.10. Fully Convolutional Networks (FCN)
  • 9.11. Neural Style Transfer
  • 9.12. Image Classification (CIFAR-10) on Kaggle
  • 9.13. Dog Breed Identification (ImageNet Dogs) on Kaggle
• 10. Natural Language Processing
  • 10.1. Word Embedding (word2vec)
  • 10.2. Approximate Training
  • 10.3. Implementation of Word2vec
  • 10.4. Subword embedding (fastText)
  • 10.5. Word Embedding with Global Vectors (GloVe)
  • 10.6. Finding Synonyms and Analogies
  • 10.7. Text Sentiment Classification: Using Recurrent Neural Networks
  • 10.8. Text Sentiment Classification: Using Convolutional Neural Networks (textCNN)
  • 10.9. Encoder-Decoder (seq2seq)
  • 10.10. Beam Search
  • 10.11. Attention Mechanism
  • 10.12. Machine Translation
• 11. Appendix
  • 11.1. List of Main Symbols
  • 11.2. Mathematical Basis
  • 11.3. Using Jupyter Notebook
  • 11.4. Using AWS to Run Code
  • 11.5. GPU Purchase Guide
  • 11.6. How to Contribute to This Book
  • 11.7. Gluonbook Package Index

Next

Preface