A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.
Traditional Machine Learning Title Dataset Description Notebooks Perceptron 2D toy data TBD Logistic Regression 2D toy data TBD Softmax Regression (Multinomial Logistic Regression) MNIST TBD Softmax Regression with MLxtend's plot_decision_regions on Iris Iris TBD
Title Dataset Description Notebooks Multilayer Perceptron MNIST TBD Multilayer Perceptron with Dropout MNIST TBD Multilayer Perceptron with Batch Normalization MNIST TBD Multilayer Perceptron with Backpropagation from Scratch MNIST TBD
Convolutional Neural Networks Title Dataset Description Notebooks Convolutional Neural Network TBD TBD CNN with He Initialization TBD TBD
Title Dataset Description Notebooks Replacing Fully-Connected by Equivalent Convolutional Layers TBD TBD
Title Dataset Description Notebooks AlexNet Trained on CIFAR-10 TBD TBD AlexNet with Grouped Convolutions Trained on CIFAR-10 TBD TBD
Title Description Daset Notebooks DenseNet-121 Digit Classifier Trained on MNIST TBD TBD DenseNet-121 Image Classifier Trained on CIFAR-10 TBD TBD
Title Dataset Description Notebooks "All Convolutionl Net" -- A Fully Convolutional Neural Network TBD TBD
Title Dataset Description Notebooks LeNet-5 on MNIST TBD TBD LeNet-5 on CIFAR-10 TBD TBD LeNet-5 on QuickDraw TBD TBD
Title Dataset Description Notebooks MobileNet-v2 on Cifar-10 TBD TBD MobileNet-v3 small on Cifar-10 TBD TBD MobileNet-v3 large on Cifar-10 TBD TBD MobileNet-v3 large on MNIST via Embetter TBD TBD
Title Dataset Description Notebooks Network in Network Trained on CIFAR-10 TBD TBD
Title Dataset Description Notebooks Convolutional Neural Network VGG-16 Trained on CIFAR-10 TBD TBD VGG-16 Smile Classifier CelebA TBD VGG-16 Dogs vs Cats Classifier TBD TBD Convolutional Neural Network VGG-19 TBD TBD
Title Dataset Description Notebooks ResNet and Residual Blocks MNIST TBD ResNet-18 Digit Classifier MNIST TBD ResNet-18 Gender Classifier CelebA TBD ResNet-34 Digit Classifier MNIST TBD ResNet-34 Object Classifier QuickDraw TBD ResNet-34 Gender Classifier CelebA TBD ResNet-50 Digit Classifier MNIST TBD ResNet-50 Gender Classifier CelebA TBD ResNet-101 Gender Classifier CelebA TBD ResNet-101 CIFAR-10 TBD ResNet-152 Gender Classifier CelebA TBD
Title Dataset Description Notebooks Multilabel DistilBERT Jigsaw Toxic Comment Challenge DistilBERT classifier fine-tuning DistilBERT as feature extractor IMDB movie review DistilBERT classifier with sklearn random forest and logistic regression DistilBERT as feature extractor using embetter IMDB movie review DistilBERT classifier with sklearn random forest and logistic regression using the scikit-learn embetter library Fine-tune DistilBERT I IMDB movie review Fine-tune only the last 2 layers of DistilBERT classifier Fine-tune DistilBERT II IMDB movie review Fine-tune the whole DistilBERT classifier
Ordinal Regression and Deep Learning Please note that the following notebooks below provide reference implementations to use the respective methods. They are not performance benchmarks.
Title Dataset Description Notebooks BatchNorm before and after Activation for Network-in-Network CIFAR-10 Classifier TBD TBD Filter Response Normalization for Network-in-Network CIFAR-10 Classifier TBD TBD
Title Dataset Description Notebooks Siamese Network with Multilayer Perceptrons TBD TBD
Fully-connected Autoencoders Title Dataset Description Notebooks Autoencoder (MNIST) TBD TBD Autoencoder (MNIST) + Scikit-Learn Random Forest Classifier TBD TBD
Convolutional Autoencoders Title Dataset Description Notebooks Convolutional Autoencoder with Deconvolutions / Transposed Convolutions TBD TBD Convolutional Autoencoder with Deconvolutions and Continuous Jaccard Distance TBD TBD Convolutional Autoencoder with Deconvolutions (without pooling operations) TBD TBD Convolutional Autoencoder with Nearest-neighbor Interpolation TBD TBD Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on CelebA TBD TBD Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on Quickdraw TBD TBD
Title Dataset Description Notebooks Variational Autoencoder TBD TBD Convolutional Variational Autoencoder TBD TBD
Conditional Variational Autoencoders Title Dataset Description Notebooks Conditional Variational Autoencoder (with labels in reconstruction loss) TBD TBD Conditional Variational Autoencoder (without labels in reconstruction loss) TBD TBD Convolutional Conditional Variational Autoencoder (with labels in reconstruction loss) TBD TBD Convolutional Conditional Variational Autoencoder (without labels in reconstruction loss) TBD TBD
Generative Adversarial Networks (GANs) Title Dataset Description Notebooks Fully Connected GAN on MNIST TBD TBD Fully Connected Wasserstein GAN on MNIST TBD TBD Convolutional GAN on MNIST TBD TBD Convolutional GAN on MNIST with Label Smoothing TBD TBD Convolutional Wasserstein GAN on MNIST TBD TBD Deep Convolutional GAN (DCGAN) on Cats and Dogs Images TBD TBD Deep Convolutional GAN (DCGAN) on CelebA Face Images TBD TBD
Graph Neural Networks (GNNs) Title Dataset Description Notebooks Most Basic Graph Neural Network with Gaussian Filter on MNIST TBD TBD Basic Graph Neural Network with Edge Prediction on MNIST TBD TBD Basic Graph Neural Network with Spectral Graph Convolution on MNIST TBD TBD
Recurrent Neural Networks (RNNs) Many-to-one: Sentiment Analysis / Classification Title Dataset Description Notebooks A simple single-layer RNN (IMDB) TBD TBD A simple single-layer RNN with packed sequences to ignore padding characters (IMDB) TBD TBD RNN with LSTM cells (IMDB) TBD TBD RNN with LSTM cells (IMDB) and pre-trained GloVe word vectors TBD TBD RNN with LSTM cells and Own Dataset in CSV Format (IMDB) TBD TBD RNN with GRU cells (IMDB) TBD TBD Multilayer bi-directional RNN (IMDB) TBD TBD Bidirectional Multi-layer RNN with LSTM with Own Dataset in CSV Format (AG News) TBD TBD
Many-to-Many / Sequence-to-Sequence Title Dataset Description Notebooks A simple character RNN to generate new text (Charles Dickens) TBD TBD
Title Dataset Description Notebooks Baseline CNN MNIST A simple baseline with traditional train/validation/test splits K-fold with pl_cross MNIST A 5-fold cross-validation run using the pl_cross library
Title Dataset Description Notebooks AutoAugment & TrivialAugment for Image Data CIFAR-10 Trains a ResNet-18 using AutoAugment and TrivialAugment
Title Dataset Description Notebooks Cyclical Learning Rate TBD TBD Annealing with Increasing the Batch Size (w. CIFAR-10 & AlexNet) TBD TBD Gradient Clipping (w. MLP on MNIST) TBD TBD
Title Dataset Description Notebooks Transfer Learning Example (VGG16 pre-trained on ImageNet for Cifar-10) TBD TBD
Visualization and Interpretation Title Dataset Description Notebooks Vanilla Loss Gradient (wrt Inputs) Visualization (Based on a VGG16 Convolutional Neural Network for Kaggle's Cats and Dogs Images) TBD TBD Guided Backpropagation (Based on a VGG16 Convolutional Neural Network for Kaggle's Cats and Dogs Images) TBD TBD
PyTorch Workflows and Mechanics PyTorch Lightning Examples Title Dataset Description Notebooks MLP in Lightning with TensorBoard -- continue training the last model TBD TBD MLP in Lightning with TensorBoard -- checkpointing best model TBD TBD
Title Dataset Description Notebooks Custom Data Loader Example for PNG Files TBD TBD Using PyTorch Dataset Loading Utilities for Custom Datasets -- CSV files converted to HDF5 TBD TBD Using PyTorch Dataset Loading Utilities for Custom Datasets -- Face Images from CelebA TBD TBD Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from Quickdraw TBD TBD Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from the Street View House Number (SVHN) Dataset TBD TBD Using PyTorch Dataset Loading Utilities for Custom Datasets -- Asian Face Dataset (AFAD) TBD TBD Using PyTorch Dataset Loading Utilities for Custom Datasets -- Dating Historical Color Images TBD TBD Using PyTorch Dataset Loading Utilities for Custom Datasets -- Fashion MNIST TBD TBD
Training and Preprocessing Title Dataset Description Notebooks PyTorch DataLoader State and Nested Iterations Toy Explains DataLoader behavior when in nested functions Generating Validation Set Splits TBD TBD Dataloading with Pinned Memory TBD TBD Standardizing Images TBD TBD Image Transformation Examples TBD TBD Char-RNN with Own Text File TBD TBD Sentiment Classification RNN with Own CSV File TBD TBD
Improving Memory Efficiency Title Dataset Description Notebooks Gradient Checkpointing Demo (Network-in-Network trained on CIFAR-10) TBD TBD
Title Description Notebooks Using Multiple GPUs with DataParallel -- VGG-16 Gender Classifier on CelebA TBD Distribute a Model Across Multiple GPUs with Pipeline Parallelism (VGG-16 Example) TBD
Title Dataset Description Notebooks PyTorch with and without Deterministic Behavior -- Runtime Benchmark TBD TBD Sequential API and hooks TBD TBD Weight Sharing Within a Layer TBD TBD Plotting Live Training Performance in Jupyter Notebooks with just Matplotlib TBD TBD
Title Dataset Description Notebooks Getting Gradients of an Intermediate Variable in PyTorch TBD TBD
TensorFlow Workflows and Mechanics Title Description Notebooks Chunking an Image Dataset for Minibatch Training using NumPy NPZ Archives TBD Storing an Image Dataset for Minibatch Training using HDF5 TBD Using Input Pipelines to Read Data from TFRecords Files TBD Using Queue Runners to Feed Images Directly from Disk TBD Using TensorFlow's Dataset API TBD
Training and Preprocessing Title Dataset Description Notebooks Saving and Loading Trained Models -- from TensorFlow Checkpoint Files and NumPy NPZ Archives TBD TBD
Title Description Notebooks TorchMetrics How do we use it, and what's the difference between .update() and .forward()?
