Harsh RB

4.1 Deep Learning

4 min read

Deep Learning Blooms Taxonomy Questions Deep Learning Assignment

UNIT I: Machine Learning Basics & Deep Feedforward Networks

Machine Learning Fundamentals

  • Learning Algorithms – Types and Frameworks
  • Model Capacity – Underfitting vs. Overfitting
  • Hyperparameters and Validation Sets – Train/Validation/Test Split, Cross-Validation
  • Estimators – Point Estimation, Interval Estimation
  • Bias-Variance Tradeoff – Decomposition of Generalization Error
  • Maximum Likelihood Estimation (MLE) – Principles and Applications
  • Bayesian Statistics – Priors, Posteriors, MAP Estimation
  • Supervised Learning Algorithms – Regression, Classification (k-NN, SVM, Decision Trees)
  • Unsupervised Learning Algorithms – Clustering (k-Means), Dimensionality Reduction (PCA)
  • Stochastic Gradient Descent (SGD) – Mini-batch, Convergence, Momentum
  • Building a Machine Learning Algorithm – Pipeline: Data → Features → Model → Evaluation
  • Challenges Motivating Deep Learning – Curse of Dimensionality, Feature Engineering Bottlenecks

Deep Feedforward Networks (Multilayer Perceptrons)

  • Learning XOR – Limitations of Linear Models, Need for Hidden Layers
  • Gradient-Based Learning – Chain Rule, Computational Graphs
  • Hidden Units – Activation Functions: ReLU, Sigmoid, Tanh, Softmax
  • Architecture Design – Depth vs. Width, Universal Approximation Theorem
  • Backpropagation and Other Differentiation Algorithms
    • Automatic Differentiation
    • Forward vs. Reverse Mode
    • Jacobian and Hessian Matrices

UNIT II: Regularization & Optimization for Deep Learning

Regularization Techniques

  • Parameter Norm Penalties – L1 (Lasso), L2 (Ridge), Elastic Net
  • Norm Penalties as Constrained Optimization – Lagrange Multipliers Interpretation
  • Regularization and Under-Constrained Problems – Ill-posed Problems, Tikhonov Regularization
  • Dataset Augmentation – Image Flips, Rotations, Noise Injection
  • Noise Robustness – Input/Weight Noise, Denoising Autoencoders
  • Semi-Supervised Learning – Leveraging Unlabeled Data
  • Multi-Task Learning – Shared Representations, Auxiliary Tasks
  • Early Stopping – Validation-Based Halting Criterion
  • Parameter Tying and Parameter Sharing – CNNs, RNNs, Weight Reuse
  • Sparse Representations – L1 Regularization, Sparse Coding
  • Bagging and Other Ensemble Methods – Random Forests, Boosting (AdaBoost, XGBoost)
  • Dropout – Training-Time Neuron Deactivation, Inference Scaling
  • Adversarial Training – Robustness to Perturbations, FGSM
  • Tangent Distance, Tangent Prop, and Manifold Tangent Classifier – Invariance Learning

Optimization for Training Deep Models

  • Learning vs Pure Optimization – Generalization ≠ Minimizing Training Loss
  • Challenges in Neural Network Optimization
    • Poor Conditioning, Local Minima, Saddle Points, Vanishing/Exploding Gradients
  • Basic Algorithms – SGD, Momentum, Nesterov Momentum
  • Parameter Initialization Strategies
    • Xavier/Glorot, He Initialization, Orthogonal Initialization
  • Algorithms with Adaptive Learning Rates
    • AdaGrad, RMSProp, Adam, Nadam

UNIT III: Convolutional Networks

Fundamentals of CNNs

  • The Convolution Operation – Kernels, Feature Maps, Stride, Padding
  • Motivation – Translation Invariance, Parameter Sharing, Hierarchical Features
  • Pooling – Max Pooling, Average Pooling, Global Pooling, Invariance Effects
  • Convolution and Pooling as an Infinitely Strong Prior – Spatial Locality, Stationarity
  • Variants of the Basic Convolution Function
    • Dilated (Atrous) Convolutions
    • Depthwise Separable Convolutions
    • Transposed Convolutions (Deconvolutions)

Advanced Topics

  • Structured Outputs – Semantic Segmentation, Object Detection Architectures
  • Data Types – Images, Volumes (3D CNNs), Sequences (1D CNNs)
  • Efficient Convolution Algorithms – FFT-based, Winograd, Im2Col
  • Random or Unsupervised Features – Random CNNs, Self-Supervised Pretext Tasks

UNIT IV: Recurrent and Recursive Networks

Sequence Modeling

  • Unfolding Computational Graphs – Time-Unrolled Representations
  • Recurrent Neural Networks (RNNs) – Vanilla RNN, Hidden State Dynamics
  • Bidirectional RNNs – Forward + Backward Context Integration
  • Encoder-Decoder Sequence-to-Sequence Architectures – Machine Translation, Text Summarization
  • Deep Recurrent Networks – Stacked RNNs, Residual Connections
  • Recursive Neural Networks – Tree-Structured Inputs (e.g., Parse Trees)

Long-Term Dependency Challenges

  • The Challenge of Long-Term Dependencies – Gradient Vanishing in Vanilla RNNs
  • Echo State Networks (ESNs) – Fixed Recurrent Weights, Readout Training
  • Leaky Units and Other Strategies for Multiple Time Scales – Time Constants, Hierarchical RNNs
  • The Long Short-Term Memory (LSTM) and Other Gated RNNs
    • LSTM: Input, Forget, Output Gates
    • GRU (Gated Recurrent Unit): Simplified Gating
  • Optimization for Long-Term Dependencies – Gradient Clipping, Curriculum Learning
  • Explicit Memory – Neural Turing Machines, Memory Networks, Differentiable Neural Computers

UNIT V: Practical Methodology & Applications

Practical Methodology

  • Performance Metrics
    • Classification: Accuracy, Precision, Recall, F1, AUC-ROC
    • Regression: MSE, MAE, R²
    • Ranking: NDCG, MAP
  • Default Baseline Models – Logistic Regression, Random Forest, Linear SVM
  • Determining Whether to Gather More Data – Learning Curves, Data Saturation
  • Selecting Hyperparameters – Grid Search, Random Search, Bayesian Optimization
  • Debugging Strategies – Gradient Checks, Activation/Gradient Distributions, Overfitting Diagnosis
  • Example: Multi-Digit Number Recognition – End-to-End System Design, Error Analysis

Applications of Deep Learning

  • Large-Scale Deep Learning – Distributed Training, Model Parallelism, Data Parallelism
  • Computer Vision
    • Image Classification (ResNet, EfficientNet)
    • Object Detection (YOLO, Faster R-CNN)
    • Segmentation (U-Net, Mask R-CNN)
  • Speech Recognition
    • End-to-End Models (DeepSpeech, Wav2Vec)
    • CTC Loss, Attention Models
  • Natural Language Processing (NLP)
    • Transformers (BERT, GPT)
    • Named Entity Recognition, Machine Translation, Question Answering
  • Other Applications
    • Recommender Systems
    • Generative Models (GANs, VAEs, Diffusion Models)
    • Reinforcement Learning (Deep Q-Networks, Policy Gradients)
    • Healthcare, Finance, Robotics, Autonomous Systems

Graph View