Merge pull request #4260 from Shantnu-singh/main

Added AdaGrad in Deep Learning Optimizers

Author: ajay-dhangar

docs/Deep Learning/Optimizers in Deep Learning/AdaGard.md

@@ -0,0 +1,109 @@
+# Add AdaGrad in Deep Learning Optimizers
+
+This section contains an explanation and implementation of the AdaGrad optimization algorithm used in deep learning. AdaGrad is known for its ability to adapt the learning rate based on the frequency of updates for each parameter.
+
+## Table of Contents
+- [Introduction](#introduction)
+- [Mathematical Explanation](#mathematical-explanation)
+  - [AdaGrad in Gradient Descent](#adagrad-in-gradient-descent)
+  - [Update Rule](#update-rule)
+- [Implementation in Keras](#implementation-in-keras)
+- [Usage](#usage)
+- [Results](#results)
+- [Advantages of AdaGrad](#advantages-of-adagrad)
+- [Limitations of AdaGrad](#limitations-of-adagrad)
+- [What Next](#what-next)
+
+## Introduction
+
+AdaGrad (Adaptive Gradient Algorithm) is an optimization method that adjusts the learning rate for each parameter individually based on the accumulated squared gradients. This allows the algorithm to perform well in scenarios where sparse features are involved, as it effectively scales down the learning rate for frequently updated parameters.
+
+## Mathematical Explanation
+
+### AdaGrad in Gradient Descent
+
+AdaGrad modifies the standard gradient descent algorithm by adjusting the learning rate for each parameter based on the sum of the squares of the past gradients.
+
+### Update Rule
+
+The update rule for AdaGrad is as follows:
+
+1. Accumulate the squared gradients:
+
+      $$
+      G_t = G_{t-1} + g_t^2
+      $$
+
+2. Update the parameters:
+
+    
+$$η = \theta_{t-1} - \frac{\eta}{\sqrt{G_t} + \epsilon} \cdot g_t$$
+
+where:
+- $G_t$ is the accumulated sum of squares of gradients up to time step $t$
+- $g_t$ is the gradient at time step $t$
+- $\eta$ is the learning rate
+- $\epsilon$ is a small constant to prevent division by zero
+
+## Implementation in Keras
+
+Here is a simple implementation of the AdaGrad optimizer using Keras:
+
+```python
+import numpy as np
+from keras.models import Sequential
+from keras.layers import Dense
+from keras.optimizers import Adagrad
+
+# Generate dummy data
+X_train = np.random.rand(1000, 20)
+y_train = np.random.randint(2, size=(1000, 1))
+
+# Define a simple model
+model = Sequential()
+model.add(Dense(64, activation='relu', input_dim=20))
+model.add(Dense(1, activation='sigmoid'))
+
+# Compile the model with AdaGrad optimizer
+optimizer = Adagrad(learning_rate=0.01)
+model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])
+
+# Train the model
+model.fit(X_train, y_train, epochs=50, batch_size=32)
+```
+
+In this example:
+- We generate some dummy data for training.
+- We define a simple neural network model with one hidden layer.
+- We compile the model using the AdaGrad optimizer with a learning rate of 0.01.
+- We train the model for 50 epochs with a batch size of 32.
+
+## Usage
+
+To use this implementation, ensure you have the required dependencies installed:
+
+```bash
+pip install numpy keras
+```
+
+Then, you can run the provided script to train a model using the AdaGrad optimizer.
+
+## Results
+
+The results of the training process, including the loss and accuracy, will be displayed after each epoch. You can adjust the learning rate and other hyperparameters to see how they affect the training process.
+
+## Advantages of AdaGrad
+
+1. **Adaptive Learning Rates**: AdaGrad adapts the learning rate for each parameter, making it effective for dealing with sparse data and features.
+2. **No Need for Manual Learning Rate Decay**: Since AdaGrad automatically decays the learning rate, it eliminates the need to manually set learning rate schedules.
+3. **Good for Sparse Data**: AdaGrad performs well on problems with sparse features, such as natural language processing and computer vision tasks.
+
+## Limitations of AdaGrad
+
+1. **Aggressive Learning Rate Decay**: The accumulated gradient sum can grow very large, causing the learning rate to become very small and eventually stopping the learning process.
+2. **Not Suitable for Non-Sparse Data**: For dense data, AdaGrad’s aggressive learning rate decay can slow down convergence, making it less effective.
+3. **Memory Usage**: AdaGrad requires storing the sum of squared gradients for each parameter, which can be memory-intensive for large models.
+
+## What Next
+
+To address these issues, various optimization algorithms have been developed, such as Adam,  which incorporate techniques. Which we'll see in next section .