Merge branch 'CodeHarborHub:main' into master2

Author: SanskarSinghiit

blog/authors.yml

@@ -76,3 +76,10 @@ DharshiBalasubramaniyam:
   title: Full Stack Developer
   url: "https://github.com/DharshiBalasubramaniyam"
   image_url: https://avatars.githubusercontent.com/u/139672976?s=400&v=4
+
+akshitha-chiluka:
+  name: Akshitha Chiluka
+  title: Software Engineering Undergraduate
+  url: https://github.com/AKSHITHA-CHILUKA
+  image_url: https://avatars.githubusercontent.com/u/120377576?v=4
+

docs/Deep Learning/Optimizers in Deep Learning/AdaGard.md

@@ -0,0 +1,109 @@
+# Add AdaGrad in Deep Learning Optimizers
+
+This section contains an explanation and implementation of the AdaGrad optimization algorithm used in deep learning. AdaGrad is known for its ability to adapt the learning rate based on the frequency of updates for each parameter.
+
+## Table of Contents
+- [Introduction](#introduction)
+- [Mathematical Explanation](#mathematical-explanation)
+  - [AdaGrad in Gradient Descent](#adagrad-in-gradient-descent)
+  - [Update Rule](#update-rule)
+- [Implementation in Keras](#implementation-in-keras)
+- [Usage](#usage)
+- [Results](#results)
+- [Advantages of AdaGrad](#advantages-of-adagrad)
+- [Limitations of AdaGrad](#limitations-of-adagrad)
+- [What Next](#what-next)
+
+## Introduction
+
+AdaGrad (Adaptive Gradient Algorithm) is an optimization method that adjusts the learning rate for each parameter individually based on the accumulated squared gradients. This allows the algorithm to perform well in scenarios where sparse features are involved, as it effectively scales down the learning rate for frequently updated parameters.
+
+## Mathematical Explanation
+
+### AdaGrad in Gradient Descent
+
+AdaGrad modifies the standard gradient descent algorithm by adjusting the learning rate for each parameter based on the sum of the squares of the past gradients.
+
+### Update Rule
+
+The update rule for AdaGrad is as follows:
+
+1. Accumulate the squared gradients:
+
+      $$
+      G_t = G_{t-1} + g_t^2
+      $$
+
+2. Update the parameters:
+
+    
+$$η = \theta_{t-1} - \frac{\eta}{\sqrt{G_t} + \epsilon} \cdot g_t$$
+
+where:
+- $G_t$ is the accumulated sum of squares of gradients up to time step $t$
+- $g_t$ is the gradient at time step $t$
+- $\eta$ is the learning rate
+- $\epsilon$ is a small constant to prevent division by zero
+
+## Implementation in Keras
+
+Here is a simple implementation of the AdaGrad optimizer using Keras:
+
+```python
+import numpy as np
+from keras.models import Sequential
+from keras.layers import Dense
+from keras.optimizers import Adagrad
+
+# Generate dummy data
+X_train = np.random.rand(1000, 20)
+y_train = np.random.randint(2, size=(1000, 1))
+
+# Define a simple model
+model = Sequential()
+model.add(Dense(64, activation='relu', input_dim=20))
+model.add(Dense(1, activation='sigmoid'))
+
+# Compile the model with AdaGrad optimizer
+optimizer = Adagrad(learning_rate=0.01)
+model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])
+
+# Train the model
+model.fit(X_train, y_train, epochs=50, batch_size=32)
+```
+
+In this example:
+- We generate some dummy data for training.
+- We define a simple neural network model with one hidden layer.
+- We compile the model using the AdaGrad optimizer with a learning rate of 0.01.
+- We train the model for 50 epochs with a batch size of 32.
+
+## Usage
+
+To use this implementation, ensure you have the required dependencies installed:
+
+```bash
+pip install numpy keras
+```
+
+Then, you can run the provided script to train a model using the AdaGrad optimizer.
+
+## Results
+
+The results of the training process, including the loss and accuracy, will be displayed after each epoch. You can adjust the learning rate and other hyperparameters to see how they affect the training process.
+
+## Advantages of AdaGrad
+
+1. **Adaptive Learning Rates**: AdaGrad adapts the learning rate for each parameter, making it effective for dealing with sparse data and features.
+2. **No Need for Manual Learning Rate Decay**: Since AdaGrad automatically decays the learning rate, it eliminates the need to manually set learning rate schedules.
+3. **Good for Sparse Data**: AdaGrad performs well on problems with sparse features, such as natural language processing and computer vision tasks.
+
+## Limitations of AdaGrad
+
+1. **Aggressive Learning Rate Decay**: The accumulated gradient sum can grow very large, causing the learning rate to become very small and eventually stopping the learning process.
+2. **Not Suitable for Non-Sparse Data**: For dense data, AdaGrad’s aggressive learning rate decay can slow down convergence, making it less effective.
+3. **Memory Usage**: AdaGrad requires storing the sum of squared gradients for each parameter, which can be memory-intensive for large models.
+
+## What Next
+
+To address these issues, various optimization algorithms have been developed, such as Adam,  which incorporate techniques. Which we'll see in next section .

docs/Deep Learning/Optimizers in Deep Learning/Adam.md

@@ -0,0 +1,116 @@
+# Add Adam in Deep Learning Optimizers
+
+This Section contains an explanation and implementation of the Adam optimization algorithm used in deep learning. Adam (Adaptive Moment Estimation) is a popular optimizer that combines the benefits of two other widely used methods: AdaGrad and RMSProp.
+
+## Table of Contents
+- [Introduction](#introduction)
+- [Mathematical Explanation](#mathematical-explanation)
+  - [Adam in Gradient Descent](#adam-in-gradient-descent)
+  - [Update Rule](#update-rule)
+- [Implementation in Keras](#implementation-in-keras)
+- [Results](#results)
+- [Advantages of Adam](#advantages-of-adam)
+- [Limitations of Adam](#limitations-of-adam)
+
+
+## Introduction
+
+Adam is an optimization algorithm that computes adaptive learning rates for each parameter. It combines the advantages of the AdaGrad and RMSProp algorithms by using estimates of the first and second moments of the gradients. Adam is widely used in deep learning due to its efficiency and effectiveness.
+
+## Mathematical Explanation
+
+### Adam in Gradient Descent
+
+Adam optimizes the stochastic gradient descent by calculating individual adaptive learning rates for each parameter based on the first and second moments of the gradients.
+
+### Update Rule
+
+The update rule for Adam is as follows:
+
+1. Compute the first moment estimate (mean of gradients):
+
+$$
+m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t
+$$
+
+2. Compute the second moment estimate (uncentered variance of gradients):
+
+$$
+v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2
+$$
+
+3. Correct the bias for the first moment estimate:
+
+$$
+\hat{m}_t = \frac{m_t}{1 - \beta_1^t}
+$$
+
+4. Correct the bias for the second moment estimate:
+
+$$
+\hat{v}_t = \frac{v_t}{1 - \beta_2^t}
+$$
+
+5. Update the parameters:
+
+$$
+\theta_t = \theta_{t-1} - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t
+$$
+
+where:
+- $\theta$ are the model parameters
+- $\eta$ is the learning rate
+- $\beta_1$ and $\beta_2$ are the exponential decay rates for the moment estimates
+- $\epsilon$ is a small constant to prevent division by zero
+- $g_t$ is the gradient at time step $t$
+
+## Implementation in Keras
+
+Simple implementation of the Adam optimizer using Keras:
+
+```python
+import numpy as np
+from keras.models import Sequential
+from keras.layers import Dense
+from keras.optimizers import Adam
+
+# Generate data
+X_train = np.random.rand(1000, 20)
+y_train = np.random.randint(2, size=(1000, 1))
+
+# Define a model
+model = Sequential()
+model.add(Dense(64, activation='relu', input_dim=20))
+model.add(Dense(1, activation='sigmoid'))
+
+# Compile the model with Adam optimizer
+optimizer = Adam(learning_rate=0.001)
+model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])
+
+# Train the model
+model.fit(X_train, y_train, epochs=50, batch_size=32)
+```
+
+In this example:
+- We generate some dummy data for training.
+- We define a simple neural network model with one hidden layer.
+- We compile the model using the Adam optimizer with a learning rate of 0.001.
+- We train the model for 50 epochs with a batch size of 32.
+
+
+## Results
+
+The results of the training process, including the loss and accuracy, will be displayed after each epoch. You can adjust the learning rate and other hyperparameters to see how they affect the training process.
+
+## Advantages of Adam
+
+1. **Adaptive Learning Rates**: Adam computes adaptive learning rates for each parameter, which helps in faster convergence.
+2. **Momentum**: Adam includes momentum, which helps in smoothing the optimization path and avoiding local minima.
+3. **Bias Correction**: Adam includes bias correction, improving convergence in the early stages of training.
+4. **Robustness**: Adam works well in practice for a wide range of problems, including those with noisy gradients or sparse data.
+
+## Limitations of Adam
+
+1. **Hyperparameter Sensitivity**: The performance of Adam is sensitive to the choice of hyperparameters ($\beta_1$, $\beta_2$, $\eta$), which may require careful tuning.
+2. **Memory Usage**: Adam requires additional memory to store the first and second moments, which can be significant for large models.
+3. **Generalization**: Models trained with Adam might not generalize as well as those trained with simpler optimizers like SGD in certain cases.