lectures/lecture-12/lecture-12.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lecture 12:  Adversarial attacks and training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Previous lecture: Few-shot learning\n",
    "- What is few-shot learning\n",
    "- Meta learning \n",
    "- Main models and approaches (ProtoNets, SiameseNetworks, MAML)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Current lecture: Adversarial attacks and training\n",
    "- Adversarial attacks\n",
    "- Adversarial training\n",
    "- Robustness of DL models (using randomized smoothing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Adversarial attacks: how they were developed.\n",
    "\n",
    "Ian Goodfellow looked at the multiclass classifier, given by the model $p(c \\vert x),$ (probability of the class condition at the input).\n",
    "\n",
    "\n",
    "Suppose we want to move smoothly from one class (cat) to another class (dog).\n",
    "\n",
    "So, you can just maximize the probability of the image of being a dog, moving in the direction of the gradient\n",
    "\n",
    "$$\\nabla \\log p(c_2 \\vert x), $$\n",
    "\n",
    "\n",
    "But they found an extremely suprising result: even **small modification** leads to large missclassification.\n",
    "\n",
    "\n",
    "<img src='fgsm_panda_image.png'>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Problem statement\n",
    "\n",
    "**White-box** small-norm attack on a deep neural network model is the following optimization task.\n",
    "\n",
    "Let $p(x) \\in \\mathbb{R}^c$ be the classifier with $c$ classes. Then the minimal norm attack is defined as the \n",
    "\n",
    "solution to the following optimization problem:\n",
    "\n",
    "\n",
    "$$\\min \\Vert \\varepsilon \\Vert\\, \\mbox{s.t. } \\arg \\max p(x + \\varepsilon) \\ne \\arg \\max p(x)$$\n",
    "\n",
    "I.e. the minimal-norm perturbation that changes the class.\n",
    "\n",
    "Existing models are notoriously unstable!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Different types of attacks\n",
    "\n",
    "There are different types of adversarial attacks (whitebox, greybox, blackbox).\n",
    "\n",
    "The can be universal or not. They can be one-shot and iterative.\n",
    "\n",
    "<img src='Taxonomy-of-adversarial-attacks-classified-according-to-the-adversarys-objective.ppm'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Whitebox attacks\n",
    "\n",
    "In **whitebox attacks** we know the weights of the model (which is not always the case).\n",
    "\n",
    "The most well-known attacks are:\n",
    "\n",
    "1. **Fast Gradient Sign Method (FGSM)**: This attack involves adding a small perturbation to the input image by computing the gradient of the loss function with respect to the input.\n",
    "\n",
    "2. **Projected Gradient Descent (PGD)**: This attack is an iterative version of FGSM, where the perturbation is added in small steps until a certain threshold is reached.\n",
    "\n",
    "3. **DeepFool**: This attack finds the minimum distance from the input image to the decision boundary of the classifier and then adds a small perturbation in that direction.\n",
    "\n",
    "4. **Carlini-Wagner (CW)** Attack: This attack is designed to minimize the distance between the original image and the adversarial image while also ensuring that the adversarial image is misclassified.\n",
    "\n",
    "5. **Universal Adversarial Perturbation (UAP)**: This attack generates a single perturbation that can be added to any input image to cause misclassification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Fast sign gradient method\n",
    "\n",
    "Fast Gradient Sign Method (FGSM) is a popular whitebox adversarial attack that involves adding a small perturbation to the input image by computing the gradient of the loss function with respect to the input. Mathematically, the FGSM attack can be expressed as:\n",
    "\n",
    "$$\n",
    "\\hat{x} = x + \\epsilon \\cdot \\mathrm{sign}(\\nabla_{x} p(y \\vert x))\n",
    "$$\n",
    "\n",
    "where $\\text{adv}{x}$ is the adversarial image, $x$ is the original image, $\\epsilon$ is the magnitude of the perturbation, $p(x, y)$ is the loss function, input image $x$, and true label $y$, and $\\nabla{x}$ is the gradient operator with respect to $x$. The sign function is used to ensure that the perturbation is added in the direction that maximizes the loss function.\n",
    "\n",
    "\n",
    "The derivation of FGSM is done by linearizing the loss function and solving the norm-constrained problem! \n",
    "\n",
    "It does not necessary gives the optimal solution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## PGD attack\n",
    "\n",
    "Projected Gradient Descent (PGD) is a more powerful whitebox adversarial attack that iteratively applies the FGSM attack with a small step size and then projects the resulting perturbed image back onto the **attack set**.\n",
    "\n",
    "Mathematically, the PGD attack can be expressed as:\n",
    "\n",
    "$$\n",
    "x_0 = x, \\quad x_{t+1} = \\text{Clip}(x_t + \\alpha \\cdot \\text{sign}(\\nabla_{x} p(y \\vert x_t))\n",
    "$$\n",
    "\n",
    "PGD is more effective, but computationally more expensive as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Carlini-Wagner attack\n",
    "\n",
    "[The Carlini-Wagner (CW)](https://ieeexplore.ieee.org/abstract/document/7958570?casa_token=Ih-Qng6pAOMAAAAA:SUZxSbOutzvlADG2k01bQDeIYA5cwIb86IhwbaAAHajbwIIaStScn6u0h1eJ516Jov331KKqhQ8ixEM) attack is a state-of-the-art whitebox adversarial attack that is designed to be more effective than PGD against defenses that use gradient masking or gradient obfuscation techniques. The CW attack formulates the problem of finding an adversarial perturbation as an optimization problem that minimizes the distance between the original image and the perturbed image, subject to a constraint on the classification loss.\n",
    "\n",
    "Mathematically, the CW attack can be expressed as:\n",
    "\n",
    "$$\n",
    "\\min_{\\delta} \\Vert \\delta \\Vert_{p} + c \\cdot f(x+\\delta, y)\n",
    "$$\n",
    "\n",
    "where $\\delta$ is the adversarial perturbation, $\\Vert \\cdot \\Vert{p}$ is a norm function, $c$ is a hyperparameter that controls the trade-off between the distance and the loss term, and $f(\\cdot, y)$ is a function that measures the classification loss of the perturbed image with respect to the target class $y$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## DeepFool\n",
    "\n",
    "[DeepFool attack](https://openaccess.thecvf.com/content_cvpr_2016/html/Moosavi-Dezfooli_DeepFool_A_Simple_CVPR_2016_paper.html) uses the idea of **linearized decision boundary**. \n",
    "\n",
    "\n",
    "Suppose we have a binary classifier, and the separation function is given as $f(x) = 0$. \n",
    "\n",
    "$f(x) > 0$ corresponds to one class, $f(x) < 0$ to another.\n",
    "\n",
    "The best attack would be given by the closest point to the boundary.\n",
    "\n",
    "To find such point, we linearize the function as\n",
    "\n",
    "$$f(x) \\approx f(x_0) + f'(x), x - x_0.$$\n",
    "\n",
    "Finding the closest point to the boundary can be done **analytically**. \n",
    "\n",
    "Then we can update iteratively\n",
    "\n",
    "**Initialize:** $x := x_0$\n",
    "\n",
    "**Iteration:** $r_i := -\\frac{f(x_i) \\nabla f(x_i)}{\\Vert \\nabla f(x_i) \\Vert^2}$, $x_{i+1} = x_i + r_i$.\n",
    "\n",
    "\n",
    "<img src='deepfool.png'>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Universal adversarial attacks\n",
    "\n",
    "Idea of universal adversarial attacks has been proposed [in the paper](https://openaccess.thecvf.com/content_cvpr_2017/html/Moosavi-Dezfooli_Universal_Adversarial_Perturbations_CVPR_2017_paper.html) \n",
    "\n",
    "The idea is to have a single image such that adding it to all images **fools** the classifier.\n",
    "\n",
    "<img src='uap.pbm'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Universal adversarial attacks: algorithms\n",
    "\n",
    "Original idea of constructing universal adversarial attacks  uses geometrical ideas\n",
    "\n",
    "<img src='uap-alg.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## UAP: numbers and how they look like\n",
    "\n",
    "Typical numbers are 60-80% of fooling!\n",
    "<img src='uap-imgs.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Black-box attacks\n",
    "\n",
    "In the black-box attacks we only have limited knowledge about the target model, i.e. \n",
    "\n",
    "1. We don't know the weights\n",
    "2. We only have access to logits or even predictions of the model.\n",
    "\n",
    "How we can construct such kind of attacks?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Some types of black-box attacks\n",
    "\n",
    "**Transfer attacks:**\n",
    "\n",
    "In this attack, the attacker trains a substitute model to mimic the behavior of the target model using only input-output pairs. The substitute model can then be used to generate adversarial examples that can fool the target model.\n",
    "\n",
    "**Query-based attacks:** In this attack, the attacker submits a large number of input queries to the target model to infer its internal behavior. This information can then be used to generate adversarial examples.\n",
    "\n",
    "**Zeroth-order optimization:** In this attack, the attacker uses only the output of the target model to generate adversarial examples, without any knowledge of the internal parameters or architecture of the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Example black-box attacks\n",
    "\n",
    "One of the simplest and efficient black-box attacks is [Square attack](https://arxiv.org/abs/1912.00049)\n",
    "\n",
    "We find squares by **random search**.\n",
    "\n",
    "The squares are sampled randomly according to some distributions.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Beyond small-norm attacks\n",
    "\n",
    "Small-norm attacks are done in the **digital domain**. But small-norm is not always the right **attack model**.\n",
    "\n",
    "Important classes of attacks include:\n",
    "\n",
    "- Sparse attacks\n",
    "- Patches\n",
    "- Semantic transformations (rotations, gaussian blurring, etc.)\n",
    "\n",
    "Surprisingly, most of them break the classifiers!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Real-world attacks\n",
    "\n",
    "A significant attention has been given to **real-world attacks**.\n",
    "<div style=\"display: flex\">\n",
    "    <div style=\"flex: 1; margin-top: 20px\">\n",
    "        <img src=\"ex1.png\" width=\"90%\">\n",
    "    </div>\n",
    "    <div style=\"flex: 1; margin-top: 20px\">\n",
    "        <img src=\"yolov2.png\" width=\"90%\">\n",
    "    </div>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## How to build real-world attacks\n",
    "\n",
    "Building real-world attacks is an interesting engineering task. \n",
    "\n",
    "Since the neural networks are unstable with respect to small perturbations, they also get really well back\n",
    "\n",
    "if the **noising process** is put back in place.\n",
    "\n",
    "I.e., when you print and you take the photo, the image is distorted and it can no longer by an **adversarial example**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Attacking in real world\n",
    "\n",
    "The first paper is [Adversarial examples in the physical world by Alexey Kurakin, Ian Goodfellow, Samy Bengio\n",
    "](https://arxiv.org/abs/1607.02533)\n",
    "\n",
    "Idea was super-simple: incorporate image transformations (rotations/blur/brightness) into the process of generating attacks.\n",
    "\n",
    "The attacks should be attacks under all of those transformations.\n",
    "\n",
    "At later [paper](https://arxiv.org/pdf/1707.07397.pdf) the approach called (expectations over transformations) has been proposed.\n",
    "\n",
    "Instead of using $f(x)$, they used\n",
    "\n",
    "$$\\hat{f}(x) = E_T T(f(x))$$ as the loss function.\n",
    "\n",
    "Other approaches include synthetic data generation, mapping real-life photos using generative adversarial networks (GANs), etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Defending against attacks\n",
    "\n",
    "Adversarial attacks pose significant danger in several scenarios, thus it is important to develop defenses.\n",
    "\n",
    "The defenses can be **empirical** or **certified**. Certified means that we guarantee that for a fixed attack model, the prediction will not change. We can do certain certification for small-norm attacks.\n",
    "\n",
    "Emprical defenses make certain modifications to the training or inference procedures."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Empirical defenses against adversarial attacks\n",
    "\n",
    "There are several standard approaches against attacks. Among them:\n",
    "\n",
    "1. **Adversarial training:** This involves training the model on both clean and adversarial examples to improve its robustness against attacks.\n",
    "2. **Defensive distillation:** This involves training the model to output softened probabilities instead of hard probabilities, which can make it more difficult for attackers to generate adversarial examples.\n",
    "3. **Randomization:** This involves adding random noise or perturbations to the input or model parameters to make it more difficult for attackers to generate adversarial example\n",
    "4. **Gradient masking:** This involves hiding the gradient information from attackers by adding noise to the gradients or using gradient obfuscation techniques.\n",
    "5. **Ensemble methods:** This involves combining multiple models to improve their robustness against attacks. Adversarial examples that fool one model may not fool another, making it more difficult for attackers to succeed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Adversarial training\n",
    "\n",
    "In adversarial training, we aim at minimizing the loss at the worst possible sample in the vicinity of the current one.\n",
    "\n",
    "\\begin{equation}\n",
    "\\min_{\\theta} \\mathbb{E}_{x,y \\sim p_{data}(x,y)} [\\max_{\\delta \\in S} \\mathcal{L}(f_{\\theta}(x+\\delta), y)]\n",
    "\\end{equation}\n",
    "\n",
    "The solution of the inner maximization problem is done by several PGD steps, giving much slower training time, but increased robustness.\n",
    "\n",
    "The idea of adversarial training has been proposed in the paper by [Madry et. al](https://arxiv.org/abs/1706.06083)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Adversarial training for free\n",
    "\n",
    "[Adversarial training for free](https://arxiv.org/abs/1904.12843) proposes two things:\n",
    "\n",
    "1. Compute the gradient with respect to the input and parameters on the same step;\n",
    "2. Train on the same mini-batch $m$ times (to mimic $m$-step PGD attacks).\n",
    "\n",
    "Leads to neglible overhead!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Fast is better than free\n",
    "\n",
    "Next step is [Fast is better than free](https://arxiv.org/pdf/2001.03994.pdf) paper, which proposes the following idea. \n",
    "\n",
    "In 'Adversarial training for free' perturbation from the previous sample is used as an initialization for the next sample.\n",
    "\n",
    "It is difficult to believe that it is a good starting point, but it is non-zero.\n",
    "\n",
    "\n",
    "Instead, the authors initialize the perturbation at random, and use 1 step of FGSM in training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Smooth adversarial training\n",
    "\n",
    "[Smooth adversarial training](https://arxiv.org/abs/2006.14536)\n",
    "\n",
    "\n",
    "Simple idea in the Quoc Le style: replace ReLU (non-smooth) with its smooth variant, significantly improves robustness."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Randomized smoothing\n",
    "\n",
    "Let $f(x)$ be our binary classifier. Existence of attacks means that under a small perturbation $f(x + \\varepsilon)$ changes a lot. Thus, it means that $f(x)$ should have **large Lipschitz constant**.\n",
    "\n",
    "One can try to make the classifier smoother by imposing certain normalization techniques, such as **spectral normalization** (which is typically a bounded norm of the linear layer).\n",
    "\n",
    "An alternative approach is to modify the inference procedure by **smoothing** over small perturbation. \n",
    "\n",
    "It is called **randomized smoothing**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Randomized smoothing: Theory \n",
    "\n",
    "Randomized smoothing has been proposed by [Cohen et. al](https://arxiv.org/abs/1902.02918)\n",
    "\n",
    "The idea is to replace the inference procedure with smoothing\n",
    "\n",
    "$$\\hat{f}(x) = E_{\\varepsilon \\sim N(0, \\sigma^2)}f(x + \\varepsilon).$$\n",
    "\n",
    "The Cohen paper uses the indicator function under smoothing. \n",
    "\n",
    "It is basically a **voting method**: we sample attacks, and select the class that is predicted the most times.\n",
    "\n",
    "<img src='randsm.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Certified radius bound\n",
    "\n",
    "Let $p_A$ and $p_B$ be the probability of true class and false class from the randomized classifier.\n",
    "\n",
    "We can not evaluate them directly, therefore we can get access to their bounds $\\underline{p_A}$ and $\\overline{p_B}$.\n",
    "\n",
    "Then, the smoothed classifier is guaranteed to get the same prediction for an $x$ within the radius\n",
    "\n",
    "$$R = \\frac{\\sigma}{2}\\left(\\Phi^{-1}(\\underline{p_A}) - \\Phi^{-1}(\\overline{p_B})\\right),$$\n",
    "\n",
    "where $\\Phi^{-1}$ is an error function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Randomized smoothing: Theory \n",
    "\n",
    "A more complicated estimate for the Randomized Smoothing has been obtained by [Salman et. al](https://arxiv.org/abs/1906.04584).\n",
    "\n",
    "Let $f(x)$ be the base classified, $f(x) \\in [0, 1]$. Let $\\hat{f}(x)$ be a smoothed classified with $\\sigma=1$.\n",
    "\n",
    "Let $\\Phi(x)$ be the error function.\n",
    "\n",
    "Then, \n",
    "\n",
    "$$g(x) = \\Phi^{-1}(\\hat{f})$$ has **Lipschitz constant one**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Randomized smoothing: discussion\n",
    "\n",
    "We have \n",
    "1. Vanilla accuracy (of the base classifier)\n",
    "2. Accuracy of the smoother classifier\n",
    "3. Certified accuracy (vs. the attack radius).\n",
    "\n",
    "If we train the base classifier only, the smoothed accuracy will be smaller.\n",
    "\n",
    "In practice, we can add maximize the accuracy of the smoothed classified, by minimizing\n",
    "\n",
    "$$L(\\hat{f}) \\rightarrow \\min.$$\n",
    "\n",
    "Note, that during training we can replace smoothing just by random Gaussian augmentation.\n",
    "\n",
    "This will give an unbiased estimate of the gradient."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## How we measure robustness of deep learning model\n",
    "\n",
    "A standard protocol to measure robustness of deep learning models is to attack them using PGD attacks.\n",
    "\n",
    "A standard benchmark is [RobustBench](https://paperswithcode.com/paper/robustbench-a-standardized-adversarial)\n",
    "\n",
    "Lets have a look at the leaderboard: https://robustbench.github.io/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "## Summary\n",
    "- Adversarial attacks\n",
    "- Adversarial training\n",
    "- Robustness of DL models (using randomized smoothing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Next lecture: Generative models I\n",
    "\n",
    "- Autoregressive models \n",
    "- Variational Autoencoders"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Слайд-шоу",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  },
  "nav_menu": {},
  "rise": {
   "scroll": true,
   "theme": "sky",
   "transition": "zoom"
  },
  "toc": {
   "navigate_menu": true,
   "number_sections": false,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}