note/GloVe.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GloVe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 问题设定"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GloVe是Global Vectors for Word Representation的缩写。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于One-hot词向量："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{aligned}\n",
    "I &= [1, 0, 0] \\\\\n",
    "Like &= [0, 1, 0] \\\\\n",
    "Apple &= [0, 0, 1] \n",
    "\\end{aligned}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "无法通过两向量夹角余弦值计算其相似度，word2vec是一种嵌入模型，通过这种模型训练出的词向量可以较好的表示出词之间的相似度，但是word2vec仅仅考虑了两个词在一段上下文的相关度，而GloVe考虑了两个词向量在全文中的相关度。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 共现矩阵（Co-occurrence Probabilities Matrix）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练GloVe模型前，首先需要构建一个共现矩阵，设词表大小为V，共现矩阵将是一个V行V列的方阵，而第i行第j列的表示了以第i个中心词$w_i$，第j个背景词$w_j$出现的次数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "假设我们有上下文："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "an\\ apple\\ a\\ day\\ keeps\\ an\\ apple\\ a\\ day\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们设定滑窗大小m等于2，我们将会有如下中心词-背景词对："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "中心词  |    背景词   |\n",
    ":---:  |    :---:   |\n",
    "an     | apple, a   |\n",
    "apple  | an, a, day |\n",
    "a      | an, apple, day, keeps   |\n",
    "day    | apple, a, keeps, an     |\n",
    "keeps  | a, day, an, apple       |\n",
    "an     | day, keeps, apple, a |\n",
    "apple  | keeps, an, a, day |\n",
    "a      | an, apple, day |\n",
    "day    | apple, a |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后遍历中心词-背景词对，更新共现矩阵，以上图为例，最后共现矩阵的结果将有如下形式："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|        |   An  | apple | a | day | keeps |\n",
    "|    -   |   -   |   -   | - |  -  |   -   |\n",
    "|   An   |   0   |   2   | 2 |  1  |   1   |\n",
    "|  apple |   2   |   0   | 2 |  2  |   1   |\n",
    "|    a   |   2   |   2   | 0 |  2  |   1   |\n",
    "|   day  |   0   |   2   | 2 |  0  |   1   |\n",
    "|  keeps |   1   |   1   | 1 |  1  |   0   |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "共现矩阵揭示了某种规律，定义共现矩阵的第i行的和为："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "X_i = \\sum^{V}_{j=1}X_{i, j}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "之后我们有条件概率，即第j列对应的词出现在第i行上下文中的条件概率："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\mathbb{P}_{i, j} = \\frac{X_{i, j}}{X_i}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "而对于某个词$w_k$，他在第i行或者第j行上下文出现的条件概率的比值："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\mathbb{P}_{i, k}}{\\mathbb{P}_{j, k}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个值是可以直接观测并计算到的，并将会有如下规律：\n",
    "- 如果$w_j$与$w_k$相关，且$w_i$与$w_k$相关，那么这个比值将会趋近于1\n",
    "- 如果$w_j$与$w_k$相关，且$w_i$与$w_k$不相关，那么这个比值将会很小\n",
    "- 如果$w_j$与$w_k$不相关，且$w_i$与$w_k$相关，那么这个比值将会很大\n",
    "- 如果$w_j$与$w_k$不相关，且$w_i$与$w_k$不相关，那么这个比值将会趋近于1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 损失函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们希望设计一个损失函数，希望对词表内每两个词对，$w_i$与$w_j$，尽可能与$w_k$在共现矩阵中对于第i, j上下文中，出现的条件概率比值相近："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\mathbb{P}_{i, k}}{\\mathbb{P}_{j, k}} = \\frac{\\exp (v^T_i v_k) }{\\exp (v^T_j v_k)}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "两边取对数，对于分子分母："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\log \\frac{X_{i, k}}{X_i} = \\log X_{i, k} - \\log X_i = v^T_i v_k \\\\\n",
    "\\log \\frac{X_{j, k}}{X_j} = \\log X_{j, k} - \\log X_j = v^T_j v_k\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到问题得到了简化，我们希望左式的分子尽可能等于右式的分子，分母亦然，则问题被简化为：对于词表内任意一组词对i, j，我们希望最小化下式："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\sum^{V}_{i=1} \\sum^{V}_{j=1} \\left( v^T_i v_j - \\log X_i - \\log(X_{i, j})\\right )^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其中偏置项$b_i, b_j$将会替换$\\log X_i$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但是并不是每一个词对都是平权的，需要考虑词频来设定每一个词对的权重："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "f(X_{i, j}) = \n",
    "\\begin{cases}\n",
    "(X_{i, j} \\ /\\ C)^{0.75}& \\text{ X > c }\\\\\n",
    "1& \\text{ X < 0}\n",
    "\\end{cases}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后我们希望最小化："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\sum^{V}_{i=1} \\sum^{V}_{j=1} f(X_{i, j}) \\left( v^T_i v_j + b_i + b_j - \\log(X_{i, j})\\right )^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后使用中心词向量$v_j$与背景词向量$v_i$的和作为中心词向量的表示。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}