PLAM-8-3 (Lesion segmentation).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 常规赛：PALM病理性近视病灶检测与分割基线方案\n",
    "本项目参考[红白黑](https://aistudio.baidu.com/aistudio/personalcenter/thirdview/277680)大佬基线：[常规赛：PALM病理性近视病灶检测与分割基线方案 基线1.0版本](https://aistudio.baidu.com/aistudio/projectdetail/1941312)\n",
    "\n",
    "**赛题简述**\n",
    "\n",
    "\tPALM病理性近视病灶检测与分割常规赛的重点是研究和发展与病理性近视诊断和患者眼底照片病变分割相关的算法。该常规赛的目标是评估和比较在一个常见的视网膜眼底图像数据集上检测病理性近视的自动算法。具体任务为：检测眼底图像是否出现视网膜萎缩病变和脱离病变，若有，需要实现病变区域的分割。\n",
    "    \n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/bcb4c79dd34242e0bd83db8ea8062d7fa6a12a59999c44f09a73c9298a45fae3)\n",
    "    \n",
    "    \n",
    "**数据基本标签**\n",
    "\n",
    "\t萎缩病变分割金标准：萎缩区域：0；背景：255；\n",
    "\t脱离病变分割金标准：脱离区域：0；背景：255。\n",
    "    \n",
    "**训练数据集**\n",
    "文件名称：Train\n",
    "\n",
    "Train里有两个文件夹，一个是fundus_images，一个是Lesion_Masks。\n",
    "\n",
    "* fundus_images文件夹内包含**800**张眼底彩照，分辨率为1444×1444，或2124×2056。命名形如H0001.jpg、N0001.jpg、P0001.jpg和V0001.jpg。\n",
    "\n",
    "* Lesion_Masks文件夹内包含两个文件夹：**Atrophy**和**Detachment**，其中，Atrophy文件夹包含fundus_images里眼底彩照的萎缩病变区域分割金标准，大小与对应的眼底彩照一致。命名前缀与对应眼底图像一致，后缀为bmp。同理，Detachment文件夹包含fundus_images里眼底彩照的脱离病变区域分割金标准，大小与对应的眼底彩照一致，命名前缀与对应眼底图像一致，后缀为bmp。**请注意，若Lesion_Masks中无某张眼底图像的病灶分割结果，说明该图像中不包含对应的病灶区域**。\n",
    "\n",
    "**测试数据集**\n",
    "\n",
    "文件名称：PALM-Testing400-Images.zip\n",
    "\n",
    "压缩包里包含400张眼底彩照，命名形如T0001.jpg。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 一、基线内容方案\n",
    "\n",
    "* 解压数据与数据划分\n",
    "\n",
    "    -- # 解压数据集\n",
    "\n",
    "    -- !unzip -oq /home/aistudio/data/data93538/常规赛：PALM病理性近视病灶检测与分割.zip -d PaddleSeg/data\n",
    "\n",
    "    -- # 划分数据\n",
    "\n",
    "    -- !python utils/dataset_splited.py\n",
    "\n",
    "* 数据标签预处理\n",
    "\n",
    "    -- # 转换标签\n",
    "\n",
    "    -- !python utils/dataset_pretrans.py\n",
    "\n",
    "    * 原分类为1分类问题，为了问题研究的充分性和更大程度上利用多分类间的类别竞争对分类结构有一个更好的指导\n",
    "\n",
    "    * 二分类问题描述，原标签为0不变，将255无效值转换为1值\n",
    "\n",
    "    * 后期提交前会后处理，消去1值，换回赛题需要的255值\n",
    "\n",
    "* 利用PaddleSeg套件加速赛题开发与测试: 使用套件config中的模型yml进行快速高效的实验开发——注意数据集yml的配置\n",
    "\n",
    "* 实现训练流程\n",
    "\n",
    "* 实现预测流程\n",
    "\n",
    "* 完成提交结果 -- 基线方案为0.67+的得分(D_iter:500, A_iter:2000)，可从训练迭代次数、损失函数、模型入手\n",
    "\n",
    "    -- # 提交结果后处理\n",
    "\n",
    "    -- utils/post_process.py\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "> 先解压需要的PadleSeg套件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 解压PaddleSeg压缩包\r\n",
    "!unzip -oq data/data88946/PaddleSeg.zip -d /home/aistudio/\r\n",
    "# 修改文件名\r\n",
    "!mv PaddleSeg-release-v2.0 PaddleSeg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "上一步mv，可以将PaddleSeg加压后的文件目录改成PaddleSeg\n",
    ">PaddleSeg下载至github的release2.0版本，为了方便大家使用，已添加在了数据集中供大家使用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 删除data目录 —— 用于数据划分时，产生了意料之外的数据扩充时的数据重置\r\n",
    "# %cd /home/aistudio/\r\n",
    "# !rm -rf PaddleSeg/data\r\n",
    "# 解压数据集到PaddleSeg目录下的data文件夹\r\n",
    "!unzip -oq /home/aistudio/data/data93538/常规赛：PALM病理性近视病灶检测与分割.zip -d PaddleSeg/data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 查看数据集文件的树形结构\r\n",
    "!tree -d PaddleSeg/data/常规赛：PALM病理性近视病灶检测与分割"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 二、比赛数据集情况\n",
    "\n",
    "PALM-Testing400-Images : 测试数据集文件夹\n",
    "\n",
    "Train : 训练数据集文件夹\n",
    "\n",
    "* Lesion_Masks ; 标注图片\n",
    "\n",
    "\t-- Detachment 视网膜脱落标注 -- 样本较少，存在同时萎缩的样本\n",
    "    \n",
    "   -- Atrophy 萎缩标注\n",
    "   \n",
    "* fundus_image : 原始图片\n",
    "\n",
    "> 注意没有验证数据集，这里提供一个简单的划分程序，划分比例为0.7\n",
    "\n",
    "> utils/dataset_splited.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "通过PIL的Image读取图片查看以下原数据与Label标注情况"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from PIL import Image\r\n",
    "import numpy as np\r\n",
    "# 读取图片\r\n",
    "png_img = Image.open('PaddleSeg/data/常规赛：PALM病理性近视病灶检测与分割/Train/fundus_image/H0003.jpg')\r\n",
    "png_img  # 展示真实图片"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "bmp_img = Image.open('PaddleSeg/data/常规赛：PALM病理性近视病灶检测与分割/Train/Lesion_Masks/Atrophy/H0003.bmp')\r\n",
    "bmp_img   # 展示萎缩标注图片"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "bmp_img = Image.open('PaddleSeg/data/常规赛：PALM病理性近视病灶检测与分割/Train/Lesion_Masks/Detachment/P0053.bmp')\r\n",
    "bmp_img   # 展示脱落标注图片"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 三、划分数据集与数据预处置\n",
    "\n",
    "当前划分比例为0.8——可在utils文件夹下的dataset_splited.py修改**train_percent**为其它值\n",
    "\n",
    "数据预处置-可在utils文件夹下的dataset_pretrans.py中查看相关代码--实现将255转化为1，原问题变二分类问题\n",
    "\n",
    "> 注意：当前数据处理中，存在对数据进行扩充，因此当前程序运行一次之后会使得原分类数据数目增加——(扩增不宜过大，否则划分数据进行训练和验证时会出现偏差问题)\n",
    "\n",
    "> 仅可运行一次，多次运行会导致填充数量过多，这是源码中扩充机制决定的(直接扩充到源文件夹中，所以下一次划分的时候就会默认把源文件中所有的文件读取)\n",
    "\n",
    "> 感兴趣可前往查看dataset_splited.py的数据扩充区"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 保证路径为初始路径\r\n",
    "%cd /home/aistudio\r\n",
    "\r\n",
    "# 划分数据\r\n",
    "!python utils/dataset_splited.py\r\n",
    "\r\n",
    "# 转换标签--预处置\r\n",
    "!python utils/dataset_pretrans.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "移除原数据，减小项目空间，减少下一次进入和退出保存时花的时间"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 移除’常规赛：PALM病理性近视病灶检测与分割‘文件夹\r\n",
    "!rm -rf PaddleSeg/data/常规赛：PALM病理性近视病灶检测与分割\r\n",
    "!rm -rf PaddleSeg/data/__MACOSX "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 四、选择比赛模型\n",
    "\n",
    "基线模型为:  配置略微修改的`PaddleSeg/configs/emanet/emanet_resnet50_os8_voc12aug_512x512_40k.yml`\n",
    "\n",
    "具体配置在\n",
    "\t\n",
    "\t-- example/unet_Deta.yml\n",
    "    * 用于分割脱落情况\n",
    "    \n",
    "\t-- example/emanet_resnet50_os8_voc12aug_512x512_40k_Atro.yml\n",
    "    * 用于分割萎缩情况\n",
    "\n",
    "详细数据集配置在\n",
    "\n",
    "\t-- example/unet_Deta.yml\n",
    "    * 用于分割脱落情况\n",
    "\n",
    "\t-- example/pascal_voc2012_Atro.yml\n",
    "    * 用于设置萎缩情况数据\n",
    "    \n",
    "\n",
    "> 针对不同分割任务配置不同的分割模型，以适配不同的任务驱动"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 1.配置_base_中对应的数据集与模型\n",
    "\n",
    "> 具体配置信息，可以在`examples`文件夹下查看相应`yml`文件，有相应的注释。\n",
    "\n",
    "**简要说明`example/pascal_voc2012_Atro.yml`中的配置要点**，以方便大家修改其他的数据集yml适配模型训练\n",
    "\n",
    "```yml\n",
    "# 该文件需要自行移动到PaddleSeg/configs/_base_下, 并修改模型文件中的_base_路径(建议)\n",
    "# 或者根据该文件中的train_dataset与val_dataset，对_base_下相应的yml进行修改\n",
    "\n",
    "# 批大小   -- 可通过训练时动态调整\n",
    "batch_size: 8\n",
    "# 迭代次数 -- 可通过训练时动态调整 \n",
    "iters: 40000\n",
    "\n",
    "# 自定义数据集加载的方式\n",
    "train_dataset:\n",
    "  # 自定义数据集加载方式：Dataset\n",
    "  type: Dataset\n",
    "  # 数据集目录--当前项目中类别Detachment的数据都放在了这里：data/Detachment\n",
    "  # 不同类别的训练，可以换成不同的数据根目录\n",
    "  # Atrophy类--对应data/Atrophy\n",
    "  dataset_root: data/Atrophy\n",
    "  # 该目录下划分数据产生的txt：data/Detachment/train_list.txt\n",
    "  train_path: data/Atrophy/train_list.txt\n",
    "  # 类别--当前已转换为2分类问题\n",
    "  num_classes: 2\n",
    "  # 预处理\n",
    "  transforms:\n",
    "    - type: ResizeStepScaling\n",
    "      min_scale_factor: 0.5\n",
    "      max_scale_factor: 2.0\n",
    "      scale_step_size: 0.25\n",
    "    # 考虑到当前数据集正确划分区域较小，RandomPaddingCrop是否有必要不清楚，可以自行尝试\n",
    "    # 下边有提供Resize处理缩放--选1缩放标准即可\n",
    "    # - type: RandomPaddingCrop\n",
    "    #   crop_size: [800, 800]\n",
    "    - type: Resize\n",
    "      target_size: [680, 680]\n",
    "    - type: RandomHorizontalFlip\n",
    "    - type: RandomVerticalFlip\n",
    "    - type: Normalize\n",
    "  # 数据集加载方式--注意要一一对应\n",
    "  mode: train\n",
    "\n",
    "val_dataset:\n",
    "  type: Dataset\n",
    "  dataset_root: data/Atrophy\n",
    "  # 注意验证数据集的path和训练数据集path的区别\n",
    "  val_path: data/Atrophy/val_list.txt\n",
    "  num_classes: 2\n",
    "  transforms:\n",
    "    # 修改padding为Resize，padding仅作填充，对于大图片无法缩放\n",
    "    - type: Resize\n",
    "      target_size: [680, 680]\n",
    "    - type: Normalize\n",
    "  # mode务必对应，否则无法索引正确的路径\n",
    "  mode: val\n",
    "\n",
    "\n",
    "# train_dataset:\n",
    "#   type: PascalVOC\n",
    "#   dataset_root: data/VOCdevkit/\n",
    "#   transforms:\n",
    "#     - type: ResizeStepScaling\n",
    "#       min_scale_factor: 0.5\n",
    "#       max_scale_factor: 2.0\n",
    "#       scale_step_size: 0.25\n",
    "#     - type: RandomPaddingCrop\n",
    "#       crop_size: [512, 512]\n",
    "#     - type: RandomHorizontalFlip\n",
    "#     - type: RandomDistort\n",
    "#       brightness_range: 0.4\n",
    "#       contrast_range: 0.4\n",
    "#       saturation_range: 0.4\n",
    "#     - type: Normalize\n",
    "#   mode: train\n",
    "\n",
    "# val_dataset:\n",
    "#   type: PascalVOC\n",
    "#   dataset_root: data/VOCdevkit/\n",
    "#   transforms:\n",
    "#     - type: Padding\n",
    "#       target_size: [512, 512]\n",
    "#     - type: Normalize\n",
    "#   mode: val\n",
    "\n",
    "# 以下参数可以在模型yml中被配置\n",
    "# 优化器选择\n",
    "optimizer:\n",
    "  type: sgd\n",
    "  momentum: 0.9\n",
    "  # 正则化\n",
    "  weight_decay: 4.0e-5\n",
    "\n",
    "# 学习率--多项式\n",
    "learning_rate:\n",
    "  value: 0.01\n",
    "  decay:\n",
    "    type: poly\n",
    "    power: 0.9\n",
    "    end_lr: 0.0\n",
    "\n",
    "# 损失配置项--可参考其它模型yml文件\n",
    "loss:\n",
    "  types:\n",
    "    - type: CrossEntropyLoss\n",
    "  coef: [1]\n",
    "```\n",
    "\n",
    "**至于模型配置，以当前使用Emanet为例说明**:\n",
    "\n",
    "> 大家在使用模型yml时，为了保证数可读取，可以使用提供的两个数据集yml，分别操作加载不同的数据\n",
    "\n",
    "```yml\n",
    "# _base_: '../_base_/pascal_voc12aug.yml'\n",
    "\n",
    "# 用调整后的pascal_voc2012.yml替换原始的_base_数据集配置文件\n",
    "# 使用新模型要保证数据集路径正确\n",
    "# 当前模型配置文件yml要移动到指定的PaddleSeg/paddleseg/emanet文件夹下\n",
    "# 数据集要移动到PaddleSeg/paddleseg/_base_文件夹下\n",
    "_base_: '../_base_/pascal_voc2012_Atro.yml'\n",
    "\n",
    "model:\n",
    "  type: EMANet\n",
    "  backbone:\n",
    "    type: ResNet50_vd\n",
    "    output_stride: 8\n",
    "    pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz\n",
    "  ema_channels: 512\n",
    "  gc_channels: 256\n",
    "  num_bases: 64\n",
    "  stage_num: 3\n",
    "  momentum: 0.25\n",
    "  concat_input: True\n",
    "  enable_auxiliary_loss: True\n",
    "  align_corners: True\n",
    "\n",
    "optimizer:\n",
    "  type: sgd\n",
    "  momentum: 0.9\n",
    "  weight_decay: 0.0005\n",
    "\n",
    "\n",
    "loss:\n",
    "  types:\n",
    "    - type: CrossEntropyLoss\n",
    "    - type: DiceLoss\n",
    "  coef: [4.,2.]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "> 将预置的基线配置yml移动到相应的文件夹下\n",
    "\n",
    "* _base_: 数据加载yml存放\n",
    "\n",
    "* emanet: emanet模型yml存放"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%cd /home/aistudio/\r\n",
    "!cp -u example/pascal_voc2012_Atro.yml PaddleSeg/configs/_base_/\r\n",
    "!cp -u example/pascal_voc2012_Deta.yml PaddleSeg/configs/_base_/\r\n",
    "# !cp -u example/emanet_resnet50_os8_voc12aug_512x512_40k_Deta.yml PaddleSeg/configs/emanet/\r\n",
    "!cp -u example/unet_Deta.yml PaddleSeg/configs/\r\n",
    "!cp -u example/emanet_resnet50_os8_voc12aug_512x512_40k_Atro.yml PaddleSeg/configs/emanet/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "\n",
    "## 2.启动训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 0.下载依赖项\n",
    "\n",
    "在平台上可以不用执行，环境支持；线下可能需要下载。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 下载依赖项，保证PaddleSeg正常运行\r\n",
    "%cd PaddleSeg\r\n",
    "%pwd\r\n",
    "!pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 1. 载入模型开始训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "> 更换自定义的模型文件时，只需要修改对应的模型yml、数据集yml(_base_中的yml)、以及替换下方的yml即可进行训练了\n",
    "\n",
    "> 不要更改输出目录，否则后边的代码也需要修改，包括预测、后处理中的所有路径"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%cd ~\r\n",
    "# %pwd\r\n",
    "# 训练分割Detachment的模型，并保存到 ./output/Detachment\r\n",
    "!python PaddleSeg/train.py --c PaddleSeg/configs/unet_Deta.yml \\\r\n",
    "--use_vdl \\\r\n",
    "--save_interval 200 \\\r\n",
    "--do_eval \\\r\n",
    "--seed 2021 \\\r\n",
    "--iters 1000 \\\r\n",
    "--learning_rate 0.01 \\\r\n",
    "--save_dir output/Detachment\r\n",
    "\r\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 训练分割Atrophy的模型，并保存到/output/Atrophy\r\n",
    "!python  PaddleSeg/train.py --c  PaddleSeg/configs/emanet/emanet_resnet50_os8_voc12aug_512x512_40k_Atro.yml \\\r\n",
    "--use_vdl \\\r\n",
    "--save_interval 200 \\\r\n",
    "--do_eval \\\r\n",
    "--seed 2021 \\\r\n",
    "--iters 3000 \\\r\n",
    "--learning_rate 0.005 \\\r\n",
    "--save_dir output/Atrophy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 3.开始预测\n",
    "\n",
    "这可以直接使用emanet进行预测，不用修改数据集yml，也不用修改相应的模型yml，注意训练权重对应即可！\n",
    "\n",
    "> 预测结果按类别分别放在`./output/result/Detachment` 和 `./output/result/Atrophy` 下\n",
    "\n",
    "> 如果使用不同的模型对不同的分割类进行讨论，注意模型yml即可\n",
    "\n",
    "> 提交结果为两种单独预测的结果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**默认使用Iou评估最好的模型训练参数--best_model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/aistudio\n",
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/layers/utils.py:26: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.\n",
      "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
      "  def convert_to_list(value, n, name, dtype=np.int):\n",
      "2021-08-16 16:08:42 [INFO]\t\n",
      "---------------Config Information---------------\n",
      "batch_size: 8\n",
      "iters: 40000\n",
      "learning_rate:\n",
      "  decay:\n",
      "    end_lr: 0.0\n",
      "    power: 0.9\n",
      "    type: poly\n",
      "  value: 0.01\n",
      "loss:\n",
      "  coef:\n",
      "  - 4.0\n",
      "  - 2.0\n",
      "  types:\n",
      "  - type: CrossEntropyLoss\n",
      "  - type: DiceLoss\n",
      "model:\n",
      "  align_corners: true\n",
      "  backbone:\n",
      "    output_stride: 8\n",
      "    pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz\n",
      "    type: ResNet50_vd\n",
      "  concat_input: true\n",
      "  ema_channels: 512\n",
      "  enable_auxiliary_loss: true\n",
      "  gc_channels: 256\n",
      "  momentum: 0.25\n",
      "  num_bases: 64\n",
      "  stage_num: 3\n",
      "  type: EMANet\n",
      "optimizer:\n",
      "  momentum: 0.9\n",
      "  type: sgd\n",
      "  weight_decay: 0.0005\n",
      "train_dataset:\n",
      "  dataset_root: PaddleSeg/data/Atrophy\n",
      "  mode: train\n",
      "  num_classes: 2\n",
      "  train_path: PaddleSeg/data/Atrophy/train_list.txt\n",
      "  transforms:\n",
      "  - max_scale_factor: 2.0\n",
      "    min_scale_factor: 0.5\n",
      "    scale_step_size: 0.25\n",
      "    type: ResizeStepScaling\n",
      "  - target_size:\n",
      "    - 680\n",
      "    - 680\n",
      "    type: Resize\n",
      "  - type: RandomHorizontalFlip\n",
      "  - type: RandomVerticalFlip\n",
      "  - type: Normalize\n",
      "  type: Dataset\n",
      "val_dataset:\n",
      "  dataset_root: PaddleSeg/data/Atrophy\n",
      "  mode: val\n",
      "  num_classes: 2\n",
      "  transforms:\n",
      "  - target_size:\n",
      "    - 680\n",
      "    - 680\n",
      "    type: Resize\n",
      "  - type: Normalize\n",
      "  type: Dataset\n",
      "  val_path: PaddleSeg/data/Atrophy/val_list.txt\n",
      "------------------------------------------------\n",
      "W0816 16:08:42.098556  4588 device_context.cc:362] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1\n",
      "W0816 16:08:42.098603  4588 device_context.cc:372] device: 0, cuDNN Version: 7.6.\n",
      "2021-08-16 16:08:47 [INFO]\tLoading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz\n",
      "2021-08-16 16:08:47,064 - INFO - Lock 140451880091280 acquired on /home/aistudio/.paddleseg/tmp/resnet50_vd_ssld_v2\n",
      "2021-08-16 16:08:47,064 - INFO - Lock 140451880091280 released on /home/aistudio/.paddleseg/tmp/resnet50_vd_ssld_v2\n",
      "2021-08-16 16:08:47 [INFO]\tThere are 275/275 variables loaded into ResNet_vd.\n",
      "2021-08-16 16:08:47 [INFO]\tNumber of predict images = 400\n",
      "2021-08-16 16:08:47 [INFO]\tLoading pretrained model from output/Atrophy/best_model/model.pdparams\n",
      "Traceback (most recent call last):\n",
      "  File \"PaddleSeg/predict.py\", line 177, in <module>\n",
      "    main(args)\n",
      "  File \"PaddleSeg/predict.py\", line 171, in main\n",
      "    stride=args.stride,\n",
      "  File \"/home/aistudio/PaddleSeg/paddleseg/core/predict.py\", line 73, in predict\n",
      "    utils.utils.load_entire_model(model, model_path)\n",
      "  File \"/home/aistudio/PaddleSeg/paddleseg/utils/utils.py\", line 38, in load_entire_model\n",
      "    load_pretrained_model(model, pretrained)\n",
      "  File \"/home/aistudio/PaddleSeg/paddleseg/utils/utils.py\", line 93, in load_pretrained_model\n",
      "    pretrained_model))\n",
      "ValueError: The pretrained model directory is not Found: output/Atrophy/best_model/model.pdparams\n"
     ]
    }
   ],
   "source": [
    "%cd ~\r\n",
    "# 预测Detachment\r\n",
    "!python PaddleSeg/predict.py --config PaddleSeg/configs/unet_Deta.yml \\\r\n",
    "--model_path output/Detachment/best_model/model.pdparams \\\r\n",
    "--image_path PaddleSeg/data/Detachment/Test \\\r\n",
    "--save_dir PaddleSeg/output/result/Detachment\r\n",
    "\r\n",
    "\r\n",
    "# # 预测Atrophy\r\n",
    "# !python PaddleSeg/predict.py --config PaddleSeg/configs/emanet/emanet_resnet50_os8_voc12aug_512x512_40k_Atro.yml  \\\r\n",
    "# --model_path output/Atrophy/best_model/model.pdparams \\\r\n",
    "# --image_path PaddleSeg/data/Atrophy/Test \\\r\n",
    "# --save_dir PaddleSeg/output/result/Atrophy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 五、预测结果后处理\n",
    "\n",
    "* 将类别值1换为255，进行赛题结果提交"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%cd /home/aistudio/\r\n",
    "!python utils/post_process.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 六、提交比赛结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 复制文件到最顶层目录\r\n",
    "%cd /home/aistudio\r\n",
    "!cp -r PaddleSeg/output/result/ Lesion_Segmentation\r\n",
    "\r\n",
    "# 过程移动文件--保证不包含生成的子目录\r\n",
    "%cd Lesion_Segmentation\r\n",
    "!cp -r Detachment/pseudo_color_prediction/. Detachment\r\n",
    "!cp -r Atrophy/pseudo_color_prediction/. Atrophy\r\n",
    "\r\n",
    "# 获取指定的提交目录格式\r\n",
    "!rm -rf Detachment/added_prediction\r\n",
    "!rm -rf Detachment/pseudo_color_prediction\r\n",
    "\r\n",
    "!rm -rf Atrophy/added_prediction\r\n",
    "!rm -rf Atrophy/pseudo_color_prediction\r\n",
    "\r\n",
    "# 压缩文件\r\n",
    "%cd /home/aistudio\r\n",
    "!zip -r Lesion_Segmentation.zip Lesion_Segmentation\r\n",
    "# 删除复制的文件\r\n",
    "!rm -rf Lesion_Segmentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# # 删除zip的文件--丢失提交结果，需重新后处理生成\r\n",
    "# !rm -rf Lesion_Segmentation.zip\r\n",
    "# # 删除预测结果--丢失预测结果，需重新预测\r\n",
    "# !rm -rf PaddleSeg/output/result\r\n",
    "# # 删除output文件夹--丢失模型参数，需重新训练\r\n",
    "# !rm -rf PaddleSeg/output\r\n",
    "# # 删除data文件夹--数据将丢失，需要重新解压，划分，预处置\r\n",
    "# !rm -rf PaddleSeg/data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 七、其它建议\n",
    "\n",
    "> 对于少类别预测较差，可以通过更换模型、数据增广等方式获取更优的预测\n",
    "\n",
    "* 1. 模型建议：注意力模型(EMANet等)或者调整unet模型\n",
    "* 2. 损失建议：多损失结构，不同的coef，针对赛题的特殊损失等(Dice等)\n",
    "* 3. 模型魔改建议：尝试对Unet添加注意力模块，修改参数，或者调整不同的backbone与indices组合\n",
    "* 4. 优化器与学习率策略的调整\n",
    "\n",
    "最后，祝大家Paddle越用越顺手，比赛越打越顺利——取得理想的成绩！"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PaddlePaddle 2.0.0b0 (Python 3.5)",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}