demo/ma/finetuner/Model_Adapter_Finetuner_Walkthrough_ResNet50_CIFAR100.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2d21a062",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/e2eAIOK/blob/main/demo/ma/finetuner/Model_Adapter_Finetuner_Walkthrough_ResNet50_CIFAR100.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf11f116",
   "metadata": {
    "id": "cf11f116"
   },
   "source": [
    "# Model Adapter Finetuner Walkthrough DEMO\n",
    "Model Adapter is a convenient framework can be used to reduce training and inference time, or data labeling cost by efficiently utilizing public advanced models and those datasets from many domains. It mainly contains three components served for different cases: Finetuner, Distiller, and Domain Adapter. \n",
    "\n",
    "This demo mainly introduces the usage of Finetuner. Take image classification as an example, it shows how to apply Finetuner for ResNet50 on CIFAR100 dataset. This demo shows how to integrate Finetuner into a general training pipeline, you can find build-in simplied demo at [here](./Model_Adapter_Finetuner_builtin_ResNet50_CIFAR100.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bf212e5",
   "metadata": {
    "id": "0bf212e5"
   },
   "source": [
    "# Content\n",
    "\n",
    "* [Overview](#Overview)\n",
    "    * [Model Adapter Finetuner Overview](#Model-Adapter-Finetuner-Overview)\n",
    "* [Getting Started](#Getting-Started)\n",
    "    * [1. Environment Setup](#1.-Environment-Setup)\n",
    "    * [2. Data Prepare](#2.-Data-Prepare)\n",
    "    * [3. Model Prepare](#3.-Model-Prepare)\n",
    "    * [4. Train](#4.-Train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe09d6e6",
   "metadata": {},
   "source": [
    "# Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c5f4657",
   "metadata": {
    "id": "9c5f4657"
   },
   "source": [
    "## Model Adapter Finetuner Overview\n",
    "Finetuner is based on pretraining and finetuning technology, it can transfer knowledge from pretrained model to target model with same network structure. \n",
    "\n",
    "Pretrained models usually are generated by pretraining process, which is training specific model  on specific dataset and has been performed by DE-NAS, PyTorch, TensorFlow, or HuggingFace. Finetunner retrieves the pretrained model with same network structure, and copy pretrained weights from pretrained model to corresponding layer of target model, instead of random initialization for target mode. With finetunner, we can greatly improve training speed, and usually achieves better performance.\n",
    "\n",
    "<img src=\"../imgs/finetuner.png\" width=\"50%\">\n",
    "<center>Model Adapter Finetuner Structure</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d46b31f",
   "metadata": {},
   "source": [
    "# Getting Started"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72acefd2",
   "metadata": {
    "id": "72acefd2"
   },
   "source": [
    "## 1. Environment Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e390f70e",
   "metadata": {
    "id": "e390f70e"
   },
   "source": [
    "### (Option 1) Use Pip install \n",
    "\n",
    "We can directly install ModelAdapter module from Intel® End-to-End AI Optimization Kit with following command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc31cafe",
   "metadata": {
    "id": "bc31cafe",
    "outputId": "046462bb-16f3-4f34-899a-963ff183ddd0"
   },
   "outputs": [],
   "source": [
    "!pip install e2eAIOK-ModelAdapter --pre"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de684a99",
   "metadata": {
    "id": "de684a99"
   },
   "source": [
    "### (Option 2) Use Docker \n",
    "\n",
    "We can also use Docker, which contains a complete environment.\n",
    "\n",
    "Step1. prepare code\n",
    "   ``` bash\n",
    "   git clone https://github.com/intel/e2eAIOK.git\n",
    "   cd e2eAIOK\n",
    "   git submodule update --init –recursive\n",
    "   ```\n",
    "    \n",
    "Step2. build docker image\n",
    "   ``` bash\n",
    "   python3 scripts/start_e2eaiok_docker.py -b pytorch112 --dataset_path ${dataset_path} -w ${host0} ${host1} ${host2} ${host3} --proxy  \"http://addr:ip\"\n",
    "   ```\n",
    "   \n",
    "Step3. run docker and start conda env\n",
    "   ``` bash\n",
    "   sshpass -p docker ssh ${host0} -p 12347\n",
    "   conda activate pytorch-1.12.0\n",
    "   ```\n",
    "  \n",
    "Step4. Start the jupyter notebook and tensorboard service\n",
    "   ``` bash\n",
    "   nohup jupyter notebook --notebook-dir=/home/vmagent/app/e2eaiok --ip=${hostname} --port=8899 --allow-root &\n",
    "   nohup tensorboard --logdir /home/vmagent/app/data/tensorboard --host=${hostname} --port=6006 & \n",
    "   ```\n",
    "   Now you can visit demso in `http://${hostname}:8899/`, and see tensorboad log in ` http://${hostname}:6006`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6502c5f4",
   "metadata": {
    "id": "6502c5f4"
   },
   "source": [
    "## 2. Data Prepare"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a4223c4",
   "metadata": {
    "id": "5a4223c4"
   },
   "source": [
    "Let's import some required modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21507528",
   "metadata": {
    "id": "21507528"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torchvision import transforms,datasets\n",
    "from torch.utils.data import DataLoader\n",
    "import torch.optim as optim\n",
    "from timm.utils import accuracy\n",
    "import timm\n",
    "import transformers\n",
    "import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66290cf3",
   "metadata": {
    "id": "66290cf3"
   },
   "source": [
    "First let's define transformer for dataset, which will be needed to augment input image. The input size of the pre-trained model is 224, while for quick test, we resize image to 112. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5ab12af",
   "metadata": {
    "id": "f5ab12af"
   },
   "outputs": [],
   "source": [
    "CIFAR_TRAIN_MEAN = (0.4914, 0.4822, 0.4465)\n",
    "CIFAR_TRAIN_STD = (0.2023, 0.1994, 0.2010)\n",
    "\n",
    "train_transform = transforms.Compose([\n",
    "  transforms.RandomCrop(32, padding=4),\n",
    "  transforms.RandomHorizontalFlip(),\n",
    "  transforms.Resize(112),\n",
    "  transforms.ToTensor(),\n",
    "  transforms.Normalize(CIFAR_TRAIN_MEAN, CIFAR_TRAIN_STD)\n",
    "])\n",
    "\n",
    "test_transform = transforms.Compose([\n",
    "  transforms.RandomCrop(32, padding=4),\n",
    "  transforms.Resize(112),\n",
    "  transforms.ToTensor(),\n",
    "  transforms.Normalize(CIFAR_TRAIN_MEAN, CIFAR_TRAIN_STD)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12824c20",
   "metadata": {},
   "source": [
    "Then let's define CIFAR100 dataset and download it with torchvision lib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee8141bc",
   "metadata": {
    "id": "ee8141bc",
    "outputId": "0170329e-e1fb-4993-b763-399b0df85606"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "data_folder='./data' # dataset location\n",
    "train_set = datasets.CIFAR100(root=data_folder, train=True, download=True, transform=train_transform)\n",
    "test_set = datasets.CIFAR100(root=data_folder, train=False, download=True, transform=test_transform)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "198903f5",
   "metadata": {
    "id": "198903f5"
   },
   "source": [
    "Finally we define dataloader, you can change batch_size and num_workers according to your own environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f17bfb4",
   "metadata": {
    "id": "1f17bfb4"
   },
   "outputs": [],
   "source": [
    "train_loader = DataLoader(dataset=train_set, batch_size=128, shuffle=True, num_workers=1, drop_last=False)\n",
    "validate_loader = DataLoader(dataset=test_set, batch_size=128, shuffle=True, num_workers=1, drop_last=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "559869f4",
   "metadata": {
    "id": "559869f4"
   },
   "source": [
    "## 3. Model Prepare"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a22f4a5",
   "metadata": {
    "id": "6a22f4a5"
   },
   "source": [
    "**Create baseline model**\n",
    "\n",
    "First we create a ResNet50 without pretrained weights as backbone model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "043a86bc",
   "metadata": {
    "id": "043a86bc"
   },
   "outputs": [],
   "source": [
    "backbone = timm.create_model('resnet50', pretrained=False, num_classes=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0e92e4e",
   "metadata": {
    "id": "f0e92e4e"
   },
   "source": [
    "**Create pretrained model**\n",
    "\n",
    "To use finetuner, we need to prepare a pretrained model to initialize the model weights. Here we use ResNet50 pretrained on ImageNet21k and you can download with following link."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "74777365",
   "metadata": {
    "id": "74777365",
    "outputId": "62aeb928-d2fd-4498-c033-f56c96280f13"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2023-03-27 10:03:21--  https://miil-public-eu.oss-eu-central-1.aliyuncs.com/model-zoo/ImageNet_21K_P/models/resnet50_miil_21k.pth\n",
      "Resolving child-prc.intel.com (child-prc.intel.com)... 10.239.120.55\n",
      "Connecting to child-prc.intel.com (child-prc.intel.com)|10.239.120.55|:913... connected.\n",
      "Proxy request sent, awaiting response... 200 OK\n",
      "Length: 186531247 (178M) [application/octet-stream]\n",
      "Saving to: ‘resnet50_miil_21k.pth’\n",
      "\n",
      "100%[======================================>] 186,531,247 3.50MB/s   in 45s    \n",
      "\n",
      "2023-03-27 10:04:08 (3.92 MB/s) - ‘resnet50_miil_21k.pth’ saved [186531247/186531247]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "! wget https://miil-public-eu.oss-eu-central-1.aliyuncs.com/model-zoo/ImageNet_21K_P/models/resnet50_miil_21k.pth && mv resnet50_miil_21k.pth data/ "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d1f65e3",
   "metadata": {},
   "source": [
    "Then we create a pretrained model which loads the weights from pre-download file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a66565a7",
   "metadata": {
    "id": "a66565a7",
    "outputId": "5c12a607-b65c-40f4-d8b7-9bd047d69702"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pretrain_path = './data/resnet50_miil_21k.pth'\n",
    "pretrained_model = timm.create_model('resnet50', pretrained=False, num_classes=11221)\n",
    "pretrained_model.load_state_dict(torch.load(pretrain_path)['state_dict'], strict=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28f7f213",
   "metadata": {
    "id": "28f7f213"
   },
   "source": [
    "**Create Finetuner** \n",
    "\n",
    "Create Fine-tuner from Model Adapter, which take pretrained model as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb4ee26e",
   "metadata": {
    "id": "cb4ee26e"
   },
   "outputs": [],
   "source": [
    "from e2eAIOK.ModelAdapter.engine_core.finetunner import BasicFinetunner\n",
    "finetuner = BasicFinetunner(pretrained_model, is_frozen=False) # Do not freeze the weight training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4895179d",
   "metadata": {
    "id": "4895179d"
   },
   "source": [
    "**Wrap model with Model Adapter**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26281765",
   "metadata": {
    "id": "26281765"
   },
   "source": [
    "Call the *make_transferrable_with_finetune()* function, which take *backbone model, finetuner and a loss function* as input. It will check the layer mapping and automatically copy the weights from pretrained model in finetuner to backbone model, an \"ERROR\" message will appear as the last layer can't match, which is normal and can be ignored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "511c36e4",
   "metadata": {
    "id": "511c36e4",
    "outputId": "55c38284-498c-4967-a2bc-655571cebaf6"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR:root:could not load layer: fc.weight; mismatch shape: target [torch.Size([100, 2048])] != pretrained [torch.Size([11221, 2048])]\n",
      "ERROR:root:could not load layer: fc.bias; mismatch shape: target [torch.Size([100])] != pretrained [torch.Size([11221])]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "could not load layer: fc.weight; mismatch shape: target [torch.Size([100, 2048])] != pretrained [torch.Size([11221, 2048])]\n",
      "could not load layer: fc.bias; mismatch shape: target [torch.Size([100])] != pretrained [torch.Size([11221])]\n"
     ]
    }
   ],
   "source": [
    "from e2eAIOK.ModelAdapter.engine_core.transferrable_model import *\n",
    "loss_fn = torch.nn.CrossEntropyLoss()\n",
    "model = make_transferrable_with_finetune(backbone, loss_fn, finetuner)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f10a7715",
   "metadata": {
    "id": "f10a7715"
   },
   "source": [
    "## 4. Train"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7300bebf",
   "metadata": {
    "id": "7300bebf"
   },
   "source": [
    "**Create optimizer**\n",
    "\n",
    "To optimize the learning, we can set a smaller learning rate for finetuned layer with pretrained weights, and a normal learning rate for remaining layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d38b9e8a",
   "metadata": {
    "id": "d38b9e8a",
    "outputId": "69b6af88-07a2-473c-ac7a-086309edfb61"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[159] params set finetuner finetuned learning rate[0.00445]\n",
      "[2] params set common learning rate [0.00753]\n"
     ]
    }
   ],
   "source": [
    "finetuned_lr = 0.00445 \n",
    "learning_rate = 0.00753\n",
    "\n",
    "finetuned_state_keys = [\"backbone.%s\"%name for name in finetuner.finetuned_state_keys] # add component prefix\n",
    "finetuner_params = {'params':[p for (name, p) in model.named_parameters() if p.requires_grad and name in finetuned_state_keys],\n",
    "                    'lr': finetuned_lr}\n",
    "remain_params = {'params':[p for (name, p) in model.named_parameters() if p.requires_grad and name not in finetuned_state_keys],\n",
    "                    'lr': learning_rate}\n",
    "print(\"[%s] params set finetuner finetuned learning rate[%s]\" % (len(finetuner_params['params']), finetuned_lr))\n",
    "print(\"[%s] params set common learning rate [%s]\" % (len(remain_params['params']), learning_rate))\n",
    "assert len(finetuner_params) > 0,\"Empty finetuner_params\"\n",
    "parameters = [finetuner_params,remain_params]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03d2cf3d",
   "metadata": {},
   "source": [
    "Create the Optimizer, here we choose *SGD*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d4f7675",
   "metadata": {
    "id": "3d4f7675"
   },
   "outputs": [],
   "source": [
    "weight_decay = 0.00115\n",
    "momentum = 0.9\n",
    "optimizer = optim.SGD(parameters,lr=learning_rate, weight_decay=weight_decay,momentum=momentum)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4121d208",
   "metadata": {
    "id": "4121d208"
   },
   "source": [
    "**Create scheduler**\n",
    "\n",
    "Here we choose a *CosineAnnealingLR* scheduler, you can also change to other schedulers for your own task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "787b00e7",
   "metadata": {
    "id": "787b00e7"
   },
   "outputs": [],
   "source": [
    "scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=200)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc25db44",
   "metadata": {
    "id": "dc25db44"
   },
   "source": [
    "**Create Trainer**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f764b69",
   "metadata": {
    "id": "28b9bc1a"
   },
   "source": [
    "Create a simple *Trainer*, which contains *train()* and *evaluate()* function for this simple ResNet50 training task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8890b41",
   "metadata": {
    "id": "a8890b41"
   },
   "outputs": [],
   "source": [
    "max_epoch = 1 # max 1 epoch\n",
    "print_interval = 10\n",
    "class Trainer:\n",
    "    def __init__(self, model, optimizer, scheduler):\n",
    "        self._model = model\n",
    "        self._optimizer = optimizer\n",
    "        self._scheduler = scheduler\n",
    "        \n",
    "    def train(self, train_dataloader, valid_dataloader, max_epoch):\n",
    "        ''' \n",
    "        :param train_dataloader: train dataloader\n",
    "        :param valid_dataloader: validation dataloader\n",
    "        :param max_epoch: steps per epoch\n",
    "        '''\n",
    "        for epoch in range(0, max_epoch):\n",
    "            ################## train #####################\n",
    "            self._model.train()  # set training flag\n",
    "            for (cur_step,(data, label)) in enumerate(train_dataloader):\n",
    "                self._optimizer.zero_grad()\n",
    "                output = self._model(data)\n",
    "                loss_value = self._model.loss(output, label) # transferrable model has loss attribute\n",
    "                loss_value.backward()       \n",
    "                if cur_step%print_interval == 0:\n",
    "                    batch_acc = accuracy(output.backbone_output,label)[0] # use output.backbone_output instead of output\n",
    "                    dt = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S') # date time\n",
    "                    print(\"[{}] epoch {} step {} : training batch loss {:.4f}, training batch acc {:.4f}\".format(\n",
    "                      dt, epoch, cur_step, loss_value.backbone_loss.item(), batch_acc.item()))\n",
    "                self._optimizer.step()\n",
    "            self._scheduler.step()\n",
    "            ################## evaluate ######################\n",
    "            self.evaluate(valid_dataloader)\n",
    "            \n",
    "    def evaluate(self, valid_dataloader):\n",
    "        with torch.no_grad():\n",
    "            self._model.eval()  \n",
    "            backbone = self._model.backbone # use backbone in evaluation\n",
    "            loss_cum = 0.0\n",
    "            sample_num = 0\n",
    "            acc_cum = 0.0\n",
    "            total_step = len(valid_dataloader)\n",
    "            for (cur_step,(data, label)) in enumerate(valid_dataloader):\n",
    "                output = backbone(data)\n",
    "                batch_size = data.size(0)\n",
    "                sample_num += batch_size\n",
    "                loss_cum += loss_fn(output, label).item() * batch_size\n",
    "                acc_cum += accuracy(output, label)[0].item() * batch_size\n",
    "                if cur_step%print_interval == 0:\n",
    "                    print(f\"step {cur_step}/{total_step}\")\n",
    "            dt = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S') # date time\n",
    "            loss_value = loss_cum/sample_num\n",
    "            acc_value = acc_cum/sample_num\n",
    "\n",
    "            print(\"[{}] evaluation loss {:.4f}, evaluation acc {:.4f}\".format(\n",
    "                dt, loss_value, acc_value))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7b229e4",
   "metadata": {
    "id": "a7b229e4"
   },
   "source": [
    "**Train and Evaluate**\n",
    "\n",
    "Let's start trainer, train for 1 epoch and evaluate the final accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80fe155a",
   "metadata": {
    "id": "80fe155a",
    "outputId": "d4c955cb-2a0a-4b60-e5a5-382c576c4b03",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-02-09 06:29:42] epoch 0 step 0 : training batch loss 4.6263, training batch acc 0.0000\n",
      "[2023-02-09 06:30:04] epoch 0 step 10 : training batch loss 4.4492, training batch acc 6.2500\n",
      "[2023-02-09 06:30:24] epoch 0 step 20 : training batch loss 4.1754, training batch acc 19.5312\n",
      "[2023-02-09 06:30:43] epoch 0 step 30 : training batch loss 3.6634, training batch acc 32.8125\n",
      "[2023-02-09 06:31:03] epoch 0 step 40 : training batch loss 3.1619, training batch acc 35.1562\n",
      "[2023-02-09 06:31:23] epoch 0 step 50 : training batch loss 2.6438, training batch acc 49.2188\n",
      "[2023-02-09 06:31:43] epoch 0 step 60 : training batch loss 1.8105, training batch acc 67.1875\n",
      "[2023-02-09 06:32:03] epoch 0 step 70 : training batch loss 1.7793, training batch acc 65.6250\n",
      "[2023-02-09 06:32:22] epoch 0 step 80 : training batch loss 1.5857, training batch acc 64.8438\n",
      "[2023-02-09 06:32:42] epoch 0 step 90 : training batch loss 1.4807, training batch acc 64.0625\n",
      "[2023-02-09 06:33:02] epoch 0 step 100 : training batch loss 1.3621, training batch acc 67.1875\n",
      "[2023-02-09 06:33:21] epoch 0 step 110 : training batch loss 1.1836, training batch acc 70.3125\n",
      "[2023-02-09 06:33:40] epoch 0 step 120 : training batch loss 1.0249, training batch acc 78.1250\n",
      "[2023-02-09 06:33:59] epoch 0 step 130 : training batch loss 1.0308, training batch acc 68.7500\n",
      "[2023-02-09 06:34:18] epoch 0 step 140 : training batch loss 0.9496, training batch acc 75.7812\n",
      "[2023-02-09 06:34:39] epoch 0 step 150 : training batch loss 1.0322, training batch acc 71.8750\n",
      "[2023-02-09 06:34:59] epoch 0 step 160 : training batch loss 0.8484, training batch acc 70.3125\n",
      "[2023-02-09 06:35:18] epoch 0 step 170 : training batch loss 1.0306, training batch acc 73.4375\n",
      "[2023-02-09 06:35:38] epoch 0 step 180 : training batch loss 0.7997, training batch acc 77.3438\n",
      "[2023-02-09 06:35:58] epoch 0 step 190 : training batch loss 0.8612, training batch acc 78.1250\n",
      "[2023-02-09 06:36:18] epoch 0 step 200 : training batch loss 0.8505, training batch acc 75.0000\n",
      "[2023-02-09 06:36:38] epoch 0 step 210 : training batch loss 0.7995, training batch acc 75.7812\n",
      "[2023-02-09 06:36:58] epoch 0 step 220 : training batch loss 0.9957, training batch acc 71.8750\n",
      "[2023-02-09 06:37:18] epoch 0 step 230 : training batch loss 0.8048, training batch acc 74.2188\n",
      "[2023-02-09 06:37:38] epoch 0 step 240 : training batch loss 0.7453, training batch acc 78.1250\n",
      "[2023-02-09 06:37:57] epoch 0 step 250 : training batch loss 0.7133, training batch acc 78.9062\n",
      "[2023-02-09 06:38:17] epoch 0 step 260 : training batch loss 0.6496, training batch acc 85.1562\n",
      "[2023-02-09 06:38:37] epoch 0 step 270 : training batch loss 0.8175, training batch acc 78.1250\n",
      "[2023-02-09 06:38:57] epoch 0 step 280 : training batch loss 0.6630, training batch acc 78.9062\n",
      "[2023-02-09 06:39:17] epoch 0 step 290 : training batch loss 0.8648, training batch acc 73.4375\n",
      "[2023-02-09 06:39:36] epoch 0 step 300 : training batch loss 0.6844, training batch acc 82.0312\n",
      "[2023-02-09 06:39:56] epoch 0 step 310 : training batch loss 0.7841, training batch acc 75.0000\n",
      "[2023-02-09 06:40:16] epoch 0 step 320 : training batch loss 0.7563, training batch acc 80.4688\n",
      "[2023-02-09 06:40:36] epoch 0 step 330 : training batch loss 0.6680, training batch acc 76.5625\n",
      "[2023-02-09 06:40:54] epoch 0 step 340 : training batch loss 0.6698, training batch acc 81.2500\n",
      "[2023-02-09 06:41:15] epoch 0 step 350 : training batch loss 0.5934, training batch acc 82.8125\n",
      "[2023-02-09 06:41:35] epoch 0 step 360 : training batch loss 0.6993, training batch acc 79.6875\n",
      "[2023-02-09 06:41:54] epoch 0 step 370 : training batch loss 0.5614, training batch acc 83.5938\n",
      "[2023-02-09 06:42:14] epoch 0 step 380 : training batch loss 0.4925, training batch acc 82.8125\n",
      "[2023-02-09 06:42:33] epoch 0 step 390 : training batch loss 0.4368, training batch acc 88.7500\n",
      "step 0/79\n",
      "step 10/79\n",
      "step 20/79\n",
      "step 30/79\n",
      "step 40/79\n",
      "step 50/79\n",
      "step 60/79\n",
      "step 70/79\n",
      "[2023-02-09 06:43:33] evaluation loss 0.6653, evaluation acc 80.3900\n"
     ]
    }
   ],
   "source": [
    "trainer = Trainer(model, optimizer, scheduler)\n",
    "trainer.train(train_loader,validate_loader,max_epoch)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "include_colab_link": true,
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}