demo/builtin/minigo/MiniGo_DEMO.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "60ca1eff",
   "metadata": {},
   "source": [
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/e2eAIOK/blob/main/demo/builtin/minigo/MiniGo_DEMO.ipynb)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a358045c",
   "metadata": {},
   "source": [
    "## MiniGo DEMO"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0ed0530e",
   "metadata": {},
   "source": [
    "MiniGo is an opensource minimalist Go engine modeled after AlphaGo Zero, which is a system that learns how to play Go at a superhuman level given only the rules of the game.\n",
    "\n",
    "Brief history about DeepMind's AlphaGo/AlphaGo Zero/AlphaZero:\n",
    "* AlphaGo(w/ training dataset) --> AlphaGo Zero(wo/ training dataset) --> AlphaZero(General AI for all board games)\n",
    "\n",
    "<img src=\"./img/minigo_intro.JPG\" width=\"600\"/><figure>AlphaGo Zero</figure>\n",
    "\n",
    "Reference: https://www.newyorker.com/science/elements/how-the-artificial-intelligence-program-alphazero-mastered-its-games"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7e90b29d",
   "metadata": {},
   "source": [
    "# Content\n",
    "* [Overview](#Overview)\n",
    "    * [Model Architecture](#Model-Architecture)\n",
    "    * [Optimizations](#Optimizations)\n",
    "    * [Performance](#Performance)\n",
    "* [Getting Started](#Getting-Started)\n",
    "    * [1. Environment Setup](#1.-Environment-Setup)\n",
    "    * [2. Workflow Prepare](#2.-Workflow-Prepare)\n",
    "    * [3. Data Prepare](#3.-Data-Prepare)\n",
    "    * [4. Train](#4.-Train)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6f2480a9",
   "metadata": {},
   "source": [
    "# Overview\n",
    "## Model Architecture\n",
    "### Overall train loop architecture of MiniGo\n",
    "<img src=\"./img/minigo_model_arch.png\" width=\"600\"/><figure>MiniGo Model Architecture</figure>\n",
    "\n",
    "### Deep network architecture based on residual blocks\n",
    "<img src=\"./img/minigo_nn_arch.png\" width=\"600\"/><figure>Neural Network Architecture</figure>\n",
    "\n",
    "### MCTS architecture (Select->Expand->Evaluate->Backup/Backpropagation)\n",
    "<img src=\"./img/minigo_mcts_arch.png\" width=\"600\"/><figure>Monte Carlo Tree Search in MiniGo</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1de113a",
   "metadata": {},
   "source": [
    "## Optimizations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bc0baf0",
   "metadata": {},
   "source": [
    "### Distributed Monte Carlo Tree Search\n",
    "* Use the rsync tool to synchronize signal, model, flags files\n",
    "* Make full use of the physical cores of all nodes in HOSTLIST for distributed MCTS\n",
    "\n",
    "``` bash\n",
    "for NODE in $HOSTLIST; do \n",
    "  if [ $NODE != $ROOTNODE ]; then\n",
    "    ml_perf/scripts/loop.sh rsync -r --append --delete-before /root/mlperf/results-19/flags root@$NODE:/root/mlperf/results-19/ 2>&1 > /dev/null &\n",
    "    ml_perf/scripts/loop.sh rsync -r --append --delete-before /root/mlperf/results-19/signal root@$NODE:/root/mlperf/results-19/ 2>&1 > /dev/null &\n",
    "    ml_perf/scripts/loop.sh rsync -r --append --delete-before /root/mlperf/results-19/models root@$NODE:/root/mlperf/results-19/ 2>&1 > /dev/null &\n",
    "  fi\n",
    "done\n",
    "```\n",
    "\n",
    "### Enable early stop and evaluate target metric during train loop\n",
    "``` python\n",
    "def post_train(state):\n",
    "    model_path = os.path.join(FLAGS.model_dir, state.train_model_name)\n",
    "    selfplay_pb = model_path + '.pb'\n",
    "\n",
    "    if (FLAGS.hostlist != None):\n",
    "        golden_chunk_dir = FLAGS.golden_chunk_local_dir\n",
    "    else:\n",
    "        golden_chunk_dir = FLAGS.golden_chunk_dir\n",
    "\n",
    "    dual_net.optimize_graph(model_path + '.pb',\n",
    "        model_path,\n",
    "        FLAGS.quantization,\n",
    "        golden_chunk_dir + '/*.zz')\n",
    "\n",
    "    dst_minigo_file = model_path + '.minigo'\n",
    "    wait(checked_run([\n",
    "         'python3', 'convert.py',\n",
    "         '--flagfile={}'.format(os.path.join(FLAGS.flags_dir, 'architecture.flags')),\n",
    "         '--input_graph={}'.format(selfplay_pb),\n",
    "         '--dst={}'.format(dst_minigo_file)]))\n",
    "\n",
    "    if FLAGS.eval or state.iter_num > 50:\n",
    "        logging.info('started evaluation minigo model for iter %s of total %s, last iter winrate is %s', state.iter_num, FLAGS.iterations, state.win_rate)\n",
    "        minigo_model_path=f\"/root/mlperf/results-19/models/0000{state.iter_num}.minigo\"\n",
    "        state.win_rate=evaluate_model(minigo_model_path)\n",
    "        logging.info('finished evaluation minigo model for iter %s, winrate is %s', state.iter_num, state.win_rate)\n",
    "        if state.win_rate >= FLAGS.winrate:\n",
    "            logging.info('we have found minigo model better than target!!! iter %s winrate %s', state.iter_num, state.win_rate)\n",
    "            metric_file = open(\"../../result/metric.txt\", \"wt\")\n",
    "            n = metric_file.write(str(state.win_rate))\n",
    "            metric_file.close()\n",
    "    \n",
    "    #resume selfplay\n",
    "    os.remove(FLAGS.pause)\n",
    "\n",
    "\n",
    "def evaluate_model(eval_model_path):\n",
    "    processes = []\n",
    "    for i, device in enumerate(FLAGS.devices):\n",
    "        a = i * FLAGS.num_games // len(FLAGS.devices)\n",
    "        b = (i + 1) * FLAGS.num_games // len(FLAGS.devices)\n",
    "        num_games = b - a;\n",
    "        \n",
    "        env = os.environ.copy()\n",
    "        env['CUDA_VISIBLE_DEVICES'] = device\n",
    "        processes.append(checked_run([\n",
    "            'numactl',\n",
    "            '--physcpubind={}'.format(i),\n",
    "            'bazel-bin/cc/eval',\n",
    "            '--flagfile={}'.format(os.path.join(FLAGS.flags_dir, 'eval.flags')),\n",
    "            '--eval_model={}'.format(eval_model_path),\n",
    "            '--target_model={}'.format(FLAGS.target),\n",
    "            '--sgf_dir={}'.format(FLAGS.sgf_dir),\n",
    "            '--parallel_games={}'.format(num_games),\n",
    "            '--eval_device=cpu',\n",
    "            '--target_device=cpu',\n",
    "            '--verbose=false'], env, False))\n",
    "    all_output = wait(processes)\n",
    "\n",
    "    total_wins = 0\n",
    "    total_num_games = 0\n",
    "    for output in all_output:\n",
    "        lines = output.split('\\n')\n",
    "\n",
    "        eval_stats, target_stats = parse_win_stats_table(lines[-7:])\n",
    "        num_games = eval_stats.total_wins + target_stats.total_wins\n",
    "        total_wins += eval_stats.total_wins\n",
    "        total_num_games += num_games\n",
    "\n",
    "    win_rate = total_wins / total_num_games\n",
    "    logging.info('Win rate %s vs %s: %.3f', eval_stats.model_name,\n",
    "                 target_stats.model_name, win_rate)\n",
    "    \n",
    "    return win_rate\n",
    "\n",
    "def parse_win_stats_table(lines):\n",
    "    result = []\n",
    "    while True:\n",
    "        # Find the start of the win stats table.\n",
    "        assert len(lines) > 1\n",
    "        if 'Black' in lines[0] and 'White' in lines[0] and 'passes' in lines[1]:\n",
    "            break\n",
    "        lines = lines[1:]\n",
    "\n",
    "    # Parse the expected number of lines from the table.\n",
    "    for line in lines[2:4]:\n",
    "        result.append(WinStats(line))\n",
    "\n",
    "    return result\n",
    "```\n",
    "\n",
    "### Fine-tune MCTS hyperparameters to trade-off exploration and exploitation\n",
    "* Trade-off MCTS exploration and exploitation:\n",
    "       If fastplay_frequency > 0, tree search is modified as follows:\n",
    "         - Each move is either a \"low-readout\" fast move, or a full, slow move.\n",
    "         The percent of fast moves corresponds to \"fastplay_frequency\"\n",
    "         - A \"fast\" move will:\n",
    "           - Reuse the tree\n",
    "           - Not mix noise in at root\n",
    "           - Only perform 'fastplay_readouts' readouts.\n",
    "           - Not be used as a training target.\n",
    "         - A \"slow\" move will:\n",
    "           - Clear the tree (*not* the cache).\n",
    "           - Mix in dirichlet noise\n",
    "           - Perform 'num_readouts' readouts.\n",
    "           - Be noted in the Game object, to be written as a training example.\n",
    "* Main application class of MiniGo MCTS Selfplayer:\n",
    "``` cpp\n",
    "  Selfplayer(\n",
    "          /* Inference flags */\n",
    "          std::string n_model,\n",
    "          int32_t cache_size_mb,\n",
    "          int32_t cache_shards,\n",
    "          /* Tree search flags */\n",
    "          int32_t num_readouts,\n",
    "          double fastplay_frequency,\n",
    "          int32_t fastplay_readouts,\n",
    "          int32_t virtual_losses,\n",
    "          double dirichlet_alpha,\n",
    "          double noise_mix,\n",
    "          double value_init_penalty,\n",
    "          bool target_pruning,\n",
    "          double policy_softmax_temp,\n",
    "          bool allow_pass,\n",
    "          int32_t restrict_pass_alive_play_threshold,\n",
    "          /* Threading flags. */\n",
    "          int32_t num_selfplay_threads,\n",
    "          int32_t num_parallel_search,\n",
    "          int32_t num_parallel_inference,\n",
    "          int32_t num_concurrent_games_per_thread,\n",
    "          /* Game flags. */\n",
    "          uint64_t seed,    \n",
    "          double min_resign_threshold,\n",
    "          double max_resign_threshold,\n",
    "          double disable_resign_pct,\n",
    "          int32_t num_games,\n",
    "          bool run_forever,\n",
    "          std::string abort_file,\n",
    "          /* Output flags. */\n",
    "          double holdout_pct,\n",
    "          std::string output_dir,\n",
    "          std::string holdout_dir,\n",
    "          std::string sgf_dir,\n",
    "          bool verbose,\n",
    "          int32_t num_output_threads,\n",
    "          /* Sample Fraction for output. */\n",
    "          double sample_frac);\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa1539a9",
   "metadata": {},
   "source": [
    "## HPO with SDA (Smart Democratization Advisor)\n",
    "SDA config\n",
    "```\n",
    "model_parameter:\n",
    "  project: sda\n",
    "  experiment: minigo\n",
    "  parameters:\n",
    "    - name: train_batch_size\n",
    "      grid:\n",
    "        - 512\n",
    "        - 1024\n",
    "        - 2048\n",
    "        - 4096\n",
    "        - 8192\n",
    "      type: int\n",
    "  metrics:\n",
    "  - name: winrate\n",
    "    objective: maximize\n",
    "  - name: training_time\n",
    "    strategy: optimize\n",
    "    objective: minimize\n",
    "```\n",
    "\n",
    "request suggestions from SDA\n",
    "\n",
    "```python\n",
    "suggestion = self.conn.experiments(self.experiment.id).suggestions().create()\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "92f794fe",
   "metadata": {},
   "source": [
    "## Framework related optimization\n",
    "### Enable parallel MCTS(Monte Carlo Tree Search) across all physical cores in the cluster\n",
    "Remark:\n",
    "* Use rsync to synchronize signal, trained model, and mcts generated datset\n",
    "* Enable numa binding to fully utilize all physical cores in the cluster\n",
    "<img src=\"./img/minigo_optimized_system_arch.JPG\" width=\"600\"/><figure>Optimized MiniGo System Architecture</figure>\n",
    "\n",
    "### Enable early stop during the train loop to leverage fast converge\n",
    "Remark:\n",
    "* 71 iteration number reduced to 55 iteration number\n",
    "* Baseline\n",
    "** Fixed iteration number and cannot guarantee convergence\n",
    "* Our Optimized MiniGo\n",
    "** Guarantee convergence and can fast converge when meet target metric\n",
    "\n",
    "### Fine-tuning MCTS hyperparameters\n",
    "Remark:\n",
    "* fine-tune MCTS hyperparameters, including concurrent_games_per_thread, fastplay_frequency, num_readouts, fastplay_readouts\n",
    "* best found MCTS hyperparameters:\n",
    "** concurrent_games_per_thread=6\n",
    "** fastplay_frequency=0.82\n",
    "** num_readouts=600\n",
    "** fastplay_readouts=60\n",
    "* MCTS performance without finetune:\n",
    "<img src=\"./img/mcts_baseline_speed.JPG\" width=\"600\"/><figure>Baseline MCTS</figure>\n",
    "* MCTS performance with finetune:\n",
    "<img src=\"./img/mcts_tuned_speed.JPG\" width=\"600\"/><figure>Finetuned MCTS</figure>\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "34348b05",
   "metadata": {},
   "source": [
    "## Performance\n",
    "\n",
    "<img src=\"./img/minigo_perf.png\" width=\"600\"/>\n",
    "\n",
    "* Distributed training with HW scaling delivered 3.57 speedup from 1 node to 4 nodes\n",
    "* Parallel selfplay and enable early stop delivered 2.50x speedup, and 8.92x speedup over baseline\n",
    "* MCTS (Monte Carlo Tree Search) hyper parameter tuning delivered 1.46x speedup, and 13.06x speedup over baseline\n",
    "    * fastplay_frequency=0.82, num_readouts=600, fastplay_readouts=60, model converged at 55 iteration\n",
    "* baseline converged and stopped at 71 epoch, optimized model converged and stopped at 55 epoch"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8e91b6e7",
   "metadata": {},
   "source": [
    "# Getting Started\n",
    "* [1. Environment Setup](#1.-Environment-Setup)\n",
    "* [2. Workflow Prepare](#2.-Workflow-Prepare)\n",
    "* [3. Data Prepare](#3.-Data-Prepare)\n",
    "* [4. Train](#4.-Train)\n",
    "\n",
    "Notes: This demo cannot run directly on colab, since it requires you to build runtime libraries on your physical machine. In order to run this demo, please follow `Environment Setup` and `Workflow Prepare` section for pre-requirements."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3420fdff",
   "metadata": {},
   "source": [
    "## 1. Environment Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f927d73b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Noted: MiniGo is only runnable with baremetal env, so we won't provide docker option\n",
    "%%bash\n",
    "# prepare model codes\n",
    "bash workflow_prepare_minigo.sh\n",
    "\n",
    "# Install e2eAIOK-sda\n",
    "pip install e2eAIOK-sda --pre\n",
    "\n",
    "# Install bazel 3.0.0\n",
    "BAZEL_VERSION=3.0.0\n",
    "wget https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION}/bazel-${BAZEL_VERSION}-installer-linux-x86_64.sh\n",
    "chmod 755 bazel-${BAZEL_VERSION}-installer-linux-x86_64.sh\n",
    "./bazel-${BAZEL_VERSION}-installer-linux-x86_64.sh\n",
    "\n",
    "# Build and install specified Intel-Tensorflow\n",
    "git clone https://github.com/Intel-tensorflow/tensorflow.git\n",
    "cd tensorflow\n",
    "git checkout 896a070312136fe944fd7a905e72f86dee9771f6\n",
    "# Apply compatibility patch for tensorflow-io\n",
    "git apply ../patches/compat.patch\n",
    "# Use all defaults\n",
    "./configure\n",
    "bazel build --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0 --copt=-O3 --copt=-Wformat \\\n",
    "--copt=-Wformat-security --copt=-fstack-protector --copt=-fPIC --copt=-fpic \\\n",
    "--linkopt=-znoexecstack --linkopt=-zrelro --linkopt=-znow --linkopt=-fstack-protector \\\n",
    "--config=mkl --define build_with_mkl_dnn_v1_only=true --copt=-DENABLE_INTEL_MKL_BFLOAT16 \\\n",
    "--copt=-march=haswell //tensorflow/tools/pip_package:build_pip_package\n",
    "# Build and install python wheel\n",
    "bazel-bin/tensorflow/tools/pip_package/build_pip_package ./tensorflow_pkg/\n",
    "pip install ./tensorflow_pkg/tensorflow-2.2.0-cp36-cp36m-linux_x86_64.whl\n",
    "\n",
    "# Install Tensorflow-io\n",
    "pip install \"tensorflow-io==0.13.0\" --no-deps\n",
    "\n",
    "# Install horovod\n",
    "# Note, please ensure that Intel MPI and GCC have been configured correctly before compling horovod\n",
    "pip install horovod==0.19.1 --no-cache-dir\n",
    "\n",
    "cd ..    \n",
    "# Install bazel 0.24.1\n",
    "BAZEL_VERSION=0.24.1\n",
    "wget https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION}/bazel-${BAZEL_VERSION}-installer-linux-x86_64.sh\n",
    "chmod 755 bazel-${BAZEL_VERSION}-installer-linux-x86_64.sh\n",
    "./bazel-${BAZEL_VERSION}-installer-linux-x86_64.sh\n",
    "\n",
    "# Compile TensorFlow C++ libraries (enter to use default config)\n",
    "./cc/configure_tensorflow.sh"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ab7d8249",
   "metadata": {},
   "source": [
    "Remark:\n",
    "* Since MiniGo is a reinforcement learning model, it generates training dataset in each iteration during train loop and doesn't need dataset. We evaluate winrate with target model(based on MLPerf submission) and final winrate>=0.5, and all our optimizations guarantee that target metric.\n",
    "\n",
    "* public reference on AlphaGo Zero: https://arxiv.org/abs/1712.01815\n",
    "\n",
    "* public reference on MiniGo: https://openreview.net/forum?id=H1eerhIpLV"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "fb018656",
   "metadata": {},
   "source": [
    "## 2. Workflow Prepare\n",
    "\n",
    "a simple example of config file, one can refer to conf/e2eaiok_defaults_minigo_example.conf for whole config file\n",
    "\n",
    "```yaml\n",
    "### GLOBAL SETTINGS ###\n",
    "observation_budget: 1\n",
    "save_path: ./result/\n",
    "hosts:\n",
    "- localhost\n",
    "rootnode: localhost\n",
    "\n",
    "## model_parameter ###\n",
    "model_parameter:\n",
    "  project: e2eaiok\n",
    "  experiment: minigo\n",
    "  parameters:\n",
    "    - name: train_batch_size\n",
    "      grid:\n",
    "        - 512\n",
    "        - 1024\n",
    "        - 2048\n",
    "        - 4096\n",
    "        - 8192\n",
    "      type: int\n",
    "  metrics:\n",
    "  - name: winrate\n",
    "    objective: maximize\n",
    "  - name: training_time\n",
    "    strategy: optimize\n",
    "    objective: minimize\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "695d2a60",
   "metadata": {},
   "source": [
    "## 3. Data Prepare\n",
    "\n",
    "No dataset needed for MiniGo workload"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "92cbf2f4",
   "metadata": {},
   "source": [
    "## 4. Train\n",
    "Noted: Below performance result is using sample dataset and small iterations to demonstrate its function. The actual performance result please refers to the [performance section](#performance).\n",
    "\n",
    "Edit config file to control SDA process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ad5636da",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " \n",
      ":: initializing oneAPI environment ...\n",
      "   bash: BASH_VERSION = 4.4.20(1)-release\n",
      "   args: Using \"$@\" for setvars.sh arguments: --force\n",
      ":: clck -- latest\n",
      ":: compiler -- latest\n",
      ":: debugger -- latest\n",
      ":: dev-utilities -- latest\n",
      ":: inspector -- latest\n",
      ":: itac -- latest\n",
      ":: mpi -- latest\n",
      ":: tbb -- latest\n",
      ":: oneAPI environment initialized ::\n",
      " \n",
      "2022-09-15 16:48:11,938 - HYDRO.AI.SDA - INFO - ### Ready to submit current task  ###\n",
      "2022-09-15 16:48:11,938 - HYDRO.AI.SDA - INFO - Model Advisor created\n",
      "2022-09-15 16:48:11,938 - HYDRO.AI.SDA - INFO - Start to init sigopt\n",
      "experiment: minigo\n",
      "metrics:\n",
      "- name: winrate\n",
      "  objective: maximize\n",
      "- name: training_time\n",
      "  objective: minimize\n",
      "  strategy: optimize\n",
      "observation_budget: 1\n",
      "parameters:\n",
      "- grid:\n",
      "  - 512\n",
      "  - 1024\n",
      "  - 2048\n",
      "  - 4196\n",
      "  - 8192\n",
      "  name: train_batch_size\n",
      "  type: int\n",
      "project: hydro.ai\n",
      "2022-09-15 16:48:12,810 - HYDRO.AI.SDA - INFO - model parameter initialized\n",
      "2022-09-15 16:48:12,810 - HYDRO.AI.SDA - INFO - start to launch training\n",
      "2022-09-15 16:48:12,810 - sigopt - INFO - Experiment budget: 1\n",
      "2022-09-15 16:48:13,685 - sigopt - INFO - training launch command: HOSTLIST=sr141,sr142 ROOTNODE=sr141 checkpoint_dir=/root/zheng/dataset/minigo/checkpoints/mlperf07 target=/root/zheng/dataset/minigo/target/target.minigo ml_perf/scripts/run_minigo.sh 19\n",
      "GCC Version: 8.4.0\n",
      "GCC 8.4.0: OK\n",
      "exporting BAZEL_BUILD_OPTS_BASIC=--cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0         --copt=-march=haswell         --copt=-mtune=broadwell                 --copt=-O3\n",
      "exporting BAZEL_BUILD_OPTS=--cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0         --copt=-march=haswell         --copt=-mtune=broadwell                 --copt=-O3 --copt=-Wformat                 --copt=-Wformat-security                 --copt=-fstack-protector                 --copt=-fPIC                 --copt=-fpic                 --linkopt=-znoexecstack                 --linkopt=-zrelro                 --linkopt=-znow                 --linkopt=-fstack-protector                 --linkopt=-pie\n",
      "exporting BAZEL_MKL_BUILD_OPTS_BASIC=--config=mkl --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0         --copt=-march=haswell         --copt=-mtune=broadwell                 --copt=-O3\n",
      "exporting BAZEL_SECURE_MKL_BUILD_OPTS=--config=mkl --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0         --copt=-march=haswell         --copt=-mtune=broadwell                 --copt=-O3 --copt=-Wformat                 --copt=-Wformat-security                 --copt=-fstack-protector                 --copt=-fPIC                 --copt=-fpic                 --linkopt=-znoexecstack                 --linkopt=-zrelro                 --linkopt=-znow                 --linkopt=-fstack-protector                 --linkopt=-pie\n",
      "+++ [[ -v HOSTLIST ]]\n",
      "+++ mpirun -bootstrap ssh -ppn 1 -hosts sr141,sr142 ml_perf/scripts/cc_libgen_parallel_selfplay.sh --base_dir=/root/mlperf/results-19 --board_size=19\n",
      "GCC Version: 8.4.0\n",
      "GCC 8.4.0: OK\n",
      "exporting BAZEL_BUILD_OPTS_BASIC=--cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0         --copt=-march=haswell         --copt=-mtune=broadwell                 --copt=-O3\n",
      "exporting BAZEL_BUILD_OPTS=--cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0         --copt=-march=haswell         --copt=-mtune=broadwell                 --copt=-O3 --copt=-Wformat                 --copt=-Wformat-security                 --copt=-fstack-protector                 --copt=-fPIC                 --copt=-fpic                 --linkopt=-znoexecstack                 --linkopt=-zrelro                 --linkopt=-znow                 --linkopt=-fstack-protector                 --linkopt=-pie\n",
      "exporting BAZEL_MKL_BUILD_OPTS_BASIC=--config=mkl --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0         --copt=-march=haswell         --copt=-mtune=broadwell                 --copt=-O3\n",
      "exporting BAZEL_SECURE_MKL_BUILD_OPTS=--config=mkl --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0         --copt=-march=haswell         --copt=-mtune=broadwell                 --copt=-O3 --copt=-Wformat                 --copt=-Wformat-security                 --copt=-fstack-protector                 --copt=-fPIC                 --copt=-fpic                 --linkopt=-znoexecstack                 --linkopt=-zrelro                 --linkopt=-znow                 --linkopt=-fstack-protector                 --linkopt=-pie\n",
      "GCC Version: 8.4.0\n",
      "GCC 8.4.0: OK\n",
      "exporting BAZEL_BUILD_OPTS_BASIC=--cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0         --copt=-march=haswell         --copt=-mtune=broadwell                 --copt=-O3\n",
      "exporting BAZEL_BUILD_OPTS=--cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0         --copt=-march=haswell         --copt=-mtune=broadwell                 --copt=-O3 --copt=-Wformat                 --copt=-Wformat-security                 --copt=-fstack-protector                 --copt=-fPIC                 --copt=-fpic                 --linkopt=-znoexecstack                 --linkopt=-zrelro                 --linkopt=-znow                 --linkopt=-fstack-protector                 --linkopt=-pie\n",
      "exporting BAZEL_MKL_BUILD_OPTS_BASIC=--config=mkl --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0         --copt=-march=haswell         --copt=-mtune=broadwell                 --copt=-O3\n",
      "exporting BAZEL_SECURE_MKL_BUILD_OPTS=--config=mkl --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0         --copt=-march=haswell         --copt=-mtune=broadwell                 --copt=-O3 --copt=-Wformat                 --copt=-Wformat-security                 --copt=-fstack-protector                 --copt=-fPIC                 --copt=-fpic                 --linkopt=-znoexecstack                 --linkopt=-zrelro                 --linkopt=-znow                 --linkopt=-fstack-protector                 --linkopt=-pie\n",
      "++ hostname\n",
      "sr142\n",
      "+ echo sr142\n",
      "+ bazel build -c opt --config=mkl --action_env=PATH --action_env=LD_LIBRARY_PATH --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0 --copt=-march=haswell --copt=-mtune=broadwell --copt=-O3 --copt=-Wformat --copt=-Wformat-security --copt=-fstack-protector --copt=-fPIC --copt=-fpic --linkopt=-znoexecstack --linkopt=-zrelro --linkopt=-znow --linkopt=-fstack-protector --linkopt=-pie --copt=-DINTEL_MKLDNN --copt=-O3 --define=board_size=19 --define=tf=1 cc:concurrent_selfplay cc:sample_records cc:eval\n",
      "++ hostname\n",
      "sr141\n",
      "+ echo sr141\n",
      "+ bazel build -c opt --config=mkl --action_env=PATH --action_env=LD_LIBRARY_PATH --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0 --copt=-march=haswell --copt=-mtune=broadwell --copt=-O3 --copt=-Wformat --copt=-Wformat-security --copt=-fstack-protector --copt=-fPIC --copt=-fpic --linkopt=-znoexecstack --linkopt=-zrelro --linkopt=-znow --linkopt=-fstack-protector --linkopt=-pie --copt=-DINTEL_MKLDNN --copt=-O3 --define=board_size=19 --define=tf=1 cc:concurrent_selfplay cc:sample_records cc:eval\n",
      "INFO: Options provided by the client:\n",
      "  Inherited 'common' options: --isatty=0 --terminal_columns=80\n",
      "INFO: Reading rc options for 'build' from /root/e2eAIOK/modelzoo/minigo/.bazelrc:\n",
      "  'build' options: --define=tf=1\n",
      "INFO: Reading rc options for 'build' from /root/e2eAIOK/modelzoo/minigo/tf_configure.bazelrc:\n",
      "  'build' options: --action_env PYTHON_BIN_PATH=/root/anaconda3/envs/minigo/bin/python --action_env PYTHON_LIB_PATH=/root/anaconda3/envs/minigo/lib/python3.6/site-packages --python_path=/root/anaconda3/envs/minigo/bin/python --action_env TF_CONFIGURE_IOS=0\n",
      "INFO: Reading rc options for 'build' from /root/e2eAIOK/modelzoo/minigo/tensorflow.bazelrc:\n",
      "  'build' options: --apple_platform_type=macos --define framework_shared_object=true --define open_source_build=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone --strategy=Genrule=standalone -c opt --announce_rc --define=grpc_no_ares=true --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include\n",
      "INFO: Found applicable config definition build:mkl in file /root/e2eAIOK/modelzoo/minigo/tensorflow.bazelrc: --define=build_with_mkl=true --define=enable_mkl=true --define=tensorflow_mkldnn_contraction_kernel=0 -c opt\n",
      "INFO: Options provided by the client:\n",
      "  Inherited 'common' options: --isatty=0 --terminal_columns=80\n",
      "INFO: Reading rc options for 'build' from /root/e2eAIOK/modelzoo/minigo/.bazelrc:\n",
      "  'build' options: --define=tf=1\n",
      "INFO: Reading rc options for 'build' from /root/e2eAIOK/modelzoo/minigo/tf_configure.bazelrc:\n",
      "  'build' options: --action_env PYTHON_BIN_PATH=/root/anaconda3/envs/minigo/bin/python --action_env PYTHON_LIB_PATH=/root/anaconda3/envs/minigo/lib/python3.6/site-packages --python_path=/root/anaconda3/envs/minigo/bin/python --action_env TF_CONFIGURE_IOS=0\n",
      "INFO: Reading rc options for 'build' from /root/e2eAIOK/modelzoo/minigo/tensorflow.bazelrc:\n",
      "  'build' options: --apple_platform_type=macos --define framework_shared_object=true --define open_source_build=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone --strategy=Genrule=standalone -c opt --announce_rc --define=grpc_no_ares=true --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include\n",
      "INFO: Found applicable config definition build:mkl in file /root/e2eAIOK/modelzoo/minigo/tensorflow.bazelrc: --define=build_with_mkl=true --define=enable_mkl=true --define=tensorflow_mkldnn_contraction_kernel=0 -c opt\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading: \n",
      "Loading: 0 packages loaded\n",
      "Analyzing: 3 targets (0 packages loaded, 0 targets configured)\n",
      "INFO: Analysed 3 targets (0 packages loaded, 0 targets configured).\n",
      "INFO: Found 3 targets...\n",
      "Loading: \n",
      "Loading: 0 packages loaded\n",
      "Analyzing: 3 targets (0 packages loaded, 0 targets configured)\n",
      "INFO: Analysed 3 targets (0 packages loaded, 0 targets configured).\n",
      "INFO: Found 3 targets...\n",
      "[0 / 1] [-----] BazelWorkspaceStatusAction stable-status.txt\n",
      "INFO: Elapsed time: 0.336s, Critical Path: 0.01s\n",
      "INFO: 0 processes.\n",
      "INFO: Build completed successfully, 1 total action\n",
      "INFO: Build completed successfully, 1 total action\n",
      "[0 / 1] [-----] BazelWorkspaceStatusAction stable-status.txt\n",
      "INFO: Elapsed time: 0.374s, Critical Path: 0.01s\n",
      "INFO: 0 processes.\n",
      "INFO: Build completed successfully, 1 total action\n",
      "INFO: Build completed successfully, 1 total action\n",
      "+++ for var_name in flag_dir golden_chunk_dir golden_chunk_tmp_dir holdout_dir log_dir model_dir selfplay_dir sgf_dir work_dir signal_dir\n",
      "+++ clean_dir /root/mlperf/results-19/flags\n",
      "+++ dir=/root/mlperf/results-19/flags\n",
      "+++ [[ /root/mlperf/results-19/flags == gs://* ]]\n",
      "+++ [[ /root/mlperf/results-19/flags = /root/mlperf/* ]]\n",
      "+++ mkdir -p /root/mlperf/results-19/flags\n",
      "+++ rm -rf '/root/mlperf/results-19/flags/*'\n",
      "+++ for var_name in flag_dir golden_chunk_dir golden_chunk_tmp_dir holdout_dir log_dir model_dir selfplay_dir sgf_dir work_dir signal_dir\n",
      "+++ clean_dir /root/mlperf/results-19/data/golden_chunks\n",
      "+++ dir=/root/mlperf/results-19/data/golden_chunks\n",
      "+++ [[ /root/mlperf/results-19/data/golden_chunks == gs://* ]]\n",
      "+++ [[ /root/mlperf/results-19/data/golden_chunks = /root/mlperf/* ]]\n",
      "+++ mkdir -p /root/mlperf/results-19/data/golden_chunks\n",
      "+++ rm -rf '/root/mlperf/results-19/data/golden_chunks/*'\n",
      "+++ for var_name in flag_dir golden_chunk_dir golden_chunk_tmp_dir holdout_dir log_dir model_dir selfplay_dir sgf_dir work_dir signal_dir\n",
      "+++ clean_dir /tmp/golden_chunks_tmp\n",
      "+++ dir=/tmp/golden_chunks_tmp\n",
      "+++ [[ /tmp/golden_chunks_tmp == gs://* ]]\n",
      "+++ [[ /tmp/golden_chunks_tmp = /root/mlperf/* ]]\n",
      "+++ [[ /tmp/golden_chunks_tmp = /tmp/* ]]\n",
      "+++ mkdir -p /tmp/golden_chunks_tmp\n",
      "+++ rm -rf '/tmp/golden_chunks_tmp/*'\n",
      "+++ for var_name in flag_dir golden_chunk_dir golden_chunk_tmp_dir holdout_dir log_dir model_dir selfplay_dir sgf_dir work_dir signal_dir\n",
      "+++ clean_dir /root/mlperf/results-19/data/holdout\n",
      "+++ dir=/root/mlperf/results-19/data/holdout\n",
      "+++ [[ /root/mlperf/results-19/data/holdout == gs://* ]]\n",
      "+++ [[ /root/mlperf/results-19/data/holdout = /root/mlperf/* ]]\n",
      "+++ mkdir -p /root/mlperf/results-19/data/holdout\n",
      "+++ rm -rf '/root/mlperf/results-19/data/holdout/*'\n",
      "+++ for var_name in flag_dir golden_chunk_dir golden_chunk_tmp_dir holdout_dir log_dir model_dir selfplay_dir sgf_dir work_dir signal_dir\n",
      "+++ clean_dir /root/mlperf/results-19/logs\n",
      "+++ dir=/root/mlperf/results-19/logs\n",
      "+++ [[ /root/mlperf/results-19/logs == gs://* ]]\n",
      "+++ [[ /root/mlperf/results-19/logs = /root/mlperf/* ]]\n",
      "+++ mkdir -p /root/mlperf/results-19/logs\n",
      "+++ rm -rf '/root/mlperf/results-19/logs/*'\n",
      "+++ for var_name in flag_dir golden_chunk_dir golden_chunk_tmp_dir holdout_dir log_dir model_dir selfplay_dir sgf_dir work_dir signal_dir\n",
      "+++ clean_dir /root/mlperf/results-19/models\n",
      "+++ dir=/root/mlperf/results-19/models\n",
      "+++ [[ /root/mlperf/results-19/models == gs://* ]]\n",
      "+++ [[ /root/mlperf/results-19/models = /root/mlperf/* ]]\n",
      "+++ mkdir -p /root/mlperf/results-19/models\n",
      "+++ rm -rf '/root/mlperf/results-19/models/*'\n",
      "+++ for var_name in flag_dir golden_chunk_dir golden_chunk_tmp_dir holdout_dir log_dir model_dir selfplay_dir sgf_dir work_dir signal_dir\n",
      "+++ clean_dir /root/mlperf/results-19/data/selfplay\n",
      "+++ dir=/root/mlperf/results-19/data/selfplay\n",
      "+++ [[ /root/mlperf/results-19/data/selfplay == gs://* ]]\n",
      "+++ [[ /root/mlperf/results-19/data/selfplay = /root/mlperf/* ]]\n",
      "+++ mkdir -p /root/mlperf/results-19/data/selfplay\n",
      "+++ rm -rf '/root/mlperf/results-19/data/selfplay/*'\n",
      "+++ for var_name in flag_dir golden_chunk_dir golden_chunk_tmp_dir holdout_dir log_dir model_dir selfplay_dir sgf_dir work_dir signal_dir\n",
      "+++ clean_dir /root/mlperf/results-19/sgf\n",
      "+++ dir=/root/mlperf/results-19/sgf\n",
      "+++ [[ /root/mlperf/results-19/sgf == gs://* ]]\n",
      "+++ [[ /root/mlperf/results-19/sgf = /root/mlperf/* ]]\n",
      "+++ mkdir -p /root/mlperf/results-19/sgf\n",
      "+++ rm -rf '/root/mlperf/results-19/sgf/*'\n",
      "+++ for var_name in flag_dir golden_chunk_dir golden_chunk_tmp_dir holdout_dir log_dir model_dir selfplay_dir sgf_dir work_dir signal_dir\n",
      "+++ clean_dir /root/mlperf/results-19/work_dir\n",
      "+++ dir=/root/mlperf/results-19/work_dir\n",
      "+++ [[ /root/mlperf/results-19/work_dir == gs://* ]]\n",
      "+++ [[ /root/mlperf/results-19/work_dir = /root/mlperf/* ]]\n",
      "+++ mkdir -p /root/mlperf/results-19/work_dir\n",
      "+++ rm -rf '/root/mlperf/results-19/work_dir/*'\n",
      "+++ for var_name in flag_dir golden_chunk_dir golden_chunk_tmp_dir holdout_dir log_dir model_dir selfplay_dir sgf_dir work_dir signal_dir\n",
      "+++ clean_dir /root/mlperf/results-19/signal\n",
      "+++ dir=/root/mlperf/results-19/signal\n",
      "+++ [[ /root/mlperf/results-19/signal == gs://* ]]\n",
      "+++ [[ /root/mlperf/results-19/signal = /root/mlperf/* ]]\n",
      "+++ mkdir -p /root/mlperf/results-19/signal\n",
      "+++ rm -rf '/root/mlperf/results-19/signal/*'\n",
      "+++ rm -f /root/mlperf/results-19/signal/abort\n",
      "+++ [[ -v HOSTLIST ]]\n",
      "+++ echo Create golden chunk dir for each node\n",
      "Create golden chunk dir for each node\n",
      "+++ mpirun -bootstrap ssh -ppn 1 -hosts sr141,sr142 mkdir -p /tmp/golden_chunks\n",
      "+++ mpirun -bootstrap ssh -ppn 1 -hosts sr141,sr142 rm -f '/tmp/golden_chunks/*'\n",
      "+++ echo Create local selfplay dir for each node\n",
      "Create local selfplay dir for each node\n",
      "+++ mpirun -bootstrap ssh -ppn 1 -hosts sr141,sr142 mkdir -p /tmp/selfplay_local/\n",
      "+++ mpirun -bootstrap ssh -ppn 1 -hosts sr141,sr142 rm -rf '/tmp/selfplay_local//*'\n",
      "+++ echo Create signal dir for each node\n",
      "Create signal dir for each node\n",
      "+++ mpirun -bootstrap ssh -ppn 1 -hosts sr141,sr142 mkdir -p /tmp/signal\n",
      "+++ mpirun -bootstrap ssh -ppn 1 -hosts sr141,sr142 rm -f '/tmp/signal/*'\n",
      "+++ BOARD_SIZE=19\n",
      "+++ python3 ml_perf/init_from_checkpoint.py --checkpoint_dir=/root/zheng/dataset/minigo/checkpoints/mlperf07 --selfplay_dir=/root/mlperf/results-19/data/selfplay --work_dir=/root/mlperf/results-19/work_dir --flag_dir=/root/mlperf/results-19/flags --board_size=19\n",
      "Copying /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay to /root/mlperf/results-19/data/selfplay\n",
      "Make dir /root/mlperf/results-19/data/selfplay/./000013\n",
      "Make dir /root/mlperf/results-19/data/selfplay/./000014\n",
      "Make dir /root/mlperf/results-19/data/selfplay/./000015\n",
      "Make dir /root/mlperf/results-19/data/selfplay/./000016\n",
      "Make dir /root/mlperf/results-19/data/selfplay/./000017\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000016/0\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000016/1\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000016/2\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000016/3\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000016/4\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000016/5\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000016/6\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000016/7\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000016/4/2019-12-24-13\n",
      "Copying 1004 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000016/4/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000016/4/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000016/2/2019-12-24-13\n",
      "Copying 1284 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000016/2/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000016/2/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000016/0/2019-12-24-13\n",
      "Copying 1117 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000016/0/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000016/0/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000016/6/2019-12-24-13\n",
      "Copying 1153 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000016/6/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000016/6/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000016/5/2019-12-24-13\n",
      "Copying 1213 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000016/5/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000016/5/2019-12-24-13\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Make dir /root/mlperf/results-19/data/selfplay/000016/7/2019-12-24-13\n",
      "Copying 1129 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000016/7/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000016/7/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000016/1/2019-12-24-13\n",
      "Copying 1269 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000016/1/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000016/1/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000016/3/2019-12-24-13\n",
      "Copying 1067 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000016/3/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000016/3/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/0\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/1\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/2\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/3\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/4\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/5\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/6\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/7\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/4/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/4/2019-12-24-14\n",
      "Copying 138 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000017/4/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000017/4/2019-12-24-13\n",
      "Copying 898 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000017/4/2019-12-24-14 to /root/mlperf/results-19/data/selfplay/000017/4/2019-12-24-14\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/2/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/2/2019-12-24-14\n",
      "Copying 162 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000017/2/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000017/2/2019-12-24-13\n",
      "Copying 1102 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000017/2/2019-12-24-14 to /root/mlperf/results-19/data/selfplay/000017/2/2019-12-24-14\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/0/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/0/2019-12-24-14\n",
      "Copying 167 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000017/0/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000017/0/2019-12-24-13\n",
      "Copying 937 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000017/0/2019-12-24-14 to /root/mlperf/results-19/data/selfplay/000017/0/2019-12-24-14\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/6/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/6/2019-12-24-14\n",
      "Copying 165 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000017/6/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000017/6/2019-12-24-13\n",
      "Copying 1005 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000017/6/2019-12-24-14 to /root/mlperf/results-19/data/selfplay/000017/6/2019-12-24-14\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/5/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/5/2019-12-24-14\n",
      "Copying 161 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000017/5/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000017/5/2019-12-24-13\n",
      "Copying 1044 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000017/5/2019-12-24-14 to /root/mlperf/results-19/data/selfplay/000017/5/2019-12-24-14\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/7/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/7/2019-12-24-14\n",
      "Copying 155 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000017/7/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000017/7/2019-12-24-13\n",
      "Copying 965 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000017/7/2019-12-24-14 to /root/mlperf/results-19/data/selfplay/000017/7/2019-12-24-14\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/1/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/1/2019-12-24-14\n",
      "Copying 167 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000017/1/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000017/1/2019-12-24-13\n",
      "Copying 1074 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000017/1/2019-12-24-14 to /root/mlperf/results-19/data/selfplay/000017/1/2019-12-24-14\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/3/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000017/3/2019-12-24-14\n",
      "Copying 143 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000017/3/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000017/3/2019-12-24-13\n",
      "Copying 957 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000017/3/2019-12-24-14 to /root/mlperf/results-19/data/selfplay/000017/3/2019-12-24-14\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/0\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/1\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/2\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/3\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/4\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/5\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/6\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/7\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/4/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/4/2019-12-24-12\n",
      "Copying 431 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000014/4/2019-12-24-12 to /root/mlperf/results-19/data/selfplay/000014/4/2019-12-24-12\n",
      "Copying 559 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000014/4/2019-12-24-11 to /root/mlperf/results-19/data/selfplay/000014/4/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/2/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/2/2019-12-24-12\n",
      "Copying 564 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000014/2/2019-12-24-12 to /root/mlperf/results-19/data/selfplay/000014/2/2019-12-24-12\n",
      "Copying 717 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000014/2/2019-12-24-11 to /root/mlperf/results-19/data/selfplay/000014/2/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/0/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/0/2019-12-24-12\n",
      "Copying 453 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000014/0/2019-12-24-12 to /root/mlperf/results-19/data/selfplay/000014/0/2019-12-24-12\n",
      "Copying 670 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000014/0/2019-12-24-11 to /root/mlperf/results-19/data/selfplay/000014/0/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/6/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/6/2019-12-24-12\n",
      "Copying 532 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000014/6/2019-12-24-12 to /root/mlperf/results-19/data/selfplay/000014/6/2019-12-24-12\n",
      "Copying 645 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000014/6/2019-12-24-11 to /root/mlperf/results-19/data/selfplay/000014/6/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/5/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/5/2019-12-24-12\n",
      "Copying 528 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000014/5/2019-12-24-12 to /root/mlperf/results-19/data/selfplay/000014/5/2019-12-24-12\n",
      "Copying 706 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000014/5/2019-12-24-11 to /root/mlperf/results-19/data/selfplay/000014/5/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/7/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/7/2019-12-24-12\n",
      "Copying 489 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000014/7/2019-12-24-12 to /root/mlperf/results-19/data/selfplay/000014/7/2019-12-24-12\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying 636 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000014/7/2019-12-24-11 to /root/mlperf/results-19/data/selfplay/000014/7/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/1/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/1/2019-12-24-12\n",
      "Copying 564 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000014/1/2019-12-24-12 to /root/mlperf/results-19/data/selfplay/000014/1/2019-12-24-12\n",
      "Copying 693 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000014/1/2019-12-24-11 to /root/mlperf/results-19/data/selfplay/000014/1/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/3/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000014/3/2019-12-24-12\n",
      "Copying 469 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000014/3/2019-12-24-12 to /root/mlperf/results-19/data/selfplay/000014/3/2019-12-24-12\n",
      "Copying 605 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000014/3/2019-12-24-11 to /root/mlperf/results-19/data/selfplay/000014/3/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/0\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/1\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/2\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/3\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/4\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/5\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/6\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/7\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/4/2019-12-24-10\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/4/2019-12-24-11\n",
      "Copying 332 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000013/4/2019-12-24-10 to /root/mlperf/results-19/data/selfplay/000013/4/2019-12-24-10\n",
      "Copying 694 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000013/4/2019-12-24-11 to /root/mlperf/results-19/data/selfplay/000013/4/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/2/2019-12-24-10\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/2/2019-12-24-11\n",
      "Copying 413 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000013/2/2019-12-24-10 to /root/mlperf/results-19/data/selfplay/000013/2/2019-12-24-10\n",
      "Copying 881 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000013/2/2019-12-24-11 to /root/mlperf/results-19/data/selfplay/000013/2/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/0/2019-12-24-10\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/0/2019-12-24-11\n",
      "Copying 376 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000013/0/2019-12-24-10 to /root/mlperf/results-19/data/selfplay/000013/0/2019-12-24-10\n",
      "Copying 736 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000013/0/2019-12-24-11 to /root/mlperf/results-19/data/selfplay/000013/0/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/6/2019-12-24-10\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/6/2019-12-24-11\n",
      "Copying 368 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000013/6/2019-12-24-10 to /root/mlperf/results-19/data/selfplay/000013/6/2019-12-24-10\n",
      "Copying 793 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000013/6/2019-12-24-11 to /root/mlperf/results-19/data/selfplay/000013/6/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/5/2019-12-24-10\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/5/2019-12-24-11\n",
      "Copying 392 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000013/5/2019-12-24-10 to /root/mlperf/results-19/data/selfplay/000013/5/2019-12-24-10\n",
      "Copying 801 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000013/5/2019-12-24-11 to /root/mlperf/results-19/data/selfplay/000013/5/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/7/2019-12-24-10\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/7/2019-12-24-11\n",
      "Copying 368 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000013/7/2019-12-24-10 to /root/mlperf/results-19/data/selfplay/000013/7/2019-12-24-10\n",
      "Copying 756 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000013/7/2019-12-24-11 to /root/mlperf/results-19/data/selfplay/000013/7/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/1/2019-12-24-10\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/1/2019-12-24-11\n",
      "Copying 403 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000013/1/2019-12-24-10 to /root/mlperf/results-19/data/selfplay/000013/1/2019-12-24-10\n",
      "Copying 821 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000013/1/2019-12-24-11 to /root/mlperf/results-19/data/selfplay/000013/1/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/3/2019-12-24-10\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000013/3/2019-12-24-11\n",
      "Copying 346 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000013/3/2019-12-24-10 to /root/mlperf/results-19/data/selfplay/000013/3/2019-12-24-10\n",
      "Copying 732 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000013/3/2019-12-24-11 to /root/mlperf/results-19/data/selfplay/000013/3/2019-12-24-11\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/0\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/1\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/2\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/3\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/4\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/5\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/6\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/7\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/4/2019-12-24-12\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/4/2019-12-24-13\n",
      "Copying 848 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000015/4/2019-12-24-12 to /root/mlperf/results-19/data/selfplay/000015/4/2019-12-24-12\n",
      "Copying 160 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000015/4/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000015/4/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/2/2019-12-24-12\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/2/2019-12-24-13\n",
      "Copying 1100 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000015/2/2019-12-24-12 to /root/mlperf/results-19/data/selfplay/000015/2/2019-12-24-12\n",
      "Copying 202 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000015/2/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000015/2/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/0/2019-12-24-12\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/0/2019-12-24-13\n",
      "Copying 982 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000015/0/2019-12-24-12 to /root/mlperf/results-19/data/selfplay/000015/0/2019-12-24-12\n",
      "Copying 116 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000015/0/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000015/0/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/6/2019-12-24-12\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/6/2019-12-24-13\n",
      "Copying 983 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000015/6/2019-12-24-12 to /root/mlperf/results-19/data/selfplay/000015/6/2019-12-24-12\n",
      "Copying 192 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000015/6/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000015/6/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/5/2019-12-24-12\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/5/2019-12-24-13\n",
      "Copying 1018 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000015/5/2019-12-24-12 to /root/mlperf/results-19/data/selfplay/000015/5/2019-12-24-12\n",
      "Copying 206 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000015/5/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000015/5/2019-12-24-13\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Make dir /root/mlperf/results-19/data/selfplay/000015/7/2019-12-24-12\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/7/2019-12-24-13\n",
      "Copying 930 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000015/7/2019-12-24-12 to /root/mlperf/results-19/data/selfplay/000015/7/2019-12-24-12\n",
      "Copying 172 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000015/7/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000015/7/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/1/2019-12-24-12\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/1/2019-12-24-13\n",
      "Copying 1085 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000015/1/2019-12-24-12 to /root/mlperf/results-19/data/selfplay/000015/1/2019-12-24-12\n",
      "Copying 186 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000015/1/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000015/1/2019-12-24-13\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/3/2019-12-24-12\n",
      "Make dir /root/mlperf/results-19/data/selfplay/000015/3/2019-12-24-13\n",
      "Copying 906 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000015/3/2019-12-24-12 to /root/mlperf/results-19/data/selfplay/000015/3/2019-12-24-12\n",
      "Copying 172 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/data/selfplay/000015/3/2019-12-24-13 to /root/mlperf/results-19/data/selfplay/000015/3/2019-12-24-13\n",
      "Copying /root/zheng/dataset/minigo/checkpoints/mlperf07/work_dir to /root/mlperf/results-19/work_dir\n",
      "Copying 4 files from /root/zheng/dataset/minigo/checkpoints/mlperf07/work_dir to /root/mlperf/results-19/work_dir/.\n",
      "Copying ml_perf/flags/19 to /root/mlperf/results-19/flags\n",
      "Copying 7 files from ml_perf/flags/19 to /root/mlperf/results-19/flags/.\n",
      "+++ echo 'Copy selfplay files to node 0 storage'\n",
      "Copy selfplay files to node 0 storage\n",
      "+++ mkdir -p /tmp/selfplay\n",
      "+++ rm -rf '/tmp/selfplay/*'\n",
      "++++ dirname /tmp/selfplay\n",
      "+++ rsync -r --append /root/mlperf/results-19/data/selfplay /tmp\n",
      "+++ ls /tmp/selfplay\n",
      "000013\t000014\t000015\t000016\t000017\n",
      "GCC Version: 8.4.0\n",
      "GCC 8.4.0: OK\n",
      "exporting BAZEL_BUILD_OPTS_BASIC=--cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0         --copt=-march=haswell         --copt=-mtune=broadwell                 --copt=-O3\n",
      "exporting BAZEL_BUILD_OPTS=--cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0         --copt=-march=haswell         --copt=-mtune=broadwell                 --copt=-O3 --copt=-Wformat                 --copt=-Wformat-security                 --copt=-fstack-protector                 --copt=-fPIC                 --copt=-fpic                 --linkopt=-znoexecstack                 --linkopt=-zrelro                 --linkopt=-znow                 --linkopt=-fstack-protector                 --linkopt=-pie\n",
      "exporting BAZEL_MKL_BUILD_OPTS_BASIC=--config=mkl --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0         --copt=-march=haswell         --copt=-mtune=broadwell                 --copt=-O3\n",
      "exporting BAZEL_SECURE_MKL_BUILD_OPTS=--config=mkl --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0         --copt=-march=haswell         --copt=-mtune=broadwell                 --copt=-O3 --copt=-Wformat                 --copt=-Wformat-security                 --copt=-fstack-protector                 --copt=-fPIC                 --copt=-fpic                 --linkopt=-znoexecstack                 --linkopt=-zrelro                 --linkopt=-znow                 --linkopt=-fstack-protector                 --linkopt=-pie\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ IFS=,\n",
      "+ for NODE in $HOSTLIST\n",
      "+ '[' sr141 '!=' sr141 ']'\n",
      "+ for NODE in $HOSTLIST\n",
      "+ '[' sr142 '!=' sr141 ']'\n",
      "+++ trap stop_selfplay EXIT\n",
      "+++ [[ ! -n '' ]]\n",
      "+++ precision=int8\n",
      "+ ml_perf/scripts/loop.sh rsync -r --append --delete-before /root/mlperf/results-19/flags root@sr142:/root/mlperf/results-19/\n",
      "+++ [[ -v HOSTLIST ]]\n",
      "+++ echo 'Run multi-node training.'\n",
      "Run multi-node training.\n",
      "+++ [[ -v TRAINHOSTLIST ]]\n",
      "+ ml_perf/scripts/loop.sh rsync -r --append --delete-before /root/mlperf/results-19/signal root@sr142:/root/mlperf/results-19/\n",
      "+++ train_host_list=sr141,sr142\n",
      "+++ echo 'Train host list = sr141,sr142'\n",
      "Train host list = sr141,sr142\n",
      "+++ echo 'Physical cores = 36'\n",
      "Physical cores = 36\n",
      "+++ echo 'Virtual cores = 72'\n",
      "Virtual cores = 72\n",
      "+++ echo 'Cores per NUMA = 18'\n",
      "Cores per NUMA = 18\n",
      "+ ml_perf/scripts/loop.sh rsync -r --append --delete-before /root/mlperf/results-19/models root@sr142:/root/mlperf/results-19/\n",
      "++++ expr 36 - 1\n",
      "+++ LAST_PHY_CORE=35\n",
      "+++ echo 'Evaluation Last physical core = 35'\n",
      "Evaluation Last physical core = 35\n",
      "+++ tee /root/mlperf/results-19/logs/train_loop.log\n",
      "++++ seq -s, 0 35\n",
      "+++ BOARD_SIZE=19\n",
      "+++ CUDA_VISIBLE_DEVICES=0\n",
      "+++ python3 ml_perf/train_loop.py --flags_dir=/root/mlperf/results-19/flags --golden_chunk_dir=/root/mlperf/results-19/data/golden_chunks --golden_chunk_local_dir=/tmp/golden_chunks --golden_chunk_tmp_dir=/tmp/golden_chunks_tmp --holdout_dir=/root/mlperf/results-19/data/holdout --log_dir=/root/mlperf/results-19/logs --model_dir=/root/mlperf/results-19/models --selfplay_dir=/tmp/selfplay --pause=/root/mlperf/results-19/signal/pause --work_dir=/root/mlperf/results-19/work_dir --log_dir=/root/mlperf/results-19/logs --flagfile=/root/mlperf/results-19/flags/train_loop.flags --window_size=5 --precision=int8 --tpu_name= --hostlist=sr141,sr142 --physical_cores=36 --virtual_cores=72 --numa_cores=18 --devices=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35 --target=/root/zheng/dataset/minigo/target/target.minigo\n",
      "++ cat /proc/cpuinfo\n",
      "++ grep 'physical id'\n",
      "++ sort -u\n",
      "++ wc -l\n",
      "++ cat /proc/cpuinfo\n",
      "++ grep 'physical id'\n",
      "++ sort -u\n",
      "++ wc -l\n",
      "+ NUMA_COUNT=2\n",
      "++ cat /proc/cpuinfo\n",
      "++ grep 'physical id'\n",
      "++ wc -l\n",
      "+ NUMA_COUNT=2\n",
      "++ cat /proc/cpuinfo\n",
      "++ grep 'physical id'\n",
      "++ wc -l\n",
      "+ VIRT_CORES=72\n",
      "++ cat /proc/cpuinfo\n",
      "++ grep 'cpu cores'\n",
      "++ head -n 1\n",
      "++ awk '//{print $4}'\n",
      "+ VIRT_CORES=72\n",
      "++ cat /proc/cpuinfo\n",
      "++ grep 'cpu cores'\n",
      "++ head -n 1\n",
      "++ awk '//{print $4}'\n",
      "+ NUMA_CORES=18\n",
      "++ expr 18 '*' 2\n",
      "+ PHY_CORES=36\n",
      "+ echo 'NUMA count = 2'\n",
      "+ echo 'Virtual cores = 72'\n",
      "+ echo 'Cores per NUMA = 18'\n",
      "+ echo 'Physical cores = 36'\n",
      "++ hostname\n",
      "+ NUMA_CORES=18\n",
      "++ expr 18 '*' 2\n",
      "+ log_dir=/root/mlperf/results-19/logs/selfplay/sr142\n",
      "+ mkdir -p /root/mlperf/results-19/logs/selfplay/sr142\n",
      "+ PHY_CORES=36\n",
      "+ echo 'NUMA count = 2'\n",
      "+ echo 'Virtual cores = 72'\n",
      "+ echo 'Cores per NUMA = 18'\n",
      "+ echo 'Physical cores = 36'\n",
      "++ hostname\n",
      "+ echo sr141\n",
      "+ ./ml_perf/scripts/start_sync_selfplay.sh --base_dir=/root/mlperf/results-19 --root_node=sr141\n",
      "+ ./ml_perf/scripts/start_sync_signal.sh --base_dir=/root/mlperf/results-19\n",
      "+ log_dir=/root/mlperf/results-19/logs/selfplay/sr141\n",
      "+ mkdir -p /root/mlperf/results-19/logs/selfplay/sr141\n",
      "+ echo sr141\n",
      "+ ./ml_perf/scripts/start_sync_selfplay.sh --base_dir=/root/mlperf/results-19 --root_node=sr141\n",
      "+ ./ml_perf/scripts/start_sync_signal.sh --base_dir=/root/mlperf/results-19\n",
      "++ tee /root/mlperf/results-19/logs/sync_selfplay_loop_sr142.log\n",
      "+ (( device=0 ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+++ whoami\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[0],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=0 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/0' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/0' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "++ hostname\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[1],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=1 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/1' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/1' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[2],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=2 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/2' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/2' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "++ tee /root/mlperf/results-19/logs/sync_selfplay_loop_sr141.log\n",
      "++ hostname\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ (( device=0 ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[3],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=3 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/3' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/3' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+++ whoami\n",
      "++ hostname\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[4],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[0],explicit'\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=4 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/4' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/4' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=0 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/0' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/0' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "++ hostname\n",
      "+ (( device++ ))\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "++ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ (( device<PHY_CORES ))\n",
      "++ ml_perf/scripts/loop.sh rsync -ar --append /tmp/selfplay_local/ root@sr141:/tmp/selfplay/\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[5],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=5 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/5' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/5' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[1],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=1 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/1' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/1' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "++ hostname\n",
      "++ hostname\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_0.log\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[2],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "++ hostname\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=2 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/2' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/2' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[6],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=6 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/6' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/6' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "++ hostname\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_1.log\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[3],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=3 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/3' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/3' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[7],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "++ hostname\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=7 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/7' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/7' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[4],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=4 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/4' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/4' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_2.log\n",
      "+ OMP_NUM_THREADS=1\n",
      "++ hostname\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[8],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=8 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/8' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/8' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_3.log\n",
      "++ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "++ ml_perf/scripts/loop.sh rsync -ar --append /tmp/selfplay_local/ root@sr141:/tmp/selfplay/\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[5],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=5 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/5' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/5' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "++ hostname\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[9],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=9 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/9' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/9' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_0.log\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[6],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=6 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/6' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/6' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_4.log\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_1.log\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[7],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=7 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/7' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/7' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[10],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=10 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/10' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/10' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_2.log\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[8],explicit'\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_5.log\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=8 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/8' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/8' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[11],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=11 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/11' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/11' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_3.log\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[9],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=9 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/9' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/9' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[12],explicit'\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_4.log\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=12 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/12' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/12' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[10],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=10 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/10' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/10' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_5.log\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[11],explicit'\n",
      "+ (( device++ ))\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ (( device<PHY_CORES ))\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=11 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/11' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/11' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[13],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=13 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/13' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/13' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[12],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=12 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/12' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/12' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ (( device<PHY_CORES ))\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[14],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=14 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/14' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/14' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[13],explicit'\n",
      "+ (( device++ ))\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ (( device<PHY_CORES ))\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=13 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/13' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/13' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[15],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=15 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/15' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/15' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[14],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=14 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/14' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/14' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ (( device<PHY_CORES ))\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[16],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=16 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/16' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/16' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[15],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=15 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/15' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/15' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[17],explicit'\n",
      "+ (( device++ ))\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=17 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/17' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/17' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device<PHY_CORES ))\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[16],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=16 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/16' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/16' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ (( device<PHY_CORES ))\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[17],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=17 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/17' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/17' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[18],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ (( device++ ))\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=18 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/18' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/18' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device<PHY_CORES ))\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[18],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=18 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/18' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/18' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[19],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=19 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/19' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/19' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[19],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=19 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/19' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/19' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[20],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=20 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/20' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/20' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[20],explicit'\n",
      "+ (( device++ ))\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ (( device<PHY_CORES ))\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=20 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/20' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/20' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[21],explicit'\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[21],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=21 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/21' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/21' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=21 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/21' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/21' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[22],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=22 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/22' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/22' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[23],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=23 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/23' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/23' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[24],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=24 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/24' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/24' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[23],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=23 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/23' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/23' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[24],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=24 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/24' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/24' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[26],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=26 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/26' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/26' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[25],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=25 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/25' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/25' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ (( device++ ))\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[22],explicit'\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ (( device<PHY_CORES ))\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[27],explicit'\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=22 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/22' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/22' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=27 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/27' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/27' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[28],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=28 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/28' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/28' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[26],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=26 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/26' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/26' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[29],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=29 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/29' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/29' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[27],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=27 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/27' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/27' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[30],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=30 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/30' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/30' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[28],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=28 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/28' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/28' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[31],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=31 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/31' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/31' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[29],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=29 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/29' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/29' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[25],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=25 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/25' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/25' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[30],explicit'\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=30 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/30' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/30' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[33],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=33 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/33' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/33' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[34],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=34 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/34' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/34' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[31],explicit'\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[32],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=31 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/31' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/31' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=32 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/32' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/32' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[32],explicit'\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[35],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=32 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/32' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/32' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=35 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/35' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/35' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[33],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=33 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/33' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/33' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ (( device++ ))\n",
      "+ (( device<PHY_CORES ))\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[35],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=35 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/35' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/35' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "+ OMP_NUM_THREADS=1\n",
      "+ KMP_AFFINITY='granularity=fine,proclist=[34],explicit'\n",
      "+ ABORT_FILE=/root/mlperf/results-19/signal/abort\n",
      "+ ml_perf/scripts/loop.sh numactl --physcpubind=34 ./bazel-bin/cc/concurrent_selfplay --device=cpu --flagfile=/root/mlperf/results-19/flags/selfplay.flags '--output_dir=/tmp/selfplay_local//$MODEL/34' '--holdout_dir=/root/mlperf/results-19/data/holdout/$MODEL/34' --model=/root/mlperf/results-19/models/%d.minigo --run_forever=1 --abort_file=/tmp/signal/abort --pause_file=/tmp/signal/pause\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_15.log\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_7.log\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_6.log\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_12.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_9.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_10.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_8.log\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_13.log\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_6.log\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_11.log\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_9.log\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_10.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_12.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_13.log\n",
      "++ hostname\n",
      "++ hostname\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_11.log\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_17.log\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_7.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_16.log\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_14.log\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_18.log\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_15.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_14.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_20.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_16.log\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_17.log\n",
      "++ hostname\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_8.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_26.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_19.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_23.log\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_22.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_26.log\n",
      "++ hostname\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_28.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_24.log\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_24.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_18.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_28.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_20.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_21.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_25.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_29.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_30.log\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_33.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_29.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_35.log\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_27.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_21.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_23.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_27.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_31.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_30.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_32.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_34.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_19.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr141/sr141_selfplay_22.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_25.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_34.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_31.log\n",
      "++ hostname\n",
      "++ hostname\n",
      "++ hostname\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_35.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_32.log\n",
      "+ tee /root/mlperf/results-19/logs/selfplay/sr142/sr142_selfplay_33.log\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231727014, \"event_type\": \"POINT_IN_TIME\", \"key\": \"submission_org\", \"value\": \"Intel\", \"metadata\": {\"file\": \"ml_perf/train_loop.py\", \"lineno\": 527}}\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231727136, \"event_type\": \"POINT_IN_TIME\", \"key\": \"submission_platform\", \"value\": \"8 nodes x 4s CPX\", \"metadata\": {\"file\": \"ml_perf/train_loop.py\", \"lineno\": 528}}\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231727136, \"event_type\": \"POINT_IN_TIME\", \"key\": \"submission_division\", \"value\": \"closed\", \"metadata\": {\"file\": \"ml_perf/train_loop.py\", \"lineno\": 529}}\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231727136, \"event_type\": \"POINT_IN_TIME\", \"key\": \"submission_status\", \"value\": \"onprem\", \"metadata\": {\"file\": \"ml_perf/train_loop.py\", \"lineno\": 530}}\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231727136, \"event_type\": \"POINT_IN_TIME\", \"key\": \"submission_benchmark\", \"value\": \"minigo\", \"metadata\": {\"file\": \"ml_perf/train_loop.py\", \"lineno\": 531}}\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231727136, \"event_type\": \"POINT_IN_TIME\", \"key\": \"cache_clear\", \"value\": true, \"metadata\": {\"file\": \"ml_perf/train_loop.py\", \"lineno\": 533}}\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231727136, \"event_type\": \"POINT_IN_TIME\", \"key\": \"filter_amount\", \"value\": 0.3, \"metadata\": {\"file\": \"ml_perf/train_loop.py\", \"lineno\": 535}}\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231727137, \"event_type\": \"POINT_IN_TIME\", \"key\": \"gradient_accumulation_steps\", \"value\": 1, \"metadata\": {\"file\": \"ml_perf/train_loop.py\", \"lineno\": 536}}\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231727137, \"event_type\": \"INTERVAL_END\", \"key\": \"init_stop\", \"value\": null, \"metadata\": {\"file\": \"ml_perf/train_loop.py\", \"lineno\": 550}}\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231727137, \"event_type\": \"INTERVAL_START\", \"key\": \"run_start\", \"value\": null, \"metadata\": {\"file\": \"ml_perf/train_loop.py\", \"lineno\": 551}}\n",
      "[2022-09-15 16:48:47] Running: python3  parse_flags_train.py  --bool_features=1  --conv_width=64  --fc_width=64  --input_features=mlperf07  --input_layout=nhwc  --keep_checkpoint_max=100  --l2_strength=0.0001  --lr_boundaries=10000  --lr_boundaries=128  --lr_boundaries=20000  --lr_rates=0.0016  --lr_rates=0.016  --lr_rates=0.016  --lr_rates=0.16  --shuffle_buffer_size=0  --shuffle_examples=false  --summary_steps=128  --train_batch_size=4196  --trunk_layers=6  --value_cost_weight=0.5\n",
      "[2022-09-15 16:48:47] parse_flags_train finished: 0.220 seconds\n",
      "[2022-09-15 16:48:47] Running: python3  parse_flags_selfplay.py  --cache_size_mb=4096  --concurrent_games_per_thread=6  --dirichlet_alpha=0.03  --disable_resign_pct=0.0  --fastplay_frequency=0.82  --fastplay_readouts=60  --holdout_pct=0.0  --max_resign_threshold=-0.9  --min_resign_threshold=-1.0  --noise_mix=0.3  --num_readouts=600  --parallel_inference=1  --parallel_search=1  --selfplay_threads=1  --target_pruning=1  --value_init_penalty=0.2  --verbose=false  --virtual_losses=4\n",
      "[2022-09-15 16:48:48] parse_flags_selfplay finished: 0.228 seconds\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231728022, \"event_type\": \"POINT_IN_TIME\", \"key\": \"window_size\", \"value\": 5, \"metadata\": {\"file\": \"ml_perf/train_loop.py\", \"lineno\": 564}}\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231728023, \"event_type\": \"POINT_IN_TIME\", \"key\": \"epoch_start\", \"value\": null, \"metadata\": {\"file\": \"ml_perf/train_loop.py\", \"lineno\": 570, \"epoch_num\": 17}}\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231728079, \"event_type\": \"POINT_IN_TIME\", \"key\": \"actual_selfplay_games_per_generation\", \"value\": 9240, \"metadata\": {\"file\": \"ml_perf/train_loop.py\", \"lineno\": 146}}\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231728079, \"event_type\": \"POINT_IN_TIME\", \"key\": \"num_games\", \"value\": 8192, \"metadata\": {\"file\": \"ml_perf/train_loop.py\", \"lineno\": 244}}\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231728079, \"event_type\": \"POINT_IN_TIME\", \"key\": \"min_selfplay_games_per_generation\", \"value\": 8192, \"metadata\": {\"file\": \"ml_perf/train_loop.py\", \"lineno\": 245}}\n",
      "[2022-09-15 16:48:48] Writing training chunks to /tmp/golden_chunks_tmp/000018.tfrecord.zz\n",
      "[2022-09-15 16:48:48] Running: bazel-bin/cc/sample_records  --compression=1  --dst=/tmp/golden_chunks_tmp/000018.tfrecord.zz  --files_per_pattern=8192  --num_read_threads=18  --num_write_threads=4  --sample_frac=0.3  --shuffle=true  /tmp/selfplay/000017/*/*/*.tfrecord.zz  /tmp/selfplay/000016/*/*/*.tfrecord.zz  /tmp/selfplay/000015/*/*/*.tfrecord.zz  /tmp/selfplay/000014/*/*/*.tfrecord.zz  /tmp/selfplay/000013/*/*/*.tfrecord.zz\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2022-09-15 16:49:00] sample_records finished: 12.389 seconds\n",
      "[2022-09-15 16:49:00] /tmp/golden_chunks_tmp/000018-*-of-*.tfrecord.zz\n",
      "[2022-09-15 16:49:00] ['/tmp/golden_chunks_tmp/000018-00000-of-00004.tfrecord.zz', '/tmp/golden_chunks_tmp/000018-00001-of-00004.tfrecord.zz', '/tmp/golden_chunks_tmp/000018-00002-of-00004.tfrecord.zz', '/tmp/golden_chunks_tmp/000018-00003-of-00004.tfrecord.zz']\n",
      "--helpfull: No such file or directory\n",
      "[2022-09-15 16:49:00] Running: scp  /tmp/golden_chunks_tmp/000018-00000-of-00004.tfrecord.zz  sr141:/tmp/golden_chunks\n",
      "[2022-09-15 16:49:00] Running: scp  /tmp/golden_chunks_tmp/000018-00001-of-00004.tfrecord.zz  sr141:/tmp/golden_chunks\n",
      "[2022-09-15 16:49:00] Running: scp  /tmp/golden_chunks_tmp/000018-00002-of-00004.tfrecord.zz  sr142:/tmp/golden_chunks\n",
      "[2022-09-15 16:49:00] Running: scp  /tmp/golden_chunks_tmp/000018-00003-of-00004.tfrecord.zz  sr142:/tmp/golden_chunks\n",
      "[2022-09-15 16:49:01] scp finished: 1.269 seconds\n",
      "[2022-09-15 16:49:01] scp finished: 1.294 seconds\n",
      "[2022-09-15 16:49:01] scp finished: 1.303 seconds\n",
      "[2022-09-15 16:49:01] scp finished: 1.346 seconds\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231741977, \"event_type\": \"POINT_IN_TIME\", \"key\": \"train_samples\", \"value\": 248886, \"metadata\": {\"file\": \"ml_perf/train_loop.py\", \"lineno\": 311}}\n",
      "[2022-09-15 16:49:01] Run bfloat16 training.\n",
      "2022-09-15 16:49:03.655443: I tensorflow_io/core/kernels/cpu_check.cc:128] Your CPU supports instructions that this TensorFlow IO binary was not compiled to use: AVX2 AVX512F FMA\n",
      "[2022-09-15 16:49:04] Running distributed: python3  train.py  --bool_features=1  --conv_width=64  --dist_train=True  --export_path=/root/mlperf/results-19/models/000018  --fc_width=64  --freeze=true  --input_features=mlperf07  --input_layout=nhwc  --keep_checkpoint_max=100  --l2_strength=0.0001  --lr_boundaries=10000  --lr_boundaries=128  --lr_boundaries=20000  --lr_rates=0.0016  --lr_rates=0.016  --lr_rates=0.016  --lr_rates=0.16  --num_examples=248886  --num_inter_threads=2  --num_intra_threads=17  --selfplay_precision=int8  --shuffle_buffer_size=0  --shuffle_examples=false  --summary_steps=128  --tpu_name=  --train_batch_size=4196  --training_seed=18  --trunk_layers=6  --use_bfloat16=True  --use_tpu=false  --value_cost_weight=0.5  --work_dir=/root/mlperf/results-19/work_dir\n",
      "[2022-09-15 16:49:04] Running: mpiexec -outfile-pattern /root/mlperf/results-19/logs/out-train-None-%r.txt -genv HOROVOD_FUSION_THRESHOLD=134217728 -genv KMP_BLOCKTIME=1 -genv KMP_HW_SUBSET=2T -genv OMP_BIND_PROC=true -genv I_MPI_PIN_DOMAIN=socket -genv PSM3_ALLOW_ROUTERS=1 -genv PSM3_RDMA=1 -genv OMP_NUM_THREADS=17 -host sr141 -n 1 -env KMP_AFFINITY=granularity=fine,compact,1,1 numactl -l -N 0 python3 train.py --flagfile=/root/mlperf/results-19/flags/train.flags --work_dir=/root/mlperf/results-19/work_dir --export_path=/root/mlperf/results-19/models/000018 --use_tpu=false --tpu_name= --training_seed=18 --num_examples=248886 --use_bfloat16=True --selfplay_precision=int8 --num_intra_threads=17 --num_inter_threads=2 --freeze=true --dist_train=True /tmp/golden_chunks/000018-00000-of-00004.tfrecord.zz : -host sr141 -n 1 -env KMP_AFFINITY=granularity=fine,compact,1,19 numactl -l -N 1 python3 train.py --flagfile=/root/mlperf/results-19/flags/train.flags --work_dir=/root/mlperf/results-19/work_dir --export_path=/root/mlperf/results-19/models/000018 --use_tpu=false --tpu_name= --training_seed=18 --num_examples=248886 --use_bfloat16=True --selfplay_precision=int8 --num_intra_threads=17 --num_inter_threads=2 --freeze=true --dist_train=True /tmp/golden_chunks/000018-00001-of-00004.tfrecord.zz : -host sr142 -n 1 -env KMP_AFFINITY=granularity=fine,compact,1,1 numactl -l -N 0 python3 train.py --flagfile=/root/mlperf/results-19/flags/train.flags --work_dir=/root/mlperf/results-19/work_dir --export_path=/root/mlperf/results-19/models/000018 --use_tpu=false --tpu_name= --training_seed=18 --num_examples=248886 --use_bfloat16=True --selfplay_precision=int8 --num_intra_threads=17 --num_inter_threads=2 --freeze=true --dist_train=True /tmp/golden_chunks/000018-00002-of-00004.tfrecord.zz : -host sr142 -n 1 -env KMP_AFFINITY=granularity=fine,compact,1,19 numactl -l -N 1 python3 train.py --flagfile=/root/mlperf/results-19/flags/train.flags --work_dir=/root/mlperf/results-19/work_dir --export_path=/root/mlperf/results-19/models/000018 --use_tpu=false --tpu_name= --training_seed=18 --num_examples=248886 --use_bfloat16=True --selfplay_precision=int8 --num_intra_threads=17 --num_inter_threads=2 --freeze=true --dist_train=True /tmp/golden_chunks/000018-00003-of-00004.tfrecord.zz\n",
      "[2022-09-15 17:03:43] mpiexec finished: 879.778 seconds\n",
      "[2022-09-15 17:03:43] 2022-09-15 16:49:06.675528: I tensorflow_io/core/kernels/cpu_check.cc:128] Your CPU supports instructions that this TensorFlow IO binary was not compiled to use: AVX2 AVX512F FMA\n",
      "2022-09-15 16:49:06.681494: I tensorflow_io/core/kernels/cpu_check.cc:128] Your CPU supports instructions that this TensorFlow IO binary was not compiled to use: AVX2 AVX512F FMA\n",
      "2022-09-15 16:48:45.373582: I tensorflow_io/core/kernels/cpu_check.cc:128] Your CPU supports instructions that this TensorFlow IO binary was not compiled to use: AVX2 AVX512F FMA\n",
      "2022-09-15 16:48:45.398397: I tensorflow_io/core/kernels/cpu_check.cc:128] Your CPU supports instructions that this TensorFlow IO binary was not compiled to use: AVX2 AVX512F FMA\n",
      "I0915 16:49:06.839362 139841635501888 train.py:292] Training on 1 records: /tmp/golden_chunks/000018-00001-of-00004.tfrecord.zz to /tmp/golden_chunks/000018-00001-of-00004.tfrecord.zz\n",
      "I0915 16:49:06.841793 139882633475904 train.py:292] Training on 1 records: /tmp/golden_chunks/000018-00000-of-00004.tfrecord.zz to /tmp/golden_chunks/000018-00000-of-00004.tfrecord.zz\n",
      "I0915 16:48:45.540552 139894849677120 train.py:292] Training on 1 records: /tmp/golden_chunks/000018-00003-of-00004.tfrecord.zz to /tmp/golden_chunks/000018-00003-of-00004.tfrecord.zz\n",
      "I0915 16:48:45.583519 139982537570112 train.py:292] Training on 1 records: /tmp/golden_chunks/000018-00002-of-00004.tfrecord.zz to /tmp/golden_chunks/000018-00002-of-00004.tfrecord.zz\n",
      "WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpc2459yoa\n",
      "W0915 16:48:45.927734 139894849677120 estimator.py:1825] Using temporary folder as model directory: /tmp/tmpc2459yoa\n",
      "WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmp3fjsbl6s\n",
      "W0915 16:49:07.249861 139841635501888 estimator.py:1825] Using temporary folder as model directory: /tmp/tmp3fjsbl6s\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/root/mlperf/results-19/work_dir', '_tf_random_seed': None, '_save_summary_steps': 128, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': intra_op_parallelism_threads: 17\n",
      "inter_op_parallelism_threads: 2\n",
      "gpu_options {\n",
      "  allow_growth: true\n",
      "}\n",
      ", '_keep_checkpoint_max': 100, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 50, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "I0915 16:49:07.250158 139882633475904 estimator.py:216] Using config: {'_model_dir': '/root/mlperf/results-19/work_dir', '_tf_random_seed': None, '_save_summary_steps': 128, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': intra_op_parallelism_threads: 17\n",
      "inter_op_parallelism_threads: 2\n",
      "gpu_options {\n",
      "  allow_growth: true\n",
      "}\n",
      ", '_keep_checkpoint_max': 100, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 50, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmp4__k6blo\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmpc2459yoa', '_tf_random_seed': None, '_save_summary_steps': 10000000, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': intra_op_parallelism_threads: 17\n",
      "inter_op_parallelism_threads: 2\n",
      "gpu_options {\n",
      "  allow_growth: true\n",
      "}\n",
      ", '_keep_checkpoint_max': 100, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 10000000, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "I0915 16:48:45.930062 139894849677120 estimator.py:216] Using config: {'_model_dir': '/tmp/tmpc2459yoa', '_tf_random_seed': None, '_save_summary_steps': 10000000, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': intra_op_parallelism_threads: 17\n",
      "inter_op_parallelism_threads: 2\n",
      "gpu_options {\n",
      "  allow_growth: true\n",
      "}\n",
      ", '_keep_checkpoint_max': 100, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 10000000, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "W0915 16:48:45.928405 139982537570112 estimator.py:1825] Using temporary folder as model directory: /tmp/tmp4__k6blo\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmp3fjsbl6s', '_tf_random_seed': None, '_save_summary_steps': 10000000, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': intra_op_parallelism_threads: 17\n",
      "inter_op_parallelism_threads: 2\n",
      "gpu_options {\n",
      "  allow_growth: true\n",
      "}\n",
      ", '_keep_checkpoint_max': 100, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 10000000, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "I0915 16:49:07.252219 139841635501888 estimator.py:216] Using config: {'_model_dir': '/tmp/tmp3fjsbl6s', '_tf_random_seed': None, '_save_summary_steps': 10000000, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': intra_op_parallelism_threads: 17\n",
      "inter_op_parallelism_threads: 2\n",
      "gpu_options {\n",
      "  allow_growth: true\n",
      "}\n",
      ", '_keep_checkpoint_max': 100, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 10000000, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmp4__k6blo', '_tf_random_seed': None, '_save_summary_steps': 10000000, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': intra_op_parallelism_threads: 17\n",
      "inter_op_parallelism_threads: 2\n",
      "gpu_options {\n",
      "  allow_growth: true\n",
      "}\n",
      ", '_keep_checkpoint_max': 100, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 10000000, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "I0915 16:48:45.930755 139982537570112 estimator.py:216] Using config: {'_model_dir': '/tmp/tmp4__k6blo', '_tf_random_seed': None, '_save_summary_steps': 10000000, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': intra_op_parallelism_threads: 17\n",
      "inter_op_parallelism_threads: 2\n",
      "gpu_options {\n",
      "  allow_growth: true\n",
      "}\n",
      ", '_keep_checkpoint_max': 100, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 10000000, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "I0915 16:48:46.143337 139894849677120 train.py:207] Real global batch size = 4196, local batch size = 1049.\n",
      "I0915 16:48:46.143449 139894849677120 train.py:265] Training, steps = 237, batch = 1049 -> 248613 examples\n",
      "I0915 16:49:07.465984 139882633475904 train.py:207] Real global batch size = 4196, local batch size = 1049.\n",
      "I0915 16:49:07.466112 139882633475904 train.py:265] Training, steps = 237, batch = 1049 -> 248613 examples\n",
      "I0915 16:48:46.147709 139982537570112 train.py:207] Real global batch size = 4196, local batch size = 1049.\n",
      "I0915 16:48:46.147817 139982537570112 train.py:265] Training, steps = 237, batch = 1049 -> 248613 examples\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n",
      "W0915 16:48:46.147836 139894849677120 deprecation.py:323] From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n",
      "W0915 16:49:07.470757 139882633475904 deprecation.py:323] From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n",
      "I0915 16:49:07.472885 139841635501888 train.py:207] Real global batch size = 4196, local batch size = 1049.\n",
      "I0915 16:49:07.473030 139841635501888 train.py:265] Training, steps = 237, batch = 1049 -> 248613 examples\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n",
      "W0915 16:48:46.152145 139982537570112 deprecation.py:323] From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n",
      "W0915 16:49:07.477641 139841635501888 deprecation.py:323] From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n",
      "I0915 16:48:46.162902 139894849677120 preprocessing.py:200] Reading tf_records from 1 inputs\n",
      "I0915 16:49:07.486228 139882633475904 preprocessing.py:200] Reading tf_records from 1 inputs\n",
      "I0915 16:48:46.167282 139982537570112 preprocessing.py:200] Reading tf_records from 1 inputs\n",
      "I0915 16:49:07.493328 139841635501888 preprocessing.py:200] Reading tf_records from 1 inputs\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:216: unbatch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.unbatch()`.\n",
      "W0915 16:48:46.457820 139894849677120 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:216: unbatch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.unbatch()`.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:219: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.\n",
      "W0915 16:48:46.461718 139894849677120 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:219: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:216: unbatch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.unbatch()`.\n",
      "W0915 16:49:07.789484 139882633475904 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:216: unbatch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.unbatch()`.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:216: unbatch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.unbatch()`.\n",
      "W0915 16:48:46.468716 139982537570112 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:216: unbatch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.unbatch()`.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:219: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.\n",
      "W0915 16:49:07.793479 139882633475904 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:219: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:219: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.\n",
      "W0915 16:48:46.472631 139982537570112 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:219: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:216: unbatch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.unbatch()`.\n",
      "W0915 16:49:07.799225 139841635501888 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:216: unbatch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.unbatch()`.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:219: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.\n",
      "W0915 16:49:07.803265 139841635501888 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:219: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "I0915 16:48:46.758236 139894849677120 estimator.py:1151] Calling model_fn.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: conv2d (from tensorflow.python.keras.legacy_tf_layers.convolutional) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.keras.layers.Conv2D` instead.\n",
      "W0915 16:48:46.758488 139894849677120 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: conv2d (from tensorflow.python.keras.legacy_tf_layers.convolutional) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.keras.layers.Conv2D` instead.\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `layer.__call__` method instead.\n",
      "W0915 16:48:46.762703 139894849677120 deprecation.py:323] From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `layer.__call__` method instead.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "I0915 16:49:08.087835 139882633475904 estimator.py:1151] Calling model_fn.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: conv2d (from tensorflow.python.keras.legacy_tf_layers.convolutional) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.keras.layers.Conv2D` instead.\n",
      "W0915 16:49:08.088072 139882633475904 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: conv2d (from tensorflow.python.keras.legacy_tf_layers.convolutional) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.keras.layers.Conv2D` instead.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "I0915 16:48:46.768161 139982537570112 estimator.py:1151] Calling model_fn.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: conv2d (from tensorflow.python.keras.legacy_tf_layers.convolutional) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.keras.layers.Conv2D` instead.\n",
      "W0915 16:48:46.768407 139982537570112 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: conv2d (from tensorflow.python.keras.legacy_tf_layers.convolutional) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.keras.layers.Conv2D` instead.\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `layer.__call__` method instead.\n",
      "W0915 16:49:08.092226 139882633475904 deprecation.py:323] From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `layer.__call__` method instead.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: batch_normalization (from tensorflow.python.keras.legacy_tf_layers.normalization) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.BatchNormalization instead.  In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.BatchNormalization` documentation).\n",
      "W0915 16:48:46.771142 139894849677120 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: batch_normalization (from tensorflow.python.keras.legacy_tf_layers.normalization) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.BatchNormalization instead.  In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.BatchNormalization` documentation).\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `layer.__call__` method instead.\n",
      "W0915 16:48:46.772548 139982537570112 deprecation.py:323] From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `layer.__call__` method instead.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: batch_normalization (from tensorflow.python.keras.legacy_tf_layers.normalization) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.BatchNormalization instead.  In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.BatchNormalization` documentation).\n",
      "W0915 16:49:08.100606 139882633475904 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: batch_normalization (from tensorflow.python.keras.legacy_tf_layers.normalization) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.BatchNormalization instead.  In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.BatchNormalization` documentation).\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: batch_normalization (from tensorflow.python.keras.legacy_tf_layers.normalization) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.BatchNormalization instead.  In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.BatchNormalization` documentation).\n",
      "W0915 16:48:46.780948 139982537570112 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: batch_normalization (from tensorflow.python.keras.legacy_tf_layers.normalization) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.BatchNormalization instead.  In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.BatchNormalization` documentation).\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "I0915 16:49:08.103891 139841635501888 estimator.py:1151] Calling model_fn.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: conv2d (from tensorflow.python.keras.legacy_tf_layers.convolutional) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.keras.layers.Conv2D` instead.\n",
      "W0915 16:49:08.104113 139841635501888 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: conv2d (from tensorflow.python.keras.legacy_tf_layers.convolutional) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.keras.layers.Conv2D` instead.\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `layer.__call__` method instead.\n",
      "W0915 16:49:08.108336 139841635501888 deprecation.py:323] From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `layer.__call__` method instead.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: batch_normalization (from tensorflow.python.keras.legacy_tf_layers.normalization) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.BatchNormalization instead.  In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.BatchNormalization` documentation).\n",
      "W0915 16:49:08.116875 139841635501888 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: batch_normalization (from tensorflow.python.keras.legacy_tf_layers.normalization) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.BatchNormalization instead.  In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.BatchNormalization` documentation).\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:575: dense (from tensorflow.python.keras.legacy_tf_layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.Dense instead.\n",
      "W0915 16:49:08.476785 139882633475904 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:575: dense (from tensorflow.python.keras.legacy_tf_layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.Dense instead.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:575: dense (from tensorflow.python.keras.legacy_tf_layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.Dense instead.\n",
      "W0915 16:48:47.160281 139894849677120 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:575: dense (from tensorflow.python.keras.legacy_tf_layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.Dense instead.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:575: dense (from tensorflow.python.keras.legacy_tf_layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.Dense instead.\n",
      "W0915 16:48:47.166304 139982537570112 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:575: dense (from tensorflow.python.keras.legacy_tf_layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.Dense instead.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:575: dense (from tensorflow.python.keras.legacy_tf_layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.Dense instead.\n",
      "W0915 16:49:08.497902 139841635501888 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:575: dense (from tensorflow.python.keras.legacy_tf_layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.Dense instead.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:444: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.\n",
      "\n",
      "W0915 16:49:09.005223 139882633475904 module_wrapper.py:138] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:444: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.\n",
      "\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:367: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.cast` instead.\n",
      "W0915 16:49:09.008785 139882633475904 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:367: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.cast` instead.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:444: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.\n",
      "\n",
      "W0915 16:48:47.689714 139982537570112 module_wrapper.py:138] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:444: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.\n",
      "\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:444: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.\n",
      "\n",
      "W0915 16:48:47.691575 139894849677120 module_wrapper.py:138] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:444: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.\n",
      "\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:367: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.cast` instead.\n",
      "W0915 16:48:47.693012 139982537570112 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:367: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.cast` instead.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:367: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.cast` instead.\n",
      "W0915 16:48:47.695047 139894849677120 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:367: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.cast` instead.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:444: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.\n",
      "\n",
      "W0915 16:49:09.036120 139841635501888 module_wrapper.py:138] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:444: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.\n",
      "\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:367: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.cast` instead.\n",
      "W0915 16:49:09.039641 139841635501888 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:367: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.cast` instead.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:415: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.cast` instead.\n",
      "W0915 16:48:47.899999 139982537570112 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:415: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.cast` instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "I0915 16:48:47.905089 139982537570112 estimator.py:1153] Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "I0915 16:48:47.905847 139982537570112 basic_session_run_hooks.py:546] Create CheckpointSaverHook.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:415: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.cast` instead.\n",
      "W0915 16:49:09.228209 139882633475904 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:415: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.cast` instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "I0915 16:49:09.233825 139882633475904 estimator.py:1153] Done calling model_fn.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:415: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.cast` instead.\n",
      "W0915 16:48:47.912051 139894849677120 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:415: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.cast` instead.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "I0915 16:49:09.234619 139882633475904 basic_session_run_hooks.py:546] Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "I0915 16:48:47.917414 139894849677120 estimator.py:1153] Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "I0915 16:48:47.918212 139894849677120 basic_session_run_hooks.py:546] Create CheckpointSaverHook.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:415: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.cast` instead.\n",
      "W0915 16:49:09.263667 139841635501888 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:415: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.cast` instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "I0915 16:49:09.269326 139841635501888 estimator.py:1153] Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "I0915 16:49:09.270148 139841635501888 basic_session_run_hooks.py:546] Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "I0915 16:49:09.778346 139882633475904 monitored_session.py:246] Graph was finalized.\n",
      "2022-09-15 16:49:09.778684: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical operations:  AVX512F\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2022-09-15 16:49:09.812872: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2600000000 Hz\n",
      "2022-09-15 16:49:09.814352: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x562e7819dbb0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
      "2022-09-15 16:49:09.814395: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n",
      "INFO:tensorflow:Restoring parameters from /root/mlperf/results-19/work_dir/model.ckpt-5672\n",
      "I0915 16:49:09.816694 139882633475904 saver.py:1293] Restoring parameters from /root/mlperf/results-19/work_dir/model.ckpt-5672\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "I0915 16:48:48.625157 139982537570112 monitored_session.py:246] Graph was finalized.\n",
      "2022-09-15 16:48:48.625541: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical operations:  AVX512F\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2022-09-15 16:48:48.631918: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2600000000 Hz\n",
      "2022-09-15 16:48:48.633051: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55a530e91540 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
      "2022-09-15 16:48:48.633095: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1077: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file utilities to get mtimes.\n",
      "W0915 16:49:09.961912 139882633475904 deprecation.py:323] From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1077: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file utilities to get mtimes.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "I0915 16:48:48.645053 139894849677120 monitored_session.py:246] Graph was finalized.\n",
      "2022-09-15 16:48:48.645413: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical operations:  AVX512F\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2022-09-15 16:48:48.652408: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2600000000 Hz\n",
      "2022-09-15 16:48:48.653407: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x557b8eb9c3f0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
      "2022-09-15 16:48:48.653435: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "I0915 16:49:10.001201 139841635501888 monitored_session.py:246] Graph was finalized.\n",
      "2022-09-15 16:49:10.001548: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical operations:  AVX512F\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2022-09-15 16:49:10.009969: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2600000000 Hz\n",
      "2022-09-15 16:49:10.010922: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55c8f5ee4d20 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
      "2022-09-15 16:49:10.010938: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "I0915 16:49:10.058501 139882633475904 session_manager.py:505] Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "I0915 16:49:10.094336 139882633475904 session_manager.py:508] Done running local_init_op.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "I0915 16:48:49.024530 139982537570112 session_manager.py:505] Running local_init_op.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "I0915 16:48:49.047545 139894849677120 session_manager.py:505] Running local_init_op.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "I0915 16:49:10.369851 139841635501888 session_manager.py:505] Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "I0915 16:48:49.063544 139982537570112 session_manager.py:508] Done running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "I0915 16:49:10.407806 139841635501888 session_manager.py:508] Done running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "I0915 16:48:49.087618 139894849677120 session_manager.py:508] Done running local_init_op.\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5672...\n",
      "I0915 16:49:11.139212 139882633475904 basic_session_run_hooks.py:614] Calling checkpoint listeners before saving checkpoint 5672...\n",
      "INFO:tensorflow:Saving checkpoints for 5672 into /root/mlperf/results-19/work_dir/model.ckpt.\n",
      "I0915 16:49:11.141494 139882633475904 basic_session_run_hooks.py:618] Saving checkpoints for 5672 into /root/mlperf/results-19/work_dir/model.ckpt.\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...\n",
      "I0915 16:48:50.073173 139982537570112 basic_session_run_hooks.py:614] Calling checkpoint listeners before saving checkpoint 0...\n",
      "INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmp4__k6blo/model.ckpt.\n",
      "I0915 16:48:50.075464 139982537570112 basic_session_run_hooks.py:618] Saving checkpoints for 0 into /tmp/tmp4__k6blo/model.ckpt.\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5672...\n",
      "I0915 16:49:11.402880 139882633475904 basic_session_run_hooks.py:626] Calling checkpoint listeners after saving checkpoint 5672...\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...\n",
      "I0915 16:48:50.123661 139894849677120 basic_session_run_hooks.py:614] Calling checkpoint listeners before saving checkpoint 0...\n",
      "INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmpc2459yoa/model.ckpt.\n",
      "I0915 16:48:50.126115 139894849677120 basic_session_run_hooks.py:618] Saving checkpoints for 0 into /tmp/tmpc2459yoa/model.ckpt.\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...\n",
      "I0915 16:49:11.460104 139841635501888 basic_session_run_hooks.py:614] Calling checkpoint listeners before saving checkpoint 0...\n",
      "INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmp3fjsbl6s/model.ckpt.\n",
      "I0915 16:49:11.464031 139841635501888 basic_session_run_hooks.py:618] Saving checkpoints for 0 into /tmp/tmp3fjsbl6s/model.ckpt.\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...\n",
      "I0915 16:48:50.345932 139982537570112 basic_session_run_hooks.py:626] Calling checkpoint listeners after saving checkpoint 0...\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...\n",
      "I0915 16:49:11.722611 139841635501888 basic_session_run_hooks.py:626] Calling checkpoint listeners after saving checkpoint 0...\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...\n",
      "I0915 16:48:50.401982 139894849677120 basic_session_run_hooks.py:626] Calling checkpoint listeners after saving checkpoint 0...\n",
      "INFO:tensorflow:loss = 4.1642804, step = 5672\n",
      "I0915 16:49:17.397910 139882633475904 basic_session_run_hooks.py:262] loss = 4.1642804, step = 5672\n",
      "INFO:tensorflow:loss = 4.09455, step = 5672\n",
      "I0915 16:49:17.402679 139841635501888 basic_session_run_hooks.py:262] loss = 4.09455, step = 5672\n",
      "INFO:tensorflow:loss = 4.0840816, step = 5672\n",
      "I0915 16:48:56.099236 139894849677120 basic_session_run_hooks.py:262] loss = 4.0840816, step = 5672\n",
      "INFO:tensorflow:loss = 4.1044617, step = 5672\n",
      "I0915 16:48:56.105665 139982537570112 basic_session_run_hooks.py:262] loss = 4.1044617, step = 5672\n",
      "INFO:tensorflow:global_step/sec: 0.292588\n",
      "I0915 16:52:08.258654 139882633475904 basic_session_run_hooks.py:702] global_step/sec: 0.292588\n",
      "INFO:tensorflow:loss = 4.0715637, step = 5722 (170.862 sec)\n",
      "I0915 16:52:08.259481 139882633475904 basic_session_run_hooks.py:260] loss = 4.0715637, step = 5722 (170.862 sec)\n",
      "I0915 16:54:37.444795 139982537570112 train.py:134] 5773: 0.293 steps per second\n",
      "INFO:tensorflow:global_step/sec: 0.292964\n",
      "I0915 16:54:37.445537 139982537570112 basic_session_run_hooks.py:702] global_step/sec: 0.292964\n",
      "I0915 16:54:37.445477 139894849677120 train.py:134] 5773: 0.293 steps per second\n",
      "INFO:tensorflow:global_step/sec: 0.293243\n",
      "I0915 16:54:58.766029 139841635501888 train.py:134] 5773: 0.293 steps per second\n",
      "INFO:tensorflow:global_step/sec: 0.292958\n",
      "I0915 16:54:37.446690 139894849677120 basic_session_run_hooks.py:702] global_step/sec: 0.292958\n",
      "INFO:tensorflow:global_step/sec: 0.292943\n",
      "I0915 16:54:58.766279 139882633475904 basic_session_run_hooks.py:702] global_step/sec: 0.293243\n",
      "I0915 16:54:58.767312 139841635501888 basic_session_run_hooks.py:702] global_step/sec: 0.292943\n",
      "I0915 16:54:58.768733 139882633475904 train.py:134] 5773: 0.293 steps per second\n",
      "INFO:tensorflow:global_step/sec: 0.292936\n",
      "I0915 16:54:58.769144 139882633475904 basic_session_run_hooks.py:702] global_step/sec: 0.292936\n",
      "INFO:tensorflow:loss = 4.1383414, step = 5772 (170.510 sec)\n",
      "I0915 16:54:58.769767 139882633475904 basic_session_run_hooks.py:260] loss = 4.1383414, step = 5772 (170.510 sec)\n",
      "INFO:tensorflow:global_step/sec: 0.262873\n",
      "I0915 16:58:08.972249 139882633475904 basic_session_run_hooks.py:702] global_step/sec: 0.262873\n",
      "INFO:tensorflow:loss = 4.0800633, step = 5822 (190.204 sec)\n",
      "I0915 16:58:08.974224 139882633475904 basic_session_run_hooks.py:260] loss = 4.0800633, step = 5822 (190.204 sec)\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5840...\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5840...\n",
      "I0915 16:59:14.448111 139882633475904 basic_session_run_hooks.py:614] Calling checkpoint listeners before saving checkpoint 5840...\n",
      "I0915 16:58:53.129374 139982537570112 basic_session_run_hooks.py:614] Calling checkpoint listeners before saving checkpoint 5840...\n",
      "INFO:tensorflow:Saving checkpoints for 5840 into /root/mlperf/results-19/work_dir/model.ckpt.\n",
      "INFO:tensorflow:Saving checkpoints for 5840 into /tmp/tmp4__k6blo/model.ckpt.\n",
      "I0915 16:59:14.448562 139882633475904 basic_session_run_hooks.py:618] Saving checkpoints for 5840 into /root/mlperf/results-19/work_dir/model.ckpt.\n",
      "I0915 16:58:53.129827 139982537570112 basic_session_run_hooks.py:618] Saving checkpoints for 5840 into /tmp/tmp4__k6blo/model.ckpt.\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5840...\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5840...\n",
      "I0915 16:58:53.129868 139894849677120 basic_session_run_hooks.py:614] Calling checkpoint listeners before saving checkpoint 5840...\n",
      "I0915 16:59:14.448612 139841635501888 basic_session_run_hooks.py:614] Calling checkpoint listeners before saving checkpoint 5840...\n",
      "INFO:tensorflow:Saving checkpoints for 5840 into /tmp/tmpc2459yoa/model.ckpt.\n",
      "INFO:tensorflow:Saving checkpoints for 5840 into /tmp/tmp3fjsbl6s/model.ckpt.\n",
      "I0915 16:58:53.130778 139894849677120 basic_session_run_hooks.py:618] Saving checkpoints for 5840 into /tmp/tmpc2459yoa/model.ckpt.\n",
      "I0915 16:59:14.449529 139841635501888 basic_session_run_hooks.py:618] Saving checkpoints for 5840 into /tmp/tmp3fjsbl6s/model.ckpt.\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5840...\n",
      "I0915 16:59:14.625483 139882633475904 basic_session_run_hooks.py:626] Calling checkpoint listeners after saving checkpoint 5840...\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5840...\n",
      "I0915 16:59:14.638520 139841635501888 basic_session_run_hooks.py:626] Calling checkpoint listeners after saving checkpoint 5840...\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5840...\n",
      "I0915 16:58:53.336978 139894849677120 basic_session_run_hooks.py:626] Calling checkpoint listeners after saving checkpoint 5840...\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5840...\n",
      "I0915 16:58:53.341241 139982537570112 basic_session_run_hooks.py:626] Calling checkpoint listeners after saving checkpoint 5840...\n",
      "I0915 17:01:00.636199 139982537570112 train.py:134] 5873: 0.261 steps per second\n",
      "INFO:tensorflow:global_step/sec: 0.260966\n",
      "INFO:tensorflow:global_step/sec: 0.259091\n",
      "I0915 17:01:00.636664 139894849677120 train.py:134] 5873: 0.261 steps per second\n",
      "I0915 17:01:00.636866 139982537570112 basic_session_run_hooks.py:702] global_step/sec: 0.260966\n",
      "I0915 17:01:21.954688 139882633475904 basic_session_run_hooks.py:702] global_step/sec: 0.259091\n",
      "INFO:tensorflow:global_step/sec: 0.260966\n",
      "I0915 17:01:21.954888 139841635501888 train.py:134] 5873: 0.261 steps per second\n",
      "I0915 17:01:00.637339 139894849677120 basic_session_run_hooks.py:702] global_step/sec: 0.260966\n",
      "INFO:tensorflow:global_step/sec: 0.260968\n",
      "I0915 17:01:21.956341 139882633475904 train.py:134] 5873: 0.261 steps per second\n",
      "I0915 17:01:21.956111 139841635501888 basic_session_run_hooks.py:702] global_step/sec: 0.260968\n",
      "INFO:tensorflow:global_step/sec: 0.260969\n",
      "I0915 17:01:21.956736 139882633475904 basic_session_run_hooks.py:702] global_step/sec: 0.260969\n",
      "INFO:tensorflow:loss = 4.1041765, step = 5872 (192.983 sec)\n",
      "I0915 17:01:21.957325 139882633475904 basic_session_run_hooks.py:260] loss = 4.1041765, step = 5872 (192.983 sec)\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5909...\n",
      "I0915 17:03:20.366278 139982537570112 basic_session_run_hooks.py:614] Calling checkpoint listeners before saving checkpoint 5909...\n",
      "INFO:tensorflow:Saving checkpoints for 5909 into /tmp/tmp4__k6blo/model.ckpt.\n",
      "I0915 17:03:20.366853 139982537570112 basic_session_run_hooks.py:618] Saving checkpoints for 5909 into /tmp/tmp4__k6blo/model.ckpt.\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5909...\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5909...\n",
      "I0915 17:03:20.366900 139894849677120 basic_session_run_hooks.py:614] Calling checkpoint listeners before saving checkpoint 5909...\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5909...\n",
      "INFO:tensorflow:Saving checkpoints for 5909 into /tmp/tmpc2459yoa/model.ckpt.\n",
      "I0915 17:03:41.684338 139841635501888 basic_session_run_hooks.py:614] Calling checkpoint listeners before saving checkpoint 5909...\n",
      "I0915 17:03:20.367796 139894849677120 basic_session_run_hooks.py:618] Saving checkpoints for 5909 into /tmp/tmpc2459yoa/model.ckpt.\n",
      "INFO:tensorflow:Saving checkpoints for 5909 into /tmp/tmp3fjsbl6s/model.ckpt.\n",
      "I0915 17:03:41.684453 139882633475904 basic_session_run_hooks.py:614] Calling checkpoint listeners before saving checkpoint 5909...\n",
      "I0915 17:03:41.685310 139841635501888 basic_session_run_hooks.py:618] Saving checkpoints for 5909 into /tmp/tmp3fjsbl6s/model.ckpt.\n",
      "INFO:tensorflow:Saving checkpoints for 5909 into /root/mlperf/results-19/work_dir/model.ckpt.\n",
      "I0915 17:03:41.685480 139882633475904 basic_session_run_hooks.py:618] Saving checkpoints for 5909 into /root/mlperf/results-19/work_dir/model.ckpt.\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5909...\n",
      "I0915 17:03:20.554218 139894849677120 basic_session_run_hooks.py:626] Calling checkpoint listeners after saving checkpoint 5909...\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5909...\n",
      "I0915 17:03:41.873648 139882633475904 basic_session_run_hooks.py:626] Calling checkpoint listeners after saving checkpoint 5909...\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5909...\n",
      "I0915 17:03:41.877823 139841635501888 basic_session_run_hooks.py:626] Calling checkpoint listeners after saving checkpoint 5909...\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5909...\n",
      "I0915 17:03:20.570525 139982537570112 basic_session_run_hooks.py:626] Calling checkpoint listeners after saving checkpoint 5909...\n",
      "INFO:tensorflow:Loss for final step: 4.064002.\n",
      "I0915 17:03:20.608705 139894849677120 estimator.py:375] Loss for final step: 4.064002.\n",
      "I0915 17:03:20.609991 139894849677120 utils.py:86] Training: 874.683 seconds\n",
      "INFO:tensorflow:Loss for final step: 4.1007457.\n",
      "I0915 17:03:41.931968 139841635501888 estimator.py:375] Loss for final step: 4.1007457.\n",
      "I0915 17:03:41.935151 139841635501888 utils.py:86] Training: 874.686 seconds\n",
      "INFO:tensorflow:Loss for final step: 4.0655293.\n",
      "I0915 17:03:41.941290 139882633475904 estimator.py:375] Loss for final step: 4.0655293.\n",
      "I0915 17:03:41.942601 139882633475904 utils.py:86] Training: 874.694 seconds\n",
      "INFO:tensorflow:Using default config.\n",
      "I0915 17:03:41.943171 139882633475904 estimator.py:1804] Using default config.\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/root/mlperf/results-19/work_dir', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "I0915 17:03:41.943528 139882633475904 estimator.py:216] Using config: {'_model_dir': '/root/mlperf/results-19/work_dir', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "I0915 17:03:41.946116 139882633475904 dual_net.py:755] Copying /root/mlperf/results-19/work_dir/model.ckpt-5909.data-00000-of-00001 to /root/mlperf/results-19/models/000018.data-00000-of-00001\n",
      "I0915 17:03:41.949675 139882633475904 dual_net.py:755] Copying /root/mlperf/results-19/work_dir/model.ckpt-5909.index to /root/mlperf/results-19/models/000018.index\n",
      "I0915 17:03:41.949830 139882633475904 dual_net.py:755] Copying /root/mlperf/results-19/work_dir/model.ckpt-5909.meta to /root/mlperf/results-19/models/000018.meta\n",
      "INFO:tensorflow:Loss for final step: 4.107523.\n",
      "I0915 17:03:20.630149 139982537570112 estimator.py:375] Loss for final step: 4.107523.\n",
      "I0915 17:03:20.633135 139982537570112 utils.py:86] Training: 874.706 seconds\n",
      "I0915 17:03:41.952477 139882633475904 dual_net.py:208] TPU inference is supported on C++ only. DualNetwork will ignore use_tpu=True\n",
      "INFO:tensorflow:Restoring parameters from /root/mlperf/results-19/models/000018\n",
      "I0915 17:03:43.040313 139882633475904 saver.py:1293] Restoring parameters from /root/mlperf/results-19/models/000018\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:812: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.compat.v1.graph_util.convert_variables_to_constants`\n",
      "W0915 17:03:43.150102 139882633475904 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/dual_net.py:812: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.compat.v1.graph_util.convert_variables_to_constants`\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/framework/graph_util_impl.py:359: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.compat.v1.graph_util.extract_sub_graph`\n",
      "W0915 17:03:43.150296 139882633475904 deprecation.py:323] From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/framework/graph_util_impl.py:359: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.compat.v1.graph_util.extract_sub_graph`\n",
      "INFO:tensorflow:Froze 77 variables.\n",
      "I0915 17:03:43.302446 139882633475904 graph_util_impl.py:431] Froze 77 variables.\n",
      "INFO:tensorflow:Converted 77 variables to const ops.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I0915 17:03:43.310590 139882633475904 graph_util_impl.py:524] Converted 77 variables to const ops.INFO:root:Training on 1 records: /tmp/golden_chunks/000018-00000-of-00004.tfrecord.zz to /tmp/golden_chunks/000018-00000-of-00004.tfrecord.zz\r\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/root/mlperf/results-19/work_dir', '_tf_random_seed': None, '_save_summary_steps': 128, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': intra_op_parallelism_threads: 17\r\n",
      "inter_op_parallelism_threads: 2\r\n",
      "gpu_options {\r\n",
      "  allow_growth: true\r\n",
      "}\r\n",
      ", '_keep_checkpoint_max': 100, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 50, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\r\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231747252, \"event_type\": \"POINT_IN_TIME\", \"key\": \"global_batch_size\", \"value\": 4196, \"metadata\": {\"file\": \"train.py\", \"lineno\": 201}}\r\n",
      "INFO:root:Real global batch size = 4196, local batch size = 1049.\r\n",
      "INFO:root:Training, steps = 237, batch = 1049 -> 248613 examples\r\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\r\n",
      "INFO:root:Reading tf_records from 1 inputs\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:216: unbatch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.data.Dataset.unbatch()`.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:219: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.\r\n",
      "INFO:tensorflow:Calling model_fn.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: conv2d (from tensorflow.python.keras.legacy_tf_layers.convolutional) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.keras.layers.Conv2D` instead.\r\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Please use `layer.__call__` method instead.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: batch_normalization (from tensorflow.python.keras.legacy_tf_layers.normalization) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use keras.layers.BatchNormalization instead.  In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.BatchNormalization` documentation).\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:575: dense (from tensorflow.python.keras.legacy_tf_layers.core) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use keras.layers.Dense instead.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:444: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.\r\n",
      "\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:367: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.cast` instead.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:415: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.cast` instead.\r\n",
      "INFO:tensorflow:Done calling model_fn.\r\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\r\n",
      "INFO:tensorflow:Graph was finalized.\r\n",
      "INFO:tensorflow:Restoring parameters from /root/mlperf/results-19/work_dir/model.ckpt-5672\r\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1077: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use standard file utilities to get mtimes.\r\n",
      "INFO:tensorflow:Running local_init_op.\r\n",
      "INFO:tensorflow:Done running local_init_op.\r\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5672...\r\n",
      "INFO:tensorflow:Saving checkpoints for 5672 into /root/mlperf/results-19/work_dir/model.ckpt.\r\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5672...\r\n",
      "INFO:tensorflow:loss = 4.1642804, step = 5672\r\n",
      "INFO:tensorflow:global_step/sec: 0.292588\r\n",
      "INFO:tensorflow:loss = 4.0715637, step = 5722 (170.862 sec)\r\n",
      "INFO:tensorflow:global_step/sec: 0.293243\r\n",
      "INFO:root:5773: 0.293 steps per second\r\n",
      "INFO:tensorflow:global_step/sec: 0.292936\r\n",
      "INFO:tensorflow:loss = 4.1383414, step = 5772 (170.510 sec)\r\n",
      "INFO:tensorflow:global_step/sec: 0.262873\r\n",
      "INFO:tensorflow:loss = 4.0800633, step = 5822 (190.204 sec)\r\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5840...\r\n",
      "INFO:tensorflow:Saving checkpoints for 5840 into /root/mlperf/results-19/work_dir/model.ckpt.\r\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5840...\r\n",
      "INFO:tensorflow:global_step/sec: 0.259091\r\n",
      "INFO:root:5873: 0.261 steps per second\r\n",
      "INFO:tensorflow:global_step/sec: 0.260969\r\n",
      "INFO:tensorflow:loss = 4.1041765, step = 5872 (192.983 sec)\r\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5909...\r\n",
      "INFO:tensorflow:Saving checkpoints for 5909 into /root/mlperf/results-19/work_dir/model.ckpt.\r\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5909...\r\n",
      "INFO:tensorflow:Loss for final step: 4.0655293.\r\n",
      "INFO:root:Training: 874.694 seconds\r\n",
      "INFO:tensorflow:Using default config.\r\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/root/mlperf/results-19/work_dir', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\r\n",
      "graph_options {\r\n",
      "  rewrite_options {\r\n",
      "    meta_optimizer_iterations: ONE\r\n",
      "  }\r\n",
      "}\r\n",
      ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\r\n",
      "INFO:root:Copying /root/mlperf/results-19/work_dir/model.ckpt-5909.data-00000-of-00001 to /root/mlperf/results-19/models/000018.data-00000-of-00001\r\n",
      "INFO:root:Copying /root/mlperf/results-19/work_dir/model.ckpt-5909.index to /root/mlperf/results-19/models/000018.index\r\n",
      "INFO:root:Copying /root/mlperf/results-19/work_dir/model.ckpt-5909.meta to /root/mlperf/results-19/models/000018.meta\r\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663232621950, \"event_type\": \"POINT_IN_TIME\", \"key\": \"save_model\", \"value\": {\"Iteration\": 18}, \"metadata\": {\"file\": \"train.py\", \"lineno\": 312}}\r\n",
      "INFO:root:TPU inference is supported on C++ only. DualNetwork will ignore use_tpu=True\r\n",
      "INFO:tensorflow:Restoring parameters from /root/mlperf/results-19/models/000018\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:812: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.compat.v1.graph_util.convert_variables_to_constants`\r\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/framework/graph_util_impl.py:359: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.compat.v1.graph_util.extract_sub_graph`\r\n",
      "INFO:tensorflow:Froze 77 variables.\r\n",
      "INFO:tensorflow:Converted 77 variables to const ops.\r\n",
      "INFO:root:Training on 1 records: /tmp/golden_chunks/000018-00001-of-00004.tfrecord.zz to /tmp/golden_chunks/000018-00001-of-00004.tfrecord.zz\r\n",
      "WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmp3fjsbl6s\r\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmp3fjsbl6s', '_tf_random_seed': None, '_save_summary_steps': 10000000, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': intra_op_parallelism_threads: 17\r\n",
      "inter_op_parallelism_threads: 2\r\n",
      "gpu_options {\r\n",
      "  allow_growth: true\r\n",
      "}\r\n",
      ", '_keep_checkpoint_max': 100, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 10000000, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\r\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231747253, \"event_type\": \"POINT_IN_TIME\", \"key\": \"global_batch_size\", \"value\": 4196, \"metadata\": {\"file\": \"train.py\", \"lineno\": 201}}\r\n",
      "INFO:root:Real global batch size = 4196, local batch size = 1049.\r\n",
      "INFO:root:Training, steps = 237, batch = 1049 -> 248613 examples\r\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\r\n",
      "INFO:root:Reading tf_records from 1 inputs\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:216: unbatch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.data.Dataset.unbatch()`.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:219: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.\r\n",
      "INFO:tensorflow:Calling model_fn.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: conv2d (from tensorflow.python.keras.legacy_tf_layers.convolutional) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.keras.layers.Conv2D` instead.\r\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Please use `layer.__call__` method instead.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: batch_normalization (from tensorflow.python.keras.legacy_tf_layers.normalization) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use keras.layers.BatchNormalization instead.  In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.BatchNormalization` documentation).\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:575: dense (from tensorflow.python.keras.legacy_tf_layers.core) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use keras.layers.Dense instead.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:444: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.\r\n",
      "\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:367: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.cast` instead.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:415: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.cast` instead.\r\n",
      "INFO:tensorflow:Done calling model_fn.\r\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\r\n",
      "INFO:tensorflow:Graph was finalized.\r\n",
      "INFO:tensorflow:Running local_init_op.\r\n",
      "INFO:tensorflow:Done running local_init_op.\r\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...\r\n",
      "INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmp3fjsbl6s/model.ckpt.\r\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...\r\n",
      "INFO:tensorflow:loss = 4.09455, step = 5672\r\n",
      "INFO:root:5773: 0.293 steps per second\r\n",
      "INFO:tensorflow:global_step/sec: 0.292943\r\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5840...\r\n",
      "INFO:tensorflow:Saving checkpoints for 5840 into /tmp/tmp3fjsbl6s/model.ckpt.\r\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5840...\r\n",
      "INFO:root:5873: 0.261 steps per second\r\n",
      "INFO:tensorflow:global_step/sec: 0.260968\r\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5909...\r\n",
      "INFO:tensorflow:Saving checkpoints for 5909 into /tmp/tmp3fjsbl6s/model.ckpt.\r\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5909...\r\n",
      "INFO:tensorflow:Loss for final step: 4.1007457.\r\n",
      "INFO:root:Training: 874.686 seconds\r\n",
      "INFO:root:Training on 1 records: /tmp/golden_chunks/000018-00002-of-00004.tfrecord.zz to /tmp/golden_chunks/000018-00002-of-00004.tfrecord.zz\r\n",
      "WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmp4__k6blo\r\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmp4__k6blo', '_tf_random_seed': None, '_save_summary_steps': 10000000, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': intra_op_parallelism_threads: 17\r\n",
      "inter_op_parallelism_threads: 2\r\n",
      "gpu_options {\r\n",
      "  allow_growth: true\r\n",
      "}\r\n",
      ", '_keep_checkpoint_max': 100, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 10000000, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\r\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231725931, \"event_type\": \"POINT_IN_TIME\", \"key\": \"global_batch_size\", \"value\": 4196, \"metadata\": {\"file\": \"train.py\", \"lineno\": 201}}\r\n",
      "INFO:root:Real global batch size = 4196, local batch size = 1049.\r\n",
      "INFO:root:Training, steps = 237, batch = 1049 -> 248613 examples\r\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\r\n",
      "INFO:root:Reading tf_records from 1 inputs\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:216: unbatch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.data.Dataset.unbatch()`.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:219: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.\r\n",
      "INFO:tensorflow:Calling model_fn.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: conv2d (from tensorflow.python.keras.legacy_tf_layers.convolutional) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.keras.layers.Conv2D` instead.\r\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Please use `layer.__call__` method instead.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: batch_normalization (from tensorflow.python.keras.legacy_tf_layers.normalization) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use keras.layers.BatchNormalization instead.  In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.BatchNormalization` documentation).\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:575: dense (from tensorflow.python.keras.legacy_tf_layers.core) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use keras.layers.Dense instead.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:444: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.\r\n",
      "\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:367: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.cast` instead.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:415: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.cast` instead.\r\n",
      "INFO:tensorflow:Done calling model_fn.\r\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\r\n",
      "INFO:tensorflow:Graph was finalized.\r\n",
      "INFO:tensorflow:Running local_init_op.\r\n",
      "INFO:tensorflow:Done running local_init_op.\r\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...\r\n",
      "INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmp4__k6blo/model.ckpt.\r\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...\r\n",
      "INFO:tensorflow:loss = 4.1044617, step = 5672\r\n",
      "INFO:root:5773: 0.293 steps per second\r\n",
      "INFO:tensorflow:global_step/sec: 0.292964\r\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5840...\r\n",
      "INFO:tensorflow:Saving checkpoints for 5840 into /tmp/tmp4__k6blo/model.ckpt.\r\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5840...\r\n",
      "INFO:root:5873: 0.261 steps per second\r\n",
      "INFO:tensorflow:global_step/sec: 0.260966\r\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5909...\r\n",
      "INFO:tensorflow:Saving checkpoints for 5909 into /tmp/tmp4__k6blo/model.ckpt.\r\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5909...\r\n",
      "INFO:tensorflow:Loss for final step: 4.107523.\r\n",
      "INFO:root:Training: 874.706 seconds\r\n",
      "INFO:root:Training on 1 records: /tmp/golden_chunks/000018-00003-of-00004.tfrecord.zz to /tmp/golden_chunks/000018-00003-of-00004.tfrecord.zz\r\n",
      "WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpc2459yoa\r\n",
      "INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmpc2459yoa', '_tf_random_seed': None, '_save_summary_steps': 10000000, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': intra_op_parallelism_threads: 17\r\n",
      "inter_op_parallelism_threads: 2\r\n",
      "gpu_options {\r\n",
      "  allow_growth: true\r\n",
      "}\r\n",
      ", '_keep_checkpoint_max': 100, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 10000000, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\r\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663231725930, \"event_type\": \"POINT_IN_TIME\", \"key\": \"global_batch_size\", \"value\": 4196, \"metadata\": {\"file\": \"train.py\", \"lineno\": 201}}\r\n",
      "INFO:root:Real global batch size = 4196, local batch size = 1049.\r\n",
      "INFO:root:Training, steps = 237, batch = 1049 -> 248613 examples\r\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\r\n",
      "INFO:root:Reading tf_records from 1 inputs\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:216: unbatch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.data.Dataset.unbatch()`.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:219: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.\r\n",
      "INFO:tensorflow:Calling model_fn.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: conv2d (from tensorflow.python.keras.legacy_tf_layers.convolutional) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.keras.layers.Conv2D` instead.\r\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Please use `layer.__call__` method instead.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:553: batch_normalization (from tensorflow.python.keras.legacy_tf_layers.normalization) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use keras.layers.BatchNormalization instead.  In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.BatchNormalization` documentation).\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:575: dense (from tensorflow.python.keras.legacy_tf_layers.core) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use keras.layers.Dense instead.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:444: The name tf.estimator.tpu.TPUEstimatorSpec is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimatorSpec instead.\r\n",
      "\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:367: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.cast` instead.\r\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/dual_net.py:415: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.cast` instead.\r\n",
      "INFO:tensorflow:Done calling model_fn.\r\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\r\n",
      "INFO:tensorflow:Graph was finalized.\r\n",
      "INFO:tensorflow:Running local_init_op.\r\n",
      "INFO:tensorflow:Done running local_init_op.\r\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...\r\n",
      "INFO:tensorflow:Saving checkpoints for 0 into /tmp/tmpc2459yoa/model.ckpt.\r\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...\r\n",
      "INFO:tensorflow:loss = 4.0840816, step = 5672\r\n",
      "INFO:root:5773: 0.293 steps per second\r\n",
      "INFO:tensorflow:global_step/sec: 0.292958\r\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5840...\r\n",
      "INFO:tensorflow:Saving checkpoints for 5840 into /tmp/tmpc2459yoa/model.ckpt.\r\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5840...\r\n",
      "INFO:root:5873: 0.261 steps per second\r\n",
      "INFO:tensorflow:global_step/sec: 0.260966\r\n",
      "INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 5909...\r\n",
      "INFO:tensorflow:Saving checkpoints for 5909 into /tmp/tmpc2459yoa/model.ckpt.\r\n",
      "INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 5909...\r\n",
      "INFO:tensorflow:Loss for final step: 4.064002.\r\n",
      "INFO:root:Training: 874.683 seconds\r\n",
      "\r\n",
      "[2022-09-15 17:03:43] Run selfplay with int8 precision.\r\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663232623905, \"event_type\": \"POINT_IN_TIME\", \"key\": \"epoch_stop\", \"value\": null, \"metadata\": {\"file\": \"ml_perf/train_loop.py\", \"lineno\": 576, \"epoch_num\": 18}}\r\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/tools/strip_unused_lib.py:88: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.compat.v1.graph_util.extract_sub_graph`\r\n",
      "[2022-09-15 17:03:43] From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/tools/strip_unused_lib.py:88: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.compat.v1.graph_util.extract_sub_graph`\r\n",
      "WARNING:tensorflow:From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/tools/optimize_for_inference_lib.py:117: remove_training_nodes (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.compat.v1.graph_util.remove_training_nodes`\r\n",
      "[2022-09-15 17:03:43] From /root/anaconda3/envs/minigo/lib/python3.6/site-packages/tensorflow/python/tools/optimize_for_inference_lib.py:117: remove_training_nodes (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\r\n",
      "Instructions for updating:\r\n",
      "Use `tf.compat.v1.graph_util.remove_training_nodes`\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2022-09-15 17:03:44] numactl -N 0 -l python3 produce_min_max_log.py --input_graph={} --flagfile=/root/mlperf/results-19/flags/architecture.flags --data_location=/tmp/golden_chunks/*.zz --num_steps=5 --batch_size=16 --random_rotation=True\n",
      "2022-09-15 17:03:44.995955: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical operations:  AVX512F\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2022-09-15 17:03:45.010873: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2600000000 Hz\n",
      "2022-09-15 17:03:45.017198: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55747f1fc5e0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
      "2022-09-15 17:03:45.017243: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n",
      "2022-09-15 17:03:45.017481: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.\n",
      "WARNING:tensorflow:From ./intel_quantization/quantize_graph/quantize_graph_base.py:608: quantize_v2 (from tensorflow.python.ops.array_ops) is deprecated and will be removed after 2017-10-25.\n",
      "Instructions for updating:\n",
      "`tf.quantize_v2` is deprecated, please use `tf.quantization.quantize` instead.\n",
      "[2022-09-15 17:03:45] From ./intel_quantization/quantize_graph/quantize_graph_base.py:608: quantize_v2 (from tensorflow.python.ops.array_ops) is deprecated and will be removed after 2017-10-25.\n",
      "Instructions for updating:\n",
      "`tf.quantize_v2` is deprecated, please use `tf.quantization.quantize` instead.\n",
      "2022-09-15 17:03:47.093119: I tensorflow_io/core/kernels/cpu_check.cc:128] Your CPU supports instructions that this TensorFlow IO binary was not compiled to use: AVX2 AVX512F FMA\n",
      "INFO:root:Reading tf_records from 1 inputs\n",
      "I0915 17:03:47.252222 140153842313024 preprocessing.py:200] Reading tf_records from 1 inputs\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:216: unbatch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.unbatch()`.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:216: unbatch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.unbatch()`.\n",
      "W0915 17:03:47.648647 140153842313024 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:216: unbatch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.unbatch()`.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:219: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:219: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.\n",
      "W0915 17:03:47.652735 140153842313024 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:219: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:223: DatasetV1.make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "This is a deprecated API that should only be used in TF 1 graph mode and legacy TF 2 graph mode available through `tf.compat.v1`. In all other situations -- namely, eager mode and inside `tf.function` -- you can consume dataset elements using `for elem in dataset: ...` or by explicitly creating iterator via `iterator = iter(dataset)` and fetching its elements via `values = next(iterator)`. Furthermore, this API is not available in TF 2. During the transition from TF 1 to TF 2 you can use `tf.compat.v1.data.make_one_shot_iterator(dataset)` to create a TF 1 graph mode style iterator for a dataset created through TF 2 APIs. Note that this should be a transient state of your code base as there are in general no guarantees about the interoperability of TF 1 and TF 2 code.\n",
      "WARNING:tensorflow:From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:223: DatasetV1.make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "This is a deprecated API that should only be used in TF 1 graph mode and legacy TF 2 graph mode available through `tf.compat.v1`. In all other situations -- namely, eager mode and inside `tf.function` -- you can consume dataset elements using `for elem in dataset: ...` or by explicitly creating iterator via `iterator = iter(dataset)` and fetching its elements via `values = next(iterator)`. Furthermore, this API is not available in TF 2. During the transition from TF 1 to TF 2 you can use `tf.compat.v1.data.make_one_shot_iterator(dataset)` to create a TF 1 graph mode style iterator for a dataset created through TF 2 APIs. Note that this should be a transient state of your code base as there are in general no guarantees about the interoperability of TF 1 and TF 2 code.\n",
      "W0915 17:03:47.951713 140153842313024 deprecation.py:323] From /root/e2eAIOK/modelzoo/minigo/preprocessing.py:223: DatasetV1.make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "This is a deprecated API that should only be used in TF 1 graph mode and legacy TF 2 graph mode available through `tf.compat.v1`. In all other situations -- namely, eager mode and inside `tf.function` -- you can consume dataset elements using `for elem in dataset: ...` or by explicitly creating iterator via `iterator = iter(dataset)` and fetching its elements via `values = next(iterator)`. Furthermore, this API is not available in TF 2. During the transition from TF 1 to TF 2 you can use `tf.compat.v1.data.make_one_shot_iterator(dataset)` to create a TF 1 graph mode style iterator for a dataset created through TF 2 APIs. Note that this should be a transient state of your code base as there are in general no guarantees about the interoperability of TF 1 and TF 2 code.\n",
      "2022-09-15 17:03:47.978150: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical operations:  AVX512F\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2022-09-15 17:03:47.997738: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2600000000 Hz\n",
      "2022-09-15 17:03:48.000739: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5654c8d5c190 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
      "2022-09-15 17:03:48.000781: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n",
      "2022-09-15 17:03:48.000940: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.\n",
      ";conv2d/Conv2D_eightbit_max_Cast__print__;__max:[1]\n",
      ";conv2d/Conv2D_eightbit_min_Cast__print__;__min:[0]\n",
      ";conv2d/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.85089421]\n",
      ";conv2d_1/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.62168598]\n",
      ";conv2d_2/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][3.79360437]\n",
      ";conv2d_3/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.97111917]\n",
      ";conv2d_4/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][6.59868717]\n",
      ";conv2d_5/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.70197749]\n",
      ";conv2d_6/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][7.60679626]\n",
      ";conv2d_7/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.83306837]\n",
      ";conv2d_8/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][8.39590359]\n",
      ";conv2d_9/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][3.45688128]\n",
      ";conv2d_10/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][9.29805374]\n",
      ";conv2d_11/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][4.99317932]\n",
      ";conv2d_12/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][14.3868504]\n",
      ";conv2d_14/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][1.03076017]\n",
      ";conv2d_13/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][4.95474195]\n",
      ";conv2d/Conv2D_eightbit_max_Cast__print__;__max:[1]\n",
      ";conv2d/Conv2D_eightbit_min_Cast__print__;__min:[0]\n",
      ";conv2d/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.4947505]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2022-09-15 17:03:58] Converted graph file is saved to: /root/mlperf/results-19/models/000018.pb\n",
      "[2022-09-15 17:03:59] Running: python3  convert.py  --bool_features=1  --conv_width=64  --dst=/root/mlperf/results-19/models/000018.minigo  --fc_width=64  --input_features=mlperf07  --input_graph=/root/mlperf/results-19/models/000018.pb  --input_layout=nhwc  --summary_steps=128  --trunk_layers=6  --value_cost_weight=0.5\n",
      "[2022-09-15 17:04:01] convert finished: 1.714 seconds\n",
      ";conv2d_1/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.69726038]\n",
      ";conv2d_2/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][4.28749228]\n",
      ";conv2d_3/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][3.34066153]\n",
      ";conv2d_4/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][6.16337395]\n",
      ";conv2d_5/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][3.46617103]\n",
      ";conv2d_6/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][7.54678202]\n",
      ";conv2d_7/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][3.60402417]\n",
      ";conv2d_8/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][11.152915]\n",
      ";conv2d_9/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][4.87781572]\n",
      ";conv2d_10/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][12.2959099]\n",
      ";conv2d_11/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][6.57900238]\n",
      ";conv2d_12/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][19.2551212]\n",
      ";conv2d_14/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][1.0842272]\n",
      ";conv2d_13/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][5.86307955]\n",
      ";conv2d/Conv2D_eightbit_max_Cast__print__;__max:[1]\n",
      ";conv2d/Conv2D_eightbit_min_Cast__print__;__min:[0]\n",
      ";conv2d/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.31115103]\n",
      ";conv2d_1/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.43950438]\n",
      ";conv2d_2/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][3.81706882]\n",
      ";conv2d_3/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.53481388]\n",
      ";conv2d_4/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][5.70735168]\n",
      ";conv2d_5/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.89082789]\n",
      ";conv2d_6/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][7.38067436]\n",
      ";conv2d_7/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.65962696]\n",
      ";conv2d_8/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][8.57310772]\n",
      ";conv2d_9/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][4.3558321]\n",
      ";conv2d_10/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][11.5576601]\n",
      ";conv2d_11/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][6.31803274]\n",
      ";conv2d_12/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][18.4378757]\n",
      ";conv2d_14/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][0.94545722]\n",
      ";conv2d_13/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][5.52064323]\n",
      ";conv2d/Conv2D_eightbit_max_Cast__print__;__max:[1]\n",
      ";conv2d/Conv2D_eightbit_min_Cast__print__;__min:[0]\n",
      ";conv2d/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.08435154]\n",
      ";conv2d_1/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.39266706]\n",
      ";conv2d_2/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][3.60771275]\n",
      ";conv2d_3/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.67695832]\n",
      ";conv2d_4/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][5.68586969]\n",
      ";conv2d_5/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.38961291]\n",
      ";conv2d_6/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][6.58029366]\n",
      ";conv2d_7/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.58448386]\n",
      ";conv2d_8/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][8.03901577]\n",
      ";conv2d_9/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][3.22239947]\n",
      ";conv2d_10/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][9.19163227]\n",
      ";conv2d_11/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][4.85502243]\n",
      ";conv2d_12/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][13.7222843]\n",
      ";conv2d_14/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][1.07349253]\n",
      ";conv2d_13/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][4.65688133]\n",
      ";conv2d/Conv2D_eightbit_max_Cast__print__;__max:[1]\n",
      ";conv2d/Conv2D_eightbit_min_Cast__print__;__min:[0]\n",
      ";conv2d/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.3003509]\n",
      ";conv2d_1/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.19808698]\n",
      ";conv2d_2/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][3.93636274]\n",
      ";conv2d_3/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][3.38569498]\n",
      ";conv2d_4/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][5.85387707]\n",
      ";conv2d_5/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.61002254]\n",
      ";conv2d_6/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][6.90551615]\n",
      ";conv2d_7/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][1.87051249]\n",
      ";conv2d_8/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][7.71753311]\n",
      ";conv2d_9/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][2.37434387]\n",
      ";conv2d_10/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][8.82016087]\n",
      ";conv2d_11/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][5.0071969]\n",
      ";conv2d_12/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][11.2927017]\n",
      ";conv2d_14/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][1.05026531]\n",
      ";conv2d_13/Conv2D_eightbit_requant_range__print__;__requant_min_max:[0][4.58893824]\n",
      ":::MLLOG {\"namespace\": \"worker1\", \"time_ms\": 1663232641428, \"event_type\": \"POINT_IN_TIME\", \"key\": \"epoch_start\", \"value\": null, \"metadata\": {\"file\": \"ml_perf/train_loop.py\", \"lineno\": 570, \"epoch_num\": 18}}\n",
      "[2022-09-15 17:04:01] Waiting for 8192 games in /tmp/selfplay/000018 (found 0)\n",
      "[2022-09-15 17:04:31] Waiting for 8192 games in /tmp/selfplay/000018 (found 0)\n",
      "[2022-09-15 17:05:01] Waiting for 8192 games in /tmp/selfplay/000018 (found 5)\n",
      "[2022-09-15 17:05:31] Waiting for 8192 games in /tmp/selfplay/000018 (found 20)\n",
      "[2022-09-15 17:06:02] Waiting for 8192 games in /tmp/selfplay/000018 (found 57)\n",
      "[2022-09-15 17:06:32] Waiting for 8192 games in /tmp/selfplay/000018 (found 111)\n",
      "[2022-09-15 17:07:03] Waiting for 8192 games in /tmp/selfplay/000018 (found 168)\n",
      "[2022-09-15 17:07:34] Waiting for 8192 games in /tmp/selfplay/000018 (found 225)\n",
      "[2022-09-15 17:08:05] Waiting for 8192 games in /tmp/selfplay/000018 (found 292)\n",
      "[2022-09-15 17:08:37] Waiting for 8192 games in /tmp/selfplay/000018 (found 364)\n",
      "[2022-09-15 17:09:08] Waiting for 8192 games in /tmp/selfplay/000018 (found 424)\n",
      "[2022-09-15 17:09:40] Waiting for 8192 games in /tmp/selfplay/000018 (found 473)\n",
      "[2022-09-15 17:10:11] Waiting for 8192 games in /tmp/selfplay/000018 (found 531)\n",
      "[2022-09-15 17:10:43] Waiting for 8192 games in /tmp/selfplay/000018 (found 595)\n",
      "[2022-09-15 17:11:14] Waiting for 8192 games in /tmp/selfplay/000018 (found 676)\n",
      "[2022-09-15 17:11:46] Waiting for 8192 games in /tmp/selfplay/000018 (found 727)\n",
      "[2022-09-15 17:12:17] Waiting for 8192 games in /tmp/selfplay/000018 (found 780)\n",
      "[2022-09-15 17:12:49] Waiting for 8192 games in /tmp/selfplay/000018 (found 834)\n",
      "[2022-09-15 17:13:20] Waiting for 8192 games in /tmp/selfplay/000018 (found 892)\n",
      "[2022-09-15 17:13:52] Waiting for 8192 games in /tmp/selfplay/000018 (found 947)\n",
      "[2022-09-15 17:14:23] Waiting for 8192 games in /tmp/selfplay/000018 (found 1026)\n",
      "[2022-09-15 17:14:55] Waiting for 8192 games in /tmp/selfplay/000018 (found 1088)\n",
      "[2022-09-15 17:15:26] Waiting for 8192 games in /tmp/selfplay/000018 (found 1162)\n",
      "[2022-09-15 17:15:58] Waiting for 8192 games in /tmp/selfplay/000018 (found 1216)\n",
      "[2022-09-15 17:16:30] Waiting for 8192 games in /tmp/selfplay/000018 (found 1289)\n",
      "[2022-09-15 17:17:01] Waiting for 8192 games in /tmp/selfplay/000018 (found 1341)\n",
      "[2022-09-15 17:17:33] Waiting for 8192 games in /tmp/selfplay/000018 (found 1407)\n",
      "[2022-09-15 17:18:05] Waiting for 8192 games in /tmp/selfplay/000018 (found 1461)\n",
      "[2022-09-15 17:18:36] Waiting for 8192 games in /tmp/selfplay/000018 (found 1523)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2022-09-15 17:19:08] Waiting for 8192 games in /tmp/selfplay/000018 (found 1580)\n",
      "[2022-09-15 17:19:40] Waiting for 8192 games in /tmp/selfplay/000018 (found 1636)\n",
      "[2022-09-15 17:20:12] Waiting for 8192 games in /tmp/selfplay/000018 (found 1694)\n",
      "[2022-09-15 17:20:44] Waiting for 8192 games in /tmp/selfplay/000018 (found 1765)\n",
      "[2022-09-15 17:21:15] Waiting for 8192 games in /tmp/selfplay/000018 (found 1818)\n",
      "[2022-09-15 17:21:47] Waiting for 8192 games in /tmp/selfplay/000018 (found 1880)\n",
      "[2022-09-15 17:22:19] Waiting for 8192 games in /tmp/selfplay/000018 (found 1939)\n",
      "[2022-09-15 17:22:51] Waiting for 8192 games in /tmp/selfplay/000018 (found 2000)\n",
      "[2022-09-15 17:23:23] Waiting for 8192 games in /tmp/selfplay/000018 (found 2067)\n",
      "[2022-09-15 17:23:55] Waiting for 8192 games in /tmp/selfplay/000018 (found 2124)\n",
      "[2022-09-15 17:24:26] Waiting for 8192 games in /tmp/selfplay/000018 (found 2195)\n",
      "[2022-09-15 17:24:58] Waiting for 8192 games in /tmp/selfplay/000018 (found 2257)\n",
      "[2022-09-15 17:25:30] Waiting for 8192 games in /tmp/selfplay/000018 (found 2315)\n",
      "[2022-09-15 17:26:02] Waiting for 8192 games in /tmp/selfplay/000018 (found 2375)\n",
      "[2022-09-15 17:26:34] Waiting for 8192 games in /tmp/selfplay/000018 (found 2435)\n",
      "[2022-09-15 17:27:06] Waiting for 8192 games in /tmp/selfplay/000018 (found 2493)\n",
      "[2022-09-15 17:27:38] Waiting for 8192 games in /tmp/selfplay/000018 (found 2550)\n",
      "[2022-09-15 17:28:10] Waiting for 8192 games in /tmp/selfplay/000018 (found 2620)\n",
      "[2022-09-15 17:28:42] Waiting for 8192 games in /tmp/selfplay/000018 (found 2683)\n",
      "[2022-09-15 17:29:14] Waiting for 8192 games in /tmp/selfplay/000018 (found 2746)\n",
      "[2022-09-15 17:29:46] Waiting for 8192 games in /tmp/selfplay/000018 (found 2827)\n"
     ]
    }
   ],
   "source": [
    "from e2eAIOK.SDA.SDA import SDA\n",
    "import yaml\n",
    "\n",
    "# create SDA settings\n",
    "settings = {}\n",
    "settings[\"enable_sigopt\"] = False\n",
    "# load MiniGo settings\n",
    "with open(\"<WORKSPACE>/e2eaiok/conf/e2eaiok_defaults_minigo_example.conf\") as f:\n",
    "    conf = yaml.load(f, Loader=yaml.FullLoader)\n",
    "settings.update(conf)\n",
    "\n",
    "sda = SDA(model=\"minigo\", settings=settings)\n",
    "sda.launch()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  },
  "vscode": {
   "interpreter": {
    "hash": "960de6b0d90e8c1b5b9252244488628d965641900559c86d48ac5737e2de9daf"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}