    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program.  If not, see <http://www.gnu.org/licenses/>.

    @author: Axel Brando Guillaumes

# Resolució d'un problema petit: la MNIST¶

Primerament obtindrem les dades amb el mateix format que tindrem posteriorment el problema de la Food-101. És a dir, com a imatges indivuales i separades en carpetes per cada classe.

## Obtenció de les imatges¶

Primer descarreguem les imatges des de l'Institutionen for systemteknik on teníem agrupades cada una de les imatges de cada classe juntes com podem observar al previsualització de la pàgina web

In [1]:

    from IPython.display import HTML
    HTML('<iframe src=http://www.isy.liu.se/cvl/ImageDB/public/MNIST_digits/ width=700 height=350></iframe>')

Out[1]:

I repartim les imatges .jpg en dues carpetes "MNIST-TEST" i "MNIST-TRAIN" conservant el nom original de "mnist_testX.jpg".

Una vegada descarregades i repartides ja podem generar les imatges individuals retallant les imatges descarregades

## Adaptació MNIST en imatges separades (retall d'imatges)¶

Amb el codi següent generem dues carpetes "TEST" i "TRAIN" que contindran \(10\) carpetes amb la nomenclatura "MNIST-X" que tindran a dins les diferents imatges anomenades "imagePixX-PixY.jpg", tal i com podem observar a la carpeta \(MNIST/\)

In [23]:

    from PIL import Image
    import os
    # size is width/height
    width = 28
    height = 28
    #Path on estan les imatges
    path = 'MNIST/'
    #Generalitzem per train i test el mateix procès
    tt = ('train','test')
    #Quants canals en total (inicial + extres) o capes volem que hi hagin
    channels = 4
    count = -1
    ch = 0

    for t in tt:
        for number in xrange (0,10):
            #Creem la carpeta si no existeix
            if not os.path.exists(path+str(t.upper())+'/MNIST-'+str(number)+'/'):
                os.makedirs(path+str(t.upper())+'/MNIST-'+str(number)+'/')
            #Carreguem la imatge
            img = Image.open(path+'MNIST-'+str(t.upper())+'/mnist_'+str(t)+''+str(number)+'.jpg')
            (w,h) = img.size

            for left in xrange (0,w,width):
                for top in xrange (0,h,height):
                    box = (left, top, left+width, top+height)
                    area = img.crop(box)
                    if (ch == 0):
                        count += 1
                        area.save(path+str(t.upper())+'/MNIST-'+str(number)+'/image'+str(count)+'.jpg', 'jpeg')
                        ch += 1
                    else:
                        area.save(path+str(t.upper())+'/MNIST-'+str(number)+'/image'+str(count)+'.jpg_'+str(ch-1), 'jpeg')
                        ch = (ch+1) % channels
            count = -1
            ch = 0

NOTA: Abans de continuar és recomenable borrar les imatges en negre. S'ha de tenir en compte que:

     Les últimes 12 imatges de test del número 0 són negres.
     Les últimes 21 imatges de test del número 1 són negres.
     Les últimes 24 imatges de test del número 2 són negres.
     Les últimes 14 imatges de test del número 3 són negres.
     Les últimes 10 imatges de test del número 4 són negres.
     Les últimes 8 imatges de test del número 5 són negres.
     Les últimes 3 imatges de test del número 6 són negres.
     Les últimes 28 imatges de test del número 7 són negres.
     Les últimes 18 imatges de test del número 8 són negres.
     Les últimes 15 imatges de test del número 9 són negres.

     Les últimes 6 imatges de train del número 0 són negres.
     Les últimes 64 imatges de train del número 1 són negres.
     Les últimes 48 imatges de train del número 2 són negres.
     Les últimes 31 imatges de train del número 3 són negres.
     Les últimes 10 imatges de train del número 4 són negres.
     Les últimes 55 imatges de train del número 5 són negres.
     Les últimes 11 imatges de train del número 6 són negres.
     Les últimes 55 imatges de train del número 7 són negres.
     Les últimes 1 imatges de train del número 8 són negres.
     Les últimes 57 imatges de train del número 9 són negres.

# Creació fitxers train.txt, val.txt i cat.txt¶

Creació test.txt i train.txt BD per MNIST NOTA: Si es vol seguir amb la nomenclatura que usen els exemples de Caffe millor renombrar test.txt -> val.txt

In [1]:

    import os
    from os import listdir
    from os.path import isdir,isfile, join

    tt = ('train','test')

    for t in tt:
        fwrite = open('MNIST/'+str(t)+'.txt','w')
        data_path = 'MNIST/'+str(t.upper())
        label = 0 #ID de la categoria

        cat = listdir(data_path)
        cat.sort()

        for c in cat:
            if isdir(join(data_path,c)):
                for f in listdir(join(data_path,c)):
                    if not isdir(f):
                        fwrite.write("%s %d\n" % (join(c+'/',f),label))
                label +=1
        fwrite.close()

## Creació carpetes train i val lmdb pel cas de Grayscale¶

A continuació exposem el contingut del respectiu create_imagenet.sh i la sortida per consola que va donar. Destaquem que en aquest cas vam activar la opció de '--gray ', per tant, indicant que era una imatge en escala de grissos el que llegia.

In [6]:

    %%bash

    #!/usr/bin/env sh
    # Create the imagenet lmdb inputs
    # N.B. set the path to the imagenet train + val data dirs

    EXAMPLE=MNIST
    DATA=MNIST
    TOOLS=caffe-master/build/tools

    TRAIN_DATA_ROOT=MNIST/TRAIN/
    VAL_DATA_ROOT=MNIST/TEST/

    # Set RESIZE=true to resize the images to 256x256. Leave as false if images have
    # already been resized using another tool.
    RESIZE=false
    if $RESIZE; then
      RESIZE_HEIGHT=256
      RESIZE_WIDTH=256
    else
      RESIZE_HEIGHT=0
      RESIZE_WIDTH=0
    fi

    if [ ! -d "$TRAIN_DATA_ROOT" ]; then
      echo "Error: TRAIN_DATA_ROOT is not a path to a directory: $TRAIN_DATA_ROOT"
      echo "Set the TRAIN_DATA_ROOT variable in create_imagenet.sh to the path" \
           "where the ImageNet training data is stored."
      exit 1
    fi

    if [ ! -d "$VAL_DATA_ROOT" ]; then
      echo "Error: VAL_DATA_ROOT is not a path to a directory: $VAL_DATA_ROOT"
      echo "Set the VAL_DATA_ROOT variable in create_imagenet.sh to the path" \
           "where the ImageNet validation data is stored."
      exit 1
    fi

    echo "Creating train lmdb..."

    GLOG_logtostderr=1 $TOOLS/convert_imageset \
        --resize_height=$RESIZE_HEIGHT \
        --resize_width=$RESIZE_WIDTH \
        --shuffle \
        --gray \
        $TRAIN_DATA_ROOT \
        $DATA/train.txt \
        $EXAMPLE/ilsvrc12_train_lmdb

    echo "Creating val lmdb..."

    GLOG_logtostderr=1 $TOOLS/convert_imageset \
        --resize_height=$RESIZE_HEIGHT \
        --resize_width=$RESIZE_WIDTH \
        --shuffle \
        --gray \
        $VAL_DATA_ROOT \
        $DATA/val.txt \
        $EXAMPLE/ilsvrc12_val_lmdb

    echo "Done."

    Creating train lmdb...
    Creating val lmdb...
    Done.

    I1104 20:42:14.956545  8512 convert_imageset.cpp:81] Shuffling data
    I1104 20:42:14.975646  8512 convert_imageset.cpp:84] A total of 60338 images.
    I1104 20:42:14.975744  8512 convert_imageset.cpp:115] Opening lmdb /home/axel/Desktop/MNIST/ilsvrc12_train_lmdb
    E1104 20:42:15.684790  8512 convert_imageset.cpp:188] Processed 1000 files.
    E1104 20:42:16.472280  8512 convert_imageset.cpp:188] Processed 2000 files.
    E1104 20:42:17.247606  8512 convert_imageset.cpp:188] Processed 3000 files.
    E1104 20:42:17.973546  8512 convert_imageset.cpp:188] Processed 4000 files.
    E1104 20:42:18.689604  8512 convert_imageset.cpp:188] Processed 5000 files.
    E1104 20:42:19.463524  8512 convert_imageset.cpp:188] Processed 6000 files.
    E1104 20:42:20.134738  8512 convert_imageset.cpp:188] Processed 7000 files.
    E1104 20:42:20.878711  8512 convert_imageset.cpp:188] Processed 8000 files.
    E1104 20:42:21.669595  8512 convert_imageset.cpp:188] Processed 9000 files.
    E1104 20:42:22.452239  8512 convert_imageset.cpp:188] Processed 10000 files.
    E1104 20:42:23.175295  8512 convert_imageset.cpp:188] Processed 11000 files.
    E1104 20:42:23.984663  8512 convert_imageset.cpp:188] Processed 12000 files.
    E1104 20:42:24.817477  8512 convert_imageset.cpp:188] Processed 13000 files.
    E1104 20:42:25.600950  8512 convert_imageset.cpp:188] Processed 14000 files.
    E1104 20:42:26.401057  8512 convert_imageset.cpp:188] Processed 15000 files.
    E1104 20:42:27.224357  8512 convert_imageset.cpp:188] Processed 16000 files.
    E1104 20:42:28.121120  8512 convert_imageset.cpp:188] Processed 17000 files.
    E1104 20:42:28.920454  8512 convert_imageset.cpp:188] Processed 18000 files.
    E1104 20:42:29.815907  8512 convert_imageset.cpp:188] Processed 19000 files.
    E1104 20:42:30.697712  8512 convert_imageset.cpp:188] Processed 20000 files.
    E1104 20:42:31.460176  8512 convert_imageset.cpp:188] Processed 21000 files.
    E1104 20:42:32.367120  8512 convert_imageset.cpp:188] Processed 22000 files.
    E1104 20:42:33.222878  8512 convert_imageset.cpp:188] Processed 23000 files.
    E1104 20:42:33.990088  8512 convert_imageset.cpp:188] Processed 24000 files.
    E1104 20:42:34.737023  8512 convert_imageset.cpp:188] Processed 25000 files.
    E1104 20:42:35.545500  8512 convert_imageset.cpp:188] Processed 26000 files.
    E1104 20:42:36.311972  8512 convert_imageset.cpp:188] Processed 27000 files.
    E1104 20:42:37.117563  8512 convert_imageset.cpp:188] Processed 28000 files.
    E1104 20:42:37.905856  8512 convert_imageset.cpp:188] Processed 29000 files.
    E1104 20:42:38.712456  8512 convert_imageset.cpp:188] Processed 30000 files.
    E1104 20:42:39.602605  8512 convert_imageset.cpp:188] Processed 31000 files.
    E1104 20:42:40.406149  8512 convert_imageset.cpp:188] Processed 32000 files.
    E1104 20:42:41.274211  8512 convert_imageset.cpp:188] Processed 33000 files.
    E1104 20:42:42.124608  8512 convert_imageset.cpp:188] Processed 34000 files.
    E1104 20:42:42.930728  8512 convert_imageset.cpp:188] Processed 35000 files.
    E1104 20:42:43.667781  8512 convert_imageset.cpp:188] Processed 36000 files.
    E1104 20:42:44.442765  8512 convert_imageset.cpp:188] Processed 37000 files.
    E1104 20:42:45.280186  8512 convert_imageset.cpp:188] Processed 38000 files.
    E1104 20:42:46.115538  8512 convert_imageset.cpp:188] Processed 39000 files.
    E1104 20:42:46.907263  8512 convert_imageset.cpp:188] Processed 40000 files.
    E1104 20:42:47.651217  8512 convert_imageset.cpp:188] Processed 41000 files.
    E1104 20:42:48.483060  8512 convert_imageset.cpp:188] Processed 42000 files.
    E1104 20:42:49.218037  8512 convert_imageset.cpp:188] Processed 43000 files.
    E1104 20:42:49.946715  8512 convert_imageset.cpp:188] Processed 44000 files.
    E1104 20:42:50.697901  8512 convert_imageset.cpp:188] Processed 45000 files.
    E1104 20:42:51.436594  8512 convert_imageset.cpp:188] Processed 46000 files.
    E1104 20:42:52.229699  8512 convert_imageset.cpp:188] Processed 47000 files.
    E1104 20:42:53.061976  8512 convert_imageset.cpp:188] Processed 48000 files.
    E1104 20:42:53.811810  8512 convert_imageset.cpp:188] Processed 49000 files.
    E1104 20:42:54.657794  8512 convert_imageset.cpp:188] Processed 50000 files.
    E1104 20:42:55.452798  8512 convert_imageset.cpp:188] Processed 51000 files.
    E1104 20:42:56.263339  8512 convert_imageset.cpp:188] Processed 52000 files.
    E1104 20:42:57.065886  8512 convert_imageset.cpp:188] Processed 53000 files.
    E1104 20:42:57.885442  8512 convert_imageset.cpp:188] Processed 54000 files.
    E1104 20:42:58.868077  8512 convert_imageset.cpp:188] Processed 55000 files.
    E1104 20:42:59.835031  8512 convert_imageset.cpp:188] Processed 56000 files.
    E1104 20:43:00.715503  8512 convert_imageset.cpp:188] Processed 57000 files.
    E1104 20:43:01.612867  8512 convert_imageset.cpp:188] Processed 58000 files.
    E1104 20:43:02.422241  8512 convert_imageset.cpp:188] Processed 59000 files.
    E1104 20:43:03.323724  8512 convert_imageset.cpp:188] Processed 60000 files.
    E1104 20:43:03.623267  8512 convert_imageset.cpp:204] Processed 60338 files.
    I1104 20:43:03.683461  8542 convert_imageset.cpp:81] Shuffling data
    I1104 20:43:03.686015  8542 convert_imageset.cpp:84] A total of 10000 images.
    I1104 20:43:03.686079  8542 convert_imageset.cpp:115] Opening lmdb /home/axel/Desktop/MNIST/ilsvrc12_val_lmdb
    E1104 20:43:04.282677  8542 convert_imageset.cpp:188] Processed 1000 files.
    E1104 20:43:04.846012  8542 convert_imageset.cpp:188] Processed 2000 files.
    E1104 20:43:05.390517  8542 convert_imageset.cpp:188] Processed 3000 files.
    E1104 20:43:06.006587  8542 convert_imageset.cpp:188] Processed 4000 files.
    E1104 20:43:06.548758  8542 convert_imageset.cpp:188] Processed 5000 files.
    E1104 20:43:07.090061  8542 convert_imageset.cpp:188] Processed 6000 files.
    E1104 20:43:07.703146  8542 convert_imageset.cpp:188] Processed 7000 files.
    E1104 20:43:08.284361  8542 convert_imageset.cpp:188] Processed 8000 files.
    E1104 20:43:08.905680  8542 convert_imageset.cpp:188] Processed 9000 files.
    E1104 20:43:09.466497  8542 convert_imageset.cpp:188] Processed 10000 files.

# Creació fitxer imatge mitja .binaryproto¶

El codi posterior correspondrà al fitxer make_imagenet_mean.sh

In [8]:

    %%bash
    #!/usr/bin/env sh
    # Compute the mean image from the imagenet training leveldb
    # N.B. this is available in data/ilsvrc12

    /home/axel/Desktop/caffe-master/build/tools/compute_image_mean /home/axel/Desktop/MNIST/ilsvrc12_train_lmdb \
      /home/axel/Desktop/MNIST/imagenet_mean.binaryproto

    echo "Done."

    Done.

    E1104 20:47:37.909678  8649 compute_image_mean.cpp:140] Processed 10000 files.
    E1104 20:47:38.556596  8649 compute_image_mean.cpp:140] Processed 20000 files.
    E1104 20:47:39.222244  8649 compute_image_mean.cpp:140] Processed 30000 files.
    E1104 20:47:39.897441  8649 compute_image_mean.cpp:140] Processed 40000 files.
    E1104 20:47:40.566815  8649 compute_image_mean.cpp:140] Processed 50000 files.
    E1104 20:47:41.224616  8649 compute_image_mean.cpp:140] Processed 60000 files.
    E1104 20:47:41.246306  8649 compute_image_mean.cpp:149] Processed 60338 files.

# Entrenament xarxa, generació fitxer prototxt¶

El contingut posterior correspondrà al fitxer train_caffenet.sh

In [90]:

    %%bash
    #!/usr/bin/env sh

    caffe-master/build/tools/caffe train \
        --solver=MNIST/Grayscale/lenet_solver.prototxt

    I1024 18:55:54.690258  8880 caffe.cpp:103] Use CPU.
    I1024 18:55:54.690579  8880 caffe.cpp:107] Starting Optimization
    I1024 18:55:54.690692  8880 solver.cpp:32] Initializing solver from parameters:
    test_iter: 100
    test_interval: 500
    base_lr: 0.01
    display: 100
    max_iter: 10000
    lr_policy: "inv"
    gamma: 0.0001
    power: 0.75
    momentum: 0.9
    weight_decay: 0.0005
    snapshot: 5000
    snapshot_prefix: "/home/axel/Desktop/MNIST/caffenet"
    solver_mode: CPU
    net: "/home/axel/Desktop/MNIST/caffenet/lenet_train_test2.prototxt"
    I1024 18:55:54.690747  8880 solver.cpp:67] Creating training net from net file: /home/axel/Desktop/MNIST/caffenet/lenet_train_test2.prototxt
    I1024 18:55:54.691124  8880 net.cpp:275] The NetState phase (0) differed from the phase (1) specified by a rule in layer mnist
    I1024 18:55:54.691160  8880 net.cpp:275] The NetState phase (0) differed from the phase (1) specified by a rule in layer accuracy
    I1024 18:55:54.691270  8880 net.cpp:39] Initializing net from parameters:
    name: "LeNet"
    layers {
      top: "data"
      top: "label"
      name: "mnist"
      type: DATA
      data_param {
        source: "/home/axel/Desktop/MNIST/ilsvrc12_train_lmdb"
        batch_size: 64
        backend: LMDB
      }
      include {
        phase: TRAIN
      }
      transform_param {
        scale: 0.00390625
        mean_file: "/home/axel/Desktop/MNIST/imagenet_mean.binaryproto"
      }
    }
    layers {
      bottom: "data"
      top: "conv1"
      name: "conv1"
      type: CONVOLUTION
      blobs_lr: 1
      blobs_lr: 2
      convolution_param {
        num_output: 20
        kernel_size: 5
        stride: 1
        weight_filler {
          type: "xavier"
        }
        bias_filler {
          type: "constant"
        }
      }
    }
    layers {
      bottom: "conv1"
      top: "pool1"
      name: "pool1"
      type: POOLING
      pooling_param {
        pool: MAX
        kernel_size: 2
        stride: 2
      }
    }
    layers {
      bottom: "pool1"
      top: "conv2"
      name: "conv2"
      type: CONVOLUTION
      blobs_lr: 1
      blobs_lr: 2
      convolution_param {
        num_output: 50
        kernel_size: 5
        stride: 1
        weight_filler {
          type: "xavier"
        }
        bias_filler {
          type: "constant"
        }
      }
    }
    layers {
      bottom: "conv2"
      top: "pool2"
      name: "pool2"
      type: POOLING
      pooling_param {
        pool: MAX
        kernel_size: 2
        stride: 2
      }
    }
    layers {
      bottom: "pool2"
      top: "ip1"
      name: "ip1"
      type: INNER_PRODUCT
      blobs_lr: 1
      blobs_lr: 2
      inner_product_param {
        num_output: 500
        weight_filler {
          type: "xavier"
        }
        bias_filler {
          type: "constant"
        }
      }
    }
    layers {
      bottom: "ip1"
      top: "ip1"
      name: "relu1"
      type: RELU
    }
    layers {
      bottom: "ip1"
      top: "ip2"
      name: "ip2"
      type: INNER_PRODUCT
      blobs_lr: 1
      blobs_lr: 2
      inner_product_param {
        num_output: 10
        weight_filler {
          type: "xavier"
        }
        bias_filler {
          type: "constant"
        }
      }
    }
    layers {
      bottom: "ip2"
      bottom: "label"
      top: "loss"
      name: "loss"
      type: SOFTMAX_LOSS
    }
    state {
      phase: TRAIN
    }
    I1024 18:55:54.691381  8880 net.cpp:56] Memory required for data: 0
    I1024 18:55:54.691453  8880 net.cpp:67] Creating Layer mnist
    I1024 18:55:54.691470  8880 net.cpp:356] mnist -> data
    I1024 18:55:54.691512  8880 net.cpp:356] mnist -> label
    I1024 18:55:54.691542  8880 net.cpp:96] Setting up mnist
    I1024 18:55:54.691683  8880 data_layer.cpp:68] Opening lmdb /home/axel/Desktop/MNIST/ilsvrc12_train_lmdb
    I1024 18:55:54.691725  8880 data_layer.cpp:128] output data size: 64,4,28,28
    I1024 18:55:54.691736  8880 base_data_layer.cpp:36] Loading mean file from/home/axel/Desktop/MNIST/imagenet_mean.binaryproto
    I1024 18:55:54.692273  8880 base_data_layer.cpp:64] Initializing prefetch
    I1024 18:55:54.692364  8880 base_data_layer.cpp:66] Prefetch initialized.
    I1024 18:55:54.692404  8880 net.cpp:103] Top shape: 64 4 28 28 (200704)
    I1024 18:55:54.692435  8880 net.cpp:103] Top shape: 64 1 1 1 (64)
    I1024 18:55:54.692461  8880 net.cpp:113] Memory required for data: 803072
    I1024 18:55:54.692509  8880 net.cpp:67] Creating Layer conv1
    I1024 18:55:54.692651  8880 net.cpp:394] conv1 <- data
    I1024 18:55:54.692705  8880 net.cpp:356] conv1 -> conv1
    I1024 18:55:54.692752  8880 net.cpp:96] Setting up conv1
    I1024 18:55:54.693794  8880 net.cpp:103] Top shape: 64 20 24 24 (737280)
    I1024 18:55:54.693848  8880 net.cpp:113] Memory required for data: 3752192
    I1024 18:55:54.693976  8880 net.cpp:67] Creating Layer pool1
    I1024 18:55:54.694010  8880 net.cpp:394] pool1 <- conv1
    I1024 18:55:54.694054  8880 net.cpp:356] pool1 -> pool1
    I1024 18:55:54.694093  8880 net.cpp:96] Setting up pool1
    I1024 18:55:54.694141  8880 net.cpp:103] Top shape: 64 20 12 12 (184320)
    I1024 18:55:54.694169  8880 net.cpp:113] Memory required for data: 4489472
    I1024 18:55:54.694208  8880 net.cpp:67] Creating Layer conv2
    I1024 18:55:54.694236  8880 net.cpp:394] conv2 <- pool1
    I1024 18:55:54.694277  8880 net.cpp:356] conv2 -> conv2
    I1024 18:55:54.694316  8880 net.cpp:96] Setting up conv2
    I1024 18:55:54.696452  8880 net.cpp:103] Top shape: 64 50 8 8 (204800)
    I1024 18:55:54.696483  8880 net.cpp:113] Memory required for data: 5308672
    I1024 18:55:54.696568  8880 net.cpp:67] Creating Layer pool2
    I1024 18:55:54.696583  8880 net.cpp:394] pool2 <- conv2
    I1024 18:55:54.696609  8880 net.cpp:356] pool2 -> pool2
    I1024 18:55:54.696631  8880 net.cpp:96] Setting up pool2
    I1024 18:55:54.696652  8880 net.cpp:103] Top shape: 64 50 4 4 (51200)
    I1024 18:55:54.696663  8880 net.cpp:113] Memory required for data: 5513472
    I1024 18:55:54.696682  8880 net.cpp:67] Creating Layer ip1
    I1024 18:55:54.696692  8880 net.cpp:394] ip1 <- pool2
    I1024 18:55:54.696714  8880 net.cpp:356] ip1 -> ip1
    I1024 18:55:54.696733  8880 net.cpp:96] Setting up ip1
    I1024 18:55:54.728989  8880 net.cpp:103] Top shape: 64 500 1 1 (32000)
    I1024 18:55:54.729023  8880 net.cpp:113] Memory required for data: 5641472
    I1024 18:55:54.729060  8880 net.cpp:67] Creating Layer relu1
    I1024 18:55:54.729070  8880 net.cpp:394] relu1 <- ip1
    I1024 18:55:54.729085  8880 net.cpp:345] relu1 -> ip1 (in-place)
    I1024 18:55:54.729095  8880 net.cpp:96] Setting up relu1
    I1024 18:55:54.729102  8880 net.cpp:103] Top shape: 64 500 1 1 (32000)
    I1024 18:55:54.729107  8880 net.cpp:113] Memory required for data: 5769472
    I1024 18:55:54.729118  8880 net.cpp:67] Creating Layer ip2
    I1024 18:55:54.729125  8880 net.cpp:394] ip2 <- ip1
    I1024 18:55:54.729136  8880 net.cpp:356] ip2 -> ip2
    I1024 18:55:54.729148  8880 net.cpp:96] Setting up ip2
    I1024 18:55:54.729550  8880 net.cpp:103] Top shape: 64 10 1 1 (640)
    I1024 18:55:54.729558  8880 net.cpp:113] Memory required for data: 5772032
    I1024 18:55:54.729590  8880 net.cpp:67] Creating Layer loss
    I1024 18:55:54.729603  8880 net.cpp:394] loss <- ip2
    I1024 18:55:54.729619  8880 net.cpp:394] loss <- label
    I1024 18:55:54.729639  8880 net.cpp:356] loss -> loss
    I1024 18:55:54.729665  8880 net.cpp:96] Setting up loss
    I1024 18:55:54.729692  8880 net.cpp:103] Top shape: 1 1 1 1 (1)
    I1024 18:55:54.729701  8880 net.cpp:109]     with loss weight 1
    I1024 18:55:54.729753  8880 net.cpp:113] Memory required for data: 5772036
    I1024 18:55:54.729763  8880 net.cpp:170] loss needs backward computation.
    I1024 18:55:54.729775  8880 net.cpp:170] ip2 needs backward computation.
    I1024 18:55:54.729784  8880 net.cpp:170] relu1 needs backward computation.
    I1024 18:55:54.729794  8880 net.cpp:170] ip1 needs backward computation.
    I1024 18:55:54.729802  8880 net.cpp:170] pool2 needs backward computation.
    I1024 18:55:54.729814  8880 net.cpp:170] conv2 needs backward computation.
    I1024 18:55:54.729823  8880 net.cpp:170] pool1 needs backward computation.
    I1024 18:55:54.729835  8880 net.cpp:170] conv1 needs backward computation.
    I1024 18:55:54.729845  8880 net.cpp:172] mnist does not need backward computation.
    I1024 18:55:54.729854  8880 net.cpp:208] This network produces output loss
    I1024 18:55:54.729882  8880 net.cpp:467] Collecting Learning Rate and Weight Decay.
    I1024 18:55:54.729899  8880 net.cpp:219] Network initialization done.
    I1024 18:55:54.729908  8880 net.cpp:220] Memory required for data: 5772036
    I1024 18:55:54.730250  8880 solver.cpp:151] Creating test net (#0) specified by net file: /home/axel/Desktop/MNIST/caffenet/lenet_train_test2.prototxt
    I1024 18:55:54.730307  8880 net.cpp:275] The NetState phase (1) differed from the phase (0) specified by a rule in layer mnist
    I1024 18:55:54.730432  8880 net.cpp:39] Initializing net from parameters:
    name: "LeNet"
    layers {
      top: "data"
      top: "label"
      name: "mnist"
      type: DATA
      data_param {
        source: "/home/axel/Desktop/MNIST/ilsvrc12_val_lmdb"
        batch_size: 100
        backend: LMDB
      }
      include {
        phase: TEST
      }
      transform_param {
        scale: 0.00390625
        mean_file: "/home/axel/Desktop/MNIST/imagenet_mean.binaryproto"
      }
    }
    layers {
      bottom: "data"
      top: "conv1"
      name: "conv1"
      type: CONVOLUTION
      blobs_lr: 1
      blobs_lr: 2
      convolution_param {
        num_output: 20
        kernel_size: 5
        stride: 1
        weight_filler {
          type: "xavier"
        }
        bias_filler {
          type: "constant"
        }
      }
    }
    layers {
      bottom: "conv1"
      top: "pool1"
      name: "pool1"
      type: POOLING
      pooling_param {
        pool: MAX
        kernel_size: 2
        stride: 2
      }
    }
    layers {
      bottom: "pool1"
      top: "conv2"
      name: "conv2"
      type: CONVOLUTION
      blobs_lr: 1
      blobs_lr: 2
      convolution_param {
        num_output: 50
        kernel_size: 5
        stride: 1
        weight_filler {
          type: "xavier"
        }
        bias_filler {
          type: "constant"
        }
      }
    }
    layers {
      bottom: "conv2"
      top: "pool2"
      name: "pool2"
      type: POOLING
      pooling_param {
        pool: MAX
        kernel_size: 2
        stride: 2
      }
    }
    layers {
      bottom: "pool2"
      top: "ip1"
      name: "ip1"
      type: INNER_PRODUCT
      blobs_lr: 1
      blobs_lr: 2
      inner_product_param {
        num_output: 500
        weight_filler {
          type: "xavier"
        }
        bias_filler {
          type: "constant"
        }
      }
    }
    layers {
      bottom: "ip1"
      top: "ip1"
      name: "relu1"
      type: RELU
    }
    layers {
      bottom: "ip1"
      top: "ip2"
      name: "ip2"
      type: INNER_PRODUCT
      blobs_lr: 1
      blobs_lr: 2
      inner_product_param {
        num_output: 10
        weight_filler {
          type: "xavier"
        }
        bias_filler {
          type: "constant"
        }
      }
    }
    layers {
      bottom: "ip2"
      bottom: "label"
      top: "accuracy"
      name: "accuracy"
      type: ACCURACY
      include {
        phase: TEST
      }
    }
    layers {
      bottom: "ip2"
      bottom: "label"
      top: "loss"
      name: "loss"
      type: SOFTMAX_LOSS
    }
    state {
      phase: TEST
    }
    I1024 18:55:54.730597  8880 net.cpp:56] Memory required for data: 0
    I1024 18:55:54.730645  8880 net.cpp:67] Creating Layer mnist
    I1024 18:55:54.730661  8880 net.cpp:356] mnist -> data
    I1024 18:55:54.730691  8880 net.cpp:356] mnist -> label
    I1024 18:55:54.730711  8880 net.cpp:96] Setting up mnist
    I1024 18:55:54.730779  8880 data_layer.cpp:68] Opening lmdb /home/axel/Desktop/MNIST/ilsvrc12_val_lmdb
    I1024 18:55:54.730806  8880 data_layer.cpp:128] output data size: 100,4,28,28
    I1024 18:55:54.730818  8880 base_data_layer.cpp:36] Loading mean file from/home/axel/Desktop/MNIST/imagenet_mean.binaryproto
    I1024 18:55:54.731557  8880 base_data_layer.cpp:64] Initializing prefetch
    I1024 18:55:54.731608  8880 base_data_layer.cpp:66] Prefetch initialized.
    I1024 18:55:54.731642  8880 net.cpp:103] Top shape: 100 4 28 28 (313600)
    I1024 18:55:54.731673  8880 net.cpp:103] Top shape: 100 1 1 1 (100)
    I1024 18:55:54.731698  8880 net.cpp:113] Memory required for data: 1254800
    I1024 18:55:54.731740  8880 net.cpp:67] Creating Layer label_mnist_1_split
    I1024 18:55:54.731770  8880 net.cpp:394] label_mnist_1_split <- label
    I1024 18:55:54.731807  8880 net.cpp:356] label_mnist_1_split -> label_mnist_1_split_0
    I1024 18:55:54.731852  8880 net.cpp:356] label_mnist_1_split -> label_mnist_1_split_1
    I1024 18:55:54.731887  8880 net.cpp:96] Setting up label_mnist_1_split
    I1024 18:55:54.731919  8880 net.cpp:103] Top shape: 100 1 1 1 (100)
    I1024 18:55:54.731945  8880 net.cpp:103] Top shape: 100 1 1 1 (100)
    I1024 18:55:54.731972  8880 net.cpp:113] Memory required for data: 1255600
    I1024 18:55:54.732009  8880 net.cpp:67] Creating Layer conv1
    I1024 18:55:54.732039  8880 net.cpp:394] conv1 <- data
    I1024 18:55:54.732080  8880 net.cpp:356] conv1 -> conv1
    I1024 18:55:54.732121  8880 net.cpp:96] Setting up conv1
    I1024 18:55:54.732395  8880 net.cpp:103] Top shape: 100 20 24 24 (1152000)
    I1024 18:55:54.732426  8880 net.cpp:113] Memory required for data: 5863600
    I1024 18:55:54.732481  8880 net.cpp:67] Creating Layer pool1
    I1024 18:55:54.732511  8880 net.cpp:394] pool1 <- conv1
    I1024 18:55:54.732583  8880 net.cpp:356] pool1 -> pool1
    I1024 18:55:54.732645  8880 net.cpp:96] Setting up pool1
    I1024 18:55:54.732681  8880 net.cpp:103] Top shape: 100 20 12 12 (288000)
    I1024 18:55:54.732707  8880 net.cpp:113] Memory required for data: 7015600
    I1024 18:55:54.732743  8880 net.cpp:67] Creating Layer conv2
    I1024 18:55:54.732772  8880 net.cpp:394] conv2 <- pool1
    I1024 18:55:54.732808  8880 net.cpp:356] conv2 -> conv2
    I1024 18:55:54.732846  8880 net.cpp:96] Setting up conv2
    I1024 18:55:54.735841  8880 net.cpp:103] Top shape: 100 50 8 8 (320000)
    I1024 18:55:54.735930  8880 net.cpp:113] Memory required for data: 8295600
    I1024 18:55:54.736063  8880 net.cpp:67] Creating Layer pool2
    I1024 18:55:54.736099  8880 net.cpp:394] pool2 <- conv2
    I1024 18:55:54.736145  8880 net.cpp:356] pool2 -> pool2
    I1024 18:55:54.736177  8880 net.cpp:96] Setting up pool2
    I1024 18:55:54.736212  8880 net.cpp:103] Top shape: 100 50 4 4 (80000)
    I1024 18:55:54.736223  8880 net.cpp:113] Memory required for data: 8615600
    I1024 18:55:54.736259  8880 net.cpp:67] Creating Layer ip1
    I1024 18:55:54.736271  8880 net.cpp:394] ip1 <- pool2
    I1024 18:55:54.736299  8880 net.cpp:356] ip1 -> ip1
    I1024 18:55:54.736330  8880 net.cpp:96] Setting up ip1
    I1024 18:55:54.767534  8880 net.cpp:103] Top shape: 100 500 1 1 (50000)
    I1024 18:55:54.767627  8880 net.cpp:113] Memory required for data: 8815600
    I1024 18:55:54.767706  8880 net.cpp:67] Creating Layer relu1
    I1024 18:55:54.767746  8880 net.cpp:394] relu1 <- ip1
    I1024 18:55:54.767791  8880 net.cpp:345] relu1 -> ip1 (in-place)
    I1024 18:55:54.767830  8880 net.cpp:96] Setting up relu1
    I1024 18:55:54.767860  8880 net.cpp:103] Top shape: 100 500 1 1 (50000)
    I1024 18:55:54.767887  8880 net.cpp:113] Memory required for data: 9015600
    I1024 18:55:54.767935  8880 net.cpp:67] Creating Layer ip2
    I1024 18:55:54.767966  8880 net.cpp:394] ip2 <- ip1
    I1024 18:55:54.768013  8880 net.cpp:356] ip2 -> ip2
    I1024 18:55:54.768056  8880 net.cpp:96] Setting up ip2
    I1024 18:55:54.768688  8880 net.cpp:103] Top shape: 100 10 1 1 (1000)
    I1024 18:55:54.768740  8880 net.cpp:113] Memory required for data: 9019600
    I1024 18:55:54.768790  8880 net.cpp:67] Creating Layer ip2_ip2_0_split
    I1024 18:55:54.768815  8880 net.cpp:394] ip2_ip2_0_split <- ip2
    I1024 18:55:54.768851  8880 net.cpp:356] ip2_ip2_0_split -> ip2_ip2_0_split_0
    I1024 18:55:54.768890  8880 net.cpp:356] ip2_ip2_0_split -> ip2_ip2_0_split_1
    I1024 18:55:54.768910  8880 net.cpp:96] Setting up ip2_ip2_0_split
    I1024 18:55:54.768924  8880 net.cpp:103] Top shape: 100 10 1 1 (1000)
    I1024 18:55:54.768936  8880 net.cpp:103] Top shape: 100 10 1 1 (1000)
    I1024 18:55:54.768944  8880 net.cpp:113] Memory required for data: 9027600
    I1024 18:55:54.768978  8880 net.cpp:67] Creating Layer accuracy
    I1024 18:55:54.768992  8880 net.cpp:394] accuracy <- ip2_ip2_0_split_0
    I1024 18:55:54.769013  8880 net.cpp:394] accuracy <- label_mnist_1_split_0
    I1024 18:55:54.769042  8880 net.cpp:356] accuracy -> accuracy
    I1024 18:55:54.769068  8880 net.cpp:96] Setting up accuracy
    I1024 18:55:54.769088  8880 net.cpp:103] Top shape: 1 1 1 1 (1)
    I1024 18:55:54.769098  8880 net.cpp:113] Memory required for data: 9027604
    I1024 18:55:54.769124  8880 net.cpp:67] Creating Layer loss
    I1024 18:55:54.769136  8880 net.cpp:394] loss <- ip2_ip2_0_split_1
    I1024 18:55:54.769157  8880 net.cpp:394] loss <- label_mnist_1_split_1
    I1024 18:55:54.769187  8880 net.cpp:356] loss -> loss
    I1024 18:55:54.769212  8880 net.cpp:96] Setting up loss
    I1024 18:55:54.769243  8880 net.cpp:103] Top shape: 1 1 1 1 (1)
    I1024 18:55:54.769253  8880 net.cpp:109]     with loss weight 1
    I1024 18:55:54.769294  8880 net.cpp:113] Memory required for data: 9027608
    I1024 18:55:54.769305  8880 net.cpp:170] loss needs backward computation.
    I1024 18:55:54.769320  8880 net.cpp:172] accuracy does not need backward computation.
    I1024 18:55:54.769330  8880 net.cpp:170] ip2_ip2_0_split needs backward computation.
    I1024 18:55:54.769341  8880 net.cpp:170] ip2 needs backward computation.
    I1024 18:55:54.769352  8880 net.cpp:170] relu1 needs backward computation.
    I1024 18:55:54.769362  8880 net.cpp:170] ip1 needs backward computation.
    I1024 18:55:54.769440  8880 net.cpp:170] pool2 needs backward computation.
    I1024 18:55:54.769454  8880 net.cpp:170] conv2 needs backward computation.
    I1024 18:55:54.769465  8880 net.cpp:170] pool1 needs backward computation.
    I1024 18:55:54.769479  8880 net.cpp:170] conv1 needs backward computation.
    I1024 18:55:54.769491  8880 net.cpp:172] label_mnist_1_split does not need backward computation.
    I1024 18:55:54.769505  8880 net.cpp:172] mnist does not need backward computation.
    I1024 18:55:54.769512  8880 net.cpp:208] This network produces output accuracy
    I1024 18:55:54.769528  8880 net.cpp:208] This network produces output loss
    I1024 18:55:54.769564  8880 net.cpp:467] Collecting Learning Rate and Weight Decay.
    I1024 18:55:54.769590  8880 net.cpp:219] Network initialization done.
    I1024 18:55:54.769603  8880 net.cpp:220] Memory required for data: 9027608
    I1024 18:55:54.769721  8880 solver.cpp:41] Solver scaffolding done.
    I1024 18:55:54.769742  8880 solver.cpp:160] Solving LeNet
    I1024 18:55:54.769804  8880 solver.cpp:247] Iteration 0, Testing net (#0)
    I1024 18:55:54.769845  8880 net.cpp:652] Copying source layer mnist
    I1024 18:55:54.769872  8880 net.cpp:652] Copying source layer conv1
    I1024 18:55:54.769923  8880 net.cpp:652] Copying source layer pool1
    I1024 18:55:54.769953  8880 net.cpp:652] Copying source layer conv2
    I1024 18:55:54.769984  8880 net.cpp:652] Copying source layer pool2
    I1024 18:55:54.770011  8880 net.cpp:652] Copying source layer ip1
    I1024 18:55:54.770112  8880 net.cpp:652] Copying source layer relu1
    I1024 18:55:54.770122  8880 net.cpp:652] Copying source layer ip2
    I1024 18:55:54.770133  8880 net.cpp:652] Copying source layer loss
    I1024 18:56:08.048274 23451 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 18:56:18.737305  6583 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 18:56:30.381639 21832 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 18:56:43.315233  4839 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 18:56:43.784041  8880 solver.cpp:298]     Test net output #0: accuracy = 0.1034
    I1024 18:56:43.784090  8880 solver.cpp:298]     Test net output #1: loss = 2.30309 (* 1 = 2.30309 loss)
    I1024 18:56:44.674151  8880 solver.cpp:191] Iteration 0, loss = 2.3028
    I1024 18:56:44.674263  8880 solver.cpp:206]     Train net output #0: loss = 2.3028 (* 1 = 2.3028 loss)
    I1024 18:56:44.674336  8880 solver.cpp:403] Iteration 0, lr = 0.01
    I1024 18:57:38.986678  8880 solver.cpp:191] Iteration 100, loss = 0.134016
    I1024 18:57:38.986852  8880 solver.cpp:206]     Train net output #0: loss = 0.134016 (* 1 = 0.134016 loss)
    I1024 18:57:38.986867  8880 solver.cpp:403] Iteration 100, lr = 0.00992565
    I1024 18:59:05.019371  8880 solver.cpp:191] Iteration 200, loss = 0.0066966
    I1024 18:59:05.019651  8880 solver.cpp:206]     Train net output #0: loss = 0.0066966 (* 1 = 0.0066966 loss)
    I1024 18:59:05.019686  8880 solver.cpp:403] Iteration 200, lr = 0.00985258
    I1024 18:59:32.896081 27279 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:00:32.988786  8880 solver.cpp:191] Iteration 300, loss = 0.00795084
    I1024 19:00:32.989152  8880 solver.cpp:206]     Train net output #0: loss = 0.00795084 (* 1 = 0.00795084 loss)
    I1024 19:00:32.989195  8880 solver.cpp:403] Iteration 300, lr = 0.00978075
    I1024 19:02:01.326431  8880 solver.cpp:191] Iteration 400, loss = 0.00475308
    I1024 19:02:01.326812  8880 solver.cpp:206]     Train net output #0: loss = 0.00475308 (* 1 = 0.00475308 loss)
    I1024 19:02:01.326830  8880 solver.cpp:403] Iteration 400, lr = 0.00971013
    I1024 19:02:58.254317 17663 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:03:27.334368  8880 solver.cpp:247] Iteration 500, Testing net (#0)
    I1024 19:03:27.334444  8880 net.cpp:652] Copying source layer mnist
    I1024 19:03:27.334461  8880 net.cpp:652] Copying source layer conv1
    I1024 19:03:27.334481  8880 net.cpp:652] Copying source layer pool1
    I1024 19:03:27.334496  8880 net.cpp:652] Copying source layer conv2
    I1024 19:03:27.334509  8880 net.cpp:652] Copying source layer pool2
    I1024 19:03:27.334523  8880 net.cpp:652] Copying source layer ip1
    I1024 19:03:27.334538  8880 net.cpp:652] Copying source layer relu1
    I1024 19:03:27.334553  8880 net.cpp:652] Copying source layer ip2
    I1024 19:03:27.334568  8880 net.cpp:652] Copying source layer loss
    I1024 19:03:41.830448 25937 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:03:57.058837  8972 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:04:12.139385 24151 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:04:27.286569  7180 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:04:27.933670  8880 solver.cpp:298]     Test net output #0: accuracy = 0.9908
    I1024 19:04:27.933775  8880 solver.cpp:298]     Test net output #1: loss = 0.0316875 (* 1 = 0.0316875 loss)
    I1024 19:04:28.851443  8880 solver.cpp:191] Iteration 500, loss = 0.0202127
    I1024 19:04:28.851555  8880 solver.cpp:206]     Train net output #0: loss = 0.0202127 (* 1 = 0.0202127 loss)
    I1024 19:04:28.851591  8880 solver.cpp:403] Iteration 500, lr = 0.00964069
    I1024 19:05:55.857671  8880 solver.cpp:191] Iteration 600, loss = 0.11313
    I1024 19:05:55.858057  8880 solver.cpp:206]     Train net output #0: loss = 0.11313 (* 1 = 0.11313 loss)
    I1024 19:05:55.858098  8880 solver.cpp:403] Iteration 600, lr = 0.0095724
    I1024 19:07:24.161579  8880 solver.cpp:191] Iteration 700, loss = 0.00286153
    I1024 19:07:24.161979  8880 solver.cpp:206]     Train net output #0: loss = 0.00286153 (* 1 = 0.00286153 loss)
    I1024 19:07:24.162020  8880 solver.cpp:403] Iteration 700, lr = 0.00950522
    I1024 19:07:25.068833  5257 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:08:52.284776  8880 solver.cpp:191] Iteration 800, loss = 0.0117198
    I1024 19:08:52.285161  8880 solver.cpp:206]     Train net output #0: loss = 0.0117198 (* 1 = 0.0117198 loss)
    I1024 19:08:52.285204  8880 solver.cpp:403] Iteration 800, lr = 0.00943913
    I1024 19:10:20.136723  8880 solver.cpp:191] Iteration 900, loss = 0.0136038
    I1024 19:10:20.137110  8880 solver.cpp:206]     Train net output #0: loss = 0.0136038 (* 1 = 0.0136038 loss)
    I1024 19:10:20.137162  8880 solver.cpp:403] Iteration 900, lr = 0.00937411
    I1024 19:10:50.660236 27810 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:11:46.804934  8880 solver.cpp:247] Iteration 1000, Testing net (#0)
    I1024 19:11:46.805300  8880 net.cpp:652] Copying source layer mnist
    I1024 19:11:46.805330  8880 net.cpp:652] Copying source layer conv1
    I1024 19:11:46.805349  8880 net.cpp:652] Copying source layer pool1
    I1024 19:11:46.805362  8880 net.cpp:652] Copying source layer conv2
    I1024 19:11:46.805377  8880 net.cpp:652] Copying source layer pool2
    I1024 19:11:46.805389  8880 net.cpp:652] Copying source layer ip1
    I1024 19:11:46.805404  8880 net.cpp:652] Copying source layer relu1
    I1024 19:11:46.805416  8880 net.cpp:652] Copying source layer ip2
    I1024 19:11:46.805430  8880 net.cpp:652] Copying source layer loss
    I1024 19:12:01.399716 28212 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:12:16.528975 11243 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:12:31.680762 26423 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:12:46.499640  9455 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:12:47.133100  8880 solver.cpp:298]     Test net output #0: accuracy = 0.988
    I1024 19:12:47.133209  8880 solver.cpp:298]     Test net output #1: loss = 0.0422123 (* 1 = 0.0422123 loss)
    I1024 19:12:47.987821  8880 solver.cpp:191] Iteration 1000, loss = 0.00498034
    I1024 19:12:47.987936  8880 solver.cpp:206]     Train net output #0: loss = 0.00498034 (* 1 = 0.00498034 loss)
    I1024 19:12:47.987972  8880 solver.cpp:403] Iteration 1000, lr = 0.00931012
    I1024 19:14:14.783875  8880 solver.cpp:191] Iteration 1100, loss = 0.00187762
    I1024 19:14:14.785198  8880 solver.cpp:206]     Train net output #0: loss = 0.00187762 (* 1 = 0.00187762 loss)
    I1024 19:14:14.785258  8880 solver.cpp:403] Iteration 1100, lr = 0.00924715
    I1024 19:15:16.143067 15405 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:15:42.097728  8880 solver.cpp:191] Iteration 1200, loss = 0.00988762
    I1024 19:15:42.097829  8880 solver.cpp:206]     Train net output #0: loss = 0.00988762 (* 1 = 0.00988762 loss)
    I1024 19:15:42.097863  8880 solver.cpp:403] Iteration 1200, lr = 0.00918515
    I1024 19:17:10.013664  8880 solver.cpp:191] Iteration 1300, loss = 0.00746505
    I1024 19:17:10.013967  8880 solver.cpp:206]     Train net output #0: loss = 0.00746505 (* 1 = 0.00746505 loss)
    I1024 19:17:10.013993  8880 solver.cpp:403] Iteration 1300, lr = 0.00912412
    I1024 19:18:36.735046  8880 solver.cpp:191] Iteration 1400, loss = 0.000956274
    I1024 19:18:36.735342  8880 solver.cpp:206]     Train net output #0: loss = 0.000956274 (* 1 = 0.000956274 loss)
    I1024 19:18:36.735358  8880 solver.cpp:403] Iteration 1400, lr = 0.00906403
    I1024 19:18:40.244703  5792 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:20:02.114861  8880 solver.cpp:247] Iteration 1500, Testing net (#0)
    I1024 19:20:02.115010  8880 net.cpp:652] Copying source layer mnist
    I1024 19:20:02.115018  8880 net.cpp:652] Copying source layer conv1
    I1024 19:20:02.115026  8880 net.cpp:652] Copying source layer pool1
    I1024 19:20:02.115031  8880 net.cpp:652] Copying source layer conv2
    I1024 19:20:02.115036  8880 net.cpp:652] Copying source layer pool2
    I1024 19:20:02.115039  8880 net.cpp:652] Copying source layer ip1
    I1024 19:20:02.115044  8880 net.cpp:652] Copying source layer relu1
    I1024 19:20:02.115048  8880 net.cpp:652] Copying source layer ip2
    I1024 19:20:02.115053  8880 net.cpp:652] Copying source layer loss
    I1024 19:20:15.060420 30520 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:20:26.675933 13553 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:20:38.574820 28742 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:20:48.134063 11796 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:20:48.386265  8880 solver.cpp:298]     Test net output #0: accuracy = 0.992
    I1024 19:20:48.386315  8880 solver.cpp:298]     Test net output #1: loss = 0.0349491 (* 1 = 0.0349491 loss)
    I1024 19:20:48.761309  8880 solver.cpp:191] Iteration 1500, loss = 0.00276966
    I1024 19:20:48.761361  8880 solver.cpp:206]     Train net output #0: loss = 0.00276966 (* 1 = 0.00276966 loss)
    I1024 19:20:48.761379  8880 solver.cpp:403] Iteration 1500, lr = 0.00900485
    I1024 19:22:08.204447  8880 solver.cpp:191] Iteration 1600, loss = 0.0012758
    I1024 19:22:08.204836  8880 solver.cpp:206]     Train net output #0: loss = 0.0012758 (* 1 = 0.0012758 loss)
    I1024 19:22:08.204876  8880 solver.cpp:403] Iteration 1600, lr = 0.00894657
    I1024 19:22:41.299787 24861 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:23:35.490855  8880 solver.cpp:191] Iteration 1700, loss = 0.00179156
    I1024 19:23:35.491224  8880 solver.cpp:206]     Train net output #0: loss = 0.00179156 (* 1 = 0.00179156 loss)
    I1024 19:23:35.491266  8880 solver.cpp:403] Iteration 1700, lr = 0.00888916
    I1024 19:25:02.686167  8880 solver.cpp:191] Iteration 1800, loss = 0.00561689
    I1024 19:25:02.686539  8880 solver.cpp:206]     Train net output #0: loss = 0.00561689 (* 1 = 0.00561689 loss)
    I1024 19:25:02.686580  8880 solver.cpp:403] Iteration 1800, lr = 0.0088326
    I1024 19:26:06.898624 16053 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:26:30.315343  8880 solver.cpp:191] Iteration 1900, loss = 0.00411707
    I1024 19:26:30.315455  8880 solver.cpp:206]     Train net output #0: loss = 0.00411707 (* 1 = 0.00411707 loss)
    I1024 19:26:30.315490  8880 solver.cpp:403] Iteration 1900, lr = 0.00877687
    I1024 19:27:56.625723  8880 solver.cpp:247] Iteration 2000, Testing net (#0)
    I1024 19:27:56.625890  8880 net.cpp:652] Copying source layer mnist
    I1024 19:27:56.625902  8880 net.cpp:652] Copying source layer conv1
    I1024 19:27:56.625911  8880 net.cpp:652] Copying source layer pool1
    I1024 19:27:56.625917  8880 net.cpp:652] Copying source layer conv2
    I1024 19:27:56.625923  8880 net.cpp:652] Copying source layer pool2
    I1024 19:27:56.625929  8880 net.cpp:652] Copying source layer ip1
    I1024 19:27:56.625936  8880 net.cpp:652] Copying source layer relu1
    I1024 19:27:56.625941  8880 net.cpp:652] Copying source layer ip2
    I1024 19:27:56.625946  8880 net.cpp:652] Copying source layer loss
    I1024 19:28:11.301493   405 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:28:26.371517 15903 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:28:41.362715 31083 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:28:56.640513 14115 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:28:57.270167  8880 solver.cpp:298]     Test net output #0: accuracy = 0.9928
    I1024 19:28:57.270270  8880 solver.cpp:298]     Test net output #1: loss = 0.0345598 (* 1 = 0.0345598 loss)
    I1024 19:28:58.129128  8880 solver.cpp:191] Iteration 2000, loss = 0.000227292
    I1024 19:28:58.129200  8880 solver.cpp:206]     Train net output #0: loss = 0.000227292 (* 1 = 0.000227292 loss)
    I1024 19:28:58.129222  8880 solver.cpp:403] Iteration 2000, lr = 0.00872196
    I1024 19:30:24.602126  8880 solver.cpp:191] Iteration 2100, loss = 0.00228695
    I1024 19:30:24.602511  8880 solver.cpp:206]     Train net output #0: loss = 0.00228695 (* 1 = 0.00228695 loss)
    I1024 19:30:24.602551  8880 solver.cpp:403] Iteration 2100, lr = 0.00866784
    I1024 19:30:30.892768  2581 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:31:52.621273  8880 solver.cpp:191] Iteration 2200, loss = 0.000236811
    I1024 19:31:52.621464  8880 solver.cpp:206]     Train net output #0: loss = 0.000236811 (* 1 = 0.000236811 loss)
    I1024 19:31:52.621482  8880 solver.cpp:403] Iteration 2200, lr = 0.0086145
    I1024 19:33:20.768113  8880 solver.cpp:191] Iteration 2300, loss = 0.0029115
    I1024 19:33:20.768427  8880 solver.cpp:206]     Train net output #0: loss = 0.0029115 (* 1 = 0.0029115 loss)
    I1024 19:33:20.768465  8880 solver.cpp:403] Iteration 2300, lr = 0.00856192
    I1024 19:33:57.903209 26209 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:34:48.791607  8880 solver.cpp:191] Iteration 2400, loss = 0.00110135
    I1024 19:34:48.791949  8880 solver.cpp:206]     Train net output #0: loss = 0.00110135 (* 1 = 0.00110135 loss)
    I1024 19:34:48.791985  8880 solver.cpp:403] Iteration 2400, lr = 0.00851008
    I1024 19:36:15.738677  8880 solver.cpp:247] Iteration 2500, Testing net (#0)
    I1024 19:36:15.738989  8880 net.cpp:652] Copying source layer mnist
    I1024 19:36:15.739012  8880 net.cpp:652] Copying source layer conv1
    I1024 19:36:15.739030  8880 net.cpp:652] Copying source layer pool1
    I1024 19:36:15.739042  8880 net.cpp:652] Copying source layer conv2
    I1024 19:36:15.739056  8880 net.cpp:652] Copying source layer pool2
    I1024 19:36:15.739069  8880 net.cpp:652] Copying source layer ip1
    I1024 19:36:15.739084  8880 net.cpp:652] Copying source layer relu1
    I1024 19:36:15.739099  8880 net.cpp:652] Copying source layer ip2
    I1024 19:36:15.739111  8880 net.cpp:652] Copying source layer loss
    I1024 19:36:29.939960  2719 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:36:45.237923 18191 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:37:00.633205   913 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:37:16.152928 16402 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:37:16.772070  8880 solver.cpp:298]     Test net output #0: accuracy = 0.9932
    I1024 19:37:16.772116  8880 solver.cpp:298]     Test net output #1: loss = 0.032235 (* 1 = 0.032235 loss)
    I1024 19:37:17.661167  8880 solver.cpp:191] Iteration 2500, loss = 0.00021334
    I1024 19:37:17.661300  8880 solver.cpp:206]     Train net output #0: loss = 0.00021334 (* 1 = 0.00021334 loss)
    I1024 19:37:17.661353  8880 solver.cpp:403] Iteration 2500, lr = 0.00845897
    I1024 19:38:23.675403 13023 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:38:44.787943  8880 solver.cpp:191] Iteration 2600, loss = 0.00293141
    I1024 19:38:44.788058  8880 solver.cpp:206]     Train net output #0: loss = 0.00293141 (* 1 = 0.00293141 loss)
    I1024 19:38:44.788091  8880 solver.cpp:403] Iteration 2600, lr = 0.00840857
    I1024 19:40:12.257936  8880 solver.cpp:191] Iteration 2700, loss = 0.000839049
    I1024 19:40:12.258384  8880 solver.cpp:206]     Train net output #0: loss = 0.000839049 (* 1 = 0.000839049 loss)
    I1024 19:40:12.258421  8880 solver.cpp:403] Iteration 2700, lr = 0.00835886
    I1024 19:41:41.049329  8880 solver.cpp:191] Iteration 2800, loss = 0.00145158
    I1024 19:41:41.049716  8880 solver.cpp:206]     Train net output #0: loss = 0.00145158 (* 1 = 0.00145158 loss)
    I1024 19:41:41.049757  8880 solver.cpp:403] Iteration 2800, lr = 0.00830984
    I1024 19:41:50.818536  4036 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:43:08.735764  8880 solver.cpp:191] Iteration 2900, loss = 0.00495344
    I1024 19:43:08.735997  8880 solver.cpp:206]     Train net output #0: loss = 0.00495344 (* 1 = 0.00495344 loss)
    I1024 19:43:08.736026  8880 solver.cpp:403] Iteration 2900, lr = 0.00826148
    I1024 19:44:35.260455  8880 solver.cpp:247] Iteration 3000, Testing net (#0)
    I1024 19:44:35.260639  8880 net.cpp:652] Copying source layer mnist
    I1024 19:44:35.260649  8880 net.cpp:652] Copying source layer conv1
    I1024 19:44:35.260658  8880 net.cpp:652] Copying source layer pool1
    I1024 19:44:35.260663  8880 net.cpp:652] Copying source layer conv2
    I1024 19:44:35.260670  8880 net.cpp:652] Copying source layer pool2
    I1024 19:44:35.260675  8880 net.cpp:652] Copying source layer ip1
    I1024 19:44:35.260682  8880 net.cpp:652] Copying source layer relu1
    I1024 19:44:35.260687  8880 net.cpp:652] Copying source layer ip2
    I1024 19:44:35.260692  8880 net.cpp:652] Copying source layer loss
    I1024 19:44:49.904296  5283 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:45:05.721585 21085 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:45:20.901792  3956 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:45:36.096451  8880 solver.cpp:298]     Test net output #0: accuracy = 0.9924
    I1024 19:45:36.096932  8880 solver.cpp:298]     Test net output #1: loss = 0.0338531 (* 1 = 0.0338531 loss)
    I1024 19:45:36.974349  8880 solver.cpp:191] Iteration 3000, loss = 0.0001377
    I1024 19:45:36.974455  8880 solver.cpp:206]     Train net output #0: loss = 0.0001377 (* 1 = 0.0001377 loss)
    I1024 19:45:36.974488  8880 solver.cpp:403] Iteration 3000, lr = 0.00821377
    I1024 19:46:15.375208 23184 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:47:02.762243  8880 solver.cpp:191] Iteration 3100, loss = 0.0131257
    I1024 19:47:02.762588  8880 solver.cpp:206]     Train net output #0: loss = 0.0131257 (* 1 = 0.0131257 loss)
    I1024 19:47:02.762626  8880 solver.cpp:403] Iteration 3100, lr = 0.0081667
    I1024 19:48:30.846729  8880 solver.cpp:191] Iteration 3200, loss = 0.00270246
    I1024 19:48:30.847064  8880 solver.cpp:206]     Train net output #0: loss = 0.00270246 (* 1 = 0.00270246 loss)
    I1024 19:48:30.847082  8880 solver.cpp:403] Iteration 3200, lr = 0.00812025
    I1024 19:49:39.706120 13593 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:49:57.998941  8880 solver.cpp:191] Iteration 3300, loss = 0.00167747
    I1024 19:49:57.999017  8880 solver.cpp:206]     Train net output #0: loss = 0.00167747 (* 1 = 0.00167747 loss)
    I1024 19:49:57.999042  8880 solver.cpp:403] Iteration 3300, lr = 0.00807442
    I1024 19:51:26.290658  8880 solver.cpp:191] Iteration 3400, loss = 0.000213729
    I1024 19:51:26.291033  8880 solver.cpp:206]     Train net output #0: loss = 0.000213729 (* 1 = 0.000213729 loss)
    I1024 19:51:26.291071  8880 solver.cpp:403] Iteration 3400, lr = 0.00802918
    I1024 19:52:53.072655  8880 solver.cpp:247] Iteration 3500, Testing net (#0)
    I1024 19:52:53.072998  8880 net.cpp:652] Copying source layer mnist
    I1024 19:52:53.073024  8880 net.cpp:652] Copying source layer conv1
    I1024 19:52:53.073043  8880 net.cpp:652] Copying source layer pool1
    I1024 19:52:53.073056  8880 net.cpp:652] Copying source layer conv2
    I1024 19:52:53.073071  8880 net.cpp:652] Copying source layer pool2
    I1024 19:52:53.073083  8880 net.cpp:652] Copying source layer ip1
    I1024 19:52:53.073097  8880 net.cpp:652] Copying source layer relu1
    I1024 19:52:53.073109  8880 net.cpp:652] Copying source layer ip2
    I1024 19:52:53.073123  8880 net.cpp:652] Copying source layer loss
    I1024 19:52:53.074592 25165 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:53:08.321238  8192 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:53:23.415853 23376 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:53:38.707955  6397 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:53:53.901252  8880 solver.cpp:298]     Test net output #0: accuracy = 0.9928
    I1024 19:53:53.901453  8880 solver.cpp:298]     Test net output #1: loss = 0.0328186 (* 1 = 0.0328186 loss)
    I1024 19:53:54.711185  8880 solver.cpp:191] Iteration 3500, loss = 0.00258751
    I1024 19:53:54.711282  8880 solver.cpp:206]     Train net output #0: loss = 0.00258751 (* 1 = 0.00258751 loss)
    I1024 19:53:54.711311  8880 solver.cpp:403] Iteration 3500, lr = 0.00798454
    I1024 19:54:06.792733   878 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:55:21.567296  8880 solver.cpp:191] Iteration 3600, loss = 0.00160017
    I1024 19:55:21.567479  8880 solver.cpp:206]     Train net output #0: loss = 0.00160017 (* 1 = 0.00160017 loss)
    I1024 19:55:21.567494  8880 solver.cpp:403] Iteration 3600, lr = 0.00794046
    I1024 19:56:48.944607  8880 solver.cpp:191] Iteration 3700, loss = 0.00166293
    I1024 19:56:48.945021  8880 solver.cpp:206]     Train net output #0: loss = 0.00166293 (* 1 = 0.00166293 loss)
    I1024 19:56:48.945065  8880 solver.cpp:403] Iteration 3700, lr = 0.00789695
    I1024 19:57:30.546828 23726 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 19:58:15.818794  8880 solver.cpp:191] Iteration 3800, loss = 0.000883576
    I1024 19:58:15.818979  8880 solver.cpp:206]     Train net output #0: loss = 0.000883576 (* 1 = 0.000883576 loss)
    I1024 19:58:15.819000  8880 solver.cpp:403] Iteration 3800, lr = 0.007854
    I1024 19:59:43.738668  8880 solver.cpp:191] Iteration 3900, loss = 5.33517e-05
    I1024 19:59:43.738950  8880 solver.cpp:206]     Train net output #0: loss = 5.33517e-05 (* 1 = 5.33517e-05 loss)
    I1024 19:59:43.738981  8880 solver.cpp:403] Iteration 3900, lr = 0.00781158
    I1024 20:00:54.301791 14915 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:01:08.069159  8880 solver.cpp:247] Iteration 4000, Testing net (#0)
    I1024 20:01:08.069195  8880 net.cpp:652] Copying source layer mnist
    I1024 20:01:08.069202  8880 net.cpp:652] Copying source layer conv1
    I1024 20:01:08.069208  8880 net.cpp:652] Copying source layer pool1
    I1024 20:01:08.069213  8880 net.cpp:652] Copying source layer conv2
    I1024 20:01:08.069218  8880 net.cpp:652] Copying source layer pool2
    I1024 20:01:08.069223  8880 net.cpp:652] Copying source layer ip1
    I1024 20:01:08.069228  8880 net.cpp:652] Copying source layer relu1
    I1024 20:01:08.069232  8880 net.cpp:652] Copying source layer ip2
    I1024 20:01:08.069237  8880 net.cpp:652] Copying source layer loss
    I1024 20:01:08.070304 27449 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:01:23.230705 10480 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:01:38.242414 25661 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:01:53.431851  8687 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:02:08.430405  8880 solver.cpp:298]     Test net output #0: accuracy = 0.992
    I1024 20:02:08.430568  8880 solver.cpp:298]     Test net output #1: loss = 0.0345766 (* 1 = 0.0345766 loss)
    I1024 20:02:09.273912  8880 solver.cpp:191] Iteration 4000, loss = 0.00343922
    I1024 20:02:09.273985  8880 solver.cpp:206]     Train net output #0: loss = 0.00343922 (* 1 = 0.00343922 loss)
    I1024 20:02:09.273998  8880 solver.cpp:403] Iteration 4000, lr = 0.0077697
    I1024 20:03:36.968215  8880 solver.cpp:191] Iteration 4100, loss = 0.000286716
    I1024 20:03:36.968533  8880 solver.cpp:206]     Train net output #0: loss = 0.000286716 (* 1 = 0.000286716 loss)
    I1024 20:03:36.968581  8880 solver.cpp:403] Iteration 4100, lr = 0.00772833
    I1024 20:05:03.379369  8880 solver.cpp:191] Iteration 4200, loss = 0.00210485
    I1024 20:05:03.379673  8880 solver.cpp:206]     Train net output #0: loss = 0.00210485 (* 1 = 0.00210485 loss)
    I1024 20:05:03.379715  8880 solver.cpp:403] Iteration 4200, lr = 0.00768748
    I1024 20:05:18.191162  1434 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:06:30.662565  8880 solver.cpp:191] Iteration 4300, loss = 0.00225787
    I1024 20:06:30.662725  8880 solver.cpp:206]     Train net output #0: loss = 0.00225787 (* 1 = 0.00225787 loss)
    I1024 20:06:30.662740  8880 solver.cpp:403] Iteration 4300, lr = 0.00764712
    I1024 20:07:56.951802  8880 solver.cpp:191] Iteration 4400, loss = 0.000458731
    I1024 20:07:56.952116  8880 solver.cpp:206]     Train net output #0: loss = 0.000458731 (* 1 = 0.000458731 loss)
    I1024 20:07:56.952155  8880 solver.cpp:403] Iteration 4400, lr = 0.00760726
    I1024 20:08:41.307680 25077 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:09:22.018496  8880 solver.cpp:247] Iteration 4500, Testing net (#0)
    I1024 20:09:22.018858  8880 net.cpp:652] Copying source layer mnist
    I1024 20:09:22.018885  8880 net.cpp:652] Copying source layer conv1
    I1024 20:09:22.018904  8880 net.cpp:652] Copying source layer pool1
    I1024 20:09:22.018916  8880 net.cpp:652] Copying source layer conv2
    I1024 20:09:22.018930  8880 net.cpp:652] Copying source layer pool2
    I1024 20:09:22.018951  8880 net.cpp:652] Copying source layer ip1
    I1024 20:09:22.018957  8880 net.cpp:652] Copying source layer relu1
    I1024 20:09:22.018961  8880 net.cpp:652] Copying source layer ip2
    I1024 20:09:22.018966  8880 net.cpp:652] Copying source layer loss
    I1024 20:09:22.022372 29743 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:09:37.140197 12774 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:09:52.284610 27955 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:10:07.570957 10985 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:10:22.602891  8880 solver.cpp:298]     Test net output #0: accuracy = 0.9936
    I1024 20:10:22.603268  8880 solver.cpp:298]     Test net output #1: loss = 0.030733 (* 1 = 0.030733 loss)
    I1024 20:10:23.461911  8880 solver.cpp:191] Iteration 4500, loss = 0.000399474
    I1024 20:10:23.461979  8880 solver.cpp:206]     Train net output #0: loss = 0.000399474 (* 1 = 0.000399474 loss)
    I1024 20:10:23.462012  8880 solver.cpp:403] Iteration 4500, lr = 0.00756788
    I1024 20:11:50.614876  8880 solver.cpp:191] Iteration 4600, loss = 0.00146791
    I1024 20:11:50.615236  8880 solver.cpp:206]     Train net output #0: loss = 0.00146791 (* 1 = 0.00146791 loss)
    I1024 20:11:50.615272  8880 solver.cpp:403] Iteration 4600, lr = 0.00752897
    I1024 20:13:06.659422 11901 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:13:19.066622  8880 solver.cpp:191] Iteration 4700, loss = 0.00284335
    I1024 20:13:19.066730  8880 solver.cpp:206]     Train net output #0: loss = 0.00284335 (* 1 = 0.00284335 loss)
    I1024 20:13:19.066763  8880 solver.cpp:403] Iteration 4700, lr = 0.00749052
    I1024 20:14:46.447500  8880 solver.cpp:191] Iteration 4800, loss = 0.00107749
    I1024 20:14:46.447813  8880 solver.cpp:206]     Train net output #0: loss = 0.00107749 (* 1 = 0.00107749 loss)
    I1024 20:14:46.447860  8880 solver.cpp:403] Iteration 4800, lr = 0.00745253
    I1024 20:16:14.396100  8880 solver.cpp:191] Iteration 4900, loss = 0.00664793
    I1024 20:16:14.396261  8880 solver.cpp:206]     Train net output #0: loss = 0.00664793 (* 1 = 0.00664793 loss)
    I1024 20:16:14.396276  8880 solver.cpp:403] Iteration 4900, lr = 0.00741498
    I1024 20:16:31.825505  1998 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:17:38.000325  8880 net.cpp:735] Serializing 9 layers
    I1024 20:17:38.011478  8880 solver.cpp:317] Snapshotting to /home/axel/Desktop/MNIST/caffenet_iter_5000.caffemodel
    I1024 20:17:38.031105  8880 solver.cpp:324] Snapshotting solver state to /home/axel/Desktop/MNIST/caffenet_iter_5000.solverstate
    I1024 20:17:38.044090  8880 solver.cpp:247] Iteration 5000, Testing net (#0)
    I1024 20:17:38.044123  8880 net.cpp:652] Copying source layer mnist
    I1024 20:17:38.044131  8880 net.cpp:652] Copying source layer conv1
    I1024 20:17:38.044138  8880 net.cpp:652] Copying source layer pool1
    I1024 20:17:38.044142  8880 net.cpp:652] Copying source layer conv2
    I1024 20:17:38.044147  8880 net.cpp:652] Copying source layer pool2
    I1024 20:17:38.044152  8880 net.cpp:652] Copying source layer ip1
    I1024 20:17:38.044157  8880 net.cpp:652] Copying source layer relu1
    I1024 20:17:38.044160  8880 net.cpp:652] Copying source layer ip2
    I1024 20:17:38.044165  8880 net.cpp:652] Copying source layer loss
    I1024 20:17:38.046298 32041 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:17:52.991899 15091 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:18:08.217923 30274 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:18:23.362880 13304 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:18:38.419692  8880 solver.cpp:298]     Test net output #0: accuracy = 0.9944
    I1024 20:18:38.419978  8880 solver.cpp:298]     Test net output #1: loss = 0.0303235 (* 1 = 0.0303235 loss)
    I1024 20:18:39.305835  8880 solver.cpp:191] Iteration 5000, loss = 0.000456518
    I1024 20:18:39.305943  8880 solver.cpp:206]     Train net output #0: loss = 0.000456518 (* 1 = 0.000456518 loss)
    I1024 20:18:39.305976  8880 solver.cpp:403] Iteration 5000, lr = 0.00737788
    I1024 20:20:05.327433  8880 solver.cpp:191] Iteration 5100, loss = 0.000582855
    I1024 20:20:05.327803  8880 solver.cpp:206]     Train net output #0: loss = 0.000582855 (* 1 = 0.000582855 loss)
    I1024 20:20:05.327844  8880 solver.cpp:403] Iteration 5100, lr = 0.0073412
    I1024 20:20:53.801770 22093 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:21:30.790508  8880 solver.cpp:191] Iteration 5200, loss = 0.000668405
    I1024 20:21:30.790834  8880 solver.cpp:206]     Train net output #0: loss = 0.000668405 (* 1 = 0.000668405 loss)
    I1024 20:21:30.790871  8880 solver.cpp:403] Iteration 5200, lr = 0.00730495
    I1024 20:22:56.470559  8880 solver.cpp:191] Iteration 5300, loss = 0.000517944
    I1024 20:22:56.470937  8880 solver.cpp:206]     Train net output #0: loss = 0.000517944 (* 1 = 0.000517944 loss)
    I1024 20:22:56.470976  8880 solver.cpp:403] Iteration 5300, lr = 0.00726911
    I1024 20:24:14.130828 12507 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:24:23.947912  8880 solver.cpp:191] Iteration 5400, loss = 0.00353877
    I1024 20:24:23.948026  8880 solver.cpp:206]     Train net output #0: loss = 0.00353877 (* 1 = 0.00353877 loss)
    I1024 20:24:23.948060  8880 solver.cpp:403] Iteration 5400, lr = 0.00723368
    I1024 20:25:50.566993  8880 solver.cpp:247] Iteration 5500, Testing net (#0)
    I1024 20:25:50.567355  8880 net.cpp:652] Copying source layer mnist
    I1024 20:25:50.567383  8880 net.cpp:652] Copying source layer conv1
    I1024 20:25:50.567401  8880 net.cpp:652] Copying source layer pool1
    I1024 20:25:50.567414  8880 net.cpp:652] Copying source layer conv2
    I1024 20:25:50.567427  8880 net.cpp:652] Copying source layer pool2
    I1024 20:25:50.567440  8880 net.cpp:652] Copying source layer ip1
    I1024 20:25:50.567453  8880 net.cpp:652] Copying source layer relu1
    I1024 20:25:50.567464  8880 net.cpp:652] Copying source layer ip2
    I1024 20:25:50.567477  8880 net.cpp:652] Copying source layer loss
    I1024 20:25:50.574589  1918 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:26:05.755672 17406 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:26:20.998002 32589 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:26:35.778409 15621 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:26:50.779052  8880 solver.cpp:298]     Test net output #0: accuracy = 0.994
    I1024 20:26:50.779157  8880 solver.cpp:298]     Test net output #1: loss = 0.0306006 (* 1 = 0.0306006 loss)
    I1024 20:26:51.668087  8880 solver.cpp:191] Iteration 5500, loss = 0.00160408
    I1024 20:26:51.668469  8880 solver.cpp:206]     Train net output #0: loss = 0.00160408 (* 1 = 0.00160408 loss)
    I1024 20:26:51.668509  8880 solver.cpp:403] Iteration 5500, lr = 0.00719865
    I1024 20:28:18.928897  8880 solver.cpp:191] Iteration 5600, loss = 0.000695632
    I1024 20:28:18.929288  8880 solver.cpp:206]     Train net output #0: loss = 0.000695632 (* 1 = 0.000695632 loss)
    I1024 20:28:18.929329  8880 solver.cpp:403] Iteration 5600, lr = 0.00716402
    I1024 20:28:39.515981 32271 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:29:45.127933  8880 solver.cpp:191] Iteration 5700, loss = 0.00213943
    I1024 20:29:45.128232  8880 solver.cpp:206]     Train net output #0: loss = 0.00213943 (* 1 = 0.00213943 loss)
    I1024 20:29:45.128269  8880 solver.cpp:403] Iteration 5700, lr = 0.00712977
    I1024 20:31:11.312753  8880 solver.cpp:191] Iteration 5800, loss = 0.000102586
    I1024 20:31:11.313128  8880 solver.cpp:206]     Train net output #0: loss = 0.000102586 (* 1 = 0.000102586 loss)
    I1024 20:31:11.313166  8880 solver.cpp:403] Iteration 5800, lr = 0.0070959
    I1024 20:32:01.893378 22668 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:32:38.408548  8880 solver.cpp:191] Iteration 5900, loss = 0.000361301
    I1024 20:32:38.408905  8880 solver.cpp:206]     Train net output #0: loss = 0.000361301 (* 1 = 0.000361301 loss)
    I1024 20:32:38.408946  8880 solver.cpp:403] Iteration 5900, lr = 0.0070624
    I1024 20:34:05.064677  8880 solver.cpp:247] Iteration 6000, Testing net (#0)
    I1024 20:34:05.065003  8880 net.cpp:652] Copying source layer mnist
    I1024 20:34:05.065032  8880 net.cpp:652] Copying source layer conv1
    I1024 20:34:05.065052  8880 net.cpp:652] Copying source layer pool1
    I1024 20:34:05.065063  8880 net.cpp:652] Copying source layer conv2
    I1024 20:34:05.065078  8880 net.cpp:652] Copying source layer pool2
    I1024 20:34:05.065090  8880 net.cpp:652] Copying source layer ip1
    I1024 20:34:05.065105  8880 net.cpp:652] Copying source layer relu1
    I1024 20:34:05.065117  8880 net.cpp:652] Copying source layer ip2
    I1024 20:34:05.065130  8880 net.cpp:652] Copying source layer loss
    I1024 20:34:05.073400  4441 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:34:20.268537 19697 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:34:35.424456  2432 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:34:51.202038 18517 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:35:05.615278  8880 solver.cpp:298]     Test net output #0: accuracy = 0.9948
    I1024 20:35:05.615437  8880 solver.cpp:298]     Test net output #1: loss = 0.0307473 (* 1 = 0.0307473 loss)
    I1024 20:35:06.510759  8880 solver.cpp:191] Iteration 6000, loss = 0.000707565
    I1024 20:35:06.510869  8880 solver.cpp:206]     Train net output #0: loss = 0.000707565 (* 1 = 0.000707565 loss)
    I1024 20:35:06.510908  8880 solver.cpp:403] Iteration 6000, lr = 0.00702927
    I1024 20:36:27.061216  9490 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:36:34.043615  8880 solver.cpp:191] Iteration 6100, loss = 0.0015943
    I1024 20:36:34.043730  8880 solver.cpp:206]     Train net output #0: loss = 0.0015943 (* 1 = 0.0015943 loss)
    I1024 20:36:34.043764  8880 solver.cpp:403] Iteration 6100, lr = 0.0069965
    I1024 20:38:01.881587  8880 solver.cpp:191] Iteration 6200, loss = 0.000185658
    I1024 20:38:01.881925  8880 solver.cpp:206]     Train net output #0: loss = 0.000185658 (* 1 = 0.000185658 loss)
    I1024 20:38:01.881963  8880 solver.cpp:403] Iteration 6200, lr = 0.00696408
    I1024 20:39:27.729962  8880 solver.cpp:191] Iteration 6300, loss = 0.000364319
    I1024 20:39:27.730131  8880 solver.cpp:206]     Train net output #0: loss = 0.000364319 (* 1 = 0.000364319 loss)
    I1024 20:39:27.730147  8880 solver.cpp:403] Iteration 6300, lr = 0.00693201
    I1024 20:39:51.097236   354 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:40:55.201653  8880 solver.cpp:191] Iteration 6400, loss = 0.000616102
    I1024 20:40:55.201788  8880 solver.cpp:206]     Train net output #0: loss = 0.000616102 (* 1 = 0.000616102 loss)
    I1024 20:40:55.201807  8880 solver.cpp:403] Iteration 6400, lr = 0.00690029
    I1024 20:42:21.374768  8880 solver.cpp:247] Iteration 6500, Testing net (#0)
    I1024 20:42:21.375146  8880 net.cpp:652] Copying source layer mnist
    I1024 20:42:21.375169  8880 net.cpp:652] Copying source layer conv1
    I1024 20:42:21.375187  8880 net.cpp:652] Copying source layer pool1
    I1024 20:42:21.375200  8880 net.cpp:652] Copying source layer conv2
    I1024 20:42:21.375213  8880 net.cpp:652] Copying source layer pool2
    I1024 20:42:21.375226  8880 net.cpp:652] Copying source layer ip1
    I1024 20:42:21.375241  8880 net.cpp:652] Copying source layer relu1
    I1024 20:42:21.375262  8880 net.cpp:652] Copying source layer ip2
    I1024 20:42:21.375277  8880 net.cpp:652] Copying source layer loss
    I1024 20:42:22.029469  7401 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:42:37.361414 22588 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:42:52.450582  5602 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:43:07.618341 20800 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:43:22.173810  8880 solver.cpp:298]     Test net output #0: accuracy = 0.9944
    I1024 20:43:22.173913  8880 solver.cpp:298]     Test net output #1: loss = 0.0290573 (* 1 = 0.0290573 loss)
    I1024 20:43:22.974530  8880 solver.cpp:191] Iteration 6500, loss = 0.000337536
    I1024 20:43:22.974658  8880 solver.cpp:206]     Train net output #0: loss = 0.000337536 (* 1 = 0.000337536 loss)
    I1024 20:43:22.974673  8880 solver.cpp:403] Iteration 6500, lr = 0.0068689
    I1024 20:44:16.022481 19643 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:44:50.160708  8880 solver.cpp:191] Iteration 6600, loss = 0.00099608
    I1024 20:44:50.161087  8880 solver.cpp:206]     Train net output #0: loss = 0.00099608 (* 1 = 0.00099608 loss)
    I1024 20:44:50.161125  8880 solver.cpp:403] Iteration 6600, lr = 0.00683784
    I1024 20:46:17.253139  8880 solver.cpp:191] Iteration 6700, loss = 0.00122824
    I1024 20:46:17.253936  8880 solver.cpp:206]     Train net output #0: loss = 0.00122824 (* 1 = 0.00122824 loss)
    I1024 20:46:17.253986  8880 solver.cpp:403] Iteration 6700, lr = 0.00680711
    I1024 20:47:38.394434 10824 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:47:41.839416  8880 solver.cpp:191] Iteration 6800, loss = 0.00108031
    I1024 20:47:41.839527  8880 solver.cpp:206]     Train net output #0: loss = 0.00108031 (* 1 = 0.00108031 loss)
    I1024 20:47:41.839560  8880 solver.cpp:403] Iteration 6800, lr = 0.0067767
    I1024 20:49:08.632081  8880 solver.cpp:191] Iteration 6900, loss = 0.000442285
    I1024 20:49:08.632338  8880 solver.cpp:206]     Train net output #0: loss = 0.000442285 (* 1 = 0.000442285 loss)
    I1024 20:49:08.632390  8880 solver.cpp:403] Iteration 6900, lr = 0.0067466
    I1024 20:50:34.369104  8880 solver.cpp:247] Iteration 7000, Testing net (#0)
    I1024 20:50:34.369371  8880 net.cpp:652] Copying source layer mnist
    I1024 20:50:34.369401  8880 net.cpp:652] Copying source layer conv1
    I1024 20:50:34.369428  8880 net.cpp:652] Copying source layer pool1
    I1024 20:50:34.369451  8880 net.cpp:652] Copying source layer conv2
    I1024 20:50:34.369475  8880 net.cpp:652] Copying source layer pool2
    I1024 20:50:34.369508  8880 net.cpp:652] Copying source layer ip1
    I1024 20:50:34.369534  8880 net.cpp:652] Copying source layer relu1
    I1024 20:50:34.369561  8880 net.cpp:652] Copying source layer ip2
    I1024 20:50:34.369587  8880 net.cpp:652] Copying source layer loss
    I1024 20:50:34.972156  9694 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:50:50.037907 24875 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:51:05.023560  7901 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:51:20.026175 23087 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:51:34.796416  8880 solver.cpp:298]     Test net output #0: accuracy = 0.9944
    I1024 20:51:34.796553  8880 solver.cpp:298]     Test net output #1: loss = 0.030245 (* 1 = 0.030245 loss)
    I1024 20:51:35.677703  8880 solver.cpp:191] Iteration 7000, loss = 0.000613304
    I1024 20:51:35.678138  8880 solver.cpp:206]     Train net output #0: loss = 0.000613304 (* 1 = 0.000613304 loss)
    I1024 20:51:35.678176  8880 solver.cpp:403] Iteration 7000, lr = 0.00671681
    I1024 20:52:02.195797 29794 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:53:03.114934  8880 solver.cpp:191] Iteration 7100, loss = 0.00039933
    I1024 20:53:03.115267  8880 solver.cpp:206]     Train net output #0: loss = 0.00039933 (* 1 = 0.00039933 loss)
    I1024 20:53:03.115281  8880 solver.cpp:403] Iteration 7100, lr = 0.00668733
    I1024 20:54:30.873641  8880 solver.cpp:191] Iteration 7200, loss = 0.000235122
    I1024 20:54:30.876655  8880 solver.cpp:206]     Train net output #0: loss = 0.000235122 (* 1 = 0.000235122 loss)
    I1024 20:54:30.876695  8880 solver.cpp:403] Iteration 7200, lr = 0.00665815
    I1024 20:55:26.570931 20974 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:55:56.655638  8880 solver.cpp:191] Iteration 7300, loss = 0.000204699
    I1024 20:55:56.655971  8880 solver.cpp:206]     Train net output #0: loss = 0.000204699 (* 1 = 0.000204699 loss)
    I1024 20:55:56.656008  8880 solver.cpp:403] Iteration 7300, lr = 0.00662927
    I1024 20:57:23.773316  8880 solver.cpp:191] Iteration 7400, loss = 0.000132924
    I1024 20:57:23.773463  8880 solver.cpp:206]     Train net output #0: loss = 0.000132924 (* 1 = 0.000132924 loss)
    I1024 20:57:23.773483  8880 solver.cpp:403] Iteration 7400, lr = 0.00660067
    I1024 20:58:49.388118  8880 solver.cpp:247] Iteration 7500, Testing net (#0)
    I1024 20:58:49.388425  8880 net.cpp:652] Copying source layer mnist
    I1024 20:58:49.388450  8880 net.cpp:652] Copying source layer conv1
    I1024 20:58:49.388469  8880 net.cpp:652] Copying source layer pool1
    I1024 20:58:49.388483  8880 net.cpp:652] Copying source layer conv2
    I1024 20:58:49.388496  8880 net.cpp:652] Copying source layer pool2
    I1024 20:58:49.388509  8880 net.cpp:652] Copying source layer ip1
    I1024 20:58:49.388556  8880 net.cpp:652] Copying source layer relu1
    I1024 20:58:49.388571  8880 net.cpp:652] Copying source layer ip2
    I1024 20:58:49.388586  8880 net.cpp:652] Copying source layer loss
    I1024 20:58:50.012787 11985 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:59:05.075253 27167 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:59:20.084126 10197 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:59:35.202414 25379 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:59:49.675127  8880 solver.cpp:298]     Test net output #0: accuracy = 0.994
    I1024 20:59:49.675235  8880 solver.cpp:298]     Test net output #1: loss = 0.0297335 (* 1 = 0.0297335 loss)
    I1024 20:59:49.676632  7795 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 20:59:50.523461  8880 solver.cpp:191] Iteration 7500, loss = 0.000633195
    I1024 20:59:50.523746  8880 solver.cpp:206]     Train net output #0: loss = 0.000633195 (* 1 = 0.000633195 loss)
    I1024 20:59:50.523777  8880 solver.cpp:403] Iteration 7500, lr = 0.00657236
    I1024 21:01:16.654639  8880 solver.cpp:191] Iteration 7600, loss = 0.00298195
    I1024 21:01:16.654964  8880 solver.cpp:206]     Train net output #0: loss = 0.00298195 (* 1 = 0.00298195 loss)
    I1024 21:01:16.655004  8880 solver.cpp:403] Iteration 7600, lr = 0.00654433
    I1024 21:02:44.482903  8880 solver.cpp:191] Iteration 7700, loss = 0.000314093
    I1024 21:02:44.483319  8880 solver.cpp:206]     Train net output #0: loss = 0.000314093 (* 1 = 0.000314093 loss)
    I1024 21:02:44.483366  8880 solver.cpp:403] Iteration 7700, lr = 0.00651658
    I1024 21:03:12.750373 30347 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:04:10.540169  8880 solver.cpp:191] Iteration 7800, loss = 0.00462593
    I1024 21:04:10.540485  8880 solver.cpp:206]     Train net output #0: loss = 0.00462593 (* 1 = 0.00462593 loss)
    I1024 21:04:10.540561  8880 solver.cpp:403] Iteration 7800, lr = 0.00648911
    I1024 21:05:36.740422  8880 solver.cpp:191] Iteration 7900, loss = 0.00260821
    I1024 21:05:36.740815  8880 solver.cpp:206]     Train net output #0: loss = 0.00260821 (* 1 = 0.00260821 loss)
    I1024 21:05:36.740855  8880 solver.cpp:403] Iteration 7900, lr = 0.0064619
    I1024 21:06:36.191633 21526 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:07:03.051265  8880 solver.cpp:247] Iteration 8000, Testing net (#0)
    I1024 21:07:03.051301  8880 net.cpp:652] Copying source layer mnist
    I1024 21:07:03.051311  8880 net.cpp:652] Copying source layer conv1
    I1024 21:07:03.051323  8880 net.cpp:652] Copying source layer pool1
    I1024 21:07:03.051331  8880 net.cpp:652] Copying source layer conv2
    I1024 21:07:03.051338  8880 net.cpp:652] Copying source layer pool2
    I1024 21:07:03.051347  8880 net.cpp:652] Copying source layer ip1
    I1024 21:07:03.051358  8880 net.cpp:652] Copying source layer relu1
    I1024 21:07:03.051367  8880 net.cpp:652] Copying source layer ip2
    I1024 21:07:03.051376  8880 net.cpp:652] Copying source layer loss
    I1024 21:07:03.670346 14267 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:07:18.778612 29448 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:07:33.743109 12480 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:07:48.635084 27660 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:08:03.239317  8880 solver.cpp:298]     Test net output #0: accuracy = 0.9936
    I1024 21:08:03.240396  8880 solver.cpp:298]     Test net output #1: loss = 0.0307348 (* 1 = 0.0307348 loss)
    I1024 21:08:04.092064  8880 solver.cpp:191] Iteration 8000, loss = 0.00214865
    I1024 21:08:04.092170  8880 solver.cpp:206]     Train net output #0: loss = 0.00214865 (* 1 = 0.00214865 loss)
    I1024 21:08:04.092202  8880 solver.cpp:403] Iteration 8000, lr = 0.00643496
    I1024 21:09:30.587731  8880 solver.cpp:191] Iteration 8100, loss = 0.000799927
    I1024 21:09:30.587901  8880 solver.cpp:206]     Train net output #0: loss = 0.000799927 (* 1 = 0.000799927 loss)
    I1024 21:09:30.587916  8880 solver.cpp:403] Iteration 8100, lr = 0.00640827
    I1024 21:10:57.810505  8880 solver.cpp:191] Iteration 8200, loss = 0.0010466
    I1024 21:10:57.810883  8880 solver.cpp:206]     Train net output #0: loss = 0.0010466 (* 1 = 0.0010466 loss)
    I1024 21:10:57.810924  8880 solver.cpp:403] Iteration 8200, lr = 0.00638185
    I1024 21:10:59.506341  8341 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:12:24.059380  8880 solver.cpp:191] Iteration 8300, loss = 0.000172352
    I1024 21:12:24.059684  8880 solver.cpp:206]     Train net output #0: loss = 0.000172352 (* 1 = 0.000172352 loss)
    I1024 21:12:24.059720  8880 solver.cpp:403] Iteration 8300, lr = 0.00635567
    I1024 21:13:50.130314  8880 solver.cpp:191] Iteration 8400, loss = 0.000218302
    I1024 21:13:50.130928  8880 solver.cpp:206]     Train net output #0: loss = 0.000218302 (* 1 = 0.000218302 loss)
    I1024 21:13:50.130992  8880 solver.cpp:403] Iteration 8400, lr = 0.00632975
    I1024 21:14:21.981431 31679 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:15:15.466727  8880 solver.cpp:247] Iteration 8500, Testing net (#0)
    I1024 21:15:15.467058  8880 net.cpp:652] Copying source layer mnist
    I1024 21:15:15.467089  8880 net.cpp:652] Copying source layer conv1
    I1024 21:15:15.467118  8880 net.cpp:652] Copying source layer pool1
    I1024 21:15:15.467138  8880 net.cpp:652] Copying source layer conv2
    I1024 21:15:15.467164  8880 net.cpp:652] Copying source layer pool2
    I1024 21:15:15.467191  8880 net.cpp:652] Copying source layer ip1
    I1024 21:15:15.467217  8880 net.cpp:652] Copying source layer relu1
    I1024 21:15:15.467247  8880 net.cpp:652] Copying source layer ip2
    I1024 21:15:15.467272  8880 net.cpp:652] Copying source layer loss
    I1024 21:15:16.108947 16552 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:15:30.903587 31735 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:15:45.878342 14766 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:16:00.616873 29948 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:16:14.922652  8880 solver.cpp:298]     Test net output #0: accuracy = 0.994
    I1024 21:16:14.922708  8880 solver.cpp:298]     Test net output #1: loss = 0.0284821 (* 1 = 0.0284821 loss)
    I1024 21:16:15.779357  8880 solver.cpp:191] Iteration 8500, loss = 0.000207761
    I1024 21:16:15.779445  8880 solver.cpp:206]     Train net output #0: loss = 0.000207761 (* 1 = 0.000207761 loss)
    I1024 21:16:15.779458  8880 solver.cpp:403] Iteration 8500, lr = 0.00630407
    I1024 21:17:41.273419  8880 solver.cpp:191] Iteration 8600, loss = 0.0013156
    I1024 21:17:41.273681  8880 solver.cpp:206]     Train net output #0: loss = 0.0013156 (* 1 = 0.0013156 loss)
    I1024 21:17:41.273697  8880 solver.cpp:403] Iteration 8600, lr = 0.00627864
    I1024 21:18:42.118592 18511 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:19:06.747373  8880 solver.cpp:191] Iteration 8700, loss = 0.00109621
    I1024 21:19:06.747484  8880 solver.cpp:206]     Train net output #0: loss = 0.00109621 (* 1 = 0.00109621 loss)
    I1024 21:19:06.747516  8880 solver.cpp:403] Iteration 8700, lr = 0.00625344
    I1024 21:20:32.686038  8880 solver.cpp:191] Iteration 8800, loss = 0.000600506
    I1024 21:20:32.686458  8880 solver.cpp:206]     Train net output #0: loss = 0.000600506 (* 1 = 0.000600506 loss)
    I1024 21:20:32.686501  8880 solver.cpp:403] Iteration 8800, lr = 0.00622847
    I1024 21:21:55.656193  8880 solver.cpp:191] Iteration 8900, loss = 0.00108805
    I1024 21:21:55.656348  8880 solver.cpp:206]     Train net output #0: loss = 0.00108805 (* 1 = 0.00108805 loss)
    I1024 21:21:55.656363  8880 solver.cpp:403] Iteration 8900, lr = 0.00620374
    I1024 21:22:00.823123  9709 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:23:20.532986  8880 solver.cpp:247] Iteration 9000, Testing net (#0)
    I1024 21:23:20.533123  8880 net.cpp:652] Copying source layer mnist
    I1024 21:23:20.533133  8880 net.cpp:652] Copying source layer conv1
    I1024 21:23:20.533143  8880 net.cpp:652] Copying source layer pool1
    I1024 21:23:20.533150  8880 net.cpp:652] Copying source layer conv2
    I1024 21:23:20.533159  8880 net.cpp:652] Copying source layer pool2
    I1024 21:23:20.533165  8880 net.cpp:652] Copying source layer ip1
    I1024 21:23:20.533175  8880 net.cpp:652] Copying source layer relu1
    I1024 21:23:20.533185  8880 net.cpp:652] Copying source layer ip2
    I1024 21:23:20.533195  8880 net.cpp:652] Copying source layer loss
    I1024 21:23:21.128978 18871 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:23:35.935672  1594 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:23:50.722964 17085 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:24:05.542022 32268 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:24:19.549345  8880 solver.cpp:298]     Test net output #0: accuracy = 0.9948
    I1024 21:24:19.549401  8880 solver.cpp:298]     Test net output #1: loss = 0.0287801 (* 1 = 0.0287801 loss)
    I1024 21:24:20.434121  8880 solver.cpp:191] Iteration 9000, loss = 0.000982145
    I1024 21:24:20.434239  8880 solver.cpp:206]     Train net output #0: loss = 0.000982145 (* 1 = 0.000982145 loss)
    I1024 21:24:20.434288  8880 solver.cpp:403] Iteration 9000, lr = 0.00617924
    I1024 21:25:46.080946  8880 solver.cpp:191] Iteration 9100, loss = 0.00016309
    I1024 21:25:46.081089  8880 solver.cpp:206]     Train net output #0: loss = 0.00016309 (* 1 = 0.00016309 loss)
    I1024 21:25:46.081109  8880 solver.cpp:403] Iteration 9100, lr = 0.00615496
    I1024 21:26:20.738178 28694 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:27:12.128509  8880 solver.cpp:191] Iteration 9200, loss = 0.000375012
    I1024 21:27:12.128881  8880 solver.cpp:206]     Train net output #0: loss = 0.000375012 (* 1 = 0.000375012 loss)
    I1024 21:27:12.128921  8880 solver.cpp:403] Iteration 9200, lr = 0.0061309
    I1024 21:28:38.331079  8880 solver.cpp:191] Iteration 9300, loss = 0.000260434
    I1024 21:28:38.331413  8880 solver.cpp:206]     Train net output #0: loss = 0.000260434 (* 1 = 0.000260434 loss)
    I1024 21:28:38.331450  8880 solver.cpp:403] Iteration 9300, lr = 0.00610706
    I1024 21:29:41.736863 19090 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:30:04.174610  8880 solver.cpp:191] Iteration 9400, loss = 0.000979604
    I1024 21:30:04.174721  8880 solver.cpp:206]     Train net output #0: loss = 0.000979604 (* 1 = 0.000979604 loss)
    I1024 21:30:04.174753  8880 solver.cpp:403] Iteration 9400, lr = 0.00608343
    I1024 21:31:27.506443  8880 solver.cpp:247] Iteration 9500, Testing net (#0)
    I1024 21:31:27.506820  8880 net.cpp:652] Copying source layer mnist
    I1024 21:31:27.506844  8880 net.cpp:652] Copying source layer conv1
    I1024 21:31:27.506862  8880 net.cpp:652] Copying source layer pool1
    I1024 21:31:27.506875  8880 net.cpp:652] Copying source layer conv2
    I1024 21:31:27.506889  8880 net.cpp:652] Copying source layer pool2
    I1024 21:31:27.506901  8880 net.cpp:652] Copying source layer ip1
    I1024 21:31:27.506916  8880 net.cpp:652] Copying source layer relu1
    I1024 21:31:27.506943  8880 net.cpp:652] Copying source layer ip2
    I1024 21:31:27.506968  8880 net.cpp:652] Copying source layer loss
    I1024 21:31:28.764153 21767 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:31:43.352231  4779 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:31:58.498486 19980 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:32:13.377471  2718 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:32:27.173216  8880 solver.cpp:298]     Test net output #0: accuracy = 0.994
    I1024 21:32:27.173272  8880 solver.cpp:298]     Test net output #1: loss = 0.0289859 (* 1 = 0.0289859 loss)
    I1024 21:32:28.009310  8880 solver.cpp:191] Iteration 9500, loss = 0.000104469
    I1024 21:32:28.009440  8880 solver.cpp:206]     Train net output #0: loss = 0.000104469 (* 1 = 0.000104469 loss)
    I1024 21:32:28.009506  8880 solver.cpp:403] Iteration 9500, lr = 0.00606002
    I1024 21:33:55.277410  8880 solver.cpp:191] Iteration 9600, loss = 0.000360392
    I1024 21:33:55.277585  8880 solver.cpp:206]     Train net output #0: loss = 0.000360392 (* 1 = 0.000360392 loss)
    I1024 21:33:55.277600  8880 solver.cpp:403] Iteration 9600, lr = 0.00603682
    I1024 21:34:03.038058  6682 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:35:21.008852  8880 solver.cpp:191] Iteration 9700, loss = 0.000644999
    I1024 21:35:21.009217  8880 solver.cpp:206]     Train net output #0: loss = 0.000644999 (* 1 = 0.000644999 loss)
    I1024 21:35:21.009258  8880 solver.cpp:403] Iteration 9700, lr = 0.00601382
    I1024 21:36:47.406687  8880 solver.cpp:191] Iteration 9800, loss = 0.000648929
    I1024 21:36:47.407044  8880 solver.cpp:206]     Train net output #0: loss = 0.000648929 (* 1 = 0.000648929 loss)
    I1024 21:36:47.407083  8880 solver.cpp:403] Iteration 9800, lr = 0.00599102
    I1024 21:37:24.315086 29240 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:38:13.379084  8880 solver.cpp:191] Iteration 9900, loss = 0.00212456
    I1024 21:38:13.379427  8880 solver.cpp:206]     Train net output #0: loss = 0.00212456 (* 1 = 0.00212456 loss)
    I1024 21:38:13.379477  8880 solver.cpp:403] Iteration 9900, lr = 0.00596843
    I1024 21:39:37.972246  8880 net.cpp:735] Serializing 9 layers
    I1024 21:39:37.984390  8880 solver.cpp:317] Snapshotting to /home/axel/Desktop/MNIST/caffenet_iter_10000.caffemodel
    I1024 21:39:38.003288  8880 solver.cpp:324] Snapshotting solver state to /home/axel/Desktop/MNIST/caffenet_iter_10000.solverstate
    I1024 21:39:38.404368  8880 solver.cpp:228] Iteration 10000, loss = 0.00111275
    I1024 21:39:38.404409  8880 solver.cpp:247] Iteration 10000, Testing net (#0)
    I1024 21:39:38.404417  8880 net.cpp:652] Copying source layer mnist
    I1024 21:39:38.404422  8880 net.cpp:652] Copying source layer conv1
    I1024 21:39:38.404428  8880 net.cpp:652] Copying source layer pool1
    I1024 21:39:38.404433  8880 net.cpp:652] Copying source layer conv2
    I1024 21:39:38.404438  8880 net.cpp:652] Copying source layer pool2
    I1024 21:39:38.404441  8880 net.cpp:652] Copying source layer ip1
    I1024 21:39:38.404446  8880 net.cpp:652] Copying source layer relu1
    I1024 21:39:38.404450  8880 net.cpp:652] Copying source layer ip2
    I1024 21:39:38.404455  8880 net.cpp:652] Copying source layer loss
    I1024 21:39:39.636479 24443 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:39:54.624178  7488 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:40:09.595444 22674 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:40:24.488375  5690 data_layer.cpp:195] Restarting data prefetching from start.
    I1024 21:40:38.111625  8880 solver.cpp:298]     Test net output #0: accuracy = 0.9944
    I1024 21:40:38.111729  8880 solver.cpp:298]     Test net output #1: loss = 0.0285009 (* 1 = 0.0285009 loss)
    I1024 21:40:38.111755  8880 solver.cpp:233] Optimization Done.
    I1024 21:40:38.111768  8880 caffe.cpp:121] Optimization Done.

## Reprendre entrenament¶

En cas que aturéssim l'execució i volguéssim reprendre-la a partir d'un punt concret de l'entremanent que tenim un 'solverstate', el codi seria el del fitxer resume_training.sh o següent:

In []:

    %%bash
    #!/usr/bin/env sh

    caffe-master/build/tools/caffe train \
        --solver=MNIST/Grayscale/lenet_solver.prototxt \
        --snapshot=MNIST/Grayscale/caffenet_train_10000.solverstate

## Altres dades recopil·lades fins el moment¶

Per tal de realitzar el train s'ha provat de les 4 maneres següents amb els següents resultats: - Test amb el paràmetre de grayscale i usant la mitja inicialment:

    - Accuracy final = 0.9898
    - Mida .caffemodel = 1724935 bytes

- Test amb el paràmetre de grayscale sense usar la mitja inicialment:

  - Accuracy final = 0.9897
  - Mida .caffemodel = 1724908 bytes

- Test Color i sense la mitja inicialment:

  - Accuracy final = 0.9909
  - Mida .caffemodel = 1728908 bytes

- Test Color i amb la mitja inicialment:

  - Accuracy final = 0.9919
  - Mida .caffemodel = 1729007 bytes

## Creació del contingut pel cas 3BandesIguals¶

A continuació crearem 3 imatges amb el mateix contingut, però amb el nom apropiat, per comparar amb la color. Això ens servirà pel cas 3BandesIguals

In [2]:

    from PIL import Image
    import os
    # size is width/height
    width = 28
    height = 28
    #Path on estan les imatges
    path = 'MNIST/'
    #Generalitzem per train i test el mateix procès
    tt = ('train','test')
    #Quants canals en total (inicial + extres) o capes volem que hi hagin
    channels = 4
    count = 0

    for t in tt:
        for number in xrange (0,10):
            #Creem la carpeta si no existeix
            if not os.path.exists(path+str(t.upper())+'/MNIST-'+str(number)+'/'):
                os.makedirs(path+str(t.upper())+'/MNIST-'+str(number)+'/')
            #Carreguem la imatge
            img = Image.open(path+'MNIST-'+str(t.upper())+'/mnist_'+str(t)+str(number)+'.jpg')
            (w,h) = img.size

            for left in xrange (0,w,width):
                for top in xrange (0,h,height):
                    box = (left, top, left+width, top+height)
                    area = img.crop(box)
                    area.save(path+str(t.upper())+'/MNIST-'+str(number)+'/image'+str(count)+'.jpg', 'jpeg')
                    area.save(path+str(t.upper())+'/MNIST-'+str(number)+'/image'+str(count)+'.jpg_'+str(0), 'jpeg')
                    area.save(path+str(t.upper())+'/MNIST-'+str(number)+'/image'+str(count)+'.jpg_'+str(1), 'jpeg')
                    count += 1
            count = 0

## Modificació Caffe per tal de llegir més bandes: MNIST 3BandesIguals i 4BandesDiferents¶

Després de realitzar les modificacions als fitxers de la Caffe, generem la lmdb per la MNIST de 3BandesIguals, és a dir, repetint dos cops la mateixa imatge per poder comparar amb la color quan la genera automàticament Caffe. Això ho conseguirem a partir del paràmetre '--channels=2 '.

In [4]:

    %%bash

    #!/usr/bin/env sh
    # Create the imagenet lmdb inputs
    # N.B. set the path to the imagenet train + val data dirs

    EXAMPLE=MNIST
    DATA=MNIST
    TOOLS=caffe-master/build/tools

    TRAIN_DATA_ROOT=MNIST/TRAIN/
    VAL_DATA_ROOT=MNIST/TEST/

    # Set RESIZE=true to resize the images to 256x256. Leave as false if images have
    # already been resized using another tool.
    RESIZE=false
    if $RESIZE; then
      RESIZE_HEIGHT=256
      RESIZE_WIDTH=256
    else
      RESIZE_HEIGHT=0
      RESIZE_WIDTH=0
    fi

    if [ ! -d "$TRAIN_DATA_ROOT" ]; then
      echo "Error: TRAIN_DATA_ROOT is not a path to a directory: $TRAIN_DATA_ROOT"
      echo "Set the TRAIN_DATA_ROOT variable in create_imagenet.sh to the path" \
           "where the ImageNet training data is stored."
      exit 1
    fi

    if [ ! -d "$VAL_DATA_ROOT" ]; then
      echo "Error: VAL_DATA_ROOT is not a path to a directory: $VAL_DATA_ROOT"
      echo "Set the VAL_DATA_ROOT variable in create_imagenet.sh to the path" \
           "where the ImageNet validation data is stored."
      exit 1
    fi

    echo "Creating train lmdb..."

    GLOG_logtostderr=1 $TOOLS/convert_imageset \
        --resize_height=$RESIZE_HEIGHT \
        --resize_width=$RESIZE_WIDTH \
        --shuffle \
        --gray \
        --channels=2 \
        $TRAIN_DATA_ROOT \
        $DATA/train.txt \
        $EXAMPLE/ilsvrc12_train_lmdb

    echo "Creating val lmdb..."

    GLOG_logtostderr=1 $TOOLS/convert_imageset \
        --resize_height=$RESIZE_HEIGHT \
        --resize_width=$RESIZE_WIDTH \
        --shuffle \
        --gray \
        --channels=2 \
        $VAL_DATA_ROOT \
        $DATA/val.txt \
        $EXAMPLE/ilsvrc12_val_lmdb

    echo "Done."

    Creating train lmdb...
    Creating val lmdb...
    Done.

    I1104 18:55:27.369652  8273 convert_imageset.cpp:81] Shuffling data
    I1104 18:55:27.399406  8273 convert_imageset.cpp:84] A total of 60338 images.
    I1104 18:55:27.399549  8273 convert_imageset.cpp:115] Opening lmdb /home/axel/Desktop/MNIST/ilsvrc12_train_lmdb
    E1104 18:55:28.711365  8273 convert_imageset.cpp:188] Processed 1000 files.
    E1104 18:55:30.216711  8273 convert_imageset.cpp:188] Processed 2000 files.
    E1104 18:55:31.623055  8273 convert_imageset.cpp:188] Processed 3000 files.
    E1104 18:55:32.995333  8273 convert_imageset.cpp:188] Processed 4000 files.
    E1104 18:55:34.350042  8273 convert_imageset.cpp:188] Processed 5000 files.
    E1104 18:55:35.584472  8273 convert_imageset.cpp:188] Processed 6000 files.
    E1104 18:55:36.904587  8273 convert_imageset.cpp:188] Processed 7000 files.
    E1104 18:55:38.438586  8273 convert_imageset.cpp:188] Processed 8000 files.
    E1104 18:55:39.966235  8273 convert_imageset.cpp:188] Processed 9000 files.
    E1104 18:55:41.242710  8273 convert_imageset.cpp:188] Processed 10000 files.
    E1104 18:55:42.466511  8273 convert_imageset.cpp:188] Processed 11000 files.
    E1104 18:55:43.701809  8273 convert_imageset.cpp:188] Processed 12000 files.
    E1104 18:55:44.938905  8273 convert_imageset.cpp:188] Processed 13000 files.
    E1104 18:55:46.058944  8273 convert_imageset.cpp:188] Processed 14000 files.
    E1104 18:55:47.192191  8273 convert_imageset.cpp:188] Processed 15000 files.
    E1104 18:55:48.436388  8273 convert_imageset.cpp:188] Processed 16000 files.
    E1104 18:55:49.539903  8273 convert_imageset.cpp:188] Processed 17000 files.
    E1104 18:55:50.621554  8273 convert_imageset.cpp:188] Processed 18000 files.
    E1104 18:55:51.908869  8273 convert_imageset.cpp:188] Processed 19000 files.
    E1104 18:55:53.099059  8273 convert_imageset.cpp:188] Processed 20000 files.
    E1104 18:55:54.192374  8273 convert_imageset.cpp:188] Processed 21000 files.
    E1104 18:55:55.148174  8273 convert_imageset.cpp:188] Processed 22000 files.
    E1104 18:55:56.124300  8273 convert_imageset.cpp:188] Processed 23000 files.
    E1104 18:55:57.152437  8273 convert_imageset.cpp:188] Processed 24000 files.
    E1104 18:55:58.130511  8273 convert_imageset.cpp:188] Processed 25000 files.
    E1104 18:55:59.087213  8273 convert_imageset.cpp:188] Processed 26000 files.
    E1104 18:56:00.072926  8273 convert_imageset.cpp:188] Processed 27000 files.
    E1104 18:56:01.054307  8273 convert_imageset.cpp:188] Processed 28000 files.
    E1104 18:56:02.053084  8273 convert_imageset.cpp:188] Processed 29000 files.
    E1104 18:56:03.065232  8273 convert_imageset.cpp:188] Processed 30000 files.
    E1104 18:56:04.056246  8273 convert_imageset.cpp:188] Processed 31000 files.
    E1104 18:56:04.962882  8273 convert_imageset.cpp:188] Processed 32000 files.
    E1104 18:56:05.935966  8273 convert_imageset.cpp:188] Processed 33000 files.
    E1104 18:56:06.950745  8273 convert_imageset.cpp:188] Processed 34000 files.
    E1104 18:56:07.992007  8273 convert_imageset.cpp:188] Processed 35000 files.
    E1104 18:56:09.238639  8273 convert_imageset.cpp:188] Processed 36000 files.
    E1104 18:56:10.404137  8273 convert_imageset.cpp:188] Processed 37000 files.
    E1104 18:56:11.445845  8273 convert_imageset.cpp:188] Processed 38000 files.
    E1104 18:56:12.438953  8273 convert_imageset.cpp:188] Processed 39000 files.
    E1104 18:56:13.422199  8273 convert_imageset.cpp:188] Processed 40000 files.
    E1104 18:56:14.452347  8273 convert_imageset.cpp:188] Processed 41000 files.
    E1104 18:56:15.475589  8273 convert_imageset.cpp:188] Processed 42000 files.
    E1104 18:56:16.488394  8273 convert_imageset.cpp:188] Processed 43000 files.
    E1104 18:56:17.564012  8273 convert_imageset.cpp:188] Processed 44000 files.
    E1104 18:56:18.624438  8273 convert_imageset.cpp:188] Processed 45000 files.
    E1104 18:56:19.641213  8273 convert_imageset.cpp:188] Processed 46000 files.
    E1104 18:56:20.647462  8273 convert_imageset.cpp:188] Processed 47000 files.
    E1104 18:56:21.635257  8273 convert_imageset.cpp:188] Processed 48000 files.
    E1104 18:56:22.632844  8273 convert_imageset.cpp:188] Processed 49000 files.
    E1104 18:56:23.678966  8273 convert_imageset.cpp:188] Processed 50000 files.
    E1104 18:56:24.773772  8273 convert_imageset.cpp:188] Processed 51000 files.
    E1104 18:56:25.927486  8273 convert_imageset.cpp:188] Processed 52000 files.
    E1104 18:56:26.977828  8273 convert_imageset.cpp:188] Processed 53000 files.
    E1104 18:56:28.015910  8273 convert_imageset.cpp:188] Processed 54000 files.
    E1104 18:56:29.062736  8273 convert_imageset.cpp:188] Processed 55000 files.
    E1104 18:56:30.060458  8273 convert_imageset.cpp:188] Processed 56000 files.
    E1104 18:56:31.065296  8273 convert_imageset.cpp:188] Processed 57000 files.
    E1104 18:56:32.051267  8273 convert_imageset.cpp:188] Processed 58000 files.
    E1104 18:56:33.075911  8273 convert_imageset.cpp:188] Processed 59000 files.
    E1104 18:56:34.129081  8273 convert_imageset.cpp:188] Processed 60000 files.
    E1104 18:56:34.466436  8273 convert_imageset.cpp:204] Processed 60338 files.
    I1104 18:56:34.586884  8285 convert_imageset.cpp:81] Shuffling data
    I1104 18:56:34.591235  8285 convert_imageset.cpp:84] A total of 10000 images.
    I1104 18:56:34.591323  8285 convert_imageset.cpp:115] Opening lmdb /home/axel/Desktop/MNIST/ilsvrc12_val_lmdb
    E1104 18:56:35.561192  8285 convert_imageset.cpp:188] Processed 1000 files.
    E1104 18:56:36.508972  8285 convert_imageset.cpp:188] Processed 2000 files.
    E1104 18:56:37.445854  8285 convert_imageset.cpp:188] Processed 3000 files.
    E1104 18:56:38.457384  8285 convert_imageset.cpp:188] Processed 4000 files.
    E1104 18:56:39.680675  8285 convert_imageset.cpp:188] Processed 5000 files.
    E1104 18:56:40.725721  8285 convert_imageset.cpp:188] Processed 6000 files.
    E1104 18:56:42.130676  8285 convert_imageset.cpp:188] Processed 7000 files.
    E1104 18:56:43.118180  8285 convert_imageset.cpp:188] Processed 8000 files.
    E1104 18:56:44.118145  8285 convert_imageset.cpp:188] Processed 9000 files.
    E1104 18:56:45.163177  8285 convert_imageset.cpp:188] Processed 10000 files.

I a continuació posariem el codi del create_imagenet.sh per la MNIST 4BandesDiferents. És a dir, agafar la imatge com a Grayscale i carregar 3 bandes extres.

In []:

    %%bash

    #!/usr/bin/env sh
    # Create the imagenet lmdb inputs
    # N.B. set the path to the imagenet train + val data dirs

    EXAMPLE=MNIST
    DATA=MNIST
    TOOLS=caffe-master/build/tools

    TRAIN_DATA_ROOT=MNIST/TRAIN/
    VAL_DATA_ROOT=MNIST/TEST/

    # Set RESIZE=true to resize the images to 256x256. Leave as false if images have
    # already been resized using another tool.
    RESIZE=false
    if $RESIZE; then
      RESIZE_HEIGHT=256
      RESIZE_WIDTH=256
    else
      RESIZE_HEIGHT=0
      RESIZE_WIDTH=0
    fi

    if [ ! -d "$TRAIN_DATA_ROOT" ]; then
      echo "Error: TRAIN_DATA_ROOT is not a path to a directory: $TRAIN_DATA_ROOT"
      echo "Set the TRAIN_DATA_ROOT variable in create_imagenet.sh to the path" \
           "where the ImageNet training data is stored."
      exit 1
    fi

    if [ ! -d "$VAL_DATA_ROOT" ]; then
      echo "Error: VAL_DATA_ROOT is not a path to a directory: $VAL_DATA_ROOT"
      echo "Set the VAL_DATA_ROOT variable in create_imagenet.sh to the path" \
           "where the ImageNet validation data is stored."
      exit 1
    fi

    echo "Creating train lmdb..."

    GLOG_logtostderr=1 $TOOLS/convert_imageset \
        --resize_height=$RESIZE_HEIGHT \
        --resize_width=$RESIZE_WIDTH \
        --shuffle \
        --gray \
        --channels=3 \
        $TRAIN_DATA_ROOT \
        $DATA/train.txt \
        $EXAMPLE/ilsvrc12_train_lmdb

    echo "Creating val lmdb..."

    GLOG_logtostderr=1 $TOOLS/convert_imageset \
        --resize_height=$RESIZE_HEIGHT \
        --resize_width=$RESIZE_WIDTH \
        --shuffle \
        --gray \
        --channels=3 \
        $VAL_DATA_ROOT \
        $DATA/val.txt \
        $EXAMPLE/ilsvrc12_val_lmdb

    echo "Done."

## Exemple classificació d'una imatge a partir de xarxa entrenada¶

A continuació el codi per donada una imatge classificar-la mitjançant un model de xarxa neuronal ja entrenada.

In [10]:

    import numpy as np
    import matplotlib.pyplot as plt
    %matplotlib inline

    # Possem la localització on es troba la carpeta mare de el framework Caffe:
    caffe_root = 'caffe-master/'
    import sys
    sys.path.insert(0, caffe_root + 'python')
    #sys.path.insert(0,'/usr/local/lib/python2.7/dist-packages')

    import caffe

    # Posem el camí al model, pretrained model weights i les imatges que volem classificar.
    MODEL_FILE = 'MNIST/MNIST-4BandesDiferents/caffenet/deploy.prototxt'
    PRETRAINED = 'MNIST/MNIST-4BandesDiferents/caffenet_iter_10000.caffemodel'
    net = caffe.Classifier(MODEL_FILE, PRETRAINED) #Carreguem la xarxa a partir d'un model
    #d'una xarxa ja entrenada i el deploy que ens marcarà la seva estructura.
    net.set_phase_test()
    net.set_mode_cpu() #Fem que només usarem GPU

Destaquem que en el fitxer 'deploy' hem de posar les dimensions de la imatge d'entrada. Per exemple, si estem en el cas de la MNIST-4BandesDiferents el deploy hem de col·locar els valors input_dim: 1, input_dim: 4, input_dim: 28, input_dim: 28. És a dir, un batch de 1, 4 canals, i unes imatges de 28x28.

### Visualització imatge i classificació d'aquesta¶

Anem a classificar un exemple d'imatge i visualitzar els seus filtres corresponents.

A continuació carreguem mitjançant la funció \(load\_image\) de Caffe una imatge especificant-li que és una imatge d'escala de grisos mitjançant el paràmetre 'color=False'

In [2]:

    IMAGE_FILE = '/home/axel/Desktop/MNIST/TEST/MNIST-0/image42.jpg'
    input_image = caffe.io.load_image(IMAGE_FILE,color=False)
    input_image2 = caffe.io.load_image(IMAGE_FILE)
    plt.imshow(input_image2)

Out[2]:

    <matplotlib.image.AxesImage at 0x7f5c1b215590>

Ara expliquem un exemple de com carregar una imatge i les \(3\) imatges següents com a canals per la 4BandesDiferents

In [12]:

    IMAGE_FILE = 'MNIST/TEST/MNIST-0/image42.jpg'
    input_image = caffe.io.load_image(IMAGE_FILE,color=False,channels=3)
    input_image2 = caffe.io.load_image(IMAGE_FILE)
    plt.imshow(input_image2)

Out[12]:

    <matplotlib.image.AxesImage at 0x7f39cb7db590>

I ara realitzem la classificació. Obtenim el nombre de classes, la classe màxima classificada i un gràfic que ens mostra per a cada classe quina puntuació té.

In [3]:

    prediction = net.predict([input_image])  # predict takes any number of images, and formats them for the Caffe net automatically
    print 'prediction shape:', prediction[0].shape
    plt.plot(prediction[0])
    print 'predicted class:', prediction[0].argmax()

    prediction shape: (10,)
    predicted class: 0

Ara desactivem l'oversampling de manera que eliminem soroll.

In [36]:

    prediction = net.predict([input_image], oversample=False)
    print 'prediction shape:', prediction[0].shape
    plt.plot(prediction[0])
    print 'predicted class:', prediction[0].argmax()

    prediction shape: (10,)
    predicted class: 0

## Visualització de filtres¶

Seguidament, visualitzem les diferents capes que està formada la nostre xarxa neuronal carregada

In [7]:

    [(k, v.data.shape) for k, v in net.blobs.items()]

Out[7]:

    [('data', (1, 4, 28, 28)),
     ('conv1', (1, 20, 24, 24)),
     ('pool1', (1, 20, 12, 12)),
     ('conv2', (1, 50, 8, 8)),
     ('pool2', (1, 50, 4, 4)),
     ('ip1', (1, 500, 1, 1)),
     ('ip2', (1, 10, 1, 1)),
     ('prob', (1, 10, 1, 1))]

I el nombre de paràmetres que conté els diferents blobs

In [8]:

    [(k, v[0].data.shape) for k, v in net.params.items()]

Out[8]:

    [('conv1', (20, 4, 5, 5)),
     ('conv2', (50, 20, 5, 5)),
     ('ip1', (1, 1, 500, 800)),
     ('ip2', (1, 1, 10, 500))]

Definim una funció \(vis\_square\) que ens permetrà fer la visualització posteriorment dels filtres

In [8]:

    # take an array of shape (n, height, width) or (n, height, width, channels)
    #  and visualize each (height, width) thing in a grid of size approx. sqrt(n) by sqrt(n)
    def vis_square(data, padsize=1, padval=0):
        #Normalitza
        data -= data.min()
        data /= data.max()

        # force the number of filters to be square
        n = int(np.ceil(np.sqrt(data.shape[0])))
        padding = ((0, n ** 2 - data.shape[0]), (0, padsize), (0, padsize)) + ((0, 0),) * (data.ndim - 3)
        data = np.pad(data, padding, mode='constant', constant_values=(padval, padval))

        # tile the filters into an image
        data = data.reshape((n, n) + data.shape[1:]).transpose((0, 2, 1, 3) + tuple(range(4, data.ndim + 1)))
        data = data.reshape((n * data.shape[1], n * data.shape[3]) + data.shape[4:])

        plt.imshow(data)

Si volem veure les mides del primer filtre de la convolucional primera conv1 seria

In [34]:

    filters.shape
    #data = filters
    #Normalitza
    # -= data.min()
    #data /= data.max()

Out[34]:

    (50, 20, 5, 5)

Mostrar el filtre en escala de grisos ho fariem de la següent manera:

In [94]:

    # the parameters are a list of [weights, biases]
    filters = net.params['conv1'][0].data
    a = plt.imshow(filters[19][3])
    a.set_cmap('gray')
    a.set_interpolation('none')
    filters[0][0].shape
    #vis_square(filters.transpoe(0, 2, 3, 1))

Out[94]:

    (5, 5)

I si ara volem fer el test sobre la segona convolucional seria:

In [33]:

    # the parameters are a list of [weights, biases]
    filters = net.params['conv2'][0].data
    a = plt.imshow(filters[0][0])
    a.set_cmap('gray')
    a.set_interpolation('none')
    filters[0][0].shape
    #vis_square(filters.transpoe(0, 2, 3, 1))

Out[33]:

    (5, 5)

I si volem vure tots els filtres d'una capa el codi podria ser:

In [15]:

    import numpy as plotting
    import matplotlib
    from pylab import *

    #creating a wide figure with 2 subplots in 1 row
    fig, axes = plt.subplots(filters.shape[0]/6, filters.shape[1], figsize=(9,3))

    i = 0
    j=0
    f = [1,3,14,19]

    for ax in axes.flatten():  # flatten in case you have a second row at some point
        img = ax.imshow(filters[f[i]][j],interpolation='none')
        img.set_cmap('gray')
        ax.set_aspect('auto')
        i +=1
        if not i%(len(f)):
            i = 0
            j +=1

    plt.colorbar(img)

Out[15]:

    <matplotlib.colorbar.Colorbar instance at 0x7f39caf5ee60>

O si volem mostrar els filtres tots junts ho fariem així:

In [82]:

    import numpy as plotting
    import matplotlib
    from pylab import *

    #creating a wide figure with 2 subplots in 1 row
    fig, axes = plt.subplots(filters.shape[0]/6, filters.shape[1], figsize=(9,3))

    i = 0
    j=0
    f = [1,3,14,19]

    for ax in axes.flatten():  # flatten in case you have a second row at some point
        img = ax.imshow(filters[f[i]][j],interpolation='none')
        img.set_cmap('gray')
        ax.set_aspect('auto')
        i +=1
        if not i%(len(f)):
            i = 0
            j +=1

    plt.colorbar(img)

Out[82]:

    <matplotlib.colorbar.Colorbar instance at 0x7fd8faadbb90>

Anem ara a mostrar les features de la conv1

In [51]:

    features = net.blobs['conv1'].data[0,19]
    a=plt.imshow(features)
    a.set_cmap('gray')
    a.set_interpolation('none')

Visualitzem ara la matriu de confusió del classificador del fitxer de la mitja generada a partir de les imatges de training amb l'script \(make\_imagenet\_mean.sh\). Concretament, visualitzarem la matriu de confusió de la quarta capa afegida extra.

In [25]:

    import numpy as np
    import matplotlib.pyplot as plt
    %matplotlib inline

    # Make sure that caffe is on the python path:
    caffe_root = '/home/axel/Desktop/caffe-master/'  # this file is expected to be in {caffe_root}/examples
    import sys
    sys.path.insert(0, caffe_root + 'python')
    #sys.path.insert(0,'/usr/local/lib/python2.7/dist-packages')

    import caffe
    from caffe.proto import caffe_pb2
    from caffe import io

    #Convertim el binaryproto mean en un numpy 'arr' mitjançant el blob
    blob = caffe.proto.caffe_pb2.BlobProto()
    data = open('/home/axel/Desktop/MNIST/caffemodels-logs/4BandesDiferents/imagenet_mean.binaryproto' , 'rb' ).read()
    blob.ParseFromString(data)
    arr = np.array( caffe.io.blobproto_to_array(blob) )
    out = arr[0]

    #Imprimim el nombre de dimensions
    print out.shape

    fig = plt.figure()
    ax = fig.add_subplot(111)
    #Optenim la matriu de confusió del quart canal
    cax = ax.matshow(out[3,:,:])
    plt.title('Matriu de confusio del classificador')
    fig.colorbar(cax)
    plt.show()

    (4, 28, 28)

Aquesta forma té sentit ja que és la unió entre els diferents nombres. La imatges s'ha d'interpretar que com més tons càlids més informació es troba.

## Test d'encert mitjançant l'extensió de Python de Caffe.¶

Anem a implementar un petit test d'imatges per veure com funciona l'extensió Python de Caffe i veure quants accerts realitza mitjançant la predicció.

Nota: Pel tipus d'implementació que té Caffe del Python wrapper la predicció acaba sent, generalment, una mica inferior a la que es realitza durant l'entrenament per consola.

In [84]:

    freadVal = open('MNIST/val.txt','r')
    IMAGE_FILE = ''
    hit = 0
    tries = 0
    for files in freadVal.readlines():
        IMAGE_FILE = 'MNIST/TEST/'+str(content[tries].split()[0])
        input_image = caffe.io.load_image(IMAGE_FILE,color=False)
        prediction = net.predict([input_image],oversample=False)
        if str(prediction[0].argmax()) == content[tries].split()[0][6]:
            hit +=1
        tries += 1
        if not tries%100:
            print "For now, accuracy: "+str(hit)+" of "+str(tries)
    freadVal.close()
    print "Accuracy: "+str(hit*100/float(tries))+" %"

    For now, accuracy: 100 of 100
    For now, accuracy: 200 of 200
    For now, accuracy: 300 of 300
    For now, accuracy: 400 of 400
    For now, accuracy: 500 of 500
    For now, accuracy: 599 of 600
    For now, accuracy: 699 of 700
    For now, accuracy: 798 of 800
    For now, accuracy: 896 of 900
    For now, accuracy: 994 of 1000
    For now, accuracy: 1094 of 1100
    For now, accuracy: 1194 of 1200
    For now, accuracy: 1294 of 1300
    For now, accuracy: 1394 of 1400
    For now, accuracy: 1492 of 1500
    For now, accuracy: 1591 of 1600
    For now, accuracy: 1690 of 1700
    For now, accuracy: 1790 of 1800
    For now, accuracy: 1889 of 1900
    For now, accuracy: 1989 of 2000
    For now, accuracy: 2088 of 2100
    For now, accuracy: 2187 of 2200
    For now, accuracy: 2284 of 2300
    For now, accuracy: 2382 of 2400
    For now, accuracy: 2481 of 2500
    Accuracy: 99.2412140575 %

## Test d'encert amb N possibilitats¶

A continuació posem el codi que serviria per realitzar un test que ens imrpimiria la ràtio d'encert amb N intents

In []:

    import datetime

    freadVal = open('MNIST/Color/val.txt','r')
    IMAGE_FILE = ''
    #Verify the last N hits
    Nhits = 5

    hit = [0]*Nhits
    tries = 0

    #Per cada fitxer
    for files in freadVal.readlines():
        #Agafem la seva ruta
        IMAGE_FILE = 'MNIST/TEST-3BandesIguals-Color/'+str(files.split()[0])
        #Carreguem la imatge i les n channels posteriors
        input_image = caffe.io.load_image(IMAGE_FILE,color=False,channels=2)
        #predim la possible class
        prediction = netColor.predict([input_image])

        #Contabilitzem els hits

        #Copiem contingut
        lPredict = prediction[0]
        #Ordenem aquesta segona dnarray i la invertim perquè el màxim estigui
        # al principi
        lPredict = np.sort(lPredict)[::-1]
        #Per cada N hits comprovem si coincideix
        # amb el valor de la classe buscada amb la N-essima posició
        for i in xrange(Nhits):
            if lPredict[i] == prediction[0][int(files.split()[1])]:
                hit[i] +=1
                break

        tries += 1
        #Cada 100 iteracions fem un print de la situació
        if not tries%100:
            print str(datetime.datetime.now().time())
            for i in xrange(Nhits):
                print "\t Accuracy at "+str(i+1)+" hits: "+str(sum(hit[:(i+1)]))+" of "+str(tries)
    freadVal.close()
    print str(datetime.datetime.now().time())
    for i in xrange(Nhits):
        print "\t Final Accuracy at "+str(i+1)+" hits: "+str(sum(hit[:(i+1)])*100/float(tries))+" %"

Un exemple de resultat seria:

     Final Accuracy at 1 hits: 97.88 %
     Final Accuracy at 2 hits: 98.94 %
     Final Accuracy at 3 hits: 99.92 %
     Final Accuracy at 4 hits: 100.0 %
     Final Accuracy at 5 hits: 100.0 %

## Comparació increment ACCURACY i decrement LOSS function¶

### Cas Grayscale¶

In [28]:

    import numpy as np
    import matplotlib.pyplot as plt
    %matplotlib inline
    regression=0
    loss,res=[],[]
    with open('MNIST/Grayscale/Consola.log') as f:
        for line in f:
            index= line.find('accuracy = ')
            if(index>0):
                if(regression):
                    index2= line.find('(*')
                    res.append(float(line[index+11:index2]))
                else:
                    res.append(100.0*float(line[index+11:]))
            index= line.find(', loss = ')
            if(index>0):
                loss.append(float(line[index+9:]))

    fig= plt.figure(figsize=(16,3))
    axes1= fig.add_subplot(1, 2, 1)
    axes2= fig.add_subplot(1, 2, 2)
    #Pintarem les dades recollides, vermella la LOSS, verda l'ACC
    axes1.plot(loss,color='r'),axes2.plot(res,color='g')
    axes1.set_ylim([min(loss),max(loss[1:])]),axes2.set_ylim([0,100]),axes2.set_xlim([0,10])
    axes1.set_title('LOSS'),axes1.set_xlabel('Num Iteracions x 100'),axes1.set_ylabel('Loss')
    axes1.yaxis.grid(),axes1.xaxis.grid()
    axes2.set_title('% ENCERT'),axes2.set_xlabel('Num Iteracions x 1000'),axes2.set_ylabel('% Accuracy')
    axes2.yaxis.grid(),axes2.xaxis.grid()
    fig.show()
    print max(res)

    99.06

### Color¶

In [27]:

    import numpy as np
    import matplotlib.pyplot as plt
    %matplotlib inline
    regression=0
    loss,res=[],[]
    with open('MNIST/Color/Consola.log') as f:
        for line in f:
            index= line.find('accuracy = ')
            if(index>0):
                if(regression):
                    index2= line.find('(*')
                    res.append(float(line[index+11:index2]))
                else:
                    res.append(100.0*float(line[index+11:]))
            index= line.find(', loss = ')
            if(index>0):
                loss.append(float(line[index+9:]))

    fig= plt.figure(figsize=(16,3))
    axes1= fig.add_subplot(1, 2, 1)
    axes2= fig.add_subplot(1, 2, 2)
    #Pintarem les dades recollides, vermella la LOSS, verda l'ACC
    axes1.plot(loss,color='r'),axes2.plot(res,color='g')
    axes1.set_ylim([min(loss),max(loss[1:])]),axes2.set_ylim([0,100]),axes2.set_xlim([0,10])
    axes1.set_title('LOSS'),axes1.set_xlabel('Num Iteracions x 100'),axes1.set_ylabel('Loss')
    axes1.yaxis.grid(),axes1.xaxis.grid()
    axes2.set_title('% ENCERT'),axes2.set_xlabel('Num Iteracions x 1000'),axes2.set_ylabel('% Accuracy')
    axes2.yaxis.grid(),axes2.xaxis.grid()
    fig.show()
    print max(res)

    99.19

### 3BandesIguals¶

In [29]:

    import numpy as np
    import matplotlib.pyplot as plt
    %matplotlib inline
    regression=0
    loss,res=[],[]
    with open('/MNIST/3BandesIguals/Consola.log') as f:
        for line in f:
            index= line.find('accuracy = ')
            if(index>0):
                if(regression):
                    index2= line.find('(*')
                    res.append(float(line[index+11:index2]))
                else:
                    res.append(100.0*float(line[index+11:]))
            index= line.find(', loss = ')
            if(index>0):
                loss.append(float(line[index+9:]))

    fig= plt.figure(figsize=(16,3))
    axes1= fig.add_subplot(1, 2, 1)
    axes2= fig.add_subplot(1, 2, 2)
    #Pintarem les dades recollides, vermella la LOSS, verda l'ACC
    axes1.plot(loss,color='r'),axes2.plot(res,color='g')
    axes1.set_ylim([min(loss),max(loss[1:])]),axes2.set_ylim([0,100]),axes2.set_xlim([0,10])
    axes1.set_title('LOSS'),axes1.set_xlabel('Num Iteracions x 100'),axes1.set_ylabel('Loss')
    axes1.yaxis.grid(),axes1.xaxis.grid()
    axes2.set_title('% ENCERT'),axes2.set_xlabel('Num Iteracions x 1000'),axes2.set_ylabel('% Accuracy')
    axes2.yaxis.grid(),axes2.xaxis.grid()
    fig.show()
    print max(res)

    99.12

### 4BandesDiferents¶

In [30]:

    import numpy as np
    import matplotlib.pyplot as plt
    %matplotlib inline
    regression=0
    loss,res=[],[]
    with open('MNIST/4BandesDiferents/Consola.log') as f:
        for line in f:
            index= line.find('accuracy = ')
            if(index>0):
                if(regression):
                    index2= line.find('(*')
                    res.append(float(line[index+11:index2]))
                else:
                    res.append(100.0*float(line[index+11:]))
            index= line.find(', loss = ')
            if(index>0):
                loss.append(float(line[index+9:]))

    fig= plt.figure(figsize=(16,3))
    axes1= fig.add_subplot(1, 2, 1)
    axes2= fig.add_subplot(1, 2, 2)
    #Pintarem les dades recollides, vermella la LOSS, verda l'ACC
    axes1.plot(loss,color='r'),axes2.plot(res,color='g')
    axes1.set_ylim([min(loss),max(loss[1:])]),axes2.set_ylim([0,100]),axes2.set_xlim([0,10])
    axes1.set_title('LOSS'),axes1.set_xlabel('Num Iteracions x 100'),axes1.set_ylabel('Loss')
    axes1.yaxis.grid(),axes1.xaxis.grid()
    axes2.set_title('% ENCERT'),axes2.set_xlabel('Num Iteracions x 1000'),axes2.set_ylabel('% Accuracy')
    axes2.yaxis.grid(),axes2.xaxis.grid()
    fig.show()
    print max(res)

    99.48