caffe stepsize rule

Caffe stepsize rules are in 

/caffe-master/src/caffe/proto/caffe.proto 

// The learning rate decay policy. The currently implemented learning rate
// policies are as follows:
//    - fixed: always return base_lr.
//    - step: return base_lr * gamma ^ (floor(iter / step))
//    - exp: return base_lr * gamma ^ iter
//    - inv: return base_lr * (1 + gamma * iter) ^ (- power)
//    - multistep: similar to step but it allows non uniform steps defined by
//      stepvalue
//    - poly: the effective learning rate follows a polynomial decay, to be
//      zero by the max_iter. return base_lr (1 - iter/max_iter) ^ (power)
//    - sigmoid: the effective learning rate follows a sigmod decay
//      return base_lr ( 1/(1 + exp(-gamma * (iter - stepsize))))
//
// where base_lr, max_iter, gamma, step, stepvalue and power are defined
// in the solver parameter protocol buffer, and iter is the current iteration.

very important issue with momentum in theano

The theano tutorial has an example of using momentum in gradient based optimization. Details are here (see [24]). The gist is here:

 # initialize momentum  
 param_update = theano.shared(param.get_value()*0., broadcastable=param.broadcastable)  
   
 # take a gradient step using momentum 
 updates.append((param, param - learning_rate*param_update))   
   
 # update momentum  
 updates.append((param_update, momentum*param_update + (1. - momentum)*T.grad(cost, param)))   
   

However, it's not quite right. Notice that the shared variables in theano are updated in parallel, not one after the other, e.g., if you have the following update rule for shared variables a and b

update=[(a,f(a,b)), (b, g(a,b))]

Then at iterate t+1, a gets f(a_t,b_t), b gets g(a_t,b_t), not g(a_{t+1},b_t)!!!

In the cited example, "param" is going to use the momentum in the last step, not the updated one! The correct way is to change the 2nd line of the above code:

 # update momentum  
 updates.append((param, param - learning_rate*momentum*param_update + (1. - momentum)*T.grad(cost, param)))  

reminder on GPU

In my deep learning research, a piece of GeForce Titan is mostly sufficient for proof-of-concept experiments. While a GeForce TitanX is enough for larger scale ones. Tesla GPUs are luxury (support double-precision float) but less a good fit in practice.

Some parameters of these gpus are as follows:

model                       memory        cores         price ($)
Geforce Titan            6G               2688          600~800 (amazon)
Geforce TitanX         12G              3072          1100 (@Xi,amazon)
Tesla K20                 5G                2496           1800 (amazon)
Tesla K40                 12G              2880           3000 (amazon)
Tesla K80                 24G              4992           4000+ (amazon)

After plugging the GPU onto the motherborad, always choose the right driver for it:  http://www.nvidia.com/Download/Find.aspx

After installing the driver, install the cuda toolkit,
http://insiderattack.blogspot.com/2014/12/installing-cuda-toolkit-65-in-ubuntu.html

Sometimes, you may want an additional cudnn package for convolutional neural network. Download cudnn library, unpack and copy its files to appropriate directories.

 sudo cp dir_of_cudnn/lib* /usr/local/cuda/lib64  
 sudo cp dir_of_cudnn/cudnn.h /usr/local/cuda/include  

Detailed guidance of other hardware can be found at 
http://timdettmers.com/2015/03/09/deep-learning-hardware-guide/

automatic indenting python when using vim editor

Create a directory under ~/.vim, called ftplugin. Then create a file called python.vim there:

 $mkdir ~/.vim/ftplugin  

 $vim ~/.vim/ftplugin/python.vim  

Insert the following into python.vim
setlocal tabstop=4
setlocal softtabstop=4
setlocal shiftwidth=4
setlocal textwidth=80
setlocal smarttab
setlocal expandtab

create a file named as .vimrc under ~

 $vim ~/.vimrc  

There insert the following text
filetype plugin indent on

Now if you write a .py script in vim, hitting tab once will indent by 4 blankspaces.

Count the Number of Files

We may want to know how many files and subdirectories there are in the current directory. The following is quick solution:

 ls . | wc -l  

Note that the "-l" is an "\ell" not one! This command lists the names of all the files and sub-directories, pipes the list to a tool called wc. Then the wc tool counts the lines of this list, which is the number of the files and sub-directories.

Sometimes, we may want to know the number of files in the sub-directories (of the current directory). This kind of recursive count can be accomplished by

 find . -type f | wc -l  

Here "-type f" restricts the count to be performed on files, not directories.

Similarly, we can do

 find . -type d | wc -l  

to recursively count the directories. But notice that the current directory . is counted in as well. Hence the returned number minus one gives the number of sub-directories.

Two-sided Print from Comand Line in Ubuntu

A network printer can be set up by running

 $ system-config-printer  

If you are assigned a user name in using that printer, you may still have to print a file from command line. The disadvantage of command line is the difficulty of page setups. The following switches between one-sided and double sided, so at least helps a little bit.

 $ lpr path/to/printed/file -U username -o sides=two-sided-long-edge   

Keep Track of Training and Testing Loss When Using Caffe

When using Caffe to train a deep neural network, how to record the training/testing loss or accuracy, throughout iterations?

I know the following works, though without understanding the details.

Run in terminal:

 $ caffe_root/build/tools/caffe train --solver=solver.prototxt 2>&1 | tee mylog.log  

Notice the appended part in red, which will log the information shown in the terminal to "mylog.log"

Then run in terminal,

 $ python caffe_root/tools/extra/parse_log.py mylog.log ./  

Now you will see under ./, there are two files, mylog.log.train, and mylog.log.test. They are two csv files that record training and testing loss.

Then you could run gnuplot to quickly visualize the loss/accuracy during iterations. But first you need to comment the 1st line of the two files by #

The .train (or .test) file is of the following form:

#NumIters,Seconds,LearningRate,accuracy,loss
0,****,****,****,****
100,****,****,****,****
.
.
.

Then in terminal run,

 $ gnuplot  
 gnuplot> set datafile separator ','  
 gnuplot> plot 'mylog.log.train' using 1:4 with line # accuracy throughout the iterations  
 gnuplot> plot 'mylog.log.train' using 1:5 with line # loss throughout the iterations