Intall fbtorch

I referred to
https://github.com/facebookarchive/fbcunn/blob/master/INSTALL.md
in order to install fbtorch.

The key step for me is

curl -sk https://raw.githubusercontent.com/soumith/fblualib/master/install_all.sh | bash -e

which tries to install Folly, fbthrift, thpp and fblualib as the prerequisites.

It turns out that Folly and fbthrift are installed smoothly. However, some errors are raised for thpp. One solution is to install thpp manually. Here is how it goes:

1. Clone version 1.0 (stead of some default one)

git clone -b v1.0 https://github.com/facebook/thpp

2. Go under thpp/thpp and edit build.sh:
change the url in line 19 to
https://github.com/google/googletest/archive/release-1.7.0.zip

Then unzip googletest-release-1.7.0.zip, and rename it to gtest-1.7.0

3. Make the following changes in thpp/thpp/detail:
line 191:

return THTensor_(max)(values, indices, t, dim); 

to

return THTensor_(max)(values, indices, t, dim, 1); 

line 195:

return THTensor_(min)(values, indices, t, dim)

to

return THTensor_(min)(values, indices, t, dim, 1); 

line 198:

return THTensor_(sum)(values, indices, t, dim); 

to

return THTensor_(sum)(values, indices, t, dim, 1); 

line 201:

return THTensor_(prod)(values, indices, t, dim); 

to

return THTensor_(prod)(values, indices, t, dim, 1); 

Now we can install thpp by ./build.sh

4. Since fblualib is not installed due to the error with thpp, we will also install it manually

git clone https://github.com/soumith/fblualib;

cd fblualib/fblualib

./build.sh

caffe stepsize rule

Caffe stepsize rules are in 

/caffe-master/src/caffe/proto/caffe.proto 

// The learning rate decay policy. The currently implemented learning rate
// policies are as follows:
//    - fixed: always return base_lr.
//    - step: return base_lr * gamma ^ (floor(iter / step))
//    - exp: return base_lr * gamma ^ iter
//    - inv: return base_lr * (1 + gamma * iter) ^ (- power)
//    - multistep: similar to step but it allows non uniform steps defined by
//      stepvalue
//    - poly: the effective learning rate follows a polynomial decay, to be
//      zero by the max_iter. return base_lr (1 - iter/max_iter) ^ (power)
//    - sigmoid: the effective learning rate follows a sigmod decay
//      return base_lr ( 1/(1 + exp(-gamma * (iter - stepsize))))
//
// where base_lr, max_iter, gamma, step, stepvalue and power are defined
// in the solver parameter protocol buffer, and iter is the current iteration.

very important issue with momentum in theano

The theano tutorial has an example of using momentum in gradient based optimization. Details are here (see [24]). The gist is here:

 # initialize momentum  
 param_update = theano.shared(param.get_value()*0., broadcastable=param.broadcastable)  
   
 # take a gradient step using momentum 
 updates.append((param, param - learning_rate*param_update))   
   
 # update momentum  
 updates.append((param_update, momentum*param_update + (1. - momentum)*T.grad(cost, param)))   
   

However, it's not quite right. Notice that the shared variables in theano are updated in parallel, not one after the other, e.g., if you have the following update rule for shared variables a and b

update=[(a,f(a,b)), (b, g(a,b))]

Then at iterate t+1, a gets f(a_t,b_t), b gets g(a_t,b_t), not g(a_{t+1},b_t)!!!

In the cited example, "param" is going to use the momentum in the last step, not the updated one! The correct way is to change the 2nd line of the above code:

 # update momentum  
 updates.append((param, param - learning_rate*momentum*param_update + (1. - momentum)*T.grad(cost, param)))  

reminder on GPU

In my deep learning research, a piece of GeForce Titan is mostly sufficient for proof-of-concept experiments. While a GeForce TitanX is enough for larger scale ones. Tesla GPUs are luxury (support double-precision float) but less a good fit in practice.

Some parameters of these gpus are as follows:

model                       memory        cores         price ($)
Geforce Titan            6G               2688          600~800 (amazon)
Geforce TitanX         12G              3072          1100 (@Xi,amazon)
Tesla K20                 5G                2496           1800 (amazon)
Tesla K40                 12G              2880           3000 (amazon)
Tesla K80                 24G              4992           4000+ (amazon)

After plugging the GPU onto the motherborad, always choose the right driver for it:  http://www.nvidia.com/Download/Find.aspx

After installing the driver, install the cuda toolkit,
http://insiderattack.blogspot.com/2014/12/installing-cuda-toolkit-65-in-ubuntu.html

Sometimes, you may want an additional cudnn package for convolutional neural network. Download cudnn library, unpack and copy its files to appropriate directories.

 sudo cp dir_of_cudnn/lib* /usr/local/cuda/lib64  
 sudo cp dir_of_cudnn/cudnn.h /usr/local/cuda/include  

Detailed guidance of other hardware can be found at 
http://timdettmers.com/2015/03/09/deep-learning-hardware-guide/

automatic indenting python when using vim editor

Create a directory under ~/.vim, called ftplugin. Then create a file called python.vim there:

 $mkdir ~/.vim/ftplugin  

 $vim ~/.vim/ftplugin/python.vim  

Insert the following into python.vim
setlocal tabstop=4
setlocal softtabstop=4
setlocal shiftwidth=4
setlocal textwidth=80
setlocal smarttab
setlocal expandtab

create a file named as .vimrc under ~

 $vim ~/.vimrc  

There insert the following text
filetype plugin indent on

Now if you write a .py script in vim, hitting tab once will indent by 4 blankspaces.

Count the Number of Files

We may want to know how many files and subdirectories there are in the current directory. The following is quick solution:

 ls . | wc -l  

Note that the "-l" is an "\ell" not one! This command lists the names of all the files and sub-directories, pipes the list to a tool called wc. Then the wc tool counts the lines of this list, which is the number of the files and sub-directories.

Sometimes, we may want to know the number of files in the sub-directories (of the current directory). This kind of recursive count can be accomplished by

 find . -type f | wc -l  

Here "-type f" restricts the count to be performed on files, not directories.

Similarly, we can do

 find . -type d | wc -l  

to recursively count the directories. But notice that the current directory . is counted in as well. Hence the returned number minus one gives the number of sub-directories.

Two-sided Print from Comand Line in Ubuntu

A network printer can be set up by running

 $ system-config-printer  

If you are assigned a user name in using that printer, you may still have to print a file from command line. The disadvantage of command line is the difficulty of page setups. The following switches between one-sided and double sided, so at least helps a little bit.

 $ lpr path/to/printed/file -U username -o sides=two-sided-long-edge