Install notes for Tensorflow

Installed CUDA 7.5 and CuDNN 5.1.3

Steps modified from https://gist.github.com/erikbern/78ba519b97b440e10640

# Need Java 1.8 for bazel, on 14.04 this is a big mess - 14.10 and up openjdk-8 is in the repos
wget http://ftp.br.debian.org/debian/pool/main/libj/libjpeg-turbo/libjpeg62-turbo_1.3.1-12_amd64.deb
wget http://ftp.br.debian.org/debian/pool/main/libj/libjpeg6b/libjpeg62_6b2-2_amd64.deb
sudo dpkg --install --recursive --auto-deconfigure libjpeg62-turbo_1.3.1-12_amd64.deb
sudo apt-get update

sudo apt-get install debian-keyring debian-archive-keyring

sudo sh -c 'echo "deb http://httpredir.debian.org/debian/ jessie-backports main" >> /etc/apt/sources.list.d/debian-jessie-backports.list'

sudo sh -c 'echo "Package: *" >> /etc/apt/preferences.d/debian-jessie-backports'

sudo sh -c 'echo "Pin: release o=Debian,a=jessie-backports" >> /etc/apt/preferences.d/debian-jessie-backports'

sudo sh -c 'echo "Pin-Priority: -200" >> /etc/apt/preferences.d/debian-jessie-backports'

sudo apt-get -t jessie-backports install openjdk-8-jre openjdk-8-jre-headless openjdk-8-jdk-headless openjdk-8-jdk

# Set both to Java 8
sudo update-alternatives --config java
sudo update-alternatives --config javac

# Now for bazel
git clone https://github.com/bazelbuild/bazel
cd bazel

git checkout tags/0.2.1
./compile.sh
sudo cp output/bazel /usr/bin

# Tensorflow
git clone --recurse-submodules https://github.com/tensorflow/tensorflow
git checkout r0.10

export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64"
export CUDA_HOME=/usr/local/cuda
TF_UNOFFICIAL_SETTING=1 ./configure
bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
pip install wheel
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg

pip install /tmp/tensorflow_pkg/tensorflow-0.10.0-cp27-none-linux_x86_64.whl

Get CuDNN from command line

Check the Dockerfile https://github.com/NVIDIA/nvidia-docker/blob/master/centos-7/cuda/7.5/runtime/cudnn5/Dockerfile

Magic fix for GPU issues in Tensorflow

E tensorflow/stream_executor/cuda/cuda_driver.cc:491] failed call to cuInit: CUDA_ERROR_UNKNOWN
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:185] libcuda reported version is: Not found: was unable to find libcuda.so DSO loaded into this program

I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:347] driver version file contents: """NVRM version: NVIDIA UNIX x86_64 Kernel Module  367.35  Mon Jul 11 23:14:21 PDT 2016
GCC version:  gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3)
"""
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] kernel reported version is: 367.35.0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:81] No GPU devices available on machine.

This error can be fixed by

sudo apt-get install nvidia-modprobe


See issue https://github.com/tensorflow/tensorflow/issues/394

Screen bindings to start paned session and rebind pane focus

Put the following lines in ~/.screenrc

bind j focus down                                                            
bind k focus up                                                              
bind l focus right                                                          
bind h focus left                                                            
                                                                                 
layout new                                                                    
split -v                                                                      
screen 0                                                                      
focus right                                                                  
split                                                                        
screen 1                                                                      
focus down

SOCKS proxy with chrome and SSH

In the particular case of an ssh proxy running, connected to something like ssh -D 4321 blah.com

google-chrome --proxy-server="socks5://127.0.0.1:4321" --host-resolver-rules="MAP * 0.0.0.0 , EXCLUDE 127.0.0.1"

OSX

 /Applications/Google\ Chrome.app/Contents/MacOSogle\ Chrome --proxy-server="socks5://127.0.0.1:4321" --host-resolver-rules="MAP * 0.0.0.0 , EXCLUDE 127.0.0.1"

Skype on Ubuntu 14.04

sudo dpkg --add-architecture i386
sudo sh -c "echo 'deb http://archive.canonical.com/ trusty partner' >> /etc/apt/sources.list.d/canonical_partner.list"
sudo apt-get update
sudo apt-get install skype
LD_PRELOAD=/usr/lib/i386-linux-gnu/mesa/libGL.so.1 skype

Install xvfb Ubuntu

sudo apt-get install xvfb x11-xkb-utils xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic x11-apps

xvfb-run app

 xvfb-run -a python get_lyrics.py http://lyrics.wikia.com/wiki/Des%27ree:Save_Me

 xvfb-run -a -e /dev/stdout python -u get_lyrics.py http://lyrics.wikia.com/wiki/Des%27ree:Save_Me 2>&1

Install MATLAB Ubuntu 14.04

sudo apt-get install libxtst6:i386
sudo apt-get install lib32z1
sudo apt-get install gcc-multilib

sudo apt-get install python-software-properties
sudo apt-get install python-software-common
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java6-installer

sudo apt-get install libxtst6:i386 libXext6:i386 libxi6:i386 libncurses5:i386 libxt6:i386 libxpm4:i386 libxmu6:i386 libxp6:i386 lib32stdc++6

sudo ln -s /lib/x86_64-linux-gnu/libc.so.6 /lib
Edit /etc/udev/rules.d/70-persistent-net.rules and change the name to eth0

Install headless bittorrent client

apt-get source bittornado 

I set up aliases to btdownloadcurses.py and btdownloadmany.py

btdownloadcurses --max_uploads 4 --max_upload_rate 32 to_be_downloaded.torrent

 
btdownloadmany --max_uploads 4 --max_upload_rate 32 .

where . is a directory full of torrent files. 

Update to latest NVIDIA drivers Ubuntu

Best:
http://askubuntu.com/questions/672047/anyone-has-successfully-installed-cuda-7-5-on-ubuntu-14-04-3-lts-x86-64

sudo add-apt-repository ppa:ubuntu-x-swat/x-updates
sudo apt-get update
sudo apt-get install nvidia-current

http://www.ubuntuupdates.org/ppa/ubuntu-x-swat

Rate limited recusrsive wget

wget -r -np --limit-rate=10m -nH -A.mp3 http://servername.com/mp3/

Streaming NES (.NSF) file audio over a network using gstreamer

Now, to check on alsa
cat /proc/asound/cards should show some available cards

alsamixer should also work if you run it from the commandline. If alsamixer gives the following error:
cannot open mixer: No such file or directory

You may need to add yourself to the audio group in /etc/group, then log out and back in to update permissions.

First, we test remote piping of sound.

On the listening machine (Macbook Air for me, so using sox vs. aplay as you would/could in Linux):
nc -l 8000 | sox -traw -r44100 -b16 -e unsigned-integer - -tcoreaudio

On the source machine (Ubuntu 12.04 for me):
cat /dev/urandom | nc ip_addr_of_listening_machine 8000

If you hear white noise, the connection between machines seems to work correctly.

Next, we want to set up gstreamer

sudo apt-get install gstreamer0.10*

Check that we can play the .nsf files from gstreamer (where ${file} is the file to play)

gst-launch-0.10 -v filesrc location="${file}" ! nsfdec ! pulsesink 

You should hear the game audio of your choice. If this doesn't work, you may need to install libgme-dev by doing:

sudo apt-get install libgme-dev

Now you can try a file over the network.

gst-launch-0.10 -v filesrc location="${file}" ! nsfdec ! audioconvert ! wavenc ! udpsink port=8888

You can then connect to this port with netcat and listen to the wavenc encoded data (basically just raw PCM, if you missed the header).

nc sourceserver 8888 | sox -traw -r44100 -b16 -e unsigned-ineger -tcoreaudio

Capturing audio output with gstreamer

To figure out which output gstreamer's pulsesink will be using, and try to capture it, check

pactl list | grep -A2 "Source #"

Look for a monitor device which seems to correspond to speakers in this list - in my example this is "alsa_output.pci-0000_00_14.2.analog-stereo.monitor", shown as source #1 in my list.

Now catch the output using gstreamer, and save to some filename

gst-launch -e pulsesrc device="alsa_output.pci-0000_00_14.2.analog-stereo.monitor" ! audioconvert ! wavenc ! filesink location=test_wav.wav

You can listen to this file (if something was playing through xmms2 already), and verify that this gstreamer command is catching the right output device. It took me a few tries to figure out which monitor output was the right one.

This filename dump can also be a fifo, which means that the output can also be used with netcat.

First, on the listening machine
nc -l 8000 > test_wav.wav

Next, on the streamer

mkfifo streamer

gst-launch -e pulsesrc device="alsa_output.pci-0000_00_14.2.analog-stereo.monitor" ! audioconvert ! wavenc ! filesink location=streamer

then in another terminal, in the same directory

nc ip_addr_of_listening 8000 < streamer

Installing xmms2 with NSF support on Ubuntu

sudo apt-get install xmms2
sudo apt-get install xmms2-plugins-*
sudo apt-get install libgme-dev

kill the currently running xmms2d process with kill, then run
xmms2-launcher

To see if things are working, add files (for example .nsf files) and play them

xmms2 add *.nsf
xmms2 list
xmms2 play

Convert many images with imagemagick

ls -1 *.png | xargs -n 1 bash -c 'convert "$0" "${0%.*}.jpg"'

find . -name \*.jpg | xargs -n 1 bash -c 'convert "$0" "${0%.*}.png"' 

Copy files from another directory when "argument list too long"

find ../COCO/images/train2014/ -name '*.jpg' -exec cp -pr {} . \;