CompSciWeek14-15
Contents
Installation Woes
- Python packages (ex: sympy)
- GNU Autotools + make-based process (ex: FFTW)
- Cmake: a developer-friendly (not user-friendly?) alternative (ex: cgal)
- Non-standard makes (ex: NAMD2 and NWChem)
- Tinkering with open-source
- The DL on Software Licenses
- Apache, BSD, GPL, Microsoft, FDL, Creative Commons
- The open-source that isn't: Canvas
The single most important idea for compiling and installing new software is to remember that the installation works for the developer's environment, and it will for you, too if your environment is setup correctly. Often times this is easier said than done.
- Package Dependencies (pdftk depends on libgcj)
- and versions of those packages - this is usually the worst part
- Shell variables
- PATH, CFLAGS, LDFLAGS, LD_LIBRARY_PATH
- Compiler version
- Machine architecture
Tips:
To find a package owning a file (on linux systems with rpm)
rpm -qf /usr/lib64/libfftw3.so.3
To find all files associated with a package
rpm -ql fftw-3.2.1-3.1.el6.x86_64
Example 1
Installing pdftk (developer Makefile)
mkdir ~/build && cd ~/build wget https://www.pdflabs.com/tools/pdftk-the-pdf-toolkit/pdftk-2.02-src.zip
Unpacking the zip file can (and often does) clobber files in your current directory, so it's always safest to unpack it in a new directory:
mkdir pdftk && cd pdftk unzip ../pdftk-2.02-src.zip ls
In this case, pdftk was nice enough to put all their files in a subfolder, so we can get rid of the temp directory.
cd .. mv pdftk/pdftk-2.02-src . rmdir pdftk
If you read the license information, you'll note that pdftk is covered by a GPL copyright. This means you have the freedom to modify and re-distribute the source (but not to change the license). The GPL is sometimes called a 'copyleft', since it is a clever hack on copyright perpetrated by Richard Stallman, the founder of the GNU project.
Make is a small language that lists the shell commands required to 'make' the 'targets'. A make target can be anything, but usually is the name of an executable file (or a file at an intermediate step) that will be built during the compilation process. Pdftk has installation instructions on its website, which outline a make-based process commonly seen for development on small projects. pdftk is large-ish because the pdf file format can do a lot, but the build process is small because it is based on java. Some standard options to make are
make -f Makefile.Redhat -j 8 -C $PWD install
The '-j 8' flag tells make to run in parallel on 8 processors. The '-C' flag tells make what directory to run make from. Here, it's for illustration, since the default is to run in the current directory. The name 'install' is the target to build. It is not a real file, but instead just has a list of commands that copy the compiled program into the filesystem. The great thing about make is that it knows what order to make all the files in.
In this instance, make will build the executable, pdftk, but will not install it - since it tries to copy it into /usr/local/bin. You should read Makefile.Redhat to find the install target. You'll notice that it's not there, but at the bottom the Makefile includes Makefile.Base. Looking at that file, you will find the install target. It is run with a single line that installs (carefully copies) pdftk from the current directory into /usr/local/bin. make install will work if you change /usr/local/bin to $(HOME)/bin. You may also consider installing the manpage (pdftk.1 from the main pdftk directory).
cd .. cp pdftk.1 $HOME/share/man/man1 export MANPATH=$MANPATH:$HOME/share/man
Example 2
Installing fftw (autotools)
A quick search turns up the main page for fftw, the Fastest Fourier Transform in the West.
mkdir ~/build && cd ~/build wget ftp://ftp.fftw.org/pub/fftw/fftw-3.3.4.tar.gz
This is a tar (tape archive) that has been gzipped (usually given the endearing name 'tarball'). You can read the contents with
tar tzf fftw-3.3.4.tar.gz | less tar xzf fftw-3.3.4.tar.gz
The first command lists the contents. It is really helpful to make sure that all the files in the tarball are in their own directory. Most tarballs should be packed this way. Ones that aren't will put files in your current directory, and may clobber other files you had there before.
cd fftw-3.3.4 less README
The second line is absolutely critical, since almost every package has some installation quirk that you won't know about until it's too late -- or until you read the instructions, either way. You'll notice this file says configure, make, make install. This means the package uses gnu autotools!
Autotools are a standard set of installation scripts that follow the general install commands above. The configure part is the most important, since it gives you a chance to put in details about your own environment. In addition to reading all install instructions, you should also run
./configure --help
This will give you a list of options that tell the compile process what compilers to use and where to find libraries and include files that the package may well depend on. The standard ones (CFLAGS, etc.) are shown in the graphic at the top of this page.
./configure --with-pic --enable-float --prefix=$HOME
The second part of autotools is (sometimes GNU) make, covered above.
make -j4 install
Example 3
Installing gromacs (CMAKE). The latest versions of gromacs use cmake - a new build tool created to replace autotools (configure). cmake uses two main files for its work:
CMakeLists.txt CMakeCache.txt
CMakeLists.txt is written by the package developers to control the build process. It typically introduces a list of required packages, calls some functions to test whether they are on the system and how to get to them (the configure process), and then lists a set of build targets. Unlike the make process, editing this file is unwise, since it mostly contains information on dependencies (which break the package if you change them).
Sadly, cmake abstracts away all the good environment variables, getting rid of your ability to directly tweak the build process. Instead, you have a perplexing array of variables that can offer more control over the process. This comes at the expense of usually not knowing how or what variables need to be changed when you have a problem. Hopefully the developers left some documentation to clear up this point. Gromacs is good at this, and lets you know the major variables to change - one influencing the location of the fftw library and the other influencing whether or not to compile with MPI.
To install gromacs, download and unpack the source as before. I'm working with gromacs-5.0-beta2 now, but you may find a different version when you read this. Next make sure your environment variables reference the compiler you intend to use. We have a module on rcslurm for the intel compiler,
module load compilers/intel/2013sp1_cluster_xe
Then generate CMakeCache.txt with the command
cmake .
This makes the cache file, which is the perplexing list of build variables referenced above. In a large project, running cmake . may produce CMakeCache.txt files in several sub-directories as well.
find . -name CMakeCache.txt
If you're unlucky, cmake will end with a version mismatch error, in which case you have to go out and get the latest version of cmake. We have a few on the rcslurm cluster, which you can load with, e.g.
module load apps/cmake/2.8.12.2
There's also an interactive way to look at the most important variables used during cmake's build. It's
ccmake .
This will come up with a list, analogous to ./configure --help with autotools. If all goes well with the cmake process, you can proceed with the usual make process
make -j8 make test make install
A word of warning is in order. Gromacs is highly optimized, and uses information specific to the machine it is compiled on. Rcslurm has different hardware (amd vs intel processors) on the head node than the cluster nodes -- leading to an "Illegal instruction" error when you compile on one and run on the other. To get around this, you have to compile on a node
rm -f CMakeCache.txt srun --ntasks-per-node 8 -N 1 --pty /bin/bash cmake . make -j8 make install
With gromacs, you will want to compile an MPI version for the cluster and a non-mpi version for the head node (to create run-files and do simple analysis).
Once you figure out the relevant cmake variables, you can save them as a build script. I've started doing this for all my builds, since it helps me keep track of what I've tried. The relevant one is here:
<source lang="bash"> module load compilers/intel/2013sp1_cluster_xe mpi/openmpi/1.6.1 apps/cmake/2.8.12.2
cmake -DCMAKE_INSTALL_PREFIX=/shares/rogers \
-DFFTW_INCLUDE_DIR=/shares/rogers/include \
-DFFTW_LIBRARY=/shares/rogers/lib \
-DGMX_MPI=ON \
.
make -j8 make install </source>
Fourier Transforms
Read through the page on Fourier transforms.
The Monte Carlo Method
- Integrals of the form: <math>\int \frac{f(x)}{g(x)} g(x) dx</math>
- Computing pi
- Parallelizing with MPI4Py
Complicated integrals are common in the hard sciences. Most of these have analytical solutions or good approximations (see [1] and the NIST Digital Library of Mathematical Functions (DLMF)). Unfortunately, sometimes there is no analytical solution. The next best thing is to use quadrature. This is just summing the function computed at a specific set of points times a specific set of weights. Some background and the relevant scipy functions are here. There is also a great page of points and weights on John Burkardt's page at FSU.
For integrals over high-dimensional spaces, and for those closely connected to probability distributions, the next next best thing is to use the Monte Carlo Method. This writes the integral <math>\int f(x) dx</math> as <math>\int \frac{f(x)}{g(x)} g(x) dx = \lim_{N\to\infty} \frac{1}{N} \sum_i \frac{f(x_i)}{g(x_i)}</math> where the points <math>x_i</math> are generated from a probability distribution, <math>P(x_i) = g(x_i)</math>
Common probability distributions are the Gaussian, <math>P(x) = (2\pi \sigma^2)^{-1/2} \exp{-\frac{(x-\mu)^2}{2\sigma^2}}</math>
and the uniform distribution <math>P(x) = 1/L, x \in (0,L)</math> (among others).
With that introduction, we can now approximate the value of <math>\pi</math> as the result of <math>\pi/4 = \int_0^1\int_0^1 (x^2 + y^1 < 1) \; dx\; dy </math> The function is one when the point x,y is inside the circle and zero otherwise. Physically, we are randomly tossing a point on the unit square and adding 1 to a total if it also happens to fall within the upper right quadrant of the unit circle.
The code is dead simple: <source lang="python"> from numpy.random import random
I = 0 for i in xrange(10000):
x = random() y = random() I += x*x + y*y < 1.0
print float(I)*4e-4 </source>
Numpy's random() function draws samples from the interval <math>(0,1)</math> with equal probability. Notice that it does not give pi exactly. It converges to pi in the limit of a large number of samples.
Parallelizing this one is easy, since each processor can run a series of Monte Carlo trials, and the only communication between the processors is averaging the final results. If we ignore that last step for now, we can run the program in parallel without any changes just by saving it to a file, say calc_pi.py and running it with mpirun
mpirun -np 4 `which python` calc_pi.py
Running may require that you save that command to a sh file, sandwiched between batch queue commands and run with, e.g. sbatch run.sh. You'll notice that all 8 programs independently run all the commands. Although all the print commands eventually find their way back to your terminal, they don't arrive at the same time (one of the most well-known perils of parallel programming). This is how MPI works - all processors run the same source program. You can get each processor to do something different by using MPI library calls.
<source lang="python">
from mpi4py import MPI
comm=MPI.COMM_WORLD size=comm.size rank=comm.rank # This is a unique number for each process name=MPI.Get_processor_name()
if rank == 0:
print "Tap dancing"
elif rank == 1:
print "Shopping @ store"
elif rank == 2:
print "Cooking roast beef"
else:
print "Staying home"
s = comm.allreduce(rank, op=MPI.SUM) if rank == 0:
print "Cumulative Node Number Sum = %d"%(s)
</source>
The last line calls MPI's allreduce, which combines values from all nodes and sends the result to all nodes. In this case the sum of all ranks is computed (which should be size(size-1)/2). Pairing the above script with the last one (and summing the Monte Carlo trials) gives an MPI program for estimating the value of pi.
