OpenFOAM 2.3.0 on Stampede

The OpenFOAM (Open Field Operation and Manipulation) Toolbox is a free, open source Computational Fluid Dynamics (CFD) software package providing an extensive range of features, from solving complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.

This document instructs Stampede cluster users how to:

Run Stampede's existing installation of OpenFOAM 2.3.0
How to create one's own OpenFOAM 2.3.0 build
Tips on visualization with OpenFOAM

Running OpenFOAM on Stampede

TACC staff has built and installed OpenFOAM 2.3.0 using Intel compiler 14.0.1.106 and mvapich2 2.0b on Stampede. Please follow the steps below to set up the environment and run a OpenFOAM tutorial example.

Set Up the Computing Environment

Set up the proper computing environment by loading the appropriate Intel compiler and loading the OpenFoam 2.3.0 module.

Assuming you have the default compiler loaded, issue the following commands:

  login1$ module swap intel intel/14.0.1.106
  login1$ module load openfoam/2.3.0

OpenFOAM places its cases in the directory defined by the FOAM_RUN environment variable. TACC staff suggests using your $WORK file space to run OpenFOAM cases. The following commands set this environment variable and create the FOAM_RUN directory under the user's $WORK directory.
```
  login1$ export FOAM_RUN=$WORK/My_OpenFOAM/2.3.0/run
  login1$ mkdir -p $FOAM_RUN
```

Run Tutorials

The following demonstrates running the OpenFOAM's Lid-driven Cavity Flow test case involving isothermal, incompressible flow in a two-dimensional square domain.

Copy OpenFOAM's tutorials into the OpenFOAM work directory created above, then change to the cavity case directory:
```
  login1$ cp -r $FOAM_TUTORIALS $FOAM_RUN 
  login1$ cd $FOAM_RUN/tutorials/incompressible/icoFoam/cavity
```
Use the pre-processing tool, blockMesh, to generate the mesh. Then run the icoFoam solver.

Do NOT run these commands on the login nodes. You may either submit a batch job, or start an interactive session using TACC's idev utility or SLURM's srun command.
```
  login1$ idev
  ...
  c557-804$ blockMesh
  c557-804$ icoFoam
  c557-804$ exit
  login1$
```

Building OpenFOAM on Stampede

The need for compiling an executable arises from most OpenFOAM users defining their own OpenFOAM cases. Cases are placed in a specific directory inside the OpenFOAM build tree by default, meaning users must have the entire writeable tree within their own directories.

Please follow these instructions precisely. These instructions may be used to build different versions of OpenFOAM, however there may be variations in the build process. Please submit a TACC support ticket upon encountering difficulties.

Build Recommendations

TACC staff recommends using the Intel 14 compiler along with mvapich2 2.0b. The Intel 14 compiler helps to avoid C++ compiler errors, and the only available MPI with Intel 14 is mvapich2 2.0b. Furthermore, using mvapich2 2.0b helps avoid ptmalloc errors when making use of python wrappers.

Build Steps

Download and unzip the OpenFOAM source code. We advise you to use TACC's $WORK filesystem to compile.

 login1$ cdw
 login1$ mkdir OpenFOAM; cd OpenFOAM
 login1$ wget http://downloads.sourceforge.net/project/foam/foam/2.3.0/ThirdParty-2.3.0.tgz
 login1$ wget http://downloads.sourceforge.net/project/foam/foam/2.3.0/OpenFOAM-2.3.0.tgz
 login1$ tar zxf OpenFOAM-2.3.0.tgz
 login1$ tar zxf ThirdParty-2.3.0.tgz

Edit the "OpenFOAM-2.3.0/etc/bashrc" file and change the following environment variables defined in this file.

a. line 45:

45 foamInstall=$HOME/$WM_PROJECT

45 foamInstall=$WORK/$WM_PROJECT

b. line 66:

66 export WM_COMPILER=Gcc

66 export WM_COMPILER=Icc

c. line 84:

84 export WM_MPLIB=OPENMPI

84 export WM_MPLIB=MV2MPI

Edit the "OpenFOAM-2.3.0/etc/config/settings.sh" file and change the following environment variables:

a. Change the default compilers on lines 89-90

 89 export WM_CC='gcc'
 90 export WM_CXX='g++'

 89 export WM_CC='mpicc'
 90 export WM_CXX='mpicxx'

b. Change the compiler and linker flags on lines 90-92:

 91 export WM_CFLAGS='-m64 -fPIC'
 92 export WM_CXXFLAGS='-m64 -fPIC'
 93 export WM_LDFLAGS='-m64'

 91 export WM_CFLAGS='-O3 -fPIC'
 92 export WM_CXXFLAGS='-O3 -fPIC'
 93 export WM_LDFLAGS='-O3'

c. add the MV2MPI case we defined in step 1 after line 379:

 379 case "$WM_MPLIB" in
 380 MV2MPI)
 381   export FOAM_MPI=mpi
 382   export MPI_ARCH_PATH="${MPICH_HOME}"
 383 ;;
 384
 385 SYSTEMOPENMPI)

Edit the "OpenFOAM-2.3.0/wmake/rules/linux64Icc/c" file and change line 5:
```
5 cc= icc
```
to
```
5 cc= mpicc
```
Edit the "OpenFOAM-2.3.0/wmake/rules/linux64Icc/c++" file and change line 5:
```
5 CC= icpc -std=c++0x
```
to
```
5 CC= mpicxx -std=c++0x
```
Set and export the following environment variables (bash examples)
```
login1$ export WM_CC=mpicc; export WM_CXX=mpicxx
```
Source the "etc/bashrc" file edited in Step 1:
```
login1$ . etc/bashrc
```
Start the build and take a 4-5 hour break
```
login1$ ./Allwmake &> build_log
```

The build process places executables in the user's $WORK/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/bin directory. To test build success, ensure this directory contains the desired executables:

login1$ which blockMesh
login1$ which icoFoam

Caveats and Troubleshooting

If the source fails to build, search the "build_log" (created in step 8) for errors. In particular watch out for MPI or scotch errors.
```
login1$ grep -i error build_log
```
Occasionally users will see errors related to the inclusion of "mpi.h" within an "extern C" block e.g., line 125 of "parallel/decompose/ptscotchDecomp/ptscotchDecomp.C"
```
  extern "C" {
  #include <stdio.h>
  #include "mpi.h"
  #include "ptscotch.h"
  }
```

Move the "include "mpi.h"" line outside the extern C scope to remedy this error.

Should you wish to add or remove features to the custom OpenFOAM installation or switch to different compilers/mpi, you must clean up the build tree manually as there is no "make clean" equivalent for OpenFOAM.
1. Delete the platforms/linux64* directories in both OpenFOAM and ThirdParty build trees
2. To reset everything
```
 login1$ cdw; cd OpenFOAM
 login1$ find ./* -iname "*.o" |xargs rm -fr
```
3. OpenFOAM requires a long list of environment variables to control the build and execution process. The commands…
```
 login1$ printenv | grep -i wm
 login1$ printenv | grep -i foam
```
  …will show you these variables.
  
  It is generally possible to reset these variables by sourcing "/etc/bashrc" a second time. You can, of course, exit the shell to start over. Alternatively, you can reset or unset variables manually.

Visualization with OpenFOAM and ParaView

OpenFOAM's visualization tool, ParaFOAM, does NOT work on Stampede. Instead, users may use the post-processing ParaView tool to visualize OpenFOAM data. ParaView is an open-source, multi-platform data analysis and visualization application.

References

Last update: May 7, 2015

User Portal

TACC User Portal