OpenFOAM 2.3.0 on Stampede

The OpenFOAM (Open Field Operation and Manipulation) Toolbox is a free, open source Computational Fluid Dynamics (CFD) software package providing an extensive range of features, from solving complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.

This document instructs Stampede cluster users how to:

Running OpenFOAM on Stampede

TACC staff has built and installed OpenFOAM 2.3.0 using Intel compiler 14.0.1.106 and mvapich2 2.0b on Stampede. Please follow the steps below to set up the environment and run a OpenFOAM tutorial example.

Set Up the Computing Environment

Set up the proper computing environment by loading the appropriate Intel compiler and loading the OpenFoam 2.3.0 module.

  • Assuming you have the default compiler loaded, issue the following commands:

      login1$ module swap intel intel/14.0.1.106
      login1$ module load openfoam/2.3.0
  • OpenFOAM places its cases in the directory defined by the FOAM_RUN environment variable. TACC staff suggests using your $WORK file space to run OpenFOAM cases. The following commands set this environment variable and create the FOAM_RUN directory under the user's $WORK directory.

      login1$ export FOAM_RUN=$WORK/My_OpenFOAM/2.3.0/run
      login1$ mkdir -p $FOAM_RUN

Run Tutorials

The following demonstrates running the OpenFOAM's Lid-driven Cavity Flow test case involving isothermal, incompressible flow in a two-dimensional square domain.

  • Copy OpenFOAM's tutorials into the OpenFOAM work directory created above, then change to the cavity case directory:

      login1$ cp -r $FOAM_TUTORIALS $FOAM_RUN 
      login1$ cd $FOAM_RUN/tutorials/incompressible/icoFoam/cavity
  • Use the pre-processing tool, blockMesh, to generate the mesh. Then run the icoFoam solver.

    Do NOT run these commands on the login nodes. You may either submit a batch job, or start an interactive session using TACC's idev utility or SLURM's srun command.

      login1$ idev
      ...
      c557-804$ blockMesh
      c557-804$ icoFoam
      c557-804$ exit
      login1$

Building OpenFOAM on Stampede

The need for compiling an executable arises from most OpenFOAM users defining their own OpenFOAM cases. Cases are placed in a specific directory inside the OpenFOAM build tree by default, meaning users must have the entire writeable tree within their own directories.

Please follow these instructions precisely. These instructions may be used to build different versions of OpenFOAM, however there may be variations in the build process. Please submit a TACC support ticket upon encountering difficulties.

Build Recommendations

TACC staff recommends using the Intel 14 (or newer) compiler along with mvapich2 2.0b. The Intel 14 compiler helps to avoid C++ compiler errors, and the only available MPI with Intel 14 is mvapich2 2.0b. Furthermore, using mvapich2 2.0b helps avoid ptmalloc errors when making use of python wrappers.

Build Steps

  1. Download and unzip the OpenFOAM source code. We advise you to use TACC's $WORK filesystem to compile.
     login1$ cdw
     login1$ mkdir OpenFOAM; cd OpenFOAM
     login1$ wget http://downloads.sourceforge.net/project/foam/foam/2.3.0/ThirdParty-2.3.0.tgz
     login1$ wget http://downloads.sourceforge.net/project/foam/foam/2.3.0/OpenFOAM-2.3.0.tgz
     login1$ tar zxf OpenFOAM-2.3.0.tgz
     login1$ tar zxf ThirdParty-2.3.0.tgz
  2. Edit the "OpenFOAM-2.3.0/etc/bashrc" file and change the following environment variables defined in this file.

    1. line 45:
      45 foamInstall=$HOME/$WM_PROJECT
      to
      45 foamInstall=$WORK/$WM_PROJECT
    2. line 66:
      66 export WM_COMPILER=Gcc
      to
      66 export WM_COMPILER=Icc
    3. line 84:
      84 export WM_MPLIB=OPENMPI
      to
      84 export WM_MPLIB=MV2MPI
  3. Edit the "OpenFOAM-2.3.0/etc/config/settings.sh" file and change the following environment variables:

    1. Change the default compilers on lines 89-90
       89 export WM_CC='gcc'
       90 export WM_CXX='g++'
      to
       89 export WM_CC='mpicc'
       90 export WM_CXX='mpicxx'
    2. Change the compiler and linker flags on lines 90-92:
       91 export WM_CFLAGS='-m64 -fPIC'
       92 export WM_CXXFLAGS='-m64 -fPIC'
       93 export WM_LDFLAGS='-m64'
      to
       91 export WM_CFLAGS='-O3 -fPIC'
       92 export WM_CXXFLAGS='-O3 -fPIC'
       93 export WM_LDFLAGS='-O3'
    3. add the MV2MPI case we defined in step 1 after line 379:
       379 case "$WM_MPLIB" in
       380 MV2MPI)
       381   export FOAM_MPI=mpi
       382   export MPI_ARCH_PATH="${MPICH_HOME}"
       383 ;;
       384
       385 SYSTEMOPENMPI)
  4. Edit the "OpenFOAM-2.3.0/wmake/rules/linux64Icc/c" file and change line 5:
    5 cc= icc
    to
    5 cc= mpicc
  5. Edit the "OpenFOAM-2.3.0/wmake/rules/linux64Icc/c++" file and change line 5:
    5 CC= icpc -std=c++0x
    to
    5 CC= mpicxx -std=c++0x
  6. Set and export the following environment variables (bash examples)
    login1$ export WM_CC=mpicc; export WM_CXX=mpicxx
  7. Source the "etc/bashrc" file edited in Step 1:
    login1$ . etc/bashrc
  8. Start the build and take a 4-5 hour break
    login1$ ./Allwmake &> build_log

The build process places executables in the user's $WORK/OpenFOAM/OpenFOAM-2.3.0/platforms/linux64IccDPOpt/bin directory. To test build success, ensure this directory contains the desired executables:

login1$ which blockMesh
login1$ which icoFoam

Caveats and Troubleshooting

  • If the source fails to build, search the "build_log" (created in step 8) for errors. In particular watch out for MPI or scotch errors.

    login1$ grep -i error build_log
  • Occasionally users will see errors related to the inclusion of "mpi.h" within an "extern C" block e.g., line 125 of "parallel/decompose/ptscotchDecomp/ptscotchDecomp.C"

      extern "C" {
      #include <stdio.h>
      #include "mpi.h"
      #include "ptscotch.h"
      }

Move the "include "mpi.h"" line outside the extern C scope to remedy this error.

  • Should you wish to add or remove features to the custom OpenFOAM installation or switch to different compilers/mpi, you must clean up the build tree manually as there is no "make clean" equivalent for OpenFOAM.

    1. Delete the platforms/linux64* directories in both OpenFOAM and ThirdParty build trees
    2. To reset everything
       login1$ cdw; cd OpenFOAM
       login1$ find ./* -iname "*.o" |xargs rm -fr
    3. OpenFOAM requires a long list of environment variables to control the build and execution process. The commands…
       login1$ printenv | grep -i wm
       login1$ printenv | grep -i foam

      …will show you these variables.

      It is generally possible to reset these variables by sourcing "/etc/bashrc" a second time. You can, of course, exit the shell to start over. Alternatively, you can reset or unset variables manually.

Visualization with OpenFOAM and ParaView

OpenFOAM's visualization tool, ParaFOAM, does NOT work on Stampede. Instead, users may use the post-processing ParaView tool to visualize OpenFOAM data. ParaView is an open-source, multi-platform data analysis and visualization application.

References

Last update: December 07, 2017