brzobohaty

8/13/2019 brzobohaty

1/65

OpenFOAMopen source CFD on ANSELM

Tom Brzobohat

Supercomputing for industry SC4IndustryResearch Programme 3 - Libraries for Parallel Computing, VB-TU Ostrava


2/65

open source CFD on ANSELM

Outline

OpenFOAM on ANSELM- Running applications

- Running applications in parallel

- Compiling applications and libraries

Benchmarks

OpenFOAM

- Introduction

- Features

- OpenFOAM Case


3/65


OpenFOAM - Introduction

What OpenFOAM is?

The OpenFOAM (Open Field Operation and Manipulation) CFD Toolbox based on C++.

OpenFOAM is free and open source software, released under theGNU General Public License.

OpenFOAM includes over 80 solver applications that simulate specific problems in engineering mechanics and over 170 utility

applications that perform pre- and post-processing tasks, e.g. meshing, data visualization, etc.

Friendly syntax for partial differential equations

Unstructured polyhedral grid capabilities

Automatic parallelization of applications written using OpenFOAM

high-level syntax

Commercial support and training provided by the developers


4/65



OpenFOAM syntax for tensor operations and partial differential equations

For example the equation is represented by the code:

Solve(

fvm::ddt(rho,U)+ fvm::div(phi,U)

- fvm::laplacian(mu,U)==

- fvc::grad(p));

+ - = p


5/65



Where we can find info?

Homepage - http://www.openfoam.com/

Wiki - http://openfoamwiki.net/

CFD Forum - http://www.cfd-online.com/Forums/openfoam/

History

The original development of OpenFOAM started in the late 1980s at Imperial College, London.

The FOAM, was sold by UK company Nabla Ltd. before being released as open source in 2004.

On 15 August 2011, OpenCFD announced its acquisition by Silicon Graphics International(SGI).

On September 12, 2012, the ESI Group announced the acquisition of OpenFOAM Ltd from SGI.


6/65


OpenFOAM

Features


7/65


OpenFOAM - Features

adjointShapeOptimizationFoam

icoFoam

MRFSimpleFoam

nonNewtonianIcoFoam

pimpleDyMFoam

pimpleFoam

pisoFoam

porousSimpleFoam

simpleFoam

SRFSimpleFoam

Incompressible flows


8/65


OpenFOAM - Features

adjointShapeOptimizationFoam

icoFoam

MRFSimpleFoam

nonNewtonianIcoFoam

pimpleDyMFoam

pimpleFoam

pisoFoam

porousSimpleFoam

simpleFoam

SRFSimpleFoam

Incompressible flows

Multiphase flows

bubbleFoam

cavitatingFoam

compressibleInterFoam

interFoam

interDyMFoam

interMixingFoam

interPhaseChangeFoam

LTSInterFoam

MRFInterFoam

MRFMultiphaseInterFoam

multiphaseInterFoam

porousInterFoam

settlingFoam

twoLiquidMixingFoam

twoPhaseEulerFoam

simpleFoam

SRFSimpleFoam


9/65


OpenFOAM - Features

rhoCentralFoam

rhoCentralDyMFoam

rhoPimpleFoam

rhoPorousMRFLTSPimpleFoam

rhoPorousSimpleFoam

rhoPorousMRFPimpleFoam

rhoSimplecFoam

rhoSimpleFoam

sonicDyMFoam

sonicLiquidFoam

Compressible flows


10/65


OpenFOAM - Features

chemFoam

coldEngineFoam

dieselEngineFoam

dieselFoam

engineFoam

fireFoam

PDRFoam

reactingFoam

rhoReactingFoam

XiFoam

Combustion

rhoCentralFoam

rhoCentralDyMFoam

rhoPimpleFoam


rhoPorousMRFSimpleFoam


rhoSimplecFoam

rhoSimpleFoam

sonicDyMFoam

sonicLiquidFoam

Compressible flows


11/65


OpenFOAM - Features

chemFoam

coldEngineFoam

dieselEngineFoam

dieselFoam

engineFoam

fireFoam

PDRFoam

reactingFoam

rhoReactingFoam

XiFoam

Combustion

Conjugate Heat TransferbuoyantBaffleSimpleFoam

buoyantBoussinesqPimpleFoam

buoyantBoussinesqSimpleFoam

buoyantPimpleFoam

buoyantSimpleFoam

chtMultiRegionFoam

rhoCentralFoam

rhoCentralDyMFoam

rhoPimpleFoam


rhoPorousMRFSimpleFoam


rhoSimplecFoam

rhoSimpleFoam

sonicDyMFoam

sonicLiquidFoam

Compressible flows


12/65


OpenFOAM - Features

RAS Turbulence Models

laminar

kEpsilon

kOmega

kOmegaSST

RNGkEpsilon

NonlinearKEShih

LienCubicKE

qZeta

LaunderSharmaKE

LamBremhorstKE

LienCubicKELowRe

LienLeschzinerLowRe

LaunderGibsonRSTM

realizableKE

SpalartAllmaras


13/65


OpenFOAM - Features

RAS Turbulence Models

laminar

kEpsilon

kOmega

kOmegaSST

RNGkEpsilon

NonlinearKEShih

LienCubicKE

qZeta

LaunderSharmaKE

LamBremhorstKE

LienCubicKELowRe

LienLeschzinerLowRe

LaunderGibsonRSTM

realizableKE

SpalartAllmaras

LES Turbulence Modelslaminar

Smagorinsky

Smagorinsky2

dynSmagorinsky

dynLagrangian

scaleSimilarity

mixedSmagorinsky

dynMixedSmagorinsky

kOmegaSSTSAS

oneEqEddy

dynOneEqEddy

locDynOneEqEddy

spectEddyVisc

LRDDiffStress

DeardorffDiffStress

SpalartAllmaras

SpalartAllmarasDDES

SpalartAllmarasIDDES


14/65


OpenFOAM - FeaturesTransport models

NewtonianCrossPowerLaw

BirdCarreau

HerschelBulkley

powerLaw

interfaceProperties


15/65




BirdCarreau

HerschelBulkley

powerLaw

interfaceProperties

ThermoPropertiesicoPolynomial

perfectGas

hConstThermo

janafThermo

specieThermo

constTransport

polynomialTransport

sutherlandTransport

hPolynomialThermo


16/65




BirdCarreau

HerschelBulkley

powerLaw

interfaceProperties

Radiation ModelsP1

fvDOM

viewFactor


perfectGas

hConstThermo

janafThermo

specieThermo

constTransport

polynomialTransport

sutherlandTransport

hPolynomialThermo


17/65




BirdCarreau

HerschelBulkley

powerLaw

interfaceProperties

Radiation ModelsP1

fvDOM

viewFactor

hPsiMixtureThermo

hsPsiMixtureThermo

hRhoMixtureThermo

hsRhoMixtureThermo

homogeneousMixture

inhomogeneousMixture

veryInhomogeneousMixture

dieselMixture

basicMultiComponentMixture

multiComponentMixture

reactingMixture

egrMixture

Reaction Models


perfectGas

hConstThermo

janafThermo

specieThermo

constTransport

polynomialTransport

sutherlandTransport

hPolynomialThermo


18/65


OpenFOAM - FeaturesMesh Generation

blockMeshextrudeMesh

extrude2DMesh

extrudeToRegionMesh

snappyHexMesh


19/65




extrude2DMesh

extrudeToRegionMesh

snappyHexMesh

Mesh ManipulationcheckMesh

mergeMeshes

moveMesh

polyDualMesh

refineMesh

renumberMesh

splitMesh

transformPoints

autoRefineMesh

modifyMesh

refineHexMesh

refinementLevel

refineWallLayer


20/65




extrude2DMesh

extrudeToRegionMesh

snappyHexMesh

Mesh Conversion

ansysToFoamcfx4ToFoam

fluentMeshToFoam

foamMeshToFluent

foamToStarMesh

gambitToFoamgmshToFoam

ideasUnvToFoam

mshToFoam

netgenNeutralToFoam

plot3dToFoamstar4ToFoam

tetgenToFoam

writeMeshObj

Mesh ManipulationcheckMesh

mergeMeshes

moveMesh

polyDualMesh

refineMesh

renumberMesh

splitMesh

transformPoints

autoRefineMesh

modifyMesh

refineHexMesh

refinementLevel

refineWallLayer


21/65


OpenFOAM - Features

Pre-Processing

applyBoundaryLayer

applyWallBoundaryConditions

changeDictionary

mapFields

setFields

viewFactorsGen


22/65


OpenFOAM - Features

Pre-Processing

applyBoundaryLayer

applyWallBoundaryConditions

changeDictionary

mapFields

setFields

viewFactorsGen

Post-ProcessingfoamDataToFluent

foamToEnsight

foamToEnsightParts

foamToGMV

foamToTecplot360

foamToVTK

Co

Lambda2

Mach


23/65


OpenFOAM case

case

system

controlDict

fvSchemes

fvSolution

decomposeParDict

constant

properties

polyMesh

Points

Cells

Faces

boundary

time directories > 0

0

processors*

OpenFOAM case

Constant- contains a full description of the case mesh in a

subdirectory polyMeshand files specifying physical properties for

the application concerned, e.g.transportProperties.

System- for setting parameters associated with the solution

procedure itself.

Time directories - containing individual files of data for

particular fields

Processors- directories are named processorN

where N=0,1 represents a processor number and contains a

time directory, containing the decomposed field descriptions, and

a constant/polyMesh directory containing the decomposed mesh

description.


24/65


OpenFOAM on ANSELM


25/65


------------------- /opt/modules/modulefiles/engineering --------------------ansys/14.5.x matlab/R2013a-EDUcomsol/43b-COM openfoam/2.2.1-gcc481-openmpi1.6.5-DPcomsol/43b-EDU openfoam/2.2.1-gcc481-openmpi1.6.5-SPlsdyna/7.x.x openfoam/2.2.1-icc-openmpi1.6.5-DPmatlab/R2013a-COM

Available engineering modules

[[email protected] ~]$ module avail

[[email protected] ~]$ module load openfoam/2.2.1-icc-openmpi1.6.5-DP

[[email protected] ~]$ source $FOAM_BASHRC

To create OpenFOAM environment on ANSELM give the commands:


26/65


Getting Started

Create a project directory within the $HOME/OpenFOAMdirectory named -2.2.1and create a directory named run within it, e.g. by

typing:

[[email protected] ~]$ mkdir -p $FOAM_RUN

[[email protected] ~]$ cd /home/brz021/OpenFOAM/brz021-2.2.1/run

Project directory is now available by typing

or

[[email protected] ~]$ cd $FOAM_RUN

Copy the tutorial examples directory in the OpenFOAM distribution to the run directory

[[email protected] ~]$ cp -r $FOAM_TUTORIALS $FOAM_RUN

Run the first example case of incompressible laminar flow in a cavity:

[[email protected] ~]$ cd $FOAM_RUN/tutorials/incompressible/icoFoam/cavity[[email protected] ~]$ blockMesh[[email protected] ~]$ icoFoam


27/65


Create a Bash script:

#!/bin/bashmodule load openfoam/2.2.1-icc-openmpi1.6.5-DPsource $FOAM_BASHRC# source to run functions. $WM_PROJECT_DIR/bin/tools/RunFunctions

cd$FOAM_RUN/tutorials/incompressible/icoFoam/cavityrunApplication blockMeshrunApplication icoFoam

pokus.sh

Running Serial Applications

Job Submission

$ qsub -A OPEN-0-0 -q qprod -l select=1:ncpus=1,walltime=03:00:00 pokus.sh


28/65


OpenFOAM on ANSELM

Running applications in parallel


29/65


#!/bin/bash#PBS N motorBike#PBS l select=2:ncpus=16#PBS l walltime=01:00:00#PBS q qprod#PBS A IT4I-1-4

module load openfoam/2.2.1-icc-openmpi1.6.5-DPsource $FOAM_BASHRCcd$FOAM_RUN/tutorials/incompressible/simpleFoam/motorBikenproc = 32

mpirun hostfile ${PBS_NODEFILE} np $nproc snappyHexMeshoverwrite parallel | tee snappyHexMesh.log

mpirun hostfile ${PBS_NODEFILE} np $nproc potentialFoam-noFunctionObject-writep parallel | tee potentialFoam.log

mpirun hostfile ${PBS_NODEFILE} np $nproc simpleFoamparallel | tee simpleFoam.log

pokusParallel.pbs



30/65



case

system

controlDict

fvSchemes

fvSolution

decomposeParDict

constant

properties

polyMesh

Points

Cells

Faces

boundary

time directories > 0

0

processors*

decomposeParDict

The mesh and fields are decomposed using

the decomposePar utility.


31/65


numberOfSubdomains 32;

method simple;

simpleCoeffs{

n ( 4 4 2 );delta 0.001;

}

hierarchicalCoeffs{

n ( 1 1 1 );delta 0.001;

order xyz;}

manualCoeffs{

dataFile "";}

decomposeParDict



32/65



simple- simple geometric decomposition in which the domain is split into pieces by

direction

hierarchical- hierarchical geometric decomposition which is the same

as simple except the user specifies the order in which the directional split is done

scotch- scotch decomposition which requires no geometric input from the user and

attempts to minimise the number of processor boundaries.

manual - manual decomposition, where the user directly specifies the allocation of

each cell to a particular processor.


33/65


Create a Bash script:

#!/bin/bashmodule load openfoam/2.2.1-icc-openmpi1.6.5-DPsource $FOAM_BASHRC# source to run functions. $WM_PROJECT_DIR/bin/tools/RunFunctions

cd$FOAM_RUN/tutorials/incompressible/simpleFoam/motorBike

runApplication blockMeshrunApplication decomposePar

pokus.sh

Job Submission

$ qsub -A OPEN-0-0 -q qprod -l select=1:ncpus=1,walltime=03:00:00 pokus.sh


$ qsub pokusParallel.pbs

CFD ANSELM


34/65


OpenFOAM on ANSELM

Compile an own solvers

CFD ANSELM


35/65


Initialize OpenFOAM environment before compiling your solver

$ cd $FOAM_RUN/

Compile an own solver

$ mkdir p applications/solvers$ cd applications/solvers

$ cpr $FOAM_SOLVERS/incompressible/icoFoam/ My_icoFoam

$ cd My_icoFoam

$ mv icoFoam.C My_icoFoam.C

Create directory applications/solvers in user directory

Copy icoFoam solvers source files

Rename icoFoam.C to My_icoFOAM.C

[[email protected] ~]$ module load openfoam/2.2.1-icc-openmpi1.6.5-DP

[[email protected] ~]$ source $FOAM_BASHRC

CFD ANSELM


36/65


Compile an own solver

$ wmake

My_icoFoam.C

EXE = $(FOAM_USER_APPBIN)/My_icoFoam

icoFoam.C

EXE = $(FOAM_APPBIN)/icoFoam

Edit filesfile in Make directory:

In directory My_icoFoam give the compilation command:

CFD ANSELM


37/65


BENCHMARKS

CFD ANSELM


38/65

Boeing 787 Dreamliner


Benchmarks



39/65


Benchmarks


106.2 M Cells

637.2 M Unknowns

Mesh generated by

snappyHexMesh

simpleFoam

k-omegaSST

2000 time step iter.

solver parametersp - GAMG

U - smoothSolver

k - smoothSolver

omega - smoothSolver

Tolerance 1e-7

relativeTolerance 0.1



40/65


Benchmarks




41/65

Lamborghini Aventador


Benchmarks



42/65

Lamborghini Aventadoropen source CFD on ANSELM

Benchmarks

225.3 M Cells

1351.8 M Unknowns

Mesh generated by

snappyHexMesh

simpleFoam

k-omegaSST


solver parametersp - GAMG

U - smoothSolver

k - smoothSolver

omega - smoothSolver

Tolerance 1e-8

relativeTolerance 0.001



43/65

Lamborghini Aventadoropen source CFD on ANSELM

Benchmarks



44/65

Hydraulic Valveopen source CFD on ANSELM

Benchmarks

52.3 M Cells

Mesh generated by

ANSYS Workbench

simpleFoam

k-omegaSST




45/65


Benchmarks



46/65


Benchmarks

3264

256

0

500

1000

1500

2000

6 14 52

Time[s]

N Cells [M]



47/65

Thermal Sensorp

Benchmarks

5.3 M Cells

Mesh generated by

ANSYS Workbench

chtMultiRegionSimpleFoam

k-epsilon


11 regions



48/65

Thermal Sensorp

Benchmarks



49/65

Lampp

Benchmarks

17.3 M Cells

Mesh generated by

ANSYS Workbench

chtMultiRegionSimpleFoam

fvDOM Radiation Model

laminar


6 Regions



50/65

Lampp

Benchmarks



51/65

Lamborghini Aventadorp

Benchmarks

Transient solver - pisoFoam

23 M Cells

nCorrectors 2

k-omegaSST

solver parametersp - PCG

U - PBiCG

k - PBiCG

omega - PBiCG



52/65

Lamborghini AventadorBenchmarks

20 000 time steps

solution time - 40 hr

256 cores



53/65

VSB Campus Benchmarks



54/65


18 M Cells

Mesh generated by

snappyHexMesh

simpleFoam

k-epsilon




55/65




56/65

snappyHexMesh



57/65

Stonavka - Ole Benchmarks



58/65

Stonavka - Ole Benchmarks



59/65

snappyHexMesh

VSB Campus

Benchmarks

0,01

0,51

1,01

1,51

0 20 40 60 80 100 120 140



60/65

Benchmarks


N cores 256 512 1024Time [h] 1.5 2 1.9

snappyHexMesh scalability



61/65

Lamborghini AventadorBenchmarks

N cores 28 256 512 1024Time [h] 3.7 2.5 3.1 3.7

snappyHexMesh scalability



62/65

Supermarine SpitfireBenchmarks



63/65

VW BeetleBenchmarks


h k


64/65

Other Open SourceBenchmarks

Elmer

Code Aster

Code Saturne

Palabos

SU 2

benchmarket.it4i.cz open source CFD on ANSELMh k


65/65

Benchmarks

Documents

brzobohaty