View
216
Download
0
Category
Preview:
Citation preview
8/13/2019 brzobohaty
1/65
OpenFOAMopen source CFD on ANSELM
Tom Brzobohat
Supercomputing for industry SC4IndustryResearch Programme 3 - Libraries for Parallel Computing, VB-TU Ostrava
8/13/2019 brzobohaty
2/65
open source CFD on ANSELM
Outline
OpenFOAM on ANSELM- Running applications
- Running applications in parallel
- Compiling applications and libraries
Benchmarks
OpenFOAM
- Introduction
- Features
- OpenFOAM Case
8/13/2019 brzobohaty
3/65
open source CFD on ANSELM
OpenFOAM - Introduction
What OpenFOAM is?
The OpenFOAM (Open Field Operation and Manipulation) CFD Toolbox based on C++.
OpenFOAM is free and open source software, released under theGNU General Public License.
OpenFOAM includes over 80 solver applications that simulate specific problems in engineering mechanics and over 170 utility
applications that perform pre- and post-processing tasks, e.g. meshing, data visualization, etc.
Friendly syntax for partial differential equations
Unstructured polyhedral grid capabilities
Automatic parallelization of applications written using OpenFOAM
high-level syntax
Commercial support and training provided by the developers
8/13/2019 brzobohaty
4/65
open source CFD on ANSELM
OpenFOAM - Introduction
OpenFOAM syntax for tensor operations and partial differential equations
For example the equation is represented by the code:
Solve(
fvm::ddt(rho,U)+ fvm::div(phi,U)
- fvm::laplacian(mu,U)==
- fvc::grad(p));
+ - = p
8/13/2019 brzobohaty
5/65
open source CFD on ANSELM
OpenFOAM - Introduction
Where we can find info?
Homepage - http://www.openfoam.com/
Wiki - http://openfoamwiki.net/
CFD Forum - http://www.cfd-online.com/Forums/openfoam/
History
The original development of OpenFOAM started in the late 1980s at Imperial College, London.
The FOAM, was sold by UK company Nabla Ltd. before being released as open source in 2004.
On 15 August 2011, OpenCFD announced its acquisition by Silicon Graphics International(SGI).
On September 12, 2012, the ESI Group announced the acquisition of OpenFOAM Ltd from SGI.
8/13/2019 brzobohaty
6/65
open source CFD on ANSELM
OpenFOAM
Features
8/13/2019 brzobohaty
7/65
open source CFD on ANSELM
OpenFOAM - Features
adjointShapeOptimizationFoam
icoFoam
MRFSimpleFoam
nonNewtonianIcoFoam
pimpleDyMFoam
pimpleFoam
pisoFoam
porousSimpleFoam
simpleFoam
SRFSimpleFoam
Incompressible flows
8/13/2019 brzobohaty
8/65
open source CFD on ANSELM
OpenFOAM - Features
adjointShapeOptimizationFoam
icoFoam
MRFSimpleFoam
nonNewtonianIcoFoam
pimpleDyMFoam
pimpleFoam
pisoFoam
porousSimpleFoam
simpleFoam
SRFSimpleFoam
Incompressible flows
Multiphase flows
bubbleFoam
cavitatingFoam
compressibleInterFoam
interFoam
interDyMFoam
interMixingFoam
interPhaseChangeFoam
LTSInterFoam
MRFInterFoam
MRFMultiphaseInterFoam
multiphaseInterFoam
porousInterFoam
settlingFoam
twoLiquidMixingFoam
twoPhaseEulerFoam
simpleFoam
SRFSimpleFoam
8/13/2019 brzobohaty
9/65
open source CFD on ANSELM
OpenFOAM - Features
rhoCentralFoam
rhoCentralDyMFoam
rhoPimpleFoam
rhoPorousMRFLTSPimpleFoam
rhoPorousSimpleFoam
rhoPorousMRFPimpleFoam
rhoSimplecFoam
rhoSimpleFoam
sonicDyMFoam
sonicLiquidFoam
Compressible flows
8/13/2019 brzobohaty
10/65
open source CFD on ANSELM
OpenFOAM - Features
chemFoam
coldEngineFoam
dieselEngineFoam
dieselFoam
engineFoam
fireFoam
PDRFoam
reactingFoam
rhoReactingFoam
XiFoam
Combustion
rhoCentralFoam
rhoCentralDyMFoam
rhoPimpleFoam
rhoPorousMRFLTSPimpleFoam
rhoPorousMRFSimpleFoam
rhoPorousMRFPimpleFoam
rhoSimplecFoam
rhoSimpleFoam
sonicDyMFoam
sonicLiquidFoam
Compressible flows
8/13/2019 brzobohaty
11/65
open source CFD on ANSELM
OpenFOAM - Features
chemFoam
coldEngineFoam
dieselEngineFoam
dieselFoam
engineFoam
fireFoam
PDRFoam
reactingFoam
rhoReactingFoam
XiFoam
Combustion
Conjugate Heat TransferbuoyantBaffleSimpleFoam
buoyantBoussinesqPimpleFoam
buoyantBoussinesqSimpleFoam
buoyantPimpleFoam
buoyantSimpleFoam
chtMultiRegionFoam
rhoCentralFoam
rhoCentralDyMFoam
rhoPimpleFoam
rhoPorousMRFLTSPimpleFoam
rhoPorousMRFSimpleFoam
rhoPorousMRFPimpleFoam
rhoSimplecFoam
rhoSimpleFoam
sonicDyMFoam
sonicLiquidFoam
Compressible flows
8/13/2019 brzobohaty
12/65
open source CFD on ANSELM
OpenFOAM - Features
RAS Turbulence Models
laminar
kEpsilon
kOmega
kOmegaSST
RNGkEpsilon
NonlinearKEShih
LienCubicKE
qZeta
LaunderSharmaKE
LamBremhorstKE
LienCubicKELowRe
LienLeschzinerLowRe
LaunderGibsonRSTM
realizableKE
SpalartAllmaras
8/13/2019 brzobohaty
13/65
open source CFD on ANSELM
OpenFOAM - Features
RAS Turbulence Models
laminar
kEpsilon
kOmega
kOmegaSST
RNGkEpsilon
NonlinearKEShih
LienCubicKE
qZeta
LaunderSharmaKE
LamBremhorstKE
LienCubicKELowRe
LienLeschzinerLowRe
LaunderGibsonRSTM
realizableKE
SpalartAllmaras
LES Turbulence Modelslaminar
Smagorinsky
Smagorinsky2
dynSmagorinsky
dynLagrangian
scaleSimilarity
mixedSmagorinsky
dynMixedSmagorinsky
kOmegaSSTSAS
oneEqEddy
dynOneEqEddy
locDynOneEqEddy
spectEddyVisc
LRDDiffStress
DeardorffDiffStress
SpalartAllmaras
SpalartAllmarasDDES
SpalartAllmarasIDDES
8/13/2019 brzobohaty
14/65
open source CFD on ANSELM
OpenFOAM - FeaturesTransport models
NewtonianCrossPowerLaw
BirdCarreau
HerschelBulkley
powerLaw
interfaceProperties
8/13/2019 brzobohaty
15/65
open source CFD on ANSELM
OpenFOAM - FeaturesTransport models
NewtonianCrossPowerLaw
BirdCarreau
HerschelBulkley
powerLaw
interfaceProperties
ThermoPropertiesicoPolynomial
perfectGas
hConstThermo
janafThermo
specieThermo
constTransport
polynomialTransport
sutherlandTransport
hPolynomialThermo
8/13/2019 brzobohaty
16/65
open source CFD on ANSELM
OpenFOAM - FeaturesTransport models
NewtonianCrossPowerLaw
BirdCarreau
HerschelBulkley
powerLaw
interfaceProperties
Radiation ModelsP1
fvDOM
viewFactor
ThermoPropertiesicoPolynomial
perfectGas
hConstThermo
janafThermo
specieThermo
constTransport
polynomialTransport
sutherlandTransport
hPolynomialThermo
8/13/2019 brzobohaty
17/65
open source CFD on ANSELM
OpenFOAM - FeaturesTransport models
NewtonianCrossPowerLaw
BirdCarreau
HerschelBulkley
powerLaw
interfaceProperties
Radiation ModelsP1
fvDOM
viewFactor
hPsiMixtureThermo
hsPsiMixtureThermo
hRhoMixtureThermo
hsRhoMixtureThermo
homogeneousMixture
inhomogeneousMixture
veryInhomogeneousMixture
dieselMixture
basicMultiComponentMixture
multiComponentMixture
reactingMixture
egrMixture
Reaction Models
ThermoPropertiesicoPolynomial
perfectGas
hConstThermo
janafThermo
specieThermo
constTransport
polynomialTransport
sutherlandTransport
hPolynomialThermo
8/13/2019 brzobohaty
18/65
open source CFD on ANSELM
OpenFOAM - FeaturesMesh Generation
blockMeshextrudeMesh
extrude2DMesh
extrudeToRegionMesh
snappyHexMesh
8/13/2019 brzobohaty
19/65
open source CFD on ANSELM
OpenFOAM - FeaturesMesh Generation
blockMeshextrudeMesh
extrude2DMesh
extrudeToRegionMesh
snappyHexMesh
Mesh ManipulationcheckMesh
mergeMeshes
moveMesh
polyDualMesh
refineMesh
renumberMesh
splitMesh
transformPoints
autoRefineMesh
modifyMesh
refineHexMesh
refinementLevel
refineWallLayer
8/13/2019 brzobohaty
20/65
open source CFD on ANSELM
OpenFOAM - FeaturesMesh Generation
blockMeshextrudeMesh
extrude2DMesh
extrudeToRegionMesh
snappyHexMesh
Mesh Conversion
ansysToFoamcfx4ToFoam
fluentMeshToFoam
foamMeshToFluent
foamToStarMesh
gambitToFoamgmshToFoam
ideasUnvToFoam
mshToFoam
netgenNeutralToFoam
plot3dToFoamstar4ToFoam
tetgenToFoam
writeMeshObj
Mesh ManipulationcheckMesh
mergeMeshes
moveMesh
polyDualMesh
refineMesh
renumberMesh
splitMesh
transformPoints
autoRefineMesh
modifyMesh
refineHexMesh
refinementLevel
refineWallLayer
8/13/2019 brzobohaty
21/65
open source CFD on ANSELM
OpenFOAM - Features
Pre-Processing
applyBoundaryLayer
applyWallBoundaryConditions
changeDictionary
mapFields
setFields
viewFactorsGen
8/13/2019 brzobohaty
22/65
open source CFD on ANSELM
OpenFOAM - Features
Pre-Processing
applyBoundaryLayer
applyWallBoundaryConditions
changeDictionary
mapFields
setFields
viewFactorsGen
Post-ProcessingfoamDataToFluent
foamToEnsight
foamToEnsightParts
foamToGMV
foamToTecplot360
foamToVTK
Co
Lambda2
Mach
8/13/2019 brzobohaty
23/65
open source CFD on ANSELM
OpenFOAM case
case
system
controlDict
fvSchemes
fvSolution
decomposeParDict
constant
properties
polyMesh
Points
Cells
Faces
boundary
time directories > 0
0
processors*
OpenFOAM case
Constant- contains a full description of the case mesh in a
subdirectory polyMeshand files specifying physical properties for
the application concerned, e.g.transportProperties.
System- for setting parameters associated with the solution
procedure itself.
Time directories - containing individual files of data for
particular fields
Processors- directories are named processorN
where N=0,1 represents a processor number and contains a
time directory, containing the decomposed field descriptions, and
a constant/polyMesh directory containing the decomposed mesh
description.
8/13/2019 brzobohaty
24/65
open source CFD on ANSELM
OpenFOAM on ANSELM
8/13/2019 brzobohaty
25/65
open source CFD on ANSELM
------------------- /opt/modules/modulefiles/engineering --------------------ansys/14.5.x matlab/R2013a-EDUcomsol/43b-COM openfoam/2.2.1-gcc481-openmpi1.6.5-DPcomsol/43b-EDU openfoam/2.2.1-gcc481-openmpi1.6.5-SPlsdyna/7.x.x openfoam/2.2.1-icc-openmpi1.6.5-DPmatlab/R2013a-COM
Available engineering modules
[brz021@login2.anselm ~]$ module avail
[brz021@login2.anselm ~]$ module load openfoam/2.2.1-icc-openmpi1.6.5-DP
[brz021@login2.anselm ~]$ source $FOAM_BASHRC
To create OpenFOAM environment on ANSELM give the commands:
8/13/2019 brzobohaty
26/65
open source CFD on ANSELM
Getting Started
Create a project directory within the $HOME/OpenFOAMdirectory named -2.2.1and create a directory named run within it, e.g. by
typing:
[brz021@login2.anselm ~]$ mkdir -p $FOAM_RUN
[brz021@login2.anselm ~]$ cd /home/brz021/OpenFOAM/brz021-2.2.1/run
Project directory is now available by typing
or
[brz021@login2.anselm ~]$ cd $FOAM_RUN
Copy the tutorial examples directory in the OpenFOAM distribution to the run directory
[brz021@login2.anselm ~]$ cp -r $FOAM_TUTORIALS $FOAM_RUN
Run the first example case of incompressible laminar flow in a cavity:
[brz021@login2.anselm ~]$ cd $FOAM_RUN/tutorials/incompressible/icoFoam/cavity[brz021@login2.anselm ~]$ blockMesh[brz021@login2.anselm ~]$ icoFoam
8/13/2019 brzobohaty
27/65
open source CFD on ANSELM
Create a Bash script:
#!/bin/bashmodule load openfoam/2.2.1-icc-openmpi1.6.5-DPsource $FOAM_BASHRC# source to run functions. $WM_PROJECT_DIR/bin/tools/RunFunctions
cd$FOAM_RUN/tutorials/incompressible/icoFoam/cavityrunApplication blockMeshrunApplication icoFoam
pokus.sh
Running Serial Applications
Job Submission
$ qsub -A OPEN-0-0 -q qprod -l select=1:ncpus=1,walltime=03:00:00 pokus.sh
8/13/2019 brzobohaty
28/65
open source CFD on ANSELM
OpenFOAM on ANSELM
Running applications in parallel
8/13/2019 brzobohaty
29/65
open source CFD on ANSELM
#!/bin/bash#PBS N motorBike#PBS l select=2:ncpus=16#PBS l walltime=01:00:00#PBS q qprod#PBS A IT4I-1-4
module load openfoam/2.2.1-icc-openmpi1.6.5-DPsource $FOAM_BASHRCcd$FOAM_RUN/tutorials/incompressible/simpleFoam/motorBikenproc = 32
mpirun hostfile ${PBS_NODEFILE} np $nproc snappyHexMeshoverwrite parallel | tee snappyHexMesh.log
mpirun hostfile ${PBS_NODEFILE} np $nproc potentialFoam-noFunctionObject-writep parallel | tee potentialFoam.log
mpirun hostfile ${PBS_NODEFILE} np $nproc simpleFoamparallel | tee simpleFoam.log
pokusParallel.pbs
Running applications in parallel
8/13/2019 brzobohaty
30/65
open source CFD on ANSELM
Running applications in parallel
case
system
controlDict
fvSchemes
fvSolution
decomposeParDict
constant
properties
polyMesh
Points
Cells
Faces
boundary
time directories > 0
0
processors*
decomposeParDict
The mesh and fields are decomposed using
the decomposePar utility.
8/13/2019 brzobohaty
31/65
open source CFD on ANSELM
numberOfSubdomains 32;
method simple;
simpleCoeffs{
n ( 4 4 2 );delta 0.001;
}
hierarchicalCoeffs{
n ( 1 1 1 );delta 0.001;
order xyz;}
manualCoeffs{
dataFile "";}
decomposeParDict
Running applications in parallel
8/13/2019 brzobohaty
32/65
open source CFD on ANSELM
Running applications in parallel
simple- simple geometric decomposition in which the domain is split into pieces by
direction
hierarchical- hierarchical geometric decomposition which is the same
as simple except the user specifies the order in which the directional split is done
scotch- scotch decomposition which requires no geometric input from the user and
attempts to minimise the number of processor boundaries.
manual - manual decomposition, where the user directly specifies the allocation of
each cell to a particular processor.
8/13/2019 brzobohaty
33/65
open source CFD on ANSELM
Create a Bash script:
#!/bin/bashmodule load openfoam/2.2.1-icc-openmpi1.6.5-DPsource $FOAM_BASHRC# source to run functions. $WM_PROJECT_DIR/bin/tools/RunFunctions
cd$FOAM_RUN/tutorials/incompressible/simpleFoam/motorBike
runApplication blockMeshrunApplication decomposePar
pokus.sh
Job Submission
$ qsub -A OPEN-0-0 -q qprod -l select=1:ncpus=1,walltime=03:00:00 pokus.sh
Running applications in parallel
$ qsub pokusParallel.pbs
CFD ANSELM
8/13/2019 brzobohaty
34/65
open source CFD on ANSELM
OpenFOAM on ANSELM
Compile an own solvers
CFD ANSELM
8/13/2019 brzobohaty
35/65
open source CFD on ANSELM
Initialize OpenFOAM environment before compiling your solver
$ cd $FOAM_RUN/
Compile an own solver
$ mkdir p applications/solvers$ cd applications/solvers
$ cpr $FOAM_SOLVERS/incompressible/icoFoam/ My_icoFoam
$ cd My_icoFoam
$ mv icoFoam.C My_icoFoam.C
Create directory applications/solvers in user directory
Copy icoFoam solvers source files
Rename icoFoam.C to My_icoFOAM.C
[brz021@login2.anselm ~]$ module load openfoam/2.2.1-icc-openmpi1.6.5-DP
[brz021@login2.anselm ~]$ source $FOAM_BASHRC
CFD ANSELM
8/13/2019 brzobohaty
36/65
open source CFD on ANSELM
Compile an own solver
$ wmake
My_icoFoam.C
EXE = $(FOAM_USER_APPBIN)/My_icoFoam
icoFoam.C
EXE = $(FOAM_APPBIN)/icoFoam
Edit filesfile in Make directory:
In directory My_icoFoam give the compilation command:
CFD ANSELM
8/13/2019 brzobohaty
37/65
open source CFD on ANSELM
BENCHMARKS
CFD ANSELM
8/13/2019 brzobohaty
38/65
Boeing 787 Dreamliner
open source CFD on ANSELM
Benchmarks
open source CFD on ANSELM
8/13/2019 brzobohaty
39/65
open source CFD on ANSELM
Benchmarks
Boeing 787 Dreamliner
106.2 M Cells
637.2 M Unknowns
Mesh generated by
snappyHexMesh
simpleFoam
k-omegaSST
2000 time step iter.
solver parametersp - GAMG
U - smoothSolver
k - smoothSolver
omega - smoothSolver
Tolerance 1e-7
relativeTolerance 0.1
open source CFD on ANSELM
8/13/2019 brzobohaty
40/65
open source CFD on ANSELM
Benchmarks
Boeing 787 Dreamliner
open source CFD on ANSELM
8/13/2019 brzobohaty
41/65
Lamborghini Aventador
open source CFD on ANSELM
Benchmarks
open source CFD on ANSELM
8/13/2019 brzobohaty
42/65
Lamborghini Aventadoropen source CFD on ANSELM
Benchmarks
225.3 M Cells
1351.8 M Unknowns
Mesh generated by
snappyHexMesh
simpleFoam
k-omegaSST
2000 time step iter.
solver parametersp - GAMG
U - smoothSolver
k - smoothSolver
omega - smoothSolver
Tolerance 1e-8
relativeTolerance 0.001
open source CFD on ANSELM
8/13/2019 brzobohaty
43/65
Lamborghini Aventadoropen source CFD on ANSELM
Benchmarks
open source CFD on ANSELM
8/13/2019 brzobohaty
44/65
Hydraulic Valveopen source CFD on ANSELM
Benchmarks
52.3 M Cells
Mesh generated by
ANSYS Workbench
simpleFoam
k-omegaSST
1000 time step iter.
open source CFD on ANSELM
8/13/2019 brzobohaty
45/65
Hydraulic Valveopen source CFD on ANSELM
Benchmarks
open source CFD on ANSELM
8/13/2019 brzobohaty
46/65
Hydraulic Valveopen source CFD on ANSELM
Benchmarks
3264
256
0
500
1000
1500
2000
6 14 52
Time[s]
N Cells [M]
open source CFD on ANSELM
8/13/2019 brzobohaty
47/65
Thermal Sensorp
Benchmarks
5.3 M Cells
Mesh generated by
ANSYS Workbench
chtMultiRegionSimpleFoam
k-epsilon
1000 time step iter.
11 regions
open source CFD on ANSELM
8/13/2019 brzobohaty
48/65
Thermal Sensorp
Benchmarks
open source CFD on ANSELM
8/13/2019 brzobohaty
49/65
Lampp
Benchmarks
17.3 M Cells
Mesh generated by
ANSYS Workbench
chtMultiRegionSimpleFoam
fvDOM Radiation Model
laminar
20000 time step iter.
6 Regions
open source CFD on ANSELM
8/13/2019 brzobohaty
50/65
Lampp
Benchmarks
open source CFD on ANSELM
8/13/2019 brzobohaty
51/65
Lamborghini Aventadorp
Benchmarks
Transient solver - pisoFoam
23 M Cells
nCorrectors 2
k-omegaSST
solver parametersp - PCG
U - PBiCG
k - PBiCG
omega - PBiCG
open source CFD on ANSELM
8/13/2019 brzobohaty
52/65
Lamborghini AventadorBenchmarks
20 000 time steps
solution time - 40 hr
256 cores
open source CFD on ANSELM
8/13/2019 brzobohaty
53/65
VSB Campus Benchmarks
open source CFD on ANSELM
8/13/2019 brzobohaty
54/65
VSB Campus Benchmarks
18 M Cells
Mesh generated by
snappyHexMesh
simpleFoam
k-epsilon
2000 time step iter.
open source CFD on ANSELM
8/13/2019 brzobohaty
55/65
VSB Campus Benchmarks
open source CFD on ANSELM
8/13/2019 brzobohaty
56/65
snappyHexMesh
open source CFD on ANSELM
8/13/2019 brzobohaty
57/65
Stonavka - Ole Benchmarks
open source CFD on ANSELM
8/13/2019 brzobohaty
58/65
Stonavka - Ole Benchmarks
open source CFD on ANSELM
8/13/2019 brzobohaty
59/65
snappyHexMesh
VSB Campus
Benchmarks
0,01
0,51
1,01
1,51
0 20 40 60 80 100 120 140
open source CFD on ANSELM
8/13/2019 brzobohaty
60/65
Benchmarks
Boeing 787 Dreamliner
N cores 256 512 1024Time [h] 1.5 2 1.9
snappyHexMesh scalability
open source CFD on ANSELM
8/13/2019 brzobohaty
61/65
Lamborghini AventadorBenchmarks
N cores 28 256 512 1024Time [h] 3.7 2.5 3.1 3.7
snappyHexMesh scalability
open source CFD on ANSELM
8/13/2019 brzobohaty
62/65
Supermarine SpitfireBenchmarks
open source CFD on ANSELM
8/13/2019 brzobohaty
63/65
VW BeetleBenchmarks
open source CFD on ANSELM
h k
8/13/2019 brzobohaty
64/65
Other Open SourceBenchmarks
Elmer
Code Aster
Code Saturne
Palabos
SU 2
benchmarket.it4i.cz open source CFD on ANSELMh k
8/13/2019 brzobohaty
65/65
Benchmarks
Recommended