FDS Application Details

EGI Activity groups

Special Interest groups

Policy groups

Virtual teams

Distributed Competence Centres

EGI Virtual teams:

Main •

Active Projects •

Closed Projects •

Guidelines

Introduction

Fire Dynamics Simulator (FDS) is a model developed by National Institute for Standard and Technology (NIST, USA) that simulates fire and predicts its effects; in the package is included Smokeview, a graphical post-processor useful for FDS results analysis. FDS solves numerically a form of Navier-Stokes equations appropriate for low-speed, thermallydriven flow with an emphasis on smoke and heat transport of fire. FDS allows modelling of fire starting from a database of standard materials distributed with the program that can be modified by the user to introduce new materials, defined with their chemical and physical characteristics and with data obtained from experimental fire tests. Dynamics of fire is simulated through the parameters that characterize each material present in the simulation domain, each one with its characteristics of in-flammability and combustion or its reaction to fire. FDS can calculate and return as output, once the simulation has been set-up to effectively calculate the quantities of interest, all the values of the variables, both scalar and vectorial, computed in every cell of the domain and useful to understand the fire and to analyse its effects (concentrations of chemical species, distributions of temperature / pressures / gas velocities / smokes, visibility, ...). FDS can be used to model the following phenomena: Low speed transport of heat and combustion products (mainly smoke) from fire; Heat transfer between the gas and solid surfaces; Pyrolysis; Fire growth; Flame spread; Activation of sprinklers and heat detectors; Fire suppression by sprinklers. FDS is widely used by fire safety professionals. One of the applications is a simulation of fire and smoke transfer in human structures, for example for design of smoke control systems and sprinkler activation studies. FDS was also used in numerous fire reconstructions including the investigation into the World Trade Centre disaster.

FDS simulations with a real scenario represent long-time, computational intensive and memory consuming jobs. In order to find the programming model exhibiting the best performance and producing authentic true results for a given fire scenario, a great number of experiments, i.e. simulation runs with various input parameters, need to be fulfilled.

Unsuppressed tunnel fires involving Heavy Goods Vehicles (HGVs) can be extremely large. The simulation of an unsuppressed fire, which could not be conducted in the tunnel, provided a means of comparing conditions in a tunnel fire with and without an active suppression system. The tunnel fire tests (described in literature) showed that very large fires (20 to 60 MW) involving HGV fuel loads create highly turbulent conditions that vary from test to test, even for apparently similar test conditions.

More information:

http://www.cfd-online.com/Wiki/FDS

http://fire.nist.gov/fds

Fire Scenario

In order to keep the computational domain within reasonable limits, only the 180m long test tunnel was included in the simulation (in the first phase). Basic layout of instrumentation in the test tunnel: We have constructed a two-lane road tunnel model with dimensions 10m x 180m x 7m with two fans located on the tunnel ceiling at the distance 50m and 140m from the left entrance of the tunnel. The fire source in the simulation was represented by burning of a flammable liquid in a pool with dimensions 2 x 3m placed in the distance 92m from the left entrance of the tunnel, 1.1m above the floor level. During the simulation, the fire did not spread along the tunnel; no other flammable obstacles were included in the simulation. The total duration of the simulation was 150s. The initial air temperature in the whole tunnel was set to 20°C. Various control devices were installed inside the tunnel in order to record mean values of gas phase quantities (soot volume fraction, visibility, temperature and carbon monoxide mass fraction) inside small testing cube-like volumes placed under the ceiling of the tunnel and at the level of human head The slices of gas temperatures, oxygen and carbon monoxide mass fractions were also recorded for several planes. The wall temperature of the tunnel ceiling was detected above the fire.

Input data required to run the model

All of the input parameters required by FDS to describe a particular scenario are conveyed via one or two text files created by the user. These files contain information about the numerical grid, ambient environment, geometry of the problem modelled, material properties, boundary conditions and the fire itself. The input file should also contain information about the desired output quantities. All blockages in the FDS model have to conform to the computational mesh. For this reason, it is usually not possible for the FDS model geometry to be exactly the same as that of the physical model (unless a very fine mesh is used). The irregular edge appearance could have been reduced by selecting a smaller cell dimension, but with a resulting rise in the number of cells and, hence, increased computational time. The grid cells can either be uniform in size (default mode) or they can be stretched in one or two of the three coordinate directions. It is possible to use more than one rectangular mesh in a calculation. This allows creation of an efficient computational domain for geometries which cannot be easily fitted into a single rectangular grid. It also allows using regions with different grid resolutions within one computational domain. Both the grid stretching and the use of multiple meshes allow the user to apply better grid resolutions in critical areas (e.g. near the fire) without unnecessarily increasing the demand for computational power by applying fine mesh to the entire computational domain. The use of multiple meshes is also required when an FDS simulation is to be run in parallel processing on more than one computer. The parallelization requires a decomposition of computational domain into computational meshes, which affect the simulation outputs. In order to use MPI version of FDS, computational domain must be decomposed into meshes and each one of them can be assigned to specific MPI process. Velocity values at the mesh boundaries are then averaged in order to maintain stability.

FDS models

FDS source code has been designed to provide 4 programming models:

1. sequential for a single CPU,
2. MPI parallel – applied on distributed memory systems: To allow executing FDS as a single parallel job on a compute cluster the MPI (Message-Passing Interface) is used. The main strategy is, that the input computational domain is broken up into multiple meshes, and then the flow field in each sub-mesh is computed as an individual MPI process on the individual computer. MPI handles the transfer of information between these processes. Usually, each mesh is assigned to its own MPI process, but it is although possible to assign multiple meshes to a single process. In this way, large meshes can be computed on dedicated processors, while smaller meshes can be grouped together in one process running on a single processor.
3. OpenMP multi-threading – applied on shared memory systems: FDS supports multi-threading implemented through the OpenMP library, which enables to employ all available processors/cores on a given machine. The number of threads need to be specified by the environment variable “OMP_NUM_THRADS”
4. a combined model MPI and OpenMP which can be applied, for example, on multi-core cluster: The combination of MPI and OpenMP approaches enables a two-level parallelization: at first the computational domain is decomposed into meshes for distributed memory, and then, within each mesh the multi-threading approach on some selected code regions is applied for shared memory.

Realization of the FDS simulation on the cluster

Several smoke transfer simulation in a road tunnel were performed using FDS and the impact of mesh decomposition of computational domain on the simulation results was under investigation. Simulations led to realistic smoke transfer behaviour and confirmed that parallel version of FDS is able to provide reasonable results.

FDS Compiling and building:

• Simulations were realized using the FDS version 5.5.3.
• Compilers GNU v.4.1.2 and 4.4.0 (gcc, gfortran, OpenMP) and the Open MPI v.1.4. were used. Note: GNU has not been supported yet in the FDS package 5.5.3, so it was necessary to create new makefiles
• We have built all of the programming models: the sequential, MPI, OpenMP and hybrid MPI+OpenMP.

Simulation input

FDS reads input parameters from a single text file which provides all the necessary information to describe the scenario. Parameters are specified by using “namelist” formatted records. There exist various third-party programs that have been developed to generate the text file containing the input parameters needed by FDS.

The computational domain representing the described fire scenario includes 12 - 25 millions rectangular mesh cells dependent on the resolution. Doing simulations with this domain consumed 8- 48 GB RAM.

Simulation output

The numerical solution to the governing equations (i.e. the user-specified output data) is written to various types of files. All output quantities must be explicitly declared (in the input file) at the start of the calculation.

Since calculations can take hours or days, there is a monitoring progress possibility (diagnostic information is written to a file) and restart feature which enables to control the stop time and the creating and dumping of restart files. A companion to FDS is a visualization program Smokeview that reads FDS output files and produces animations on the computer screen. Simulations with the described fire scenario produced output data files of amount 7-20 GB.

Simulation results

The main goal of this work was to investigate the impact of partitioning the computational domain on the reliability of simulation results, and along with we struggled to find a FDS scenario exhibiting the best performance.

Simulation support

In order to automate the process of a great number of simulation runs with a minimum of human intervention, for each of the configurations we have developed a fds manager script (written in Shell) which is responsible for the completion of the following actions:

1. It accepts and checks the input arguments specifying the FDS input file, and the required cluster configuration (the number of nodes, the number of cores, and eventually the number of MPI processes and number of threads). Arguments are passed to the subsequent operations.
2. It produces the corresponding job submission script which serves as an input to the Portable Batch System (PBS).
3. It provides for the execution of the FDS simulation using the previously generated job submission script.

To submit and start a batch job on a compute cluster the qsub command is used, to which the jobsubmission script is passed as argument. A number of options on qsub allow the specification of attributes which may affect the behaviour of the job execution. When running the pure MPI version of the FDS, MPI processes are allocated by the (default) slots strategy, where one MPI process is mapped per processor-core. When running the hybrid MPI+OpenMP version, MPI processes are enforced to be allocated so, that each MPI process is mapped per node which has a sufficient number of cores available, free allocated cores are reserved to fork the OpenMP threads.

Requirements for running FDS simulations on the cluster

Hardware requirements

FDS has been designed to run on a single computer using only one CPU, or on multiple computers and multiple banks of memory (multi-core processors, compute clusters), for operating systems: MS Windows, Mac OS X, Unix, and Linux.

For running FDS efficiently, that is to have computation times inside reasonable ranges (a couple of days), it requires:

• a computational resource as fast as possible - the CPU speed will determine how long the computation will take to finish,
• a substantial amount of RAM - it will determine how many mesh cells can be held in memory,
• a large hard disk to store the output of the calculations.

The amount of consumed RAM and storage space is highly dependent on the input scenario – the computational domain, which may include millions of mesh cells. Following some user experience, for 1 million mesh cells about 1 GB of memory seems to be enough (minimum 512MB).

Software requirements

FDS is a software package including

• the Fortran source code,
• general purpose makefiles for FDS compiling and building for various platforms, OSs, and compilers,
• tools and files developed for the verification and validation purposes.

Preliminaries for building a FDS on the given platform and OS are: Fortran 90 and C compilers, and parallel versions of FDS ask for the MPI and OpenMP libraries to be installed.

Simulation running

Experiments were carried out on the HP blade cluster located at the Institute of Informatics, Slovak Academy of Sciences, Bratislava. The cluster configuration (at that time):

• 152 cores (152*2 if considering HyperThreading)
• 16 blade nodes, each comprising of two quad-core CPU Intel Xeon X5570 @ 2.9GHz, with 8MB cache, 24-48GB of RAM, 500GB hard disk
• 6 nodes are interconnected by the Infiniband network of bandwidth 40Gbit/s per link and direction; here GNU compilers and Open MPI are installed
• 10 nodes are interconnected by the Ethernet network of bandwidth 4Gbit/s; these nodes serve also for supporting EGI VOs (ESR, VOCE)
• Disk storage: 48TB disk array

Simulation experiments were performed on 6 nodes connected by the Infiniband.

Requirements for running FDS simulations on the Grid

Hardware

• high-performance, reliable and failure-free computing resources involving a sufficient amount of memory
• a storage resource to place output results

Installed software

• Fortran 90 and C compilers including OpenMP library
• MPI and MPI-Start

Grid middleware functionalities

• the proxy credential delegation and renewal
• the submission of parallel MPI and OpenMP jobs
• monitoring the status of the submitted job
• the job cancellation
• the job output retrieval
• optionally, the job perusal (real time output retrieval) enabling the inspecting the job output in real time
• the basic data management services