7/28/2019 Once Fallas en Simulacin

1/3

Eleven Critical Pitfalls in Simulation Modelingby

Averill M. Law, Ph.D.

The following material is a synopsis of some of the ideas presented in my three-daysimulation short courses.

Pitfall Number 1: Failure to have a well-defined set of objectives at the beginning

of the simulation study.

We recommend making a list of the specific questions that the model is to address and

also the performance measures that will be used to evaluate the efficacy of various

system configurations. Otherwise, it will be impossible to determine the appropriate

level of model detail.

Pitfall Number 2: Failure to communicate with the decision-maker (or the client)

on a regular basis.

This is essential to ensure that the correct problem is solved and to promote model

credibility. There are many valid (technically sound) models that are not used in thedecision-making process because they are not credible.

Pitfall Number 3: Lack of knowledge of simulation methodology and also of

probability and statistics.

A significant percentage of the people involved in simulation modeling are only trainedin how to use a particular simulation software package, which we feel is definitely not

sufficient. Most experts in simulation modeling would agree that "programming" of the

model represents only 25 to 50 percent of a sound simulation study. The simulation

analyst must also be knowledgeable in simulation methodology (validating a model,

selecting input probability distributions, designing and analyzing simulation

experiments, etc.) and also probability and statistics (probability distributions,confidence intervals, etc.).

Pitfall Number 4: Inappropriate level of model detail.

A very common pitfall for beginning simulation analysts is to have an excessive level of

model detail. We recommend starting "with a moderately detailed" model which is

embellished as needed. The adequacy of a particular version of the model is determined

in part by having the model reviewed by "subject-matter experts" and by the decision-

maker (or client). Modeling each aspect of the system will seldom be required to make

effective decisions, and will also be infeasible due to time, money, or computer

constraints.

7/28/2019 Once Fallas en Simulacin

2/3

Pitfall Number 5: Failure to collect good system data.

If one is modeling an existing system, it is important to collect data on key system

random variables. (For a manufacturing system, key random variables would probably

include times to failure and times to repair for each machine.) Often this is not donebecause of project time constraints or because the simulation analyst does not realize

that this is an important consideration.

Pitfall Number 6: Belief that so-called easy-to-use simulation packages require a

significantly lower level of technical competence.

Some people believe that a so-called "easy-to-use" simulation package will makeperforming a simulation study a much easier task. This type of software can reduce the

time to "program" a model for problems of modest complexity. However, for most real-

world problems, programming in some form is required. Furthermore, the simulation

modeler will still have to be concerned with formulating the problem, collecting and

analyzing data, validating the model, modeling system randomness, designing and

analyzing simulation experiments, and managing the overall simulation project. These

activities require a significant amount of technical competence and experience.

Pitfall Number 7: Blindly using simulation software without understanding its

underlying assumptions.

To facilitate ease of use, simulation-software vendors have added to their softwarepowerful "macro" blocks (or modeling constructs) that model a significant part of a

real-world system. However, these blocks are often not well documented, possibly

resulting in the development of an invalid model. For example, one encounters many

types of conveyors in practice, yet some simulation products only offer a few conveyor

options.

Pitfall Number 8: Misuse of animation.

Animation is useful for communicating the essence of a simulation model to decision-

makers (who may not understand all of its technical details), for debugging simulation

computer programs, and for suggesting improved operational procedures for a system.

However, the efficacy of a particular system design should be decided by applying

appropriate statistical procedures to carefully designed simulation experiments. Justbecause a "short" run of the animated simulation model seems okay, this does not mean

that the model is either debugged or valid.

Pitfall Number 9: Replacing a probability distribution by its mean.

A common (but unfortunate) practice in simulation modeling is to represent a source of

system randomness by the perceived mean value rather than its correspondingprobability distribution. For example, consider a single-server queueing system where

the mean interarrival time and mean service time are 1 minute and 0.99 minute,

respectively. Suppose that the interarrival times and the service times actually each havean exponential distribution. Then the long-run average number in queue is

7/28/2019 Once Fallas en Simulacin

3/3

approximately 98. Suppose, on the other hand, that a simulation analyst takes every

interarrival time to be a constant 1 minute and every service time to be a constant 0.99

minute. Then no customer ever waits in the queue! Thus, it is not sufficient to just get

the mean correct we also have to represent variability in an appropriate way.

Pitfall number 10: Using an inappropriate probability distribution.

It is important to model each source of system randomness by an appropriate probability

distribution. For example, many simulation practitioners represent the time to do some

task by a normal distribution (a symmetric distribution). However, we have never seen atask-time data set that was actually normally distributed. In practice, most histograms

have a longer right tail (positive skewness). Consider the single-server queueing systemfrom Pitfall Number 9. If one models the service-time distribution by a symmetric

distribution when, in fact, it is positively skewed, then the average number in queuemay be significantly underestimated.

Pitfall Number 11: Failure to perform a proper output-data analysis.

A stochastic simulation model does not produce the true performance measures for the

model it only produces statistical estimates of them. A simulation analyst must

properly choose the simulation run length, the length of the warmup period (if one is

appropriate), and the number of independent model replications (each using different

random numbers). We recommend that confidence intervals be constructed for

important performance measures. Note, however, that this can not be easily done using

the data from one simulation run, since these data will not be independent (an

assumption of classical statistics). It is also not possible to get legitimate standard

deviation (or variance) estimates from one simulation run, yet a number of simulationsoftware products provide these automatically.