Using Benchmarking Data Effectively

Large-scale studies in the chemical and petroleum refining industries identify relationships between maintenance and reliability performance and plant characteristics and practices.

Maintenance benchmarking studies have been carried out for 20 years or more, but only now, however, has it become possible to begin to develop the full potential of this type of study. This is for two reasons:

1. Plants have become more rigorous in their recording of the use of maintenance resources and of outages due to maintenance, and therefore are able to provide more comprehensive data for comparative studies.

2. Software to perform advanced statistical analysis of databases efficiently and economically is increasingly being used to analyze study data.

The result is a powerful new tool for plants to set realistic performance targets on the one hand and to discover their best current means to achieve the goals on the other.

The benchmarks referred to here are data representing the range of observed performance in studies sponsored by the Chemical Manufacturers Association (CMA) and conducted by Solomon Associates. In the maintenance function, benchmark data is typically of three types:

Reliability and maintenance performance indices. These indices depict both resource consumption and reliability results. They are presented either as quartile results (that is, the values representing the average performances of the first, second, third, and fourth 25 percent of the sample when ranked by performance), or as range charts showing the distribution of performance from the leading to the trailing performer.

Plant characteristics. Plant characteristics are important factors in determining performance. The benchmarks, which cover such factors as plant age, feedstock type, location, operating severity, and equipment complexity, are presented therefore as the average value of the plants in each quartile of performance. Range graphs additionally will show the plants characteristic ranking relative to others in the sample.

Maintenance function practices and organization factors. Physical factors are only a part of the complex interactions which influence maintenance performance. Factors such as ratio of craftsmen to planners, percent of emergency work, and type of contract are all influential and analyzed to detect relationships with performance results. These benchmarks therefore are similarly presented as performance quartile averages and range graphs.

What is maintenance?
As for any function in industry, there must exist an agreed definition of what responsibilities and activities constitute maintenance before its performance can usefully be discussed, measured, or compared. CMA and Solomon use the following definition: "Maintenance is responsible for conserving an industrial facility at its designed level of performance and for managing the necessary resources."

Conserving a facility at its designed level of performance points to the first key objective of maintenancethat is, to ensure the designed level of facility reliability. It is therefore necessary to define reliability: "Reliability is the percentage of time that a facility is mechanically available to perform as designed when operated under design conditions."

The second key objective of maintenance is obviously "to accomplish its mission at a minimum overall cost/lost margin."

Managers want to use comparative industry data on reliability and maintenance to set "best performance" targets for their plants, and that is the role of these comparative studies.

To derive credible targets, however, four dimensions must first be factored into the target-setting process. These dimensions are: Categories of maintenance, process characteristics and equipment complexity, components of overall maintenance effectiveness, and performance sustainability.

Categories of maintenance
When analyzing maintenance performance it is essential to measure separately the different categories of equipment maintenance. Different equipment categories not only are designed by different engineering functions but also are maintained by separate craft groups. At any given plant, high or low reliability and maintenance performance in one category is usually quite independent of that in another.

There are four equipment category families that cover the facilities at petroleum refining and chemical plants. They correspond broadly to the traditional grouping of maintenance technician crafts:

  • Fixed plant: Pipefitters, welders, inspectors, boilermakers, insulators, painters, riggers, scaffolders, equipment operators, general labor, and other civil crafts
  • Rotating and reciprocating mechanical equipment: Millwrights and machinery, workshop, and pump mechanics
  • Electrical equipment: High and low voltage electricians, HVAC technicians
  • Instrumentation: Instrument, analyzer, and control systems technicians.

To study maintenance performance, it is important to separately identify and measure the performance of each category. Initiatives that improve maintenance cost and reliability in the fixed plant category differ considerably from those that achieve similar results for rotating equipment. Fixed plant maintenance is dominated by metallurgical and inspection criteria and the need to perform the work during turnarounds. Rotating equipment involves seal technology, lubrication, vibration monitoring, and not least the standards applied to standby equipment. Summing the results of these two categories of performances into one performance metric not only will reveal little, it actually may mask important information in the separate categories.

Combining all four main categories into a single overall maintenance performance metric will reveal even less. Therefore, these data are separated in reports of overall maintenance performance.

Process characteristics and equipment complexity
Every site is unique. Differences in layout, processes, feedstock, products, plant complexity, and equipment redundancy therefore dictate the need for appropriate comparative metrics that account for the differences.

The most influential factor on maintenance cost is obviously the scale of the facility maintained. In recent years, expressing maintenance cost as a percent of facility replacement value has normalized the influence of size. In studies of specific industry sectors, other size normalizing factors are used (capacity, throughput, etc.).

The second family of factors that can influence maintenance cost and process reliability is the location of the facility with respect to the environment. Extremes of temperature, humidity, air borne aggressive agents, and proneness to unusual damage through cyclones, floods, earthquakes, etc., can affect performance significantly. The effects of the design standards used as a consequence of these factors can be both positive and negative on the performance metric. The designs can improve reliability directly; moreover, the effect of any additional maintenance cost on the performance metric may be masked when the resulting higher plant replacement value is used as the divisor.

A third family of physical factors that influence performance is the nature of the plant. Such factors may include age, construction standards, site acreage, and equipment crowding.

A fourth family is the nature of the process. This includes batch or continuous production modes, feed and product changes that require maintenance intervention, operating severity, corrosivity, etc. Performance metrics which are specific to a given product line family will go some way to normalizing these effects. They are not, however, the complete answer. Often facilities in the same product line family will have been built to different standards and may utilize different process variants.

The fifth family of factors that influence maintenance cost and process reliability is the actual amount of equipment in the process. This "equipment complexity" can be measured by the equipment count of a given family of equipment per plant replacement value. It seems obvious that plants with a higher number of heat exchangers per billion dollars of plant value will have higher heat exchanger maintenance costs.

The final category of equipment characteristics that must be mentioned is equipment redundancy. Plants with a lower percentage of spared pumps, for example, will tend to have higher production losses resulting from rotating equipment reliability problems.

Overall maintenance effectiveness
In a project to improve maintenance performance it is wrong to focus on one or two of its components and neglect others. There are at least five interdependent components of overall reliability and maintenance effectiveness. These include:

  • The value of production losses caused by maintenance activity
  • The direct costs (labor, contracts, and materials) of maintenance work
  • Overhead costs of maintenance support staff (supervisors, planners, schedulers, reliability engineers, managers, etc.)
  • The cost of operator time spent administering maintenance (scheduling, work permits, preparing plant for maintenance, etc.), and spent carrying out assigned maintenance tasks (condition monitoring, preventive routines, repairs, etc.)
  • The costs of tying up capital in an inventory of spare parts and materials.

The following examples of interdependence illustrate this dimension:

  • More maintenance by operators reduces the need for craftsman resources
  • More (up to a limit) preventive maintenance reduces production losses due to maintenance-caused stoppages
  • Too little maintenance overhead support (planning, scheduling, reliability engineering) results in increased direct maintenance costs
  • Too much maintenance overhead support adds to the total cost of maintenance
  • Too little spare parts inventory reduces service levels and increases repair duration and therefore lost production
  • Too high a level of inventory owned and managed by the plant increases the interest to be paid on the capital tied up in spares.

The performance target-setting process therefore must be based on a target objective of overall maintenance effectiveness that optimizes the balance of the targets of the individual components. Five performance indices comprise overall maintenance effectiveness—Reliability Loss Index, Direct Maintenance Cost Index, Indirect Maintenance Cost Index, Operator Maintenance Cost Index, and Spares Holding Cost Index.

Reliability Loss Index. When production is stopped or slowed, an opportunity for potentially profitable sales is lost and schedules are disrupted. But what is the value of the lost production when substantial spare capacity exists, or where profit margins are temporarily negative? The premise of most studies is that reliability always has economic value. To a large degree, plant reliability is also necessary to attain fundamental safety and environmental objectives.

In a growing business, additional capacity created by superior reliability earns an investment credit at least equal to the cost of building new capacity. Therefore, achieving more capacity through reliability is worth at least the anticipated return on the cost of building that capacity. Conversely, losing capacity costs at least the anticipated return on that capacity.

In studies, the percent of planned and unplanned maintenance outages and slowdowns is analyzed by causal equipment categories. Identifying the Reliability Loss Index by equipment causal component is fundamental in gaining an accurate understanding of the impact that the site severity and complexity and organizational factors may have on different areas of reliability.

The benefit of evaluating the Reliability Loss Index, a cost, as opposed to the more current Mechanical Availability Index is that reliability losses and maintenance costs are evaluated thereby on the same basis and therefore can be combined to give an overall maintenance effectiveness performance.

Maintenance Cost Indices (direct, indirect, and operator involvement). All the principal components of direct, indirect and operator maintenance costs are recorded to obtain a total cost index.

Direct costs are analyzed further by equipment category. Identifying maintenance costs at the equipment level is fundamental to understanding the impact of site severity, complexity, and organizational factors on the different cost areas.

Indirect and operator-related maintenance costs normally cannot be collected by equipment category and they are calculated pro-rata to the direct maintenance costs for the same equipment family.

Spares Holding Cost Index. The cost of holding inventory (like any investment) consists mostly of the interest to be paid on the capital tied up. Typically, a standard percentage rate per year is applied to spares inventory value to evaluate the cost of this component. Inventory data is collected for each category of equipment and the associated holding cost is calculated.

Performance sustainability
Plants can take steps that, in the short term, enable them to achieve unusually good performance levels in one or two components of reliability or maintenance cost. Often these can be unsustainable in the longer term. Typical examples are:

  • Delaying necessary tank maintenance overhauls for many years to reduce costs
  • Adopting a policy of temporary repairs on corroded pipework to reduce costs
  • Downsizing support staff to a level that provides insufficient support to equipment or craftsmen
  • Over-extending turnaround intervals and/or excessively reducing overhaul content to delay production losses until the next turnaround

World-class sustainable performance is not that which a few plants can occasionally achieve. Rather it is the best performance that a significant number of plants can achieve year in and year out.

Sustainable reliability or maintenance performance is therefore: Performance that can be achieved on a steady basis over time, preserving the integrity of the process and physical characteristics of the facility. This concept introduces three notions into the measurement of performance: Annualization of data, steady state operations, and performance trends.

Annualization of data. While accounting procedures generally regroup statistics on an annual basis, actual maintenance activity certainly does not occur in such a systematic way. For example, tankage is overhauled on a cycle of the order of 10 years, the components of better performing pumps need replacement at an interval of around 48 months, filters need cleaning once a week, and so on. Both maintenance cost and reliability performance therefore are greatly distorted when the events of only a specific 12-month accounting period are computed. A true reflection of performance can be obtained only if cost and lost production are annualized. However, this involves special computing only in the case that the events are of a frequency of more than 6 months. Events that occur more frequently and consistently normally will average out on an annualized basis.

Steady state operations. The need for maintenance is heightened when the production process is changed and out-of-specification operating conditions occur. In the extreme case, pilot plants of prototype processes require permanent maintenance department attention. It is fair to say, therefore, that the concept of sustainable performance is alien to non-steady-state operations and comparison of performance is unrealistic.

Performance trends. Performance metrics, to be truly useful, must account for and include trends in the measured parameters. These are usually economic in nature, inflation and price drift (up or down) being the most obvious. These are accounted for in most performance metrics by their design as ratios of concurrent costs and plant values.

Calculating sustainable performance targets
To gain credibility and acceptance, performance targets must be realistic. That is, they have to be achievable within a context understood by those responsible for achieving them. To set realistic and sustainable targets, management will best achieve its purposes by adopting the following approach which factors in the different dimensions of performance described previously.

Choose a group of better performers in their industry sector based on first quartile performance in overall maintenance effectiveness. This metric is the sum of the five measurable components of reliability and maintenance.

For direct costs and production losses in each equipment category, set targets based on the first quartile average of plant subgroups with similar equipment characteristics. (Note that the first quartile average should be calculated excluding plants with performance values that are outside the two-sigma range.)

For operator cost, overhead cost, and spares holding cost, set targets based on the first quartile average of plant subgroups with similar overall characteristics. (Again, the first quartile average should be calculated excluding plants with performance values that are outside the two-sigma range.)

The targets derived from this systematic approach provide balanced, equipment category, maintenance component, and overall target objectives. This is more realistic than a target derived by examination of individual component performance values in isolation.

The technique of summing performance components, coupled with the use of performance levels of real sites in the target-setting process, provides the best means of gaining commitment to the new targets and increasing the chance of lasting success. This would seem to be demonstrated overall by the maintenance cost index trends of plants participating in the CMA studies.

Using benchmarks to achieve goals
Once realistic and sustainable performance targets have been derived, set, and accepted, the next management task is to decide which organization practices should be promoted to achieve closure of the different performance gaps between the plant's actual performances and the sustainable targets.

Comparative studies of work practices and maintenance organization are the means to guide decisions. Organization and practices data can be classed in seven interdependent categories (an example of the data collected is given for each):

  • Reliability program (formalized improvement projects, etc.)
  • Engineering standards (specification of standby criteria, etc.)
  • Support staff (ratio of craftsmen per supervisor, etc.)
  • Organization (sharing resources across the plant, etc.)
  • Procedures (percent of work that is condition monitoring, etc.)
  • Craftsmen (percent overtime, etc.)
  • Contractors (percent of unit rate contracts, etc.)

Multivariable statistical analysis of plant characteristics, work practices, and organization features enables us to propose links between such features and their leverage on performance metrics. Statistical analysis of these features will potentially demonstrate that a given feature has an influence on performance, and the weight of the feature on performance.

The result is a model formula such as the following:

Electrician hours per year per million dollars plant value = K + a times number of motors per billion dollars plant value - b times percent of motors which are spared + c times product line age + d times percent of electrical work that is emergency  e times percent of electrical work that is condition monitoring + f times number of electricians per supervisor. (K, a, b, c, d, e, and f represent appropriate constants.)

It is our experience from studies that usually less than 10 of the first most significantly correlated features will achieve a correlation coefficient of better than 0.8.

An adequately large and well-designed database will deliver the means to provide knowledge of realistic/sustainable performance targets for individual plants within the database range, and identification of those organizational practices that will help to achieve quantified performance improvement.

The current participation of product lines in the CMA study is olefins, 24; olefin intermediates, 59; chlorinated hydrocarbons, 18; primary aromatics, 19; aromatics intermediates, 10; polyolefins, 23; thermoplastics and elastomers, 32; other petrochemicals, 17; chlor alkali, 21; refineries, 120; and others, 14.

In addition to the sheer size of the database, the quality of the performance targets predicted and the plant characteristics, organization, practices, and statistical models of performance developed depends on three factors: The ongoing development of the design of the database and questionnaire, the quality and validity of the data provided, and the thoroughness of the statistical analysis.

In successive years the conclusions obtained from these studies become more definitive as the scope of the database increases, the data collection by participants improves, and the database design itself evolves. MT

Information in this article is based on studies sponsored by the Chemical Manufacturers Association, Washington, D.C., and a paper presented at the National Petrochemical and Refiners Association Maintenance Conference held in Austin, TX, May 24, 2000. The CMA benchmark study is compiled on a 2-year cycle.

Michael Hernu is a senior consultant at Solomon Associates, Inc., Dallas, TX 75240. a benchmarking and management consulting firm; telephone (972) 385-8600.