Process-oriented organizations drive value by improving their business processes and equipment performance. At the same time, however, a number of applications, including asset management, work process improvement, defect elimination and preventive maintenance, among others, can be powerful but incomplete applications when seeking to sustain a competitive edge.

To implement and sustain high-performing, reliable cultures, managers need to be as rigorous about diagnosing, designing and implementing changes to the human decision-making process as they are with their business and equipment processes. Equipment and process reliability ultimately rest with human reliability. Thus, cultural change at its deepest level requires examining human reasoning and its resulting decisions.

To establish a culture-of-reliability requires going beyond the traditional stew of copycat approaches and learning how to: (1) use actionable tools to implement and sustain reliability improvements and bottom-line impact by (2) collecting cultural action data and (3) learning how to use that data to uncover hidden bottlenecks to performance.

In the quest for high performance, well-intentioned managers often launch cultural change efforts using what they believe to be applied methods, like employee surveys, team building, empowerment, leadership style, systems thinking, formal performance appraisal, 360° feedback, you name it, only to be disillusioned in the end by the fact that more change efforts fail than succeed. Although they may be well-accepted, traditional change methods are not precise enough to create and sustain cultures-of-reliability and typically evolve into the next flavor of the month.

The learning exercise
For the past 16 years I have been conducting a specific learning exercise related to cultural change. The purpose is to help participants understand why implementation is so hard. There are five objectives for the session:

1. To discover root cause of implementation barriers;
2. To illustrate the interdependent relationship between learning and error;
3. To determine how participants personally feel when they make mistakes;
4. Based on their experience of error, to understand how humans design a culture-in-action to avoid errors and mistakes; and
5. To determine the costs of error avoidance to business and human dignity.

To start, participants construct a definition of competitive learning which, at its root, is defined as the detection and correction of mistakes, errors, variance, etc., at ever-increasing rates of speed and precision—the heart of reliability. Through poignant illustrations, they learn that their organizations tend to focus on making fast decisions ("time is money"), timelines, milestones etc., but at a cost to precision, the quality of the decision.

Based on that definition, the participants are asked to reflect on a recent performance mistake they have made on the job or in life. The response from hundreds of them—male and female, Fortune 500 executives, managers, supervisors, engineers, technicians and craftsmen—are very consistent. When they make an error they feel: shame, anger, frustration, stupid, embarrassed, inadequate with an impulse to hide the error and, at the same time, a desire to fix it. The result is an emotionally charged picture of wanting to fix mistakes coupled with an overwhelming response to hide them for fear of blame.

As the exercise unfolds, participants gain insight into how learning and mistakes, trial and error shape performance and how ineffective learning patterns persist for years. For example, individuals from process industries have revealed they've known that less-than-effective outages and turnarounds have existed for years; that "lessons-learned" sessions don't successfully address operations and maintenance infighting and squabbles over what quality work means and the validity of data; that stalled work management initiatives or reprisals for management decisions are a fact of life; etc. The list goes on and on. Discovering why his division had not been able to penetrate a market for over 20 years, one vice president-level participant summed up the dilemma this way: "The costs [of ineffective learning] are so high, they are un-estimateable."

Through collective reflection in a larger group, participants come to realize that they all experience learning in very similar ways. They also come to learn that their reasoning is very similar. They typically espouse that continuous learning is important and mistakes are OK, but, in the final analysis, mistakes are categorized as critical incidents on performance appraisals or simply seen as ineffectiveness.

When performance appraisal is tied to pay, rewards and promotion, participants indicate that they would have to be foolish, if they "didn't put the best spin" and save face at any cost. "I have a mortgage to pay" is how many respondents put it. At the same time, they acknowledge learning does occur, but at a rate that leaves much to be desired. "It's not all bad," is how many participants put it. Yet, this is not really a case of being bad. Rather, it is a case of sincere, hard-working people unknowingly designing a culture with a set of unintended outcomes.

At this point, participants begin to gain insight: they say one thing and do another. Moreover, they come to understand that it is easy to see defensive patterns in others, but not so easy to see defensive patterns in themselves. Not surprising, being defensive is espoused as not ok. Hence, good team players should be open to feedback. Not being open would be admitting a mistake, the very essence of pain.

In the final phase of the learning exercise, participants come to recognize that they have a strong desire to learn and they seek noble goals, but that fears of retribution for telling the truth, blame, fear of letting someone down or fear of failure, whether in substance or perception, contribute to a sense of loss of control. Unfortunately, this situation violates the first commandment of management: BE IN CONTROL.

The need for control translates into a hidden performance bottleneck, given the complexity of job interdependencies and systemic error. As one individual noted, "I can't control what I can't control, but I am held accountable. Accountability translates into who to blame." Participants acknowledge that they subtly side-step difficult issues and focus on the more routine, administrative issues, thereby reducing emotional pain and conflict in the short term. They acknowledge that they bypass the potential for higher performance by not reflecting on gaps in decision-making.

Ironically, as these decision bottlenecks limit performance, expectations for better performance increase, often resulting in unrealistic timelines and more stress. Executives complain they just don't get enough change fast enough, and middle managers and individual contributors complain of "micro-management." Sound familiar?

The end result is that sincere attempts to improve the status quo slowly are cocreatively undermined and inadequate budgets and unrealistic timeframes are set. Good soldiers publicly salute the goals, but privately resist because their years of experience have taught them to think in terms of "what's the use of telling the truth as I see it; this, too, will pass." Ultimately, many see the "other guy(s)" or group as the problem and wonder why we can't "get them" in line. This is the heart of an organizational fad—something that often is labeled as the lack of accountability.

Culture-in-action
Based on participants' data generated from this learning exercise and action data recorded and collected from the field (see Part III of this series for the data collection method), a culture-inaction model, similar to that shown in Fig. 1, is created and verified with illustrations. Participants consistently agree this type of model is accurate and reflects their own current cultures-in-action.

Underlying assumptions…
The culture-in-action model is rooted in human reasoning. Given the assumptions of avoiding mistakes and being in control to win and look competent in problem resolution, the reasoning path is clear. The behaviors make perfectly good sense.

Behavior…
When seeking solutions, multiple perspectives will proliferate on which solution is best, some with more risk, some with less. Think of it as inference stacking. A complex web of cause and effect, solutions and reasons why something will or will not work are precariously stacked one upon the other, up to a dizzying height.

Determining whose perspective is right is problematic ("Your guess is as good as mine"). Hence, controlling the agenda to reduce frustration either by withholding information ("Don't even go there") or aggressively manipulating people to submit or comply with someone else's views to get things done is a logical conclusion based on the underlying assumptions.

It is not surprising that executives seek to control their organizations and focus on objectives—and when they do this that middle managers privately feel out of control because they think they are not trusted to implement initiatives or handle day-to-day routines. This leads to the following managerial dilemma: If I voice my real issues, I will not be seen as a good team player. If I stay silent, I will have to pretend to live up to unrealistic expectations. Either way is no win (a real double bind).

To overcome this dilemma, people verify and vent their emotions one-on-one, i.e. in hallways, restrooms and offices. This way, they avoid confronting the real issue of how they are impacted by others, which is diffi- cult to discuss in a public forum ("Don't want to make a career-threatening statement"). Instead, they seek thirdparty validation that their beliefs are the right ones to hold ("Hey, John, can you believe what just happened in that meeting? I don't think that strategy is going to work; didn't we try it 10 years ago?"). Even the best-performing teams demonstrate some of these performance-reducing characteristics. The culture becomes laden with attributions about others' motivation, intent and effectiveness and it is labeled "politics."

Results…
Routine problems often are uncovered, organizations do learn, but the deeper performance bottlenecks, hidden costs, sources of conflict and high-performance opportunities are missed because the focus is on putting the "best spin" on "opportunities for improvement" with a twist of language to avoid the "mistake" word. That's because mistakes are bad and people don't like to discuss them. Interestingly enough, there are even objections to using the word "error" during the process of the exercise. It is not surprising that when trying to learn and continuously improve a turnaround, business process or project, for example, people privately will conclude "Oh, boy, here we go again. Another wasted meeting debating the same old issues." Negative attributions proliferate ("They don't want to learn") and underlying tension grows.

At this stage of the process, the pattern begins to repeat itself. As the project effort falls behind, expectations build. Typically, someone will be expected to "step up" and be the hero. With eyes averted, looking down, uncomfortable silence, someone "steps up" and often gets rewarded. Yet this heroic reward doesn't address root cause (i.e. what accounted for the errors and frustration in the first place). Side-stepping or avoiding the more difficult-to-discuss issues don't help uncover root cause, but, rather, lead to fewer errors being discovered. As a result, the business goal is pushed a little further out and economic vulnerability is increased.

If the market is robust, errors and mistakes may mean little to a business. The demand can be high if you have the right product, at the right time. As competition increases, however, or the market begins to falter, the ability to remain competitive and achieve what the organization has targeted is crucial. Competitive learning is the only weapon an organization has to maintain its edge in the marketplace.

Major culture-in-action features
In summary, the major features of a true culture-in-action are:

• Avoidance of mistakes and errors at all cost;
• Little active inquiry to test negative attributions;
• Little personal reflection (i.e. "How am I a part of the problem?");
• Little discussion of personal performance standards by which we judge others; and
• Little agreement on what valid data would look like.

As the exercise winds down, it's not long before someone asks, "So how do you get out of this status quo loop?" When this question comes, because it always does, I turn it back to the group and ask how they would alter this cultural system? The reaction is always the same—silence and stares. No wonder. The answer is not intuitively obvious, even to the most seasoned of practitioners and theorists.

The short answer is rather than "get" anyone anywhere, change has to be based on individual reflection and actionable tools driven through collaborative design and invitation. These actionable tools balance the playing field, at all levels, by helping create informed choice through daily decision-making reflection. Traditional intervention methods focus on changing behavior, learning your style or type, building a vision, etc. There are any number of approaches, all very powerful but incomplete without addressing the underlying reasoning (root cause) that is informing the behavior in the first place.

Coming next month
In Part II, a culture of reliability will be defined, as well as the role of reflection in organizational performance and the actionable tools of collaborative design. MT

Brian Becker is a senior project manager with Reliability Management Group (RMG), a Minneapolis-based consulting firm. With 27 years of business experience, he has been both a consultant and a manager. Becker holds a Harvard doctorate with a management focus. For more information, e-mail: bbecker@rmgmpls.com

Wireless Technology now provides secure, reliable communication for remote field sites and applications where wires cannot be run for practical or economic reasons. For maintenance purposes, wireless can be used to acquire condition monitoring data from pumps and machines, effluent data from remote monitoring stations, or process data from an I/O system.

For example, a wireless system monitors a weather station and the flow of effluent leaving a chemical plant. The plant's weather station is 1.5 miles from the main control room. It has a data logger that reads inputs from an anemometer to measure wind speed and direction, a temperature gauge and a humidity gauge. The data logger connects to a wireless remote radio frequency (RF) transmitter module, which broadcasts a 900MHz, frequency hopping spread spectrum (FHSS) signal via a YAGI directional antenna installed at the top of a tall boom located beside the weather station building. This posed no problem.

However, the effluent monitoring station was thought to be impossible to connect via wireless. Although the distance from this monitoring station to the control room is only one-quarter mile, the RF signal had to pass through a four-story boiler building. Nevertheless, the application was tested before installation, and it worked perfectly. The lesson here is that wireless works in places where you might think it can't. All you have to do is test it.

There are many flavors of wireless, and an understanding is needed to determine the best solution for any particular application.Wireless can be licensed or unlicensed, Ethernet or serial interface, narrow band or spread spectrum, secure or open protocol,Wi-fi…the list goes on. This article provides an introduction to this powerful technology.

The radio spectrum
The range of approximately 9 kilohertz (kHz) to gigahertz (GHz) can be used to broadcast wireless communications. Frequencies higher than these are part of the infrared spectrum, light spectrum, X-rays, etc. Since the RF spectrum is a limited resource used by television, radio, cellular telephones and other wireless devices, the spectrum is allocated by government agencies that regulate what portion of the spectrum may be used for specific types of communication or broadcast.

In the United States, the Federal Communications Commission (FCC) governs the allocation of frequencies to non-government users. FCC has limited the use of Industrial, Scientific, and Medical (ISM) equipment to operate in the 902-928MHz, 2400-2483.5MHz and 5725-5875MHz bands,with limitations on signal strength, power, and other radio transmission parameters. These bands are known as unlicensed bands, and can be used freely within FCC guidelines. Other bands in the spectrum can be used with the grant of a license from the FCC. (Editor's Note: For a quick definition of the various bands in the RF spectrum, as well as their uses, log on to: http://encyclopedia.thefreedictionary. com/radio+frequency )

Licensed or unlicensed
A license granted by the FCC is needed to operate in a licensed frequency. Ideally, these frequencies are interference-free, and legal recourse is available if there is interference. The drawbacks are a complicated and lengthy procedure in obtaining a license, not having the ability to purchase off-the-shelf radios since they must be manufactured per the licensed frequency, and, of course, the costs of obtaining and maintaining the license.

License-free implies the use of one of the frequencies the FCC has set aside for open use without needing to register or authorize them. Based on where the system will be located, there are limitations on the maximum transmission power. For example, in the U.S., in the 900MHz band, the maximum power may be 1 Watt or 4 Watts EIRP (Effective Isotropic Radiated Power).

The advantages of using unlicensed frequencies are clear: no cost, time or hassle in obtaining licenses; many manufacturers and suppliers who serve this market; and lower startup costs, because a license is not needed. The drawback lies in the idea that since these are unlicensed bands, they can be "crowded" and, therefore, may lead to interference and loss of transmission. That‘s where spread spectrum comes in. Spread spectrum radios deal with interference very effectively and perform well, even in the presence of RF noise.

Spread spectrum systems
Spread Spectrum is a method of spreading the RF signal across a wide band of frequencies at low power, versus concentrating the power in a single frequency as is done in narrowband channel transmission. Narrowband refers to a signal which occupies only a small section of the RF spectrum, whereas wideband or broadband signal occupies a larger section of the RF spectrum. The two most common forms of spread spectrum radio are frequency hopping spread spectrum (FHSS), and direct sequence spread spectrum (DSSS). Most unlicensed radios on the market are spread spectrum.

As the name implies, frequency hopping changes the frequency of the transmission at regular intervals of time. The advantage of frequency hopping is obvious: since the transmitter changes the frequency at which it is broadcasting the message so often, only a receiver programmed with the same algorithm would be able to listen and follow the message. The receiver must be set to the same pseudo-random hopping pattern, and listen for the sender's message at precisely the correct time at the correct frequency. Fig. 1 shows how the frequency of the signal changes with time. Each frequency hop is equal in power and dwell time (the length of time to stay on one channel). Fig. 2 shows a two dimensional representation of frequency hopping, showing that the frequency of the radio changes for each period of time. The hop pattern is based on a pseudo random sequence.

DSSS combines the data signal with a higher data-rate bit-sequence-also known as a ‘chipping code'-thereby "spreading" the signal over greater bandwidth. In other words, the signal is multiplied by a noise signal generated through a pseudo-random sequence of 1 and -1 bits. The receiver then multiplies the signal by the same noise to arrive at the original message (since 1 x 1 = 1 and -1 x -1 = 1).

When the signal is "spread," the transmission power of the original narrowband signal is distributed over the wider bandwidth, thereby decreasing the power at any one particular frequency (also referred to as low power density). Fig. 3 shows the signal over a narrow part of the RF spectrum. In Fig. 4, that signal has been spread over a larger part of the spectrum, keeping the overall energy the same, but decreasing the energy per frequency. Since spreading the signal reduces the power in any one part of the spectrum, the signal can appear as noise. The receiver must recognize this signal and demodulate it to arrive at the original signal without the added chipping code. FHSS and DSSS both have their place in industry and can both be the "better" technology based on the application. Rather than debating which is better, it is more important to understand the differences, and then select the best fit for the application. In general, a decision involves:

• Throughput
• Colocation
• Interference
• Distance
• Security

Throughput
Throughput is the average amount of data communicated in the system every second. This is probably the first decision factor in most cases. DSSS has a much higher throughput than FHSS because of a much more efficient use of its bandwidth and employing a much larger section of the bandwidth for each transmission. In most industrial remote I/O applications, the throughput of FHSS is not a problem.

As the size of the network changes or the data rate increases, this may become a greater consideration. Most FHSS radios offer a throughput of 50-115 kbps for Ethernet radios.Most DSSS radios offer a throughput of 1-10 Mbps. Although DSSS radios have a higher throughput than FHSS radios, one would be hard pressed to find any DSSS radios that serve the security and distance needs of the industrial process control and SCADA market. Unlike FHSS radios, which operate over 26MHz of the spectrum in the 900MHz band (902-928MHz), and DSSS radios, which operate over 22MHz of the 2.4GHz band, licensed narrow band radios are limited to 12.5kHz of the spectrum.Naturally, as the width of the spectrum is limited, the bandwidth and throughput will be limited as well.Most licensed frequency narrowband radios offer a throughput of 6400 to 19200 bps.

Collocation
Collocation refers to having multiple independent RF systems located in the same vicinity. DSSS does not allow for a high number of radio networks to operate in close proximity as they are spreading the signal across the same range of frequencies. For example, within the 2.4GHz ISM band, DSSS allows only three collocated channels. Each DSSS transmission is spread over 22MHz of the spectrum, which allows only three sets of radios to operate without overlapping frequencies.

FHSS, on the other hand, allows for multiple networks to use the same band because of different hopping patterns. Hopping patterns which use different frequencies at different times over the same bandwidth are called orthogonal patterns. FHSS uses orthogonal hopping routines to have multiple radio networks in the same vicinity without causing interference with each other. That is a huge plus when designing large networks, and needing to separate one communication network from another. Many lab studies show that up to 15 FHSS networks may be collocated, whereas only 3 DSSS networks may be collocated. Narrowband radios obviously cannot be collocated as they operate on the same 12.5MHz of the spectrum.

Interference
Interference is RF noise in the vicinity and in the same part of the RF spectrum. A combining of the two signals can generate a new RF wave or can cause losses or cancellation in the intended signal. Spread Spectrum in general is known to tolerate interference very well, although there is a difference in how the different flavors handle it.When a DSSS receiver finds narrowband signal interference, it multiplies the received signal by the chipping code to retrieve the original message. This causes the original signal to appear as a strong narrow band; the interference gets spread as a low power wideband signal and appears as noise, and thus can be ignored.

In essence, the very thing that makes DSSS radios spread the signal to below the noise floor is the same thing that allows DSSS radios to ignore narrowband interference when demodulating a signal. Therefore, DSSS is known to tolerate interference very well, but it is prone to fail when the interference is at a higher total transmission power, and the demodulation effect does not drop the interfering signal below the power level of the original signal.

Given that FHSS operates over 83.5MHz of the spectrum in the 2.4GHz band, producing high power signals at particular frequencies (equivalent to having many short synchronized bursts of narrowband signal) it will avoid interference as long as it is not on the same frequency as the narrowband interferer.Narrowband interference will, at most, block a few hops which the system can compensate for by moving the message to a different frequency. Also, the FCC rules require a minimum separation of frequency in consecutive hops, and therefore the chance of a narrowband signal interfering in consecutive hops is minimized.

When it comes to wideband interference, DSSS is not so robust. Since DSSS spreads its signal out over 22MHz of the spectrum all at once at a much lower power, if that 22MHz of the spectrum is blocked by noise or a higher power signal, it can block 100% of the DSSS transmission, although it will only block 25% of the FHSS transmission. In this scenario, FHSS will lose some efficiency, but not be a total loss.

In licensed radios the bandwidth is narrow, so a slight interference in the range can completely jam transmission. In this case, highly directional antennas and band pass filters may be used to allow for uninterrupted communication, or legal action may be pursued against the interferer.

802.11 radios are more prone to interference since there are so many readily available devices in this band. Ever notice how your microwave interferes with your cordless phone at home? They both operate in the 2.4GHz range, the same as the rest of 802.11 devices. Security becomes a greater concern with these radios.

If the intended receiver of a transmitter is located closer to other transmitters and farther from its own partner, it is known as a Near/Far problem. The nearby transmitters can potentially drown the receiver in foreign signals with high power levels. Most DSSS systems would fail completely in this scenario. The same scenario in a FHSS system would cause some hops to be blocked but would maintain the integrity of the system. In a licensed radio system, it would depend on the frequency of the foreign signals. If they were on the same or close frequency, it would drown the intended signal, but there would be recourse for action against the offender unless they have a license as well.

Distance
Distance is closely related to link connectivity, or the strength of an RF link between a transmitter and a receiver, and at what distance they can maintain a robust link. Given that the power level is the same, and the modulation technique is the same, a 900MHz radio will have higher link connectivity than a 2.4GHz radio. As the frequency in the RF spectrum increases, the transmission distance decreases if all other factors remain the same. The ability to penetrate walls and object also decreases as the frequency increases.Higher frequencies in the spectrum tend to display reflective properties. For example, a 2.4GHz RF wave can bounce off reflective walls of buildings and tunnels. Based on the application, this can be used as an advantage to take the signal farther, or it may be a disadvantage causing multipath, or no path, because the signal is bouncing back.

FCC limits the output power on spread spectrum radios. DSSS consistently transmits at a low power, as discussed above, and stays within the FCC regulation by doing so. This limits the distance of transmission for DSSS radios, and thus this may be a limitation for many of the industrial applications. FHSS radios, on the other hand, transmit at high power on particular frequencies within the hopping sequence, but the average power on the spectrum is low, and therefore can meet with the regulations. Since the actual signal is transmitting at a much higher power than the DSSS, it can travel further.Most FHSS radios are capable of transmitting over 15 miles, and longer distances with higher gain antennas.

802.11 radios, although available in both DSSS as well as FHSS, have a high bandwidth and data rate, up to 54Mbps (at the time of this publication). But it is important to note that this throughput is for very short distances, and downgrades very quickly as the distance between the radio modems increases. For example, a distance of 300 feet would drop the 54Mbps rate down to 2Mbps. This makes this radio ideal for a small office or home application, but not for many industrial applications where there is a need to transmit data over several miles.

Since narrowband radios tend to be a lower frequency, they are a good choice in applications where FHSS radios cannot provide adequate distance. A proper application for narrow band licensed radios is when there is a need to use a lower frequency to either travel over a greater distance, or be able to follow the curvature of the earth more closely and provide link connectivity in areas where line of sight is hard to achieve.

Security
Since DSSS signals run at such low power, the signals are difficult to detect by intruders. One strong feature of DSSS is its ability to decrease the energy in the signal by spreading the energy of the original narrowband signal over a larger bandwidth, thereby decreasing the power spectral density. In essence, this can bring the signal level below the noise floor, thereby making the signal "invisible" to would-be intruders. On the same note, however, if the chipping code is known or is very short, then it is much easier to detect the DSSS transmission and retrieve the signal since it has a limited number of carrier frequencies. Many DSSS systems offer encryption as a security feature, although this increases the cost of the system and lowers the performance, because of the processing power and transmission overhead for encoding the message.

For an intruder to successfully tune into a FHSS system, he needs to know the frequencies used, the hopping sequence, the dwell time and any included encryption. Given that for the 2.4GHz band the maximum dwell time is 400ms over 75 channels, it is almost impossible to detect and follow a FHSS signal if the receiver is not configured with the same hopping sequence, etc. In addition, most FHSS systems today come with high security features such as dynamic key encryption and CRC error bit checking.

Today,Wireless Local Area Networks (WLAN) are becoming increasingly popular. Many of these networks use the 802.11 standard, an open protocol developed by IEEE.Wi-fiis a standard logo used by the Wireless Ethernet Compatibility Alliance (WECA) to certify 802.11 products. Although industrial FHSS radios tend to not be Wi-fi, and therefore not compatible with these WLANs, there may be a good chance for interference due to them operating in the same bandwidth. Since most Wi-fiproducts operate in the 2.4 or 5GHz bands, it may be a good idea to stick with a 900MHz radio in industrial applications, if the governing body allows this range (Europe allows only 2.4GHz, not 900MHz). This will also provide an added security measure against RF sniffers (a tool used by hackers) in the more popular 2.4 band.

Security is one of the top issues discussed in the wireless technology sector. Recent articles about "drive-by hackers" have left present and potential consumers of wireless technology wary of possible infiltrations. Consumers must understand that 802.11 standards are open standards and can be easier to hack than many of the industrial proprietary radio systems.

The confusion about security stems from a lack of understanding of the different types of wireless technology. Today, Wi-fi(802.11a, b, and g) seems to be the technology of choice for many applications in the IT world, homes and small offices. 802.11 is an open standard in which many vendors, customers and hackers have access to the standard.While many of these systems have the ability to use encryption like AES and WEP, many users forget or neglect to enable these safeguards which would make their systems more secure.Moreover, features like MAC filtering can also be used to prevent unauthorized access by intruders on the network. Nonetheless, many industrial end users are very wary about sending industrial control information over standards that are totally "open."

So, how do users of wireless technology protect themselves from infiltrators? One almost certain way is to use non- 802.11 devices that employ proprietary protocols that protect networks from intruders. Frequency hopping spread spectrum radios have an inherent security feature built into them. First, only the radios on the network that are programmed with the "hop pattern" algorithm can see the data. Second, the proprietary, non-standard, encryption method of the closed radio system will further prevent any intruder from being able to decipher that data.

The idea that a licensed frequency network is more secure may be misleading. As long as the frequency is known, anyone can dial into the frequency, and as long as they can hack into the password and encryption, they are in. The added security benefits that were available in spread spectrum are gone since licensed frequencies operate in narrowband. Frequency hopping spread spectrum is by far the safest, most secure form of wireless technology available today.

Mesh radio networks
Mesh radio is based on the concept of every radio in a network having peer-topeer capability. Mesh networking is becoming popular since its communication path has the ability to be quite dynamic. Like the worldwide Web, mesh nodes make and monitor multiple paths to the same destination to ensure that there is always a backup communication path for the data packets.

There are many concerns that developers of mesh technology are still trying to address, such as latency and throughput. The concept of mesh is not new. The internet and phone service are excellent mesh networks based in a wired world. Each node can initiate communication with another node and exchange information.

Summary
In conclusion, the choice of radio technology to use should be based on the needs of the application. For most industrial process control applications, proprietary protocol license-free frequency hopping spread spectrum radios (Fig. 5) are the best choice because of lower cost and higher security capabilities in comparison to licensed radios.When distances are too great for a strong link between FHSS radios with repeaters, then licensed narrowband radios should be considered for better link connectivity. The cost of licensing may offset the cost of installing extra repeaters in a FHSS system.

As more more industrial applications require greater throughput, networks employing DSSS that enable TCP/IP and other open Ethernet packets to pass at higher data rates will be implemented. This is a very good solution where PLCs (Programmable Logic Controllers), DCS (Distributed Control Systems) and PCS (Process Control Systems) need to share large amounts of data with one another or upper level systems like MES (Manufacturing Execution Systems) and ERP (Enterprise Resource Planning) systems.

When considering a wireless installation, check with a company offering site surveys that allow you to install radios at remote locations to test connectivity and throughput capability. Often this is the only way to ensure that the proposed network architecture will satisfy your application requirements. These demo radios also let you look at the noise floor of the plant area, signal strength, packet success rate and the ability to identify if there are any segments of the license free bandwidth that are currently too crowded for effective communication throughput. If this is the case, then hop patterns can be programmed that jump around that noisy area instead of through it. MT

Gary Mathur is an applications engineer with Moore Industries-International, in North Hills, CA. He holds Bachelor's and Masters degrees in Electronics Engineering from Agra University, and worked for 12 years with Emerson Process Management before joining Moore. For more information on the products referenced in this article, telephone: (818) 894-7111; e-mail: GMathur@miinet.com

Richard L. Dunn, Executive Director, Foundation for Industrial Maintenance Excellence

Now in its seventeenth year, the NAME Award is widely regarded as the most prestigious recognition in the maintenance function. Awards are presented to individual plants on the basis of their maintenance departments’ ability to provide “capacity assurance for operational excellence” in the areas of organization, work processes and materials management.

In many ways, the two winners represent the breadth of the possible paths to maintenance excellence. One is a large plant, the other small; one a large maintenance organization, the other not. One plant is primarily a round-the-clock continuous process operation, the other a manufacturer of discrete products. One has a long tradition of striving for and exemplifying maintenance excellence, the other has come to this level only recently.

Alcoa Mt. Holly is a 1.5 million-square-foot aluminum smelter that produces about 500 million pounds of aluminum ingots annually. Its 160 maintenance employees support the 24/7 operation of the plant through a wide variety of preventive and predictive maintenance activities, major equipment overhauls and operation and maintenance of the plant’s substation. In recommending the plant for the NAME Award, evaluators noted its long history of outstanding work planning and scheduling, as well as its excellent communications and cooperation with all production areas.

Dodge Marion manufactures mounted tapered/spherical roller bearings in its 174,000- square-foot facility. Its nine-person maintenance department has developed a strong preventive and predictive maintenance program using various total productive maintenance (TPM) processes.

Both plants have demonstrated enviable records for reliability. Furthermore, both demonstrate that a foundation of sound preventive maintenance practices coupled with a plant-wide respect for the value of maintenance is essential to overall excellence.

Established in 1990 as a way to encourage best maintenance practices and a way to honor those who achieve them, the NAME Award program has presented 20 awards over the years with several awards in some years and none in others. In 2000, the volunteers who administer the award program incorporated as the not-for-profit Foundation for Industrial Maintenance Excellence (FIME) to ensure the program’s continuance and independence from commercial influence. The Board of Directors is made up of past award winners and others with a demonstrated devotion to the values the award represents.

To be eligible, a plant must submit a comprehensive application by June 30 in the year of entry. This application is reviewed by the Board of Directors to determine eligibility for an onsite audit. Following this audit, the Board of Directors again meets to decide if the applicant qualifies in all respects for the award.

The NAME Award recognizes that the Alcoa Mt. Holly and Dodge Marion plants have demonstrated their maintenance competence at a world-class level. The Foundation for Industrial Maintenance Excellence is proud to honor their achievements.

Rick Dunn participated in the establishment of the North American Maintenance Excellence Award and has been active in its activities since inception. He was appointed Executive Director when the NAME Award program was incorporated as the Foundation for Industrial Maintenance Excellence. Information on the NAME Award program is available online at www.nameaward.com

Bob Williamson, Contributing Editor

Maintenance & Reliability is, and has been, a woefully overlooked career. We need our nation's best and brightest young minds in Maintenance & Reliability careers NOW! What are we doing to attract and retain them?

What are we doing to train them to maintain the highest levels of equipment performance and reliability? What are we doing to promote pride in workmanship? The situation in many plants is already dire…and getting worse. You can see, hear and sense it everywhere, especially out on the plant floor.

Who's got time for training
"I learned this job years ago from one of the best. I was under his wing for nearly eight months learning all the aspects of the precision work on this one type of machinery. In the 35 years I have worked here, I have never seen such a lack of training of our new guys. They get a few days training at best. Why, we even have some of the new employees teaching the newer employees how to work on this equipment. Pretty scary if you ask me! Most of them have never even seen the manual that came with these machines, the one that I learned from years ago. The only copy we have now is locked up in the maintenance office. Doesn't anyone in top management care anymore?"

The skilled mechanic quoted above was truly concerned. We had just discovered that another mechanic at one point cranked down on one of the precision adjustments so far that it badly damaged the machine. The procedure in the equipment manual was not followed. Even though it was still running and making acceptable parts, the $10,000 precision cylinder had been scored beyond repair and there was no spare in stock. After a 12-week estimated delivery time, it would take several more days to replace the damaged parts.

We've always done it that way
In another plant, I noticed that four finethreaded machine adjustment bolts had been beaten severely with a hammer. They were so mushroomed that a wrench would no longer fit. ("That's why we have Channel Lock pliers.") Logically, and mechanically, any adjustment had to be made by turning the threaded adjusters. No other movement was possible. When asked, the mechanics all responded:

"Why do we hit the adjusters with a hammer? That's the way we were taught. I guess we've always done it that way."

We couldn't find the manual
A one-year-old machine's programmable controller was operated with a touch screen panel. While working on a processing line that fed this final stage unit, we noticed a gaggle of people gathered around the panel poking at it. Then they just wandered away. As we attempted to start up the machine, we discovered that the program had been erased and the machine would not cycle properly. Searching for the machine's O&M manual, we discovered it underneath a workbench…and half of it was missing! As one individual later explained:

"Somebody must have messed with the program, again. If you touch this icon, then this one, it erases the program. I figured that out the hard way since we've never really had training on the programming controls. The manual has some of the control panel information, but it's still not easy to understand."

Sure we do regular preventive maintenance
During a hands-on PM workshop on a large integrated manufacturing line, one person discovered a loose bolt (no, it was not a maintenance person). Upon further investigation, we discovered that only one of the four bolts holding this unit together and in alignment was actually in place. One was missing, another one was completely broken off and a third bolt had the head sheared off. The remaining bolt was doing the work of four and was the only link between full operation and catastrophic downtime. After two hours of disassembly and repair, the broken bolt problems were corrected. The situation, evidently, came as surprise to at least one staffer:

"I don't understand how we could have missed that one. Our monthly PM was just completed a few days ago."

What's changed
We are in the midst the "de-skilling" of the American industrial workforce—not by design, but by default. It's not a new phenomenon either. This frightening trend has been overlooked by far too many of our business, government and academic decision-makers for far too long. We are at a near-critical point-of-no-return as the critical mass of skilled and knowledgeable people leave today's workplace. Too many of today's maintenance, reliability and operations personnel have not been adequately trained and qualified to do the jobs they are asked to do day in and day out. Many, if not most, younger and newer employees may not have the same basic skills and knowledge as those whom they are replacing.

Unfortunately, today's decision-makers often ASSUME the fundamental skills and knowledge that were "common" when they began working 30-plus years ago are the same today. While we hate to be the bearer of bad tidings, these decision-makers are sorely wrong! There has been a fundamental paradigm shift and it is hurting our capital-intensive industries' performance and reliability.

Think about it. How many of today's older teenagers and twenty-somethings ever have:

• Built a birdhouse, a utility box or a shed?
• Changed the oil and filter in a car or truck?
• Disassembled a lawnmower, a motorcycle, a jet ski or a snowmobile engine, put it back together and have it run?
• Assembled a radio, a computer or an electronic robot?
• Glazed a wood frame window?
• Rebuilt an automobile engine?
• Made something useful on a lathe or milling machine?
• Owned and used a set of mechanic's or carpenter's tools?
• Used a volt-ohmmeter to check a circuit?
• Welded an angle iron frame or built a metal stand?
• Soldered copper tubing or brazed steel tubing?
• Installed and wired a doorbell?

Not many parents spend time with their children and teenagers making things, building projects or doing repairs around the home these days. Many of the fundamental skills and knowledge we took for granted in the 1960s, 70s and early 80s are apparently no longer valued. Luckily, there still are some very good high school vocational programs out there and some very good post-secondary technical colleges too—despite thousands of schools and programs being closed over the years. But, there simply are not enough schools and programs to address the problem we have now—a problem that's going to get worse before it gets worse.

An overlooked career
As shown in the findings of our 2007 Salary Survey beginning on page 38, Maintenance & Reliability technician jobs can pay quite well. Some industries pay in the $30 per hour range and higher. So, why do countless newly-minted high school grads take jobs that pay less than $10 per hour—and, hop from job to job for years until they find their niche? Why do they go on to a four-year college to try and figure out what career they want to pursue in life? (If you are asking me, that is really an expensive "career education" program!)

We should promote careers in Maintenance & Reliability (not just "maintenance jobs")! Clean up the workplace and give career-day tours. Help teachers and students understand that good money can be made in a rewarding career with a one- or two-year technical degree. Begin attracting the best and the brightest. Offer high-school cooperative education experience in your plant.

Trainers and coaches
Recruit a few of your senior, highly skilled maintenance personnel to be trainers and on-job coaches. Have them dedicate time documenting proper maintenance and reliability procedures for your critical equipment. Set new expectations; insist that critical maintenance tasks follow "standard procedures" or "standard job plans." Train everyone who needs to know—everyone who touches the critical equipment—to follow these new standards. Then, hold everyone accountable for following these procedures. Problems will begin disappearing!

Show everybody that you care about how your equipment and plant are maintained. Be proud of your workmanship. Share a positive vision for careers in this arena. Let's make 2008 the year of "Transforming Careers in Maintenance & Reliability."

Jane Alexander, Editor-In-Chief

Quick-Eye
Quick-Eye digital video systems help manufacturing and packaging line operators improve production efficiency. Quick-Eye captures high-speed video and replays product and equipment issues in slow motion. It is portable and can be moved to problem areas with little setup time. Operators can eliminate bottlenecks and address the root causes of problems faster.

Quick-Eye offers high frame rates, high resolution, multihour video buffer, image analysis, etc. According to IVS, this affordable and simple-to-use technology provides an immediate return on investment (ROI).

WebScanPRO
WebScanPRO provides advanced monitoring and sheet break analysis for the paper industry and other web process manufacturers, such as non-woven fabrics and plastic sheet. Fast, precise and digitally simple, it, too, offers fast return on investment by continuously recording events that cause machine problems, poor quality and sheet breaks with some of the industry’s most advanced technology, including:

• 100% noise-free digital video;
• 90 or 200 frames per second at 659x493 resolution, assuring 100% monitoring on the fastest paper machine;
• Up to 1/100,000 sec shutter speed;
• Video synchronized to 1-frame and sheet break events saved without operator's assistance. WebScanPRO offers exhaustive image analysis, including:
• Grayscale of each frame is displayed with buffered and event video;
• Real-time regions of interest (ROI) alert operators to changes in video. ROI can be defined for any camera;
• Digital live video broadcast over the mill network accessible on any computer;
• WebScanPRO is always on. It never misses a frame; simultaneous video capture, live video, viewing video in the buffer, viewing sheet breaks, ROI image analysis and grayscale analysis;
• Paper-machine proven lights and camera enclosures.

Industrial Video Solutions, Inc.
McLean, VA

We asked the questions. Here are our findings. How do you stack up?

After a three-year absence, our annual Salary Survey is back to help you determine how your income stacks up in relation to other maintenance and reliability professionals in today’s industrial arena.

Please note that our 2007 Salary Survey goes well beyond anecdotal information to reflect concrete data regarding the actual state of this industry’s employment marketplace. The data we used to compile this survey was obtained from a random sample of Maintenance Technology and Lubrication Management & Technology readers who completed an anonymous on-line survey. We believe the survey findings reported here to be both accurate and representative of what’s happening in the maintenance and reliability community.

A basic profile
When Maintenance Technology conducted its first salary survey in 1998, average respondent income was $58,748, (including overtime and bonus, which all averages in our findings reflect). Nine years later, the average expected income for 2007 is $86,251—a 32% increase. This also reflects a 3% increase from the average salary of $83,678 that this year’s respondents report having received in 2006.

Furthermore, expected income for 2007 is ranging from $26,000 to $250,000, in comparison to a range of $12,000 to $160,000 in 1998 and $26,000 to $235,000 in 2006.

 

For those paid on an hourly basis—23.68% of our survey respondents—the average pay rate is $28.30 per hour, equating to an average expected 2007 income of $69,238.

As shown in Fig. 1, the highest percentage of our respondents report an expected 2007 income in the $70,000 to $79,999 range. This also is where the median income, $78,000, is found.

Changes with age
Age of our survey respondents ranged from 26 to 71 years old, with an average of 50.2 years. Half of them are between 45 and 56 years of age. In addition, a large number of respondents are seasoned veterans, having spent an average of 22.2 years working in their fields.

Based on age, the average income increased from $63,333.33 for respondents in their 20s to a high of $88,674 for those in their 50s. For those in their 60s and above, the average reported income dropped by slightly more than $4000. More results are shown in Fig. 2.

The learning curve
Of the survey respondents, 30.2% indicate a trade school diploma as their highest level of educational achievement; 25.9% have a two-year community college degree; 34.5% have a four-year college or university degree; and 9.6% have a masters or doctorate graduate university degree. So how do these educational levels relate to salary compensation?

Typically, the higher the level of education respondents have achieved, the higher their average level of income is. Trade school graduates report an average 2007 income of $74,355; two-year community college graduates report $77,439; four-year college or university graduates report $97,375; those with advanced degrees report $107,301. Each level of education includes a wide range of salaries, as depicted in Fig. 3.

Outside of a formal education, 19% of respondents also hold one or more professional licenses or certifications, which include P.E., CMRP, CPMM and CPE. The average income for Professional Engineers (P.E.) is $113,316; the average income for those designated solely as Certified Maintenance and Reliability Professionals (CMRP) is $85,340; the average income for those designated solely as AFE Certified Plant Maintenance Managers (CPMM) is $77,000. (Note: Too small a number of AFE Certified Plant Engineers (CPE) or those with combinations of certification provided their expected 2007 income to report an accurate average.)

Income by facility size
Survey respondents were asked to indicate the number of workers at their location of employment. The results were as follows: 12% are employed at facilities of one to 49 employees; 9% at facilities of 50 to 99 employees; 20% at facilities of 100 to 249 employees; 18% at facilities of 250 to 499 employees; 13% at facilities of 500 to 999 employees; and 28% at facilities with 1000 or more employees.

Related to salary, respondents working at facilities of 50 to 99 employees report the lowest average income at $68,998. Respondents working at facilities with 1000 or more employees record the highest average salary at $96,748. Fig. 4 displays the results from the remaining facility sizes.

Industry type
We also asked survey respondents to specify the industry sector of their company/facility. The results, combined into five general categories derived from the North American Industry Classification System (NAICS), include processing, manufacturing, utilities, service and nonindustrial industries.

Based on responses, 40.1% of respondents work in processing industries; 22.1% in manufacturing; 14.6% in utilities; 6.8% in services; and 16.3% in non-industrial. Those in processing report the highest average salary at $94,346. The lowest average salary based on industry, $71,673, is reported by those working in the non-industrial sector. Fig. 5 displays full results.

 

Who’s doing what
Our survey asked respondents to indicate their level of work involvement. Results show that 13% chose corporate or multiplant; 15% plant or facility manager; 21% reliability or maintenance manager; 6% reliability engineer; 6% reliability technician; 7% maintenance engineer; 9% maintenance technician; 13% supervisor; 10% “other.”

As might have been expected, the average expected income for 2007 was the highest for those involved with corporate or multiplant levels, at $104,746, as is seen in Fig. 6. This is the same result we have found in the seven previous years of our survey. Those involved at the level of maintenance technician indicate the lowest average income at $62,100.

In the nuclear power industry, the primary mission of a root-cause investigation is to understand how and why a failure or a condition adverse to quality has occurred so that it can be prevented from recurring. This is a good practice for many reasons—and a lawful requirement mandated by 10CFR50, Appendix B, Criterion XVI.

To successfully carry out this mission, a root-cause investigation needs to be evidence-driven in accordance with a rigorous application of the bedrock of all root-cause methodologies: the Scientific Method. Consistent with the Scientific Method, underlying assumptions have to be questioned and conclusions have to be consistent with the available evidence, as well as with proven scientific facts and principles.

Sometimes root-cause investigations fail to fulfill their primary mission and the failure recurs. In that regard, diagnosing the root cause of root-cause investigation failures is, in itself, an interesting topic. Here are three common reasons why some root-cause investigations fail their mission.

Reason #1: The Tail Wagging the Dog
As a root-cause investigation proceeds and information about the failure event accumulates, some initial hypotheses can be readily falsified by the preliminary evidence and dismissed from consideration. The diminished pool of remaining hypotheses will likely have some attributes in common. More work is then usually needed to uncover additional evidence to discriminate which of the remaining hypotheses specifically apply.

At this point in the investigation, it may become apparent what the final root cause might be—especially if the remaining pool of hypotheses is small and they all share several important attributes. At the same time, it also becomes apparent what the corresponding corrective actions might be.

By anticipating which corrective actions are more palatable to the client or management, the investigator may begin to unconsciously—or perhaps even consciously—steer the remainder of the investigation to arrive at a root cause whose corresponding corrective actions are less troublesome.

Evidence that appears to support the root cause and lead to more palatable corrective actions is actively sought, while evidence that might falsify the favored root cause is not actively sought. Evidence that could falsify a favored root cause may be dismissed as being irrelevant or not needed. It may be tacitly assumed to not exist, to have disappeared or to be too hard or too expensive to find. It may even just be ignored because so much evidence already exists to support the favored root cause that the investigator presumes he already has the answer.

In logic, this is defined as an a priori methodology. This is where an outcome or conclusion is decided beforehand, and the subsequent investigation is conducted to find support for the foregone conclusion. In this case, the investigator has decided what corrective actions he wants based on convenience to his client or management. Subsequently, he uses the remainder of the investigation to seek evidence that points to a root-cause that corresponds to the corrective actions he desires.

What Really Happened: Failure Of A Zener Diode This X-ray radiograph shows a 1N752A-type Zener diode that was manufactured without a die-attach at one end of the die, and with only marginal die-attach at the other end. This die-attach defi ciency caused the component to fail unexpectedly in an intermittent fashion. In turn, this led to a failure in the voltage regulator system of an emergency diesel generator system, causing it to be temporarily taken out of service. The failure of this Zener diode occurred in a circuit board that had seen less than 40 hours of actual service time, although the circuit board itself was over 27 years old. It had been a spare board kept in inventory. Going to this level of detail to gather evidence might seem extreme. This particular evidence, however, was fundamental to validating the hypothesis that the rootcause in this case was a random failure due to a manufacturing defect, and falsifying the hypothesis that the failure was caused by an infant mortality type failure. In the nuclear power industry, this distinction is significant.

Here is an example: A close-call accident involved overturning a large, heavy, lead-lined box mounted on a relatively tall, small-wheeled cart. The root-cause investigation team found that the box and wheeled cart combination was intrinsically unstable. The top-heavy cart easily tipped when the cart was moved and the front wheels had to swivel, or when the cart was rolled over a carpet edge or floor expansion joint.

The investigation team also found that the personnel who moved the cart in the course of doing cleaning work in the area had done so in violation of an obviously posted sign. The sign stated that prior to moving the cart a supervisor was to be contacted. The personnel, however, inadvertently moved the cart—without contacting a supervisor—in order to clean under and around it.

The easy corrective actions in this case would be to chastise the personnel for not following the posted rules and to strengthen work rule adherence through training and administrative permissions. There is ample evidence to back-fit a root cause to support these actions. Also, such a root-cause finding—and its corresponding corrective actions—are consistent with what everyone else in the industry has done to address the problem, as noted in ample operational experience reports. In the nuclear power industry, the "bandwagon" effect of doing what other plants are doing is very strong.

In short, the aforementioned corrective actions are attractive because they appeal to notions of personal accountability, are cheap to do and can quickly dispose of the problem. Consequently, the root cause of the close-call accident was that the workers failed to follow the rules.

Unfortunately, when the cart and box combination is rolled to a new location, the same problem could recur. The procedure change and additional training might not have fixed the instability problem. While the new administrative permissions and additional training could reduce the probability of recurrence, they would not necessarily eliminate it. When the cart is rolled many times to new locations, it is probable that the problem will eventually recur and perhaps cause a significant injury. This situation is similar to the hockey analogy of "shots on goal." Even the best goalkeeper can be scored upon if there are enough shots on goal.

Reason #2: Putting Lipstick on a Corpse
In this instance, a failure event has already been successfully investigated. A root cause supported by ample evidence has been determined. Vigorous attempts to falsify the root-cause conclusion have failed. Ok…so far, so good.

On the other hand, perhaps the root-cause conclusion is related to a deficiency involving a friend of the investigator, a manager known to be vindictive and sensitive to criticism or some company entity that, because of previous problems, can't bear criticism. The latter could include an individual that might get fired if he is found to have caused the problem, an organization that might be fined or sued for violating a regulation or law or a department that might be re-organized or eliminated for repeatedly causing problems. In other words, the root-cause investigator is aware that the actual consequences of identifying and documenting the root cause may be greater than just the corrective actions themselves.

When faced with this dilemma, some investigators attempt to "word-smith" the root-cause report in an eff ort to minimize perceived negative findings and to emphasize perceived positive findings. Instead of using plain, factually descriptive language to describe what occurred, less precise and more positive- sounding language is used. This is called "word-smithing" a report.

"Word-smithed" reports are relatively easy to spot. Instead of using plain modifiers like "deficient" or "inadequate" to describe a process, euphemistic phrases like "less than sufficient" or "less than adequate" are used. Instead of reporting that a component has failed a surveillance test, the component is reported to have "met 95% of its expected goals." Likewise, instead of reporting that a fire occurred, it is reported that there was a "minor oxidation-reduction reaction that was temporarily unsupervised."

In such cases, the root-cause report becomes a quasi-public relations document that sometimes has conflicting purposes. Since it is a root-cause report, its primary purpose is supposed to be a no-nonsense, fact-based document that details what went wrong and how to fix it. However, a secondary, perhaps conflicting, purpose is introduced when the same document is used to convince the reader that the failure event and its root cause are not nearly as significant or serious as the reader might otherwise think.

With respect to recurrence, there are two problems with "word-smithing" a root-cause report. Corrective actions work best when they are specific and targeted. A diluted or minimized root-cause, however, is oft en matched to a diluted or minimized corrective action. There is a strong analogy to the practice of medicine in this instance. When a person has an infection, if the degree of infection is underestimated, the medicine dose may be insufficient and the infection may come back.

The second problem is that by putting a positive "spin" on the problem, management may not properly support what needs to be done to fix the problem. Thus, the report succeeds in convincing its audience that the failure event is not a serious problem.

Reason #3: Elementary My Dear Watson
In some ways, root-cause investigations are a lot like "whodunit" novels. Some plant personnel simply can't resist making a guess about what caused the failure in the same way that mystery buffs often try to second guess who will be revealed to be the murderer at the end of the story. It certainly is fun for a person—and perhaps even a point of pride—if his/her initial guess turns out to be right. Unfortunately, there are circumstances when such a guess can jeopardize the integrity of a root-cause investigation.

The circumstances are as follows:

• The guess is made by a senior manager involved in the root-cause process.
• The plant has an authoritarian, chain-of-command style organization.
• The management culture puts a high premium on being "right," and has a zero-defects attitude about being "wrong." the scenario goes something like this:
• A failure event occurs or a condition adverse to quality is discovered.
• Some preliminary data is quickly gathered about conditions in the plant when the failure occurred.
• From this preliminary data, a senior manager guesses that the root-cause will likely be x, because:
  • (1) he/she was once at a plant where the same thing occurred; or
  • (2) applying his/her own engineering acumen, he/she deduces the nature of the failure from the preliminary data, like a Sherlock Holmes or a Miss Marple.
• Not being particularly eager to prove their senior manager wrong and deal with the consequences, the root-cause team looks for information that supports the manager's hypothesis.
• Not surprisingly, the teams find some of this supporting information; the presumption is then made that the cause has been found and field work ceases.
• A report is prepared, submitted and approved, possibly by the same senior manager that made the Sherlockian guess.
• The senior manager takes a bow, once again proving why he is a senior manager.

The deficiency in this scenario that can lead to recurrence is the fact that falsification of the favored hypothesis was not pursued. Once a cause was presumed to have been found, significant evidence gathering ceased. (Why waste resources when we already have the answer?) As a result, evidence that may have falsified the hypothesis, or perhaps supported an alternate hypothesis, was left in the field. Again, this is another example of an a priori methodology: where the de facto purpose of the investigation is to gather information that supports the favored hypothesis.

In this regard, there is a famous experiment about directed observation that applies. Test subjects in the experiment were told to watch a volleyball game carefully because they would be questioned about how many times the volleyballs would be tipped into to air by the participants. This they did.

In fact, the test subjects did this so well, they ignored a person dressed in a gorilla suit who sauntered through the gaggle of volleyball players as they played. When the test subjects were asked about what they had observed, they all reported dutifully the number of times the ball was tipped but no one mentioned the gorilla. When they were told about the gorilla, they were incredulous and did not believe that they had missed seeing a gorilla…until they were shown the tape a second time. At that point, they all observed the gorilla. MT

Randall Noon is currently a root-cause team leader at Cooper Nuclear Station. A licensed professional engineer in both the United States and Canada, he has been investigating failures for 30 years. Noon is the author of several articles and texts on failure analysis, including the Engineering Analysis of Fires and Explosions and Forensic Engineering Investigations. He also has contributed two chapters to the popular college text, Forensic Science, edited by James and Nordby. E-mail: rknoon@nppd.com

Stainless steel is the fastest growing metal market in the world, not only for its popularity in kitchen appliances but industrial applications as well. The Outokumpu Stainless Hot Rolled Plate (HRP) factory in Degerfors, Sweden serves the latter.

Outokumpu Stainless is one of the world's four largest producers of hot rolled plate with one of the widest range of products and steel grades within the stainless steel industry. Our Degerfors factory alone produces 120 thousand tons per year. The plates are extremely resistant to corrosion and wear, making them popular in challenging applications and environments including pulp & paper, oil & gas, chemicals and power generation.

Because our customers depend on us to keep our production lines running, we looked outside the company for maintenance assistance. Gradually, we increased our reliance on contracted maintenance services ("outsourcing") and raised the bar to higher standards. The strategy has led to our current full-service, performance-oriented, maintenance- management agreement.

Outsourcing evolution
When the plant opened in 1996, we had extensive knowledge of stainless steel production, but little in terms of equipment maintenance. To alleviate the burden, some maintenance tasks were managed internally and others were contracted out on an hourly basis to various service providers. At its peak, about 100 individuals were involved in plant maintenance activities.

For three years, our operational effectiveness (OE) and production availability were high, yet our maintenance costs were prohibitive. The break/fix approach was expensive, and tensions ran high between maintenance and production personnel.

By 1999, maintenance was still not a core competency for us. Thus, we resolved to forgo all maintenance responsibility and consolidate it under a single, more conducive contract. We chose to contract 100% of our corrective and preventive maintenance activities in Degerfors under a jointly developed, hourly-based ABB Full Service maintenance agreement.

The agreement established performance objectives that subjected the service provider to bonuses or penalties depending on its performance. This approach allowed the contractor to share the risks and rewards of plant maintenance, and provided the incentive to continuously improve performance. Soon, we had approximately 65 ABB Service employees working at the plant.

In 2001, the arrangement was transitioned from hourly rates to a fixed price so that we could have more predictable budgets. Performance incentives still provided rewards or penalties depending on the results achieved.

By 2006, an enhanced four-year contract was negotiated. Plant management, production and maintenance personnel were all involved in developing the new agreement, setting target performance levels and specifying when and how long the machines would be stopped should corrective maintenance be required. More services were added to the agreement and caps were established on certain service costs.

We began conducting weekly management meetings with the provider to assess equipment status, production schedules and maintenance priorities. In our plant, production is moving all the time and production priorities change every week. When corrective action is required, maintenance personnel are reassigned to the highest priority tasks based on equipment criticality and bottleneck location. Our priority classifications are as follows:

• Level One – Accident risk: Equipment problems that pose a potential danger for the operator are of first concern. All other maintenance is stopped.
• Level Two – Outage in the hot part of production: Equipment trouble in the hot rolling mill can destroy a lot of materials and suffer the greatest costs.
• Level Three – Process transition: Bottlenecks in moving from one machine to another affect production throughput and must be minimized.

Operational benefits
One of the greatest advantages of our maintenance outsourcing agreement is having another company at the table. It provides a new way of thinking about maintenance and a new perspective on problems. We can be experts at producing plates, while our contracted service provider can focus on keeping our machines running. Moreover, we can put much greater pressure on an outside party than we would on our own employees.

When the Maximo system was brought in, we saw a tremendous improvement. Our previous maintenance management system was wholly inadequate, and work instructions were often written on paper. Now, all of ABB's maintenance practices and records are tracked in the new system. Outokumpu also uses the system to manage spare parts.

Our costs have decreased as a result of streamlined operations and better maintenance planning, giving us the ability to do more with less. Maintenance costs now are on par with other departments, while OE and production availability remain high.

The four-year agreement duration also has given our service provider greater incentive to invest more in its maintenance processes, since it now can be assured of seeing the return on its investments before the contract expires.

Convincing results
Among other things, since 2001, our full service maintenance agreement has helped us:

• Decrease our total maintenance cost by 24%
• Reduce our maintenance cost per produced ton by 58%
• Achieve our current customer satisfaction score of 91.2%

What's most impressive is that, in the same timeframe, we've raised our production volume by 80%—to 120 thousand tons. In 2006, as part of our agreement, we added overall equipment effectiveness (OEE) as an additional metric. Much more preventive work is being done now, and the work is being completed more quickly and efficiently.

Ongoing improvement
The performance incentives in the full-service agreement benefit Outokumpu through ongoing operational improvements and the service provider through financial rewards. As such, we are always trying to do things better. Utilizing the industry's best maintenance practices and systems will facilitate our mutual desire for continuous improvement.

Our greatest test was convincing the corporate office of our strategy's value. Because Outokumpu's vision is to be number one in stainless, with success based on operational effectiveness, management questioned whether maintenance outsourcing fit with our corporate goals. Once we explained the arrangement, including the benchmarking, the best practices and the bottom-line benefits, management supported our approach. By entrusting an outside service provider with all our maintenance requirements under the full-service, performance-driven agreement, Outokumpu corporate and the Degerfors plant can look forward to further cost reductions and operational improvements. MT

Mladen Perkovic is production manager for the Outokumpu Stainless Hot Rolled Plate (HRP) Plant.

That’s right. I want to thank our loyal readers, contributors and partners for a great run. This issue marks the end of Maintenance Technology’s special year-long 20th Anniversary Celebration. It also marks the beginning of our next 20 successful years of publishing. Projecting our future (and also being a grandfather), I think the words of Buzz Lightyear sum it up best: “To infinity, and beyond…”

Maintenance Technology was founded 20 years ago by a dedicated team of individuals who saw a need to serve maintenance practitioners by promoting Best Practices throughout industry. For the past two decades, that’s exactly what we’ve been doing—delivering the best-read, most-preferred, monthly, independent and audited publication in the market to ever-savvier, increasingly hard-working maintenance and reliability professionals across virtually all industry sectors. Supported by practitioners, industry experts and suppliers who are willing to share their knowledge, skills, experience and technologies/methodologies with you, this powerful, high-quality editorial is now—and always will be—designed to help our readers successfully meet their capacity assurance needs.

Although many things have changed over the past 20 years, Maintenance Technology has stayed the course, never deviating from our primary mission and strategies. We serve our readers. We engage our readers. We listen to our readers. Doing so has led us to grow in some unexpected and exciting ways.

Five years ago, we developed and began presenting Maintenance & Reliability Technology Summit (MARTS) an annual professional development program that has become one of the premier learning and networking events for the maintenance and reliability community. In 2004, we began publishing another standalone magazine, now known as Lubrication Management & Technology, dedicated to improving industrial lubrication programs. More recently, we have begun producing regular quarterly supplements like Utilities Manager and The Fundamentals, focusing, respectively, on energy efficiency and a backto- basics approach to maintenance and reliability. These are just a few of the many things that have helped Maintenance Technology maintain its position as the leading publication in our market. Along with other yet-to-be-determined offerings, they will be among the things that help us grow and better serve you and future generations of maintenance and reliability professionals over the next 20 years.

Because we could not have gotten where we are today without the help of many individuals and organizations, we put a lot of stock in giving something back “to the good of the order.” For example, while building Maintenance Technology into the publication that it is today, we were one of the founding entities of the Society for Maintenance and Reliability Practitioners (SMRP). We also continue to be strongly involved in industry activities such as MER (the Maintenance Excellence Roundtable), NAME/FIME (the North American Maintenance Excellence Award), STLE, ARC, MIMOSA and FSA (the Fluid Sealing Association), among others. We view our participation in these diverse types of initiatives as something that truly helps set a reader-driven publication such as Maintenance Technology ahead of the pack—and that’s a place we always want to be!

It’s been a tremendous 20 years. All of those involved with Maintenance Technology, including past and present staff, contributors, associations, valued advertising partners and you—our loyal readers—deserve my heartfelt appreciation. Again, thank you all! MT

Ken Bannister, Contributing Editor

"ABC Corporation of Smalltown, USA, today announced a manufacturing sales order worth over $100 million…"

Such announcements are commonplace in today's business press, leaving little doubt that the sales and marketing department are still revered as corporate heroes when a large sales order is closed. Getting that order to the customer, however, as ordered, on time and with first-time quality requires the effort of many unsung heroes within the plant.

From the post-war 1950s to the 1980s, North American corporate philosophy surrounding the sales process often was "close the sale and we'll worry about design, quality and delivery later." Since many sales were closed in a wine-anddine forum, and in a somewhat indiscriminate consumer culture of the time that tended to be accepting of poor design, quality and delivery, countless corporations were successful in spite of themselves.

That all changed when the Japanese singlehandedly raised the bar, having been attributed largely with the responsibility for raising consumer awareness and expectations surrounding quality and service throughout the 1970s and 1980s. This state of affairs finally forced the North American industrial giants into compete mode by the 1990s. New heightened consumer awareness resulted in an intelligent customer who was unafraid to demand quality products at reasonable prices, delivered on time. Competing in this new world order forced many corporations to rethink their sales strategies.

A renewed sales approach
To be considered a viable contender in today's marketplace, a corporation must attain quality assurance certification. Many customers demand ISO 9000 or TS 16949 certification (quality assurance through audited documentation and procedural control to a defined international standard) as a contract bidding prerequisite.

With ISO/QS 9000 certification, the customer is assured that a qualified maintenance program is in effect and also that manufacturing equipment is being maintained to a specified level of reliability. Through this link, the maintenance department is now established integrally with the sales department.

Building on this newly established integration, the modern-day sales approach utilizes a corporate team effort to put together a winning sales proposal. This sales process calls for the salesperson to listen and document the customer's requirements exactly, so that these requirements can then be reviewed by a multi-faceted manufacturing and sales team comprising members of the finance, engineering, production, purchasing and maintenance departments. Because many sales contracts contain penalty clauses for poor quality and poor delivery, the sales team must ensure corporate capability to attain and maintain a sustained level of production throughput for the duration of the sales contract. This only can be assured by the maintenance department.

Further links between maintenance and the sales force recently have been established through the implementation of Lean Manufacturing initiatives, in which the sales department is no longer called upon to fire-sell surplus "made for inventory" product. Instead, sales is made intimately aware of current long-term/short-term surplus manufacturing equipment capability that can be tapped into and sold competitively with high profit margins and low manufacturing cost in a pull manufacturing environment.

Maintenance facilitating the sales process
Although maintenance rarely involves itself in the sales process, it can assist in the sales effort through the provision of reliable equipment performance information, such as:

Throughput capability report—Covering the manufacturing equipment or line intended to produce the new parts, this is essential information in determining the ability to deliver the requested product volumes. Through the process of analyzing the specified equipment maintenance history, a detailed downtime record is used to compare against the machine's design throughput figure, so that a true throughput measure can be predicted. If throughput requirement is more than capability for the plant in question, alternate manufacturing requirements will be needed to provide quality and delivery.

Maintenance cost report—Once again, knowing what equipment will be used to manufacture the product helps the maintenance department establish past maintenance costs for the equipment in question. These cost figures are then averaged out and a projected maintenance cost can be established for the proposed production contract term. Such projections allow the sales department to accurately calculate operation and manufacturing costs. In turn, controlled costs allow corporations to lower profit margins and be more competitive.

Maintenance health reports—Standard maintenance health reports, such as Availability, Reliability, PM completion, Overall Equipment Effectiveness, etc., are all excellent reports for the sales department to have in its possession. Providing these reports are favorable; they can be used within a sales presentation to bolster confidence in the corporation's ability to deliver the required goods and services being bid on to potential and actual customers.

In the course of the sales process, maintenance also may be called upon to directly interface with potential customers and provide them with a tour of the maintenance facilities, as well as present an overview of the maintenance process.

The benefits
Establishing a partnership with the sales department allows the maintenance department to once again be recognized as an entity within the corporation. Being aware of pending contracts lets maintenance better plan any equipment maintenance and overhaul requirement so as to be ready for the production contract in the event the sales department is successful. Involvement in the front end of the sales process also allows maintenance to keep in check the possibility of ‘overselling' the plant design capacity. Inattention to this element can accelerate maintenance demands quickly, increasing equipment downtime and, ultimately, leading to corporate losses.

Ken Bannister is lead partner and principal consultant for Engtech Industries, Inc. Telephone: (519) 469-9173; e-mail: kbannister@engtechindustries.com

The enabling toolbox to attain the highest levels of capacity assurance has been filling up with a variety of solutions. From our perspective “sustainability” tops the list.

The International Trade Association (ITA) defines sustainability as “the creation of manufactured products using processes that are nonpolluting, conserve energy and safe for employees, communities and consumers.” Specialized technologies and expertise offers various avenues to turn the process into progress.

Sustainability programs among manufacturers have accrued significant gains with successes large and small. For those most engaged, organizations have been able to engineer annual reductions in CO2 emissions and water consumption; made recycling of scrap metal routine and profitable; equipped pumps with frequency controls to promote energy efficiency; and have moved from harmful chemicals and lubricants to environmentally friendly and lubrication-free solutions.

Countless other examples abound to affirm the viability, achievability and rationale of sustainability programs, regardless of a manufacturer’s size or industry. For those seeking to make their own inroads with sustainability, these keys for success can help unlock programs and keep them moving forward:

• Evaluate your operations top-to-bottom. Energy and environmental analyses can be conducted to pinpoint areas where high energy consumption is the norm and check chemical treatments, lubrication use and other operating processes to determine environmental risk. Improvements linked to opportunities then can be introduced, based on remedial action recommendations.
• Establish goals and targets. Analyses additionally provide benchmarking data for arriving at realistic objectives and measuring subsequent results that contribute to the business goals.
• Apply new technologies. Targets of opportunity for sustainability improvements can be found in virtually every piece of equipment and among all applications and processes within industry. Great strides have been made in the evolution of relevant technologies and these can be enlisted as appropriate to meet particular sustainability goals.
• Promote equipment reliability. Practices aimed at improving and enhancing the reliability and efficiency of assets can pay big dividends. Regularly monitored and well-maintained equipment can save energy, increase uptime, drive profitability and advance sustainability objectives.
• Manage information effectively. Data is “king” for documenting and quantifying sustainability efforts and satisfying mandated obligations for environmental, health and safety compliance reporting. Customized Web-based EHS (environmental, health and safety) information management systems offer solutions to electronically automate key EHS functions, including audits, chemicals management, regulatory reporting and sustainability metrics. This can drastically reduce the time and money usually spent in collecting, analyzing, re-formatting and preparing data. Capabilities expand with live CO2 footprint tracking and performance tracking and measurement.

Immediately and over time, sustainability programs can allow operations to reap the inherent rewards in reduced operating costs, increased productivity, generated energy savings and enhanced asset reliability. In today’s competitive business arena, making room for sustainability programs simply makes perfect sense.

Bob Williamson, Contributing Editor

“What is the formula for determining the optimum maintenance staffing level for our plant?” someone recently asked. I have asked the very same question about optimum maintenance staffing levels for over 20 years. It’s a tough one to answer.

Unfortunately, there seems to be no logical or easy answer to this seemingly straightforward question. I’m sure there may be some readers out there who have mastered this mythical formula, or have come up with an effective method for their respective situations. Still, I feel obligated to share my own thoughts as to the difficulties associated with maintenance staffing levels as we wrestle with maintenance costs, reliability improvement and an era of skills shortages.

Plant staffing levels can be determined by a number of different methods. For example, determining the number of operators for machinery, material handling or control stations is a relatively simple task due to the number of operating positions, job tasks, narrowly focused scope of work and specific but limited skills and knowledge requirements. On the other hand, determining the number of maintenance mechanics or technicians is not so simple—in fact, in some plants it is extremely complex. I’ve heard of formulas based on headcount per installed horsepower, mechanics per replacement cost or technicians per square foot. Why don’t these work across the board? Here are the BIG variables that affect maintenance staffing level decisions:

Variable #1 – scope of work
The breadth and depth of job-performance requirements varies widely in today’s industries from extremely-narrow, single-task, repetitive job tasks to broad, multi-skill job roles. Maintenance is rarely a narrowly focused job role, either geographically in the plant or intellectually in the skills and knowledge requirements.

In general, maintenance includes very broad core job skills and knowledge such as in-depth principles of mechanical, machine repair, electrical, instrumentation/controls, machining, etc. We also must include equipment-specific, facilityspecific task skills and knowledge. Then there are advanced trouble-shooting and problem-solving skills and knowledge. Furthermore, we cannot ignore the specialized skills and knowledge requirements for condition monitoring and predictive maintenance. In many plants I’ve often heard this scope of work scenario described as “an inch wide and a mile deep for equipment operators” and “a mile wide and a mile deep for maintenance technicians.”

Variable #2 – individual competency
The second big variable for maintenance headcount is the skill set of each person—individual competency. If all maintenance people had the same level of skills and knowledge, there could be an easy answer to the question of “optimum maintenance staffing levels.” BUT, there is a lack of comprehensive skills standards as applied to industrial maintenance job roles especially in the areas of equipment-specific tasks.

Today, many plants do not FORMALLY train and qualify all of the maintenance staff to address the maintenance and reliability needs of the plant’s equipment and facilities. Why is it that equipment and plant operators typically receive job- and task-specific training and qualification, but the maintenance staff rarely does? There seems to be an “assumed” higher level of maintenance competency than what actually exists, or an over-simplification of the job roles that gets some plants into deep trouble.

Variable #3 – equipment reliability
Highly reliable plants and equipment can be managed with relatively fewer maintenance technicians than comparable highly reactive plants. If you have a very RELIABLE plant and equipment, the maintenance workloads are usually very well defined in terms of scope, skills and duration due to planned, scheduled and preventive/predictive maintenance. And, when jobs are assigned only to qualified maintenance technicians, accurate staffing level decisions are much easier.

Reactive or repair-based maintenance is highly unpredictable in terms of scope, skills and duration due to high levels of unplanned, unscheduled, reactive work loads. A wide variety of individual competencies also adds to the sporadic nature of equipment problems. It is almost impossible to plan anything day-to-day, let alone the proper staffing levels.

Variable #4 – historical information
Work orders capture a whole host of information about maintenance and repair work, including problems, causes, corrective action and laborhours worked by named technicians. Sadly, there is a huge void of decision-making information if the plant or facility does not use work orders or does not reinforce the need for accurate equipment and work history information. Staffing levels are arbitrary, repetitive problems are not identified, common causes are overlooked, improper actions and rework go unnoticed.

An analysis of comprehensive maintenance work order information often reveals that most of the root causes of the perceived “maintenance problems” with the plant and equipment are outside the direct control of the maintenance staff. Other departments and/or personnel that must be involved in improving “maintenance” include maintenance and repair parts procurement, inventory control, operations management and staff, process technicians, engineering, production scheduling, etc. The maintenance department alone cannot make equipment reliable.

Variable #5 – maintenance & reliability trends
Many business decision-makers do not have enough information to truly understand maintenance and the BIG maintenance staffing variables outlined here. Regrettably, for decades “maintenance” has been treated as an overhead expense line item and a “non-value-adding” activity in many business operations. Some business decision-makers also perceive maintenance technicians as “fixers” rather than “preventers” of equipment problems.

Current information about maintenance workforce demographics, hiring trends, retirement forecasting and knowledge retention often is overlooked, not fully understood and/or not factored into the staffing level decisions. More and more plants will experience higher maintenance costs and higher turnover of top skilled people as the “maintenance skills shortages” grips our nation’s business and industries, furthering the inability to determine proper maintenance staffing levels.

Real-world example
I spent time recently with a business that modifies and tunes high-performing street motorcycles using a custom-designed chassis dynamometer. One of the shop’s modifications includes changing from chain-driven to gear-driven cams and new push rods. This design eliminates high amounts of friction and improves engine torque and horsepower measured at the rear wheel. In one case, when the new cams and push rods were installed in a customer’s motorcycle, the dynamometer test revealed a sizeable LOSS in torque and horsepower!

The highly experienced mechanic who installed the new gear cams and push rods did not notice anything unusual during the assembly and adjustment. I noticed that he was impressively meticulous about his work. There was, however, a real problem somewhere.

A second highly experienced mechanic disassembled and inspected the new cams and push rods and immediately spotted a problem: The wrong push rods were installed!

When the two mechanics met and discussed the findings, they discovered that the new push rods were in the WRONG package from the manufacturer! Even though the shrink-wrapped labeled package indicated otherwise, they were not the correct parts for this engine and were about 5/8” too short. Because of their design, the first mechanic was able to easily adjust the push rod length and set the proper valve clearance. But, adjusted to their maximum limit, they flexed while running under load, limiting the valve travel and causing a reduction in torque and horsepower.

In this real-world example, both mechanics were trained, experienced and more than adequately skilled to work on the motorcycle in question. The second mechanic, though, had been factory trained and certified during the past 10 years. Both had made this particular modification hundreds of times, yet what was clear to the certified mechanic was overlooked by the uncertified experienced mechanic. While the root cause of the problem was obvious, neither mechanic had ever experienced mislabeled parts from this specific high-performance parts manufacturer.

Such subtle differences in today’s mechanics’ skill sets—or competencies—can create or eliminate maintenance-induced failures and the need to rework a recently completed job. Think how much difference there is among all the maintenance technicians’ skill sets and competencies in your plant or facility.

Plants and equipment would be highly reliable with a relatively smaller maintenance workforce if everyone were highly skilled and knowledgeable and only assigned to jobs that they were qualified to perform—right the first time. Gee, aircraft mechanics and top NASCAR race team mechanics do that all the time.

Bottom line
An analysis of plant equipment, chronic and sporadic problems and overall equipment effectiveness losses can lead to the determination of the required “skill sets” to achieve optimum levels of equipment performance and reliability. Until these “skill sets” become core competencies for maintenance staffing, I believe it is IMPOSSIBLE to use a formula to determine the optimum maintenance staffing levels.

To many plant-level professionals, the term “asset management” is synonymous with equipment maintenance or field device management. But, to business managers responsible for process manufacturing operations, asset management implies the effective deployment of all assets within their operational domain to meet business objectives. These assets include: plant equipment, energy, raw material, products, people, facilities, instrumentation, automation, information and even time.

Maintenance is certainly one important element of an overall asset management solution set, but maintenance improvements by themselves will not maximize the business performance of the manufacturing asset base. In fact, attempting to optimize plant operations while improving maintenance, without regard to the operational consequences, or vice versa, actually can degrade performance. True business optimization requires a holistic balance of tradeoffs between maintenance and operations—as dictated by the corporate profitability strategy, not by the isolated improvement objectives of either maintenance or operations.

Five years ago or so, balancing requirements of maintenance and operations across the enterprise, affordably, was more of a vision than a reality. Today, serviceoriented architectures, systems integration standards, high-speed networking and numerous other technologies have advanced to a new level of performance and economy, making platforms that unify maintenance, plant floor, business and customer systems a reality.

Balancing availability and utilization
Maintenance functions typically strive to maximize asset availability while the operations functions strive for maximum asset utilization. Although there are no industry-standard definitions, asset availability often is represented by the percentage of time the plant asset base is available for operating over any given period of time, and as the percentage of total output from an asset base divided by the theoretical maximum output over a period of time. Because it is impossible for a refinery to be 100% available and 100% utilized simultaneously, the only way to manage assets from a business perspective is to manage both holistically, according to the business measures such as plant profitability. Asset performance management (APM) is the holistic approach that has emerged to describe the practice of balancing availability and utilization around business performance.

APM includes what traditionally has been known as enterprise asset management (EAM), but that is only part of it. Although the initial EAM vision did indeed seek to bring together business and operations enterprise functions, in practice, it has focused primarily on optimizing availability. Traditional EAM tactics for optimizing availability have included maintenance repair and operating (MRO) inventory management, condition based maintenance and preventive maintenance. Although these are indeed critical to refinery health and success, most of the maintenance strategies are still at the isolated unit and equipment level—and of these, many are focused even further on instruments and valves.

Similarly, tactics to optimize utilization have been traditionally isolated at the advanced process control level, including such technologies as model-based production management, multivariable control, recipe management and online process optimization.

Moving to APM
Moving from traditional EAM to APM requires extending the scope of traditional EAM systems in at least two directions. One direction involves seamless integration with business and customer systems; the other involves seamless integration with production and production optimization systems. This extension includes the following elements:

• Comprehensive enterprise asset management system with reliability centered maintenance (RCM) and extended condition based maintenance
• Integrated and open field device management
• Condition management, not just condition monitoring
• Knowledge management
• Decision support
• Effective measurement systems
• Improved communication and collaboration

EAM is still pivotal
EAM remains very much at the core of APM. It automates management of the complete lifecycle of plant assets from the device level up to the overall plan. The core capabilities of the Enterprise Asset Manager are as follows:

• Work requests/work order management
• Work crew planning and scheduling
• Workflow and approvals management
• Maintenance cost tracking and analysis
• MRO inventory management, shipping/receiving and supply chain management
• Contract and warranty management

The EAM function also provides a central collection point and access point for asset information, such as cost, performance and history. The EAM component adds value in a number of key operational areas. A study by A.T. Kearney found that the following benefits are achievable from EAM:

• Improved throughput—uptime increases within 10-25%
• Reduced operating expense—labor productivity increased between 20 and 30%; overtime costs reduced between 20 and 35%
• Reduced inventory—MRO Inventories reduced 15-25%; with MRO Supply Chain Savings between 15 and 25%
• General improvements—improvement in health, safety and environmental compliance, reductions in costs of outages and emergency repairs

Benefits such as these are heightened when EAM also includes integrated field device management.

Multi-protocol field device management
Field device management improves flow of operating data from field devices to the EAM system. One of the main benefits of fieldbus technology is the capability to utilize advanced device management applications in host systems that can interact with asset performance diagnostics resident in their intelligent field devices. But EAM systems have had little access to such information when host and field devices come from different vendors.

Although each device had Device Description technology that supported its configuration, this alone was inadequate and, until recently, there were no other standards. However, a consortium of process manufacturers and vendors has collaborated on an open field device toolkit (FDT) that, when combined with recent enhanced data description language (EDDL) developments, has changed the picture significantly.

FDT technology is ideal for making advanced “plug-in” applications, including highly capable valve testing plugins that attach to the host’s device management software in a standard manner. And, through the efforts of the multi-vendor EDDL cooperation teams, the recent EDDL enhancements address one of the key limitations of earlier device description technology by allowing the device vendor to organize the data shown on simple live data screens on the host system and provide the menus to organize the user selection of displays. Invensys, for example, has recently introduced a field device management toolset that lets users take advantage of any EDDL, Enhanced EDDL and/or FDT host deliverables supplied by the device vendor.

In cases where the device vendor supplies only traditional device descriptions (EDDL), the Invensys field device manager lets the users add functions, such as organizing their own live data maintenance screens and watch windows for each model of field device. The field device manager also lets users set up templates for the commissioning behavior and attach supporting manuals, repair procedures, and any other Windows files they find useful in device maintenance.

If the device vendor supplies enhanced device descriptions (Enhanced EDDL), this reduces template setup work because the device vendor has already organized many of the configuration and maintenance displays. Those displays may contain gauge style indicators, trend waveforms and graphic images.

And, if the device vendor supplies both enhanced device descriptions and an FDT device type manager plug-in, users can realize maximum device management capabilities. FDT technology enables the device vendors to go beyond the capabilities of even enhanced device descriptions. With FDT, the device vendor can program a rich graphical user interface (GUI) application as a plug-in to any other FDTcompliant host system engineering application. The plant maintenance staff would call up this application when they want to analyze the health and performance of a specific model of field device or run comprehensive diagnostic tests and archive the test results.

Field device management that supports both EDDL and FDT helps boost engineering and maintenance productivity over the entire lifecycle of an intelligent field device. Reusable engineering is facilitated through customizable templates for each FOUNDATION fieldbus device model, making it easier, for example, for technicians at all skill levels to correctly replace a failed device.

The value delivered by effective field device management can be considerable. In most complex process environments, up to 20% of the ongoing maintenance cost is associated with intelligent devices, sensors and other devices that act as the eyes and ears of the plant. By using a common toolset that works on the wide diversity of intelligent devices from multiple vendors, this cost can be reduced by up to 40%. For a shop with maintenance spend of $50,000,000, these savings can amount to several hundred thousand dollars per year. Customers are able to greatly increase the number of loops managed per individual, where on average, clients are doubling the loops managed per person.

Field device management enables advancement from condition monitoring, which is characteristic of conventional EAM systems, to condition management, which is essential for the new era of asset performance management.

Condition management
Where many technologies provide basic condition monitoring, describing what is happening with the system, field device management enables condition management, which guides in improving asset performance to achieve specific business objectives. Condition management helps move from the reactive or preventive mode of operations to a proactive and predictive environment. Ultimately, it is this linkage between the real-time and operational environment that moves an organization from asset management to asset performance management.

Condition management has three phases: collecting information (which is comparable to traditional condition monitoring); analyzing information to spot trends and areas requiring action; and acting on the results. Also, where traditional condition monitoring tends to be equipment or area focused, condition management takes a complete contextual view in bringing together operations, maintenance and engineering to resolve critical business issues. Where the previous era of condition monitoring focused on gathering plant level data and making it available as information, condition management goes the next step, advancing information to knowledge and action.

The difference is much more than semantic. Where condition monitoring will help you estimate when a valve might need to be replaced from a wear perspective, condition management might add the business context, assisting you in balancing the risks and gains of replacing that valve this month or next and also ensures that all the key stakeholders are engaged in the decision.

Condition management also extends to fully integrate with DCS/PLC, safety and equipment diagnostic systems, ideally presenting information from these systems through business intelligence frameworks.

The value of condition management is clear. A recent industry survey shows that on average, more than 5% of production is lost every year to unplanned or unexpected outages. For a plant with a total production value of $50,000,000 per year, this amounts to $2,500,000 annually in lost production. The role of condition management is to monitor the key assets that have the largest impact on production, providing early warning of any impending failure, allowing the plant personnel to proactively deal with the issue before it causes a costly shutdown and/or extended outage. In our example, using a conservative estimate of a 30% reduction in outages yields an annual return of more than $750,000.

This predictive capability is further extended by condition management’s ability to collect key performance data to support RCM (Reliability Centered Maintenance) analysis. Based on an independent industry survey conducted by Invensys, more than 50% of preventive maintenance, while valuable in terms of preventing outages, is unnecessary and can often introduce problems. By analyzing the RCM data collected via condition management, organizations can greatly reduce the level of unnecessary maintenance, delivering a further 10-20% reduction in maintenance spending.

Decision support
Condition management also fuels decision support systems that further integrate and present data from additional sources, including all other components of the EAM system: real-time, historical and analytic plant operations data and other plant and business information, including (especially) financials and customer order management systems.

Such data can be presented through role-specific “digital dashboards” (similar to Fig. 2) tailored to show only the information that users require to make informed decisions within their roles and in the context of the key performance indicators and dynamic performance measures for their department, plant or overall operation. These dashboards can combine multiple formats—meters, graphs/charts, tables, raw statistics and spatial data, with full drill-down and drill-around capabilities.

Knowledge management
The real-time integration is much more than a simple process of catching an alarm/alert and generating work requests. It requires full workflow capability that enables engagement of key individuals in the resolution, including operations, engineering and maintenance with full visibility at the management level. It also requires direct connection to devices, system and process alarms/alerts through control system historians and equipment condition monitoring solutions to provide the complete set of information required to understand the context of the potential issue.

In addition, this integration must extend to an HMI in the control/operations environment to allow the operations personnel to spot potential issues immediately, drill down into the details and history and fully interact with the maintenance team and maintenance application(s).

The information solution must further include the ability to capture all the key readings and trends for the critical assets that are necessary to support reliability and availability analysis, which is fundamental in supporting the move to a proactive and predictive approach to operations and maintenance.

Today’s technologies allow us to more effectively capture this wealth of data. Knowledge management is the process of capturing the context and interrelationships of the data points to deliver usable and actionable information. With the “greying” of the workforce, a systematic and automated approach to knowledge capture is fundamental —and a critical element in the move to APM.

The measure of success
Although general descriptors of asset availability and utilization provide a sense of the operation of an asset base, they are lacking in specificity and in any real-time context. Both are required to provide operations and maintenance with an effective performance measurement system. A more specific approach would be to measure the following dimensions:

• Effective Asset Availability—the maximum output possible from an asset set in the current state divided by the theoretical maximum output.
• Effective Asset Utilization—the current output from an asset set divided by the maximum output possible in the current state.

These definitions preserve the integrity of the initial descriptors, and provide real-time context, as well as more accurate assessment of the performance of the operators and maintenance teams. Regardless of which definition is used, it is clear that there is a strong relationship between availability and utilization. It turns out that the relationship tends toward the inverse (Fig. 1) as availability and utilization approach their maximums. This inverse relationship presents a challenge to the maintenance and operations teams in industrial plants because the better they do their individual jobs, the more they will tend to negatively impact each other.

Balancing these factors requires an effective business measurement system that can provide business value insight into the desired operational balance between effective availability and effective utilization. Invensys has a patented approach to real-time business measurement that does exactly this: dynamic performance measures (DPM). DPMs measure the business value of base assets, asset sets or groups of asset sets as a real-time vector that represents the true value that they generate. Instead of optimizing availability or utilization, manufacturers are now able to optimize the business value.

The real-time business performance data (DPMs) enables a much more effective approach to true asset management. Rather than merely managing the availability of some instrumentation, plant personnel can drive business performance from asset sets, up to and including the entire plant. To bridge the gap between the traditional approach to asset management and asset performance management, a threelevel model has been developed (Fig. 2).

In this model, the Base Asset Management level represents the narrow approach traditionally deployed in industrial plants in which base assets, such as instruments and valves, were independently managed from an asset availability perspective. As a matter of fact, even this level is an expansion on the traditional approach to asset management since it includes effective utilization as well as effective availability improvements.

The second level, Asset Set Optimization, goes beyond traditional asset management by combining assets into logical production sets through the use of advanced technologies, such as first principle models so an entire logical set of assets can be effectively managed.

The third level, Business Performance Management, is an all-encompassing level in which advanced technologies and business measures, such as predictive maintenance and multivariable predictive control, can be used in balance with each other to maximize the business value generated of groups of asset sets.

Almost all organizations have the core elements in place that are required for utilizing asset performance management data to drive business value. When implementing this approach, however, it is important to evaluate the current state of the plant. In particular, there are five key elements that must be assessed:

• What is the culture of the company? Are the company and employees ready and willing to change and do they recognize the issues involved with changing?
• Do the employees have the skill base that is required for implementation?
• What are the current business processes in place? Business processes must be evaluated to see if they are current, optimized and built on best practices and benchmarks.
• What is the current technology level? The technology must be assessed to see if the company is based on current and open standards, and what level of enterprise integration is employed.
• Is corporate knowledge readily available and accessible?

While being up-to-date on all these elements is not necessary, all must be evaluated to determine the current status of the plant and how and where to move forward. This will aid implementation, help to identify risks and establish a phased plan for full implementation.

Neil Cooper is vice president, Asset Performance Management Solutions, a key member of the Invensys Process Systems (IPS) Global Marketing Group. Prior to joining Invensys four years ago, Cooper was the president of Indus Canada.

The Ocean Spray Cranberries Inc. [OSC] Bordentown, NJ facility was getting a bit “long in the tooth.” Originally constructed in the late 19th century as a Worsted Wool plant and then converted by OSC in 1946, it operated for many years with minimal preventive maintenance. Over time, the facility began having difficulty meeting its obligations. In August 2004, newly appointed plant manager Tim Haggerty and corporate continuous improvement manager Jerry Langley invited Charles Brooks Associates, Inc. (CBA) to conduct a maintenance benchmarking evaluation and help develop an action plan to improve the overall performance of the facility. This process had been used in other OSC facilities for a number of years with great success. Over the course of the next two years, the Bordentown plant began making remarkable improvements in both production and maintenance performance.

During the initial benchmarking by CBA back in 2004, the facility scored 466 out of a possible 1000 on a worldclass maintenance scale. Specific opportunity areas were identified during the survey and recommendations were communicated to the management team during the wrapup meeting. The team was encouraged to:

• Develop and communicate a coordinated plan for asset care improvement.
• Develop effective metrics for measuring maintenance contribution to plant performance.
• Restructure the maintenance function to support the operations.
• Improve the utilization of Maximo:
  • Capture of maintenance costs
  • Analysis of asset performance data
• Determine if assets were capable of performing at required levels (and upgrading if necessary).
• Evaluate maintenance skills and provide the required high-impact training.
• Determine the appropriate maintenance approach for each asset, including center-lining.
• Improve the execution of the planned maintenance process:
  • Level 1 (Operators’) PMs
  • Maintenance PMs
  • Overhauls

The local OSC management team began to evaluate its staffing and processes immediately after the benchmarking and discovered that it was not prepared to take the plant to the next level. At that point, plant manager Haggerty chose to move the facility out of the reactive maintenance mode into a planned maintenance mode through a series of staff moves and strategic hires.

A plant-wide strategy
In September 2005, Charles Brooks Associates began providing interim maintenance management by assigning Dan Simpson to the facility on a full-time basis. A thorough analysis was made of all hot-fill bottling equipment to determine what steps were required to bring the facility up to acceptable standards. OEMs and vendor representatives had been contracted prior to Simpson’s arrival to conduct complete technical evaluations of their equipment.

The result of the evaluations was a plant revitalization strategy that identified over $325,000 in parts that were required. One fact was evident: the equipment was not receiving the preventive maintenance attention it required. The following month (October 2005) the plant conducted its first “revitalized” maintenance down day. The postmortem revealed the following:

1. Hourly employees from both production and maintenance completed a great deal of very necessary work.
2. The planning process and the presentation of THE PLAN from the maintenance department needed improvement.
3. More and better interaction with production operations was needed. Production participation in the preventive maintenance (PM) process needed to be expanded to enhance equipment knowledge and increase available PM resources.
4. There was a definite need for a “make ready” meeting prior to the down day kickoff, as well as a need for a process to allow the team’s progress to be tracked throughout the day.
5. It was determined that critical equipment (filler, capper, labeler, case packer) must receive preventive care during every down day, both from production and maintenance.

As time went on, each and every down day was evaluated and improvements were made before the next down day was scheduled. The maintenance team included the maintenance manager, maintenance supervisors, maintenance planners and hourly employees who had specific technical knowledge of the critical equipment. Twelve-hour maintenance down days were scheduled every week for each of the packaging lines.

Delivering via a reshaped program
When new technical services manager, Herb Nielson, came onboard, he began to reshape the technical team, including the maintenance operation. One notable change was the assignment of Phil Camerota as permanent maintenance manager. Camerota had a background in aircraft maintenance with the U.S. Air Force—he definitely understood the importance of prevention. His experience with aircraft coupled with his plant experience gave him a full appreciation of what could be accomplished.

To support the planned maintenance effort, two maintenance planners were assigned to work with the existing Maximo maintenance management software. Tom Krepp and Warren Bell assumed the roles as maintenance planners, conducting both pre- and post-down day meetings. Down day schedules were prepared and monitored, assigning work orders to individuals on an hour-by-hour basis. By continuously monitoring the progress of the down days, resources could be shifted from one activity to another to ensure that the most critical activities always were completed.

To address the technical skill needs of the maintenance department, Chris Guldner of CBA worked with the OSC maintenance team to develop and implement a skills evaluation process. The process included supervisory observations, self evaluations and the use of the Minimizer, a device for evaluating mechanical and electrical ability. The results of the skills evaluations were used to create individual development plans for maintenance personnel.

Working with the new production manager, Bill Garcia, the maintenance team has been able to signifi- cantly improve the performance of the packaging assets of the facility. The cooperation between maintenance and production has improved, and the results show it:

• 25% improvement in bottling throughput efficiency performance
• 25% reduction in unplanned downtime
• 24% increase in bottling output (cases/day)
• 1,068,240 additional cases produced in a 5-month period due to increased line efficiencies.
• $1,068,240 of additional available sales volume

Haggerty, Nielson and Camerota all recognize that none of this progress could have occurred had it not been for the buy-in and support of the front-line mechanics, electricians and controls technicians. As Haggerty says, “All of those individuals are professional in their own right, and as such have been looking for a professional, specific, process to follow. These people have always wanted to do the right job, with the right equipment and parts, but past decision-making put them in a position of having to do the ‘best job they could with whatever they had available.’ This new focus and drive has really made a significant step improvement, not only in the operations, but in the daily work life of each maintenance employee.”

The sweet taste of success
While a great deal has been accomplished, there is much left to be done as the facility strives to become a world-class bottling facility. The OSC team at Bordentown has accepted the challenge and is moving ahead implementing new tools, conducting additional training and exploring creative ways to improve asset performance.

In July 2006, Charles Brooks Associates once again conducted a maintenance benchmarking of the Bordentown plant. The survey revealed the percentage of maintenance work that was planned in advance had increased from less than 20% in 2004 to 71% in 2006. Maintenance personnel are now recording over 90% of all hours worked on work orders and maintenance schedules are being honored. Production and maintenance managers meet every day to discuss asset performance and the top three downtime causes are analyzed. The Bordentown facility improved its overall maintenance score by 39% and recorded “best in class” in the areas of planning and scheduling and maintenance procedures.

Bordentown plant manager Tim Haggerty has spent 33 years in container manufacturing and beer/ juice packaging, including 27 years with Coors and 6 years with Ocean Spray Cranberries, Inc.

This article describes the successful removal of a tube side cover plate diaphragm (gasket) from a high-pressure heat exchanger. The diaphragm was replaced with a metal pressure-energized seal ring. While the technology has been utilized in piping and offshore applications for several years, this retrofit was its first known application on a refinery heat exchanger of this magnitude. Modification of the cover plate to accept the pressure-energized seal ring eliminated the need to reinstall the metal diaphragm gasket, thereby saving 75% of the reassembly cost of the exchanger.

Utilized in conjunction with mechanical multi-jack bolt tensioners, this innovative retrofit has eliminated a recurring, costly problem (in both maintenance cost and loss of opportunity). This additional reliability coupled with the significant future cost savings from less downtime justified the retrofit—not including the cost savings foreseen by preventing unplanned outages from the exchanger.

It is anticipated that this modification eventually could change the way that refiners will specify how high-pressure heat exchangers in hydrocracking services are designed and constructed.

The problem
It is not uncommon for diaphragm plates in high-pressure heat exchangers to develop cracks in their seal welds. The diaphragm, which is generally a thin plate of alloy steel, serves as the gasket and corrosion resistant liner for the channel cover (Fig. 1). This arrangement is common for heat exchangers in hydrogen services at operating pressures above 1600 psig in our refinery’s Gas Oil Hydrotreater (GOHT) unit. We have had multiple diaphragm leaks over the past several years.

Until recently, the repair process had been the same. It entailed removing the diaphragm, machining the channel face, welding and re-machining a nickel “butter-coat” layer on to the channel face and finally welding on a new diaphragm under controlled heat. This repair would suffice for a time, until some process upset or other anomaly would create another cracking “event.”

This procedure had become the standard operation of repair and, in turn, our “insanity clause,” as we continued to perform the same repair steps repeatedly, and yet, after returning the exchanger to service, would expect a different result.

Criticality of process [1]
Hydrotreaters process feedstock for fluid catalytic cracking units (FCCU) and hydrocrackers. The economic impact of these conversion units are crucial to a refinery’s profit and loss statement as well, maybe even more so than hydrotreaters. Downtime in a hydrotreater forces refinery logistical issues such as throughput curtailments in the FCCU and hydrocracking units. Their close integration to each other (and the bottom line) emphasizes the need for unit availability. Equipment reliability is critical to success.

From a business perspective, hydrotreater units are required to meet the low sulfur fuel specifications that are now in effect. Hydrotreater units are critical to a refinery’s balance sheet. The cost associated with taking one out of service can be dwarfed when compared to the potential lost income. Hydrotreaters are often large volume units. The 3-2-1 and 2-1-1 crack spreads have been favorable for refiners in the recent years; even further elevated the last few years following both hurricanes Katrina and Rita.

Hydrotreater units are expensive to build. Their high pressure and elevated temperatures necessitate heavy wall vessels, piping and ancillary equipment. Their severe service often requires exotic metallurgy. The preferred method of fabrication is butt-welding due to material and equipment costs and to prevent possible leak locations. These constraints often minimize block valve installations between equipment and any possibility of isolating or “bypassing” equipment, without taking the complete unit down.

Hydrotreater units also are expensive to start up and shutdown. They are labor- and maintenance-equipment intensive. Large volumes of inert gas are required to cool and protect the multimillion-dollar catalyst beds and equipment. There is always some associated risk involved in starting up or shutting down a hydrotreater unit. Rapid thermal gradients can damage equipment and the repair or replacement time could be several weeks or months.

Why diaphragms crack [2, 3, 4]
Diaphragms are generally fabricated from thin stainless steel (SS) sheets (0.100”-0.125”) such as 304 or 304L. Diaphragms are GTAW fillet welded to exchanger channels in a lap-joint configuration along the edge of the diaphragm plate. This joint configuration has limited transverse strength due to the limited size of the effective fillet leg or cross sectional area of the fillet weld reinforcement. Because of the thin diaphragm, there is high welding residual stress adjacent to the weld.

There are several reasons why cracks can develop. They include:

• Tensile overload caused by differences in the thermal expansion of the low alloy steel channel (carbon or Cr- Mo) and the SS diaphragm. Some exchanger channels contain effluent streams up to 730 F and diaphragms can be over 80” in diameter.
• Chloride Stress Corrosion Cracking (SCC) – A salt which drops out in hydrotreater effluent exchangers is ammonium chloride. The diaphragm is susceptible to this failure mode due to its high residual stress. The crevice between the diaphragm and channel contain concentrated chlorides and aggravate the cracking mechanism.
• Polythionic Acid Stress Corrosion Cracking – Hydrotreater effluent systems use austenitic SS for sulfidation resistance. Polythionic acids are formed in the process during shutdown periods when the prevalent metal sulfide scale reacts with oxygen and water condensed during the steam out cleaning process. These acids cause SCC in SS, which is sensitized from welding or from operating temperatures in excess of 750 F.

SCC is the result of combined mechanical stresses with corrosion reactions. The combination of a susceptible alloy, sustained tensile stress and a particular environment lead to the eventual cracking of the alloy. It is difficult to alleviate the environmental conditions that lead to SCC. Chloride levels required to produce stress corrosion are very small, generally below the macroscopic yield stress. The stresses are often externally applied but are quite often residual stresses associated with fabrication, welding or even thermal cycling. Unfortunately, stress relieving heat treatments cannot completely eliminate all the residual stress.

Knowing, and ultimately reducing (or eliminating) the important variables of SCC propagation is the best avenue for success. These variables again are:

• The level of stress,
• The presence of oxygen,
• The concentration of the chloride,
• The elevated temperature and
• The conditions of the heat transfer (often the design).

This failure mode is not uncommon for exchangers of a certain age, design and service. Diaphragm fillet welds encounter high stresses from the combination of high hoop stress and large compressive stresses generated from the cover plate bolting. This cracking is common in the diaphragm welds of high-pressure heat exchangers in hydrotreating units found throughout our nation’s refineries and abroad.

Seeking a solution
Faced with repeated failures of the diaphragm welds in hydrotreater exchangers—14 exchangers, seven in two separate trains (see Fig. 2)—and the economic impact of these exchangers to the refinery, developing a solution to this phenomenon became a priority.

The Reliability and Maintenance groups designed or investigated a handful of possible solutions, but none of them received total buy-in. Then, following several months of study, an innovative, alternate solution was identified. It involved eliminating the diaphragm plate entirely and replacing it with the pressureenergized seal ring (Fig. 3). The solution from Taper-Lok® was simple, effective and quite field compatible within the timeframe of a shutdown.

Seal concept…
The Taper-Lok metal pressure-energized seal ring was designed to use on piping applications for topsides of offshore platforms, flow lines, production risers, manifolds, chemical plants, refineries, power generation, supercritical wet oxidation and numerous other practical applications. Most assemblies consist of a male flange, female flange, seal ring, and a set of studs and nuts. The pressure-energized seal ring seats into a pocket in the female flange and is wedged and seated by a male nose located on the male flange.

Utilizing this concept, the exchangers channel cylinder would contain the female pocket, while the channel cover would have the male nose geometry.

In the pre-bolted condition, the Taper-Lok seal ring lip stands off of the face of the channel. The converging seal surfaces are brought together like a wedge during bolt up. This wedging motion forces the seal ring onto the male nose and into the female pocket forcing a compressive hoop stress. Minimal bolt load is required to achieve the required contact stress on the seal surfaces.

The converging angles of the seal ring create a wedge or “doorstop” effect. As the equipment internal pressures increases, the seal seats tighter into this sealing wedge.

Taper-Lok seals are made from the same material as the process equipment (exchanger channel and cover) to ensure that thermal expansions are consistent across all components. The effects of bi-metallic (galvanic) corrosion are eliminated. A baked-on moly coating is applied to the seal to prevent galling.

This promising sealing technology required minor modifications to the heat exchangers. It was simple, reusable, provided a metal-to-metal seal and took very little time to make the modification. This application, however, was unique in that there was no published history utilizing the seal on a refining exchanger of this pressure and severity.

Reliability assessment and risk mitigation
Since this could have been the first use of this type of seal on a fixed equipment cover, a reliability assessment had to be conducted and the risks needed to be identified and subsequently mitigated. All known applications for Taper-Lok pressure- energized seals were researched. These seals, it was discovered, had been used in many different types of connections, including:

• weld neck flanges
• blind flanges
• closures
• clamps
• swivel flanges
• misalignment flanges
• tube sheets

These seals also have performed well in temperatures from cryogenic to 1600 F and at pressures to 40,000 psig. Applications of note included:

• heat exchanger internals
• hydrogen processing
• high temperature measurement equipment
• offshore (both sub-sea and topside)
• high pressure compressor connections

A study conducted by JP Kenny proved to be helpful. It compared the Taper-Lok to standard ANSI bolted connections. While the study did not focus on welded diaphragm connections, it did point out some of the beneficial characteristics of the seal. The results of the study showed that the pressure seal was preferable to ANSI standard gaskets connections.[6]

A design of the seal for one of our heat exchangers was created and calculations according to ASME Section VIII, Division 1 Unfired Pressure Vessels [7] were conducted to ensure code compliance. Because the seal is self-energizing, the gasket factor “m” and the minimum design seating stress “y” are both zero, the required bolt load is reduced and equals the hydrostatic end load of the closure only.

A finite element analysis (FEA) was conducted by an independent third-party engineering firm.[8] Analysis consisted of both 2D and 3D nonlinear models with contact elements. Both models showed a wide contact area with pressures at the sealing surfaces to be in excess of 20ksi. The analysis verified that the seal would be kept in an elastic state and that the stresses in the components would be below code the allowable limits.

Even though all data suggested that the Taper-Lok seal would work in our application, we were still concerned with the possibility of a leak or failure of some sort. Without published history, we needed a fallback plan. It was determined that when we implemented the Taper-Lok sealing system in one of our exchangers, we would build a new channel cover with the male nose geometry as opposed to retrofitting our existing closure. This would then require only the cutting of the female pocket into the exchanger channel. In the event of something unexpected showing up during the retrofit, we could reuse the old closure and weld back a diaphragm plate and seal the opening as we had always done.

Implementation
Description of modification…
The modification centered on the elimination of the welded diaphragm gasket and implementation of the double angled pressure seal.

Simple modification procedure…

1. Remove/replace channel cover plate
A new channel cover plate was fabricated to reduce downtime. Originally, the existing channel cover plate was to be modified via a rapid turnaround machining effort to allow the cover to accept the tapered pressure seal; a minor effort that was not difficult. The original channel cover plate was salvaged for modification and installation on the sister exchanger in the second train. It was also available to reinstall if any unforeseen problem existed with the retrofit.

2. Remove metal diaphragm gasket/seal
The diaphragm seal was removed in a multi-step process that began with drilling a hole through the diaphragm and performing a safety check for any residual hydrocarbon. Once complete, the center of the diaphragm was removed by arc gouging, being careful not to cut close to the inside diameter of the channel. The remaining diaphragm, including the fillet weld that attaches the diaphragm to the channel, were machine cut from the channel. The channel was also faced to insure a true, flat surface.

3. Field machine female pocket into exchanger channel (Fig. 4)
The Taper-Lok pressure-energized seal geometry requires two sealing surfaces. One a female pocket, the other a male nose. The female pocket was machine cut into the exchanger channel while the new, fabricated channel cover plate featured the male nose. During assembly, an additional benefit to the design was observed. This male nose on the channel cover plate configuration acted as a guide.

4. Insert pressure seal ring into pocket (Fig. 5)
Since the Taper-Lok geometry of the female seal pocket is angled, the seal ring was installed into the female pocket and held in place by fricton, providing a hands free, safe installation of the channel cover plate.

5. Insert channel cover plate on studs and pressure seal

6. Pre-tension studs (Fig. 6)
To ensure that the channel cover plate was assembled square and free from misalignment and to reduce bolt interactions, hydraulic tensioning equipment was utilized. Four tensioners were used at 90 degrees and the tensioners were kept under load while all nuts were installed and hand tightened.

A second deviation from the exchanger’s original design was applied at this time. In lieu of the traditional heavy hex nuts, mechanical multi-jack bolt tensioners were utilized. This also proved quite fruitful as torque wrenches were then used to apply the proper torque required to seat the pressure- energized seal ring. The traditional impact (and accompanying crane used to hold it in place) was rendered obsolete.

Hot torquing was not necessary after the installation, even after the unit had gone through a few cycles. This is attributed to the spring effect that the seal and component geometry create during and after seal seating. This spring effect refrains the bolts from relaxing.

Conclusion
Several benefits were realized from this retrofit using the Taper-Lok® sealing technology. One of the most important benefits was the elimination of the diaphragm cracking, which, in turn, increased equipment reliability and unit availability. Since the Taper-Lok seal is fabricated from the same material as the pressure parts (channel and cover), bi-metallic or galvanic corrosion cannot occur. All thermal expansion observed during operation is constant. The new seal or “gasket” remains in compression and in an elastic state. The seal is self-energized, creating a tighter seal with any increase in pressure. Elements that promote the cracking are eliminated.

Following the first retrofit, a second exchanger was retrofit in March 2006. The remaining 12 exchangers are scheduled to be retrofit during the next two scheduled GOHT outages.

Additional benefits—some initially unforeseen—were the reduced costs or downtime from several items, including:

• the reduction in the exchanger turnaround time from six shifts to three shifts
• the utilization of one crane in lieu of two (one for cover plate and one for impact)
• the elimination of any hydrogen bake out from weld contamination and weld dilution on alloy exchangers with a stainless steel corrosion overlay
• the elimination of machining of weld buildup (Nickel butter coat)
• the elimination of seal weld and metal diaphragm seal
• the elimination of NDE to search for cracking throughout the entire process

Downtime has decreased since the root cause of the process leaks (cracking of diaphragm welds) at the cover have been eliminated. The retrofit was deemed a success. No downside opportunities have been observed or foreseen. MT

Doug Hughes is turn around superintendent at Valero Refining – Texas City, TX. E-mail: doug.hughes@valero.com

James Cesarini is president of Petro Spect, based in Texas City, TX. E-mail: chezo@petrospect.com

Erick Howard is vice president Engineering of Taper-Lok, in Houston, TX. E-mail: ehoward@taper-lok.com

How you rally and arm your troops is crucial to the outcome of this ongoing battle.

As the well-known civil engineer and author Henry Petroski put it, “Success is foreseeing failure.” It’s only four words, but it defines what every preventive maintenance (PM) program should strive for. Whether you have been working for many years or are a newcomer, your most effective weapon against breakdown is a well-organized and on-schedule PM program.

It also doesn’t matter what field you are in or how large or small your organization is. The fact remains, if you do preventive maintenance correctly and consistently, it works. Proper staffing levels are required to stay on schedule and staying on schedule is the cornerstone of all successfully run programs. You still can run your program behind schedule, but breakdowns will increase and your program will become less and less efficient. Your troops will be doing more breakdown maintenance than preventive maintenance, a sorry situation for everyone concerned (including your company, your department and your customers).

We are all familiar with the various predictive (PdM) technologies that have emerged to supplement PM programs (i.e. infrared scanning, oil analysis, vibration analysis, etc.). But, while such technologies are very effective and can add much more refinement to a program, they still are supplemental in nature. They do not take the place of a PM program, but enhance it. There is no substitute for the day-in-day-out battles that must be waged to keep breakdowns at bay.

The dictionary defines “breakdown” in terms of “failing to function.” A breakdown can be a complete failure, i.e. a motor burns out. It can be a partial failure, i.e. a motor overheats but still runs. It can be an intermittent failure, i.e. a motor stops and starts for no apparent reason. Or, it can be a calibration failure, i.e. a thermostat won’t control temperature properly. Whatever the form, though, be it major or minor, a breakdown is always a problem that needs to be corrected.

Let’s look over the accompanying chart. As you can see, the planning, strategy and tactics to conquer breakdowns start at the headquarters.

Planning
Planning is the first and most important step. According to John Wooden, former UCLA basketball coach, “Failing to plan is planning to fail.” It’s the same in maintenance as it is in basketball.

Take the time needed—months if necessary—to develop your plan and always involve your troops in the planning phase. Furthermore, “know your enemy.”

• Do you have 1000 motors that you have to keep running 16 hours a day, 300 days a year?
• Do you have computer room air conditioners that never can be shut down? How can you schedule the maintenance these air conditioners need?

You get the idea. You have to know your plant, your systems and their needs—and you need to develop your strategy around this information.

To give you an example, our program has 225 pumps that must have preventive maintenance done twice a year, on schedule. This is part of our overall planning strategy— to do the PMs needed to keep these pumps operating at their designed efficiency. We have a planned strategy for all of the equipment in our program. 

Tactics
Let’s assume that you have enough troops to move against the enemy. If you were setting up your program in a new plant that had yet to start up, adequate staffing should have been part of your overall planning. There is information available to help you plan tasks and how long they should take to complete.

If you are coming into an existing plant, your troop levels may or may not be sufficient. If you have enough troops, good! If you don’t, you’ll have to do your best to control and defeat breakdowns working under a disadvantage.

Refer to our chart again—you will notice we have the enemy surrounded. He cannot break out as long as we keep pressure on him and keep him under control. Keeping him besieged is your tactical objective.

I’ve set the stage for our attack to begin. We have a sound plan, we have developed tactics and we have enough troops to control the growth of our enemy. Now what?

Gather your troops and explain your overall strategy (they should have been in the planning phase). Discuss the benefits of the program if it is done correctly and on schedule as much as humanly possible.

What are the benefits?

1. Increased reliability and life of equipment
2. Fewer major repairs and downtime
3. Shift from breakdown maintenance to preventive maintenance
4. Fewer emergencies
5. Better customer relations
6. Less work stress = POM (Peace of Mind!)
7. Increased profits
8. Glory for you and your troops (A Bonus!)

Attack with vigor
Look at the chart, again. The following items are your prongs of attack. 

Persistence (refusing to give up)…
Next to your overall strategy, “persistence” may be one of the most critical elements in your program. You must—at all costs—keep your program active and vigorous. In our own organization, I try to ensure that preventive maintenance is done every workday. If you are understaffed, you will have to prioritize and be willing to adapt to changing conditions. As a maintenance manager, it is YOUR responsibility to make sure the work gets done.

Training …
Train your troops both on the job and in formal settings. New technology is a blessing and a bane. It gives YOU an edge—but, it also gives breakdown an edge. Your troops must be properly prepared to meet these new threats.

Standards (the level of requirement)…
What brand of line starters do you prefer? Will you accept sloppy and shoddy workmanship? Will you tolerate late shipments from vendors? These are questions of standards or levels of requirements. Your preventive maintenance will live or die based on what kind of standards you set for it. Set high standards and have your troops, contractors and others who report to you rise to meet them. Don’t lower your standards to meet theirs.

Routine…
The next prong of attack is the routine (course of action or schedule). Your schedule is your guide. Routine for your organization could be quarterly, bi-annual, yearly, whatever. Only you and your troops can determine that. Don’t, however, underestimate the role of “routine.” Without it, you are like a ship at sea without a rudder.

Documentation…
The documentation that you do and the level of quality and importance that you place on it will be another major factor in your success or failure. Your office staff must be involved, hopefully from the very beginning. Moreover, they must be properly trained and committed to the program and its success.

Auxiliaries (contractors and vendors)…
Your auxiliaries also play key roles in your conquest of breakdowns. If you can’t get the logistical support to the front when you need it, breakdowns will begin to defeat you and instead of being on the attack, you will be in retreat, Sit down with your key contractors and vendors and discuss their roles in the program. Let them know what is expected of them—and that they are essential to your success. If they won’t or can’t conform to your high standards, muster them out and recruit new auxiliaries to fill the gaps in your lines.

Observation (paying attention)…
Sounds simple enough. How hard can paying attention really be? Talk with your troops about taking ownership of their equipment, their areas and their customers. Your troops are your eyes and ears out on the front lines; they are the ones that hold back the “hordes of breakdown.” Their input is essential to the conduct of the war. Always be on guard. As the great Neil Young album noted, “Rust never sleeps.” It’s true.

Ingenuity (inventive skills)…
This last assault is what separates the winners from the losers. Everyone in your program will contribute to this effort by presenting new ideas (i.e. “Let’s switch to this new grease for motor bearings, it protects better and will last longer without drying out”).

Try to foster in your troops a climate where their ideas and ingenuity are valued and used. If they can improve the program, they will take ownership of it—it will become “their” program. As a result, they will nurture it and believe in it and its value. This is another way to make constant improvements. Remember, if you are not moving forward, you are falling behind.

Taking up the flag
There’s no industry where preventive maintenance won’t pay off. Now that you know what it will take to make you, your troops and your PM program a success or failure, roll up your sleeves, put on your battle gear and get out there and conquer breakdowns.

As The Battle Rages On

So what is breakdown doing while you are doing your best to defeat it? It’s getting older and next to neglect that’s what is going to give you the most problems. Certainly, the other factors are important—but they are easier to control. The older the equipment, the more it has worn out. If you have a 35-year-old air handler, it will take more of your resources than a 10-year old air handler in good condition. (Age really can be your enemy, too.)

John Camillo is an engineering supervisor at the Princeton Healthcare System in Princeton, NJ. He also is the Engineering Department’s training coordinator. Camillo studied at the Philadelphia Wireless Technical Institute, majoring in HVAC-R. Over the past 40 years, he has worked in various industries, including hospitality, aviation, missile reentry, mill/canning and healthcare, where he has specialized in developing and implementing PM programs. E-mail: jcamillo@princetonhcs.org

As a number of Louisiana refineries are finding, flexible sensor technology can deliver an especially cost-effective way to measure temperatures almost anywhere in a system.

As a systems integrator that services refineries and chemical plants in Louisiana, we have found our customers’ biggest maintenance headaches come in the area of temperature sensor replacement. Rigid temperature sensor technology, used inside thermowells for 20-50 years, is the nightmare of every maintenance department. Problems using rigid sensors include stocking difficulties, finding suitable replacements, ordering the correct length and size, and being unable to install a replacement sensor in an existing thermowell.

Flexible temperature sensors, on the other hand, offer a universal solution for all maintenance dilemmas. A flexible sensor fits nearly everywhere, can be cut to the correct length and reduces the number of spare parts a plant has to keep on hand.

Rigid sensor challenges
A standard rigid temperature sensor, made by virtually every sensor manufacturer in the world, consists of a sensor element—thermocouple (T/C) or Resistance Temperature Device (RTD)—protected inside a rigid stainless steel shaft in a 2” sensitive area, forming what most users know as a “fixed length sensor” (Fig. 1). These fixed length sensors are either spring loaded (for use with thermowells), welded to a hex nipple for a fixed immersion length into a process or sealed with epoxy, exposing the sensor leads for external measurement connections.

Typically, the T/C or RTD element is embedded inside the bottom two inches of a stainless steel tube, which is then filled with mineral insulated powder (MGO) and sealed with epoxy to prevent moisture penetration. The rigid sensor assembly fits into the thermowell beneath the connection head. The wires from the sensor are then terminated in the enclosed head and connected to extension wires using a terminal block, or attached directly to a transmitter. Wiring is then run back to the control room, usually encased inside conduit for long wire runs.

The first problem posed by rigid sensors is the difficulty involved in replacing a faulty sensor. Typically, a maintenance technician has to remove the enclosure cap, disconnect the wires from the transmitter or terminal block, disassemble the union, conduit and fittings attached to the transmitter and thermowell, and then move them out of the way before he or she can pull the rigid sensor out of the thermowell. Depending on the age of the installation, the corroded conditions of the conduit or junction, and the amount of room available, this can be an arduous task, particularly on the top of towers or columns, or in close confinement areas.

The next problem involves determining the correct length of the replacement sensor. In many cases, a maintenance technician may know that the sensor needs to be replaced, but doesn’t know the exact length of the rigid sensor. If the loop is critical, the plant may not want to pull out the old sensor yet. Instead, they will make all the necessary measurements first, order a new sensor and wait for it to arrive. In that case, the technician will have to make multiple visits to the sensor—first to determine as much information about the installation as possible, including sensor type, connection style (nipple union nipple, direct thread, lagging length, approximate insertion length, etc.)—and then go back to stores to try to find a best fit, probably returning with a number of different sizes to avoid a third trip. Of course, the unused sensors then have to be returned to stores (another trip)!

In some cases, the technician leaves the old system intact, gets on the phone to a sensor representative, and the two of them make an educated guess based on a thermowell’s length, size of the union, length of nipples, etc. At least one sensor manufacturer we deal with admits that they only get it right about 85% of the time when they have to guess. Another solution is for a maintenance tech to carry 8-10 spare sensors out into the plant, in hopes that one of them will be the right size. All this could be avoided, of course, with proper documentation—that is, the size and type of each temperature sensor should be recorded for future reference when replacements are needed. This, however, can be a daunting task, considering that some plants have hundreds, if not thousands, of temperature sensors. Plus, engineering drawings do not always represent the “as built” installation.

Once a replacement sensor is found, ideally it will slide back into the thermowell. Unfortunately, thermowells can cause other problems. Some thermowells will “sag” (bend) when exposed to high temperatures over prolonged periods, as is the case with flare stacks (Fig. 2). It may be possible to extract the existing sensor from a sagging thermowell, but it is usually impossible to install a new rigid sensor into a sagging thermowell. Instead, the thermowell itself must be replaced.

Thermowells also can accumulate debris, which makes it difficult to install a new replacement sensor. In areas with high humidity, such as Louisiana and other southern states, thermowells can fill up with assorted contaminants that condense out of the air. When the rigid sensor is removed, this debris can then prevent a new sensor from being fully inserted back into the well.

Finally, the length of a rigid sensor can affect accuracy of the measurement: A rigid sensor inside a short, 2”-3” thermowell may not be measuring the correct process temperature. This is because a sensor with a rigid metal sheath is not measuring just the process inside a short thermowell; some of the sensor’s sheath protrudes up into the nipple, union or enclosure, which is outside the process. Such a sensor actually measures part of the process temperature and part of the ambient temperature outside the process. This situation typically will result in erroneous temperature readings with possibly adverse effects on process control. In one case at a tire plant, the lower inaccurate reading resulted in higher process temperatures that, in turn, caused the thermowells to overheat and sag. Sagging thermowells resulted because the actual temperature was much higher than the sensor could record.

Flexible solutions
Even if an application spec calls for a rigid sensor, a flexible sensor can fill the requirement. A flexible sensor typically consists of a 1” stainless steel sensor element and lead wires that are protected either with Teflon or fiberglass insulation. Flexible sensor wires can be trimmed to the correct length depending on assembly (Fig. 3).

The sensor element is held in place with a spring at the top of the thermowell (Fig. 4). The spring keeps the sensor in constant contact with the bottom of the thermowell, allowing the best heat transfer to the sensor. If there are large open areas within the union/nipple junction, spacers can be used to facilitate insertion through these areas.

Replacing a flexible sensor in the field is much simpler, compared to rigid sensors. To insert a new flexible sensor in place of an existing one, a technician only has to remove the cap, disconnect the sensor wires, remove the transmitter or terminal block and pull out the old sensor. It is not necessary to disassemble the union, conduit or any other fittings.

Because a flexible sensor can be trimmed to the correct length, a technician only has to carry a single sensor to the field. Flexible sensors typically are available in various lengths to accommodate nearly every size of thermowell or application a plant may have.

In the case of a sagging thermowell, if the rigid sensor can be removed, a flexible sensor can be installed without replacing the old thermowell. We usually purchase flexible sensors that are slightly smaller in diameter than rigid sensors. The most popular rigid temperature sensors built in the U.S. have a ¼” (0.25”) O.D. metal shaft. Most thermowells installed today have a 0.260” internal bore (in Europe a 7mm bore is used). We order flexible sensors with a 6mm O.D. (slightly smaller than 0.25”), making it easier to slide into a sagging well or into dirty thermowells that have built-up or caked-on debris inside them.

Because a flexible sensor has a 1” sensor with flexible fiberglass or Teflon insulated lead wires and a spring, it can be trimmed to fit even the smallest of thermowells. Furthermore, since the spring and lead wires cannot conduct ambient temperatures to the sensor, outside measurement errors cannot exist. Like the previously mentioned tire plant, we’ve had several applications on other flare stacks where we installed flexible sensors and the process engineers were surprised to see that their stacks were operating at much higher temperatures than the previous rigid sensors had indicated. The energy and fuel cost savings obtained from operating these stacks at the proper temperatures paid for the replacement sensors many times over.

Intriguing applications
Flexible sensors offer several interesting ways to approach temperature measurement applications and their problems. For example, the intense humidity in Louisiana causes “Green Rot” at the wire termination points with thermocouples, so engineers and technicians try to avoid as many termination points as possible. Because a flexible sensor can be made with any length of wire, we now have several plants in the area that do not use terminal blocks anymore; instead, the wires are run directly to temperature transmitters located in a separate cabinet. The sensor wires are run inside rigid or flexible conduit, all the way from the thermowell to the remote mounted transmitter, without using any intervening termination blocks. This eliminates one major source of failure.

Another plant noted that since the sensor wire did not carry any dangerous voltage or current, it was not necessary to encase it inside conduit. Therefore, all their sensor cables run directly from the thermowell to a remote transmitter without conduit (Fig. 5). The flexible insulation covering the sensor wires is sufficient to protect it from most environments, but stainless steel braid or flex armor can be added at very little cost.

In one application, a plant had a burner with dozens of temperature sensors, but none could be replaced without shutting down the entire burner. It was simply too hot for a tech to walk into the burner while it was operating. By using flexible sensors inside long protection tubes attached to the points of measurement, it was possible to slide a flexible sensor in and out of the tube from a safe location without shutting down the burner.

In a similar situation, a refinery had a problem with calibrating and replacing sensors with transmitters on top of columns or towers. It was physically dangerous for a tech to climb to such heights while the hot process was running, and try to safely work with rigid conduit, fittings and transmitters. This refinery replaced all its rigid sensors with flexible units and installed the transmitters at the bottom of the towers for easy access. Again, because a flexible sensor can be made to any wire length, the transmitter could be calibrated or replaced from the bottom of the tower, and the flexible sensors were easier to change out if they failed.

Adding up the savings
Over the last year, several refineries in Louisiana have begun systematically replacing all their rigid temperature sensors with flexible sensors because of the cost savings they expect to gain.

• Maintenance will be easier, take less time and cause fewer shutdowns or process interruptions.
• Fewer thermowells will have to be replaced because of sagging or foreign debris that clogs the wells.
• Only two or three standard sensor lengths will be needed for an entire plant, reducing the spare parts inventory.
• The refineries will get better measurements in shorter thermowell applications, leading to increased accuracy and energy savings.

Although conventional rigid temperature sensors have proven to be a workhorse for the past 50 years, modern flexible sensors are now starting to replace them across Louisiana.

Robert Poole is an engineer with Process Measurements & Monitors, in Baton Rouge, LA.

Continuous vibration monitoring of pump stations at a major wastewater treatment plant pays off for the City of Tampa.

The plant has developed and is currently executing an optimization program that includes automation of processes and procedures when possible, and reducing scheduled vs. unscheduled downtime and maintenance, transitioning from a reactive to proactive organization ready to address issues and problems. Because the Howard F. Curren facility is the City of Tampa’s (COT) only wastewater treatment facility, it is imperative to minimize flow interruptions, unscheduled downtime and overflows.

The use of reliable pumps to transport wastewater from various locations in the city is critical for maximizing flows and maintaining biological efficiencies by producing a constant flow. When the pumps fail, backup pumps are used to keep the flow going. Failures often can be very damaging to the pumps and auxiliary equipment. Installing a protection system that monitors the vibration levels and can be integrated to a shutdown circuit can minimize flow interruptions and the amount of costly damage to that equipment. The price of a new pump motor can be as high as $450,000; the cost to repair an existing unit can approach $175,000 after a catastrophic failure. In an effort to help prevent these types of failures, the HFCAWTP and Connection Technology Center, Inc., a vibration analysis hardware and process equipment manufacturer, investigated different equipment and system options for monitoring this crucial application.

The application
There are eight major pump stations that collect the wastewater and deliver it to the treatment plant. Each major pump station has many smaller stations that will feed it—either through pump systems or gravity feed. There are approximately 224 pump stations within this system.

Three types of pumps setups are typical of these stations: Direct coupled, submersible and vertical shaft. The direct coupled stations will have the motor and the pump on the same floor, with the motor in an overhung position and supported over the pump. The vertical shaft stations will have the motor and clutch or VFD-controlled motor typically two stories above the pump, with the shaft coupled in one or two places.

Each major lift station has three or more motor-pump systems, with one pump typically running at a time to ensure system redundancy. Major failures can cause overflow issues, not to mention extensive damage or complete failure with auxiliary equipment such as valves, VFDs and wiring.

For the purposes of determining where to install a protective system, the major stations were identified as the areas to have the critical equipment monitored.

Vibration considerations
The general vibration considerations that are periodically monitored in the pumping systems at this plant include cavitation, mechanical failure and mis-alignment. Cavitations often will accelerate the mechanical failures of the pump, such as discharge valve failures and impeller wear. Faults due to mechanical issues also are accelerated due to increased flow. Possible mechanical failures include breaking or dropping the impeller or impeller shaft and/or bearing failures.

Other unique vibration considerations at this plant are associated with the alignment of the vertical shafts to pumps, requiring coupling shafts up to 20 feet in length, and accessibility of the equipment, which is often very difficult.

The application is made even more challenging by the fact that these remote pump stations are not manned, and the periodic monitoring may not be sufficient to capture any transient type of faults that could lead to failures.

Process/protection considerations
Periodic monitoring may be sufficient to identify general, long-term machinery conditions, but to capture transient conditions that can cause catastrophic failures, continual monitoring is required. Because the pump stations are unmanned, a system is in place to alert a technician at the plant that there is an issue with the pump station equipment. If there is an issue, corrective actions may be necessary in order to prevent the premature failure of the equipment and overflows.

Ensuring this capability required integration of the vibration system with the plant SCADA system. The output parameters of the vibration system, in this case 4-20mA output proportional to the overall vibration levels of the equipment, will feed into the SCADA system and allow the technician to observe a "status" of the equipment at the stations. This is an ideal situation, as many issues can be identified quickly before the effects of a catastrophic failure occur. However, this integration is often difficult based on the available resources of both the SCADA system and the plant personnel to integrate this.

Another solution that can be implemented as a stand-alone or integrated with the SCADA system is to provide a local relay or shutdown system that can be tied into the motor control circuit to shut down the pump system in the event of a catastrophic failure. Such a solution can limit the extent of the damage to the pump and limit/prevent the damage to auxiliary equipment, as well as minimize interruptions of the flow to the plant.

Equipment & system selection considerations
For the initial unit, a system of low-cost accelerometers mounted to mounting targets connected to a remotely mounted process controller enclosure was specified, with integration to the main plant SCADA system. The equipment was selected based on the following considerations.

Accelerometer selection…
To select the proper accelerometer for the monitoring of components, the following vibration frequency criteria was taken into consideration:

• Pump vane frequencies
• Pump cavitations frequencies
• Motor fault frequencies
• No clearance issues that would require low-profile sensors
• Historical vibration data and experience with the equipment

Frequencies for detecting vibration faults should be within the frequency response of the selected accelerometer. For accelerometer specification, the motor and pump vane frequencies did not require a special frequency response, and a standard, 100 mV/g accelerometer, with a frequency response between 0.5 - 15000 Hz, was selected for this application.

Mounting hardware selection…
To provide the optimum vibration transfer between the machine surface and the accelerometer, a mounting system that utilizes the full frequency span of the accelerometer needed to be considered. A mounting target attached to the prepared machine surface (prepared with an installation tool kit [MH117-1B] that can be resharpened for multiple installations) with an adhesive was selected. The adhesive-mounted target facilitates excellent vibration transfer, and the full frequency range of the sensor can be utilized. Another advantage to the adhesive-mounted target is that the machine surface does not need to be drilled and tapped. A flat mounting target with a ¼-28 threaded hole was selected for this function.

Cable selection…
In light of the environment, the cable connecting the accelerometer to the enclosure needed to be robust, chemical resistant, water resistant and reliable in caustic conditions. A Teflon-jacketed cable with molded connector and stainless steel locking ring was chosen.

Signal conditioner selection…
Because of the required inputs into the process controller, a field-configurable signal conditioner with a display that can be easily seen in a variety of lighting conditions was chosen, as each pump that is monitored can have unique vibration levels. The signal conditioner also needed to be able to re-transmit the 4-20mA outputs in order to eventually integrate with another process control system and SCADA. Power for the signal conditioner(s) and the sensors are provided by the internal process controller.

Process controller selection…
The selected process controller allowed for field configuration, incorporated a display that permitted visual identifi- cation of the vibration level and included a power supply for the signal conditioners.

The ability to set up two different alarm levels, as well as a time delay to prevent "nuisance alarms" that might occur if a spike in vibration levels due to a transient event also was determined to be important for this system. The controllers are powered from 120 VAC input into the enclosure, which was provided by the facility.

Enclosure selection…
The selected enclosure allowed for easy wiring into and out of it. This enclosure also has proven to be unaffected in a highly corrosive atmosphere. The process controllers and the signal conditioners were factory-wired. The wiring of the sensors into the enclosure, any re-transmitted signals out of the enclosure and 120 VAC power into the enclosure were done through pre-defined cable entry and exit cord grips/conduit. The wiring was attached at a termination block that was clearly identified for the type of connection required. (See Fig. 1 for an example of the termination identification.)

The easy wiring minimized the time required to install sensor cables and integrate the components of the system into the enclosure, and ensured that the system was completely integrated prior to delivery.

Financial analysis
Justification for the Howard F. Curren Advanced Wastewater Treatment Plant project was determined based on a review of the approximate cost of a pump station motor repair versus the price of a typical two-channel monitoring system. The repair cost for an 800 hp motor could go as high as $175,000. The price of the monitoring system was approximately $2500—or roughly $1500 per measurement point.

The initial approval to outfit one major lift station was decided in 2006, and a unit has been in service since that time. The project justification was further underscored by a subsequent motor failure at another pump station. The estimated cost of that motor repair was close to $160,000—a fact that renewed interest in the relatively low-cost 24 hour protection device.

Approved monitoring setup
The approved system was to be used as a monitor to notify the plant of problems with the pump or motor, especially during off-hour operation. As shown in Fig. 2, this system consists of two permanently mounted sensors, with cable from the sensor wired to the enclosure. Mounted inside are: two process controllers, two signal conditioners, and two transmitters (for the 4-20 mA output process signals). The box also has a window to permit viewing of the process controller displays for overall vibration level readings.

• The signal conditioner was scaled to less than 0.51.0 IPS, with a frequency range between 5 and 50 Hz.
• Two relay outputs were configured based on experience in required alarm settings. The baseline vibration on the machine was observed to be 0.2 IPS, peak. From there, relay/alarm settings were set at 0.35 IPS, peak for the first level, and 0.65 IPS, peak for the second alarm level, with time delays of approximately 30 seconds for each level. If the vibration does not maintain that amplitude (or greater) for that length of time continuously, the relay does not activate. The levels, time delays and relay action (latching, latching with clear, manual reset) can be adjusted on the process controllers.
• The system was mounted at a lift station with a flow capacity of approximately 35 MGD and connected to the main plant SCADA system. Relays are in place to shut down the pump/motor if there is an event that could cause serious damage to the equipment. Sensor location selection The sensor mounting locations were selected based on historical data and accessibility of the measurement location point. In order to monitor the pump and motor, for the direct driven system, a sensor was placed on both pump and motor. Enclosure mounting location selection The cable was routed from the pump and motor to the enclosure, which was mounted on a fixed wall. This is located near the shut-off switch, which was installed to protect the pump and motor equipment. Major benefits of the system can be seen in the following features and capabilities:
• A turn-key system solution
• Easy wiring terminations
• Field-configurable signal conditioners and process controllers
• Allows for re-transmission of the process signal
• Allows for integration into a SCADA system
• Allows for settings to shut down the equipment
• Two relays with independent input levels with latching options
• User-friendly components
• Permits access to "live" data to hard to inaccessible points
• Offers multi-functions vibration and temperature

Results
The installed system has identified possible pump cavitations occurring in the early morning hours during low-flow periods. These types of cavitations can escalate rapidly, putting a pump and motor in danger. For example, another station at this plant that did not have the approved system in place subsequently failed—possibly due to cavitation—requiring repairs to the equipment and costly unscheduled downtime.

Conclusion
The following factors were critical in convincing management that vibration monitoring has benefits to the Predictive Maintenance Program and City of Tampa (COT) and could be considered for expansion into other pump stations:

1. Cost of the equipment is much less than the cost of repair or replacement of pump and motor
2. The system protects critical equipment with relays to trigger alarms or shutdown
3. 4-20mA outputs feed into SCADA system for continuous, online monitoring.
4. Continuous monitoring can identify possible issues that would not have been observed otherwise.
5. Protecting pump and motor systems during increasedflow events can reduce unscheduled maintenance or repair by alerting the plant of issues before they become catastrophic.
6. The system permits easy access of dynamic data for route collection and/or detailed analysis.
7. Required maintenance will be identified more precisely and accurately, thus reducing unscheduled downtime, repair cost and overflow issues.

Tom LaRocque is the engineering manager for Connection Technology Center, Inc., in Victor, NY. A Certified Vibration Analyst: Category III, he holds a B.S. in Engineering from Clarkson University. LaRocque is a member of the Central New York Chapter of the Vibration Institute. Telephone: (585) 924-5900 ext. 817; e-mail: tlarocque@ctconline.com

Gary Kaiser is a senior application engineer for Connection Technology Center, Inc. A Certified Vibration Analyst: Category III, he previously worked for Eastman Kodak for 23 years. While at Kodak, Kaiser spent 9 years in the vibration analysis group. He also is a member of the Central New York Chapter of the Vibration Institute. E-mail: kaiserg@ctconline.com

Joe Spencer is a mechanical specialist with the City of Tampa, fl. A Certified Vibration Analyst, he has 30 years of field maintenance experience.

Jane Alexander, Editor-In_Chief

Capacity assurance is not a new term—it's been around for many years. Those of you in the maintenance and reliability community are no doubt quite familiar with it, since it's all about maximizing uptime, minimizing downtime, running safely, cleanly, efficiently and profitably.

The task of keeping modern plants running at peak capacity, however, goes well beyond the area of traditional maintenance and reliability (although those elements are more important than ever as key capacity assurance components). It encompasses all activities necessary for ensuring that your equipment and systems are capable of operating at prescribed output and quality levels whenever scheduled or needed. In other words, capacity assurance is the "fat rabbit" everyone in a company is chasing 24/7/365—and we do mean everyone. Therefore, being successful in this chase requires a "holistic," integrated approach to maintenance, operations and management.

We at Maintenance Technology have long recognized how critical it is for you in industry to be able to catch the capacity assurance rabbit quickly—continuously. In fact, we've been championing the types of integrated approaches and solutions that help you get the job done for more than 20 years. Today, though, and into the future, with so much riding on a company's ability to assure capacity, we feel compelled to be more specific in our own approach.

Time has marched on. Technologies, applications, operating parameters and business environments have changed. So have your jobs, your time constraints and your information needs. What has not changed is the importance of capacity assurance across your operations—and the fact that countless organizations are pushed to get more, more, more of it with less, less, less.

Putting our editorial spotlight on "capacity assurance" as opposed to "plant equipment reliability, maintenance and asset management" will allow us to better serve you and other busy readers. You've been seeing us move in that direction for some time, placing increased emphasis on failure avoidance and the operating equipment and systems where preventive and predictive maintenance technologies are applied than we have in the past. Our quarterly supplements, "Utilities Manager" (focusing on successful demand-side energy solutions for plants and facilities) and "The Fundamentals" (taking a back-to-basics approach to maintenance and reliability), are two other prime examples of our sharpened focus. Now, going forward, you can expect even more great "new" things from us.

You know it and we know it... Excellence in capacity assurance is vital to industrial profit and world-class quality. In our view, it's one of the fattest rabbits out there. Maintenance Technology is proud to be your partner in this noble and exciting chase.

Bob Williamson, Contributing Editor

Capacity assurance is not a new term—it's been around for many years. Those of you in the maintenance and reliability community are no doubt quite familiar with it, since it's all about maximizing uptime, minimizing downtime, running safely, cleanly, efficiently and profitably.

The task of keeping modern plants running at peak capacity, however, goes well beyond the area of traditional maintenance and reliability (although those elements are more important than ever as key capacity assurance components). It encompasses all activities necessary for ensuring that your equipment and systems are capable of operating at prescribed output and quality levels whenever scheduled or needed. In other words, capacity assurance is the "fat rabbit" everyone in a company is chasing 24/7/365—and we do mean everyone. Therefore, being successful in this chase requires a "holistic," integrated approach to maintenance, operations and management.

We at Maintenance Technology have long recognized how critical it is for you in industry to be able to catch the capacity assurance rabbit quickly—continuously. In fact, we've been championing the types of integrated approaches and solutions that help you get the job done for more than 20 years. Today, though, and into the future, with so much riding on a company's ability to assure capacity, we feel compelled to be more specific in our own approach.

Time has marched on. Technologies, applications, operating parameters and business environments have changed. So have your jobs, your time constraints and your information needs. What has not changed is the importance of capacity assurance across your operations—and the fact that countless organizations are pushed to get more, more, more of it with less, less, less.

Putting our editorial spotlight on "capacity assurance" as opposed to "plant equipment reliability, maintenance and asset management" will allow us to better serve you and other busy readers. You've been seeing us move in that direction for some time, placing increased emphasis on failure avoidance and the operating equipment and systems where preventive and predictive maintenance technologies are applied than we have in the past. Our quarterly supplements, "Utilities Manager" (focusing on successful demand-side energy solutions for plants and facilities) and "The Fundamentals" (taking a back-to-basics approach to maintenance and reliability), are two other prime examples of our sharpened focus. Now, going forward, you can expect even more great "new" things from us.

You know it and we know it... Excellence in capacity assurance is vital to industrial profit and world-class quality. In our view, it's one of the fattest rabbits out there. Maintenance Technology is proud to be your partner in this noble and exciting chase.