Experimental cooling setups

…And it’s been over a year again.

In my defense, during that year we’ve been working pretty hard to get the Exactum Greenhouse to produce foodstuff, with more than moderate success. But this post is not about the Greenhouse or the chilies contained therein. This post is about the Nidec DR04XLG-12PUS1 dual fan units that have been failing systematically in the Helsinki Chamber.

For over three years, some of our servers have withstood the harsh winters and exceptionally sunny summers of our nordic climate. But not all. A batch of HP DL360 g3 servers did all fail, and in all failures the root cause was the same: the CPU fan block.

Now that the defense of my Ph.D. thesis looms nearer, we are trying to finalize an article about the long term effects of free air cooling. Generally speaking, all other components have not exhibited systematic failures. Since our set of hardware is pretty diverse, it seems that our experiment has been somewhat of a success.

But for the article, we need to delve deeper, and try to explain what precisely goes wrong with the CPU fan blocks.

The outer part of the dual fan cut open for multimeter measurements. One of the units is working, the other two dead.

As mentioned before, the Nidec dual fan model is designated DR04XLG-12PUS1. It consists of a three-blade “inner” unit designated R04G-12PS1 B, and a five-blade “outer” unit designated D04XL-12PU B. It is the outer unit which fails. Previously, I had thought that the outer unit still spins up and rotates, but this proved to be false. Of the failing CPU fan blocks, most of the outer fans fail completely and do not spin up. Occasional spasms may be seen, but that is all.

Luckily, there wasn’t just a single failure. Of the 5-6 CPU fan blocks, almost every single one displayed the “1611 CPU Zone Fan Assembly Failures“. This means that we have plenty of broken fans to experiment with, and that is precisely what we did.

With the help of Mikko “dogo” Rantanen, we cut open several of the broken fan units. We also cut open a fan unit which still worked in order to get a comparison point. Then, we started to figure out what is the difference between a live and a dead unit.

Our initial hypothesis was that the coil wires might be faulty. This could have meant that the insulation around the wires had oxidized and caused a short circuit, preventing the motor to spin up. In order to test this hypothesis we measured the resistance across the circuit board’s connections to the coil wires.

The three solder points to the coil wires are marked A, B, and C.

The circuit board has three connection points to the coil wires. We designated these points as A, B, and C. Their pairwise resistances are as follows

• A-B ~4,8 Ω
• B-C ~2,4 Ω
• A-C ~4,8 Ω

Except in a failing unit. For a unit that refuses to spin up, the pairwise resistances are higher, but not precisely double:

• A-B ~8,2 Ω
• B-C ~16,2 Ω
• A-C ~8,2 Ω

What does this mean? We don’t precisely know yet. What we know at this point is this:

1. There seemed to be no short circuit, so our initial hypothesis was abandoned
2. The soldering points seem to be working as they should, so it’s not a problem with the solder
3. We can deduce if a unit has failed based on the resistance readouts

Perhaps during next week, we will find out more.

I confess to having had a bad case of writer’s block with this blog. Subsequently, there have been no updates for the past seven months. I’m correcting this issue now.

The cause for the blog blackout has been a number of fan failures affecting a specific brand and model of servers in the Helsinki Chamber (HC). As a scientist, I’ve tried to figure out conclusive proof for the cause of the faults, and having been unable to do so, I have been unwilling to publish partial results. My hope is that this blog post might attract the web searches of a system administrator who has experienced similar failures and been able to deduct their root cause.

In our case, the failing servers are 1U sized HP DL360 G3 models. In the two years we’ve been running experiments using free air-side cooling using direct, unconditioned outside air, these servers have been the only ones to exhibit systematic failures.  This kind of failures has also been called common-mode (CMF) or common-cause failures (CCF). It is of particular interest to me, as this phenomenon is the very one that I originally set out to study with the direct free air cooling experiments.

Figure 1: HP DL360 G3 CPU fan block

Caveat emptor

We have been using the same HP DL360 G3 models in our regular data center for years, and in no way have they been more or less prone to failures than other server brands or models. I am not claiming that these models are flawed although they do fail regularly when cooled with outside air. Similarly, I can not claim that direct free air cooling would be an unfeasible technique: the numerous other models we have used have not exhibited CMFs. What can be said is that with a reasonable probability, there exists a server fan type which is unsuitable for direct free air cooling, and a number of other server fan types which remain suitable.

Symptoms

The servers in question start to fail with the following warnings in their HP Integrated Management Log (IML).

Event: 20 Added: 06/14/2011 02:32
CRITICAL: Machine Environment - Fan Failure (Fan 1, Location CPU).
Event: 21 Added: 06/14/2011 02:32
CRITICAL: OS Class - Automatic Operating System Shutdown Initiated Due to Fan Failure.
Event: 22 Added: 06/13/2011 23:59
CAUTION: POST Messages - POST Error: 1611-CPU Zone Fan Assembly Failure Detected.
Event: 23 Added: 06/13/2011 23:59
CAUTION: POST Messages - POST Error: Fan Solution Not Sufficient.
Event: 24 Added: 06/14/2011 00:59

The errors are duplicated into syslog through the hpasmd daemon, if it is running:

Oct 23 09:00:15 lost25 hpasmd[747]: CRITICAL: hpasmd: Fan Failure (Fan 1, Location CPU)
Oct 23 09:00:15 lost25 hpasmd[747]: CRITICAL: hpasmd: Automatic Operating System Shutdown
Initiated Due to Fan Failure

Oct 23 09:00:22 lost25 hpasmd[747]: NOTICE: hpasmd: Fan Failure (Fan 1, Location CPU)
has been repaired
Oct 23 09:00:22 lost25 hpasmd[747]: NOTICE: hpasmd: Automatic Operating System Shutdown
Initiated Due to Fan Failure has been repaired

What happens is that the CPU fan block depicted in Fig.1 tells the system management board that there is a failing fan. As there does not seem to be any redundancy in the block despite the four fan assemblies, even a single faulty fan is enough to cause the errors to surface in the block. I have experimented this by shifting fans one-by-one from a malfunctioning server to a correctly functioning server, until the latter starts to show the same symptoms as the previous.

Initially, the errors are simply warnings and the malfunctioning server will recover, aborting the automatic operating system shutdown. Later on, this will happen less and less, causing the server to go into a delayed reboot loop. The OS will shut down and the server will remain shut down for a varying number of minutes, and after this, the system board will try again. After a few minutes of operating, a new critical warning is logged and the OS is shut down again.

Attempted repairs

Despite the logged errors, visual inspection reveals that the fan assemblies remain operational and keep rotating until the shutdown. Since we have been running the servers in pretty low temperatures, even a completely dead fan block would probably have been sufficient for normal operation.

In our case, a total of seven HP DL360 G3 fan blocks have failed with this type of problem. We initially installed five units in the HC, and after discovering these problems, I replaced two fan blocks with used fan blocks from spare servers. I also reconstructed three more “correct” fan blocks by marking the failed fans and shifting unmarked fans until a fan block no longer reported problems.

These replacements rule out individual events like power spikes which might have destroyed the fans. Likewise, web searches reveal no error reports supporting the idea that the problems would be caused by faulty firmware, perhaps solvable through an upgrade. As only the fan blocks placed in the HC have failed, the root cause does not seem to be a manufacturing or handling error either. Finally, the problems are not caused by the fan assemblies clogging with pollen or lint, which I have verified by breaking down the assemblies into their base components.

The end result is that all seven fan blocks ultimately caused delayed reboot loops, and we were forced to remove the HP DL360 G3:s from the HC. The identical models purchased at the same time in our regular data center have not caused problems, and neither has the 2U-sized HP DL380 G3 which is still installed in the HC.

Figure 2: Nidec DR04XLG-12PUS1 fan units

Failing fans

The fan used in the CPU fan block are Nidec DR04XLG-12PUS1 40*40*48 mm units. A web search will yield other users who have had problems with this fan type, but not enough to claim that the model would be systematically erroneous. Our own control groups also disproves this idea, as the servers in our regular data centers have not failed.

What is peculiar about these fans is that they are double fan units, i.e., there are two fans connected in a serial fashion. In addition, the second fan is reversed and rotates in the opposite direction. This behaviour is our current best guess on what goes wrong.

Buest guess at cause

The motor section of the Nidec fans is visible in Figure 2. As in normal fans, the motor is located in the center of the fan unit and the fan blades rotate in front of the motor unit. My theory is that in a normal fan, the blades work somewhat like an umbrella, pushing the humidity in the air away from the motor shielded in the middle of the unit.

Since in the units the second fan unit is reversed, the internals of the motor become more prone to any humidity in the air. This might cause transient faults which the fan unit then reports to the fan block, and onwards to the system management board.

Solutions?

So far, we have figured out no solutions for the problem. What I’m planning is an emulated experiment by removing one of the DL360 G3:s from the control group and trying to see if I can make the server fail indoors by raising the relative humidity of the air. If so, the cause of the problems is the humidity combined with the reversed air flow.

If there are any readers with hands-on experience with this type of errors, we are interested in hearing from you. My user account is pervila at the Department of CS servers, so you can easily figure out my e-mail address.

I contacted Halton‘s Risto Kosonen and enquired whether Halton would be interested in participating in our research. After a few phone discussions, Risto caught the idea and promised to consider it. Roughly a week from that, I got a call from Exactum‘s doorman. We had received quite a hefty package.

The outdoors ventilation grille in the picture above is designed to cover a building’t supply air intake. The fins are form roughly 45 degree angles and permit air trough with a minimized resistance. Rain is almost completely blocked by the sharp upwards turn, and the water will gather at the bottom of the grill, from where it will simply flow down a connecting wall.

We used the grille for the exactly opposite purpose, and replaced our home-grown exhaust cover with the grille. As we didn’t want to waste the exhaust cover, it was repurposed as the supply cover. Thus, exhaust air flows straight through the grille and out, but supply air is forced to make the 90 degree turns described earlier. The following video displays the current configuration.

Helsinki Chamber with improved air flow (on YouTube)

With these changes, we are able to supply air with a temperature elevation of within 1 C of the ambient temperature. The drawbacks are that we can no longer view the computers leds through the clear plexiglass windows. In addition, the wing nuts we have been using to secure the plexiglass covers are clearly suboptimal for the grille and our cover hat. Finally, snow remains a problem with both covers. Very fine snow could very well be carried either with the intake suction or through the rear grille, given a sufficient wind speed.

The next step is to connect both covers with sliders so that an administrator can easily view what the situation inside the chamber is. As for snow, I have no idea yet. Switching to the plexiglass covers avoid the problem, but this is troublesome if an admin operates more than a single chamber.

After quite a lot of design work, I decided to ask Ville Hautakangas for help. Ville helped me to improve the design and solved most of the manufacturing problems. Our solution consisted of a “cover hat” for the exhaust chamber, similar in design to some ventilation pipes used in housing and appartment buildings.

A master craftsman at work

As materials, we used 5 mm thick corrugated plastic sheets from Zymotec. Model airplane enthusiasts have used the same material for miniature construction, since the sheets are easy to handle and quite strong. We had previous experience with corrugated plastic sheets from the second version of our home-grown cold aisle containment setup, which I will describe later on in this blog.

Our idea for the cover was to force the exhaust air flow around two 90 degree angles and out from the exhaust chamber. For the air flow, this is a moderate flow impediment, but still manageable. Conversely, water can only reach the inner wall of the exhaust cover, from where it will run down through the open bottom section of the exhaust cover.

Sensor temperatures relative to ambient, May 2011

The graph above depicts sensor temperatures from the supply and exhaust chambers relative to the ambient (climate) temperature shown in green. The first part of the graph show that the exhaust section has gradually become quite warm, with elevations spiking 35 C above ambient 10 to 15 C, resulting in absolute temperatures of 45 to 50 C in the exhaust section.

The first drop in the graph occurs on 5.5.2011 (middle of week 18) after the installation of our home-grown rear cover. For many hours, everything seemed to be perfect with drops of over 20 C for the lower exhaust sensor, for example. Unfortunately, the following day showed that the temps were mostly normalizing. The effect is visible as the spike that immediately follows the surge.

This caused much head-scratching, as the exhaust air flow had definitely improved. Ville turned out to be right in his guesstimate: the supply chamber had to be improved as well. We had encountered both exhaust overpressure and supply underpressure, and both had to be resolved separately.

(The graph is somewhat of a spoiler, for in the end, our solutions did work.)

Over the past two months, we have encountered steadily raising temperatures within the prototype Helsinki Chamber. After some investigation, we had to conclude that both the exhaust and supply air flows were insufficient. During experiments, two interesting new phenomena occurred

1. Supply underpressure, whenever the supply is unsufficient
2. Exhaust overpressure, whenever the exhaust is insufficient

During supply underpressure, the servers are simply provided insufficient supply air. This causes suction on the supply chamber, which forces some of the exhaust air to reverse flow through the servers and back into the supply chamber. Exhaust overpressure works similarly, but the cause is the insufficient air flow out of the exhaust chamber.

Both effects can occur on their own, and both can be verified by allowing the HC to run without either the front or back covers. Moreover, both effects have their counterparts in production environments using either cold or hot aisle containment. Supply underpressure can happen whenever the CRACs do not supply enough cool air into the cold aisle. Exhaust overpressure is more elusive, but will happen if a hot aisle has insufficient exhaust air suction.

These effects were initially very depressing, because from an engineering perspective they signalled that only a full “wind tunnel” would be ideal for server air flow. Such a tunnel would be very tricky to construct without also allowing rain or snow inside the chambers. This meant that a certain researcher had to return to the design whiteboard.

Initial sketches on how to fix the problem