Experimental cooling setups

You know what this thing needs? A twitter-interface, that’s what!

You can now follow eight of the servers inside the Helsinki Chamber via Twitter. Each of the eight machines is running its own little monitoring-software that listens to our @helsinkichamber twitter-account and responds with information about it. We think this is really cool, and urge you to start stalking us there.

A hot day in Helsinki? Just @mention on of our servers and see if they’re frying inside their chassis!

A freakishly cold night? Had to pull another blanket on just so you wouldn’t freeze in your bed? That’s nothing! Our Helsinki Chamber just spent the night standing outside, in the cold, running menial and repetitive tasks! Ask for a graph from all of the servers and see which one froze the most!

Sounds interesting? Let’s introduce our servers:

• lost20: A Dell Poweredge running a 3,2 GHz Intel Xeon CPU. It’s hobbies include tennis and prime numbers.
• lost24: An old HP Proliant DL380G3 that still mourns the loss of Michael Jackson. On its free time lost24 tries to moonwalk, but you know, servers ain’t got no legs…
• lost25: The oldest of the four Proliant DL360-brothers in our rack, it was born 2 minutes before the others. Lost25 was always the responsible one of the four, and is considering a career in politics.
• lost26: The second oldest brother collects pins in its spare time and keeps them inside its spare 3.5″ drive bay.
• lost27: The third of the bunch is kind of a dandy. He prefers classic menswear and reeeeally hates it when the other servers come to work wearing cargo-pants and t-shirts with wolves on them.
• lost28: The youngest of the four likes pizza, is kinda reckless, wears an orange bandana and is deadly with nunchucks. Also thinks its a turtle.
• lost29: Still thinks jiggawatts is a real number.
• lost31: Is really just a combination of clever engineering, silicon, plastic, copper and aluminum, and has no persona, soul or consciousness. If we interpreted lost31:s log-files correctly, it wishes to join the local community theater. Either that, or sshd is mis-configured.

All kidding aside, @mention helsinkichamber, name the server (or servers) you want to query, and what you want from them.

So for the latest temperatures from lost20 you would say:

Or for graphs of the latest 24 hours from lost20 you would say:

So, what do you think? Are they crash-y? Could be. Did one of the servers just overheat? Maybe, that would actually be kinda something… Would you like to know more about the servers? Hit us up with a comment.

Have fun!

An open research problem concerns whether a given machine M is operating correctly at a given point in time t. There are a number of proprietary checking programs that can be run against a specific sub-component, e.g., hard disk, but these are mostly offline tests and unsuitable for our operating environment. It would have quickly become tedious to regularly reboot the servers with MS-DOS diskettes to run diagnostic programs.

Thus, we decided on a more pragmatic test. Our servers are defined to be operating correctly if they can successfully

1. Compress the Linux v. 2.6.32.8 source code directory into a tarball file temp.tar.bz2
2. Calculate the md5sum hash over the compressed tarball file temp.tar.bz2
3. Verify that the md5sum hash does not differ from its initial value, as calculated indoors and assumed to be correct

This pattern is repeated every 20 minutes. It is designed to stress both the disk, CPU, and memory subsystems. If a deviation of the calculated md5sum pattern is found, the file temp.tar.bz2 is saved for further analysis. All checksum files are also transferred via scp to an external host to mitigate the effects of correlated disk failures (yielding broken arrays etc). This results in a small amount of additional load per server.

Interestingly, we do find broken md5sum hashes. In phase 2 we detected 6 differing md5sum calculations out of a total of 119 516 check executions. Four of the mismatches were found in the control group and two in the test (tent) group. By examining the resulting tarballs with the bzip2recover program, we found that only singular blocks had become corrupted in the packing process.

It is still unknown why the md5sums sometimes fail. Our best hypothesis is that the errors are caused by memory page faults, since none of the rack enclosures equipped with ECC DIMMs have snown problems. The six faulty sums were found from three identical COTS machines, making it possible that the batch of memory modules used in the original setups was partially faulty. By estimating the amount of memory pages read and written in the 119 516 check executions, we project the failure ratio to lie in the ballpark of one per 2,5 billion.

For phase 3, the current plan is to let the servers execute some well-known collaborative computing program, for example folding@home. By doing this, our equipment stress would perhaps resemble normal operating environments somewhat better. If you have ideas about suitable client programs, do not hesitate to leave comments on this post.