Beowulf Linux Cluster

starting

Beowulf Clusters are scalable performance clusters based on commodity hardware, on a private system network, with open source software (Linux) infrastructure.

Each consists of a cluster of PCs or workstations dedicated to running high-performance computing tasks. The nodes in the cluster don't sit on people's desks; they are dedicated to running cluster jobs. It is usually connected to the outside world through only a single node.

What are Beowulf systems being used ? 

Traditional technical applications such as simulations, biotechnology, and petro-clusters; financial market modeling, data mining and stream processing; and Internet servers for audio and games.

One question that is commonly enough asked on the beowulf list is "How hard is it to build or care for a beowulf?"

Mind you, it is quite possible to go into beowulfery with no more than a limited understanding of networking, a handful of machines (or better, a pocketful of money) and a willingness to learn, and over the years I've watched and sometimes helped as many groups and individuals (including myself) in many places went from a state of near-total ignorance to a fair degree of expertise on little more than guts and effort.

However, this sort of school is the school of hard (and expensive!) knocks; one ought to be able to do better and not make the same mistakes and reinvent the same wheels over and over again, and this book is an effort to smooth the way so that you can.

One place that this question is often asked is in the context of trying to figure out the human costs of beowulf construction or maintenance, especially if your first cluster will be a big one and has to be right the first time. After all, building a cluster of more than 16 or so nodes is an increasingly serious proposition. It may well be that beowulfs are ten times cheaper than a piece of "big iron'' of equivalent power (per unit of aggregate compute power by some measure), but what if it costs ten times as much in human labor to build or run? What if it uses more power or cooling? What if it needs more expensive physical infrastructure of any sort?

These are all very valid concerns, especially in a shop with limited human resources or with little linux expertise or limited space, cooling, power. Building a cluster with four nodes, eight nodes, perhaps even sixteen nodes can often be done so cheaply that it seems ''free'' because the opportunity cost for the resources required are so minimal and the benefits so much greater than the costs. Building a cluster of 256 nodes without thinking hard about cost issues, infrastructure, and cost-benefit analysis is very likely to have a very sad outcome, the least of which is that the person responsible will likely lose their job.

If that person (who will be responsible) is you, then by all means read on. I cannot guarantee that the following sections will keep you out of the unemployment line, but I'll do my best.