"We've gone from needing small clusters to needing bigger and bigger ones," said associate professor of crop sciences Matt Hudson, "When I first got here, I had my own cluster in the closet downstairs. It was pretty reasonable. It had 20 nodes and did everything we wanted."
Like Hudson, many people on campus have built computer clusters within their departments or colleges, leading to duplication of resources. As researchers generate increasing amounts of data, these clusters are rapidly outgrown.
The Illinois Campus Cluster is expected to address this problem. Individuals, groups, and campus units can invest in compute and storage resources on the cluster, which is housed at the Advanced Computation Building, a facility specially designed to support HPC systems.
Researchers are guaranteed access to the number and type of nodes in which they invested, and they can also access any unused nodes. In this way, they get more computing capacity for their money.
Pooling computer resources helps researchers control expenses and reduce overhead, shortens startup time, reduces the need for space renovation, and frees up space formerly used for local clusters. It also reduces the campus carbon footprint.
Why do ACES scientists need bigger and faster computers?
They need them because, over time, their ability to generate data has outpaced the speed and capacity of computers. This applies to environmental scientists and agricultural engineers as well as biologists such as Hudson, who uses DNA sequencers capable of generating billions of base pairs per day.
In most clusters previously in operation in ACES, communication between the nodes limits the processing speed. Because of this bottleneck, adding more machines does not make the cluster faster, so the size of the clusters has been limited.
"Everyone says that genome sequencing is getting cheaper and easier," explained Hudson, "but it turns out that some plants are very difficult to do, and they are very refractory to the methods that are out there now."
"The dedicated biofuel crops, like sugar cane, Miscanthus and switchgrass, have been tough, so we haven't completed a genome sequence of any of those yet," he added. One reason is that the cells are polyploid, meaning that they contain more than two sets of chromosomes.
Moreover, the genomes are packed with repetitive transposons, which are short, mobile DNA sequences that can replicate and be inserted at random sites in the chromosome. "Trying to figure out which one goes where, especially with the newer technologies, is difficult," Hudson explained.
Another problem is that the datasets, particularly for biology, are very large, often several terabytes (one trillion bytes). "When the dataset is stored on a hard disk or a set of hard disks, all of the worker nodes have to be able to access it. The speed of getting the data on and off the hard disks becomes limiting too," said Hudson. The campus cluster uses a new way of storing data, called General Parallel File System (GPFS), that alleviates this problem.
ACES was the second college to buy into the campus cluster, with 36 compute nodes, 432 cores, 1.4 terabytes of random access memory (RAM), and 20 terabytes of additional storage. Researchers in crop sciences, agricultural and biological engineering, animal sciences, and food science and human nutrition will be using it. Any researcher with an ACES affiliation can apply for an account to use the system.
Hudson has already started working with the cluster. "It's had a few teething troubles," he admitted, "but it's very, very fast."
The idea is that researchers will keep investing in successive versions of the cluster. Hudson said, "Each successive generation is going to be bigger and better than the last one."
"My hope is that having this cluster available to the college is going to make people able to do a lot of things they couldn't do before," he added. "Not many colleges of ag have access to this kind of thing, and not many of them are seeing the potential in this, so there is definitely an opportunity for us to be a leader in this area."