The OSUOSL fosters open source development projects as a service to the community. The lab “is all about using open standards to promote technologies that help the University stimulate its lasting attitude of inquiry and social responsibility,” according to the FAQ at its Web site.

Corey Shields, lead systems engineer for the lab, oversees the management of 60 servers that are logged to a central host using syslog-ng and stunnel. Before Splunk, the log host environment was set up to output files into directory hierarchies, with new files for each day. When there was a problem with a server, Shields had to search through all the log files manually to find out what was wrong. “To find problems in this setup, there is a lot of grep and awking to do,” Shields says, “and that is when you know what you are greping for.” Looking through the logs by hand took a lot of time, especially when multiple servers were performing the same functions and generating gigabytes of log data every day.

Shields remembered seeing Splunk demonstrated at San Francisco’s LinuxWorld Conference and Expo last year. “I thought it was a pretty good idea.” Splunk is kind of like Google for server logs. It “sucks up every type of log you care to feed it, indexes them, and then makes them easily searchable via a nifty AJAX-enabled Web interface,” writes OpenSolaris.org’s Ben Rockwood.

Shields says installing Splunk on a Hewlett-Packard ProLiant DL140 server running Debian took just a few minutes. “Everything is pre-packaged, and the installation asks a few questions. It indexes data in quite a few different ways. I played around with some of the methods for a day, and settled on having syslog-ng output data to a named pipe which Splunk then watches,” he says. “This allowed us to keep our existing log host configuration, and use Splunk to supplement it.”

“I had a slight challenge in the beginning getting data into Splunk,” Shields says. He posted a question on the user forums and got a “quick response” that got him back on track.

“Almost immediately Splunk showed its worth in helping to find problems I didn’t even notice the symptoms of,” he says. “I was [using Splunk to] browse the logs of one of our development testbeds and noticed a cron job that was running every minute out of an old account from a developer who had left the group six months before. Given the alternative of just looking through the log one page at a time, I would not have been scouting for possible problems.”

Shields says the time-saving element of Splunk has proved invaluable to the lab. He hopes that in the near future Splunk will provide greater reporting opportunities. “Statistics are important,” he says. “They can mean bragging rights, new job lines, resource needs. When you have a cluster of machines all performing the same job, the difficulty of collecting statistics seems to increase with the size of your resources. I would like to see awstats-like reporting from any given search or data set on the fly. Overlay a couple of search results graphed on top of each other and you get to compare trends. Now, wouldn’t that be great?”

Splunk is available either as a freeware download or a commercial application called Splunk Professional, which is priced based on the amount of data that needs to be indexed per day and charged on an annual basis. A company that needs to index up to 500 megabytes per day would pay $2,500 a year.

Patrick McGovern, one of the founders of Splunk, says his company is looking at the possibility of opening parts of the source code within the year. “At this point we are doing a bit of a hybrid,” McGovern says. “We’re providing the software at zero cost, and we are providing all the API to allow developers to extend and customize our search engine to suit their needs. We’re also keeping parts of the code closed, so that large data centers have a reason to purchase the professional version of our software.”