The OSUOSL fosters open source development projects as a service to the community. The lab “is all about using open standards to promote technologies that help the University stimulate its lasting attitude of inquiry and social responsibility,” according to the FAQ at its Web site.

Corey Shields, lead systems engineer for the lab, oversees the management of 60 servers that are logged to a central host using syslog-ng and stunnel. Before Splunk, the log host environment was set up to output files into directory hierarchies, with new files for each day. When there was a problem with a server, Shields had to search through all the log files manually to find out what was wrong. “To find problems in this setup, there is a lot of grep and awking to do,” Shields says, “and that is when you know what you are greping for.” Looking through the logs by hand took a lot of time, especially when multiple servers were performing the same functions and generating gigabytes of log data every day.

Shields remembered seeing Splunk demonstrated at San Francisco’s LinuxWorld Conference and Expo last year. “I thought it was a pretty good idea.” Splunk is kind of like Google for server logs. It “sucks up every type of log you care to feed it, indexes them, and then makes them easily searchable via a nifty AJAX-enabled Web interface,” writes OpenSolaris.org’s Ben Rockwood.

Shields says installing Splunk on a Hewlett-Packard ProLiant DL140 server running Debian took just a few minutes. “Everything is pre-packaged, and the installation asks a few questions. It indexes data in quite a few different ways. I played around with some of the methods for a day, and settled on having syslog-ng output data to a named pipe which Splunk then watches,” he says. “This allowed us to keep our existing log host configuration, and use Splunk to supplement it.”

“I had a slight challenge in the beginning getting data into Splunk,” Shields says. He posted a question on the user forums and got a “quick response” that got him back on track.

“Almost immediately Splunk showed its worth in helping to find problems I didn’t even notice the symptoms of,” he says. “I was [using Splunk to] browse the logs of one of our development testbeds and noticed a cron job that was running every minute out of an old account from a developer who had left the group six months before. Given the alternative of just looking through the log one page at a time, I would not have been scouting for possible problems.”

Shields says the time-saving element of Splunk has proved invaluable to the lab. He hopes that in the near future Splunk will provide greater reporting opportunities. “Statistics are important,” he says. “They can mean bragging rights, new job lines, resource needs. When you have a cluster of machines all performing the same job, the difficulty of collecting statistics seems to increase with the size of your resources. I would like to see awstats-like reporting from any given search or data set on the fly. Overlay a couple of search results graphed on top of each other and you get to compare trends. Now, wouldn’t that be great?”

Splunk is available either as a freeware download or a commercial application called Splunk Professional, which is priced based on the amount of data that needs to be indexed per day and charged on an annual basis. A company that needs to index up to 500 megabytes per day would pay $2,500 a year.

Patrick McGovern, one of the founders of Splunk, says his company is looking at the possibility of opening parts of the source code within the year. “At this point we are doing a bit of a hybrid,” McGovern says. “We’re providing the software at zero cost, and we are providing all the API to allow developers to extend and customize our search engine to suit their needs. We’re also keeping parts of the code closed, so that large data centers have a reason to purchase the professional version of our software.”

CHW’s primary business is providing acute care, defined as treatment for a short-term or episodic illness or health problem. Erik Leader, chief technical architect for CHW, says his company used to focus solely on a variety of proprietary applications, a mix which included “all the major players in the field. We have just about one of everything.” As CHW grew through the acquisition of other hospitals, the IT mix became a nearly unmanageable hodgepodge of disparate systems, many times further separated by the lack of interoperability provided by proprietary standards. “Over the last five years we’ve focused on standardization and consolidation,” Leader says. “We decided to move to more open standards so we weren’t dependent on one particular vendor or technology.”

As Leader began exploring the open source world, he discovered the open source community and liked what he saw. “[The] community was embracing open standards and technologies, especially in the healthcare area, so we started pursuing that. What we found was that a lot of our vendors were also looking at open source and moving toward open standards.”

Leader says CHW’s open source strategy focuses on the long term. “How can we leverage open standards and open source applications to improve cost and reliability and access to patient information for our caregivers? We found that most proprietary solutions in identity management security were rather confining,” he says. “We have lots of regulatory compliance we need to keep at the state and federal level. It would be very difficult to continue to meet those needs with the systems we had.” With regulations such as HIPAA, health care providers must show that patient care records are maintained with the strictest privacy.

Because of the importance of high level of trust needed in identity management, one of the first open source implementations at CHW was the Open Lightweight Directory Access Protocol (OpenLDAP) used by Novell in its eDirectory application. LDAP uses a directory tree to store information in a central repository. IT departments like to use LDAP as a means of identifying and authorizing users on a network. “We looked at Microsoft -– they support some of the [LDAP] standards; and Oracle’s identity management; and Sun’s Netscape ID management.” Ultimately, Leader says CHW went with Novell because of what he calls its overall commitment to open source. “[Open source] is a better, less expensive way of providing IT support to our caregivers,” he says. “Part of it is the fact that the code is open and available to everybody, so there’s a constant inspection of it going on by people who wish to make it stronger. We’ve found that the correction of security errors and holes is faster in the open source community than in proprietary solutions. We see many more security issues with our proprietary solutions.”

One of the challenges Leader faces in his drive toward open source adoption is resistance from his own industry. “Not all of our software vendors are yet embracing open source,” he says. “In general, IT in the healthcare industry tends to lag. They are slow to adopt and risk-averse.” That resistance has pushed CHW out to the bleeding edge of open source in healthcare, past the larger and more established software vendors to the newer, smaller companies. “Once we got into that we found there were a lot of companies that had begun supporting open standards,” Leader says. “In many cases we partnered with them to help them identify direction and to jointly manage the risk of moving in that direction.”

The changes in the infrastructure don’t disrupt the end users. “We try to make all of our changes as transparent to the caregivers as possible,” Leader says. “A lot of what we do they don’t really know that its being done with open source -– nor do they really care.” CHW has begun pilots and assessments of Linux desktops, but “we don’t have a whole lot of it out there,” he says.

CHW is progressing in its move toward complete open source penetration by upgrading existing solutions as it becomes necessary. “We have roughly 200 application servers running on [SUSE] platforms,” Leader says. “We’re rapidly moving toward 100% open source.”