Reinventing Fermilab's Scientific Computing Grid Accounting with the Elastic Stack
Elastic{on} 2018 talk on distributed computing monitoring and accounting at Fermilab
[2018-02-28]#development #monitoring
One of the great privileges I have from working at Fermilab is representing the lab and high energy physics within the monitoring community. In 2018 I was invited to speak at Elastic{on} to talk about Fermilab and how we use Elasticsearch to monitor distributed scientific computing.
My talk presents an overview of the history of Fermilab–having just celebrated our 50th anniversery– scientific and distributed computing, and then a focus on the GRACC accounting project, which reinvented the system used to track usage across the Open Science Grid.
This project took an aging Java application with a strict MySQL database and archaic web frontend, and coverted it into a scalable system based on microservices and Elasticsearch, with a modern open- source interactive frontend, that provides a much more performant system that requires fewer maintenance and development cycles.
