Luke Hospadaruk
Ithaka
Luke Hospadaruk is an engineer at Ithaka, he’s worked on data processing & ETL systems there for several years, and has designed and developed multiple iterations of data processing infrastructure. He loves to use clojure to put together effective and powerful systems.
Data-Shuffler: Bringing Order to The Galaxy (of Data Process)
Data-Shuffler is an ETL system implemented in Clojure on top of Apache Spark whose primary goals are testability, reliability, and collaboration. Clojure has allowed us to construct a robust ETL foundation that enables the entire organization to participate in a performant, well tested ETL system to get the datasets and insight they need, as well as contribute to organization-wide data and reporting initiatives. Using clojure.spec in particular has allowed us to focus on the model and the underlying features, while giving us the flexibility to compose and manipulate already-implemented functionality to meet unanticipated needs quickly as they arise.