The Project
FIBI is a sophisticated user of various database and big data technologies, which underpin its
day-to-day activities as a financial services institution. In 2020, these were primarily MongoDB
databases and numerous Hadoop clusters.
However, the Cloudera acquisition of Hortonworks in 2018 presented an issue for FIBI. Cloudera
was not only altering licensing mechanisms for their newly acquired Hortonworks Data Platform
and Hadoop customers, but they were also changing the deployment methods for the technology.
Additionally, Cloudera was seeking to pull the disparate customer base onto a new, singular
platform.
“Amongst other things, the new management UI that Cloudera proposed for Hortonworks customers
just wasn’t going to work for us… this presented us with a unique chance to try something new”
states Denis, a lead DBA at FIBI who is responsible for big data technologies.
FIBI selected Kafka as the platform for their upcoming big data projects, but it wasn’t set to
be a straightforward project. Hadoop had been in use at FIBI for many years, and the ETL
processes that had been established internally were not able to be re-used onto the new
Kafka-based platform.
FIBI’s technology and infrastructure teams were also lacking an understanding of Kafka, which
would make delivering a complex project like this impossible. There were also numerous security
integrations to be considered, given that FIBI operates in a heavily regulated industry.