FIBI is a sophisticated user of various database and big data
technologies, which underpin its day-to-day activities as a
financial services institution. In 2020, these were primarily
MongoDB databases and numerous Hadoop clusters.
However, the Cloudera acquisition of Hortonworks in 2018 presented
an issue for FIBI. Cloudera was not only altering licensing
mechanisms for their newly acquired Hortonworks Data Platform and
Hadoop customers, but they were also changing the deployment methods
for the technology. Additionally, Cloudera was seeking to pull the
disparate customer base onto a new, singular platform.
“Amongst other things, the new management UI that Cloudera
proposed for Hortonworks customers just wasn’t going to work for
us… this presented us with a unique chance to try something new”
states Denis, a lead DBA at FIBI who is responsible for big data
FIBI selected Kafka as the platform for their upcoming big data
projects, but it wasn’t set to be a straightforward project. Hadoop
had been in use at FIBI for many years, and the ETL processes that
had been established internally were not able to be re-used onto the
new Kafka-based platform.
FIBI’s technology and infrastructure teams were also lacking an
understanding of Kafka, which would make delivering a complex
project like this impossible. There were also numerous security
integrations to be considered, given that FIBI operates in a heavily