As they moved to modernize their data platform, Pipl followed Google's recommendation and chose us to re-architect and re-deploy their many Elasticsearch clusters on GCP. We worked closely with the team at Pipl to ensure both performance and cost of those clusters is at an optimum at all times.

Pipl is a market-leading digital identity verification and investigation data service, trusted by enterprises and individuals worldwide.

BigData Boutique partnered with Pipl on a complex project involving modernizing their infrastructure, seamlessly migrating their Elasticsearch clusters onto Google Cloud Platform (GCP), and provided data grooming and re-architecture.

BigData Boutique was recommended by Google, as a leading light in the fields of GCP and Elasticsearch. Through BigData Boutique’s work, Pipl now runs a cutting edge tech stack and has the ability to undertake more challenging projects than ever before.

Legacy Cloud Services Provider

Pipl’s existing cloud services provider (CSP) left a lot to be desired. It did not support Pipl’s pipeline of ambitious engineering projects. Last year, Pipl made a strategic decision to move to a leading CSP. As part of that process, they selected Google Cloud Platform (GCP) as their new provider.

Unoptimized Elasticsearch Clusters

“We had a very big, and not properly managed Elasticsearch cluster”, stated Uri Harel, Pipl’s Head of Infrastructure. Not only was this unwieldy and difficult to update, but it was hampering the development team’s ambitions to use Elasticsearch for more advanced and intelligent projects.

These issues with Elasticsearch configuration and architecture are what led Pipl to search for outside help from someone with serious expertise in the field.

No down time during the migration

As with many other tech organizations, Pipl’s customer experience is key. For Pipl, the migration of internal and production applications to Google Cloud Platform had to be carried out without any disruption or downtime. This was a challenging request - there were unruly Elasticsearch clusters and vast amounts of improperly indexed data to be considered.

BigData Boutique were able to bring expertise and a bespoke plan to support Pipl in achieving their goals. BigData Boutique’s reputation and track record for success in the areas of Elasticsearch, Google Cloud Platform, and zero-downtime cloud migrations made them an easy choice for Pipl.

Uri, Head of Pipl’s Infrastructure was happy to note, “BigData Boutique took overall responsibility and ownership for the whole project, from start to finish”. Not only did this give the team at Pipl confidence in the delivery, but ensured that there was no disruption to business during discovery, migration, and implementation.

The Seamless Move to Google Cloud Platform

BigData Boutique set up new Elasticsearch clusters on Google Cloud Platform - “with a clever piece of engineering, they set up some kind of mechanism that kept them synchronized with the old production clusters that were still running”.

The actual migration took around 3 months, owing to the amount of data to be moved and the complexity of the project. BigData Boutique ran the old production cluster fully synchronized with the new Google Cloud Platform cluster. All this was done whilst maintaining a logical separation between the old and new instances.

“That was a big goal for our migration - to have zero downtime for the move - which BigData Boutique [successfully achieved]”.

Knowledge Sharing

The BigData Boutique team were passionate about sharing their knowledge with Pipl’s engineers. This extended beyond the scope of the project, with BigData Boutique keen to build a lasting relationship that involved imparting knowledge throughout their engagement.

“We had a real lack of knowledge on Elastic, and that’s why we looked to BigData Boutique”. This enabled Pipl’s team to use new knowledge to drive forward some of their more challenging projects.

A Bespoke Elasticsearch Cluster

Pipl’s platform relies on being able to query vast amounts of structured and unstructured data at high speeds, so the new Elasticsearch cluster had to be optimized in a way that would require minimal tuning moving forward.

“BigData Boutique worked really closely with the engineers to make sure the right planning had gone into setting up the new Elasticsearch cluster on Google Cloud Platform”, said Pipl’s Head of Infrastructure.

“A lot of work was done by BigData Boutique on the structure of the cluster, with efficient coding to optimize the performance”.

Access to a whole new Cloud Platform

“The migration was really effectively managed into two waves: the offline and internal systems, and the customer-facing systems”, which was critical in guaranteeing no disruption to Pipl’s customers, says Uri.

“Towards the end of the migration project, Pipl had two identical clusters running, one on the old infrastructure and one on Google Cloud Platform”.

Eventually, all the data sources were managed from GCP. This exclusive access to GCP has enabled Pipl to take advantage of a vast array of tooling. This greater functionality, flexibility and resilience is propelling Pipl forward to exciting new challenges.

Prepared for the future

Previously, Pipl lacked expertise in Elasticsearch. That’s now a thing of the past: “We can now architect our data so we don’t have to just add another node every couple of weeks”. This capability has enabled Pipl to take control of their Elasticsearch cluster and use their new expertise when implementing their strategic goals.

Ongoing Partnership

Pipl were so impressed with the level of professionalism, expertise, and support shown throughout the engagement that BigData Boutique remained part of the team. On top of the huge successes of the initial scoped project, Pipl continues to engage BigData Boutique in several areas.

“BigData Boutique continues to work on the Google Cloud clusters, both on day to day maintenance, and also on some special projects”. Pipl’s engineering team are also leveraging BigData Boutique’s considerable knowledge for retention and backup projects, setting up and improving monitoring and alerting, as well as other data platform development projects. “We have someone to go to for questions around Elasticsearch now”, noted one engineer.

Fundamentally, BigData Boutique provided Pipl with a cutting edge stack and the knowledge to continue to excel in their field. Whilst big data search technology is still a challenging field, the team at Pipl are happy that they have a reliable partner to lean on.

“We are constantly exploring the potential for further use of Elasticsearch, and they are a very valuable resource for that”.