To power financial prosperity around the world, Intuit needs to create personalized product experience and new data centric products. Some of these use cases include Enabling 360 Customer View for Personalization and Targeting, building Ecosystem for Data Exchange between internal and 3rd party and personalize financial offerings, creating platform for Personalized security experience based on risk factors of people and devices.

Unlike workflow centric products (for example, tax processing, accounting transactions), these use cases are often information-intensive and require real-time access to a large amount of connected data associated with people, organizations and things they own.

To achieve this, we have created a platform called Unified Profile Service utilizing Flink. This platform is intended to provide the strategic data asset of a trusted, real-time, unified and connected view of people, organization and things they own. We have abstracted re-usable components such as sources, sinks, transformations etc and created a template. Utilizing this template our Product teams are able to rapidly test domain specific transformations and computations by creating and deploying Flink Jobs. This platform is running in production on AWS EMR, powering multiple use cases, ingesting and processing billions of events per day.

In this talk, we will be discussing the design details of this Platform built leveraging Flink and Flink APIs as well as challenges faced along the way. We will begin by talking about the various components of the pipeline such as Identity Stitching, Entity Resolution, Reconciliation and Data Persistence. We will then dig in to the technical details of how we abstracted away these common components and created a template. We will also talk about how we update Consumer’s Financial Identity Graph in real-time through custom built AWS Dynamodb and Neptune Sink using Flink’s Connector API.

Finally we will touch on lessons learnt along the way as we deployed the platform in production and offer advice on things to avoid as well as how to take things to the next level.

Conference Program