Description:

BlackRock has open sourced one of its internally used library that is used for extracting data from different sources, doing translation, validation and writing to the destination in format desired as described in a configuration file. This library aims at separating the code to do repetitive tasks in generating interface files from the business logic of data extraction, translation and formatting. Extract Transform and Transfer is a design pattern term coined to differ it from ETL pattern where the data is then loaded in a new data warehouse. This is just in memory translation before generating the output in desired format.

BlackRock uses a lot of open source libraries from Python community and we would like to give back this library to our community that we have built using open source frameworks like Pandas and Great Expectations. During this talk, we would share more about what we have built, how other developers can use it and contribute to it. We will share some examples of writing the configuration for ETT use cases so that developers/business users can directly use it for their ETT use cases.

Prerequisites:

No Prerequisites

Content URLs:

Our medium Blog on this: https://engineering.blackrock.com/introducing-ingen-cb6a5083788c

Speaker Info:

Swarna Dhakad, is a Senior Engineering Team Director at BlackRock's Aladdin Wealth Tech. She has about 20 years of work experience across leading investment banking organisations like Morgan Stanley, Nomura prior to BlackRock, building full stack scalable applications in React, Java, Python technologies. She is very passionate about building reusable components and InGen is a brain child of hers.

Linked in Bio: https://www.linkedin.com/in/swarna-dhakad/

Piyush Ranjan, Engineer III at BlackRock's Aladdin Wealth Tech, is an expert engineer in backend technolgies like Java, Python, Go etc. Piyush has been a major part of the entire InGen built out and is very passionate about thoughtful technical solutions.

Linked in Bio: https://www.linkedin.com/in/shpiyu/

Speaker Links:

https://engineering.blackrock.com/introducing-ingen-cb6a5083788c https://github.com/blackrock/ingen https://www.linkedin.com/in/swarna-dhakad/ https://www.linkedin.com/in/shpiyu/