AB InBev – Grupo Modelo
I joined as a Data Engineer on an Azure ETL team and was promoted to Team Lead six months later. Besides leading the team, I built a Ruby on Rails app that automated the preparation of each ETL and was adopted by every data engineering pod.
WHAT I BUILT
01ETL pipelines with TalendExtraction from SAP and Vertica into Azure.
The ETL process was orchestrated in Talend Open Studio: it extracted data from SAP, Vertica and other sources, transformed it with SQL files and loaded it into Azure, partly into Azure Databricks. Each flow was defined in a master Excel file with the tables, fields, SQL files and load strategy. Some runs took 2 to 3 hours.
02Load strategiesFull and incremental loads.
Each ETL used one of two strategies: a full load, which brings all the data on every run, or an incremental load, which brings only new records (for example, the previous day’s), checks them against the destination and drops duplicates so source and destination match.
03ETL prep automationFrom 2–3 hours to 5 minutes per ETL.
Before each pipeline run, someone had to fill in the master Excel file by hand and write the extraction and transformation SQL, even though little changed from one ETL to the next; data-entry mistakes caused failures and delays. It started as a Ruby script and, once it proved itself, I turned it into a full Ruby on Rails app with a UI: it connects to the database, takes the request parameters (source, destination, table and load strategy) and generates every file ready for Talend. Prep went from 2–3 hours to about 5 minutes with no data-entry errors, and I shared it with every pod.
04Talend pipeline maintenanceFixes and optimizations in Java.
The base pipeline was built by a global team and showed failures as it was used. As practically the only person on the team with Java experience, I went into its code, fixed and optimized many parts, and built and repaired pipelines in Talend. The failures that remained were usually external: timeouts or service and database outages.
05Team leadershipUp to six people, with Scrum.
As Team Lead I coordinated a team of up to six people, developers and QA: sprint planning, code review and performance follow-ups with Scrum and Azure DevOps.