Data Engineering

Weather and Financial Data Pipeline

Developed a data pipeline and backend for a large-scale financial analysis application. The app integrated weather data, energy prices, and site-specific data to calculate expected solar site revenues, used by the finance team to make investment decisions. Integrated financial data (Bloomberg API) with weather data (SolarGIS API) and site-specific data (Google Maps API) and processed using Apache Spark. Time series data on solar production from existing solar sites was also integrated into the tool, data was streamed into a data warehouse using Apache Kafka.

Technologies Used

Python

PostgresSQL

Apache Spark

Apache Kafka

Ecommerce Data Warehouse + CRM

Created a Snowflake data warehouse for an eBay seller. The client wanted both a CRM and a data warehouse to track their customers and their orders. They wanted to see the revenue generated by each customer and which were their best selling items. I used the eBay API to load data into the data warehouse and then used the data warehouse as a source for the CRM. The data warehouse served a dashboard which tracked the key metrics that the client wanted to see. I also built an ETL pipeline to allow the client to load Excel files into the data warehouse. Used dbt to help transform the data into the models used in the dashboard. Built the dashboard using React.

Technologies Used

Python

PostgresSQL

dbt

Docker

React