by @Nattaphat and grammar, sentence revised by Gemini
In this week, I shifted from basic scripts to managed orchestration. I began by refactoring my Week 1 ETL setup, integrating Kestra into my docker-compose environment file and configured it with an API key to leverage the AI Copilot features.
Through this process, I explored three primary categories of workflows:
-
Local Database integration
I successfully implemented a Simple ETL script that processes and loads data into a local database, serving as the foundation for more complex orchestration.
-
Cloud-Native ETL & ELT
I configured flows to connect seamlessly with Google Cloud Platform (GCP). This required a deep dive into managing secrets and credentials. Interestingly, I strayed from the instructor’s method to find a more efficient custom configuration for my environment like in this image.
-
Scheduling & Backfilling
I explored Triggers, which allow for automated monthly ETL/ELT runs. A key takeaway was the Backfill concept—the ability to manually trigger a flow for specific historical date ranges to ensure data consistency.
Overcoming Technical Hurdles(again and always)
The journey wasn’t without its “bugs”. I encountered a technical issue within Kestra while attempting to set up the AI Chat feature using my own API key. While this prevented me from following that specific part of the tutorial, it was a great lesson in troubleshooting modern orchestration tools.
Key Takeaways
Though this week was significantly more detail-intensive, it solidified my understanding of several core concepts:
- 📈Workflow-as-Code: Managing complex pipelines through declarative configuration.
- 📈Orchestration: Using Kestra to bridge the gap between local and cloud environments.
- 📊ETL vs ELT: Observing the tangible difference in execution speed and architecture when shifting processing power to the cloud.
Link to week 2 journey:
Link to my main journey page: **https://lnkd.in/giwBqS8w**