Role: GCP Data Engineer (Mid Level)
Position Overview:
We are seeking a mid-level GCP Data Engineer with 4+ years of experience in ETL, Data Warehousing, and Data Engineering. The ideal candidate will have hands-on experience with GCP tools, solid data analysis skills, and a strong understanding of Data Warehousing principles.
Qualifications:
· 4+ years of experience in ETL & Data Warehousing
· Should have excellent leadership & communication skills
· Should have experience in developing Data Engineering solutions Airflow, GCP BigQuery, Cloud Storage, Dataflow, Cloud Functions, Pub/Sub, Cloud Run, etc.
· Should have built solution automations in any of the above ETL tools
· Should have executed at least 2 GCP Cloud Data Warehousing projects
· Should have worked at least 2 projects using Agile/SAFe methodology
· Should Have mid level experience in Pyspark and Teradata
· Should Have mid level experience in
· Should have working experience on any DevOps tools like GitHub, Jenkins, Cloud Native, etc & on semi-structured data formats like JSON, Parquet and/or XML files & written complex SQL queries for data analysis and extraction
· Should have in depth understanding on Data Warehousing, Data Analysis, Data Profiling, Data Quality & Data Mapping
Responsibilities:
· Analyze the different source systems, profile data, understand, document & fix Data Quality issues
· Gather requirements and business process knowledge in order to transform the data in a way that is geared towards the needs of end users
· Write complex SQLs to extract & format source data for ETL/data pipeline
· Create design documents, Source to Target Mapping documents and any supporting documents needed for deployment/migration
· Design, Develop and Test ETL/Data pipelines
· Design & build metadata-based frameworks needs for data pipelines
· Write Unit Test cases, execute Unit Testing and document Unit Test results
· Deploy ETL/Data pipelines
· Use DevOps tools to version, push/pull code and deploy across environments
· Support team during troubleshooting & debugging defects & bug fixes, business requests, environment migrations & other adhoc requests
· Do production support, enhancements and bug fixes
· Work with business and technology stakeholders to communicate EDW incidents/problems and manage their expectations
· Leverage ITIL concepts to circumvent incidents, manage problems and document knowledge
· Perform data cleaning, transformation, and validation to ensure accuracy and consistency across various data sources
· Stay current on industry best practices and emerging technologies in data analysis and cloud computing, particularly within the GCP ecosystem
Skills:- GCP, Data engineering, ETL and Datawarehousing