A data-driven organization is looking for a Data Engineer to join their team. The Data Engineer will be responsible for designing, developing, and maintaining robust data pipelines and architectures that support the organization’s data-driven initiatives. The role involves contributing to all phases of the data engineering lifecycle, ensuring scalability, reliability, and performance. The Data Engineer demonstrates technical leadership by solving complex problems, mentoring junior engineers, and actively improving data engineering practices.
Main Responsibilities:
- Design and implement scalable and efficient data pipelines.
- Collaborate with stakeholders to gather requirements and identify data sources.
- Develop and optimize ETL processes.
- Create and maintain data models.
- Build and manage cloud-based data architectures.
- Implement monitoring and alerting systems.
- Contribute to data governance initiatives.
- Utilize advanced data processing frameworks like Apache Spark, Apache Kafka, and Flink.
- Maintain and enhance CI/CD pipelines.
- Perform code reviews.
- Document processes, architectures, and technical workflows.
- Mentor junior engineers.
- Identify opportunities for process optimization and automation.
- Collaborate with cross-functional teams.
Required Knowledge and Experience:
- Minimum 3 - 4 years of experience as a Data Engineer.
- In-depth understanding of core data engineering concepts and principles.
- Advanced proficiency in Python.
- Extensive experience with Python libraries and frameworks.
- Strong knowledge of both SQL and NoSQL databases.
- Advanced understanding of data modeling concepts and techniques.
- Proficient in various data processing methods.
- Extensive experience with data processing frameworks like Apache Spark, Apache Kafka, and Apache Flink.
- Advanced knowledge of file processing concepts.
- Strong understanding of data governance principles.
- Comprehensive understanding of the end-to-end data engineering lifecycle.
- Experience with CI/CD pipelines and automation for data engineering workflows.
- Advanced understanding of various data architectures.
- Proficient with version control systems (e.g., Git).
- Advanced understanding of data visualization tools (e.g., Tableau, Power BI).
- Strong understanding of cloud computing in AWS and GCP stacks.
- Basic experience with Infrastructure as Code (IaC) tools like Terraform or CloudFormation.
- Proven experience leading projects with Objectives and Key Results (OKRs).