Clash
of Jobs
Browse All Jobs
Job Description
The company is looking for a Lead Data Engineer to join the Data Governance team. The Lead Data Engineer will collaborate with multiple stakeholders to develop data objects and pipelines on the AWS Cloud Platform. They will work with other IT teams on data solution projects, delivering products that integrate various system data sources and harmonize processes and data models. The Lead Data Engineer will oversee the completion of data and business intelligence (BI) related projects and will collaborate with other teams within the organization.
Role involves:
- Building and maintaining orchestration workflows to automate data pipeline execution, scheduling, and monitoring.
- Collaborating with data analysts and business stakeholders to understand their data needs and translate them into technical solutions.
- Contributing to the development and improvement of data engineering processes and standards.
- Optimizing data pipelines for performance, reliability, and cost-effectiveness, leveraging AWS best practices and cloud-native technologies.
- Designing and implementing robust and scalable data architectures on the AWS platform, including data lakes, data warehouses, and data pipelines.
- Developing and maintaining ETL/ELT processes using AWS services such as AWS Glue, AWS Lambda and AWS Step Functions.
- Building and managing data pipelines for ingesting, transforming, and loading data from various sources, including databases, APIs, and cloud storage.
- Ensuring data quality, security, and compliance with industry best practices and regulatory requirements.
- Troubleshooting and resolving data-related issues, including performance bottlenecks and data integrity problems.
- Mentoring junior data engineers and provide guidance on best practices and technologies.
- Staying up-to-date with the latest advancements in AWS technologies and data engineering best practices.
Requirements:
- Minimum 7 years of experience building and deploying large-scale data processing pipelines in a production environment.
- Hands-on experience in designing and building data pipelines on AWS cloud infrastructure.
- Strong proficiency in AWS services such as Amazon S3, AWS Glue, AWS Lambda, Amazon Redshift, AWS Step Functions etc.
- Strong experience with Apache Spark for data processing and analytics.
- Hands-on experience on orchestrating and scheduling data pipelines using AppFlow, Event Bridge and Lambda.
- Solid understanding of data modeling, database design principles, and SQL and Spark SQL.
- Experience with version control systems (e.g., Git) and CI/CD pipelines.
- Excellent communication skills and the ability to collaborate effectively with cross-functional teams.
- Strong problem-solving skills and attention to details
Role offers:
- Healthcare Options
- Medical, Dental, Vision, Prescription
- Dependent and Health Care Flexible Spending Accounts
- 401(k) savings plan with company match
- Flexible based Paid Time Off
- Employee Assistance Program
- Legal Assistance Plan
- Tuition Reimbursement
- Employee Stock Purchase Plan
- Exclusive discounts on cell phones, gyms, and everyday purchases
