Senior Python Engineer, DataHub Ingestion Framework

Job Description

Acryl Data is seeking a Senior Python Engineer to lead the development of DataHub's ingestion framework. This role is crucial for connecting diverse data systems and powering metadata collection capabilities. The engineer will build scalable, fault-tolerant ingestion systems, clean APIs for the connector ecosystem, and event-driven architectures for real-time metadata processing. They will also work on schema mapping between diverse systems and DataHub's unified model, as well as versioning systems for AI assets.Role involves:

• Building scalable, fault-tolerant ingestion systems
• Creating clean, intuitive APIs for the connector ecosystem
• Developing event-driven architectures for real-time metadata processing
• Implementing schema mapping between diverse systems and DataHub's unified model
• Designing versioning systems for AI assets

Requirements:

• 4+ years of experience building production-grade distributed systems
• Advanced Python expertise with a focus on API design
• Experience with high-scale data processing or integration frameworks
• Strong systems knowledge and distributed architecture experience
• A track record of solving complex technical challenges

Role offers:

• Competitive salary
• Equity
• Medical, dental, and vision insurance
• Remote friendly work environment
• Work from home and monthly co-working space budget