Data Engineer, DaaS

Job Description

PayPay is seeking a Data Engineer to join its Data Pipeline team. This role involves creating, deploying, and managing a resilient Data Engineering Platform using technologies like Databricks, Delta Lake, Spark, and Scala. The Data Engineer will contribute to scaling the platform across the organization, ensuring high performance, reliability, and efficiency.

PayPay is a FinTech company that has grown to over 69M users since its launch in 2018. The company values constant risk-taking and challenges to realize a vision that no one else can imagine.

Role involves:

• Creating and managing robust data ingestion pipelines using Databricks, Airflow, Kafka, and Terraform.
• Optimizing large-scale data pipelines for high performance and reliability.
• Developing data processing workflows using Databricks, Snowflake, Delta Lake, and Spark.
• Maintaining and improving the Data Lakehouse using Unity Catalog.
• Constructing automation, frameworks, and tools to streamline data engineering workflows.
• Collaborating across teams to facilitate smooth data flow and integration.
• Enforcing best practices in observability, data governance, security, and regulatory compliance.

Requirements:

• Minimum 5 years of experience as a Data Engineer or similar role.
• Hands-on experience with Databricks, Snowflake, Delta Lake, Spark, and Scala.
• Proven ability to design, build, and operate Data Lakes or Data Warehouses.
• Proficiency with Data Orchestration tools (Airflow, Dagster, Prefect).
• Familiarity with Change Data Capture tools (Canal, Debezium, Maxwell).
• Strong command of at least one primary language (Scala, Python, etc.) and SQL.
• Experience with data catalog and metadata management (Unity Catalog, Lakeformation).
• Experience in Infrastructure as Code (IaC) using Terraform.
• Excellent problem-solving and debugging abilities.
• Strong communication and collaboration skills.

Role offers:

• Full-time employment with flexible working style (Hybrid Workstyle including Remote and office).
• Super Flex Time (No Core Time).
• Social Insurance, 401K, Translation/Interpretation support, VISA sponsor + Relocation support.
• Paid leave and Personal leave.