Staff Software Engineer - Spark and OLAP Internals

Job Description

Adyen is seeking a Staff Software Engineer to enhance its distributed systems, focusing on Spark and OLAP internals. The ideal candidate will improve the operations and scalability of existing offerings and introduce new ones. This role involves scaling and tuning Hadoop and Spark infrastructure, as well as iterating on the OLAP platform using Apache Druid.Adyen is a financial technology platform that provides payments, data, and financial products in a single solution for global customers.What this role involves:

• Designing, building, and optimizing data clusters for scalability and fault tolerance.
• Improving the Spark, Hadoop, Kubernetes, Deltalake, and Druid ecosystems.
• Evolving open-source projects internally and contributing code upstream.
• Educating and growing Adyen’s internal knowledge on relevant topics.

Requirements:

• Experience scaling and tuning large deployments of Spark-on-k8s and Spark-on-Hadoop.
• Knowledge of Hadoop and the HDFS protocol.
• Experience designing and tuning shuffle-heavy systems.
• Familiarity with lakehouse file formats (Delta, Iceberg, Hudi).
• Experience with OLAP technologies (Clickhouse, Apache Druid, Apache Pinot, Apache Doris).
• Open-source contributions in relevant technologies.
• Strong communication skills and ability to work with diverse stakeholders.
• Ability to troubleshoot and resolve issues in large-scale production environments.

What Adyen offers:

• Opportunity to tackle unique technical challenges at scale.
• A collaborative team environment.
• The chance to deliver innovative and ethical solutions.