Senior Digital Engineer

Job description

ROLE SUMMARY Data Engineers with hands-on expertise in PySpark and Fivetran to design, build, and maintain scalable data ingestion and transformation pipelines, at two levels: Junior (execution-focused, guided work) and Senior (ownership of architecture, performance, and mentoring). Both levels will work on integrating data from SaaS, database, and API sources into the cloud data warehouse, and building distributed processing jobs to support analytics and reporting. SENIOR DATA ENGINEER REQUIREMENTS - [Must-Have] 5+ years of hands-on experience with PySpark (DataFrames, Spark SQL, RDDs, performance tuning, job optimization) in production. - [Must-Have] 3+ years configuring and managing Fivetran connectors at scale (custom connectors, schema drift handling, sync orchestration, troubleshooting). - [Must-Have] Strong SQL and proven data modeling / warehouse design experience (Snowflake / Redshift / BigQuery). - [Must-Have] Deep experience with at least one major cloud platform (AWS, Azure, or GCP), including cost and performance optimization. - [Must-Have] Proven ability to design end-to-end pipeline architecture and lead technical decisions. - Experience mentoring junior engineers and reviewing code/pipeline designs. NICE-TO-HAVE SKILLS (EITHER LEVEL) - Experience with Databricks and Delta Lake. - Familiarity with orchestration tools (Airflow, Databricks Workflows). - dbt experience for transformation/modeling. - Exposure to streaming technologies (Kafka, Spark Structured Streaming). - Relevant certifications (Databricks Certified Data Engineer, Fivetran certification, cloud data engineer certs). - Prior experience in [specific industry, e.g., healthcare, finance, retail], if relevant. RESPONSIBILITIES - Senior: Design, build, and optimize PySpark ETL/ELT pipelines for large-scale batch and/or streaming data. - Own Fivetran connector strategy across [X] source systems, including custom connector development. - Define data architecture and standards; review junior engineers' pipeline designs. - Collaborate with analytics/BI teams and business stakeholders to define data requirements. - Implement data quality frameworks, validation, and monitoring across pipelines.