Weekend Drive-Virtual-Saturday

Job description

**Strong expertise in SQL, PySpark, and Python to build scalable batch and distributed data pipelines.** - **Ability to write and tune complex SQL for analytics on distributed query engines.** - **Experience designing high-performance data processing using Spark SQL and DataFrames.** - **Proficiency in developing Python-based automation scripts and reusable data engineering components.** - **Hands-on experience with distributed SQL engines such as Starburst or similar technologies.** - **Practical knowledge of lakehouse architecture using Delta Lake, Iceberg, or equivalent platforms.** - **Experience working with cloud object storage such as S3, ADLS, or similar environments.** - **Understanding of ETL/ELT processes and managed data services for ingestion and transformation.** - **Solid knowledge of relational and non-relational databases as source systems.** - **Ability to implement data quality checks, including validation, monitoring, and reconciliation.** - **Experience in analytical data modeling, including dimensional and star schema design.** - **Familiarity with Airflow, Autosys, or similar orchestration tools.** - **Exposure to CI/CD, DevOps practices, and Git for collaborative pipeline development.** - **Strong skills in debugging data issues, validating transformations, and optimizing performance.** - **Proven problem-solving ability for complex data engineering challenges.** - **Ability to leverage AI coding tools like GitHub Copilot, Devin, or Claude to improve productivity and code quality.**