AWS Lead Data Engineer – R01564133

Job description

AWS Lead Data Engineer - R01564133 Lead Data Engineer Primary Skills • Athena, Step Functions, Spark - Pyspark, ETL Fundamentals, SQL (Basic + Advanced), Glue, Python, Lambda, Data Warehousing, EBS /EFS, AWS EC2, Lake Formation, Aurora, S3, Modern Data Platform Fundamentals, PLSQL, Data Modelling Fundamentals, Cloud front Specialization • AWS Data EngineerIng Basic: Senior Data Engineer Job requirements Job Description – AWS Senior Data Engineer Company: Brillio Technologies Role: Senior Data Engineer Experience: 5–10 Years Location: India (Hybrid/Remote) Employment Type: Full-time Role Overview We are looking for a Senior Data Engineer with strong Databricks and Big Data expertise to design, build, and optimize scalable data pipelines and platforms. The ideal candidate will have hands-on experience with Spark, PySpark, AWS ecosystem, and modern data architectures, along with exposure to APIs and real-time data processing. Key Responsibilities • Design and build scalable data pipelines using Spark / PySpark on Databricks • Develop and optimize ETL/ELT workflows for large-scale data processing • Work with AWS EMR, S3, and Hadoop ecosystem for distributed data processing • Build and maintain data lake and data warehouse solutions • Develop and integrate APIs for data ingestion and consumption • Implement data processing workflows using Airflow / Autosys • Optimize data performance using partitioning, caching, and query tuning • Handle structured and semi-structured data from multiple sources • Ensure data quality, governance, and reliability of pipelines • Collaborate with cross-functional teams (Analytics, Product, Engineering) Technical Skills (Must-Have) • Strong expertise in: • - Apache Spark / PySpark • Databricks (mandatory) • SQL (PostgreSQL or similar) • Hands-on experience with: • - AWS EMR, S3 • Hadoop, Hive ecosystem • Programming skills: • - Python (must-have) • Scala (good exposure) • Strong knowledge of: • - UNIX / Shell scripting • ETL pipelines and data engineering fundamentals Additional Skills • Experience with: • - Elasticsearch (data storage & retrieval) • Workflow orchestration tools (Airflow, Autosys) • Exposure to: • - API development & integration • Version control tools (Git / SVN) • Basic knowledge of HTML (for web/data tasks) Experience Required • Minimum 5+ years of Data Engineering experience • Strong hands-on experience in Databricks + Big Data stack • Proven experience working on large-scale distributed systems Education • Bachelor’s or Master’s degree in Computer Science, Engineering, or related field Why Join Brillio • Work on cloud-native, large-scale data platforms • Exposure to cutting-edge Databricks & AWS ecosystems • Collaborative and high-performance engineering culture