AWS Lead Data Engineer – R01564133

Job description

# **AWS Lead Data Engineer - R01564133** **Lead Data Engineer** ###### **Primary Skills** - Athena, Step Functions, Spark - Pyspark, ETL Fundamentals, SQL (Basic + Advanced), Glue, Python, Lambda, Data Warehousing, EBS /EFS, AWS EC2, Lake Formation, Aurora, S3, Modern Data Platform Fundamentals, PLSQL, Data Modelling Fundamentals, Cloud front ###### **Specialization** - AWS Data EngineerIng Basic: Senior Data Engineer ###### **Job requirements** **Job Description – AWS Senior Data Engineer** **Company:** Brillio Technologies **Role:** Senior Data Engineer **Experience:** 5–10 Years **Location:** India (Hybrid/Remote) **Employment Type:** Full-time **Role Overview** We are looking for a **Senior Data Engineer with strong Databricks and Big Data expertise** to design, build, and optimize scalable data pipelines and platforms. The ideal candidate will have hands-on experience with **Spark, PySpark, AWS ecosystem, and modern data architectures**, along with exposure to APIs and real-time data processing. **Key Responsibilities** - Design and build **scalable data pipelines** using Spark / PySpark on Databricks - Develop and optimize **ETL/ELT workflows** for large-scale data processing - Work with **AWS EMR, S3, and Hadoop ecosystem** for distributed data processing - Build and maintain **data lake and data warehouse solutions** - Develop and integrate **APIs for data ingestion and consumption** - Implement data processing workflows using **Airflow / Autosys** - Optimize data performance using **partitioning, caching, and query tuning** - Handle structured and semi-structured data from multiple sources - Ensure **data quality, governance, and reliability** of pipelines - Collaborate with cross-functional teams (Analytics, Product, Engineering) **Technical Skills (Must-Have)** - Strong expertise in: - - **Apache Spark / PySpark** - **Databricks (mandatory)** - **SQL (PostgreSQL or similar)** - Hands-on experience with: - - **AWS EMR, S3** - **Hadoop, Hive ecosystem** - Programming skills: - - **Python (must-have)** - **Scala (good exposure)** - Strong knowledge of: - - **UNIX / Shell scripting** - **ETL pipelines and data engineering fundamentals** **Additional Skills** - Experience with: - - **Elasticsearch (data storage & retrieval)** - **Workflow orchestration tools (Airflow, Autosys)** - Exposure to: - - **API development & integration** - Version control tools (**Git / SVN**) - Basic knowledge of **HTML** (for web/data tasks) **Experience Required** - Minimum **5+ years of Data Engineering experience** - Strong hands-on experience in **Databricks + Big Data stack** - Proven experience working on **large-scale distributed systems** **Education** - Bachelor’s or Master’s degree in **Computer Science, Engineering, or related field** **Why Join Brillio** - Work on **cloud-native, large-scale data platforms** - Exposure to **cutting-edge Databricks & AWS ecosystems** - Collaborative and high-performance engineering culture