Data Engineer

Job description

**Data Engineer** **Job Description:** **Position: AWS- Data Engineer** **Experience: 1-4 Years** **Location: Sector 43, Gurgaon, Haryana** **Job Insight:** **We are seeking a skilled and experienced Data Engineer to join our dynamic team. You will be** **responsible for Data Ingestion and Migration, ETL Job Scripting(Python, Pyspark, SQL, DBT),** **Data Modeling and structuring of our data lake infrastructure. You will collaborate with data** **architects, data scientists, and other stakeholders to ensure efficient data storage, retrieval, and** **processing capabilities. The ideal candidate will have 2 to 4 years of experience in data** **engineering with a focus on data lake technologies.** **Responsibilities:** **1. Design and implement salable data lake and data replication/migration architectures using** **technologies such as Qlik Replicate, AWS DMS, fivetran, Glue, EMR,DBT, Airflow, AWS S3,** **AWS Redmine, Snowflake.** **2. Develop and maintain data ingestion pipelines to efficiently collect and store structured and** **unstructured data from various sources.** **3. Optimize data lake performance and reliability by implementing best practices in data** **partitioning, indexing, and compression.** **4. Collaborate with data scientists and analysts to understand data requirements and implement** **data transformations and aggregations as needed.** **5. Ensure data lake security and compliance with regulatory requirements by implementing** **access controls, encryption, and auditing mechanisms.** **6. Monitor data lake performance and troubleshoot issues to ensure high availability and** **reliability.** **7. Document data lake architecture, processes, and procedures for knowledge sharing and** **future reference.** **8. Stay updated on emerging trends and technologies in data engineering and contribute to** **continuous improvement initiatives.** **Required Skills:** **1. Bachelor's degree in Computer Science, Engineering, or a related field.** **2. Minimum 1 year of experience in data engineering with a focus on data lake technologies.** **3. Proficiency in programming languages such as Python, Pyspark, SQL, DBT.** **4. Hands-on experience with data lake technologies and cloud data storage solutions like AWS** **S3, Glue, Athena.** **5. Experience with data ETL tools and frameworks such as Apache NiFi, Apache Kafka, AWS** **Glue, AWS DMS, Qlik Replication, AWS Athena, AWS Redshift.** **6. Familiarity with data governance, security, and compliance requirements.** **7. Excellent problem-solving skills and ability to work effectively in a fast-paced environment.** **8. Strong communication and collaboration skills to work effectively with cross-functional teams.** **Good to have skills:** **9. Understanding of data modeling concepts and experience with schema design for structured** **and semi-structured data.** **10. Experience with Agile process methodology** **11. Exposure to Quicksight, Apache Superset will be a plus**