Big Data Engineer

Job description

Role: Senior Data Engineer Skills: SCALA, Spark, Azure Data lake, Azure Data factory, Kafka Exp: 7+ Years Location: Gr Noida/ Hyderabad/ Pune Job Description: • Data Engineers with engineering skills to build pipelines within the analytics platforms, moving ingested data from raw to modelled and ultimately to dimensional models that can be accessed by BI tools or ML models. • Primary duties include - Build data pipelines for complex sized Data and Analytics solutions (e.g. Data Lake, Data Warehouse, Business Intelligence Dashboards) in a team. • Translate functional and non-functional data and analytics requirements into fit for purpose technical design. • Ensure solution performance, business edge cases and security related issues are addressed while developing software. Debug issues of complexity, resolve blockers and follow design documents with minimal supervision. • This includes Production incidents. Complete data engineering coding tasks on problems of moderate to high scope and complexity. • Demonstrate good coding principles. Conduct code review for peers. Ensure solutions adhere to published data privacy and cybersecurity principles. • Work with senior and lead data engineers in the technical design process by contributing to the analysis of data application, data integration, large data storage or data pipeline requirements. • Update data inventories and registries as required to keep metadata and data lineage up to date, following agreed Data Governance standards, guidelines and principles. Carry out unit testing independently. • Troubleshoot issues, and fix defects that are of moderate to high complexity. Adhere to published coding standards, guidelines and best practices and contribute to Data Engineering Playbooks and other data technology blueprints. • Required technical skills: Microsoft Azure and Cloudera technology stacks, Hadoop, Spark, Scala, Hive, H-Base, Sqoop, Oozie, Apache Nifi, Airflow, HDFS, ADLS (Gen 2), Azure Data Factory (ADF), Data Bricks, Streaming (e.g. Spark Streaming), Kafka, Snowflake, SQL, GIT Bitbucket, Azure DevOps. • Team player: good attitude and willingness to learn and adapt. Familiar with Agile and Waterfall methodologies Please share your update CV with below details on radhika.singhal@coforge.com Total Exp: Current Location: Preferred Location: CTC: Expected CTC: