ADB + Datastage

Job description

Expereince : 4-6yrs Work Mode : hybrid Work location : Hyderabad/Bangalore/Chennai/Pune Primary skil: Datastage +Databricks As a Data Engineer, you will be responsible for: • Design & Hands-on engineering of Big Data and cloud environments that support our clients advanced analytics and data science initiatives. • Provide support in defining the scope and sizing of work • Working closely with various enterprise architects Information security teams, Data management team, to ensure the architected solution meets all the needs of a customer, from a functionality perspective and IT solution engineering perspective. • Translate business requirements into technology solutions • Mentor and guide Jr. team members to deliver the solutions on-time Technical Skills: • Overall 6-9 years of experience of experience with Data Management, Big Data, Data Warehousing and Analytics. • At least 1 years of experience in designing and migrating Oracle, DataStage based Enterprise Wherehouse to Azure using Databricks. • 1-2 years of experience in designing and developing ETL solutions using Data Stage. • Understanding and reverse engineering of DataStage parallel ,server and sequence ETL jobs to Databricks SparkSQL/PySpark/Spark Declarative pipelines •Analysis and converting custom stages, local containers, shared containers, routines and functions to Databricks SparkSQL/PySpark code. • Analysis of the DataStage Configuration & Batch-job Shell scripts. • Deep expertise in reverse engineering of Data Stage code and producing complexity reports. • 3+ years of hands-on experience in creating an end-to-end lineage of all inbound and outbound file/data delivery including regulatory or operational reporting • 3+ years of hands-on experience with ADF, Pyspark • 1-2 years of experience with Tidal. • One to two years of experience in Collibra DQ and DG is plus. • Hands on working experience with scripting languages such as Python, Pyspark, Spark SQL or Shell Scripting, etc