Data Engineer

Job description

**Job description** **Role: Data Engineer** **Exp: 5-8 Years** **Responsibilities** - Build document ingestion pipelines from SharePoint, DMS and file stores into vector databases, including chunking and pre-processing - Build ETL/ELT pipelines from structured enterprise sources (ERP, CRM, databases) into the AI data layer - Design and manage a feature store — curated, versioned features for ML models - Implement data quality validation, lineage tracking and alerting - Build data access interfaces used by AI Engineers and Backend Engineers - Orchestrate pipelines using Airflow or Prefect — scheduling, retry logic, SLA monitoring **Required Skills** - Python — for transformation logic, custom operators and data processing - SQL — advanced. Window functions, CTEs, query optimisation. Can read and interpret query plans. - Workflow orchestration — Airflow or Prefect in a production environment. Not just writing DAGs — operating them, handling failures, managing backfills. - Cloud data storage — S3, ADLS or GCS. At least one cloud warehouse (Snowflake, BigQuery or Azure Synapse). - Data modelling — understands medallion architecture (bronze/silver/gold) and can design a schema for a given use case. **Additional Skills** - Vector store ingestion — has built a pipeline that loads embeddings into a vector database. Understands chunking strategy and its effect on retrieval quality. - Feature store — has contributed to or built a feature store. Understands offline vs online feature serving and training-serving skew. - dbt — models, tests, documentation - Data contracts — has defined or enforced a data contract between producer and consumer - Real-time pipelines — Kafka, Azure Event Hub or Kinesis