Senior Software Engineer - Data Engineer

Job description

**About this role:** Wells Fargo is seeking a **Senior Software Engineer Data Engineering** **In this role, you will:** - Lead moderately complex initiatives and deliverables within technical domain environments - Contribute to large scale planning of strategies - Design, code, test, debug, and document for projects and programs associated with technology domain, including upgrades and deployments - Review moderately complex technical challenges that require an in-depth evaluation of technologies and procedures - Resolve moderately complex issues and lead a team to meet existing client needs or potential new clients needs while leveraging solid understanding of the function, policies, procedures, or compliance requirements - Collaborate and consult with peers, colleagues, and mid-level managers to resolve technical challenges and achieve goals - Lead projects and act as an escalation point, provide guidance and direction to less experienced staff **Required Qualifications:** - 4+ years of Software Engineering experience, or equivalent demonstrated through one or a combination of the following: work experience, training, military experience, education **Desired Qualifications:** - Design, build, test, deploy, and maintain **large-scale structured and unstructured data pipelines** using Python, SQL, Apache Spark, and modern data lake/lakehouse technologies. - Develop and optimize **metadata-driven** pipelines, wrappers, ingestion frameworks, and validation layers to support GFCM data workflows. - Build and maintain **high-quality ELT/ETL pipelines** following best practices in reliability, performance, observability, and reusability. **Distributed Computing & Lakehouse Engineering** - Engineer and optimize Spark pipelines for large-scale batch and streaming workloads (partitioning, caching, Catalyst optimization, AQE, Tungsten). - Work with open table formats such as **Iceberg** , **Delta** , or **Hudi** for versioned data, time-travel, compaction, and schema evolution. - Implement Medallion (Bronze/Silver/Gold) architecture patterns for modern lakehouse systems. **Data Quality, Testing & Observability** - Implement automated data quality frameworks using tools such as Great Expectations / Deequ or custom validators. - Build data health monitoring frameworks with SLAs/SLOs, anomaly detection, and lineage capture. - Ensure strong validation layers during Data Center exits, migration programs, and DPC onboarding. **API & Microservice Development** - Build RESTful and metadata APIs using Python frameworks (FastAPI/Flask) to enable secure, governed data access. - Collaborate with application teams to integrate data access patterns and platform services. **Cloud & Platform Engineering** - Design and deploy data pipelines in cloud platforms (AWS, Azure, GCP) leveraging managed compute, orchestration, and storage. - Build CI/CD workflows and infrastructure automation using Jenkins, GitHub Actions, Azure DevOps, Terraform, or Helm. - Apply secure engineering principles including IAM, secrets management, encryption standards, and network controls. **Orchestration & Scheduling** - Build resilient orchestration flows using **Autosys** or equivalent tools. - Apply modular design with retries, alerts, SLAs, and workflow dependency management. **Collaboration & Delivery** - Work with cross-functional Agile teams (Product, Architecture, QA, Treasury SMEs). - Analyze technical requirements, evaluate design alternatives, and provide recommendations aligned with enterprise standards. - Independently deliver complex engineering tasks and contribute to architecture/roadmap discussions. **Job Expectations:** - 4+ years of hands-on experience with **Python** , **SQL** , and **bash** scripting for automation. - Strong experience building **big data pipelines** using Apache Spark, Hive, Hadoop. - Experience with **Autosys/Airflow** or similar orchestration tools. - Working knowledge of **REST APIs** , **Object Storage** , **Dremio** , and CI/CD pipelines. - Strong troubleshooting and problem-solving capabilities. - Solid foundation in **data modeling** (conceptual/logical/physical) and database design. - Experience architecting pipelines using distributed systems patterns (shuffle optimization, spill, broadcast, storage layouts). - Experience with streaming frameworks like **Spark Structured Streaming** or **Apache Flink** . **Data Lakehouse & Storage** - Hands-on with optimization techniques: clustering, Z-ordering, vectorized IO (Parquet/ORC), compaction. - Experience implementing Medallion architectures and governed ingestion zones. **Data Quality & Governance** - Knowledge of data governance platforms (Collibra, Alation, Purview). - Understanding of financial data controls, validation rules, reconciliation checks, and compliance (SOX/PCI). - Experience implementing lineage, observability, drift detection. **Platform & DevOps** - Cloud-native engineering experience serverless, managed Spark, event-driven architectures. - Familiarity with containerization (Docker, K8s) and workflow operators. - Strong experience implementing test automation for data pipelines (unit, contract, integration tests). **GenAI for Data Engineering** - Applying GenAI for metadata extraction, data anomaly detection, automated documentation, or pipeline optimization. **Domain Expertise** - Exposure to financial risk, treasury functions, or Collateral Management processes.