Principal Site Reliability Engineer

Job description

**About the Role** Oracle Cloud Infrastructure (OCI) is looking for a Principal Site Reliability Engineer (IC4) to help build, operate, and evolve highly available, scalable, and resilient cloud services. As a Principal SRE, you will take technical ownership of reliability across complex, distributed systems operating at cloud scale. You will work closely with software engineering, architecture, security, and operations teams to influence service design, improve availability and performance, automate operational work, and ensure our services meet their reliability objectives. This role goes beyond operating production systems. You will identify systemic reliability risks, influence architecture and engineering decisions, lead complex incident investigations, build automation, improve observability, and drive long-term engineering improvements. You will also serve as a technical leader within the team, mentoring engineers and helping establish strong SRE practices across services. Career Level - IC4 **Key Responsibilities** **Reliability Architecture Capacity Engineering** - Design and influence architectures for highly available, resilient, scalable, and operationally efficient cloud services. - Partner with software development teams during design and implementation to ensure reliability, scalability, observability, security, and operability are built into services from the beginning. - Identify architectural and operational risks across multiple services and drive engineering improvements to address them. - Define and improve Service Level Indicators (SLIs), Service Level Objectives (SLOs), monitoring strategies, and reliability standards. - Forecast infrastructure and service capacity based on workload growth, utilization trends, architecture changes, and customer demand. - Identify capacity risks and bottlenecks before they impact customers and drive appropriate mitigation plans. - Lead technical prototypes and evaluations for new infrastructure, reliability patterns, and operational technologies. **Production Engineering Service Lifecycle** - Own and continuously improve the operational health of production services. - Analyze service telemetry, operational data, and reliability trends to identify systemic issues and improvement opportunities. - Drive improvements across availability, latency, performance, scalability, security, recoverability, and operational efficiency. - Establish mechanisms to detect reliability degradation before it becomes customer impacting. - Lead complex service lifecycle activities including upgrades, migrations, security updates, disaster recovery, capacity expansion, and decommissioning. - Identify recurring operational issues and convert them into engineering problems with sustainable solutions. **Automation Toil Reduction** - Identify high-impact opportunities to eliminate repetitive operational work through software engineering and automation. - Design and build scalable automation, tooling, and frameworks for deployment, monitoring, diagnostics, mitigation, remediation, and service lifecycle management. - Develop automated mechanisms for detecting and recovering from common failure scenarios. - Establish engineering standards for operational tooling, ensuring automation is reliable, testable, maintainable, observable, and safe. - Measure operational toil and drive initiatives that improve engineering efficiency and reduce manual intervention. - Review and improve automation developed by other engineers. **Observability Performance Engineering** - Define and improve observability strategies across services using metrics, logs, traces, dashboards, and alerting. - Develop meaningful service health indicators that accurately reflect customer experience. - Analyze production workloads to identify performance bottlenecks, resource inefficiencies, scaling limitations, and reliability risks. - Drive improvements to monitoring and alerting to improve signal quality and reduce operational noise. - Use production data and reliability trends to influence architecture, capacity planning, and engineering priorities. **Technical Leadership Engineering Excellence** - Provide technical leadership for reliability initiatives spanning multiple services or engineering teams. - Influence architecture and design decisions by identifying reliability, scalability, operational, and failure-mode considerations. - Lead technical discussions and design reviews for complex infrastructure and reliability challenges. - Establish and promote engineering best practices for operating large-scale distributed systems. - Mentor SREs and software engineers in troubleshooting, incident management, automation, observability, and reliability engineering. - Review designs, operational readiness, automation, and implementation approaches and provide actionable technical feedback. - Raise the overall technical and operational maturity of the team. **Cross-Team Collaboration Communication** - Partner with software engineering, architecture, security, networking, infrastructure, and operations teams to solve complex reliability problems. - Clearly communicate service health, operational risks, capacity constraints, incident impact, and reliability priorities to technical and non-technical stakeholders. - Anticipate the operational impact of infrastructure, architecture, feature, and tooling changes across multiple services. - Drive alignment across teams when reliability improvements require changes across organizational boundaries. - Provide clear technical recommendations supported by production data and engineering analysis. **Continuous Improvement Innovation** - Evaluate emerging technologies, engineering approaches, and SRE practices that can improve reliability, scalability, security, or operational efficiency. - Identify systemic weaknesses in existing operational processes and drive improvements. - Use operational data, incident trends, and engineering metrics to pri