Staff Software Engineer, Observability

Summary We have an opening for a Staff Software Engineer on our Infrastructure Team, with specific focus on Observability - both internal and customer-facing . This is an opportunity to join a mission-critical engineering team that is driving the productivity and reliability of Temporal’s developers and core platforms, respectively. We are a passionate team of talented developers who truly care about our mission and enjoy working deeply across the entire Temporal ecosystem to be a force multiplier across the organization. What You'll Do Build: • Lead the end-to-end Software Development Lifecycle: goals & requirements solicitation, design & review, implementation, operationalization & deployment, support & maintenance. • Lead feature design, review with stakeholders, iterate to incorporate feedback and drive consensus. • Clearly document design choices and operational knowledge to successfully deploy and manage the software you develop. • Provide appropriate test and production readiness coverage for unit, integration, and performance of your feature ownership area. Own: • Set a high bar for technical excellence and take pride in the software you develop. • Design and build multi-component, distributed systems that operate at scale. • Investigate issues with a methodical approach to identify a root cause. • Understand performance and reliability implications of design options at scale. Make related tradeoffs. • Participate in the team’s on-call rotation. Learn: • Expert-level knowledge of architecture and services of assigned domain. Strong command over all aspects of the Temporal ecosystem. This role provides a strong opportunity for diverse technical exposure! • Investigate and understand ways to best leverage Temporal’s own software to power our mission. • Deeply understand the needs of Temporal internal developers and external customers, and leverage that knowledge for product development and feature design. Collaborate: • Lead and mentor fellow team members. • Participate in design reviews and contribute to design of other features. • Share design principles for building reliable systems at scale. What You'll Bring • User-first mindset. You’re excited by the opportunity to empower others through tooling, and enjoy deeply internalizing user goals and use cases to build effective solutions. • Motivated by impact. You are driven by a desire to make positive things happen. • Strong opinions about tools and technology that are equally balanced by a pragmatic drive for impact. • Ability to work in a self-directed manner in a fast-paced environment. • Excellent collaboration and communication skills. Skills & Technologies • Demonstrated ability to develop horizontally scalable, resilient, and high performance distributed systems in a production environment. • Experience designing, implementing, deploying, and supporting large scale, geographically distributed observability and/or high throughput data streaming/processing pipelines, or similar. • Expert in one or more high-level programming languages, preferably Go. • Expert-level Kubernetes skills. • Expert-level query development skills, preferably SQL. • Hands-on experience with one or more cloud providers, preferably AWS, or GCP. • Thorough understanding of computer architecture, operating systems, and networking. • Familiarity with best practices regarding monitoring, instrumenting, and configuring infrastructure. Team Highlights • Core observability tech stack: Clickhouse, Prometheus, Grafana, Loki, Thanos, Kubernetes, Temporal. • Our recent observability project had the biggest impact to Temporal margins of any engineering effort in company history while also unlocking unprecedented operator capabilities.  • “Observability is the beating heart of our ability to deliver a mission critical service” -Preeti Somal, Senior VP of Engineering. Compensation • The estimated pay range for this role is $212,000 - $286,200 depending on qualifications and location. • This role is eligible to participate in Temporal's equity plan.