Senior Staff Applied AI Engineer - Context Retrieval

About the Company

Requirements

• ● 10+ years of software engineering experience, with significant time spent building production retrieval, search, or RAG systems at scale.
• ● Deep Information Retrieval (IR) expertise : lexical retrieval (BM25, Lucene/Elasticsearch/OpenSearch), dense retrieval (embeddings, ANN indexes - FAISS, ScaNN, HNSW), hybrid retrieval, and learning-to-rank.
• ● Hands-on experience with modern LLM-era retrieval : RAG architectures, query rewriting, re-ranking with cross-encoders, long-context strategies, and grounding techniques that reduce hallucination.
• ● Experience designing agentic systems on top of retrieval - search planners, multi-hop / iterative retrieval, self-reflection and sufficiency checks, tool-using agents that decide what to fetch and verify what came back.
• ● Strong grasp of relevance evaluation : nDCG, MRR, Precision@K, Recall@K; offline/online experimentation; LLM-as-judge frameworks; building human labeling pipelines.
• ● Experience working across structured and unstructured data - you've indexed and ranked over tables, code, and documents in the same system, and have opinions about how to do it well.
• ● Track record of building 0 1 : you've stood up a retrieval system from an empty repo, made the foundational architectural decisions, and grown it into something that customers depend on.
• ● Demonstrated ability to operate as a technical leader : setting direction across teams, mentoring senior engineers, and influencing roadmap with research, product, and platform partners.
• ● Experience building retrieval over enterprise SaaS sources (permissions, freshness, multi-tenancy, ACL-aware indexing).
• ● Background in agentic systems, tool use, or multi-turn retrieval for LLM agents.
• ● Contributions to open-source IR/search projects, or publications at SIGIR, KDD, WWW, EMNLP, or similar venues.
• ● Experience training or fine-tuning embedding models, rerankers, or query understanding models.
• ● Foundational impact. Retrieval is the single biggest lever on agent quality. The stack you build will sit underneath every Databricks agent and every customer-built agent on our platform.
• ● Greenfield with scale. You get the rare combination of starting from a clean sheet and having immediate access to massive enterprise scale, real customer data, and a world-class research org.
• ● The right team. You'll work alongside engineers and researchers behind Lakehouse, Apache Spark , Delta Lake, MLflow, MosaicML, and DBRX.
• ● This role is based in our Mountain View, CA or San Francisco, CA office. Hybrid in-office collaboration expected.
• ● Databricks is committed to fair and equitable compensation practices. The pay range(s) for this role is listed below and represents the expected salary range for non-commissionable roles or on-target earnings for commissionable roles. Actual compensation packages are based on several factors that are unique to each candidate, including but not limited to job-related skills, depth of experience, relevant certifications and training, and specific work location. Based on the factors above, Databricks anticipates utilizing the full width of the range. The total compensation package for this position may also include eligibility for annual performance bonus, equity, and the benefits listed above. For more information regarding which range your location is in visit our page here .

Benefits

• ● At Databricks, we strive to provide comprehensive benefits and perks that meet the needs of all of our employees. For specific details on the benefits offered in your region click here .
• ● At Databricks, we are committed to fostering a diverse and inclusive culture where everyone can excel. We take great care to ensure that our hiring practices are inclusive and meet equal employment opportunity standards. Individuals looking for employment at Databricks are considered without regard to age, color, disability, ethnicity, family or marital status, gender identity or expression, language, national origin, physical and mental ability, political affiliation, race, religion, sexual orientation, socio-economic status, veteran status, and other protected characteristics.
• ● If access to export-controlled technology or source code is required for performance of job duties, it is within Employer's discretion whether to apply for a U.S. government license for such positions, and Employer may decline to proceed with an applicant on this basis alone.