Your career compass awaits
Create a free account to unlock
- ✓ See how you match this role
- ✓ AI resume tailored to this specific job
- ✓ Inside Track Companion — find your insider contact
- ✓ Skills gap analysis and upskill plan
- ✓ Interview prep kit for this role
- ✓ Relocation concierge — salary, tax, cost of living
Free — no credit card required
Staff / Senior Software Engineer (Agentic Search) - Crawler
ICT
💰 Salary Not shared
📊 Market 10,000 EUR–15,000 EUR/mo (120,000 EUR–180,000 EUR/yr)
About the Company
Nebius is leading a new era in cloud infrastructure for the global AI economy. It builds a full stack AI cloud platform that supports developers and enterprises from data and model training through to production deployment, eliminating the cost and complexity of large in house AI/ML infrastructure. The company, built by engineers for engineers, handles large scale GPU orchestration, inference optimization, compute, storage, networking and applied AI. Listed on Nasdaq (NBIS) and headquartered in Amsterdam, Nebius has a global footprint with R&D hubs across Europe, the UK, North America and Israel, and a team of over 1,500 employees, including hundreds of engineers with deep expertise in hardware, software and AI R&D.
About the Role
We are looking for a Senior Software Engineer to work on the content acquisition and crawling infrastructure of a novel search engine tailored for agentic AI consumption. In this role you will focus on building systems that discover, fetch, and continuously refresh content from the open web and other large scale data sources, designing distributed crawling, scheduling, and ingestion infrastructure that operates at internet scale while balancing coverage, freshness, resource efficiency, and reliability. You will work on systems that process billions of URLs, manage high throughput data flows, and ensure high quality content is consistently available to downstream indexing and retrieval systems.
Responsibilities
- ● Design, implement, and operate web scale crawling systems for acquiring content from the internet.
- ● Build ingestion workflows for internal and external data sources, including crawlers, structured feeds, and partner integrations.
- ● Develop crawl scheduling, prioritisation, recrawl policies, and freshness strategies.
- ● Build systems for URL discovery, deduplication, content extraction, and crawl orchestration.
- ● Ensure reliable operation of crawling infrastructure under high throughput conditions.
- ● Define observability and quality metrics for crawl coverage, freshness, throughput, and content quality.
Confirm Application
Are you applying to ?
We'll track this in your dashboard as Applied.
Saving to your tracker…
Stand out: message someone at Nebius.
A short, personal note to a recruiter can get your application looked at sooner.
⚡ Full Resume Sandbox Canvas