Radar with Jeff Kao
Notas del episodio
Radar processes billions of location events daily, powering geofencing and location APIs for companies like Uber, Lyft, and thousands of other apps. When their existing infrastructure started hitting performance and cost limits, they built HorizonDB, a specialized database which replaced both Elasticsearch and MongoDB with a custom single binary written in Rust and backed by RocksDB.
In this episode, we dive deep into the technical journey from prototype to production. We talk about RocksDB internals, finite-state transducers, the intricacies of geospatial indexing with Hilbert curves, and why Rust's type system and performance characteristics made it the perfect choice for rewriting critical infrastructure that processes location data at massive scale.
About Radar
Radar is the leading geofencing and location platform, trusted by companies like Uber, Lyft, and thousands of apps to power location-based experiences. Processing billions of location events daily, Radar provides geofencing APIs, geocoding, and location tracking that enables developers to build powerful location-aware applications. Their infrastructure handles massive scale with a focus on accuracy, performance, and reliability.
About Jeff Kao
Jeff Kao is a Staff Engineer at Radar, where he led the development of HorizonDB, Radar's geospatial database written in Rust. His work replaced Elasticsearch and MongoDB with a custom Rust stack built on RocksDB, achieving dramatic improvements in performance and cost efficiency. Jeff has deep experience with geospatial systems and previously open-sourced Node.js TypeScript bindings for Google's S2 library. He holds a degree from the University of Waterloo.
Links From The Episode
Official Links
In this episode, we dive deep into the technical journey from prototype to production. We talk about RocksDB internals, finite-state transducers, the intricacies of geospatial indexing with Hilbert curves, and why Rust's type system and performance characteristics made it the perfect choice for rewriting critical infrastructure that processes location data at massive scale.
About Radar
Radar is the leading geofencing and location platform, trusted by companies like Uber, Lyft, and thousands of apps to power location-based experiences. Processing billions of location events daily, Radar provides geofencing APIs, geocoding, and location tracking that enables developers to build powerful location-aware applications. Their infrastructure handles massive scale with a focus on accuracy, performance, and reliability.
About Jeff Kao
Jeff Kao is a Staff Engineer at Radar, where he led the development of HorizonDB, Radar's geospatial database written in Rust. His work replaced Elasticsearch and MongoDB with a custom Rust stack built on RocksDB, achieving dramatic improvements in performance and cost efficiency. Jeff has deep experience with geospatial systems and previously open-sourced Node.js TypeScript bindings for Google's S2 library. He holds a degree from the University of Waterloo.
Links From The Episode
- Radar Blog: High-Performance Geocoding in Rust - The blog post, which describes Radar's migration from Elasticsearch and MongoDB to Rust and RocksDB
- FourSquare - The compay Jeff worked at before
- Ruby - The basis for Rails
- PagerDuty) - Another company Jeff worked at. Hes' been around!
- CoffeeScript - The first big JavaScript alternative before TypeScript
- Scala - A functional JVM based language
- Wikipedia: MapReduce - Distributed application of functional programming
- Wikipedia: Algebraic Data Types - The concept behind Rust's Enums, also present in e.g. Scala
- Kotlin - Easier than Scala, better than Java
- Apache Lucene - The core of ElasticSearch
- Discord Blog: Why Discord is switching from Go to Rust - Always the #1 result in searches for Rust migrations
- Radar Blog: Introducing HorizonDB - A really nice write up of Horizon's architecture
- RocksDB - The primary storage layer used in HorizonDB
- MyRocks - A MySQL Storage Engine using RocksDB, written by Facebook
- MongoRocks - A MongoDB Storage Layer using RocksDB
- CockroachDB - PostgreSQL compatible, distributed, SQL Database
- InfluxDB - A timeseries database that used RocksDB at one point, and our very first guest in this Podcast!
- sled - An embedded database written in Rust
- rocksdb - Rust bindings for RocksDB
- H3 - Uber's Geo Hashing using hexagons
- S2 - Google's Geo Hashing library
- Wikipedia: Hilbert Curve - A way to map 2 dimensions onto 1 while retaining proximity
- Wikipedia: Finite-State Transducer - A state machine used for efficiently looking up if a word exists in the data set
- Rust in Production: Astral - Our episode with Charlie Marsh about tooling for the Python ecosystem
- burntsushi - A very prolific Rust developer, now working at Astral
- fst - FST crate from burntsushi
- Wikipedia: Trie - A tree-structure using common prefixes
- Wikipedia: Levenshtein Distance - The number of letters you have to change, add, or remove to turn word a into word b
- tantivy - A full-text search engine, written in Rust, inspired by Lucene
- LightGBM - A gradient boosted tree, similar to a decision tree
- fastText - A text classification library from Meta
- Wikipedia: Inverted Index - An index used for e.g. full text search
- Wikipedia: Okapi BM25 - The ranking algorithm used in tantivy
- Wikipedia: tf-idf - A classic and simple ranking algorithm
- Roaring Bitmaps - A very fast bitset library used in many places
- corrode.dev: Be Simple - A sentiment right down Matthias' alley
- loco-rs - Rust on Rails
Official Links
Capítulos
Cargando capítulos…