A Billion Data Points Per Second

An airhacks.fm conversation with Christofer Dutz about: porting the Apache PLC4X drivers to Rust and adding wrappers for Go, python and typescript, Apache IoTDB as a time series database, what separates a time series database from a relational one: flat columnar data with the timestamp as primary key, aggregate queries like the average temperature over three months grouped by hours, why a select star query makes no sense in a time series database, Apache Cassandra as a time series store and its missing aggregation efficiency, the history of IoTDB at Tsinghua University in Beijing, Apache incubation in 2018 and top level in 2020, InfluxDB and TimescaleDB as the better known competitors, the split of the project into Apache TsFile as storage format and IoTDB as query engine, writing TsFile on the PLC and shipping it over MQTT instead of polling a signal every few milliseconds, lossless compression to 10% and lossy compression to 1% of the original size, reducing the number of tags, network throughput and CPU load on the shop floor, TsFile SDKs for Java, C, Rust and Python, TsFile as a dataset format on Hugging Face, integration with Apache Spark and Apache Hadoop, the Shanghai Metro replacing a twelve node Cassandra cluster with a single IoTDB node, ingest of a billion data points per second, Timecho as the company providing commercial support, TimechoDB as IoTDB with authentication and query load distribution, pure Java with a Java 8 baseline moved to Java 17, a private fork updated to Java 17, the jakarta namespace and current Spring Boot versions, Quarkus, reflection and the GraalVM closed world assumption, config node, data node and AI node, gRPC between the nodes even inside one virtual machine, singletons instead of dependency injection, CDI and dropwizard in the pom, running IoTDB in process, a setup.exe install compared to a Cassandra cluster with docker and kubernetes, starting an inference process in process instead of over HTTP, lightmetal and zsmith as an example, Vidocq as a CDI implementation without external dependencies, the JDBC driver, the REST API and an Apache Drill plug-in, push queries that call back whenever the query produces a new result, hierarchies of aggregation levels with per level retention from milliseconds on the shop floor to years for business analysis, the tree identifier of company, site, line, machine and machine part, the table model added for users who think in table algebra, data types, missing updates, delete and automatic purging by retention, replication so queries run on a second instance, the built-in MCP server giving Claude access to an IoTDB instance, the AI node running models inside the database, data gravity, a KNX house logging into IoTDB on a Raspberry Pi, an IoTDB app running in process on a Bosch Rexroth ctrlX PLC

Christofer Dutz on linkedin: christofer-dutz