Skip to content

From the One Billion Row Challenge to a Multi-Threaded Parquet Library

September 14, 2026 · 59 min

Show notes

An airhacks.fm conversation with Gunnar Morling about: returning guest discussion four years after episode "#174 Kafka Connect CLI, JFR Unit, OSS Archetypes and JPMS", the One Billion Row Challenge (1BRC) as a Java community coding challenge, aggregating a 13 gigabyte semicolon separated file of one billion temperature measurements per weather station, baseline single threaded idiomatic Java at five minutes versus a 1.5 second winning entry, two orders of magnitude speedup, parallel processing scaling near linearly across cores, memory mapped files via the Foreign Function and Memory API for arbitrarily large files, reading eight bytes at once into a long, SIMD parsing via the incubating Vector API, GraalVM native binaries, no third party libraries and single source file rule, results on par with C, Thomas Würthinger and the GraalVM team participation, HTMLDB a Java 25 executable shebang script storing records as semantic HTML pages using the built in XML parser, agents reading XML better than JSON, Hardwood a minimal dependency multi threaded Java library and CLI for reading and writing Apache Parquet files, columnar versus row based formats, efficient encodings including delta encoding and bit packing, trading CPU for disk size, range requests to read specific columns from object storage like S3, the dependency heavy Parquet Java library pulling in the Hadoop stack, zero mandatory dependencies with optional modules, a hand written AWS Signature Version 4 signer using the JDK HTTP client instead of the AWS SDK, verifying the signer against the AWS TCK test suite, owning code with LLMs, optional dependency for IAM instance profiles, Zstandard and Snappy compression as domain specific dependencies, Java LLM code generation quality and import handling, AIRails.dev skills based on Boundary Control Entity, spec driven development storing the spec in package-info Java Doc using EARS notation, the code review pyramid as a review skill, GraalVM WebAssembly web image compiling a Java text user interface to analyze Parquet files in the browser, an agent skill for Parquet analysis in natural language, Apache Iceberg and S3 Tables, Apache Flink integration, the shift in the reuse versus own trade-off with LLMs, Agent zSmith a zero dependency Java agent harness, var over explicit types

Gunnar Morling on twitter: @gunnarmorling

All episodes of airhacks.fm podcast with adam bien

Settings

Display

Language

Decides which stations you see first on the home page.

History

Recently played stations and episodes and your latest searches are only stored in this browser.

Notifications

A short notice when a podcast you follow has a new episode. Turn off single podcasts with the bell on their page.

Account

Save favorites – in the app and on the website.

Keyboard shortcuts

/
Open search
Space
Play or pause
← →
In the player: back 15 s, forward 30 s (episodes) · in a row: next tile
↑ ↓
In a row: target within the tile
Home / End
In a row: first or last tile
Esc
Close window or menu
?
This overview
Now playing

Go to station page

Play next

    Add episodes with “Play next” – they play after the current episode.