IndexTables 0.5.5 continues the shift toward a fully native execution stack.
The headline change is a complete reimplementation of the transaction log in Rust, replacing the previous Scala-based design with a native module built on Arrow FFI. This release also introduces a DataSource short name and adds built-in profiling for the FFI read path.
IndexTables 0.5.4 introduces streaming companion synchronization, unified native memory management, and Arrow FFI across all data paths. Together these changes simplify the architecture, reduce serialization overhead, and make IndexTables behave more like a native Spark component rather than an external indexing engine.
More importantly, this release continues a broader shift in IndexTables toward incremental lakehouse indexing and fully columnar execution. Companion indexes can now stay synchronized with table changes continuously, while Arrow-native data movement and unified memory management allow Spark to treat indexing and search workloads as first-class citizens inside the execution engine.
This release also adds a streaming columnar reader shared across all split types and introduces a new complete read mode designed for ETL workloads.
Every database has indexes.
You don’t copy the table into the index.
You don’t govern it separately.
You don’t build ETL pipelines to keep it in sync.
An index is an acceleration structure, not a second source of truth.
So why does every search platform ask you to copy your data first?
With IndexTables 0.5.0, that changes.
Introducing Companion Mode — a fundamentally new way to add full-text search to your existing Delta Lake tables, Apache Iceberg tables, or raw Parquet datasets, without duplicating your data.
We're excited to announce the release of IndexTables 0.4.5, a milestone update that advances performance, stability, usability, and cloud-scale readiness for large-scale data workloads.
Since the 0.4.0 line, the IndexTables project has stayed focused on a simple goal: bring blazing-fast full-text search and analytics to Spark, running entirely inside your existing Spark cluster—no external services or additional infrastructure required. The result is dramatically faster search and aggregation compared to traditional table formats, with 100×–1000× speedups observed in real-world applications for interactive log analytics and cybersecurity workloads.
But performance is only the starting point.