A Nikkei NEEDS licensee planning a research-scale ingest should start with 2,104 seconds for about 414 million rows, roughly 2.9 seconds per trading day. The 59.8x in the abstract comes later. That independent beta pass covered one ticker (7203) and 730 trading days across 2023 to 2025, using an 18-core i9-9980XE with 128 GiB. The authors present it as evidence of guard-respecting operation and end-to-end usability, separate from the controlled benchmarks. The planning rate of roughly 197,000 rows a second is my inference from their log. Their own fair comparison appears in the abstract, where the engine-matched 34.3x accompanies the 59.8x. They also footnote the 209-row index summary, where Polars runs at 0.3x of the pandas C-engine baseline, and exclude it from aggregate speedup claims.
Li, Hayashi, Nakatsuma and Romero built a Python library around one proprietary deliverable. Nikkei NEEDS distributes Tokyo Stock Exchange tick data in zipped CSVs, covering four file codes: individual-stock ticks, stock summary, index ticks and index summary. The delivery spans two schema eras, uses Japanese-language column documentation and encodes categorical fields as numeric codes. A complete 2016 to 2025 delivery across all four types contains 35,080 archives and occupies 1.725 TiB. Individual-stock ticks account for 34,720 archives and 1.72 TiB. Each trading day is divided among numbered parts, while the individual-stock layout produces 95 cleaned fields, most of them belonging to a ten-level quote ladder on each side.
tse_tick reads those files into typed Polars frames or writes a Hive-partitioned zstd Parquet store arranged as {type}/date=YYYYMMDD/ticker=CODE.parquet, with embedded DuckDB handling queries. Its one-shot ZIP reader and its ingest-then-query route use the same parsing and cleaning core. The library initially reads every field as a string, then casts explicitly, a sensible choice for ragged lines and quote columns full of blanks. The paper contains no return, Sharpe or signal. Its measurements are parse time, peak memory, storage size and query latency. The authors put the contribution in the engineering rather than the parser itself.
The engineering claims carry the paper.
We could not test any of it. Our platform has no NEEDS archives and no tick, quote or order-book data at all; the nearest available market is US equities at daily and one-minute bars. A parser for a Japanese vendor's ten-level ladder has no corresponding input there. A US substitution would reproduce neither the schemas nor the part layout or book features.
What the 410x contains
The query headline comes from a single-ticker, one-hour slice that returns 52 rows. Reading the Hive-partitioned Parquet store through DuckDB takes 0.0169 s, versus 6.9223 s for pandas scanning an equivalent monolithic CSV. The middle row printed in the paper matters more: DuckDB needs 0.8123 s to scan that same CSV. The engine change therefore contributes roughly 8.5x. Storage layout and partition pruning supply the remaining 48x. A shop already using DuckDB or Arrow is buying only the second factor.
The parse comparison receives the same care from the authors. A January 2017 part containing 4,777,240 rows and 95 columns runs in a median 4.774 s on 16 threads, compared with 285.552 s for the original pandas prototype. That prototype relied on the Python CSV engine because the NEEDS files have ragged lines, which the C engine rejects without additional configuration. The engine-matched pandas baseline takes 163.829 s, leaving a 34.3x gap bounded by observed extremes of 32.4x and 36.7x. Single-threaded Polars completes the job in 22.789 s, or 7.2x. The paper supplies enough disclosure to identify the fair comparison directly.
Throughput does not transfer cleanly between these tests. The headline parse covers one of nine parts for one day. On that monolithic job, peak process memory reached 13.1 GiB for Polars, 9.7 GiB for the engine-matched pandas baseline and 28.1 GiB for the prototype, all on a 32 GB machine. Polars uses 3.4 GiB more than the fair baseline. The beta ingest covered ticker 7203 across periods of 246, 241 and 243 trading days, moving about 414 million rows in 2,104 s on different hardware (i9-9980XE, 128 GiB). These are different jobs with different rates. Parsing one part without writing a store reaches about 1.0 million rows a second. Ingesting while also writing Parquet runs at roughly 197,000 a second. The latter is the paper's only research-scale, end-to-end ingestion result. The same appendix includes a whole-corpus query that scanned all 730 daily Parquet partitions with Polars and returned 22.3 million rows in 7.39 s.
A separate engineering change supports the query result. Since version 0.15.0, every individual_stock partition includes an Int32 HHMMSS Effective Time key. A bounded window can therefore use Parquet row-group statistics instead of computing an expression that prevents their use. On a real 2.56-million-row, single-ticker partition, the change made a one-minute slice 7.64x faster and a five-minute window 5.65x faster, at +0.52% storage.
Bounded memory, within a limit
The abstract gives an unqualified guarantee: writes stream in bounded morsels, making peak memory independent of trading-day size (24.5 GB to 2.4 GB on the worst measured day). Section 4.3 reports the measurement as 24.52 GB to 2.40 GB, a 10x reduction with a byte-identical store. It also supplies the missing boundary. Streaming applies to individual_stock days only when the ticker filter contains at most 64 codes. Full-frame days, summary types and index types follow a concatenate path sized from a per-day estimate. The condition is explicit in the section and absent from the abstract. A cross-sectional microstructure study covering the whole TSE name list falls outside the path that receives the 10x reduction.
Part-pruning has a similar boundary. NEEDS orders each day's numbered parts by ascending stock code, leaving one ticker in a contiguous run. The scanner opens that run and the appendix part. Across 18 days sampled over three years, the median gain was 6.3x, with a range from 1.3x to 25.8x, and every sampled day matched the full scan row for row. If the ascending layout cannot be verified, the scanner reverts to a full scan, preserving the full scan's correctness. A broad ticker set reaches every part, leaving nothing to prune on a many-ticker ingest in my reading.
Vendor conventions still govern the optimization. TSE began issuing alphanumeric codes such as 162A in 2024, breaking code comparison. Treating each code as a fixed-width four-character token restored pruning. On one affected 27-part day, a single-ticker read fell from 472 s to 13 s.
Worker selection uses the smaller of the logical core count and the capacity allowed by 70% of available RAM. On the 16-thread reference machine, a ticker-filtered ingest covering 19 trading days and 4.07 million retained rows scales to 1.9x with two workers, 3.3x with four and 3.6x with five. A request for eight workers was clamped because each streaming worker was assigned 3.0 GB against 70% of available RAM. During the beta run, 36 workers (108 GB) were reduced to 26 (78 GB). Both constants are hard-coded, so fast NVMe paired with modest RAM will leave some cores unused.
One-day caps
read_ticks and query_ticks stop at 10 million rows. The benchmark part contains 4,777,240 rows, and its trading day occupies nine parts, placing the cap inside a single full-day cross-section. The paper directs larger extracts to export_query or extract_to_store, both uncapped. The query cap now checks one row beyond the limit and raises a TruncationWarning.
Before 0.11.5, truncation was silent. An author's alpha pass on Python 3.14 also exposed a full-day one-shot read that exhausted memory through an untrappable Rust panic. Both faults were repaired three days later and received regression tests. The current one-shot path checks a 5 GB decompressed-input ceiling in advance and returns a catchable error. This review concerns the repaired version.
One intentional behavior remains unresolved for users comparing exports between machines. Equal-timestamp rows have no fixed tie order, so a slice exported on two platforms will be multiset-equal without being byte-identical.
The case for paying
The correctness controls make the strongest commercial case. A gate requires pandas and Polars to produce identical cleaned output for all four types, with floats matching within 1e-6. Every worker count produces a byte-identical store, including after the streaming rewrite. Coverage markers record the actual contents of each partition instead of assuming that file existence means completion, the kind of mistake that can quietly corrupt a ticker-scoped store.
The test suite collected 637 tests, with 587 passing and 50 skipped when licensed data was absent. On Linux with data, 636 pass and 1 skip. CI covers 3.9, 3.11 and 3.13. Each repaired defect retains a dedicated regression file. The authors also disclose AI assistance in drafting the manuscript and parts of the software.
The benchmark comparison remains limited to pandas under two engine configurations. We did not find parse-path results against ArcticDB, a DuckDB-only ingest, Dask or Spark. The related-work section describes ArcticDB as a storage solution rather than a parsing one. Without an institutional license, none of these results can be reproduced. The benchmark suite does ship and can read a local mirror, allowing a licensee to rerun every benchmark in Section 7. The paper also cites JPX Market Innovation & Research's ten-level order-book historical dataset, launched 29 June 2026. That product offers another route to TSE book depth and bypasses this deliverable entirely.
Existing NEEDS users with in-house parsing code should focus on coverage markers and pruning when considering a switch. Buyers still choosing a source should price the JPX Market Innovation & Research product first. A full-universe ingest benchmark would change my view: one month of all listed codes on the concatenate path, with peak memory reported.