An afternoon port: Taking Jieba to Rust with Codex

Agents changed the cost of replacing a dependency

When an impressive new model comes out, I like to throw a real project at it. This time, I tried porting the parts of Jieba we used at Speak from Python to Rust.

Chinese text doesn’t have spaces between words, so Speak uses Jieba to find those boundaries. We use them to compare a learner’s speech transcript with the target words in a lesson and to prepare Chinese text with word-by-word pinyin. The library hadn’t been maintained for a while, and its dependencies complicated our builds. It was also a good experiment: put strings in, get segmented text out, and compare the results automatically.

I built the feature and shipped it to production in one afternoon. I asked Claude Code to plan a drop-in Jieba replacement with a Rust core, Python bindings, uv, Python 3.11+, CPU and memory comparisons, and tasks small enough for smaller models. Then I asked it to implement PLAN.md and use subagents where appropriate. To my surprise, the model required very little steering and was mostly able to execute on the task entirely autonomously.

Checking the result

I focused on the behavior we actually used, including segmentation, token offsets, part-of-speech tagging, and dictionary updates. A Rust core with Python bindings let us keep the same interface while moving the expensive work into Rust.

The existing implementation gave us a reference, which is really helpful when automating a re-write. On the 1,633-sample Chinese benchmark corpus, outputs matched for every compared operation.

Initial benchmarks on my Mac M5 Max showed that the Rust port ran bulk operations 3.6–5.7× faster than Python. A later, separate Linux benchmark compared an optimized Rust version with its predecessor: part-of-speech tagging was 5.09× faster, and dictionary-resident memory use was 38.2% lower, though some general segmentation operations were slightly slower.

Some dependencies are worth another look

Rewriting even a small dependency used to be a substantial project. Agents make some of those projects cheap enough to try, especially when you can check the behavior automatically.

Owning the library also means we can handle edge cases there, rather than keep adding workarounds in our product code. We’re responsible for maintaining it, but we can change it to fit what we need.

If there’s a small dependency you’ve wanted to replace or adapt, it’s worth trying again.

Come build with us: speak.com/careers