Kotoshu — News

kotoshu 1.0.6: four fixes and an honest library

2026-09-15T00:00:00.000Z

Expired caches resolve instead of raising, the dictionary catalog reaches its repo again, a suggestion crash on tied scores is gone, and every URL the library builds now points somewhere live.

1. An expired-but-present cache resolves: Kotoshu.correct? no longer raises ResourceNotSetupError for a language whose cached dictionary metadata predates bookkeeping, and kotoshu setup --list agrees with resolve. TTL now means refresh-at-setup, never absence-at-read.

2. Dictionaries::Catalog fetches from the v1 branch - every catalog download had been a 404 against a branch the dictionary repository abandoned. NgramSuggest no longer crashes with comparison of Array with Array when tied root scores span mixed dictionary entry shapes; the sort compares scores only. The deprecated dictionaries URL default and a dead model-URL fallback joined the fix.

3. The typo layer hardened: a prebuilt matrix now carries the sha of the exact full-tier vocabulary it was derived over, and an arm that does not pair with the cached tier derives instead of answering wrong slates. The signature file parses for the first time, all 32 example walkthroughs run, and kotoshu-lsp 0.1.2 floors on the 1.0 engine line.

kotoshu 1.0.5: typo arming becomes a download

2026-09-14T00:00:00.000Z

The typo-retrieval layer now ships prebuilt matrices for all six full-feature languages, setup backfills them on existing installs, and the engine records how it armed - plus a weekly guard against stale extension pins. The Rust crate follows at 0.2.1.

1. kotoshu setup LANG --typo now ends with the language's 26 MB KTM1 matrix in the cache, and an install set up before the matrices existed picks one up on its next setup run instead of deriving the index at load time. Arming measures about half a second where deriving took 25 to 45 seconds, and German - the language where the hybrid's benchmark win is largest - arms in 511 ms against 44.9 s derived.

2. The engine records which path won: Kotoshu::Typo::Engine#armed_via answers :matrix or :derived, Kotoshu::ResourceManager.setup?(lang, resource: :typo_matrix) probes the artifact, and kotoshu setup --list prints per-resource detail such as en: spelling, model, typo, typo-matrix. A stale extension can no longer silently fall back to deriving - a weekly workflow compares the extension's kotoshu-rs pin against main and fails loudly with the fix recipe, and the release verify job installs the shipped gem and asserts the native surface before green-lighting a cut.

3. The Rust crate follows at 0.2.1 with the public matrix surface (TypoEngine::from_matrix, TypoIndex::parse_ktm1/write_rows), and the model registry grew a fetchability gate: every primary, mirror, and vocab URL is probed with ranged requests on each pull request and before every release, so a dead mirror can never reach a tag again.

The hybrid typo-retrieval layer is live across the ecosystem

2026-09-14T00:00:00.000Z

A 0.481 MB character bi-encoder now retrieves the twenty nearest vocabulary entries for a misspelling and the fastText full tier rescores them, with the layer wired end to end: registry v1.7.0 carries the artifacts, kotoshu 1.0.3 fetches them with one setup flag, and the Rust crate and WebAssembly package expose the same engine.

1. The registry v1.7.0 release promotes the typo bi-encoder from an opt-in mirror to a first-class artifact pair, generated by the descriptor-driven release flow with the validator enforcing that the model and its character vocabulary travel together. kotoshu 1.0.3 adds kotoshu setup LANG --typo: one flag fetches the pair plus the full tier, reports the layer as cached on later runs, and arms it behind KOTOSHU_TYPO_RETRIEVAL with a byte-identical passthrough when off.

2. The engine reproduces the frozen benchmark exactly through the shipped path - top1 0.118772 and top5 0.269829 on the 2,509 real English pairs, identical to the measured verdict - at 1.8 ms per query after the index builds eagerly with the Ruby GVL released (a 25-second derivation no longer pauses any thread). The pure-Rust forward pass matches onnxruntime to cosine 1.0. kotoshu-native 0.1.1 on PyPI now carries all 21 platform wheels, and the crate 0.2.0 and @kotoshu/wasm 1.1.0 cut the same surface for Cargo and npm. Prebuilt 26 MB KTM1 matrices - one per full-feature language, served by the registry - arm the same engine in about half a second instead of a 25-to-45-second index derivation.

kotoshu 1.0.2: the Rust engine installs precompiled

2026-09-12T00:00:00.000Z

gem install kotoshu now resolves a precompiled native platform gem on Linux, macOS, and Windows, so the Rust engine runs with no compiler toolchain, and the backend default became auto: use the native engine when it loads, fall back to pure Ruby when it does not.

1. Five platform gems ship alongside the pure-Ruby gem — x86_64-linux, aarch64-linux, arm64-darwin, x86_64-darwin, and x64-mingw-ucrt — each carrying the compiled extension inside the gem and built natively on its own runner. The CI matrix proves every one by installing the platform gem into a toolchain-free environment and asserting that the native engine engages. RubyGems picks the platform gem automatically on a matching machine, and every other platform, including musl Linux, resolves the pure-Ruby gem exactly as before.

2. The backend setting now defaults to auto, which selects the native engine when the extension loads and silently uses pure Ruby when it does not; KOTOSHU_BACKEND=native and KOTOSHU_BACKEND=ruby still force an engine explicitly. No output changes anywhere: both engines remain pinned to the frozen 2,630-vector conformance contract. The Docker CI image follows in the same arc by dropping its build-tool layer in favor of the platform gem.

kotoshu 1.0.1: deterministic ranking by construction

2026-09-12T00:00:00.000Z

The gem now embeds the frozen Kelly frequency tiers, so a machine with a cold cache ranks suggestions exactly like the frozen conformance vectors, without any download, CI seeding, or dataset switch at cache expiry.

1. This release closes the last dataset-divergence path. The frequency provider now falls back from the cache to the frozen embedded tiers before the local YAML file, and the embedded table is generated from the same arrays the Rust engine compiles in, with provenance and the upstream checksum stored inside the file. The empty-cache conformance comparison over all 2,630 vectors passes byte for byte with no cache present at all.

2. The patch also threads the baseline suggestion-skip through the Jekyll generator, so baselined builds stop paying the sweep for covered words, matching the CLI and Rake paths. kotoshu-server 1.0.1 fixes its runtime floor, because the ~> 0.6 cap it had shipped with silently excluded the 1.x gem.

Synthetic typo corpora, published error budgets, and suggestion reporting

2026-09-12T00:00:00.000Z

Four new 5,000-pair dictionary-grounded typo corpora, a published per-tier error budget, one-click wrong-suggestion reporting in the playground, and a rebuilt Docker CI image.

1. The synthetic corpora are verifiable by construction: every correction is a dictionary stem, every misspelling is reachable by a declared keyboard-noise operation, and the whole file is a seeded pure function of the dictionary pin. The headline measurement is that even the full tier can embed only about 1.5% of these realistic misspellings, which quantifies the gap between embedding tiers and actual misspellings and motivates the dictionary sweep.

2. The performance documentation now carries the tier error budget, listing worst-case rank correlation and top-1 agreement across all 55 languages next to the gates that enforce them, so every claim about a cheaper tier carries its measured number.

3. The playground popover gained a report link: one click opens a pre-filled issue with the word, the offered suggestions, the language, and the engine version, and nothing else. Verified reports feed the corpora.

4. The Docker CI image could not install kotoshu at all once the native extension shipped; a newly added version guard caught the failure on its first run, and the image now builds the Rust accelerator properly.

kotoshu 1.0.0: the stability freeze

2026-09-10T00:00:00.000Z

The coordinated 1.0 cut: the Ruby gem, @kotoshu/wasm, and kotoshu-server all reach 1.0.0 with the public surface frozen and the 2,630-vector Ruby==Rust conformance contract gated in CI on both sides.

1. Version 1.0 freezes the strict two-stage resource model, in which setup is never implicit and the hot path never downloads; the CLI exit codes 0 through 3 and the JSON and SARIF output shapes; the kotoshu.resources/v1 registry schema and the /v1 HTTP prefix; the extension points for custom dictionary types and suggestion algorithms; and the behavioral contract itself, which is the set of 2,630 frozen conformance vectors that the Ruby and Rust engines must both match byte for byte, now enforced by CI jobs in both repositories.

2. The last blocker before the cut was found by running the audit checklist, not by a user: suggestion ranking silently changed when the frequency cache passed its 7-day TTL, because the reader fell back to a differently-curated local dataset. Fixed, tested with the cache deliberately expired, and the compare job added so the invariant can never drift silently again.

3. At 1.0 the gem is on RubyGems, @kotoshu/wasm is on npm with loadPack language packs for English, German, and Portuguese, and kotoshu-server ships model-aware checking. The GitHub action runs in six repositories including this one, and kotoshu-lsp, PyPI, crates.io, the Go client, and the zero-install playground complete the channel picture.

kotoshu 0.11.1: baselines that work on real repositories

2026-09-10T00:00:00.000Z

The baseline command now accepts directories and globs, ignores the personal dictionary, and skips the suggestion sweep, and check --baseline no longer generates suggestions for the occurrences it suppresses.

1. Dogfooding action-kotoshu@v2 on the gem repository itself surfaced four defects in baseline init, all fixed: directories and globs expand exactly like check directory mode; a target matching no file is refused with the same File not found instead of silently vanishing; the personal dictionary is never consulted, so baselines freeze what every machine sees; and file reads scrub invalid bytes.

2. The baseline generator also skips the suggestion sweep whose output it was discarding, which turned 49 CPU-minutes over the gem repository into 84 seconds. The gate side received the same treatment, so check --baseline only generates suggestions for occurrences that will surface. On this repository's own CI gate, 1,905 covered occurrences no longer generate suggestions, and the check phase dropped from 13 minutes to 14 seconds with identical output.

3. The gem repository now runs its own spellcheck workflow — the sixth repo in the rollout and the first Ruby one, closing the loop that found these bugs.

loadPack ships in @kotoshu/wasm 0.5.0

2026-09-10T00:00:00.000Z

A single language pack now carries the dictionary, the embedding tier, the vocabulary, and the bucket table, and one loadPack call turns one fetch into the complete set of engine handles.

1. loadPack(packBytes) parses the KPK1 container, verifies the checksum of every section before constructing anything, and returns dictionary and model handles, with the model already bucket-attached when the pack carries a bucket table. The package published through keyless publishing after the owner cleared a publishing-policy setting on the npm side.

2. The playground uses it for English, German, and Portuguese, so three or more round trips collapse into one, and per-artifact loading remains the fallback for languages without a pack.

Language packs and the typo-biencoder hybrid in registry v1.6.0

2026-09-10T00:00:00.000Z

One fetch now loads a whole language through KPK1 packs for English, German, and Portuguese, and the typo-biencoder hybrid becomes the first alternative model to beat the full tier on every gate.

1. A language pack concatenates the dictionary files, the mini tier, the vocabulary, and the bucket table into one KPK1 artifact with per-section checksum footers, so three round trips become one. loadPack on the wasm surface and pack mode in @kotoshu/worker both fall back to per-artifact loading when no pack exists.

2. The model bake-off closed with a winner: the 0.48 MB typo-biencoder hybrid, which retrieves with the tiny bi-encoder and rescores with fastText, beats the full tier on every real-pair component at 3 to 4 ms per lookup. English gains +6.3 points on top-5 with a confidence interval of +4.9 to +7.7 over 2,509 pairs, German gains +28.6, Spanish gains +10, and no component regresses. The hybrid is registered opt-in as kotoshu://models/typo/typo-biencoder, and the candidates rejected on measurement along the way were int4, the MiniLM rerank, ModernBERT, and the standalone bi-encoder.

3. The registry now carries 383 release assets, and the lid primary URLs work again after pointing at unattached assets since v1.5.0.

The engine worker ships as @kotoshu/worker 0.1.0

2026-09-09T00:00:00.000Z

The playground worker protocol ships on npm as createEngine, covering load, check, suggest-batch, semantic, and detect, with the same pinned resources, the same caching, and no install step.

1. @kotoshu/worker exposes createEngine(onMessage), which speaks the full protocol the playground runs: dictionary loading with progress, batched priority suggestions, the opt-in semantic tier with buckets, and language detection, plus a ready new Worker() entry and the pure semantic-merge half. It runs in the browser and in Node.

2. CI now enforces what docs used to claim: latency gates (en 56 ms average vs the 120 ms budget, pt 346 vs 700) and frozen wasm memory ceilings (en 46.9/64 MB resident with tier and buckets).

Bucket tables for 47 languages in registry v1.5.0

2026-09-08T00:00:00.000Z

Model-generated candidates now reach out-of-vocabulary misspellings in 45 more languages, and eight languages were rejected on measurement with full evidence ladders.

1. Forty-five new bucket-table siblings of 10 to 12 MB each let semanticSuggest embed the out-of-vocabulary n-grams the vocabulary lacks, in every language the fidelity gates passed. French rose from 0.696 to 1.000, Spanish from 0.667 to 1.000, and Russian from 0.623 to 1.000 in out-of-vocabulary resolution on real-corpus probes.

2. Eight languages (ar, cs, fa, he, ja, pl, vi, zh) were rejected on measurement, with the gates never weakened; full ladder evidence is committed for a future decision. Serbian and Swedish ship at K=65536 with a recorded size deviation.

3. The registry now holds 213 resources at revision 7, and the playground picks up every addition automatically.

All staged languages reachable; Korean and Nepali gain full support in kotoshu 0.11.0

2026-09-07T00:00:00.000Z

setup no longer gates on a module, so 95 staged languages work out of the box with script-aware fallbacks, and Korean and Nepali join the full-feature 35.

1. The basic tier (plan 107) derives AVAILABLE_LANGUAGES from the dictionaries manifest instead of a hardcoded module list, so every staged language installs and checks, with script-aware tokenizers and per-script keyboard fallbacks applied automatically. Modules are now upgrades rather than gates, and nn is wired as the 33rd full-feature language.

2. Korean and Nepali reach full feature (plan 108) through a Hangul eojeol tokenizer on the Dubeolsik 2-set grid (KS X 5002) and a Devanagari tokenizer that keeps matras and conjuncts whole on the InScript grid, covering the last big-population languages without support. The total is 35 full-feature languages, with live specimen round trips in both scripts.

3. The cut also carries the 1.0 public-surface cleanup: the global DictCommand leak is fixed, dead command classes and unused aliases are removed (1,095 lines), and the experimental checkers are documented. The suite runs 3,985 examples with conformance untouched.

/v1/detect covers 169 languages in kotoshu-server 0.1.2

2026-09-07T00:00:00.000Z

Detection prefers the lid-176 model through the gem 0.10.0 native path, and the engine field reports which detector served the answer.

1. POST /v1/detect returns { language, confidence, engine }: lid-176 (176 languages, lazy model setup on first request) when kotoshu >= 0.10.0 and the native extension can load the model; heuristic (the 7-language fallback) otherwise - old gems, KOTOSHU_BACKEND=ruby, offline setup failures, or KOTOSHU_DETECT=heuristic. Setup failures log once and degrade; never a 5xx.

2. The release also fixes a pre-existing failing build on main, where App::VERSION had drifted from the release-workflow bumps, and documents the engine field in the README and the OpenAPI schema.

Personal dictionaries in the check path and native detection in kotoshu 0.10.0

2026-09-07T00:00:00.000Z

kotoshu check finally consults the personal dictionary, and Kotoshu.detect_language identifies 176 languages through the pure-Rust lid reader.

1. The personal dictionary reaches the check path (plan 105): a word added with kotoshu personal add stops being an error on the next check - case-insensitive, no metadata, one load per process, mirroring the editor integration. Opt out with --no-personal or KOTOSHU_PERSONAL_DICTIONARY=false. Until now the README story was false on the CLI path.

2. Kotoshu.detect_language -> { code, score } (plan 106): the lid-176 model through the native extension, ~42 microseconds warm, 55/55 parity with the reference on the frozen corpus (en 0.87, de 0.99, ja 0.998, ar 0.97 live). Pure-Ruby installs keep the 7-language heuristic; kotoshu-server /v1/detect upgrades to the 176-language path in 0.1.2.

Twelve more languages ship in the gem with kotoshu 0.9.3

2026-09-07T00:00:00.000Z

Full-feature batch 3 reaches RubyGems with 32 languages carrying modules and keyboards, plus the right-to-left tokenizer fix, on the 0.9.2 indexed-sweep base.

1. Cut from main after PR #151: ar, id, fa, he, bg, sr, hr, sk, sl, lt, lv, et as full-feature modules (national keyboard grids, AVAILABLE_LANGUAGES 20 to 32, 49 engine-verified specimens). The script-aware word regex fix means kotoshu check -l ar|fa|he extracts real words from RTL documents.

2. The release sits on the 0.9.2 sweep index, so the new languages get the indexed sweep from the first install, and gem install kotoshu now resolves 0.9.3 for lsp and server users too.

Language detection and OOV bucket tables in @kotoshu/wasm 0.4.0

2026-09-07T00:00:00.000Z

loadLid and detectLanguage bring 176-language identification, and bucket-backed out-of-vocabulary embedding lets Teh reach The through the model path.

1. The lid-176 model (~1 MB + vocab) ships through the registry v1.4.0 mirror. The playground detect control loads it once, scores the editor text, and proposes a language switch — never auto-switches an explicit choice.

2. Bucket-table siblings for English and German close the model-side Teh gap: n-grams absent from the vocabulary fall back to hashed bucket rows, while in-vocabulary behavior stays byte-identical.

Twelve more full-feature languages bring the total to 32

2026-09-07T00:00:00.000Z

Arabic, Indonesian, Persian, Hebrew, and eight European languages join the full-feature set: gem modules, national keyboards, and engine-verified specimens.

1. Plan 100 batch 3 promotes ar, id, fa, he, bg, sr, hr, sk, sl, lt, lv, and et from the degraded path to full feature. Enumeration was data-driven: every language present in both the dictionaries manifest and the models registry that lacked a module, ranked by speakers. ko and ne rank higher by speakers but need Hangul jamo and Devanagari tokenizers outside this batch.

2. Six national keyboard grids (Arabic 101, Persian ISIRI 9147, Hebrew SI-1452, Bulgarian BDS, Serbian Cyrillic, Croatian/Slovenian QWERTZ) plus five parameterized Latin family members. AVAILABLE_LANGUAGES grows 20 to 32. A real bug fixed on the way: the script-aware word regex never reached the ar/fa/he tokenizers, so Kotoshu.check extracted zero words from RTL documents — those tokenizers now declare their script sets (fa includes ZWNJ).

3. Forty-nine specimen pairs engine-verified across the twelve languages, plus RTL sentence round trips. Suite 3,887 examples, 0 failures.

GitHub Action v2 adds gem directory mode and baselines

2026-09-07T00:00:00.000Z

The files input now walks the whole repository tree, and baselines, include/exclude, and SARIF suppression notes ship in the wrapper that CI actually runs.

1. Version 1 expanded globs in bash and had no baseline input, even after the gem shipped directory mode and baselines in 0.8.0. Version 2 passes paths straight to kotoshu check, adds baseline, include, exclude, show_suppressed, and category inputs, and merges per-file SARIF runs so upload-sarif accepts the report.

2. A self-test workflow runs the action against a fixture repository with two misspellings and a committed baseline: baselined entries pass as notes, new errors fail, and gitignored files stay out. The action requires gem 0.8.0 or newer, and the v2 tag was cut by the owner.

The sweep index reaches Ruby in kotoshu 0.9.2

2026-09-07T00:00:00.000Z

Per-word sweep invariants now live in a lazily memoized per-dictionary index, which gives 2.2x on the English average and up to 6x on short words, with byte-identical outputs.

1. This release is the twin of the wasm 0.3.2 release: Kotoshu::Suggestions::SweepIndex memoizes character lengths, Soundex codes, and length buckets on the dictionary, built at the first sweep. The edit-distance strategy draws its length window from the buckets with word-list order restored so ranking tie order is untouched, the phonetic strategy compares memoized codes, and the n-gram gate reads indexed lengths; the strategies also stop copying the whole word list per sweep.

2. Warm benchmarks on full cached dictionaries under MRI 3.4.8 show the English average dropping from 1,472 to 661 ms per suggest (2.2x), the Spanish average from 2,591 to 1,396 ms (1.9x), and short words gaining up to 6.2x, with Teh at 185 ms, wrold at 656 ms, and gatoss at 1,249 ms. Mutation safety is explicit: every add_word, remove_word, clear, and merge path resets the memo, pinned by specs on real Hunspell and PlainText dictionaries.

3. Outputs byte-identical across 14 en+es combinations; the 2,630 conformance vectors untouched and replay green; suite 3,795 examples, 0 failures; rubocop clean over 447 files. Soundex now lives once in Algorithms::Soundex, shared by strategy and index.

The per-dictionary sweep index ships in @kotoshu/wasm 0.3.2

2026-09-07T00:00:00.000Z

Per-word sweep invariants now live in a per-dictionary index built once at the first sweep, so heavy dictionaries gain another 1.2 to 6x, Portuguese foremost.

1. The 0.3.1 rewrite still walked the whole word list per sweep, decoding each word before its gate — the length window, the Soundex code, the n-gram length bound — all values that never change with the query. 0.3.2 builds them once per dictionary: char lengths, Soundex codes packed into four bytes, and length buckets that let the edit-distance sweep touch only its two-length window (word-list order restored before the ranking sort, so tie order and outputs are untouched).

2. Benchmarks over the pinned dictionaries show the average per sweep dropping for Portuguese from 1,398 to 423 ms with the worst case from 1,931 to 656 ms, for Spanish from 522 to 89 ms, for Norwegian from 346 to 277 ms, for German from 184 to 148 ms, for Russian from 226 to 205 ms, for Italian from 144 to 127 ms, and for English from 52 to 45 ms. The one remaining heavy case is a short French word whose length window covers a dense slice of the French vocabulary, which is the floor for this algorithm as frozen by the conformance contract.

3. Conformance unchanged at 2,630 vectors, outputs byte-identical; every transposed-word typo in the benchmark still recovers its intended word in the top five.

The sweep fix reaches Ruby in kotoshu 0.9.1

2026-09-07T00:00:00.000Z

Pure-Ruby suggestions drop from tens of seconds to under a second on full dictionaries, and the Damerau sweep ships for gem users, so Teh suggests The in first place.

1. The patch release carries both suggestion-quality changes for gem users. The Damerau edit sweep charges transpositions one step, enumerates substitutions and insertions over the aff TRY string, and validates forms through affix-aware lookup, which puts Teh to The and definately to definitely in first place. The sweep-performance port builds one word index per generate instead of scanning the dictionary per keyboard variant, adds a length pre-gate ahead of the n-gram scan, computes Jaccard in one pass without the intermediate key-union array, and builds Soundex in reused buffers.

2. Benchmarked on full en_US (48,262 words, MRI 3.4.8, pure-Ruby backend): Teh 20.6 s to 0.59 s, mispellings 140.2 s to 0.75 s, recieve 36.2 s to 0.70 s, definately 55.7 s to 0.87 s, asdfghjkl 39.6 s to 0.86 s. Suggestion outputs are byte-identical — the 2,630 conformance vectors are untouched, and equivalence fuzz against the old algorithms (6,004 find-word inputs, 22,011 soundex inputs, 2,986 similarity pairs) found zero mismatches.

3. The dictionary data files that had ridden along under docs/verification-evidence are no longer packaged — the gem is 269 library files, no stray hunspell data. Suite 3,777 examples, 0 failures; rubocop clean.

Suggestion sweeps complete in under a second with @kotoshu/wasm 0.3.1

2026-09-07T00:00:00.000Z

The suggestion sweep is 10 to 200x faster on full dictionaries, returning the same suggestions byte for byte in milliseconds instead of seconds.

1. A full-en_US sweep had been taking seconds, with Teh at 3.6 s, recieve at 17.3 s, and mispellings at 38.4 s measured over the published 0.3.0 module, because four allocation and scan patterns compounded per dictionary word: find_word scanned the whole word list per keyboard variant with a lowercase allocation per word, the n-gram strategy built a map of String trigrams per dictionary word, Soundex allocated a String per word, and the edit loops allocated a fresh char vector per word.

2. The sweep now builds one word index per call (exact set plus lowercase-form map, borrowed for words already lowercase), packs trigrams into fixed-width integer keys in a reused scratch map with a length pre-gate, compares fixed Soundex keys, and reuses char buffers. Outputs are byte-identical: the 2630 conformance vectors all pass, and the benchmark set returns the same lists — Teh 320 ms, mispellings 189 ms, definately 145 ms, recieve 93 ms.

3. The playground also sweeps as one batch now: the misspellings pane sends a single message for all its words, the worker walks the queue with a yield between words so a popover click jumps ahead, and a fresh check replaces the pending batch instead of stacking behind it.

The Damerau sweep ranks the right word first in @kotoshu/wasm 0.3.0

2026-09-07T00:00:00.000Z

The Damerau edit sweep and model-generated candidates reach the browser: Teh now suggests The first, definately suggests definitely, and the playground shows every load and inference step.

1. The engine-side fix changes the suggestion sweep to enumerate adjacent transpositions at cost 1, substitutions and insertions over the aff TRY string, and deletions, each validated through affix-aware lookup so that dictionary forms like definite/IYVP surface with their surface spelling. The root causes fixed were that the sweep had only ever enumerated stems, so suffixed words like definitely could not appear at any distance; that the INITCAP form The was lookup-valid but charged case plus transposition as two steps; and that the phonetic and keyboard strategies carried private edit-distance copies that still charged swaps double. Acceptance on full en_US, byte-identical across both engines: Teh to The at rank 1, definately to definitely at rank 1, with recieve to receive and wrold to world unregressed.

2. semanticSuggest(model, word, k) joins the wasm surface: the int8 tier embeds an out-of-vocabulary word through its character n-grams and returns the nearest vocabulary words — candidate generation the edit-distance sweep cannot produce. The playground merges those neighbors into suggestion lists ahead of its context rerank, labeled with their cosine.

3. The playground itself grew instrumentation: every artifact download — engine, dictionary, the optional semantic tier — reports a phase-labeled byte bar, checking reports the busy dot and elapsed milliseconds, the misspellings pane sweeps with per-word spinners and a done-of-total count, and the suggestion popover closes with a sweep and rerank timing footer. 2630 conformance vectors regenerated for the new sweep, zero divergence between the Ruby and Rust engines.

kotoshu-server and kotoshu-lsp 0.1.1 republish with their real contents

2026-09-06T00:00:00.000Z

The two ecosystem gems republish with their actual contents after the empty 0.1.0 cuts: kotoshu-server brings semantic models over HTTP, and kotoshu-lsp brings a personal dictionary that un-flags words.

1. kotoshu-server 0.1.1 makes gem install kotoshu-server work. The 0.1.0 cut had been published with an empty file list because its gemspec collected files through git ls-files in a build environment without git. Version 0.1.1 ships the real server: /v1/check with the optional model flag providing cascade-guarded ONNX reranking memoized per language, /v1/languages reporting model availability, and boot-time opt-in through KOTOSHU_SERVER_MODEL_LANGS and KOTOSHU_SERVER_MODEL_TIER, bringing semantic quality to every SDK user with no implicit downloads.

2. kotoshu-lsp 0.1.1 reads the personal dictionary at ~/.config/kotoshu/personal.dic, reloads it when the file changes, and answers kotoshu.addToPersonalDictionary server-side, so adding a word republishes diagnostics in any LSP editor and the flag clears immediately. The first-diagnostic column bug (a −11 offset on didOpen) is fixed as well.

3. Both published through RubyGems trusted publishing — the OIDC exchange, the version bump, and the push all ran in CI with no stored credentials. The empty 0.1.0s stay listed: rubygems does not allow self-yanking versions older than 30 days; install unpinned and you get 0.1.1.

kotoshu 0.9.0 adds Norwegian and fixes dictionary corruption

2026-09-06T00:00:00.000Z

Norwegian Bokmål and the no alias join the twenty language modules, and the long-reported remove_word corruption bug is fixed on RubyGems.

1. Norwegian joins the module roster: registry v1.3.0 ships nb converted from fastText’s Bokmål-dominated cc.no while nn keeps cc.nn — and the gem now resolves both nb and the ISO macro-language no to the same Bokmål module, with å æ ø tokenizing as word characters through the shared Latin tokenizer.

2. The dictionary-mutation bugs first reported in PR #93 are fixed and released: remove_word deleted the wrong word after a prior removal, the PlainText length index drifted out of sync on add and remove, and the suggest hot path could download implicitly — it is now cache-only with offline mode enforced, and KOTOSHU_OFFLINE genuinely works through default configuration.

3. Also in the cut: NearestNeighbor accepts the full cosine range (in-vocabulary words no longer crash semantic analysis on rounding drift or true negatives), and the plan corpus — every TODO.impl record from 01 to 98 — is finally tracked in the repository.

kotoshu 0.8.0 ships the integrations wave

2026-09-06T00:00:00.000Z

The wave-1 program reaches RubyGems in one cut, bringing Unicode word detection for Greek and Cyrillic, directory-mode checking, baselines, inline ignores, and framework integrations for Rails, RSpec, Rake, and Jekyll.

1. kotoshu 0.8.0 puts the whole wave-1 program on RubyGems in one cut: the thirteen language modules and nineteen keyboard layouts, directory-mode checking, CI baselines, inline ignore directives, a pre-commit hook, and four framework integrations, with no new runtime dependencies anywhere. The release also repairs remote kotoshu setup for staged languages through a sublayout-first, flat-fallback strategy, lets kotoshu/tasks and kotoshu/jekyll load standalone, and compares baseline paths canonically.

2. Unicode word detection is the headline fix: word extraction had accepted only ASCII letters, so Greek and Ukrainian users could not check any text through the CLI. Extraction now follows the configured language’s tokenizer, using Greek for el and Cyrillic for uk with the in-word apostrophe kept (Мар’яна), and Latin capitals such as Å and Ä for Latin languages, while languages without a script tokenizer keep the historical behavior and the frozen conformance vectors are unchanged. Swedish dictionary loading is fixed alongside: COMPOUNDRULE )k uses ) itself as a flag character, and every flag is now escaped at compile time so the affix reader no longer raises.

3. Directory mode — kotoshu check DIR [DIR ...] — checks every file with a known text extension under --include / --exclude control, honoring .gitignore and .ignore through a standard glob subset (last-match wins, anchoring, nested scoping; files inside ignored directories cannot be re-included) and skipping hidden files plus .git, node_modules, vendor, and target by default. JSON and SARIF emit one combined document with per-file entries, --baseline applies per file, and single-file and stdin behavior is byte-identical to before.

4. Ignores and baselines arrive as a pair: kotoshu:disable-line, disable-next-line [WORDS], and the nestable disable-file / enable-file block are recognized in each format’s comment syntax, with suppressed words moving to suppressed_errors — listed by --show-suppressed and marked in result JSON and SARIF. kotoshu baseline init records existing debt count-based, so baselines survive reformatting; check --baseline lets covered errors pass, fails new ones, and reports stale entries as the debt shrinks.

5. The integrations are opt-in and dependency-free: an ActiveModel SpellingValidator that raises one validation error per misspelling with the top suggestion in the message (real ActiveModel standalone — Rails not required), RSpec matchers whose failures list each misspelling and its suggestions, a Rake task over repository text files, and a safe Jekyll generator that fails the build on new spelling errors while a .kotoshu-baseline.json in the site source keeps baselined debt from blocking builds. The pre-commit hook (id kotoshu, language: system) ships too — honestly documented as requiring Ruby and the gem on PATH.

6. Norwegian is not in this cut. The registry added nb at v1.3.0 today, but the gem-side no → nb alias and the Norwegian language module are next-cut work; 0.8.0’s thirteen new modules are the wave-1 thirteen and no more.

Semantic reranking in the browser with @kotoshu/wasm 0.2.0

2026-09-06T00:00:00.000Z

The wasm engine gains the semantic path with loadModel and rerank over the int8 tiers, and CORS-open mirrors put all 55 languages within reach of a browser tab.

1. loadModel(model_bytes, vocab_bytes) loads an int8-per-row embedding tier — mini ~3 MB, fluency ~15 MB — from ONNX bytes plus its .vocab.json sibling, returning a KotoshuModel that is the wasm twin of the gem’s ONNX provider, scored in pure Rust. rerank(model, word, context) returns the mean-cosine context score in [-1, 1] — 0.0 when the word or every token is out of vocabulary — so suggestion lists can be reordered in the browser the way the server does it. TypeScript declarations ship in the package.

2. The mirrors make that fetchable: registry mirror URLs send Access-Control-Allow-Origin: * while GitHub release assets send no CORS header at all, so a browser tab can resolve a model tier for any of the 55 languages and pull it without a server in between. Version 0.2.0 was built and signed by the kotoshu-rs release workflow with a provenance attestation and zero dependencies.

3. The Ruby API got its hosted reference the same morning: the YARD documentation is live at kotoshu.github.io/kotoshu/ — served from the site domain, deployed from the gem repository’s main branch — and already covers the 0.8.0 surfaces: model tiers and the confidence cascade, inline ignores, CI baselines, and the pre-commit hook.

Semantic models cover 55 languages; Norwegian Bokmål arrives

2026-09-06T00:00:00.000Z

The registry adds Norwegian Bokmål, with nb converted from fastText’s cc.no while nn keeps cc.nn, for a total of 55 languages and 165 tiered models behind the same eval gates.

1. Registry v1.3.0 ships nb × 3 tiers — mini 2.9 MiB, fluency 14.5 MiB, full 114.4 MiB — bringing the registry to 55 languages, 165 resources, and 7.1 GiB at ONNX opset 11, behind the same measured per-tier evals that gate every other language.

2. fastText publishes no cc.nb: the nb models are converted from cc.no and carry Bokmål provenance — models/nb/metadata.json records the source URL and sha256 of cc.no.300.vec — while nn ships separately from cc.nn, so both written standards of Norwegian are in the registry under their own codes.

3. The registry itself carries no no entries: the ISO macro-language code resolves to nb on the engine side, and that alias — with the Norwegian language module — is next-cut gem work, not part of 0.8.0.

4. That answers the no→nb/nn question this record left as an owner decision at v1.2.0, and it exhausts the convertible pool: what remains — fi sourcing and the nds resource-spec call — is owner decision, not pending work.

Semantic models cover 54 languages

2026-09-05T00:00:00.000Z

The models registry grows from 22 to 54 languages, with 32 new languages shipped including the first right-to-left languages, behind unchanged eval gates.

1. Registry v1.2.0 ships 32 new languages × 3 tiers for a total of 54 languages and 162 registry resources. Arabic, Persian, and Hebrew arrive as the registry’s first right-to-left languages, with new dictionaries verified against upstream license headers where none existed.

2. The gates did not move, with fluency held at rank correlation 0.97 and top-1 0.95 and mini at 0.90 and 0.85, never weakened, and the whole batch clears them comfortably, with fluency at 0.9999 to 1.0000 correlation and mini near-lossless. The keyboard-aware evaluation grew ten more curated national grids so the newcomer languages are judged on their own keyboards.

3. Not everything that converts ships. The nds language passed both gates but is ISO 639-2 only, while the registry’s language contract is two-letter codes; fi has no license-clear Hunspell source upstream; and fo, rw, ie, fur, and tlh have no fastText Common Crawl vectors at all. Every drop is recorded with its reason in the README.

4. The candidate pool is now exhausted: every convertible language either ships or is dropped for a stated reason, so the coverage backlog is empty. What remains, namely the no→nb/nn alias, fi sourcing, and the nds Resource Spec call, is an owner decision rather than pending work.

Nineteen languages reach full-feature support

2026-09-05T00:00:00.000Z

Thirteen language modules and a wave of keyboard layouts land in the gem, and full-feature support grows from six languages to nineteen.

1. Language modules for ca, cs, da, el, hu, it, nl, pl, ro, sv, tr, uk, and vi — thin compositions over the shared Latin tokenizer and normalizer, with real care where the script demands it: Turkish dotless-i case folding, Greek final-sigma and accent normalization, and Greek and Cyrillic script tokenizers that keep non-Latin words whole.

2. Keyboard layouts grow from five to nineteen: the Turkish-Q, Ukrainian-JCUKEN, and Greek-Phonetic national grids are mirrored from the models-repo eval harness with a drift spec guarding the mirror, and an eleven-member Latin family carries the real Nordic å/æ/ø and å/ä/ö keys over the qwerty and qwertz base grids.

3. The CLI learns directory mode — kotoshu check DIR walks trees, honors .gitignore and .ignore, and applies baselines per file — with baselines wired into CI so existing errors never block a build while new ones still fail it.

4. Framework integrations with no new runtime dependencies: a Rails/ActiveModel spelling validator, RSpec matchers that print each misspelling with its suggestions, a Rake task over repository text files, and a Jekyll generator that fails the build on new spelling errors.

Semantic models cover 22 languages

2026-09-05T00:00:00.000Z

The models registry grows from 9 to 22 languages, with thirteen new FastText reranking tiers shipped behind the same eval gates.

1. Registry v1.1.0 adds ca, cs, da, el, hu, it, nl, pl, ro, sv, tr, uk, and vi — every batch-one language whose dictionary exists — for a total of 22 languages × 3 tiers, 66 registry resources.

2. The gates did not move: every fluency tier holds rank correlation 0.9999 with top-1 of at least 0.958, every mini tier is near-lossless, and a language that fails its gates ships nothing.

3. The keyboard-aware eval grew with them — QWERTY/QWERTZ supplements plus curated Turkish-Q, Ukrainian ЙЦУКЕН, and Greek-phonetic noise grids — so newcomer languages are judged on their own keyboards.

4. The released registry.json is byte-identical to the committed one: local rebuilds reproduce the CI sha256 exactly, so future expansions are verifiable, not trusted.

Keyless publishing on every channel

2026-09-05T00:00:00.000Z

Releases now reach npm, RubyGems, and crates.io through CI-held OIDC trust — no long-lived tokens stored anywhere.

1. npm publishes @kotoshu/client and @kotoshu/wasm keyless: the release workflow exchanges GitHub’s OIDC identity for registry trust, and every artifact carries a provenance attestation.

2. crates.io trusted publishing (RFC 3691) is registered and verified for the kotoshu crate — the exchange was proven by a smoke dispatch before the first release. The first publish of a crate still needs a token; keyless covers 0.1.1 onward.

3. RubyGems is wired the same way — the gem’s release workflow plus the owner-side registration — so gem push never runs on a stored credential.

kotoshu 0.7.0 ships model tiers and the native extension

2026-09-05T00:00:00.000Z

The biggest gem release so far brings tiered models with a registry, the native extension, and a conformance-frozen engine.

1. The model tiers are full at about 120 MB, fluency at about 15 MB as the new default, and mini at about 3 MB, and they resolve through the models repository’s registry with SHA-256-verified primary, mirror, and vocab downloads. KOTOSHU_MODEL_TIER, setup --model --tier, and a tier-less legacy-cache bridge keep older caches loading.

2. The optional native extension compiles the Rust core into the gem, KOTOSHU_BACKEND=ruby|native|auto selects it, and the Ruby and Rust engines agree on all 2,630 frozen conformance vectors.

3. A confidence cascade skips the ONNX rerank when the composite strategies are already confident — semantic quality where it matters, plain speed where it does not.

4. Hunspell correctness: CHECKCOMPOUNDPATTERN replacement support, dot-split casing, INITCAP ngram-root skip, and AF alias flag parsing. The integrational suite runs 176 examples with zero failures.

The engine ships on npm and crates.io

2026-09-05T00:00:00.000Z

@kotoshu/client and @kotoshu/wasm are on npm and the kotoshu crate is on crates.io, all at 0.1.0.

1. @kotoshu/client 0.1.0 is the typed HTTP client for Node 18+, Deno, Bun, and browsers, covering check, suggest, detect, and correct.

2. @kotoshu/wasm 0.1.0 is the whole engine compiled to WebAssembly at 291 KiB gzipped; a consumer constructs a dictionary from .aff/.dic contents and checks words in-process and offline.

3. kotoshu 0.1.0 on crates.io is the Rust core itself, exposing Dictionary::load, correct, ranked suggest, the batch C ABI, and the models registry behind the resources feature.

Python packages ship on PyPI

2026-09-04T00:00:00.000Z

The Python HTTP client and the maturin wheel that embeds the Rust engine both ship at 0.1.0.

1. kotoshu 0.1.0 is the Python client, exposing Client.check, suggest, detect, and correct over the HTTP API with the same result types the other SDKs use.

2. kotoshu-native 0.1.0 is a maturin-built wheel embedding the Rust engine. KOTOSHU_BACKEND=native|http|auto picks the backend, and word-level checking runs fully offline with no server involved.

3. Result handling is identical across backends: the same Suggestion rows come back whether the engine runs in-process or over the wire.

Engine correctness wave: compound semantics and green conformance

2026-09-04T00:00:00.000Z

CHECKCOMPOUNDPATTERN replacement semantics are ported, and all 2,630 conformance vectors pass in both the Ruby and the Rust engine.

1. The gem’s CHECKCOMPOUNDPATTERN replacement semantics, the subtlest rule in the compound family, are ported to the Rust core, with the conformance vector pack regenerated from the gem’s engine.

2. All 2,630 frozen vectors, comprising 1,315 correct and 1,315 suggest cases, pass in the Ruby engine, the Rust engine, and over the C ABI. Both sides of the conformance suite run in CI, so behavior cannot drift.

Models registry v1.0.1 serves nine languages in three tiers

2026-09-03T00:00:00.000Z

FastText-ONNX embedding models for nine languages ship in full, fluency, and mini tiers.

1. One registry, addressed as kotoshu://models/{lang}/{tier}, resolves every model with SHA-256 checksums, license, and minimum engine version; clients try the primary URL, then the mirror, then the vocab.

2. Per-tier evaluation reports are published for each language. fluency is int8 with the top 50,000 words and near-lossless, while mini is the wasm and edge tier at about 3 MB.

3. The full tier at about 120 MB also serves from a media-host mirror, so the first setup does not lean on release bandwidth.

kotoshu.org goes live

2026-09-01T00:00:00.000Z

The ecosystem site opens as a catalog of the project, its languages, and its documentation.

1. The site carries one page per audience, one page per language, seven documentation pages, and a playground that runs the real engine against any kotoshu-server.

2. The design treats the site itself as a dictionary, with headwords, part-of-speech lines, and numbered senses throughout.

The ecosystem around the gem: LSP, server, Go client, and Action

2026-08-01T00:00:00.000Z

kotoshu-lsp, kotoshu-server, kotoshu-go, and action-kotoshu all reach their first releases.

1. kotoshu-lsp provides diagnostics, quick-fixes, and hover suggestions in any editor that speaks LSP.

2. kotoshu-server provides seven JSON endpoints over Rack, Sinatra, and Puma, forming the deployment surface for every SDK.

3. kotoshu-go and action-kotoshu provide a Go client with context-aware methods and a GitHub Action that lands SARIF in the Security tab.