Piotr Wilkin (ilintar) 0ec191e1d7 vocab: add gemma4 tokenizer tests, fix edge case (#21534) ... * YATF (Yet Another Tokenizer Fix) for Gemma 4. With tests! * Remove unnecessary hash from update script. * minor: move constant		2026-04-09 11:41:14 +02:00
..
templates	common : add gemma 4 specialized parser (#21418)	2026-04-04 20:39:00 +02:00
.editorconfig
ggml-vocab-aquila.gguf
ggml-vocab-baichuan.gguf
ggml-vocab-bert-bge.gguf
ggml-vocab-bert-bge.gguf.inp	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-bert-bge.gguf.out	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-command-r.gguf
ggml-vocab-command-r.gguf.inp	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-command-r.gguf.out	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-deepseek-coder.gguf
ggml-vocab-deepseek-coder.gguf.inp	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-deepseek-coder.gguf.out	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-deepseek-llm.gguf
ggml-vocab-deepseek-llm.gguf.inp	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-deepseek-llm.gguf.out	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-falcon.gguf
ggml-vocab-falcon.gguf.inp	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-falcon.gguf.out	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-gemma-4.gguf	vocab: add gemma4 tokenizer tests, fix edge case (#21534)	2026-04-09 11:41:14 +02:00
ggml-vocab-gemma-4.gguf.inp	vocab: add gemma4 tokenizer tests, fix edge case (#21534)	2026-04-09 11:41:14 +02:00
ggml-vocab-gemma-4.gguf.out	vocab: add gemma4 tokenizer tests, fix edge case (#21534)	2026-04-09 11:41:14 +02:00
ggml-vocab-gpt-2.gguf
ggml-vocab-gpt-2.gguf.inp	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-gpt-2.gguf.out	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-gpt-neox.gguf
ggml-vocab-llama-bpe.gguf
ggml-vocab-llama-bpe.gguf.inp	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-llama-bpe.gguf.out	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-llama-spm.gguf
ggml-vocab-llama-spm.gguf.inp	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-llama-spm.gguf.out	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-mpt.gguf
ggml-vocab-mpt.gguf.inp	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-mpt.gguf.out	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-nomic-bert-moe.gguf	tests : improve UGM tokenizer test coverage (#13773)	2025-05-25 16:22:29 +02:00
ggml-vocab-phi-3.gguf
ggml-vocab-phi-3.gguf.inp	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-phi-3.gguf.out	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-qwen2.gguf
ggml-vocab-qwen2.gguf.inp	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-qwen2.gguf.out	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-refact.gguf
ggml-vocab-refact.gguf.inp	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-refact.gguf.out	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-starcoder.gguf
ggml-vocab-starcoder.gguf.inp	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00
ggml-vocab-starcoder.gguf.out	convert : allow partial update to the chkhsh pre-tokenizer list (#13847)	2025-05-30 12:24:37 +02:00