goerch b08e75baea Fixing the last deviations from sentencepiece indicated by test-tokenizer-1 (#3170) ... * Fix für #2721 * Reenable tokenizer test for LLaMa * Add `console.cpp` dependency * Fix dependency to `common` * Fixing wrong fix. * Make console usage platform specific Work on compiler warnings. * Adapting makefile * Remove trailing whitespace * Adapting the other parts of the makefile * Fix typo. * Fixing the last deviations from sentencepiece indicated by test-tokenizer-1 * Simplify logic * Add missing change... * Fix ugly compiler warning * llama_tokenize should accept strings containing NUL now * Adding huichen's test case		2023-09-16 13:41:33 +02:00
..
CMakeLists.txt
test-c.c
test-double-float.cpp
test-grad0.cpp
test-grammar-parser.cpp
test-llama-grammar.cpp
test-opt.cpp	check C++ code with -Wmissing-declarations (#3184)	2023-09-15 15:38:27 -04:00
test-quantize-fns.cpp	check C++ code with -Wmissing-declarations (#3184)	2023-09-15 15:38:27 -04:00
test-quantize-perf.cpp	check C++ code with -Wmissing-declarations (#3184)	2023-09-15 15:38:27 -04:00
test-sampling.cpp	check C++ code with -Wmissing-declarations (#3184)	2023-09-15 15:38:27 -04:00
test-tokenizer-0-falcon.cpp
test-tokenizer-0-falcon.py
test-tokenizer-0-llama.cpp	Fixing the last deviations from sentencepiece indicated by test-tokenizer-1 (#3170)	2023-09-16 13:41:33 +02:00
test-tokenizer-0-llama.py
test-tokenizer-1-llama.cpp	Fixing the last deviations from sentencepiece indicated by test-tokenizer-1 (#3170)	2023-09-16 13:41:33 +02:00