llama_cpp_for_radxa_dragon_.../examples
staviq 1a159553f9
tokenizer : special token handling (#3538)
* Rewrite special token handling from #1931

* shorten param name, add st verification by type

* use offsets instead of copy by substr

* formatting, remove copying iterator on delete

* llama : normalize code-style

* swift fix

* print pfx/sfx if verb, main: split pfx input sfx

* dont add space when using special tokens

* minor : comment + spacing

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-10-17 18:11:01 +03:00
..
baby-llama
batched
batched-bench
batched.swift
beam-search
benchmark
convert-llama2c-to-ggml
embd-input
embedding
export-lora
finetune
gguf
gptneox-wip
infill
jeopardy
llama-bench
llava
main
main-cmake-pkg
metal
parallel
perplexity
quantize
quantize-stats
save-load-state
server
simple
speculative
train-text-from-scratch
alpaca.sh
chat-13B.bat
chat-13B.sh
chat-persistent.sh
chat-vicuna.sh
chat.sh
CMakeLists.txt
gpt4all.sh
json-schema-to-grammar.py
llama.vim
llama2-13b.sh
llama2.sh
llm.vim
make-ggml.py
Miku.sh
reason-act.sh
server-llama2-13B.sh