llama : custom attention mask + parallel decoding + no context swaps by ggerganov · Pull Request #3228

The author discusses a workaround for an issue with the `./bin/parallel` external alternative-prompt file option not working with `-f file.txt`. They inserted a few lines into `parallel.cpp` to make it run, despite not being a `C/C++` specialist. The solution involves creating a plain text file with questions, which is read by `common.cpp` and used to replace the default `k_prompts` inside `parallel.cpp`. The author provides a sample output and the additional lines of code they used.

The additional code includes a function to split strings and a section inside `main()` that loads prompts from an external file if any are available. If no new questions are provided, the program proceeds with built-in defaults. If new prompts are provided, they are outputted and copied to `k_prompts`. The author also includes diagnostic non-functional code to output each line of the updated `k_prompts` vector. The solution was tested on an M2 MAX 32GB and MacOS Sonoma 14.0.

Key takeaways:

The author has made modifications to the `parallel.cpp` file to allow it to read prompts from an external file, as the `./bin/parallel` external alternative-prompt file option was not working.
The prompts are read from a plain text file and transferred to replace the default `k_prompts` inside `parallel.cpp`.
A sample output is provided to demonstrate the functionality of the modified code, showing the prompts being loaded and the responses generated.
The author has also included the additional lines of code used to split the string and load the prompts from the external file.

llama : custom attention mask + parallel decoding + no context swaps by ggerganov · Pull Request #3228 · ggerganov/llama.cpp

Key takeaways:

Comments (0)

Newsletter