The testing process involves sending numerous prompts (up to 130 for some bias categories) to the LLMs and evaluating their responses for sensitive words or unexpected unethical responses. The LLM's score is determined by the percentage of tests it passes, indicating its level of bias.
Key takeaways:
- The LLM observatory uses a scientific approach to test biases in LLMs.
- It utilizes LangBiTe, an open-source framework that includes a library of prompts to test various biases such as lgtbiq+phobia, ageism, misogyny/misandry, political bias, racism, religious discrimination, and xenophobia.
- The testing process involves sending many prompts (up to 130 for some bias categories) to the LLMs and evaluating their responses for sensitive words or unexpected unethical responses.
- The score of the LLM is determined by the percentage of tests it passes.