The article also warns of "availability cascades," a phenomenon where irrational risk assessments become self-reinforcing due to viral hysteria. This is particularly relevant to X-risk, which is susceptible to this process due to the ease with which the public can imagine it, aided by decades of popular science fiction. The author concludes by emphasizing the importance of an unbiased appraisal of X-risk in AI alignment, advocating for a collective immunity against factors that jeopardize rational thought, stifle progress, or seek to assign power to an "elite" class.
Key takeaways:
- The Messiah Bias is a cognitive bias that can threaten the field of AI Alignment, where individuals or organizations make high estimates of X-risk to play grandiose roles such as having crucial knowledge, dictating actions and access to resources, and playing the savior.
- Identifying this bias involves looking for signs of extreme X-risk estimates and the desire to realize grandiose roles, as well as irrational reasoning and appeals to probability and authority.
- Availability Cascades, where irrational risk assessments become self-reinforcing due to viral hysteria, can also contribute to the overestimation of X-risk. This is often fueled by "availability entrepreneurs" who profit from or have an agenda related to the fear.
- AI Alignment should be approached with an unbiased appraisal of X-risk, and efforts should be made to resist factors that jeopardize rational thought, stifle progress, or concentrate power in an "elite" class.