The news highlights the ethical concerns technology companies face when adopting generative AI tools. OctoML was aware of the nature of the content Civitai was generating and chose to continue profiting from the relationship. The company uses AWS to provide services to its clients, meaning the nonconsensual AI generated images Civitai users are making are being generated on Amazon servers, generating revenue for Amazon. The legality of AI-generated child sexual abuse material (CSAM) has not been broadly tested in American courts.
Key takeaways:
- Seattle-based startup OctoML, which helps companies optimize and deploy their machine learning models, continued to work with AI model sharing and image generating platform Civitai despite internal concerns that the platform was generating content that could be categorized as child pornography.
- Internal Slack messages and documents viewed by 404 Media revealed that OctoML co-founder Thierry Moreau had expressed concerns about the ethical and legal risks of generating images for Civitai, which is backed by venture capital firm Andreessen Horowitz.
- Despite these concerns, OctoML decided to continue its relationship with Civitai, but chose not to advertise the partnership as it had previously done. The company has raised $132 million in funding and is an AWS partner, meaning the images were generated on Amazon servers.
- The case highlights the ethical challenges faced by technology companies as they adopt generative AI tools, particularly in relation to the generation of synthetic pornography and nonconsensual images of real people.