Generative AI and Child Safety: A Convergence of Innovation and Exploitation

In June 2025, Australia’s eSafety Commissioner published a sobering analysis as part of their “Converge” blog series, which examines the intersection of emerging technologies and online harms. This edition focuses on a disturbing reality: generative AI is being weaponized to create child sexual exploitation and abuse (CSEA) material at an alarming and accelerating rate.

The statistics are stark. The blog reveals that in 2024, the National Centre for Missing and Exploited Children reported an 1,325% increase in reports involving AI-generated CSEA material—jumping from 4,700 reports in 2023 to 67,000 reports in 2024. Australia’s eSafety investigators noted a 218% increase in reports of AI-generated CSEA material from 2023 to 2024 alone.

But what makes this analysis particularly valuable is that it goes beyond documenting the problem. It outlines Australia’s regulatory framework for addressing AI-generated harm, emphasizes the critical role of Safety by Design principles, and details specific steps the tech industry must take to prevent the weaponization of their products.

As the blog states plainly: “While responsibility must primarily sit with those who choose to perpetrate abuse, we cannot ignore how technology is weaponised. The tech industry must take responsibility to address the weaponisation of their products and platforms.”

The Unseen Threat: AI-Generated CSEA Is Already Here

The blog begins by dispelling a dangerous notion: that AI-generated CSEA material represents a future threat we need to prepare for. The reality is more urgent.

Not Hypothetical—Present and Growing

“To understand the gravity of the situation, we must first recognise the misuse of AI for generating CSEA material is not a hypothetical risk – it is a present and growing threat.”

The numbers tell the story:

National Centre for Missing and Exploited Children (NCMEC):

2023: 4,700 reports of AI-generated CSEA material
2024: 67,000 reports of AI-generated CSEA material
Increase: 1,325%

eSafety Australia Investigators:

2023 to 2024: 218% increase in reports of AI-generated CSEA material

This exponential growth reflects both the increasing availability of generative AI tools and the growing willingness of perpetrators to weaponize them.

The “No Real Child” Misconception

The blog confronts a dangerous misconception head-on: “that AI-generated CSEA material isn’t harmful because no ‘real’ child was involved.”

This misconception is not only false—it’s dangerous. The harm is real in multiple ways:

Digital Permanence and Psychological Trauma “Once created, this content can exist forever in the digital world, constantly resurfacing and causing psychological trauma to victims who may never even be aware of its existence until it’s too late.”

Re-traumatization of Survivors “For survivors of abuse, seeing synthetic versions of their suffering can reignite trauma and deepen their victimisation.”

When AI-generated CSEA is based on real images of victim-survivors—their faces manipulated into new abusive contexts—the harm is direct and ongoing. Each new synthetic image extends their victimization.

Cultural Normalization “Synthetic versions of CSEA also contribute to a culture that diminishes the seriousness of CSEA and its harmful impacts.”

The proliferation of AI-generated CSEA material risks normalizing the sexual exploitation of children, desensitizing viewers, and creating a culture where such content is treated as less serious because it’s “just AI.”

A Double-Edged Sword in the Fight Against CSEA

One of the most nuanced aspects of the blog is its recognition that AI technology itself is not inherently harmful. The same technology that can be weaponized to create CSEA material can also be leveraged to detect and remove it.

AI for Detection and Protection

The blog highlights several positive applications:

Identifying and Removing CSEA Material “AI technology is used to identify new CSEA material, supporting platforms in removing this material, and aiding law enforcement by triaging suspected CSEA material for human review.”

Machine learning models can flag harmful content faster than human investigators, potentially enabling early intervention.

Educational Nudges AI systems can prompt users with educational messages when they attempt harmful searches, potentially deterring exploitation before it occurs.

Ethical Dataset Development: “My Pictures Matter” The blog highlights “My Pictures Matter,” a crowdsourcing initiative led by AiLECS Lab (AI for Law Enforcement and Community Safety), a collaboration between the Australian Federal Police and Monash University.

This initiative creates an ethically sourced dataset of “safe” childhood photos for machine learning research to counter CSEA. To qualify as a “safe” image, it must:

Be collected with full consent
Not contain child nudity
Not depict illegal activity, violence, or abuse

These crowdsourced images train algorithms to recognize “safe” images of children, which can then inform technologies that assess whether digital files contain “unsafe” imagery.

AI for Exploitation

But the blog doesn’t shy away from the uncomfortable reality:

“While we should recognise these advances, we must also confront the uncomfortable reality: the same technology that can protect children is being weaponised by perpetrators to exploit them.”

Perpetrators are using AI to:

Create hyper-realistic synthetic images of child abuse using completely synthetic models of children
Manipulate existing images of victim-survivors, placing their faces or bodies into new abusive contexts
Generate convincing content that is increasingly difficult to distinguish from authentic CSEA material

The accessibility and sophistication of these tools are both increasing rapidly.

A Growing Crisis: Challenges in Policing AI-Generated Abuse

The blog outlines several critical challenges that AI-generated CSEA material creates for regulators, law enforcement, and child protection agencies.

The Proliferation of “Nudify” Apps

“Generative AI tools that can be used to produce explicit content – such as ‘nudify’ apps that digitally remove clothing – are proliferating online. They are widely available, and quick and easy to use. Many are also free.”

These tools allow perpetrators to take innocuous photos of children and manipulate them to generate explicit material. The ease of use and widespread availability create a dangerous environment where:

Innocuous Photos Become Weapons “AI’s ability to create detailed and authentic-looking images means even innocuous photos of children can be manipulated to generate explicit material.”

A family photo posted on social media can be scraped and manipulated without the knowledge or consent of the child or their family.

Children Exploiting Peers “The accessibility of such platforms has also made it easier for children to create inappropriate or harmful material of their peers.”

The blog notes that children themselves are using these tools to create explicit images of classmates, creating new forms of peer harassment and exploitation.

Extortion and Exploitation “Synthetic images of children are being used for extortion and exploitation.”

AI-generated images are being used as tools for sextortion, with perpetrators threatening to distribute synthetic explicit images unless victims comply with demands.

Resource Misdirection: The Synthetic vs. Real Challenge

One of the most troubling challenges the blog identifies is the difficulty in determining whether content is AI-generated, AI-modified, or authentic.

Why This Matters: “This is incredibly important, because if it is authentic, it may depict a child in need of rescuing.”

Law enforcement resources are finite. If investigators are spending time trying to identify and rescue synthetic children, they may be missing opportunities to rescue real children currently being abused.

Visually Indistinguishable Content “According to the Internet Watch Foundation, some AI-generated CSEA material is now ‘visually indistinguishable’ from authentic child abuse content.”

As AI technology improves, the ability to distinguish synthetic from real becomes increasingly difficult, even for trained investigators.

Intentional Obfuscation “Research by the Stanford Cyber Policy Center also found concerning anecdotal evidence that some perpetrators intentionally add flaws to non-synthetic CSEA material, in the hopes that law enforcement might misdirect their efforts by believing the material is AI-generated and purely synthetic.”

This represents a particularly insidious development: perpetrators are deliberately trying to make authentic CSEA material look AI-generated, hoping investigators will deprioritize it as not depicting a real child in danger.

Australia’s Regulatory Framework: The Online Safety Act 2021

Unlike many jurisdictions still grappling with how to address AI-generated harms, Australia has already established a regulatory framework with teeth.

The Online Safety Act 2021 (OSA)

The blog explains that the OSA empowers eSafety to take concrete action:

Investigation and Removal Authority “The Online Safety Act 2021 (OSA) empowers eSafety to investigate and remove harmful material, including AI-generated CSEA, under defined categories of illegal and restricted content.”

This means eSafety can take direct action when AI-generated CSEA material is identified, not just issue recommendations.

Transparency Notices “The OSA also gives eSafety the authority to issue legal transparency notices under the Basic Online Safety Expectations (BOSE, or the Expectations).”

The Expectations include foundational steps that service providers are expected to take to ensure safety for their users, including:

Safe use of certain features like generative AI
Protections for encrypted services
Accountability for anonymous accounts
Responsible recommender systems

Published Transparency Reports “To date, eSafety has published five transparency reports. In some cases, these transparency reports have revealed that service providers are not doing enough.”

This public accountability mechanism allows eSafety to shine a light on companies that are failing to meet their safety obligations.

Industry Codes and Standards

The OSA provides for industry codes and standards which set out mandatory requirements on key services, including those in the generative AI ecosystem.

Six Industry Codes Apply To:

Social media services
App distribution services
Hosting services
Internet carriage services
Equipment providers
Search engine services

Two Industry Standards: Two industry standards drafted by eSafety came into effect on 22 December 2024 in relation to unlawful and seriously harmful material such as CSEA.

These standards include:

A range of obligations on services to detect and remove CSEA
Specific obligations on generative AI services that pose a risk of generating this material (such as nudify services)
Obligations on model distribution platforms that enable access to the underlying models

Enforcement Powers

Critically, the blog emphasizes that these are not voluntary guidelines:

“eSafety can assess and investigate a service provider’s compliance with the relevant standards and is empowered with a range of enforcement options in cases of non-compliance.”

While eSafety announced that enforcement would begin after June 2025 (following an implementation phase for industry), the message is clear: compliance is mandatory, and companies that fail to meet their obligations will face consequences.

“Where necessary, eSafety will use the full range of its enforcement powers to ensure compliance.”

Safety by Design: A Proactive Framework

At the heart of eSafety’s approach is the Safety by Design initiative, which puts user protection at the heart of all stages of the product and service lifecycle.

The Core Principle

Rather than treating safety as an afterthought or a bolt-on feature, Safety by Design requires companies to integrate child safety measures at every stage of the AI lifecycle.

As the blog states: “By integrating child safety measures at every stage of the AI lifecycle, we can significantly reduce the risk of abuse.”

Critical Steps for Companies

The blog outlines five critical steps that technology companies must take, noting that these align with obligations under the Industry Codes and Standards under the OSA:

1. Responsible Data Curation

The Requirement: “Companies should make sure training datasets are free of CSEA material.”

Why It Matters: “A study by the Stanford Internet Observatory revealed that some AI platforms had used a dataset containing known CSEA content, underscoring the need for rigorous data vetting.”

If generative AI models are trained on datasets that include CSEA material, they become more capable of generating similar content. This contamination of training data must be prevented.

Key Practice: “Content relating to children must be separated from adult sexual content in training data to limit the ability for models to generate CSEA material.”

This separation is crucial—if models learn associations between images of children and sexual content during training, they become more likely to generate inappropriate combinations when prompted.

2. Content Transparency Mechanisms Including Labelling and Source Watermarking

The Requirement: “Embedding visible or invisible markers into AI-assisted, AI-enhanced or AI-generated content can help law enforcement differentiate between synthetic and real CSEA material, enabling more effective resource allocation.”

Why It Matters: This directly addresses the resource misdirection challenge outlined earlier. If AI-generated content is clearly marked as synthetic, investigators can prioritize their efforts toward identifying and rescuing real children.

Implementation: Watermarking can be:

Visible: Clear markers that content is AI-generated
Invisible: Embedded metadata or steganographic markers that can be detected by specialized tools but don’t alter the visual appearance

3. Strict and Enforceable Usage Policies

The Requirement: “Platforms must prohibit the generation of CSEA content and enforce these rules.”

Implementation: This includes:

Providing clear user reporting mechanisms
Implementing AI tools that detect and prevent the creation of harmful material
Actually enforcing policies (not just having them on paper)

Why It Matters: Usage policies only work if they’re enforced. Too often, companies have policies prohibiting harmful content but fail to invest in the systems and personnel needed to actually enforce those policies.

4. Transparency and Accountability

The Requirement: “AI developers should work closely with regulators, sharing insights into how their models operate and what safeguards are in place to prevent misuse.”

Stakeholders: This transparency should extend to:

Regulators
Users
Researchers

Why It Matters: Without transparency, it’s impossible to assess whether companies are meeting their safety obligations. Regulators need to understand how models work to identify potential vulnerabilities, and researchers need access to information to develop better safety measures.

5. Reviewing and Assessing Products Regularly

The Requirement: “Addressing the risk of CSEA on a platform needs to be an ongoing and evolving process – not a check box review.”

Why It Matters: AI technology is evolving rapidly. A model that was safe six months ago may have new vulnerabilities today due to changes in how it’s used, new jailbreaking techniques, or integration with other systems.

Safety assessment cannot be a one-time exercise—it must be continuous and adaptive.

Alignment with Regulatory Obligations

The blog notes that these five steps are “closely aligned with specific obligations applying to high impact generative AI services under the ‘Designated Internet Services’ Standard” and “highly relevant to services’ expectations under the Basic Online Safety Expectations.”

In other words, these aren’t just best practices—they’re regulatory requirements for companies operating in Australia.

The Whole-of-Ecosystem Approach

One of the most important aspects of the blog is its recognition that addressing AI-generated CSEA requires action across the entire digital ecosystem, not just from AI developers.

Different Actors, Different Responsibilities

Search Engine Providers: “Search engine providers should de-index platforms which have been explicitly created to generate CSEA material from appearing in their search results.”

If nudify apps and CSEA generation platforms are easily discoverable through search engines, their availability increases dramatically.

App Stores: “App stores should remove apps explicitly created to nudify.”

Apple’s App Store and Google Play Store serve as gatekeepers for mobile applications. They must take responsibility for ensuring harmful apps are not distributed through their platforms.

Social Media and Messaging Platforms: These platforms should:

Prohibit AI-generated CSEA on their services
Detect and report such content
Remove it promptly
Enforce their terms in relation to preventing the advertising of nudify services

Even if a platform doesn’t create AI-generated CSEA, if it allows such content to be shared or allows nudify services to be advertised, it becomes part of the exploitation ecosystem.

Why This Matters

“The sheer scale and complexity of this issue demands a holistic, collaborative approach.”

No single actor can solve this problem alone. A nudify app blocked from one app store but available through direct download, easily found through search engines, and with content shared across social media platforms will still cause significant harm.

Effective protection requires coordinated action across the entire ecosystem.

Ensuring a Safer Future: Education and Empowerment

The blog emphasizes that regulation and enforcement alone are not sufficient. A comprehensive approach must also include education and empowerment.

Strength-Based Approach

“Promoting a strength-based approach is critical to this work. This must be underpinned by education to support the development of critical digital literacy, including AI literacy.”

Rather than focusing solely on restrictions and prohibitions, effective child safety also requires:

Teaching children about AI technology and how it works
Developing critical thinking skills to evaluate AI-generated content
Empowering young people to recognize and report harmful uses
Building resilience and support systems

Elevating Children’s Voices

“Equally, we must elevate voices of children and young people in these discussions and embed lived experiences in policy.”

Too often, policy about children is developed without input from children themselves. Effective policy requires understanding:

How young people actually use technology
What harms they’re experiencing
What protections they find helpful vs. restrictive
How policy impacts their ability to learn, create, and connect online

Multi-Stakeholder Collaboration

The blog emphasizes that achieving a safer digital future requires a united effort from:

Companies that develop and deploy AI
Regulators
Policymakers
Academia
Law enforcement
Educators
Parents and carers
Children and young people themselves

Looking Ahead: More Sophisticated Exploitation

The blog doesn’t shy away from uncomfortable predictions about the future:

“Looking ahead, we must expect more sophisticated forms of exploitation, including a combination of generated audio and video, becoming more common.”

As AI technology advances, we can expect:

More realistic synthetic content that is harder to distinguish from authentic material
Multi-modal exploitation combining generated images, video, and audio
Personalized targeting using AI to create synthetic content featuring specific children
Greater accessibility as tools become easier to use and more widely available

But Also Hope

“But with foresight and concerted action, we can shape a safer digital world – one where innovation and responsibility go hand in hand.”

The message is clear: while the challenges are significant and growing, they are not insurmountable. With the right combination of:

Regulatory frameworks with enforcement power
Industry accountability and Safety by Design principles
Whole-of-ecosystem collaboration
Education and empowerment
Continuous adaptation to emerging threats

We can create a digital environment where children can benefit from AI innovation while being protected from exploitation.

The Time to Act Is Now

The blog concludes with an urgent call to action:

“The rise of generative AI presents both an extraordinary opportunity and an unprecedented risk. If left unchecked, the misuse of this technology will continue to have devastating consequences for children.”

The statistics speak for themselves—a 1,325% increase in reports of AI-generated CSEA in a single year is not a trend we can afford to ignore or address slowly.

“Whether you’re at the forefront of AI innovation, enforcing policy, or safeguarding children, the message is clear: we must not wait for the next crisis to spur us into action.”

How This Connects to CAIROS AI’s Mission

Australia’s eSafety Commissioner blog reinforces several principles that are central to our work:

Safety by Design Requires Proactive Testing

The emphasis on integrating child safety measures “at every stage of the AI lifecycle” aligns with our approach of proactive red-teaming before deployment. You cannot ensure safety if you only assess it after a product is already in children’s hands.

Specialized Expertise Is Essential

The blog’s recognition that individual actors are ill-equipped to assess AI tools for child safety underscores the need for specialized expertise. Red-teaming AI systems for child safety vulnerabilities requires:

Understanding of how perpetrators exploit technology
Knowledge of child development and age-appropriate content
Technical expertise in AI systems and jailbreaking techniques
Familiarity with regulatory requirements across jurisdictions

Transparency and Accountability Demand Documentation

eSafety’s transparency notices and enforcement powers highlight the need for documented, rigorous evaluation. Companies must be able to demonstrate—not just claim—that they’ve taken steps to prevent weaponization of their products.

Regulatory Compliance Requires Continuous Assessment

The blog’s emphasis that “addressing the risk of CSEA on a platform needs to be an ongoing and evolving process – not a check box review” aligns with our understanding that child safety assessment must be continuous and adaptive.

Third-Party Validation Is Necessary

The call for transparency and accountability to regulators, along with mandatory standards and enforcement mechanisms, recognizes that self-assessment is insufficient when children’s safety is at stake.

A Model for Global Action

Australia’s approach—combining clear regulatory authority, mandatory industry standards, transparency requirements, and enforcement powers—provides a potential model for other jurisdictions grappling with how to address AI-generated child exploitation.

The key elements:

Clear legal authority to investigate and remove harmful content
Mandatory standards (not voluntary guidelines) for industry
Whole-of-ecosystem accountability involving all actors in the digital supply chain
Safety by Design principles embedded throughout the product lifecycle
Transparency mechanisms including public reporting
Enforcement powers with real consequences for non-compliance

Combined with education, empowerment, and multi-stakeholder collaboration, this comprehensive approach offers a pathway toward ensuring that AI innovation and child safety can coexist.

As eSafety concludes: “With collaboration, transparency and accountability, and by embedding safety into every layer of AI’s development, we can harness its potential to create a safer digital future.”

Read the full blog: Generative AI and child safety: A convergence of innovation and exploitation – eSafety Commissioner, Australia

CAIROS AI provides the specialized red-teaming and assessment capabilities that Safety by Design frameworks like eSafety’s require. Our expert-led testing helps organizations identify child safety vulnerabilities before deployment, implement effective safeguards throughout the AI lifecycle, and demonstrate compliance with emerging regulatory requirements like Australia’s Online Safety Act standards.