Toxicity Detection Block

The Toxicity Detection Block leverages pre-trained machine learning models to score text for toxicity. It is highly useful for moderating user-generated content before saving it to a database or displaying it to other users.

Configuration

Output Variable

Specify a Variable Name to store the analysis results.

Success Payload

The output is an object containing scores for various toxicity categories (from 0.0 to 1.0):

{
  "toxic": 0.92,
  "severe_toxic": 0.01,
  "obscene": 0.85,
  "threat": 0.0,
  "insult": 0.70,
  "identity_hate": 0.0
}

Errors & Responses

If an error occurs, the variable will be populated with:

{
  "error_key": "INPUT_MISSING",
  "error_message": "Text to analyze is required."
}

Common Error Keys: INPUT_MISSING, RUNTIME_ERROR

You can use the scores in a Compare Block to reject the request if the content violates your community guidelines (e.g. toxic > 0.8).