Scenarios API

The Scenarios API allows you to create, run, and manage test scenarios for your tool collections.

List Scenarios

GET /api/scenarios

Retrieve a paginated list of public scenarios with optional filtering.

Query Parameters

limit - Number of results (default: 20, max: 50)
offset - Pagination offset
collectionId - Filter by collection ID
tags - Filter by tags (comma-separated)
sortBy - Sort field: qualityScore, totalRuns, createdAt, lastRunAt

Example Request

curl "https://tpmjs.com/api/scenarios?limit=10&sortBy=qualityScore"

Example Response

{
  "data": [
    {
      "id": "clu123abc456",
      "collectionId": "clx789def",
      "prompt": "Scrape the main article content from a news website...",
      "name": "Scrape article content",
      "description": "Tests web scraping capability on news sites",
      "tags": ["web-scraping", "content-extraction"],
      "qualityScore": 0.85,
      "totalRuns": 12,
      "lastRunAt": "2024-01-20T10:30:00Z",
      "lastRunStatus": "pass",
      "consecutivePasses": 5,
      "consecutiveFails": 0,
      "createdAt": "2024-01-01T00:00:00Z",
      "collection": {
        "id": "clx789def",
        "name": "Web Scraping Toolkit",
        "slug": "web-scraping-toolkit",
        "username": "johndoe"
      }
    }
  ],
  "pagination": {
    "limit": 10,
    "offset": 0,
    "hasMore": true
  }
}

Get Scenario

GET /api/scenarios/:id

Retrieve detailed information about a specific scenario.

Path Parameters

id - Scenario ID

Example Request

curl "https://tpmjs.com/api/scenarios/clu123abc456"

Example Response

{
  "success": true,
  "data": {
    "id": "clu123abc456",
    "collectionId": "clx789def",
    "prompt": "Scrape the main article content from a news website and extract the headline, author, publication date, and body text.",
    "name": "Scrape article content",
    "description": "Tests web scraping capability on news sites",
    "tags": ["web-scraping", "content-extraction", "news"],
    "assertions": null,
    "qualityScore": 0.85,
    "totalRuns": 12,
    "lastRunAt": "2024-01-20T10:30:00Z",
    "lastRunStatus": "pass",
    "consecutivePasses": 5,
    "consecutiveFails": 0,
    "createdAt": "2024-01-01T00:00:00Z",
    "updatedAt": "2024-01-20T10:30:00Z",
    "collection": {
      "id": "clx789def",
      "name": "Web Scraping Toolkit",
      "slug": "web-scraping-toolkit"
    }
  }
}

List Collection Scenarios

GET /api/collections/:id/scenarios

Retrieve all scenarios for a specific collection.

Path Parameters

id - Collection ID

Query Parameters

limit - Number of results (default: 20, max: 50)
offset - Pagination offset

Example Request

curl "https://tpmjs.com/api/collections/clx789def/scenarios"

Example Response

{
  "success": true,
  "data": {
    "scenarios": [
      {
        "id": "clu123abc456",
        "name": "Scrape article content",
        "prompt": "Scrape the main article content...",
        "qualityScore": 0.85,
        "totalRuns": 12,
        "lastRunStatus": "pass",
        "tags": ["web-scraping"]
      }
    ]
  }
}

Create Scenario

POST /api/scenarios

Create a new scenario manually. Requires authentication and collection ownership.

Request Body

{
  "collectionId": "clx789def",
  "prompt": "Search for weather information in Tokyo and return the current temperature",
  "name": "Tokyo weather lookup",
  "description": "Tests weather search functionality",
  "tags": ["weather", "search", "api"]
}

TypeScript Example

const response = await fetch('https://tpmjs.com/api/scenarios', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': 'Bearer YOUR_API_KEY'
  },
  body: JSON.stringify({
    collectionId: 'clx789def',
    prompt: 'Search for weather information in Tokyo',
    name: 'Tokyo weather lookup',
    tags: ['weather', 'search']
  })
});

const { data: scenario } = await response.json();
console.log('Created scenario:', scenario.id);

Generate Scenarios

POST /api/collections/:id/scenarios/generate

AI-generate scenarios based on the collection's tools. Requires authentication and collection ownership.

Path Parameters

id - Collection ID

Request Body

{
  "count": 3,
  "skipSimilarityCheck": false
}

Example Request

curl -X POST "https://tpmjs.com/api/collections/clx789def/scenarios/generate" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"count": 3}'

Example Response

{
  "success": true,
  "data": {
    "scenarios": [
      {
        "scenario": {
          "id": "clu111aaa",
          "prompt": "Scrape the main article content from a news website...",
          "name": "News article extraction",
          "tags": ["web-scraping", "content-extraction"]
        },
        "similarity": {
          "maxSimilarity": 0.15,
          "mostSimilar": null
        }
      },
      {
        "scenario": {
          "id": "clu222bbb",
          "prompt": "Extract all product images from an e-commerce page...",
          "name": "E-commerce image extraction",
          "tags": ["web-scraping", "images", "e-commerce"]
        },
        "similarity": {
          "maxSimilarity": 0.72,
          "mostSimilar": {
            "id": "clu000xxx",
            "name": "Product page scraping"
          }
        }
      }
    ]
  }
}

Similarity Warning

When similarity.maxSimilarity exceeds 0.7, the scenario is similar to an existing one. The mostSimilar field shows which scenario it overlaps with.

Run Scenario

POST /api/scenarios/:id/run

Execute a scenario and return the results. Requires authentication. Consumes one run from your daily quota.

Path Parameters

id - Scenario ID

Example Request

curl -X POST "https://tpmjs.com/api/scenarios/clu123abc456/run" \
  -H "Authorization: Bearer YOUR_API_KEY"

Example Response

{
  "success": true,
  "data": {
    "runId": "run_abc123def456",
    "status": "completed",
    "success": true,
    "evaluator": {
      "model": "claude-3-5-sonnet-latest",
      "verdict": "pass",
      "reason": "The agent successfully extracted the article headline, author name, publication date, and full body text from the news website. All required fields were present and correctly formatted."
    },
    "usage": {
      "inputTokens": 892,
      "outputTokens": 353,
      "totalTokens": 1245,
      "executionTimeMs": 2341
    },
    "timestamps": {
      "startedAt": "2024-01-20T10:30:00Z",
      "completedAt": "2024-01-20T10:30:02Z",
      "createdAt": "2024-01-20T10:30:00Z"
    },
    "quotaRemaining": 46
  }
}

Failure Response

{
  "success": true,
  "data": {
    "runId": "run_xyz789",
    "status": "completed",
    "success": false,
    "evaluator": {
      "model": "claude-3-5-sonnet-latest",
      "verdict": "fail",
      "reason": "The agent was unable to complete the task. The target website returned a 403 Forbidden error, preventing the scraping operation."
    },
    "usage": {
      "inputTokens": 456,
      "outputTokens": 123,
      "totalTokens": 579,
      "executionTimeMs": 3102
    },
    "quotaRemaining": 45
  }
}

Get Scenario Run History

GET /api/scenarios/:id/runs

Retrieve the run history for a scenario. Requires authentication if the scenario belongs to a private collection.

Path Parameters

id - Scenario ID

Query Parameters

limit - Number of results (default: 10, max: 50)
offset - Pagination offset

Example Request

curl "https://tpmjs.com/api/scenarios/clu123abc456/runs?limit=5" \
  -H "Authorization: Bearer YOUR_API_KEY"

Example Response

{
  "data": [
    {
      "id": "run_abc123",
      "status": "completed",
      "success": true,
      "evaluator": {
        "model": "claude-3-5-sonnet-latest",
        "verdict": "pass",
        "reason": "Successfully extracted article content"
      },
      "assertions": null,
      "usage": {
        "inputTokens": 892,
        "outputTokens": 353,
        "totalTokens": 1245,
        "executionTimeMs": 2341
      },
      "timestamps": {
        "startedAt": "2024-01-20T10:30:00Z",
        "completedAt": "2024-01-20T10:30:02Z",
        "createdAt": "2024-01-20T10:30:00Z"
      }
    }
  ],
  "pagination": {
    "limit": 5,
    "offset": 0,
    "hasMore": true
  }
}

Check Prompt Similarity

POST /api/scenarios/check-similarity

Check if a prompt is similar to existing scenarios in a collection. Useful before creating manual scenarios to avoid duplicates.

Request Body

{
  "prompt": "Scrape article headlines from a news website",
  "collectionId": "clx789def"
}

Example Response

{
  "success": true,
  "data": {
    "maxSimilarity": 0.78,
    "similar": [
      {
        "id": "clu123abc456",
        "name": "Scrape article content",
        "prompt": "Scrape the main article content from a news website...",
        "similarity": 0.78
      },
      {
        "id": "clu789xyz",
        "name": "Extract news headlines",
        "prompt": "Get all headlines from the front page of a news site...",
        "similarity": 0.65
      }
    ]
  }
}

Similarity Threshold

A maxSimilarity above 0.7 indicates significant overlap with existing scenarios. Consider modifying your prompt to test different aspects of your tools.

Get Featured Scenarios

GET /api/scenarios/featured

Retrieve featured scenarios for the homepage showcase. Returns a mix of high-quality, diverse, and recently successful scenarios.

Query Parameters

limit - Number of results (default: 10, max: 20)

Example Request

curl "https://tpmjs.com/api/scenarios/featured?limit=5"

Example Response

{
  "success": true,
  "data": [
    {
      "id": "clu123abc456",
      "name": "Scrape article content",
      "qualityScore": 0.95,
      "totalRuns": 50,
      "lastRunStatus": "pass",
      "collection": {
        "name": "Web Scraping Toolkit",
        "slug": "web-scraping-toolkit",
        "username": "johndoe"
      }
    }
  ]
}

Error Responses

Common error codes and responses

401 Unauthorized

{
  "success": false,
  "error": "Unauthorized",
  "message": "API key is required for this endpoint"
}

403 Forbidden

{
  "success": false,
  "error": "Forbidden",
  "message": "You don't have permission to access this scenario"
}

404 Not Found

{
  "success": false,
  "error": "Not Found",
  "message": "Scenario not found"
}

429 Rate Limited

{
  "success": false,
  "error": "Rate Limit Exceeded",
  "message": "Daily scenario run quota exceeded. Resets at midnight UTC.",
  "quotaRemaining": 0,
  "resetAt": "2024-01-21T00:00:00Z"
}