realroc/ama-script-abuse-screening

AMA Script Abuse Screening

Use this skill to triage behavioral abuse on an AI chat product's conversation collection — script callers, prompt-injection templates, and free-quota farming — as opposed to content violations (which the sibling shumei-user-violation-audit skill handles via Shumei text-risk API).

The screening is anchored on the assumption that a normal end user has:

• A non-empty request_client (web / desktop / app)
• A non-curl, non-empty browser
• A populated session_id for almost every request
• A small handful of long, natural-language query values per session
• A coherent topic across the day's query history

A script caller diverges on at least one of these axes, often several at once.

Required Inputs

Before running, identify these from the user or local context:

• Time window — a single CST date or a [start, end) range; defaults to "yesterday CST".
• Mongo access — either:
  • mongo_uri + mongo_db for direct read access, or
  • the name of a backend container that already has pymongo + the MONGO__* env populated (preferred for production), in which case the skill is invoked via docker exec <container> python3 /tmp/<script>.py.
• Optional whitelist — list of usernames or IPs known to be internal smoke-test / monitoring accounts that must be excluded from "confirmed abuse" (e.g. an internal canary user that legitimately uses browser=curl).

The script is read-only for MongoDB. It writes a JSON triage report and a ban-candidate CSV; nothing else.

Schema Assumptions

The conversation collection is expected to expose these top-level fields (field names match the typical AMA backend; for a different schema, edit the field references in the script directly — there is no CLI remap):

| Field | Type | Notes | |---|---|---| | _id | ObjectId | Used for time-window filtering (avoids slow request_time scans on unindexed deployments) | | request_time | float | Unix seconds, UTC | | user | str | Account identifier; phone-registered users prefixed with # | | ama_uid | str | Anonymous device id; populated by the web client | | ip | str | Client IP | | query | str | User prompt | | engine / model | str | Each user prompt fans out to 5–6 engines in PK mode — that 1-to-N expansion is expected | | role | str | assistant / mcp / mcp_assistant | | browser / device | str | UA hints | | request_client / request_client_version | str | web / desktop / app / '' | | session_id / conv_id | str | Empty session_id on a real client is the strongest single signal |

Available secondary indexes typically include user_1_request_time_-1, request_time_-1, engine_1_request_time_1. ip is usually NOT indexed, so range queries should be anchored on _id (which is) and filtered on ip in the projection stage — never count_documents({"ip": ...}) over a multi-day window without an _id bound.

Detection Rules

The script applies the following heuristics. Each rule emits a per-user score; a user with ≥ 2 rules tripped and ≥ 50 daily requests goes on the ban candidate list.

1. Prompt-injection templates — query regex matches any of:

   • (?i)cron\s*job
   • (?i)You are running as
   • \[SILENT\]
   • 127\.0\.0\.1.*health / localhost.*health
   • <\|+DSML\|+

   Extend by editing DEFAULT_INJECTION_PATTERNS in the script or by passing your own injection_patterns list via --config.

2. Forged / scripted user agent — browser in {"curl", ""} AND request_client == "" AND volume ≥ 20/day, excluding the configured whitelist.

3. Probe-word flood — any of {"ssf", "ping", "hi", "hello", "test", "你好", "在吗"} appearing ≥ 20 times for one user in one day.

4. Low-distinct repetition — same user with ≥ 80 daily requests and ≤ 5 distinct query values (Note: with 5-engine PK fanout, 5 unique prompts × 5 engines = 25 docs is normal; the threshold is set above that).

5. Sessionless calls — same user with ≥ 50 requests where session_id ∈ {None, ""}.

6. Cross-IP burst — same user with ≥ 10 distinct ip values in one day AND avg requests-per-ip ≥ 10 (legitimate roaming rarely hits both bounds).

7. Multi-account IP — same ip shared by ≥ 15 distinct user values in one day, and the IP geolocates outside the product's primary market (geolocation step is optional; left to the caller).

Output

The script writes two files in the working directory:

• script_abuse_triage_<date>.json — full triage report:
  • rule_counts: hit count per rule type
  • tier_1_confirmed: users tripping ≥ ban_min_rules rules with daily volume ≥ ban_min_daily_reqs
  • tier_2_suspicious: users that tripped at least one rule but missed the tier-1 thresholds
  • multi_account_ips: IPs hosting ≥ multi_account_ip_min_users distinct users (IP-centric, not per-user)
• ban_candidates_<date>.csv — flat ban list: username, primary_ip, daily_count, tripped_rules, evidence_query

The script also prints a stdout summary suitable for a 24 KB TAT response cap.

Workflow

1. Confirm the time window with the user (default: "yesterday CST").
2. Confirm whitelist usernames / IPs (internal smoke-test, monitoring agents).
3. Run the script against Mongo. If conversation lives behind a backend container, copy the script into the container first (docker cp <script> <container>:/tmp/) and docker exec it there.
4. Inspect the tier-1 list. For every ban candidate, eyeball at least three raw query values and the cross-week request counts before recommending a ban — a single noisy day can be a false positive.
5. Hand the operator the CSV and the JSON evidence.

Run

# Single CST day (defaults to yesterday) — the supported default cadence.
python3 scripts/screen_script_abuse.py --date 2026-05-22

# Custom range and a whitelist. Keep the window ≤ 3 days unless you are
# running against a read replica during off-peak hours — every per-rule
# aggregate scans the full `_id` range with `allowDiskUse=True`, so a wide
# window can spill to disk multiple times on primaries.
python3 scripts/screen_script_abuse.py \
  --start 2026-05-22 --end 2026-05-25 \
  --whitelist-user internal-canary-user \
  --whitelist-ip 1.2.3.4

# Show config example
python3 scripts/screen_script_abuse.py --print-example-config

Mongo connection — either pass --mongo-uri + --mongo-db, or rely on env vars MONGO__MONGO_URL / MONGO__MONGO_USER / MONGO__MONGO_PASSWD / MONGO__MONGO_DATABASE (the convention inside the AMA backend container).

Tuning

All thresholds in the Detection Rules section are configurable via the thresholds block of a JSON config passed with --config (see --print-example-config for the full schema). The defaults were calibrated on roughly 170 k daily conversation docs; scale them proportionally for smaller or larger products.

When a user lands on tier 1 but their queries read as genuine multi-engine PK (e.g. literary roleplay, design iteration, code Q&A), prefer rate-limit over ban — the data shape is similar but the intent is different. The skill report includes a evidence_query column specifically so the operator makes that call on a human read of the actual prompt, not just the counters.

Privacy

The triage report contains usernames, IPs, and prompt fragments. Treat the output as PII; do not commit it to git or paste it into public channels.