openclaw/Chinese NLP Toolkit

Chinese NLP Toolkit

Process and analyze Chinese text with specialized NLP capabilities.

Core Capabilities

1. Text Segmentation (分词)

Chinese has no word boundaries. Segmentation is the foundation of all Chinese NLP.

Approach: Use rule-based heuristics when no library is available:

• Dictionary matching (maximum forward/backward matching)
• Context-aware: "南京市长江大桥" → ["南京市", "长江大桥"] not ["南京", "市长", "江大桥"]
• Domain-specific terms should be added as custom dictionary entries

Common Ambiguities: | Text | Wrong Split | Correct Split | |------|-------------|---------------| | 雨伞 | 雨/伞 | 雨伞 (compound) | | 结婚的和尚未结婚的 | 结婚/的/和尚/未/结婚/的 | 结婚/的/和/尚未/结婚/的 | | 项目部 | 项目/部 | 项目部 (compound) |

2. Sentiment Analysis (情感分析)

Beyond positive/negative — Chinese sentiment is nuanced:

Intensity levels: 强烈负面 < 偏负面 < 中性 < 偏正面 < 强烈正面

Chinese-specific signals:

• Rhetorical questions often indicate negative sentiment: "这也算好？"
• Sarcasm markers: "呵呵", "厉害了", "也是醉了", "你开心就好"
• Intensifiers: "非常", "特别", "简直了", "超级"
• Diminishers: "还行吧", "马马虎虎", "凑合"

Emoji contribution (critical for social media):

• 😊👍❤️ = positive amplification
• 😤👎💔 = negative amplification
• 🙄🙄🙄 = sarcasm/disdain (intensity scales with repetition)

3. Keyword Extraction (关键词提取)

For Chinese text, prioritize:

• Noun phrases (名词短语)
• Domain-specific terminology
• Named entities (人名、地名、机构名)

Method: TF-IDF adapted for Chinese + positional weighting (first/last sentences carry more weight in Chinese writing).

4. Text Summarization (文本摘要)

Chinese-specific rules:

• Summarize to 20-30% of original length
• Preserve key numbers, names, and claims
• Chinese articles often "bury the lead" — the conclusion may be more important than the introduction
• Extract key sentences using positional + keyword scoring

5. Readability Scoring (可读性评分)

Rate Chinese text on a 1-10 scale considering:

• Average sentence length (characters per sentence)
• Vocabulary difficulty (HSK level estimate)
• Clause density ( commas per sentence)
• Use of classical Chinese elements
• Technical jargon density

| Score | Level | Target Audience | |-------|-------|-----------------| | 1-3 | Easy | General public | | 4-6 | Moderate | Educated readers | | 7-8 | Hard | Domain experts | | 9-10 | Very Hard | Academic specialists |

6. Format Conversion

| Conversion | Example | |---|---| | Simplified → Traditional | 体验 → 體驗 | | Traditional → Simplified | 體驗 → 体验 | | Chinese → Pinyin | 你好 → nǐ hǎo | | Chinese → Zhuyin | 你好 → ㄋㄧˇ ㄏㄠˇ |

Workflow

When Processing Chinese Text:

1. Detect variant: Simplified (简体) or Traditional (繁体)?
2. Segment: Break into meaningful units
3. Analyze: Apply the requested analysis type(s)
4. Report: Present results with Chinese annotations

Output Format

原文：[original text]
分词：[segmented text with / separators]
关键词：[top 5-10 keywords with relevance scores]
情感：[sentiment label + confidence + key signals]
摘要：[summarized text]
可读性：[score/10 + brief explanation]

Edge Cases

• Mixed-language text: Handle code-switching naturally ("这个bug太坑了") — don't force Chinese segmentation on English words
• Internet slang: Recognize common abbreviations (yyds, xswl, nbcs, awsl) and expand for formal analysis
• Poetry/classical Chinese: Flag as special case — modern NLP rules don't apply; use classical grammar patterns
• Dialectal text: Flag non-Mandarin text (Cantonese, Shanghainese written forms) — analysis may be unreliable
• Zero-width characters: Chinese text sometimes contains invisible characters (U+200B, U+FEFF) that affect processing

Common Tasks & Prompts

• "Analyze the sentiment of this Chinese review"
• "Extract keywords from this article"
• "Summarize this Chinese news article in 100 characters"
• "Rate the readability of this document"
• "Convert this to Traditional Chinese with pinyin annotation"
• "Segment this Chinese text and identify named entities"