RonanCodes/ingest-hackernews

Ingest Hacker News Thread

Extract the full content of a Hacker News post and its comment tree.

Extraction Method

1. Parse the URL to extract the item ID:

   • Pattern: https://news.ycombinator.com/item?id={id}
   • Also accept bare IDs (just the number)

2. Fetch the full thread via Algolia HN API (free, no auth, returns entire comment tree):

curl -s "https://hn.algolia.com/api/v1/items/{id}"

3. Parse JSON response for the top-level post:

   • title — post title
   • url — linked URL (if link post; null for Ask HN / Show HN text posts)
   • author — HN username of poster
   • created_at — ISO timestamp
   • points — score
   • text — body text (for self/text posts; HTML-encoded)
   • children — array of comment objects (recursive)

4. Parse each comment in children recursively:

   • author — commenter username
   • text — comment body (HTML-encoded, strip tags to markdown)
   • created_at — timestamp
   • children — nested replies
   • Skip items where author is null (deleted/dead comments)

5. Convert HTML entities in text fields to readable markdown:

   • <p> to double newline
   • <a href="..."> to [text](url) markdown links
   • <code> / <pre> to backtick code spans / fenced code blocks
   • <i> to *italic*
   • ' and other entities to literal characters

Building the Markdown Output

Construct a flat markdown representation of the thread with indentation to show nesting depth:

# {title}

**Author:** {author} | **Points:** {points} | **Date:** {created_at}
**HN URL:** https://news.ycombinator.com/item?id={id}
**Link:** {url} (if present)

## Post Body
{text content if self-post, otherwise "Link post — see URL above."}

## Comments ({total_comment_count} comments)

**{commenter1}** ({timestamp}):
Comment text here...

> **{reply_author}** ({timestamp}):
> Reply text here, indented with blockquote...
>
> > **{nested_reply_author}** ({timestamp}):
> > Deeper reply, double-indented...

**{commenter2}** ({timestamp}):
Another top-level comment...

Use > blockquote markers for nesting depth:

• Top-level comments: no prefix
• 1 level deep: >
• 2 levels deep: > >
• 3+ levels deep: cap at > > > (deeper replies still shown but capped at 3 levels of indent for readability)

Post-Extraction

• Save to vault's raw/hn-{id}-{slug}.md where {slug} is a kebab-case version of the title (truncated to ~50 chars), with YAML header:

---
source-url: https://news.ycombinator.com/item?id={id}
title: "{title}"
author: "{author}"
date-fetched: {today}
source-type: discussion
points: {points}
comment-count: {total_comments}
linked-url: "{url or none}"
---

• Include the full thread content (post + comments) below the frontmatter

Handling Large Threads

• The Algolia API returns the full tree in one call, but very large threads (1000+ comments) may produce a lot of content.
• For threads with more than 200 top-level comments, include only the top ~100 most substantial comment sub-trees (those with the most replies or longest text) and note the truncation in the raw file.
• Always include a comment count summary at the top of the Comments section.

Fallback

If the Algolia API fails or returns an error:

1. Try the official HN Firebase API as a fallback (requires multiple calls):

# Get the item
curl -s "https://hacker-news.firebaseio.com/v0/item/{id}.json"
# Then fetch each child comment by ID from the `kids` array
curl -s "https://hacker-news.firebaseio.com/v0/item/{kid_id}.json"

2. Note: the Firebase API requires one call per comment (no tree endpoint), so for large threads this is slow. Use only as a fallback.

Dependencies

None — uses only curl.