How change detection works

Watching a page for changes is easy to start and hard to get right. The hard part is not fetching the page. It is deciding what actually counts as new.

Feeds and streaming sources are event-native: an RSS item or a WebSocket frame already marks its own arrival, so Hypeline reads the new item and moves on. A feed-less HTML page gives you nothing to go on. It has no notion of "an event", and most of what visibly changes between two fetches is noise: a timestamp ticks over, an ad rotates, a view counter climbs, a related-articles rail reshuffles, a theme flips from light to dark overnight. Compare the raw HTML and every one of those reads as a change. This page is about how Hypeline separates a real edit from that churn on the HTML tier.

Real changes only

Hypeline reduces each capture to the page's main content and ignores the surrounding noise: navigation, ads, and sidebars, plus the timestamps, counters, and template shuffling that move on every load. A change event fires only when the meaningful content actually changes.

The engine is deliberately conservative about periodic and cyclical churn, so a page that merely alternates between templates or rotates a pool of related links does not read as an edit. Numbers and dates in the body itself are always preserved, because a changed price, rate, or deadline is exactly the change you want to hear about.

What you receive

When a real change is detected, the HTML event carries a structured diff of the content, so you get what changed without diffing it yourself:

{
  "added":   ["A new paragraph that appeared."],
  "removed": ["A line that was deleted."],
  "changed": [{ "before": "Price: 100", "after": "Price: 120" }],
  "truncated": false
}

The diff is computed over the extracted text with a word- and number-aware tokenizer, and is capped in size (truncated is true when a very large change exceeds the cap). Only whole-content HTML change events carry a diff; feed and streaming events, and the first snapshot of a new HTML source, carry none.

JavaScript and single-page apps

A lot of the modern web returns an empty shell to a plain HTTP request and paints the real content only after its JavaScript runs. Hypeline detects when a feed-less page behaves this way and renders it in a real browser (bundled in the engine), then runs the post-hydration HTML through the same change detection. The URL is re-validated for safety before the browser touches it, and the browser only ever returns HTML, so a JavaScript-rendered page is compared the same way a static one is. No configuration is required; the escalation is automatic and cached per source.

When you need to override extraction

Automatic extraction handles the common case. When you need to pin change detection to a specific region of a page, or extract a field out of a JSON endpoint, set a selector on the source (CSS, XPath, or jq). The selector replaces automatic extraction, and change detection then works exactly as described above over the selected content.