Hypelinedocs
Quickstart

Sources

What a source is, the three source tiers Hypeline detects automatically, how it watches each one politely and adaptively, and how to check a source's health.

A source is any URL you want turned into a live stream of new-content events. You add one with a single POST /v1/sources (with a sources:write key), and the engine works out what it is and how to watch it.

bash
curl -X POST https://api.hypeline.io/v1/sources \
  -H "Authorization: Bearer $HYPELINE_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/news"}'

The three tiers

The engine fetches the URL once and classifies it, so you do not declare its type:

TierWhat it isHow it is watched
Streaming (push)A ws:// / wss:// endpoint that pushes frames.A connection held open; frames arrive in seconds.
Feed (feed)RSS, Atom, or JSON Feed.Polled and parsed on an adaptive cadence.
Feed-less HTML (html)A page with no feed, watched by diffing.Fetched, extracted, and diffed on an adaptive cadence.

Detection is body-based, not a guess from the URL or Content-Type: a URL whose body parses as a feed is a feed source even when served as text/html, and an HTML page that advertises a feed with <link rel="alternate"> is followed to it. The response echoes back the detected type and the evidence for it, so a wrong guess is visible and you can override it by passing type explicitly on create.

Streaming sources are filtered, not archived

A streaming source can emit a very high volume, so it only admits frames that match a filter or alert. You store and receive the slice you asked for, never the whole stream.

Safe by construction

Because a source URL is user-supplied, add-time classification runs behind an SSRF guard: the fetch is pinned to a validated public IP, and a URL that resolves to a private, loopback, link-local, or cloud-metadata address is refused before it can become a source. The guard fails closed, so a URL that cannot be safely resolved is never added.

Watched politely and adaptively

The engine matches its cadence to each source and stays a good citizen of the sites it watches:

  • Adaptive cadence. A source that just changed is polled more often; one that keeps returning the same content backs off toward a ceiling of tens of minutes, never hours.
  • Conditional requests. If-None-Match / If-Modified-Since mean an unchanged source costs a 304 and almost nothing.
  • robots.txt is honored on the crawl tier, with per-host rate limiting and backoff.
  • A per-source circuit breaker backs a failing source off after repeated errors and recovers it automatically once it responds again.

Health

Every source carries a health signal you can read, so a source that quietly stops producing is visible rather than silent:

bash
curl -H "Authorization: Bearer $HYPELINE_KEY" \
  https://api.hypeline.io/v1/sources/{id}/health
json
{
  "score": 0.94,
  "flag": "ok",
  "reason": "producing on cadence",
  "last_success_at": "2026-07-11T09:31:00Z"
}

The score runs from 0 to 1 and the flag is ok, degraded, or rotten. A source that is merely quiet (returning 304/no-change on cadence) is not penalized; the score falls when a source stops responding or stops yielding content it used to yield. GET /v1/sources returns the whole list worst-health-first, so the sources that need attention are at the top.

Going deeper