How it works
The path a source takes through the Hypeline engine, from a URL you add to a signed event you receive: detection, per-tier ingestion, change detection, matching, and a signed stream you consume over plain HTTP.
Hypeline turns any source you point it at into one unified, deduplicated live stream of new-content events. You add a URL, the engine works out what it is, watches it at a cadence that fits its type, and emits a normalized, signed event the moment something genuinely new appears. This page is the map of that path. Each stage links to the page that covers it in depth.
1. You add a source
You add a source with a single POST /v1/sources carrying its URL. The engine fetches it once and classifies it, so you do not have to declare what kind of source it is:
- A
ws://orwss://URL is a streaming source. - A body that parses as RSS, Atom, or JSON Feed is a feed source, even if it is served as
text/html. An HTML page that advertises a feed is followed to it. - Anything else is a feed-less HTML source that the engine watches by diffing.
Classification runs behind a strict SSRF guard: the fetch is pinned to a validated public IP, and a URL that resolves to a private, loopback, link-local, or cloud-metadata address is refused before it ever becomes a source. The guard fails closed, so a URL that cannot be safely resolved is never added.
2. The engine watches it at the right cadence
Each tier is consumed the way it is meant to be, which is why latency is honest and per-tier rather than one universal number:
| Tier | How it is watched | Typical latency |
|---|---|---|
| Streaming | A connection held open, frames pushed as they arrive. | Seconds |
| Feed | Polled and parsed, on an adaptive cadence. | Minutes |
| Feed-less HTML | Fetched, extracted, and diffed on an adaptive cadence. | Minutes to tens of minutes |
Polling is adaptive and polite. A source that just changed is checked more often; a source that keeps returning the same content backs off toward a ceiling measured in tens of minutes, never hours. The engine sends conditional requests (If-None-Match / If-Modified-Since) so an unchanged source costs a 304 and almost nothing, honors robots.txt on the crawl tier, and trips a per-source circuit breaker that backs off a failing source and recovers it automatically.
3. It decides what is genuinely new
Feeds and streams announce their own new items. Feed-less HTML pages do not, and most of what changes between two fetches is noise. The engine extracts the main content with go-trafilatura, strips volatile chrome in the page's own language, fingerprints the result with a 64-bit simhash, and compares it against a rolling window of recent captures, so a rotated ad or a ticking timestamp never reads as a change. Pages that render their content with JavaScript are escalated to a real bundled browser and compared on their post-hydration HTML. The full mechanism is on how change detection works.
4. It becomes one normalized, signed event
Whatever the tier, the output is one shape: a StreamEvent with a stable schema (id, source_type, url, title, tags, emitted_at, and more). Feed-less HTML change events also carry a structured diff of exactly what changed. Every event is signed at the source with an Ed25519 content-layer signature the moment it is born, so you can prove any event is authentic with only a public key, on any transport. See verifying events.
5. You filter it down to what you care about
You do not consume the firehose. The stream is filtered server-side with repeatable query parameters (source, source_type, tag, keyword), backed by an Aho-Corasick keyword pre-filter and PostgreSQL full-text search, so a Boolean or phrase keyword query is matched in one pass and confirmed with stemming. You only receive, and only store, the slice you asked for.
6. You consume it over plain HTTP
The stream is Server-Sent Events over GET /v1/stream, so anything that can read an HTTP response can consume it. A dropped connection resumes gap-free from the last event id via Last-Event-ID, and you can backfill from a known point with ?since=. Named channels expose stable, pre-filtered views of the same bus. The quickstart gets you from a key to a live stream in a single curl.