Streaming API
The SSE event stream contract, generated from the AsyncAPI spec.
This reference is generated directly from our AsyncAPI specification (asyncapi.json), emitted from the Go source of truth (the same StreamEvent schema as the API reference), so it always matches the published contract. Subscribe over Server-Sent Events with an API key in the X-API-Key header (or as a Bearer token); the response content-type is text/event-stream.
How the stream frames events
Each SSE frame carries id: <uuidv7> (also the resume cursor), event: set to the public source tier (push, feed, or html), and data: holding the StreamEvent JSON. Two control frames are not payload messages and so do not appear in the message list below: an id:-only priming frame sent on backfill/resume to advance your cursor, and a : keepalive comment line that keeps the connection warm.
Resuming without gaps
Reconnect and pass the last id you saw as the Last-Event-ID request header (or ?since=<uuidv7>); the stream replays everything after it. A cursor older than your retained window returns 410 Gone, so treat a 410 as "start fresh from the live tip" rather than an error to retry.
Latency is per-tier
Latency is honest and per-source-tier: push sources arrive in seconds, RSS/JSON feeds in minutes, and feed-less HTML-diff sources in minutes to tens of minutes. The event: label tells you which tier an event came from; it is a latency framing, never a reason to branch consumer logic.
Hypeline Stream API (events)
Subscribe to Hypeline's unified, deduplicated live event stream over Server-Sent Events. Latency is honest and per-source-tier: push sources arrive in seconds, RSS/JSON feeds in minutes, and feed-less HTML-diff sources in minutes to tens of minutes. Resume gap-free with the Last-Event-ID cursor; a cursor older than the retained window returns 410 Gone. This document is written from the subscriber's (consumer's) viewpoint.
Servers
api.hypeline.io(https) Authenticated SSE endpoint. AsyncAPI has no SSE protocol, so SSE is modeled as https plus prose and http bindings. Authenticate with X-API-Key or a Bearer token; the response content-type is text/event-stream. (The host is a documented placeholder, cosmetic to codegen.)
Channels
SSE
/v1/streamLive normalized event stream (SSE)
Each SSE frame is `id: <uuidv7>` (also the Last-Event-ID resume cursor), `event: <public tier: push|feed|html>`, and `data: <StreamEvent JSON>`. Two control frames are NOT payload messages and are documented in prose only: an `id:`-only priming/cursor frame on backfill/resume, and a `: keepalive` comment line. Filter with ?source, ?source_type, ?tag and ?keyword; backfill with ?since=<uuidv7> or the Last-Event-ID request header.
Operations
receive
streamSubscribe to the live normalized event stream (SSE).
Query parameters
| Field | Type | Description |
|---|---|---|
keyword | string | Repeatable keyword filter. |
since | string | Backfill floor: replay events after this UUIDv7 id. |
source | string | Restrict to a source id (a trailing .* is a prefix match). |
source_type | "push" | "feed" | "html" | |
tag | string | Repeatable tag filter. |
Messages
feed Feed-tier event
A normalized event whose SSE `event:` label is `feed`.
Headers
| Field | Type | Description |
|---|---|---|
Last-Event-ID | string <uuid> | Request header: resume cursor of the last event the client saw. |
id | string <uuid> | SSE id: field; the UUIDv7 event id and the Last-Event-ID resume cursor. |
Payload: application/json StreamEvent
html HTML-tier event
A normalized event whose SSE `event:` label is `html`.
Headers
| Field | Type | Description |
|---|---|---|
Last-Event-ID | string <uuid> | Request header: resume cursor of the last event the client saw. |
id | string <uuid> | SSE id: field; the UUIDv7 event id and the Last-Event-ID resume cursor. |
Payload: application/json StreamEvent
push Push-tier event
A normalized event whose SSE `event:` label is `push`.
Headers
| Field | Type | Description |
|---|---|---|
Last-Event-ID | string <uuid> | Request header: resume cursor of the last event the client saw. |
id | string <uuid> | SSE id: field; the UUIDv7 event id and the Last-Event-ID resume cursor. |
Payload: application/json StreamEvent
Schemas
StreamEvent
| Field | Type | Description |
|---|---|---|
$schema | string <uri> | A URL to the JSON Schema for this object. |
authorrequired | string | Item author, when known. |
contentrequired | string | Extracted main content (boilerplate removed). |
content_hashrequired | string | null | Hash of the extracted content for the trivial-unchanged case; null until populated. |
diff | ContentDiff | Itemized old->new change on html-tier whole-content events: added/removed/changed segments; null (omitted) on other tiers and cold starts. |
emitted_atrequired | string <date-time> | When the Engine emitted this normalized event onto the stream. |
fetched_atrequired | string <date-time> | When the Engine fetched the item. |
idrequired | string | UUIDv7 event id; also the SSE Last-Event-ID resume cursor |
key_idrequired | string | kid of the signing key; resolve the matching public JWK at /.well-known/jwks.json. Empty until signed. |
published_atrequired | string | null | Source-reported publish time; may be absent. |
schema_versionrequired | integer <int64> | Contract version of this event (currently 1). |
signaturerequired | string | Content-layer Ed25519 signature over this event (RFC 8785 JCS canonical form, signature field excluded), base64url-unpadded of the 64 raw bytes. Empty until the event is signed. |
signed_atrequired | string | null | When the signature was produced (RFC 3339 UTC); covered by the signature. null until signed. |
simhashrequired | string | null | 64-bit near-duplicate fingerprint of the extracted content, as a decimal string (exceeds JS Number precision as an integer); null until populated. |
source_idrequired | string | The source's stable UUID — the same 'id' POST /v1/sources returns. Correlate an event to a source directly by this id; ?source= filters on it too. |
source_typerequired | string | Source tier label: push|feed|html|… Latency framing is per-tier; never a behavioural switch for consumers. |
source_urlrequired | string | The source's ingest key: the feed/page URL that produced this event (the push-tier adapter constant for born-live stream sources). |
tagsrequired | array | null | Server-side filter tags attached to this event. |
titlerequired | string | Item title / headline. |
urlrequired | string | Canonical URL of the item, when known. |
ContentDiff
| Field | Type | Description |
|---|---|---|
addedrequired | array | null | |
changedrequired | array | null | |
removedrequired | array | null | |
truncatedrequired | boolean |
Create-sourceRequest
| Field | Type | Description |
|---|---|---|
$schema | string <uri> | A URL to the JSON Schema for this object. |
browser_steps | object | Optional ordered interactive pre-capture steps (jsonb): [{op,selector,value}]. Validated at set time (structure + same-origin goto + selector denylist); login/secret steps rejected (deferred). Echo-only here; durable storage is the html_sources.browser_steps column read by the ingest loader. |
categorize | boolean | Enable the generic content-categorization capability for this source (default false). |
cluster | boolean | Enable the generic cross-source clustering capability for this source (default false). |
preprocess | object | Optional per-source canonicalization knobs (jsonb); omit for all-defaults. |
schedule | string | Optional poll interval as a Go duration (e.g. "5m"); empty = tier default. |
selector | string | Optional content-selector override (css:/xpath:/jq:/jqraw:/json: or a bare CSS selector); validated at set time, empty = structural auto-detect. |
tags | array | null | Optional general label tags (e.g. ["competitors"]); bare values are normalized to label:<slug>. Reserved prefixes cohort:/region:/topic:/outlet: are rejected. |
type | "push" | "feed" | "html" | Optional operator override of the detected connector tier. |
urlrequired | string <uri> | The source URL to add (any language/region; a feed, a streaming source, or a feed-less web page). |
Detect-sourceRequest
| Field | Type | Description |
|---|---|---|
$schema | string <uri> | A URL to the JSON Schema for this object. |
urlrequired | string <uri> | The URL to classify. |
DetectDTO
| Field | Type | Description |
|---|---|---|
$schema | string <uri> | A URL to the JSON Schema for this object. |
confidencerequired | string | Detection confidence: high|medium|low|unknown. |
discovered_feed_url | string | A hidden feed URL discovered on an HTML page, if any. |
evidencerequired | string | Human-readable evidence for the classification. |
supportedrequired | boolean | False = recognized tier with no connector yet. |
typerequired | string | Classified connector tier: push|feed|html. |
DiffSegment
| Field | Type | Description |
|---|---|---|
after | string | |
before | string |
ErrorDetail
| Field | Type | Description |
|---|---|---|
location | string | Where the error occurred, e.g. 'body.items[3].tags' or 'path.thing-id' |
message | string | Error message text |
value | object | The value at the given location |
ErrorModel
| Field | Type | Description |
|---|---|---|
$schema | string <uri> | A URL to the JSON Schema for this object. |
detail | string | A human-readable explanation specific to this occurrence of the problem. |
errors | array | null | Optional list of individual error details |
instance | string <uri> | A URI reference that identifies the specific occurrence of the problem. |
status | integer <int64> | HTTP status code |
title | string | A short, human-readable summary of the problem type. This value should not change between occurrences of the error. |
type | string <uri> | A URI reference to human-readable documentation for the error. |
HealthDTO
| Field | Type | Description |
|---|---|---|
$schema | string <uri> | A URL to the JSON Schema for this object. |
flagrequired | string | Maintenance flag: ok|degraded|rotten. |
last_checked_atrequired | string <date-time> | Last time the source was polled, if any. |
last_success_atrequired | string <date-time> | Last successful fetch, if any. |
rankrequired | number <double> | Importance-weighted maintenance rank; higher = more urgent. |
reason | string | Human explanation of the dominating penalty. |
scorerequired | number <double> | Health score in [0,1]; 1 = healthy. |
HistoryPage
| Field | Type | Description |
|---|---|---|
$schema | string <uri> | A URL to the JSON Schema for this object. |
eventsrequired | array | null | The events on this page, ascending by id (the UUIDv7 keyset cursor). |
next_cursorrequired | string | null | Pass back as ?cursor= to fetch the next page; null when this is the last page. |
retention_clampedrequired | boolean | True when the requested 'from' predated the key's retention window and was clamped up to window_from. |
window_fromrequired | string | RFC3339 effective lower bound actually queried after the retention clamp (the tier floor when no wider 'from' was allowed). |
PickerBox
| Field | Type | Description |
|---|---|---|
hrequired | number <double> | Element height in CSS px. |
selectorrequired | string | Generated CSS selector for this element (compiles with cascadia). |
tagrequired | string | Lowercased HTML tag name. |
textrequired | string | Trimmed element text, capped. |
wrequired | number <double> | Element width in CSS px. |
xrequired | number <double> | Page-absolute CSS px (scrollX included). |
yrequired | number <double> | Page-absolute CSS px (scrollY included). |
PickerDTO
| Field | Type | Description |
|---|---|---|
$schema | string <uri> | A URL to the JSON Schema for this object. |
boxesrequired | array | null | Clickable element boxes with generated selectors. |
dprrequired | number <double> | Device pixel ratio the screenshot was taken at. |
formatrequired | string | Screenshot image format (jpeg). |
screenshotrequired | string | Bare base64 (no data: prefix) of the JPEG screenshot. |
viewport_hrequired | integer <int64> | Captured page height in CSS px (height-capped). |
viewport_wrequired | integer <int64> | Capture viewport width in CSS px. |
RenderDocBody
| Field | Type | Description |
|---|---|---|
$schema | string <uri> | A URL to the JSON Schema for this object. |
block | array | null | Optional resource block list (e.g. font, media). Empty on a visual capture so images load. |
format | string | Image encoding for a png output: png|jpeg|webp. |
full_page | boolean | Capture the full scrollable page rather than the viewport. |
output | array | null | Subset of ["html","png","pdf"]; default ["png"]. One output streams raw bytes, multiple outputs return a base64 JSON envelope. |
timeout_ms | integer <int64> | Per-request time budget in milliseconds (clamped to the server maximum). |
urlrequired | string | The absolute http(s) URL to render. Validated against the SSRF classifier before navigation. |
viewport | RenderDocViewport | Optional viewport override (clamped to server maxima). |
wait | RenderDocWait | Optional content-settle wait before capture. |
RenderDocViewport
| Field | Type | Description |
|---|---|---|
height | integer <int64> | Viewport height in CSS pixels (clamped to the server maximum). |
mobile | boolean | Emulate a mobile device. |
scale | number <double> | Device scale factor (clamped to the server maximum). |
width | integer <int64> | Viewport width in CSS pixels (clamped to the server maximum). |
RenderDocWait
| Field | Type | Description |
|---|---|---|
ms | integer <int64> | Extra settle delay in milliseconds after load. |
selector | string | Wait for this CSS selector to appear before capture. |
RenderMultiResponse
| Field | Type | Description |
|---|---|---|
$schema | string <uri> | A URL to the JSON Schema for this object. |
final_urlrequired | string | |
html | string | |
pdf_base64 | string | |
png_base64 | string |
SourceDTO
| Field | Type | Description |
|---|---|---|
$schema | string <uri> | A URL to the JSON Schema for this object. |
browser_steps | object | Ordered interactive pre-capture steps (jsonb): [{op,selector,value}]. Ops: click|click_if_exists|type|select|scroll|wait_for_element|wait_for_text|press_enter|goto. goto is same-origin-only. Login/secret steps are not yet supported (deferred). Durable storage is the html_sources.browser_steps column read by the ingest loader. |
categorize | boolean | Whether the generic content-categorization capability is enabled for this source (default false). When on and the build-time master toggle is set, emitted events get topic:<slug> tags. |
cluster | boolean | Whether the generic cross-source clustering capability is enabled for this source (default false). When on and the build-time master toggle is set, events group into cross-source stories. |
created_atrequired | string <date-time> | When the source was first added. |
detected_type | string | What add-time detection classified the URL as, before any operator override. |
detection_confidencerequired | string | Detection confidence: high|medium|low|unknown. |
detection_evidence | string | Human-readable evidence for the detected type (so a wrong guess is diagnosable). |
discovered_feed_url | string | A hidden <link rel=alternate> feed discovered on an HTML page, if any. |
enabledrequired | boolean | Whether the source is actively polled. |
follow_one_hoprequired | boolean | Whether a listing item is followed one hop to its article (INGEST-06). |
health_flagrequired | string | Maintenance flag: ok|degraded|rotten. |
health_rankrequired | number <double> | Importance-weighted maintenance rank; higher = more urgent. |
health_reason | string | Human explanation of the dominating health penalty. |
health_scorerequired | number <double> | Health score in [0,1]; 1 = healthy. |
idrequired | string | Stable source identifier (UUIDv7). |
labels | array | null | General user label tags on this source (e.g. ["competitors"]); each maps to a label:<x> tag stamped onto emitted events. |
moderequired | string | Extraction mode: auto|whole|listing. |
preprocess | object | Per-source canonicalization knobs (jsonb): json_sort, line_sort, dedupe_lines, strip_obfuscation, whitespace, pdf_text. Omit for all-defaults. Durable storage is the html_sources.preprocess column read by the ingest loader. |
schedule | string | Poll interval as a Go duration string (e.g. "5m"); empty = tier default. |
selector | string | Optional CSS selector override for feed-less HTML extraction. |
supportedrequired | boolean | False = the tier is recognized but has no connector yet (e.g. a non-Bluesky push source). |
typerequired | string | Effective connector tier: push|feed|html. Config-time selector, never a consumer branch. |
updated_atrequired | string <date-time> | When the source config was last changed. |
urlrequired | string | The user-supplied source URL (one row per URL, the highway model). |
Update-sourceRequest
| Field | Type | Description |
|---|---|---|
$schema | string <uri> | A URL to the JSON Schema for this object. |
browser_steps | object | Replace the source's ordered interactive pre-capture steps (jsonb); validated at set time (structure + same-origin goto + selector denylist), login/secret steps rejected (deferred). Echo-only here; nil = leave unchanged. Durable storage is the html_sources.browser_steps column. |
categorize | boolean | Enable/disable the generic content-categorization capability for this source. |
cluster | boolean | Enable/disable the generic cross-source clustering capability for this source. |
enabled | boolean | Enable/disable polling. |
follow_one_hop | boolean | Follow a listing item one hop to its article. |
mode | "auto" | "whole" | "listing" | Extraction mode. |
preprocess | object | Replace the source's per-source canonicalization knobs (jsonb); omit to leave unchanged. |
schedule | string | Poll interval as a Go duration string. |
selector | string | CSS selector override. |
type | "push" | "feed" | "html" | Connector tier override. |