clone

a desktop tv and program guide for over-the-air broadcasts: what's on air, what your tuner gets, live tv, recordings

Take the guide's listings from Schedules Direct, with an account

FelineStateMachine committed 1cf22a2
internal/sd keeps the ZIP code's antenna lineup from Schedules Direct, the
licensed source of Gracenote's data for personal use, in place of the
public grid, which now answers only browsers: two weeks of listings with
long descriptions, cast, original air dates, movie runtimes and star
ratings, and sports teams. Station and program IDs are Gracenote's either
way, so channels, series rules and recordings carry over.

It's a careful client, as Schedules Direct requires: it logs in only when
its day-long token runs out (keeping it across restarts), adds the lineup
once, updates once a day at a minute of its own, asks for the
station-days whose checksums changed and the programs whose did, renews
an expired token once, and stops asking altogether on a refusal (a wrong
password, a locked or expired account, too many logins, a reply that
isn't JSON) until the account is saved again. It identifies itself as
airwaves/<version>, under the personal research terms.

The admin page signs in (Guide listings): the server keeps the user name
and the password's SHA-1, which the API takes, readable by it alone and
out of backups. The app's guide still shows its listing hours, now with
stars, runtime, first air dates, teams and cast where known; recording
rules look the two weeks ahead, and the emulated HDHomeRun's XMLTV
carries them. Without an account nothing changes.
commit
1cf22a2fe329ba6a71a6fc84927e85c0317b95a9
parent
acf1dea856cfc3a78faca667e07415d274c820e2
author
FelineStateMachine <226808055+FelineStateMachine@users.noreply.github.com>
kind
feat risk score 1.9 of 3, Jev's estimate
20 files changed, +2042 -34
MREADME.md23+-
Mcmd/airwavesd/main.go4+
Mfrontend/dist/app.js28+-
Mfrontend/dist/index.html1+
Mfrontend/dist/style.css2+
Minternal/admin/admin.go61+++-
Minternal/admin/ui/admin.css2+
Minternal/admin/ui/admin.js124+++++
Minternal/admin/ui/index.html1+
Minternal/guide/gracenote.go17+-
Minternal/guide/gracenote_test.go4+-
Minternal/lineup/lineup.go38+-
Ainternal/sd/account.go89++++
Ainternal/sd/client.go681++++++++++++++++++++++++++++++
Ainternal/sd/sd_test.go345+++++++++++++++
Ainternal/sd/source.go292+++++++++++++
Ainternal/sd/state.go307++++++++++++++
Minternal/service/antenna.go21+-
Minternal/service/hdhr.go5+-
Minternal/service/service.go31+-
M README.md +19 -4
@@ -177,15 +177,29 @@ an amber ring.
177177 | [FCC TV Query](https://www.fcc.gov/media/television/tv-query) | Licensed transmitters: RF channel, ERP, height, location | 7 days |178178 | [AWS Terrain Tiles](https://registry.opendata.aws/terrain-tiles/) | Elevation along each path (SRTM, USGS NED and others) | permanent |179179 | [RabbitEars.Info](https://www.rabbitears.info/market.php?request=atsc3) | ATSC 3.0 hosts and services | 3 days |180-| [Gracenote TV listings](https://tvlistings.gracenote.com/) | Lineup `USA-OTA<zip>` and schedules | 4 hours |180+| [Gracenote TV listings](https://tvlistings.gracenote.com/) | Lineup `USA-OTA<zip>` and schedules, without a Schedules Direct account | 4 hours |181+| [Schedules Direct](https://www.schedulesdirect.org/) | Lineup `USA-OTA-<zip>` and two weeks of schedules and programs, with an account | Updated daily |181182 | [Zippopotam.us](https://zippopotam.us/) | ZIP centroid | 1 year |182183 183184 Caches live in `~/Library/Caches/airwaves` (on the server, `~/airwaves/data/cache`);184185 app settings in `~/Library/Application Support/airwaves/settings.json`.185186 186187 The Gracenote grid endpoint is the public JSON behind tvlistings.gracenote.com. It is187-undocumented and meant for personal use; [Schedules Direct](https://www.schedulesdirect.org)188-is the supported source if this grows beyond that.188+undocumented and meant for personal use, and it answers only browsers.189+190+**Schedules Direct.** With an account at [Schedules Direct](https://www.schedulesdirect.org)191+(the licensed source of the same Gracenote data, for personal use), the listings come from192+there instead: two weeks of them, with long descriptions, cast, original air dates, movie193+runtimes and star ratings, and sports teams. Sign in on the admin page (Guide listings); the194+server keeps the user name and the password's SHA-1, which is what the API takes, in195+`<data>/schedulesdirect`, readable by it alone, and never in backups or the app. It adds the196+ZIP code's antenna lineup to the account, updates once a day at a minute of its own, asks only197+for the station-days that changed, and stops asking altogether when the account is refused198+(a wrong password, a locked or expired account) until it's saved again, as Schedules Direct199+requires. The app's guide still shows its listing hours; recording rules look the two weeks200+ahead, and the emulated HDHomeRun's XMLTV carries them all. Airwaves identifies itself as201+`airwaves/<version>`; it is used under Schedules Direct's personal research terms, for one202+household.189203 190204 ## How reception is estimated191205
@@ -234,7 +248,7 @@ It serves HTTP on port 5004 (`AIRWAVES_HDHR_LISTEN`, default `:5004`; empty or
234248 | `/discover.json`, `/lineup.json`, `/lineup_status.json` | The HDHomeRun device API |235249 | `/auto/v<number>`, `/tuner0/v<number>` | A channel's stream, e.g. `/auto/v9.1` |236250 | `/channels.m3u` | Playlist, with `url-tvg` pointing at the guide |237-| `/xmltv.xml` | 48 hour guide from Gracenote listings and custom channel schedules; channel ids `airwaves.<number>` |251+| `/xmltv.xml` | The guide as far as the listings go (two days from Gracenote's grid, two weeks from Schedules Direct) and custom channel schedules; channel ids `airwaves.<number>` |238252 | `/channel-logos/<number>` | A custom channel's logo, also on the API port; the playlist and guide link to it |239253 240254 Discovery answers on UDP 65001, only to other machines: requests from the server itself
@@ -616,6 +630,7 @@ internal/terrain Terrarium elevation tiles and path profiles
616630 internal/reception propagation model and antenna presets (otascan)617631 internal/atsc3 RabbitEars ATSC 3.0 list parser618632 internal/guide Gracenote lineup and listings633+internal/sd Schedules Direct listings: account, daily updates of what changed619634 internal/lineup merges all of the above into a Report, and describes the tuner's channels from it620635 internal/signal what the tuners measured per RF channel, kept across restarts621636 internal/tuner shares the HDHomeRun's tuners: one per RF channel, by priority
M cmd/airwavesd/main.go +4 -0
M frontend/dist/app.js +23 -5
@@ -721,17 +721,34 @@ function step(dir) {
721721 function chipsFor(p, ch) {722722 const out = [];723723 for (const fl of (p && p.flags) || []) out.push(`<span class="chip ${fl === 'Live' ? 'live' : fl === 'New' ? 'new' : ''}">${esc(fl)}</span>`);724+ if (p && p.stars) out.push(`<span class="chip" title="Gracenote's rating, out of 4">${stars(p.stars)}</span>`);724725 if (p && p.rating) out.push(`<span class="chip">${esc(p.rating)}</span>`);726+ if (p && p.runtime) out.push(`<span class="chip">${p.runtime >= 60 ? `${Math.floor(p.runtime / 60)} h ${p.runtime % 60 ? `${p.runtime % 60} min` : ''}` : `${p.runtime} min`}</span>`);725727 for (const t of (p && p.tags) || []) out.push(`<span class="chip">${esc(t)}</span>`);726728 return out.join('');727729 }728730 731+// stars draws a rating out of 4 in stars, halves included.732+const stars = (n) => '\u2605'.repeat(Math.floor(n)) + (n % 1 >= 0.5 ? '\u00bd' : '');733+729734 function episodeLine(p) {730735 if (!p) return '';731736 const se = p.season && p.episode ? `S${p.season} E${p.episode}` : '';732- return [p.episodeTitle && `"${p.episodeTitle}"`, se, p.year].filter(Boolean).join(' ');737+ // A repeat says when it first aired.738+ const first = p.originalAirDate && !(p.flags || []).includes('New') ? `first aired ${firstAired(p.originalAirDate)}` : '';739+ const game = p.awayTeam && p.homeTeam && !p.episodeTitle ? `${p.awayTeam} at ${p.homeTeam}` : '';740+ return [p.episodeTitle && `"${p.episodeTitle}"`, game, se, p.year, first].filter(Boolean).join(' ');741+}742+743+// firstAired names an original air date ("2019-03-05") as "Mar 5, 2019".744+function firstAired(d) {745+ const [y, m, day] = d.split('-').map(Number);746+ return new Date(y, m - 1, day).toLocaleDateString([], { month: 'short', day: 'numeric', year: 'numeric' });733747 }734748 749+// castLine names a program's cast: "With Missy Peregrym, Zeeko Zaki".750+const castLine = (p) => (p && p.cast && p.cast.length ? `With ${p.cast.slice(0, 5).map((c) => c.replace(/ \(.*\)$/, '')).join(', ')}` : '');751+735752 function renderBanner() {736753 updateMediaSession();737754 showTimeshift();
@@ -2174,15 +2191,15 @@ function renderGuideActions(ch, p) {
21742191 }21752192 21762193 // fitGuideDetail leaves out what of the program's details doesn't fit2177-// above the grid at this window and interface size: its description2178-// first, then its tags, its episode line and the hint. The title and the2194+// above the grid at this window and interface size: its cast first, then2195+// its description, its tags, its episode line and the hint. The title and the21792196 // actions always show, the actions on one row that scrolls sideways when21802197 // even that isn't enough.21812198 function fitGuideDetail() {21822199 const g = $('#guide');21832200 const top = $('.g-top', g);21842201 const box = $('.g-detail', g);2185- const parts = ['desc', 'meta', 'ep', 'hint'].map((k) => f(g, k));2202+ const parts = ['cast', 'desc', 'meta', 'ep', 'hint'].map((k) => f(g, k));21862203 for (const el of parts) el.classList.remove('gd-gone');21872204 box.classList.remove('one-row');21882205 const cs = getComputedStyle(top);
@@ -2246,7 +2263,7 @@ function renderGuideDetail(ch, p) {
22462263 const g = $('#guide');22472264 if (!ch) {22482265 // No channels under this filter.2249- for (const k of ['kicker', 'title', 'ep', 'meta', 'desc', 'actions', 'hint']) f(g, k).textContent = '';2266+ for (const k of ['kicker', 'title', 'ep', 'meta', 'cast', 'desc', 'actions', 'hint']) f(g, k).textContent = '';22502267 f(g, 'art').classList.remove('has');22512268 return;22522269 }
@@ -2258,6 +2275,7 @@ function renderGuideDetail(ch, p) {
22582275 // listed it shows on its own, with the channel's description.22592276 const genres = p ? p.genres || [] : ch.own && ch.category !== 'Other' ? [ch.category] : [];22602277 f(g, 'meta').innerHTML = chipsFor(p, ch) + genres.map((x) => `<span class="chip">${esc(x)}</span>`).join('');2278+ f(g, 'cast').textContent = castLine(p);22612279 f(g, 'desc').textContent = p ? p.description || '' : ch.description || '';22622280 const live = !p || p._s <= now;22632281 const ok = keyHint('Enter', 'Enter');
M frontend/dist/index.html +1 -0
M frontend/dist/style.css +2 -0
M internal/admin/admin.go +59 -2
@@ -35,6 +35,7 @@ import (
3535 "airwaves/internal/guide"3636 "airwaves/internal/jellyfin"3737 "airwaves/internal/phase"38+ "airwaves/internal/sd"3839 "airwaves/internal/vchan"3940 )4041
@@ -58,6 +59,9 @@ type Server struct {
5859 // ChannelsDir and MusicDir are where the channels and the weather5960 // music folders are as the host sees them, for the page's hints.6061 ChannelsDir, MusicDir string62+ // Listings is the guide's Schedules Direct account; nil on a server63+ // without an antenna.64+ Listings Listings6165 6266 bk backupState6367 mu sync.Mutex
@@ -108,6 +112,8 @@ func (s *Server) Handler() http.Handler {
108112 }))109113 mux.HandleFunc("GET /admin/api/state", s.getState)110114 mux.HandleFunc("PUT /admin/api/account", s.putAccount)115+ mux.HandleFunc("PUT /admin/api/listings", s.putListings)116+ mux.HandleFunc("DELETE /admin/api/listings", s.deleteListings)111117 mux.HandleFunc("POST /admin/api/account/quickconnect", s.startQuickConnect)112118 mux.HandleFunc("GET /admin/api/account/quickconnect/{id}", s.checkQuickConnect)113119 mux.HandleFunc("GET /admin/api/library/{kind}", s.getLibrary)
@@ -508,10 +514,14 @@ func (s *Server) getState(w http.ResponseWriter, r *http.Request) {
508514 list = append(list, antennaJSON{Number: n, Name: name})509515 }510516 slices.SortFunc(list, func(a, b antennaJSON) int { return guide.CompareNumbers(a.Number, b.Number) })511- writeJSON(w, map[string]any{517+ state := map[string]any{512518 "account": s.accountStatus(r.Context(), false), "channels": chans, "categories": vchan.Categories, "antenna": list,513519 "channelsDir": s.ChannelsDir, "musicDir": s.MusicDir,514- })520+ }521+ if s.Listings != nil {522+ state["listings"] = s.Listings.Status()523+ }524+ writeJSON(w, state)515525 }516526 517527 func (s *Server) find(number string) (vchan.Entry, bool) {
@@ -1111,3 +1121,50 @@ func writeErr(w http.ResponseWriter, code int, err error) {
11111121 w.WriteHeader(code)11121122 _ = json.NewEncoder(w).Encode(map[string]string{"error": err.Error()})11131123 }1124+1125+// ---------- listings ----------1126+1127+// Listings is a Schedules Direct account and how its listings stand.1128+type Listings interface {1129+ Status() sd.Status1130+ // SetAccount keeps a login (an empty password keeps the saved one for1131+ // the same user) and updates with it; an empty user removes it.1132+ SetAccount(user, password string) error1133+}1134+1135+// putListings saves the Schedules Direct login and starts an update with1136+// it; the page follows the update in the state.1137+func (s *Server) putListings(w http.ResponseWriter, r *http.Request) {1138+ if s.Listings == nil {1139+ writeErr(w, http.StatusNotFound, errors.New("this server has no antenna, so no listings"))1140+ return1141+ }1142+ var req struct{ User, Password string }1143+ if !decode(w, r, &req) {1144+ return1145+ }1146+ req.User = strings.TrimSpace(req.User)1147+ if req.User == "" {1148+ writeErr(w, http.StatusBadRequest, errors.New("enter the Schedules Direct user name"))1149+ return1150+ }1151+ if err := s.Listings.SetAccount(req.User, req.Password); err != nil {1152+ writeErr(w, http.StatusBadRequest, err)1153+ return1154+ }1155+ writeJSON(w, s.Listings.Status())1156+}1157+1158+// deleteListings removes the Schedules Direct account; the guide goes back1159+// to Gracenote's public grid.1160+func (s *Server) deleteListings(w http.ResponseWriter, r *http.Request) {1161+ if s.Listings == nil {1162+ writeErr(w, http.StatusNotFound, errors.New("this server has no antenna, so no listings"))1163+ return1164+ }1165+ if err := s.Listings.SetAccount("", ""); err != nil {1166+ writeErr(w, http.StatusInternalServerError, err)1167+ return1168+ }1169+ writeJSON(w, s.Listings.Status())1170+}
M internal/admin/ui/admin.css +2 -0
M internal/admin/ui/admin.js +124 -0
@@ -375,6 +375,7 @@
375375 return;376376 }377377 renderAccount();378+ renderListings();378379 let t = hashTarget();379380 if (!t) {380381 // The first channel there is to edit; Jellyfin ones once there's an account.
@@ -396,6 +397,7 @@
396397 stateReq = (async () => {397398 try {398399 const prevAcct = JSON.stringify(S.state && S.state.account);400+ const prevLs = JSON.stringify(S.state && S.state.listings);399401 const prevMissing = JSON.stringify(S.ch && S.ch.unmatched);400402 S.state = normalizeState(await api('GET', '/state'));401403 S.stateErr = null;
@@ -403,6 +405,7 @@
403405 else if (S.sel && !isNewTarget(S.sel)) S.ch = findCh(S.sel);404406 renderChannels();405407 if (JSON.stringify(S.state.account) !== prevAcct && !S.qc && !acctFormInUse()) renderAccount();408+ if (JSON.stringify(S.state.listings) !== prevLs && !lsFormInUse()) renderListings();406409 if (E.yt) {407410 ytStateChanged();408411 } else if (E.kicker) {
@@ -726,6 +729,127 @@
726729 n.focus();727730 });728731 732+ // ---------- Side: guide listings ----------733+ // The guide's Schedules Direct account: a line while it works, the form734+ // with none, when it stopped, or on Change. An update is followed every735+ // few seconds until it's done.736+ function lsFormInUse() {737+ const box = $('#listings');738+ return box.contains(document.activeElement) && isTyping(document.activeElement);739+ }740+741+ const lsDate = (t) => new Date(t).toLocaleDateString([], { month: 'short', day: 'numeric' });742+743+ function renderListings() {744+ const box = $('#listings');745+ const l = S.state && S.state.listings;746+ box.hidden = !l;747+ if (!l) { box.replaceChildren(); return; }748+ const hadFocus = box.contains(document.activeElement);749+ const configured = !!l.user;750+ if (l.updating) followListings();751+ if (configured && !l.stopped && !S.lsOpen) {752+ const what = l.updating ? 'Fetching the listings'753+ : l.ok ? `${l.stations} stations through ${lsDate(l.through)}`754+ : l.error ? `Not updated: ${l.error}` : 'Waiting for the first update';755+ const dot = l.ok && !l.updating ? '' : l.error && !l.updating ? ' bad' : ' wait';756+ box.replaceChildren(h('div', { class: 'acct-line' },757+ h('span', { class: 'dot' + dot, 'aria-hidden': 'true' }),758+ h('span', { class: 'who' },759+ h('b', { text: 'Schedules Direct' }),760+ h('span', { text: `${l.user} | ${what}`, title: l.expires ? `Account until ${lsDate(l.expires)}` : '' })),761+ h('button', { class: 'btn ghost small', type: 'button', id: 'ls-change', onclick: () => { S.lsOpen = true; renderListings(); $('#ls-user').focus(); } }, 'Change')));762+ if (hadFocus) $('#ls-change').focus();763+ return;764+ }765+ let cls = 'acct-box', title = 'Guide listings', lead;766+ if (!configured) {767+ cls += ' setup';768+ lead = h('p', { class: 'lead' }, "Sign in to Schedules Direct for two weeks of listings, with cast, first air dates and ratings. Until then the guide uses Gracenote's public listings.");769+ } else if (l.stopped) {770+ cls += ' fail';771+ title = 'Listings stopped';772+ lead = h('p', { class: 'lead err' }, h('b', { text: l.stopped }), ' Nothing more is asked of Schedules Direct until you save the account again.');773+ } else {774+ lead = h('p', { class: 'lead' }, `Signed in to Schedules Direct as ${l.user}.`);775+ }776+ const msg = h('p', { class: 'acct-msg' + (S.lsMsg ? ' ' + S.lsMsg.cls : ''), role: 'status', text: S.lsMsg ? S.lsMsg.text : '' });777+ const form = h('form', { class: 'acct-form', onsubmit: saveListings },778+ h('label', { class: 'field' }, h('span', null, 'User name'),779+ h('input', { class: 'input', name: 'user', id: 'ls-user', value: l.user || '', autocomplete: 'username', spellcheck: 'false', autocapitalize: 'off' })),780+ h('label', { class: 'field' }, h('span', null, 'Password'),781+ h('input', { class: 'input', name: 'password', type: 'password', autocomplete: 'current-password', placeholder: configured ? 'saved' : '' })),782+ configured ? h('p', { class: 'help' }, 'Leave the password empty to keep the saved one. Only its SHA-1, which Schedules Direct takes, is kept on the server.')783+ : h('p', { class: 'help' }, 'Only the password\'s SHA-1, which Schedules Direct takes, is kept on the server.'),784+ h('div', { class: 'acct-actions' },785+ h('button', { class: 'btn', type: 'submit', id: 'ls-submit', disabled: S.lsBusy }, S.lsBusy ? 'Saving' : 'Save and fetch'),786+ configured && !l.stopped ? h('button', { class: 'btn ghost', type: 'button', onclick: () => { S.lsOpen = false; S.lsMsg = null; renderListings(); } }, 'Cancel') : null,787+ configured ? h('button', { class: 'btn ghost danger', type: 'button', onclick: removeListings }, S.lsRemove ? 'Press again to remove' : 'Remove') : null),788+ msg);789+ box.replaceChildren(h('div', { class: cls }, h('h2', null, title), lead, form));790+ if (hadFocus) {791+ const el = box.querySelector('#ls-submit:not([disabled]), input');792+ if (el) el.focus();793+ }794+ }795+796+ async function saveListings(e) {797+ e.preventDefault();798+ const f = e.currentTarget;799+ const body = { user: f.user.value.trim(), password: f.password.value };800+ if (!body.user) { S.lsMsg = { text: 'Enter the Schedules Direct user name.', cls: 'err' }; renderListings(); return; }801+ S.lsBusy = true;802+ renderListings();803+ try {804+ S.state.listings = await api('PUT', '/listings', body);805+ S.lsOpen = false;806+ S.lsMsg = null;807+ toast('Saved. Fetching two weeks of listings from Schedules Direct.');808+ } catch (err) {809+ S.lsMsg = { text: err.message, cls: 'err' };810+ }811+ S.lsBusy = false;812+ renderListings();813+ followListings();814+ }815+816+ async function removeListings() {817+ if (!S.lsRemove) {818+ S.lsRemove = true;819+ renderListings();820+ setTimeout(() => { S.lsRemove = false; if (!lsFormInUse()) renderListings(); }, 4000);821+ return;822+ }823+ S.lsRemove = false;824+ try {825+ S.state.listings = await api('DELETE', '/listings');826+ S.lsOpen = false;827+ toast("Removed. The guide goes back to Gracenote's public listings.");828+ } catch (err) {829+ toast(err.message, true);830+ }831+ renderListings();832+ }833+834+ // followListings reads the listings' state every few seconds while an835+ // update runs.836+ let lsTimer = 0;837+ function followListings() {838+ if (lsTimer) return;839+ lsTimer = setTimeout(async () => {840+ lsTimer = 0;841+ try {842+ const st = await api('GET', '/state');843+ const was = JSON.stringify(S.state.listings);844+ S.state.listings = st.listings;845+ if (JSON.stringify(st.listings) !== was && !lsFormInUse()) renderListings();846+ if (st.listings && (st.listings.updating || (st.listings.user && !st.listings.ok && !st.listings.error && !st.listings.stopped))) followListings();847+ } catch {848+ followListings();849+ }850+ }, 3000);851+ }852+729853 // ---------- Side: account ----------730854 function acctFormInUse() {731855 const box = $('#account');
M internal/admin/ui/index.html +1 -0
M internal/guide/gracenote.go +15 -2
@@ -57,6 +57,17 @@ type Program struct {
5757 // AudioLang is the sound's language as tagged ("eng", "jpn") where a5858 // custom channel knows it.5959 AudioLang string `json:"audioLang,omitempty"`60+61+ // From Schedules Direct's fuller listings: when the episode first62+ // aired ("2019-03-05"), the cast as "Name (Character)", a movie's63+ // running time in minutes and Gracenote's star rating (1 to 4), and a64+ // game's teams.65+ OriginalAirDate string `json:"originalAirDate,omitempty"`66+ Cast []string `json:"cast,omitempty"`67+ Runtime int `json:"runtime,omitempty"`68+ Stars float64 `json:"stars,omitempty"`69+ HomeTeam string `json:"homeTeam,omitempty"`70+ AwayTeam string `json:"awayTeam,omitempty"`6071 }6172 6273 // Guide is a lineup with listings, keyed by channel ID.
@@ -194,7 +205,7 @@ func (g *Guide) merge(resp *gridResponse, seen map[string]bool) {
194205 ID: ch.ChannelID,195206 Number: ch.ChannelNo,196207 CallSign: ch.CallSign,197- Network: networkName(ch.AffiliateName),208+ Network: NetworkName(ch.AffiliateName),198209 Logo: imageURL(ch.Thumbnail),199210 })200211 }
@@ -266,7 +277,9 @@ var acronyms = map[string]bool{
266277 "LC": true, "LAFF": true, "ABC": true, "CBS": true, "NBC": true, "FOX": true, "CW": true,267278 }268279 269-func networkName(affiliate string) string {280+// NetworkName is an affiliate's name as the guide shows it: "NBC" for281+// "NATIONAL BROADCASTING COMPANY", "Start TV" for "START TV NETWORK".282+func NetworkName(affiliate string) string {270283 if n, ok := networkNames[affiliate]; ok {271284 return n272285 }
M internal/guide/gracenote_test.go +2 -2
M internal/lineup/lineup.go +27 -11
@@ -107,7 +107,10 @@ type Builder struct {
107107 // Terrain, when set, has every transmitter's reception estimated from108108 // the terrain between it and the location. Without it the report is109109 // the records alone, and its stations are ordered by distance.110- Terrain *terrain.Source110+ Terrain *terrain.Source111+ // Listings, when it has any, are the guide's listings in place of112+ // Gracenote's public grid: Schedules Direct's, kept by the server.113+ Listings func() *guide.Guide111114 Progress func(string)112115 }113116
@@ -162,14 +165,24 @@ func (b *Builder) Build(ctx context.Context, opt Options) (*Report, *guide.Guide
162165 }163166 rep.ATSC3 = relevantHosts(hosts, rep.Stations)164167 165- b.progress("Loading listings for lineup %s", guide.LineupID(opt.ZIP))166- guideKey := fmt.Sprintf("guide-%s-%dh", opt.ZIP, opt.GuideHours)167- g, err := store.Load(ctx, b.Cache, guideKey, 4*time.Hour, opt.Refresh, func(ctx context.Context) (*guide.Guide, error) {168- return guide.Fetch(ctx, b.HTTP, opt.ZIP, time.Now().Add(-30*time.Minute), opt.GuideHours)169- })170- if err != nil {171- rep.Warnings = append(rep.Warnings, "Listings unavailable: "+err.Error())172- g = nil168+ var g *guide.Guide169+ if b.Listings != nil {170+ g = b.Listings()171+ }172+ if g != nil {173+ b.progress("Listings from Schedules Direct, lineup %s", g.Lineup)174+ rep.Sources = slices.DeleteFunc(rep.Sources, func(s Source) bool { return s.Name == gracenoteSource })175+ rep.Sources = append(rep.Sources, Source{"Schedules Direct", "https://www.schedulesdirect.org/", "Over-the-air lineup and program guide (Gracenote data)"})176+ } else {177+ b.progress("Loading listings for lineup %s", guide.LineupID(opt.ZIP))178+ guideKey := fmt.Sprintf("guide-%s-%dh", opt.ZIP, opt.GuideHours)179+ g, err = store.Load(ctx, b.Cache, guideKey, 4*time.Hour, opt.Refresh, func(ctx context.Context) (*guide.Guide, error) {180+ return guide.Fetch(ctx, b.HTTP, opt.ZIP, time.Now().Add(-30*time.Minute), opt.GuideHours)181+ })182+ if err != nil {183+ rep.Warnings = append(rep.Warnings, "Listings unavailable: "+err.Error())184+ g = nil185+ }173186 }174187 175188 rep.Channels = buildChannels(rep, g)
@@ -177,13 +190,16 @@ func (b *Builder) Build(ctx context.Context, opt Options) (*Report, *guide.Guide
177190 return rep, g, nil178191 }179192 180-const terrainSource = "AWS Terrain Tiles"193+const (194+ terrainSource = "AWS Terrain Tiles"195+ gracenoteSource = "Gracenote TV Listings"196+)181197 182198 var sources = []Source{183199 {"FCC TV Query", "https://www.fcc.gov/media/television/tv-query", "Licensed transmitters, power, height, location"},184200 {terrainSource, "https://registry.opendata.aws/terrain-tiles/", "Terrain profiles for reception estimates"},185201 {"RabbitEars.Info", "https://www.rabbitears.info/market.php?request=atsc3", "ATSC 3.0 (NextGen TV) hosting"},186- {"Gracenote TV Listings", "https://tvlistings.gracenote.com/", "Over-the-air lineup and program guide"},202+ {gracenoteSource, "https://tvlistings.gracenote.com/", "Over-the-air lineup and program guide"},187203 {"Zippopotam.us", "https://zippopotam.us/", "Location lookup"},188204 }189205
A internal/sd/account.go +89 -0
@@ -0,0 +1,89 @@
1+package sd2+3+import (4+ "crypto/sha1"5+ "encoding/hex"6+ "encoding/json"7+ "errors"8+ "io/fs"9+ "os"10+ "path/filepath"11+ "strings"12+)13+14+// Account is a Schedules Direct login as the server keeps it: the user15+// name and the SHA-1 of the password, which is what the API takes, so the16+// password itself is never stored.17+type Account struct {18+ User string `json:"user"`19+ PasswordSHA1 string `json:"passwordSha1"`20+}21+22+// Set reports whether there's a login.23+func (a Account) Set() bool { return a.User != "" && a.PasswordSHA1 != "" }24+25+// HashPassword is the form of a password the API takes: its SHA-1, in hex.26+func HashPassword(password string) string {27+ sum := sha1.Sum([]byte(password))28+ return hex.EncodeToString(sum[:])29+}30+31+// LoadAccount reads the login kept at path; a missing file is no login.32+func LoadAccount(path string) (Account, error) {33+ var a Account34+ raw, err := os.ReadFile(path)35+ if errors.Is(err, fs.ErrNotExist) {36+ return a, nil37+ }38+ if err != nil {39+ return a, err40+ }41+ err = json.Unmarshal(raw, &a)42+ a.User = strings.TrimSpace(a.User)43+ return a, err44+}45+46+// SaveAccount keeps a at path, readable by this user only; an empty47+// account removes the file.48+func SaveAccount(path string, a Account) error {49+ if !a.Set() {50+ err := os.Remove(path)51+ if errors.Is(err, fs.ErrNotExist) {52+ return nil53+ }54+ return err55+ }56+ raw, err := json.MarshalIndent(a, "", " ")57+ if err != nil {58+ return err59+ }60+ return writeFile(path, raw, 0o600)61+}62+63+// writeFile replaces path with raw atomically, with the given mode.64+func writeFile(path string, raw []byte, mode os.FileMode) error {65+ if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {66+ return err67+ }68+ f, err := os.CreateTemp(filepath.Dir(path), filepath.Base(path)+".*")69+ if err != nil {70+ return err71+ }72+ _, err = f.Write(raw)73+ if err == nil {74+ err = f.Chmod(mode)75+ }76+ if err == nil {77+ err = f.Sync()78+ }79+ if cerr := f.Close(); err == nil {80+ err = cerr81+ }82+ if err == nil {83+ err = os.Rename(f.Name(), path)84+ }85+ if err != nil {86+ os.Remove(f.Name())87+ }88+ return err89+}
A internal/sd/client.go +681 -0
@@ -0,0 +1,681 @@
1+package sd2+3+import (4+ "bytes"5+ "cmp"6+ "context"7+ "encoding/json"8+ "errors"9+ "fmt"10+ "io"11+ "net/http"12+ "slices"13+ "strconv"14+ "strings"15+ "time"16+)17+18+// API is the Schedules Direct JSON API.19+const API = "https://json.schedulesdirect.org/20141201/"20+21+// Request limits.22+const (23+ maxStations = 5000 // per /schedules and /schedules/md524+ maxPrograms = 5000 // per /programs25+)26+27+// StopError is a refusal that ends updating until the account is28+// changed: a wrong password, a locked, expired or disabled account, too29+// many logins, or a request Schedules Direct calls a client bug. Trying30+// again would only count against the account.31+type StopError struct {32+ Code int33+ Message string34+}35+36+func (e *StopError) Error() string {37+ if e.Code == 0 {38+ return e.Message39+ }40+ return fmt.Sprintf("%s (Schedules Direct code %d)", e.Message, e.Code)41+}42+43+// apiError is any other error response.44+type apiError struct {45+ Code int46+ Response string47+ Message string48+ Status int49+}50+51+func (e *apiError) Error() string {52+ return fmt.Sprintf("Schedules Direct: %s (code %d, HTTP %d)", cmp.Or(e.Message, e.Response), e.Code, e.Status)53+}54+55+// stopCodes end updating until the account changes.56+var stopCodes = map[int]string{57+ 4001: "The Schedules Direct account has expired",58+ 4002: "Schedules Direct didn't take the password's form",59+ 4003: "Schedules Direct doesn't know that user name and password",60+ 4004: "The Schedules Direct account is locked for a while after failed logins",61+ 4005: "Schedules Direct has turned off this account's access; open a ticket with them",62+ 4007: "Schedules Direct has turned off this application",63+ 4008: "The Schedules Direct account isn't active",64+ 4009: "Too many logins to Schedules Direct today",65+ 4010: "Schedules Direct saw this account used from too many places",66+ 1003: "Schedules Direct wants a User-Agent",67+ 1011: "Schedules Direct wants JSON",68+ 2055: "Schedules Direct refused a debug request",69+ 9999: "Schedules Direct reported an internal failure",70+}71+72+// client is one update's conversation with Schedules Direct.73+type client struct {74+ base string // API, or a test server's75+ http *http.Client76+ ua string77+ account Account78+ st *state79+ // relogged is set once a request has been retried with a new token.80+ relogged bool81+}82+83+// envelope is the code every response object may carry.84+type envelope struct {85+ Code *int `json:"code"`86+ Response string `json:"response"`87+ Message string `json:"message"`88+}89+90+// do makes a request with JSON in and out. Errors in the response, by its91+// code, become *StopError or *apiError; an expired token is renewed once.92+func (c *client) do(ctx context.Context, method, path string, in, out any) error {93+ var body io.Reader94+ if in != nil {95+ raw, err := json.Marshal(in)96+ if err != nil {97+ return err98+ }99+ body = bytes.NewReader(raw)100+ }101+ req, err := http.NewRequestWithContext(ctx, method, cmp.Or(c.base, API)+path, body)102+ if err != nil {103+ return err104+ }105+ req.Header.Set("User-Agent", c.ua)106+ if in != nil || method == http.MethodPut {107+ req.Header.Set("Content-Type", "application/json")108+ }109+ if path != "token" {110+ req.Header.Set("Token", c.st.Token)111+ }112+ resp, err := c.http.Do(req)113+ if err != nil {114+ return fmt.Errorf("Schedules Direct: %w", err)115+ }116+ defer resp.Body.Close()117+ raw, err := io.ReadAll(io.LimitReader(resp.Body, 512<<20))118+ if err != nil {119+ return fmt.Errorf("Schedules Direct: %w", err)120+ }121+ trimmed := bytes.TrimSpace(raw)122+ if len(trimmed) == 0 || trimmed[0] != '{' && trimmed[0] != '[' {123+ // Not JSON at all: a load balancer's refusal, or worse. Stop124+ // rather than knock again.125+ return &StopError{Message: fmt.Sprintf("Schedules Direct answered HTTP %d with something other than JSON; this server may be blocked", resp.StatusCode)}126+ }127+ if trimmed[0] == '{' {128+ var env envelope129+ if err := json.Unmarshal(trimmed, &env); err == nil && env.Code != nil && *env.Code != 0 {130+ code := *env.Code131+ if code == 4006 && path != "token" && !c.relogged {132+ // The token ran out: one new one, one retry.133+ c.relogged = true134+ c.st.Token = ""135+ if err := c.login(ctx, true); err != nil {136+ return err137+ }138+ return c.do(ctx, method, path, in, out)139+ }140+ if msg, ok := stopCodes[code]; ok {141+ return &StopError{Code: code, Message: msg}142+ }143+ return &apiError{Code: code, Response: env.Response, Message: env.Message, Status: resp.StatusCode}144+ }145+ }146+ if resp.StatusCode >= 400 {147+ return &apiError{Status: resp.StatusCode, Message: http.StatusText(resp.StatusCode)}148+ }149+ if out == nil {150+ return nil151+ }152+ if err := json.Unmarshal(trimmed, out); err != nil {153+ return fmt.Errorf("Schedules Direct %s: %w", path, err)154+ }155+ return nil156+}157+158+// login gets a token, unless the saved one has more than an hour left. A159+// token lasts a day from its first use, and asking again within that day160+// gives the same one; fresh asks for a new one.161+func (c *client) login(ctx context.Context, fresh bool) error {162+ if !fresh && c.st.Token != "" && time.Until(c.st.TokenExpires) > time.Hour {163+ return nil164+ }165+ in := map[string]any{"username": c.account.User, "password": c.account.PasswordSHA1}166+ if fresh || c.st.Token != "" {167+ in["newToken"] = true168+ }169+ var out struct {170+ Token string `json:"token"`171+ TokenExpires int64 `json:"tokenExpires"`172+ ServerTime int64 `json:"serverTime"`173+ }174+ if err := c.do(ctx, http.MethodPost, "token", in, &out); err != nil {175+ return err176+ }177+ if out.Token == "" || out.TokenExpires == 0 {178+ return errors.New("Schedules Direct gave no token; it may be offline")179+ }180+ // The expiry by the server's clock, carried over to ours.181+ life := time.Duration(out.TokenExpires-cmp.Or(out.ServerTime, time.Now().Unix())) * time.Second182+ c.st.Token, c.st.TokenExpires = out.Token, time.Now().Add(life)183+ return nil184+}185+186+// update brings the state up to date: the account, the antenna lineup of187+// zip, the schedules that changed and the programs they air.188+func (c *client) update(ctx context.Context, zip string, now time.Time) error {189+ if err := c.login(ctx, false); err != nil {190+ return err191+ }192+ id := "USA-OTA-" + zip193+ modified, err := c.status(ctx, id)194+ if err != nil {195+ return err196+ }197+ if modified == "" {198+ if err := c.addLineup(ctx, id); err != nil {199+ return err200+ }201+ modified = "added"202+ }203+ if c.st.Lineup != id || c.st.LineupModified != modified || len(c.st.Stations) == 0 {204+ if err := c.lineup(ctx, id); err != nil {205+ return err206+ }207+ c.st.LineupModified = modified208+ }209+ if err := c.schedules(ctx, now); err != nil {210+ return err211+ }212+ if err := c.programs(ctx); err != nil {213+ return err214+ }215+ c.st.prune(now)216+ return nil217+}218+219+// status reads the account: when it expires, whether the service is up,220+// and when the lineup id last changed ("" when the account hasn't got it).221+func (c *client) status(ctx context.Context, id string) (string, error) {222+ var out struct {223+ Account struct {224+ Expires string `json:"expires"`225+ MaxLineups int `json:"maxLineups"`226+ } `json:"account"`227+ Lineups []struct {228+ Lineup string `json:"lineup"`229+ ID string `json:"ID"`230+ Modified string `json:"modified"`231+ IsDeleted bool `json:"isDeleted"`232+ } `json:"lineups"`233+ SystemStatus []json.RawMessage `json:"systemStatus"`234+ }235+ if err := c.do(ctx, http.MethodGet, "status", nil, &out); err != nil {236+ return "", err237+ }238+ if t, err := time.Parse(time.RFC3339, out.Account.Expires); err == nil {239+ c.st.Expires = t240+ }241+ for _, raw := range out.SystemStatus {242+ var sys struct {243+ Status, Message string244+ }245+ if json.Unmarshal(raw, &sys) == nil && strings.EqualFold(sys.Status, "Offline") {246+ return "", fmt.Errorf("Schedules Direct is offline: %s", sys.Message)247+ }248+ break // the first entry is the current one249+ }250+ for _, l := range out.Lineups {251+ if cmp.Or(l.Lineup, l.ID) == id && !l.IsDeleted {252+ return cmp.Or(l.Modified, "unknown"), nil253+ }254+ }255+ return "", nil256+}257+258+// addLineup adds the antenna lineup to the account, once: Schedules259+// Direct allows a few changes a day.260+func (c *client) addLineup(ctx context.Context, id string) error {261+ err := c.do(ctx, http.MethodPut, "lineups/"+id, nil, nil)262+ var ae *apiError263+ switch {264+ case errors.As(err, &ae) && ae.Code == 2100: // already there265+ return nil266+ case errors.As(err, &ae) && ae.Code == 2107:267+ return errors.New("Schedules Direct is preparing this ZIP code's antenna lineup; trying again later")268+ case errors.As(err, &ae) && (ae.Code == 4100 || ae.Code == 4101):269+ return &StopError{Code: ae.Code, Message: "The Schedules Direct account can't take another lineup (" + ae.Message + "); remove one on schedulesdirect.org"}270+ case errors.As(err, &ae) && ae.Code >= 2101 && ae.Code <= 2106:271+ return &StopError{Code: ae.Code, Message: "Schedules Direct has no antenna lineup for ZIP code " + strings.TrimPrefix(id, "USA-OTA-")}272+ }273+ return err274+}275+276+// lineup reads the stations of lineup id and their channel numbers.277+func (c *client) lineup(ctx context.Context, id string) error {278+ var out struct {279+ Map []struct {280+ StationID string `json:"stationID"`281+ Major int `json:"atscMajor"`282+ Minor int `json:"atscMinor"`283+ RF int `json:"uhfVhf"`284+ Channel json.RawMessage `json:"channel"`285+ } `json:"map"`286+ Stations []struct {287+ StationID string `json:"stationID"`288+ Name string `json:"name"`289+ CallSign string `json:"callsign"`290+ Affiliate string `json:"affiliate"`291+ Logos []struct {292+ URL string `json:"URL"`293+ Category string `json:"category"`294+ } `json:"stationLogo"`295+ Logo *struct {296+ URL string `json:"URL"`297+ } `json:"logo"`298+ } `json:"stations"`299+ }300+ if err := c.do(ctx, http.MethodGet, "lineups/"+id, nil, &out); err != nil {301+ return err302+ }303+ number := map[string]string{}304+ for _, m := range out.Map {305+ switch {306+ case m.Major > 0:307+ number[m.StationID] = fmt.Sprintf("%d.%d", m.Major, m.Minor)308+ default:309+ var ch string310+ if json.Unmarshal(m.Channel, &ch) == nil && ch != "" {311+ number[m.StationID] = strings.TrimLeft(ch, "0")312+ }313+ }314+ }315+ var stations []station316+ for _, s := range out.Stations {317+ n := number[s.StationID]318+ if n == "" {319+ continue320+ }321+ sta := station{ID: s.StationID, Number: n, CallSign: s.CallSign, Name: s.Name, Affiliate: s.Affiliate}322+ // The logo for a light background (the guide's logo chips), else323+ // any.324+ for _, want := range []string{"dark", "gray", ""} {325+ for _, l := range s.Logos {326+ if sta.Logo == "" && (want == "" || l.Category == want) {327+ sta.Logo = l.URL328+ }329+ }330+ }331+ if sta.Logo == "" && s.Logo != nil {332+ sta.Logo = s.Logo.URL333+ }334+ stations = append(stations, sta)335+ }336+ if len(stations) == 0 {337+ return fmt.Errorf("Schedules Direct's lineup %s has no stations", id)338+ }339+ c.st.Lineup, c.st.Stations = id, stations340+ if c.st.Days == nil {341+ c.st.Days = map[string]map[string]day{}342+ }343+ keep := map[string]bool{}344+ for _, s := range stations {345+ keep[s.ID] = true346+ }347+ for id := range c.st.Days {348+ if !keep[id] {349+ delete(c.st.Days, id)350+ }351+ }352+ return nil353+}354+355+// dates are the UTC days to keep: yesterday (US evenings fall on the next356+// UTC day) and Days from today.357+func dates(now time.Time) []string {358+ out := []string{}359+ d := now.UTC().AddDate(0, 0, -1)360+ for range Days + 1 {361+ out = append(out, d.Format(time.DateOnly))362+ d = d.AddDate(0, 0, 1)363+ }364+ return out365+}366+367+type stationDates struct {368+ StationID string `json:"stationID"`369+ Date []string `json:"date"`370+}371+372+// schedules fetches the station-days whose checksum changed since the373+// last update, and only those.374+func (c *client) schedules(ctx context.Context, now time.Time) error {375+ ds := dates(now)376+ var ask []stationDates377+ for _, s := range c.st.Stations {378+ ask = append(ask, stationDates{StationID: s.ID, Date: ds})379+ }380+ changed := map[string][]string{}381+ for chunk := range slices.Chunk(ask, maxStations) {382+ var out map[string]json.RawMessage383+ if err := c.do(ctx, http.MethodPost, "schedules/md5", chunk, &out); err != nil {384+ return err385+ }386+ for id, raw := range out {387+ // A station with nothing comes as [] rather than {}.388+ var days map[string]struct {389+ Code int `json:"code"`390+ MD5 string `json:"md5"`391+ }392+ if json.Unmarshal(raw, &days) != nil {393+ continue394+ }395+ for date, d := range days {396+ if d.Code == 0 && d.MD5 != "" && c.st.Days[id][date].MD5 != d.MD5 {397+ changed[id] = append(changed[id], date)398+ }399+ }400+ }401+ }402+ if len(changed) == 0 {403+ return nil404+ }405+ var get []stationDates406+ for id, ds := range changed {407+ slices.Sort(ds)408+ get = append(get, stationDates{StationID: id, Date: ds})409+ }410+ slices.SortFunc(get, func(a, b stationDates) int { return strings.Compare(a.StationID, b.StationID) })411+ for chunk := range slices.Chunk(get, maxStations) {412+ var out []json.RawMessage413+ if err := c.do(ctx, http.MethodPost, "schedules", chunk, &out); err != nil {414+ return err415+ }416+ for _, raw := range out {417+ c.schedule(raw)418+ }419+ }420+ return nil421+}422+423+// schedule keeps one station-day of a /schedules response; errors for424+// one station-day (deleted, queued, out of range) skip it.425+func (c *client) schedule(raw json.RawMessage) {426+ var s struct {427+ StationID string `json:"stationID"`428+ Code int `json:"code"`429+ Programs []struct {430+ ProgramID string `json:"programID"`431+ AirDateTime string `json:"airDateTime"`432+ Duration int `json:"duration"`433+ MD5 string `json:"md5"`434+ New bool `json:"new"`435+ Premiere bool `json:"premiere"`436+ LiveTapeDelay string `json:"liveTapeDelay"`437+ IsPremiereOrFinale string `json:"isPremiereOrFinale"`438+ AudioProperties []string `json:"audioProperties"`439+ VideoProperties []string `json:"videoProperties"`440+ Ratings []struct {441+ Body string `json:"body"`442+ Code string `json:"code"`443+ } `json:"ratings"`444+ } `json:"programs"`445+ Metadata struct {446+ MD5 string `json:"md5"`447+ StartDate string `json:"startDate"`448+ Code int `json:"code"`449+ } `json:"metadata"`450+ }451+ if json.Unmarshal(raw, &s) != nil || s.StationID == "" {452+ return453+ }454+ if s.Code == 2201 || s.Metadata.Code == 2201 { // the station is gone455+ delete(c.st.Days, s.StationID)456+ c.st.LineupModified = "" // read the lineup again next time457+ return458+ }459+ if s.Code != 0 || s.Metadata.StartDate == "" {460+ return461+ }462+ d := day{MD5: s.Metadata.MD5}463+ for _, p := range s.Programs {464+ start, err := time.Parse(time.RFC3339, p.AirDateTime)465+ if err != nil || p.ProgramID == "" || p.Duration <= 0 {466+ continue467+ }468+ a := airing{469+ ProgramID: p.ProgramID, Start: start, Seconds: p.Duration, MD5: p.MD5,470+ New: p.New, Live: p.LiveTapeDelay == "Live",471+ Premiere: p.Premiere || strings.Contains(p.IsPremiereOrFinale, "Premiere"),472+ Finale: strings.Contains(p.IsPremiereOrFinale, "Finale"),473+ }474+ for _, prop := range slices.Concat(p.AudioProperties, p.VideoProperties) {475+ if tag := tags[strings.ToLower(prop)]; tag != "" && !slices.Contains(a.Tags, tag) {476+ a.Tags = append(a.Tags, tag)477+ }478+ }479+ for _, r := range p.Ratings {480+ if a.Rating == "" || strings.Contains(r.Body, "USA") {481+ a.Rating = ratingCode(r.Code)482+ }483+ }484+ d.Airings = append(d.Airings, a)485+ }486+ if c.st.Days[s.StationID] == nil {487+ c.st.Days[s.StationID] = map[string]day{}488+ }489+ c.st.Days[s.StationID][s.Metadata.StartDate] = d490+}491+492+// tags are the guide's names for an airing's sound and picture.493+var tags = map[string]string{494+ "cc": "CC", "stereo": "Stereo", "dvs": "DVS", "sap": "SAP", "dd 5.1": "5.1", "surround": "5.1",495+ "subtitled": "Subtitled", "hdtv": "HD",496+}497+498+// ratingCode writes a rating as the guide does: "TV14" as "TV-14".499+func ratingCode(code string) string {500+ if rest, ok := strings.CutPrefix(code, "TV"); ok && rest != "" && rest[0] != '-' {501+ return "TV-" + rest502+ }503+ return code504+}505+506+// programs fetches the programs aired whose details changed (or were507+// never fetched).508+func (c *client) programs(ctx context.Context) error {509+ want := map[string]bool{}510+ for _, days := range c.st.Days {511+ for _, d := range days {512+ for _, a := range d.Airings {513+ if p, ok := c.st.Programs[a.ProgramID]; !ok || p.MD5 != a.MD5 {514+ want[a.ProgramID] = true515+ }516+ }517+ }518+ }519+ ids := slices.Sorted(func(yield func(string) bool) {520+ for id := range want {521+ if !yield(id) {522+ return523+ }524+ }525+ })526+ for chunk := range slices.Chunk(ids, maxPrograms) {527+ var out []json.RawMessage528+ if err := c.do(ctx, http.MethodPost, "programs", chunk, &out); err != nil {529+ return err530+ }531+ for _, raw := range out {532+ if id, p, ok := parseProgram(raw); ok {533+ c.st.Programs[id] = p534+ }535+ }536+ }537+ return nil538+}539+540+// sdProgram is the part of a /programs entry kept.541+type sdProgram struct {542+ ProgramID string `json:"programID"`543+ Code int `json:"code"`544+ MD5 string `json:"md5"`545+ Titles []struct {546+ Title120 string `json:"title120"`547+ } `json:"titles"`548+ EpisodeTitle150 string `json:"episodeTitle150"`549+ Descriptions struct {550+ Short []sdDescription `json:"description100"`551+ Long []sdDescription `json:"description1000"`552+ } `json:"descriptions"`553+ OriginalAirDate string `json:"originalAirDate"`554+ Genres []string `json:"genres"`555+ EntityType string `json:"entityType"`556+ Cast []struct {557+ BillingOrder string `json:"billingOrder"`558+ Role string `json:"role"`559+ Name string `json:"name"`560+ CharacterName string `json:"characterName"`561+ } `json:"cast"`562+ Metadata []map[string]struct {563+ Season int `json:"season"`564+ Episode int `json:"episode"`565+ } `json:"metadata"`566+ ContentRating []struct {567+ Body string `json:"body"`568+ Code string `json:"code"`569+ } `json:"contentRating"`570+ Movie *struct {571+ Year flexInt `json:"year"`572+ Duration int `json:"duration"`573+ QualityRating []struct {574+ RatingsBody string `json:"ratingsBody"`575+ Rating string `json:"rating"`576+ } `json:"qualityRating"`577+ } `json:"movie"`578+ EventDetails *struct {579+ Teams []struct {580+ Name string `json:"name"`581+ IsHome bool `json:"isHome"`582+ } `json:"teams"`583+ } `json:"eventDetails"`584+ HasImageArtwork bool `json:"hasImageArtwork"`585+}586+587+type sdDescription struct {588+ Language string `json:"descriptionLanguage"`589+ Description string `json:"description"`590+}591+592+// flexInt reads a number that may come as a string.593+type flexInt int594+595+func (n *flexInt) UnmarshalJSON(raw []byte) error {596+ s := strings.Trim(string(raw), `"`)597+ if s == "" || s == "null" {598+ *n = 0599+ return nil600+ }601+ v, err := strconv.Atoi(s)602+ *n = flexInt(v)603+ return err604+}605+606+// parseProgram turns a /programs entry into what's kept; ok is false for607+// an entry that's an error (queued, unknown).608+func parseProgram(raw json.RawMessage) (string, program, bool) {609+ var sp sdProgram610+ if json.Unmarshal(raw, &sp) != nil || sp.ProgramID == "" || sp.Code != 0 || sp.MD5 == "" {611+ return "", program{}, false612+ }613+ p := program{614+ MD5: sp.MD5, EpisodeTitle: sp.EpisodeTitle150, OriginalAirDate: sp.OriginalAirDate,615+ Genres: sp.Genres, Kind: sp.EntityType, Art: sp.HasImageArtwork,616+ Description: cmp.Or(english(sp.Descriptions.Long), english(sp.Descriptions.Short)),617+ }618+ if len(sp.Titles) > 0 {619+ p.Title = sp.Titles[0].Title120620+ }621+ for _, m := range sp.Metadata {622+ if g, ok := m["Gracenote"]; ok {623+ p.Season, p.Episode = g.Season, g.Episode624+ }625+ }626+ for _, r := range sp.ContentRating {627+ if p.Rating == "" || strings.Contains(r.Body, "USA") || strings.Contains(r.Body, "Motion Picture") {628+ p.Rating = ratingCode(r.Code)629+ }630+ }631+ cast := slices.Clone(sp.Cast)632+ slices.SortStableFunc(cast, func(a, b struct {633+ BillingOrder string `json:"billingOrder"`634+ Role string `json:"role"`635+ Name string `json:"name"`636+ CharacterName string `json:"characterName"`637+ }) int {638+ return strings.Compare(a.BillingOrder, b.BillingOrder)639+ })640+ for _, m := range cast {641+ if len(p.Cast) == 6 || m.Name == "" {642+ continue643+ }644+ if m.CharacterName != "" {645+ p.Cast = append(p.Cast, m.Name+" ("+m.CharacterName+")")646+ } else {647+ p.Cast = append(p.Cast, m.Name)648+ }649+ }650+ if sp.Movie != nil {651+ p.Year, p.Runtime = int(sp.Movie.Year), sp.Movie.Duration/60652+ for _, q := range sp.Movie.QualityRating {653+ if v, err := strconv.ParseFloat(q.Rating, 64); err == nil && strings.EqualFold(q.RatingsBody, "Gracenote") {654+ p.Stars = v655+ }656+ }657+ }658+ if sp.EventDetails != nil {659+ for _, t := range sp.EventDetails.Teams {660+ if t.IsHome {661+ p.HomeTeam = t.Name662+ } else {663+ p.AwayTeam = t.Name664+ }665+ }666+ }667+ return sp.ProgramID, p, true668+}669+670+// english picks the English description, else the first.671+func english(ds []sdDescription) string {672+ for _, d := range ds {673+ if strings.HasPrefix(d.Language, "en") {674+ return d.Description675+ }676+ }677+ if len(ds) > 0 {678+ return ds[0].Description679+ }680+ return ""681+}
A internal/sd/sd_test.go +345 -0
@@ -0,0 +1,345 @@
1+package sd2+3+import (4+ "context"5+ "encoding/json"6+ "fmt"7+ "net/http"8+ "net/http/httptest"9+ "strings"10+ "sync"11+ "testing"12+ "time"13+)14+15+// fakeSD is just enough of Schedules Direct: one OTA lineup of two16+// stations, a day of airings each, and the programs they air.17+type fakeSD struct {18+ mu sync.Mutex19+ password string // the SHA-1 it accepts20+ calls map[string]int21+ asked []string // schedule station-days requested22+ programs []string // program IDs requested23+ added bool24+ expire bool // the next request gets 4006 once25+ md5 string26+}27+28+func (f *fakeSD) ServeHTTP(w http.ResponseWriter, r *http.Request) {29+ f.mu.Lock()30+ defer f.mu.Unlock()31+ path := strings.TrimPrefix(r.URL.Path, "/")32+ f.calls[r.Method+" "+path]++33+ reply := func(v any) { _ = json.NewEncoder(w).Encode(v) }34+ fail := func(status, code int, msg string) {35+ w.WriteHeader(status)36+ reply(map[string]any{"code": code, "response": msg, "message": msg})37+ }38+ if !strings.HasPrefix(r.UserAgent(), "airwaves/") {39+ fail(400, 1003, "NO_USER_AGENT")40+ return41+ }42+ if path == "token" {43+ var in struct{ Username, Password string }44+ _ = json.NewDecoder(r.Body).Decode(&in)45+ if in.Password != f.password {46+ fail(400, 4003, "INVALID_USER")47+ return48+ }49+ now := time.Now().Unix()50+ reply(map[string]any{"code": 0, "token": "tok", "tokenExpires": now + 86400, "serverTime": now})51+ return52+ }53+ if r.Header.Get("Token") != "tok" {54+ fail(403, 4006, "TOKEN_EXPIRED")55+ return56+ }57+ if f.expire {58+ f.expire = false59+ fail(403, 4006, "TOKEN_EXPIRED")60+ return61+ }62+ today := time.Now().UTC().Format(time.DateOnly)63+ switch path {64+ case "status":65+ lineups := []map[string]any{}66+ if f.added {67+ lineups = append(lineups, map[string]any{"lineup": "USA-OTA-80302", "modified": "2026-10-01T00:00:00Z"})68+ }69+ reply(map[string]any{"code": 0, "account": map[string]any{"expires": "2026-10-19T00:00:00Z", "maxLineups": 4},70+ "lineups": lineups, "systemStatus": []any{map[string]any{"status": "Online"}}})71+ case "lineups/USA-OTA-80302":72+ if r.Method == http.MethodPut {73+ f.added = true74+ reply(map[string]any{"code": 0, "response": "OK", "changesRemaining": 5})75+ return76+ }77+ reply(map[string]any{78+ "map": []any{79+ map[string]any{"stationID": "20370", "uhfVhf": 36, "atscMajor": 2, "atscMinor": 1},80+ map[string]any{"stationID": "21252", "uhfVhf": 35, "atscMajor": 4, "atscMinor": 1},81+ },82+ "stations": []any{83+ map[string]any{"stationID": "20370", "callsign": "KWGNDT", "name": "KWGNDT (KWGN-DT)", "affiliate": "CW",84+ "stationLogo": []any{map[string]any{"URL": "https://example.com/kwgn-white.png", "category": "white"}, map[string]any{"URL": "https://example.com/kwgn-dark.png", "category": "dark"}}},85+ map[string]any{"stationID": "21252", "callsign": "KCNCDT", "affiliate": "CBS"},86+ },87+ })88+ case "schedules/md5":89+ out := map[string]any{}90+ for _, id := range []string{"20370", "21252"} {91+ out[id] = map[string]any{today: map[string]any{"code": 0, "md5": f.md5}}92+ }93+ out["99999"] = []any{} // a station with nothing94+ reply(out)95+ case "schedules":96+ var in []stationDates97+ _ = json.NewDecoder(r.Body).Decode(&in)98+ var out []any99+ start := time.Now().UTC().Truncate(time.Hour)100+ for _, s := range in {101+ for _, d := range s.Date {102+ f.asked = append(f.asked, s.StationID+"@"+d)103+ progs := []any{104+ map[string]any{"programID": "EP013657560045", "airDateTime": start.Format(time.RFC3339), "duration": 3600, "md5": "p1",105+ "new": true, "audioProperties": []string{"cc", "stereo", "dvs"}, "videoProperties": []string{"hdtv"},106+ "ratings": []any{map[string]any{"body": "USA Parental Rating", "code": "TV14"}}},107+ map[string]any{"programID": "MV000087710000", "airDateTime": start.Add(time.Hour).Format(time.RFC3339), "duration": 7200, "md5": "p2"},108+ }109+ if s.StationID == "21252" {110+ progs = []any{map[string]any{"programID": "SP003190180000", "airDateTime": start.Format(time.RFC3339), "duration": 10800, "md5": "p3", "liveTapeDelay": "Live"}}111+ }112+ out = append(out, map[string]any{"stationID": s.StationID, "programs": progs, "metadata": map[string]any{"md5": f.md5, "startDate": d}})113+ }114+ }115+ reply(out)116+ case "programs":117+ var ids []string118+ _ = json.NewDecoder(r.Body).Decode(&ids)119+ f.programs = append(f.programs, ids...)120+ all := map[string]any{121+ "EP013657560045": map[string]any{"programID": "EP013657560045", "md5": "p1", "titles": []any{map[string]any{"title120": "The Voice"}},122+ "episodeTitle150": "The Knockouts", "originalAirDate": "2026-10-07", "genres": []string{"Reality", "Music"},123+ "descriptions": map[string]any{"description100": []any{map[string]any{"descriptionLanguage": "en", "description": "Short."}},124+ "description1000": []any{map[string]any{"descriptionLanguage": "en", "description": "The knockouts continue, longer."}}},125+ "cast": []any{map[string]any{"billingOrder": "02", "name": "Second"}, map[string]any{"billingOrder": "01", "name": "First", "characterName": "Coach"}},126+ "metadata": []any{map[string]any{"Gracenote": map[string]any{"season": 28, "episode": 12}}}, "entityType": "Episode"},127+ "MV000087710000": map[string]any{"programID": "MV000087710000", "md5": "p2", "titles": []any{map[string]any{"title120": "The Wizard of Oz"}},128+ "entityType": "Movie", "contentRating": []any{map[string]any{"body": "Motion Picture Association", "code": "PG"}},129+ "movie": map[string]any{"year": "1939", "duration": 6120, "qualityRating": []any{map[string]any{"ratingsBody": "Gracenote", "rating": "4"}}}},130+ "SP003190180000": map[string]any{"programID": "SP003190180000", "md5": "p3", "titles": []any{map[string]any{"title120": "NFL Football"}},131+ "entityType": "Sports", "genres": []string{"Sports event", "Football"},132+ "eventDetails": map[string]any{"teams": []any{map[string]any{"name": "Kansas City Chiefs"}, map[string]any{"name": "Denver Broncos", "isHome": true}}}},133+ }134+ var out []any135+ for _, id := range ids {136+ if p, ok := all[id]; ok {137+ out = append(out, p)138+ } else {139+ out = append(out, map[string]any{"programID": id, "code": 6000})140+ }141+ }142+ reply(out)143+ default:144+ fail(404, 1008, "INCORRECT_REQUEST")145+ }146+}147+148+func newSource(t *testing.T) (*Source, *fakeSD) {149+ t.Helper()150+ f := &fakeSD{password: HashPassword("hunter2"), calls: map[string]int{}, md5: "day1"}151+ srv := httptest.NewServer(f)152+ t.Cleanup(srv.Close)153+ s := &Source{Dir: t.TempDir(), ZIP: "80302", HTTP: srv.Client(), UserAgent: "airwaves/test", BaseURL: srv.URL + "/"}154+ return s, f155+}156+157+func (f *fakeSD) count(call string) int {158+ f.mu.Lock()159+ defer f.mu.Unlock()160+ return f.calls[call]161+}162+163+// TestUpdate: a first update logs in, adds the antenna lineup and fetches164+// everything; a second asks what changed and fetches nothing more; the165+// listings come out as the guide has them.166+func TestUpdate(t *testing.T) {167+ s, f := newSource(t)168+ if err := s.SetAccount("viewer", "hunter2"); err != nil {169+ t.Fatal(err)170+ }171+ ctx := context.Background()172+ a, _ := LoadAccount(s.accountPath())173+ s.update(ctx, a)174+ st := s.Status()175+ if !st.OK || st.Error != "" || st.Lineup != "USA-OTA-80302" || st.Stations != 2 || st.Programs == 0 || st.Expires.IsZero() {176+ t.Fatalf("status = %+v", st)177+ }178+ if f.count("POST token") != 1 || f.count("PUT lineups/USA-OTA-80302") != 1 || len(f.programs) != 3 {179+ t.Fatalf("calls %v, programs %v", f.calls, f.programs)180+ }181+182+ // Nothing changed: no new login, no schedules, no programs.183+ asked := len(f.asked)184+ s.update(ctx, a)185+ if f.count("POST token") != 1 || len(f.asked) != asked || len(f.programs) != 3 || f.count("PUT lineups/USA-OTA-80302") != 1 {186+ t.Fatalf("second update: calls %v, asked %d more", f.calls, len(f.asked)-asked)187+ }188+ // A changed day is fetched again, its unchanged programs not.189+ f.mu.Lock()190+ f.md5 = "day2"191+ f.mu.Unlock()192+ s.update(ctx, a)193+ if len(f.asked) == asked || len(f.programs) != 3 {194+ t.Fatalf("changed day: asked %d more, programs %v", len(f.asked)-asked, f.programs)195+ }196+197+ g := s.Guide()198+ if g == nil || len(g.Channels) != 2 || g.Channels[0].Number != "2.1" || g.Channels[0].CallSign != "KWGNDT" || g.Channels[0].Logo != "https://example.com/kwgn-dark.png" {199+ t.Fatalf("guide channels = %+v", g)200+ }201+ ps := g.Programs["20370"]202+ if len(ps) != 2 {203+ t.Fatalf("2.1 programs = %+v", ps)204+ }205+ ep := ps[0]206+ if ep.Title != "The Voice" || ep.EpisodeTitle != "The Knockouts" || ep.Description != "The knockouts continue, longer." ||207+ ep.Season != "28" || ep.Episode != "12" || ep.SeriesID != "SH01365756" || ep.Rating != "TV-14" ||208+ strings.Join(ep.Flags, ",") != "New" || strings.Join(ep.Tags, ",") != "CC,Stereo,DVS,HD" ||209+ strings.Join(ep.Cast, ",") != "First (Coach),Second" || ep.OriginalAirDate != "2026-10-07" || strings.Join(ep.Genres, ",") != "music,Reality" {210+ t.Errorf("episode = %+v", ep)211+ }212+ if mv := ps[1]; mv.Year != "1939" || mv.Runtime != 102 || mv.Stars != 4 || mv.Rating != "PG" || mv.SeriesID != "" || mv.Genres[0] != "movie" {213+ t.Errorf("movie = %+v", mv)214+ }215+ game := g.Programs["21252"][0]216+ if game.HomeTeam != "Denver Broncos" || game.AwayTeam != "Kansas City Chiefs" || game.Flags[0] != "Live" || game.Genres[0] != "sports" {217+ t.Errorf("game = %+v", game)218+ }219+220+ // Kept across a restart.221+ again := &Source{Dir: s.Dir, ZIP: "80302", HTTP: s.HTTP, UserAgent: "airwaves/test", BaseURL: s.BaseURL}222+ if g := again.Guide(); g == nil || len(g.Programs["20370"]) != 2 || again.Status().Lineup != "USA-OTA-80302" {223+ t.Fatalf("after a restart: %+v", again.Status())224+ }225+}226+227+// TestWrongPassword: a refused login stops updating, with no more logins,228+// until the account is saved again.229+func TestWrongPassword(t *testing.T) {230+ s, f := newSource(t)231+ if err := s.SetAccount("viewer", "wrong"); err != nil {232+ t.Fatal(err)233+ }234+ ctx, cancel := context.WithCancel(context.Background())235+ defer cancel()236+ done := make(chan struct{})237+ go func() { s.Run(ctx); close(done) }()238+ for deadline := time.Now().Add(5 * time.Second); s.Status().Stopped == ""; time.Sleep(10 * time.Millisecond) {239+ if time.Now().After(deadline) {240+ t.Fatal("not stopped")241+ }242+ }243+ if st := s.Status(); !strings.Contains(st.Stopped, "user name and password") || st.OK {244+ t.Fatalf("status = %+v", st)245+ }246+ s.Kick()247+ time.Sleep(100 * time.Millisecond)248+ if n := f.count("POST token"); n != 1 {249+ t.Fatalf("%d logins after a refusal", n)250+ }251+ // Saving the account again tries again, once.252+ if err := s.SetAccount("viewer", "hunter2"); err != nil {253+ t.Fatal(err)254+ }255+ for deadline := time.Now().Add(5 * time.Second); !s.Status().OK; time.Sleep(10 * time.Millisecond) {256+ if time.Now().After(deadline) {257+ t.Fatalf("not updated after fixing the password: %+v", s.Status())258+ }259+ }260+ if n := f.count("POST token"); n != 2 {261+ t.Fatalf("%d logins", n)262+ }263+ cancel()264+ <-done265+}266+267+// TestTokenExpired: a token refused mid-update is renewed once, and the268+// request retried.269+func TestTokenExpired(t *testing.T) {270+ s, f := newSource(t)271+ _ = s.SetAccount("viewer", "hunter2")272+ a, _ := LoadAccount(s.accountPath())273+ s.update(context.Background(), a)274+ f.mu.Lock()275+ f.expire = true276+ f.md5 = "day3"277+ f.mu.Unlock()278+ s.update(context.Background(), a)279+ if st := s.Status(); !st.OK || f.count("POST token") != 2 {280+ t.Fatalf("status %+v, %d logins", st, f.count("POST token"))281+ }282+}283+284+// TestNotJSON: a page that isn't JSON (a load balancer's block) stops285+// updating rather than knocking again.286+func TestNotJSON(t *testing.T) {287+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {288+ w.WriteHeader(http.StatusForbidden)289+ fmt.Fprint(w, "<html>blocked</html>")290+ }))291+ defer srv.Close()292+ s := &Source{Dir: t.TempDir(), ZIP: "80302", HTTP: srv.Client(), UserAgent: "airwaves/test", BaseURL: srv.URL + "/"}293+ _ = s.SetAccount("viewer", "hunter2")294+ a, _ := LoadAccount(s.accountPath())295+ s.update(context.Background(), a)296+ if st := s.Status(); !strings.Contains(st.Stopped, "other than JSON") {297+ t.Fatalf("status = %+v", st)298+ }299+}300+301+func TestPasswordHash(t *testing.T) {302+ if got := HashPassword("password"); got != "5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8" {303+ t.Fatalf("SHA-1 = %s", got)304+ }305+ s, _ := newSource(t)306+ _ = s.SetAccount("viewer", "hunter2")307+ raw, _ := LoadAccount(s.accountPath())308+ if raw.PasswordSHA1 != HashPassword("hunter2") || strings.Contains(fmt.Sprint(raw), "hunter2") {309+ t.Fatalf("kept %+v", raw)310+ }311+ // An empty password keeps the saved one for the same user.312+ if err := s.SetAccount("viewer", ""); err != nil {313+ t.Fatal(err)314+ }315+ if a, _ := LoadAccount(s.accountPath()); a.PasswordSHA1 != HashPassword("hunter2") {316+ t.Fatal("the saved password was dropped")317+ }318+ if err := s.SetAccount("someone", ""); err == nil {319+ t.Fatal("a new user without a password was taken")320+ }321+}322+323+// TestRunOncePerDay: after an update, Run waits for the next day, however324+// short the listings are (here, a day of them).325+func TestRunOncePerDay(t *testing.T) {326+ s, f := newSource(t)327+ _ = s.SetAccount("viewer", "hunter2")328+ ctx, cancel := context.WithCancel(context.Background())329+ done := make(chan struct{})330+ go func() { s.Run(ctx); close(done) }()331+ for deadline := time.Now().Add(5 * time.Second); !s.Status().OK; time.Sleep(10 * time.Millisecond) {332+ if time.Now().After(deadline) {333+ t.Fatal("no update")334+ }335+ }336+ time.Sleep(300 * time.Millisecond)337+ cancel()338+ <-done339+ if n := f.count("POST schedules/md5"); n != 1 {340+ t.Fatalf("%d updates, want one", n)341+ }342+ if next := s.Status().Next; time.Until(next) < 12*time.Hour {343+ t.Fatalf("next update %s", next)344+ }345+}
A internal/sd/source.go +292 -0
@@ -0,0 +1,292 @@
1+// Package sd keeps a ZIP code's over-the-air listings from Schedules Direct2+// (schedulesdirect.org), the licensed source of Gracenote's TV data for3+// personal use: two weeks of schedules with descriptions, cast, original4+// air dates, ratings, movie runtimes and star ratings, and sports teams.5+//6+// It is a careful client. It logs in only when its token runs out, asks for7+// what changed since its last update and nothing else, updates once a day,8+// and stops trying altogether when the account is refused, until the9+// account is changed: Schedules Direct blocks clients that keep failing.10+package sd11+12+import (13+ "context"14+ "errors"15+ "log"16+ "math/rand/v2"17+ "net/http"18+ "path/filepath"19+ "sync"20+ "time"21+22+ "airwaves/internal/guide"23+)24+25+// Days is how many days of listings are kept.26+const Days = 1427+28+// Status is how the listings stand, for the admin page.29+type Status struct {30+ // User is the account's user name; "" with no account set.31+ User string `json:"user,omitempty"`32+ // OK is set when the last update worked.33+ OK bool `json:"ok"`34+ // Stopped says why updating stopped until the account is changed:35+ // a refused login, an expired or locked account.36+ Stopped string `json:"stopped,omitempty"`37+ // Error is the last update's failure, when it failed.38+ Error string `json:"error,omitempty"`39+ // Expires is when the Schedules Direct account runs out.40+ Expires time.Time `json:"expires,omitzero"`41+ // Lineup is the account's antenna lineup, "USA-OTA-80302".42+ Lineup string `json:"lineup,omitempty"`43+ Stations int `json:"stations,omitempty"`44+ Programs int `json:"programs,omitempty"`45+ // Updated is when the listings were last brought up to date, Through46+ // how far ahead they reach, and Next when they're next updated.47+ Updated time.Time `json:"updated,omitzero"`48+ Through time.Time `json:"through,omitzero"`49+ Next time.Time `json:"next,omitzero"`50+ // Updating is set while an update runs.51+ Updating bool `json:"updating,omitempty"`52+}53+54+// Source keeps the listings of ZIP's antenna lineup up to date.55+type Source struct {56+ // Dir keeps the account and what was fetched.57+ Dir string58+ // ZIP is the antenna's.59+ ZIP string60+ // HTTP makes the requests; UserAgent names this program and its61+ // version, as Schedules Direct requires.62+ HTTP *http.Client63+ UserAgent string64+ // BaseURL is the API's; API by default.65+ BaseURL string66+ // OnUpdate, when set, is told of new listings.67+ OnUpdate func()68+69+ once sync.Once70+ updates chan struct{}71+72+ mu sync.Mutex73+ status Status74+ st *state75+ g *guide.Guide76+}77+78+func (s *Source) accountPath() string { return filepath.Join(s.Dir, "account.json") }79+func (s *Source) statePath() string { return filepath.Join(s.Dir, "listings.json") }80+81+func (s *Source) init() {82+ s.once.Do(func() {83+ s.updates = make(chan struct{}, 1)84+ a, err := LoadAccount(s.accountPath())85+ if err != nil {86+ log.Printf("listings: %v", err)87+ }88+ st, err := loadState(s.statePath())89+ if err != nil {90+ log.Printf("listings: %v; starting afresh", err)91+ }92+ s.mu.Lock()93+ defer s.mu.Unlock()94+ s.st = st95+ s.status.User = a.User96+ s.status.Stopped = st.Stopped97+ if st.Updated.After(time.Time{}) {98+ s.status.OK = st.Stopped == ""99+ s.status.Updated, s.status.Lineup = st.Updated, st.Lineup100+ s.status.Expires = st.Expires101+ }102+ s.g = st.guide(s.ZIP, time.Now())103+ s.count()104+ })105+}106+107+// count notes the listings' size in the status. Called with s.mu held.108+func (s *Source) count() {109+ s.status.Stations, s.status.Programs, s.status.Through = 0, 0, time.Time{}110+ if s.g == nil {111+ return112+ }113+ s.status.Stations = len(s.g.Channels)114+ for _, ps := range s.g.Programs {115+ s.status.Programs += len(ps)116+ if n := len(ps); n > 0 && ps[n-1].End.After(s.status.Through) {117+ s.status.Through = ps[n-1].End118+ }119+ }120+}121+122+// Status reports how the listings stand.123+func (s *Source) Status() Status {124+ s.init()125+ s.mu.Lock()126+ defer s.mu.Unlock()127+ return s.status128+}129+130+// Guide is the listings, nil while there are none: no account, or none131+// fetched yet.132+func (s *Source) Guide() *guide.Guide {133+ s.init()134+ s.mu.Lock()135+ defer s.mu.Unlock()136+ if s.status.User == "" {137+ return nil138+ }139+ return s.g140+}141+142+// SetAccount keeps a new login and updates with it at once; an empty user143+// removes the account and its listings. An empty password keeps the saved144+// one for the same user.145+func (s *Source) SetAccount(user, password string) error {146+ s.init()147+ old, _ := LoadAccount(s.accountPath())148+ a := Account{User: user, PasswordSHA1: HashPassword(password)}149+ switch {150+ case user == "":151+ a = Account{}152+ case password == "" && user == old.User:153+ a.PasswordSHA1 = old.PasswordSHA1154+ case password == "":155+ return errors.New("enter the Schedules Direct password")156+ }157+ if err := SaveAccount(s.accountPath(), a); err != nil {158+ return err159+ }160+ s.mu.Lock()161+ // A new login: try again, with a token of its own.162+ s.st.Stopped, s.st.Token, s.st.TokenExpires = "", "", time.Time{}163+ s.status = Status{User: a.User}164+ if a.User == "" {165+ s.st = &state{}166+ s.g = nil167+ }168+ err := s.st.save(s.statePath())169+ s.mu.Unlock()170+ if a.Set() {171+ s.Kick()172+ } else if s.OnUpdate != nil {173+ s.OnUpdate() // back to the public grid174+ }175+ return err176+}177+178+// Kick asks for an update now.179+func (s *Source) Kick() {180+ s.init()181+ select {182+ case s.updates <- struct{}{}:183+ default:184+ }185+}186+187+// Run keeps the listings up to date until ctx ends: once a day, at a time188+// of its own (Schedules Direct asks clients not to all come at once), and189+// on Kick. After a failure it tries again in an hour; a refused account190+// stops it until the account changes.191+func (s *Source) Run(ctx context.Context) {192+ s.init()193+ // The same minute of the day every day, picked once per server.194+ offset := time.Duration(rand.IntN(24*60)) * time.Minute195+ for {196+ a, _ := LoadAccount(s.accountPath())197+ next := time.Time{}198+ if a.Set() {199+ s.mu.Lock()200+ stopped, updated := s.st.Stopped, s.st.Updated201+ s.mu.Unlock()202+ if stopped == "" {203+ due := updated.Add(24 * time.Hour)204+ if updated.IsZero() || !time.Now().Before(due) {205+ s.update(ctx, a)206+ continue207+ }208+ // The server's own minute of the day, once a day has passed.209+ if next = due.Truncate(24 * time.Hour).Add(offset); next.Before(due) {210+ next = next.Add(24 * time.Hour)211+ }212+ }213+ }214+ s.mu.Lock()215+ s.status.Next = next216+ s.mu.Unlock()217+ var wake <-chan time.Time218+ if !next.IsZero() {219+ wake = time.After(time.Until(next))220+ }221+ select {222+ case <-ctx.Done():223+ return224+ case <-s.updates:225+ s.mu.Lock()226+ stopped := s.st.Stopped227+ s.mu.Unlock()228+ // A stopped account waits for SetAccount, which clears it.229+ if a, _ := LoadAccount(s.accountPath()); a.Set() && stopped == "" {230+ s.update(ctx, a)231+ }232+ case <-wake:233+ }234+ }235+}236+237+// update brings the listings up to date with a, and notes how it went.238+// After a failure that isn't the account's, it waits an hour.239+func (s *Source) update(ctx context.Context, a Account) {240+ // This update answers any Kick waiting.241+ select {242+ case <-s.updates:243+ default:244+ }245+ s.mu.Lock()246+ s.status.Updating = true247+ st := s.st.clone()248+ s.mu.Unlock()249+250+ c := &client{base: s.BaseURL, http: s.HTTP, ua: s.UserAgent, account: a, st: st}251+ err := c.update(ctx, s.ZIP, time.Now())252+ var stop *StopError253+ if errors.As(err, &stop) {254+ st.Stopped = stop.Error()255+ }256+ if err == nil {257+ st.Updated = time.Now()258+ }259+ if serr := st.save(s.statePath()); serr != nil {260+ log.Printf("listings: %v", serr)261+ }262+263+ s.mu.Lock()264+ s.st = st265+ s.status.Updating = false266+ s.status.Stopped, s.status.Lineup, s.status.Expires = st.Stopped, st.Lineup, st.Expires267+ if err == nil {268+ s.status.OK, s.status.Error, s.status.Updated = true, "", st.Updated269+ s.g = st.guide(s.ZIP, time.Now())270+ s.count()271+ } else {272+ s.status.OK, s.status.Error = false, err.Error()273+ }274+ s.mu.Unlock()275+276+ if err == nil && s.OnUpdate != nil {277+ s.OnUpdate()278+ }279+ switch {280+ case err == nil:281+ log.Printf("listings: up to date from Schedules Direct, %d programs on %d stations", s.status.Programs, s.status.Stations)282+ case stop != nil:283+ log.Printf("listings: stopped until the account is changed: %v", err)284+ default:285+ log.Printf("listings: %v; trying again in an hour", err)286+ select {287+ case <-ctx.Done():288+ case <-time.After(time.Hour):289+ case <-s.updates:290+ }291+ }292+}
A internal/sd/state.go +307 -0
@@ -0,0 +1,307 @@
1+package sd2+3+import (4+ "cmp"5+ "encoding/json"6+ "errors"7+ "fmt"8+ "io/fs"9+ "maps"10+ "os"11+ "slices"12+ "strconv"13+ "strings"14+ "time"15+16+ "airwaves/internal/guide"17+)18+19+// state is what an update leaves for the next: the token, the lineup, and20+// the listings by station and day with the checksums Schedules Direct21+// gave them, so only what changed is fetched again.22+type state struct {23+ Token string `json:"token,omitempty"`24+ TokenExpires time.Time `json:"tokenExpires,omitzero"`25+ // Stopped says why updating stopped until the account changes.26+ Stopped string `json:"stopped,omitempty"`27+ Expires time.Time `json:"expires,omitzero"` // the account28+ Updated time.Time `json:"updated,omitzero"`29+30+ Lineup string `json:"lineup,omitempty"`31+ LineupModified string `json:"lineupModified,omitempty"`32+ Stations []station `json:"stations,omitempty"`33+ // Days holds each station's schedule by date ("2026-10-12", UTC).34+ Days map[string]map[string]day `json:"days,omitempty"`35+ // Programs are the programs aired, by program ID.36+ Programs map[string]program `json:"programs,omitempty"`37+ // Art is the best picture of each series or program, by the ID it was38+ // asked for: a program's, or its series' first 10 characters.39+ Art map[string]string `json:"art,omitempty"`40+}41+42+type station struct {43+ ID string `json:"id"`44+ Number string `json:"number"` // "2.1"45+ CallSign string `json:"callSign"`46+ Name string `json:"name,omitempty"`47+ Affiliate string `json:"affiliate,omitempty"`48+ Logo string `json:"logo,omitempty"`49+}50+51+type day struct {52+ MD5 string `json:"md5"`53+ Airings []airing `json:"airings"`54+}55+56+type airing struct {57+ ProgramID string `json:"programId"`58+ Start time.Time `json:"start"`59+ Seconds int `json:"seconds"`60+ MD5 string `json:"md5"`61+ New bool `json:"new,omitempty"`62+ Live bool `json:"live,omitempty"`63+ Premiere bool `json:"premiere,omitempty"`64+ Finale bool `json:"finale,omitempty"`65+ Tags []string `json:"tags,omitempty"` // CC, Stereo, DVS...66+ Rating string `json:"rating,omitempty"`67+}68+69+type program struct {70+ MD5 string `json:"md5"`71+ Title string `json:"title"`72+ EpisodeTitle string `json:"episodeTitle,omitempty"`73+ Description string `json:"description,omitempty"`74+ OriginalAirDate string `json:"originalAirDate,omitempty"`75+ Season int `json:"season,omitempty"`76+ Episode int `json:"episode,omitempty"`77+ Genres []string `json:"genres,omitempty"`78+ Kind string `json:"kind,omitempty"` // Show, Episode, Sports, Movie79+ Cast []string `json:"cast,omitempty"`80+ Year int `json:"year,omitempty"`81+ Runtime int `json:"runtime,omitempty"` // minutes82+ Stars float64 `json:"stars,omitempty"`83+ Rating string `json:"rating,omitempty"`84+ HomeTeam string `json:"homeTeam,omitempty"`85+ AwayTeam string `json:"awayTeam,omitempty"`86+ Art bool `json:"art,omitempty"` // Schedules Direct has pictures of it87+}88+89+const stateVersion = 190+91+type stateFile struct {92+ Version int `json:"version"`93+ *state94+}95+96+// loadState reads the state kept at path; a missing or unreadable file is97+// a fresh start (an unreadable one is set aside as path.bad).98+func loadState(path string) (*state, error) {99+ f := stateFile{state: &state{}}100+ raw, err := os.ReadFile(path)101+ if errors.Is(err, fs.ErrNotExist) {102+ return f.state, nil103+ }104+ if err == nil {105+ err = json.Unmarshal(raw, &f)106+ }107+ if err != nil || f.Version != stateVersion {108+ _ = os.Rename(path, path+".bad")109+ if err == nil {110+ err = fmt.Errorf("listings saved by another version")111+ }112+ return &state{}, fmt.Errorf("%s: %w", path, err)113+ }114+ return f.state, nil115+}116+117+func (st *state) save(path string) error {118+ raw, err := json.Marshal(stateFile{Version: stateVersion, state: st})119+ if err != nil {120+ return err121+ }122+ return writeFile(path, raw, 0o600)123+}124+125+// clone copies st for an update to change, so a failed update leaves the126+// listings as they were.127+func (st *state) clone() *state {128+ c := *st129+ c.Stations = slices.Clone(st.Stations)130+ c.Days = make(map[string]map[string]day, len(st.Days))131+ for id, days := range st.Days {132+ c.Days[id] = maps.Clone(days)133+ }134+ c.Programs = maps.Clone(st.Programs)135+ c.Art = maps.Clone(st.Art)136+ if c.Programs == nil {137+ c.Programs = map[string]program{}138+ }139+ if c.Art == nil {140+ c.Art = map[string]string{}141+ }142+ return &c143+}144+145+// prune drops days before yesterday, and programs and pictures nothing146+// airs any more.147+func (st *state) prune(now time.Time) {148+ oldest := now.UTC().AddDate(0, 0, -1).Format(time.DateOnly)149+ used := map[string]bool{}150+ for id, days := range st.Days {151+ for date, d := range days {152+ if date < oldest {153+ delete(days, date)154+ continue155+ }156+ for _, a := range d.Airings {157+ used[a.ProgramID] = true158+ }159+ }160+ if len(days) == 0 {161+ delete(st.Days, id)162+ }163+ }164+ for id := range st.Programs {165+ if !used[id] {166+ delete(st.Programs, id)167+ }168+ }169+ for id := range st.Art {170+ if !used[id] && !slices.ContainsFunc(slices.Collect(maps.Keys(used)), func(p string) bool { return seriesKey(p) == id }) {171+ delete(st.Art, id)172+ }173+ }174+}175+176+// seriesKey is what a program's series pictures are kept under: its ID's177+// first 10 characters.178+func seriesKey(programID string) string {179+ if len(programID) < 10 {180+ return programID181+ }182+ return programID[:10]183+}184+185+// SeriesID is a program's series as the guide's grid gives it, so series186+// recording rules carry over: "SH" and the series number, for episodes and187+// shows; "" for movies and the like.188+func SeriesID(programID string) string {189+ if len(programID) < 10 || !strings.HasPrefix(programID, "EP") && !strings.HasPrefix(programID, "SH") {190+ return ""191+ }192+ return "SH" + programID[2:10]193+}194+195+// imageBase is where Gracenote's pictures are served, the same files196+// Schedules Direct names.197+const imageBase = "https://zpmc.tmsimg.com/assets/"198+199+// guide is the listings as the guide has them, from a day before now on;200+// nil while there are none.201+func (st *state) guide(zip string, now time.Time) *guide.Guide {202+ if len(st.Stations) == 0 || len(st.Days) == 0 {203+ return nil204+ }205+ g := &guide.Guide{Lineup: st.Lineup, Fetched: st.Updated, Programs: map[string][]guide.Program{}}206+ from := now.Add(-24 * time.Hour)207+ for _, sta := range st.Stations {208+ g.Channels = append(g.Channels, guide.Channel{209+ ID: sta.ID, Number: sta.Number, CallSign: sta.CallSign, Network: guide.NetworkName(sta.Affiliate), Logo: sta.Logo,210+ })211+ for _, d := range st.Days[sta.ID] {212+ for _, a := range d.Airings {213+ end := a.Start.Add(time.Duration(a.Seconds) * time.Second)214+ if !end.After(from) {215+ continue216+ }217+ g.Programs[sta.ID] = append(g.Programs[sta.ID], st.program(a, end))218+ }219+ }220+ }221+ slices.SortFunc(g.Channels, func(a, b guide.Channel) int { return guide.CompareNumbers(a.Number, b.Number) })222+ for id, ps := range g.Programs {223+ slices.SortFunc(ps, func(a, b guide.Program) int { return a.Start.Compare(b.Start) })224+ // Days overlap at midnight: an airing listed on both stays once.225+ g.Programs[id] = slices.CompactFunc(ps, func(a, b guide.Program) bool { return a.Start.Equal(b.Start) })226+ if n := len(ps); n > 0 {227+ g.Start = cmp.Or(minTime(g.Start, ps[0].Start), ps[0].Start)228+ g.End = maxTime(g.End, ps[n-1].End)229+ }230+ }231+ return g232+}233+234+func (st *state) program(a airing, end time.Time) guide.Program {235+ p := st.Programs[a.ProgramID]236+ out := guide.Program{237+ Start: a.Start, End: end, Title: cmp.Or(p.Title, "To be announced"), EpisodeTitle: p.EpisodeTitle,238+ Description: p.Description, Rating: cmp.Or(a.Rating, p.Rating), Tags: a.Tags,239+ ProgramID: a.ProgramID, SeriesID: SeriesID(a.ProgramID), OriginalAirDate: p.OriginalAirDate, Cast: p.Cast,240+ Runtime: p.Runtime, Stars: p.Stars, HomeTeam: p.HomeTeam, AwayTeam: p.AwayTeam, Genres: genres(a.ProgramID, p),241+ }242+ if p.Season > 0 && p.Episode > 0 {243+ out.Season, out.Episode = strconv.Itoa(p.Season), strconv.Itoa(p.Episode)244+ }245+ if p.Year > 0 {246+ out.Year = strconv.Itoa(p.Year)247+ }248+ for flag, on := range map[string]bool{"New": a.New, "Live": a.Live, "Premiere": a.Premiere, "Finale": a.Finale} {249+ if on {250+ out.Flags = append(out.Flags, flag)251+ }252+ }253+ slices.Sort(out.Flags)254+ if img := cmp.Or(st.Art[a.ProgramID], st.Art[seriesKey(a.ProgramID)]); img != "" {255+ out.Image = imageBase + img + "?w=360"256+ }257+ return out258+}259+260+// filterGenres maps Schedules Direct's genres to the guide's filters.261+var filterGenres = map[string][]string{262+ "news": {"news"}, "sports event": {"sports"}, "sports non-event": {"sports"}, "sports talk": {"sports"},263+ "children": {"kids", "family"}, "family": {"family"}, "animated": {"family"}, "talk": {"talk"},264+ "documentary": {"documentary"}, "music": {"music"}, "musical": {"music"},265+}266+267+// genres are a program's genres: the guide's filter keys ("sports",268+// "movie", "news"...), then the rest as Schedules Direct names them.269+func genres(id string, p program) []string {270+ var out []string271+ seen := map[string]bool{}272+ add := func(g string) {273+ if k := strings.ToLower(g); g != "" && !seen[k] {274+ seen[k] = true275+ out = append(out, g)276+ }277+ }278+ if strings.HasPrefix(id, "MV") || p.Kind == "Movie" {279+ add("movie")280+ }281+ if strings.HasPrefix(id, "SP") || p.Kind == "Sports" {282+ add("sports")283+ }284+ for _, g := range p.Genres {285+ for _, k := range filterGenres[strings.ToLower(g)] {286+ add(k)287+ }288+ }289+ for _, g := range p.Genres {290+ add(g)291+ }292+ return out293+}294+295+func minTime(a, b time.Time) time.Time {296+ if a.IsZero() || b.Before(a) {297+ return b298+ }299+ return a300+}301+302+func maxTime(a, b time.Time) time.Time {303+ if b.After(a) {304+ return b305+ }306+ return a307+}
M internal/service/antenna.go +15 -6
M internal/service/hdhr.go +4 -1
M internal/service/service.go +30 -1
@@ -33,6 +33,7 @@ import (
3333 "airwaves/internal/hdhr"3434 "airwaves/internal/lineup"3535 "airwaves/internal/phase"36+ "airwaves/internal/sd"3637 "airwaves/internal/signal"3738 "airwaves/internal/store"3839 "airwaves/internal/stream"
@@ -152,6 +153,9 @@ type Options struct {
152153 // the recorded files; recording needs both.153154 DVRPath string154155 RecordingsDir string156+ // ListingsDir keeps a Schedules Direct account and its listings,157+ // which replace Gracenote's public grid once there's an account.158+ ListingsDir string155159 // NoAntenna serves custom channels only: no tuner, lineup or listings156160 // from the ZIP code, reception reports or recording.157161 // The location (Config) is then only for the weather, and there's none
@@ -199,6 +203,7 @@ type Service struct {
199203 streamErr error200204 tuners *tuner.Manager201205 dvr *dvr.Manager206+ listings *sd.Source202207 weather *weather.Source203208 // wx is the weather channel, when there's a WeatherStar display;204209 // wxStreams when it can be rendered to video for HDHomeRun clients.
@@ -266,6 +271,17 @@ func New(opt Options) (*Service, error) {
266271 s.weather = weather.NewSource(s.http)267272 s.streams, s.streamErr = stream.NewServer(false)268273 274+ if !opt.NoAntenna && opt.ListingsDir != "" {275+ s.listings = &sd.Source{276+ Dir: opt.ListingsDir, ZIP: s.cfg.ZIP, HTTP: s.http, UserAgent: "airwaves/" + Version,277+ // New listings make the next snapshot rebuild with them.278+ OnUpdate: func() {279+ s.mu.Lock()280+ s.built = time.Time{}281+ s.mu.Unlock()282+ },283+ }284+ }269285 if !opt.NoAntenna {270286 s.tuners = tuner.NewManager()271287 if opt.DVRPath != "" && opt.RecordingsDir != "" {
@@ -308,6 +324,10 @@ func New(opt Options) (*Service, error) {
308324 // Library is the folder channels, or nil when there's no channels folder.309325 func (s *Service) Library() *vchan.Library { return s.library }310326 327+// Listings is the Schedules Direct account and its listings, or nil on a328+// server without an antenna.329+func (s *Service) Listings() *sd.Source { return s.listings }330+311331 // WeatherChannel is the weather channel, or nil when there's no WeatherStar312332 // display.313333 func (s *Service) WeatherChannel() *vchan.Weather { return s.wx }
@@ -485,7 +505,10 @@ func (s *Service) Snapshot(ctx context.Context, refresh bool) (*Snapshot, error)
485505 if len(a.Channels) == 0 && a.Tuner.Reason != "" {486506 out.Report.Warnings = append(slices.Clone(out.Report.Warnings), "No antenna channels: "+a.Tuner.Reason)487507 }488- out.Guide = listedOnly(snap.Guide, a.Channels)508+ s.mu.Lock()509+ hours := s.cfg.GuideHours510+ s.mu.Unlock()511+ out.Guide = listedOnly(snap.Guide, a.Channels, time.Now().Add(-time.Hour), time.Now().Add(time.Duration(hours)*time.Hour))489512 out.Antenna = a490513 return out, nil491514 }
@@ -624,6 +647,9 @@ func (s *Service) snapshot(ctx context.Context, refresh bool) (*Snapshot, error)
624647 625648 // The records only: no terrain, so no estimates.626649 b := &lineup.Builder{HTTP: s.http, Cache: s.cache, Progress: s.opt.Progress}650+ if s.listings != nil {651+ b.Listings = s.listings.Guide652+ }627653 opt := lineup.Options{ZIP: cfg.ZIP, RadiusKm: cfg.RadiusKm, GuideHours: cfg.GuideHours, Refresh: refresh}628654 if cfg.Lat != 0 && cfg.Lon != 0 {629655 opt.Point = &geo.Point{Lat: cfg.Lat, Lon: cfg.Lon}
@@ -964,6 +990,9 @@ func (s *Service) recordingChannels(ctx context.Context) (*guide.Guide, []lineup
964990 // tuner and keeps up with it, measures the signal while tuners are in use,965991 // and makes recordings.966992 func (s *Service) Run(ctx context.Context) {993+ if s.listings != nil {994+ go s.listings.Run(ctx)995+ }967996 if s.tuners != nil {968997 s.findTuner(ctx)969998 go s.sample(ctx)