Adding a show waited for yt-dlp's one listing job at a time, which the YouTube channels hold for minutes while they list a whole catalog for their schedules. Its two minutes ran out while it waited, so it failed the moment its turn came, and pressing Add again only queued more. Now adding a show lists it at once, a few seconds' work, as a search does. The background listings and lookups still take turns, but their time starts when their turn does, so a busy channel no longer fails them. Podcasts has Refresh under the shows in the rail (also R), for someone who wants the newest now rather than at the next listing, every 6 hours while YouTube's feeds are down. The server lists every show again, a few at once, and the app says how many episodes are new and when every show was last checked. In Settings, Add says it's still adding when pressed again, a channel found says how many subscribers it has as YouTube does (3.39M), and one already a show says Added. Names from YouTube lose the spaces around them.
- commit
- 7c7538696d7df66799a95ff8c3c2fe3ab17da0e8
- parent
- 6129c5cc215d1cf96465e8d95f51f0b14a43c950
- author
- FelineStateMachine <226808055+FelineStateMachine@users.noreply.github.com>
- kind
- feat risk score 1.9 of 3, Jev's estimate
| M | README.md | 11 | ++- |
| M | app.go | 9 | +++ |
| M | frontend/ | 8 | ++- |
| M | frontend/ | 104 | +++++++++++++++++++++++++++--- |
| M | frontend/ | 6 | ++ |
| M | frontend/ | 1 | + |
| M | frontend/ | 2 | + |
| M | frontend/ | 4 | + |
| M | frontend/ | 2 | + |
| M | internal/ | 4 | + |
| M | internal/ | 9 | +++ |
| M | internal/ | 6 | ++ |
| M | internal/ | 21 | ++++-- |
| M | internal/ | 52 | ++++++++++++++- |
| M | internal/ | 54 | ++++++++++++--- |
| M | internal/ | 10 | +++ |
| M | internal/ | 6 | +- |
@@ -165,7 +165,7 @@ had.
165165 | --- | --- |166166 | G | Guide: arrows move, and Up and Down wrap round the channels. The filters (Favorites, Sports...; also `[` `]`) are Down from the top bar, which is Left from the earliest program. Enter watches what's on, or shows a program to come's actions; hold Enter, or I, for any program's: watch, record (which opens to this one, every episode or new only), favorite, hide. Home jumps to now, R / Shift R / N record, Shift F favorite, H hide (U undoes) |167167 | D | Recordings: Left (or `[` `]`) for Library, Coming up, Series. Enter shows a recording's actions: resume, from the start, watched, delete (press twice); for one recording now, watch from the start, 30 more minutes, stop (keeping it) and delete; Shift Enter starts over, W marks watched, Delete twice removes; on a series K sets how many to keep, N new episodes only |168-| P | Podcasts: video podcasts from YouTube, on demand. The rail has Latest (every show's newest), Continue and each show; Enter shows an episode's actions: play or resume, from the start, watched. Shows are added in Settings, Podcasts, by search or a channel's or playlist's address |168+| P | Podcasts: video podcasts from YouTube, on demand. The rail has Latest (every show's newest), Continue and each show, and under them Refresh (also R), which looks for new episodes now; Enter shows an episode's actions: play or resume, from the start, watched. Shows are added in Settings, Podcasts, by search or a channel's or playlist's address |169169 | W | Weather: current conditions, alerts, the next 24 hours, 7-day, radar loop and satellite; Up and Down move through them, Left to watch the weather channel |170170 | `,` or S | More: Settings, a tab for each area in the rail (Display, Watching, Channels, Reception, Recordings, Server, About). Up and Down pick a tab, Enter goes in, Back returns to the tabs; Page Up and Down (a controller's LB and RB, also `[` `]`) change tab |171171 | A | Settings' Reception tab: what the tuners measured, by RF channel (the ones the scan found first), with the transmitters licensed on each. Up and Down pick one, Enter watches its first channel; above them the tuner, Measure now, which reads every RF channel on a free tuner and shows its progress, and Scan for channels (press twice) |@@ -289,7 +289,7 @@ come, on now or not yet processed, are left out.
289289 each was watched (and any older ones part watched), backed up like the recordings:290290 291291 - every 6 hours it lists the show's newest 40 videos with yt-dlp, and every 30 minutes reads292- its channel's feed for new uploads in between;292+ its channel's feed for new uploads in between; Refresh, in the app, lists every show now;293293 - a video new to a show is looked up (its length, date, description and picture), a dozen294294 an update, a few seconds apart, so YouTube isn't asked too much;295295 - the newest unwatched episode of each show has where it streams from found ahead, so it@@ -299,9 +299,14 @@ Nothing is downloaded: an episode streams from YouTube while it's watched, H.264
299299 1080p with English sound, its picture copied rather than transcoded, and played like a300300 recording, from where it was left. One show failing to list says so on that show alone.301301 302+Listing a show takes a few seconds. Adding one, or Refresh, does it at once; the background303+listings and lookups take turns with the YouTube channels' jobs, one yt-dlp job at a time,304+since listing a channel's whole catalog for its schedule can take minutes.305+302306 | | |303307 | --- | --- |304-| `GET /api/podcasts` | `{"shows": [{"id", "title", "url", "thumbnail", "minMinutes", "error", "episodes": [{"id", "showId", "title", "description", "published", "duration", "thumbnail", "position", "watched"}]}], "updating"}`, episodes newest first |308+| `GET /api/podcasts` | `{"shows": [{"id", "title", "url", "thumbnail", "minMinutes", "error", "episodes": [{"id", "showId", "title", "description", "published", "duration", "thumbnail", "position", "watched"}]}], "updating", "listed"}`, episodes newest first; `listed` is since when every show has been listed |309+| `POST /api/podcasts/refresh` | Lists every show now, a few at once, and replies as `GET /api/podcasts`; new episodes are looked up after |305310 | `GET /api/podcasts/search?q=` | YouTube channels: `[{"id", "title", "url", "thumbnail", "description", "subscribers"}]` |306311 | `POST /api/podcasts/shows` | `{"url", "minMinutes"}`: adds a show, listed before the reply, which is the show; 400 for what isn't one |307312 | `DELETE /api/podcasts/shows/{id}` | Takes a show out |@@ -347,6 +347,15 @@ func (a *App) Podcasts() (*podcast.State, error) {
347347 return c.Podcasts(a.ctx)348348 }349349 350+// RefreshPodcasts lists every show again now, for the newest episodes.351+func (a *App) RefreshPodcasts() (*podcast.State, error) {352+ c, _, err := a.server()353+ if err != nil {354+ return nil, err355+ }356+ return c.RefreshPodcasts(a.ctx)357+}358+350359 // SearchPodcasts finds YouTube channels to add as shows.351360 func (a *App) SearchPodcasts(query string) ([]podcast.Found, error) {352361 c, _, err := a.server()@@ -137,7 +137,13 @@
137137 <!-- Podcasts, an app: video podcasts from YouTube, played on demand, laid138138 out as Recordings' library is (js/podcasts.js). -->139139 <section id="podcasts" class="view railed" aria-label="Podcasts">140- <nav class="rail"><div class="rail-menu p-menu" data-f="menu"></div></nav>140+ <nav class="rail">141+ <div class="rail-menu p-menu" data-f="menu"></div>142+ <div class="p-refresh" hidden>143+ <button type="button" class="btn ghost fx" data-f="refresh">Refresh</button>144+ <div class="p-rnote" data-f="rnote"></div>145+ </div>146+ </nav>141147 <div class="main p-main">142148 <article class="r-detail" data-f="detail"></article>143149 <div class="r-grid" data-f="grid"></div>@@ -8,9 +8,10 @@
88 // show's newest), Continue (episodes part watched) and each show; beside it99 // the episode picked, above a grid of the list's episodes. Its parts take1010 // the focus in turn (data-zone): the grid, the rail's menu (Left from the11-// grid's left edge), and the episode's actions (OK): Play or Resume, From12-// the start, Watched. Back goes from the actions to the grid.13-const podUI = { zone: 'grid', act: '', list: 'latest', sel: {}, at: 0, timer: 0, searching: false };11+// grid's left edge) and Refresh under it, and the episode's actions (OK):12+// Play or Resume, From the start, Watched. Back goes from the actions to13+// the grid.14+const podUI = { zone: 'grid', act: '', list: 'latest', sel: {}, at: 0, timer: 0, searching: false, refreshing: false };1415 1516 // asEpisode marks an episode for the player (state.recording), which plays1617 // recordings too.@@ -66,6 +67,7 @@ function podZone(zone, act = '') {
6667 if (state.dock >= 0 || state.view !== 'podcasts') return;6768 if (zone === 'grid') return blurIn($('#podcasts'));6869 if (zone === 'menu') return void focusEl($(`#podcasts [data-list="${CSS.escape(podList()[0])}"]`));70+ if (zone === 'refresh') return void focusEl(f($('#podcasts'), 'refresh'));6971 const acts = $$('#podcasts .rd-actions .act.fx').filter(isShown);7072 if (!acts.length) return podZone('grid');7173 focusEl(acts.find((b) => b.dataset.act === podUI.act) || acts[0]);@@ -132,12 +134,55 @@ function renderPodcasts() {
132134 </article>`;133135 }).join('') || (cur[0] !== 'latest' && cur[0] !== 'continue' && (shows.find((x) => x.id === cur[0]) || {}).error134136 ? `<p class="r-empty">${esc(shows.find((x) => x.id === cur[0]).error)}</p>` : '');137+ renderPodRefresh(v);135138 podRefocus();136139 }137140 141+// renderPodRefresh shows Refresh, under the lists when there are shows,142+// and when every show was last listed.143+function renderPodRefresh(v) {144+ const listed = state.podcasts && Date.parse(state.podcasts.listed);145+ $('.p-refresh', v).hidden = !podShows().length;146+ f(v, 'refresh').classList.toggle('busy', podUI.refreshing);147+ f(v, 'refresh').textContent = podUI.refreshing ? 'Refreshing' : 'Refresh';148+ f(v, 'rnote').textContent = podUI.refreshing ? 'Looking for new episodes'149+ : listed ? `Every show checked ${ago(listed)}` : '';150+}151+152+// refreshPodcasts has the server list every show again now, and says what153+// came of it. The new episodes are looked up after (their descriptions,154+// their dates to the minute), so they're loaded again in a bit.155+async function refreshPodcasts() {156+ if (podUI.refreshing || !podShows().length) return;157+ const had = new Set(podShows().flatMap((s) => (s.episodes || []).map((ep) => ep.id)));158+ podUI.refreshing = true;159+ renderPodcasts();160+ let st = null;161+ try {162+ st = await api().RefreshPodcasts();163+ } catch (e) {164+ toast(String(e && e.message ? e.message : e), 6000);165+ }166+ podUI.refreshing = false;167+ if (st) {168+ state.podcasts = st;169+ podUI.at = Date.now();170+ const fresh = podShows().flatMap((s) => s.episodes || []).filter((ep) => !had.has(ep.id)).length;171+ const failed = podShows().filter((s) => s.error);172+ const why = failed.length ? `. ${failed.length === 1 ? failed[0].title : `${failed.length} shows`} couldn't be checked` : '';173+ toast(`${fresh ? `${fresh} new episode${fresh === 1 ? '' : 's'}` : 'No new episodes'}${why}`, failed.length ? 5000 : 2500);174+ if (fresh) {175+ podUI.sel = {}; // the newest, where they are176+ clearTimeout(podUI.timer);177+ podUI.timer = setTimeout(loadPodcasts, 20_000);178+ }179+ }180+ if (state.view === 'podcasts') renderPodcasts();181+}182+138183 function podHint(ep) {139184 if (onRemote()) return 'OK for the actions, Left for the shows';140- return hints([['Enter', 'Enter', 'for the actions'], ...(resumable(ep) ? [['', 'Shift Enter', 'plays from the start']] : []), ['w', 'W', ep.watched ? 'unwatched' : 'watched']]);185+ return hints([['Enter', 'Enter', 'for the actions'], ...(resumable(ep) ? [['', 'Shift Enter', 'plays from the start']] : []), ['w', 'W', ep.watched ? 'unwatched' : 'watched'], ['r', 'R', 'refreshes']]);141186 }142187 143188 function wirePodcasts() {@@ -148,6 +193,7 @@ function wirePodcasts() {
148193 podUI.list = b.dataset.list;149194 renderPodcasts();150195 });196+ f(v, 'refresh').addEventListener('click', refreshPodcasts);151197 v.addEventListener('click', (e) => {152198 const b = e.target.closest('[data-act]');153199 if (b) return podAct(b.dataset.act);@@ -200,12 +246,14 @@ async function podAct(act) {
200246 // library.201247 function podcastsKey(e) {202248 const lists = podLists();249+ if (e.key === 'r' || e.key === 'R') return void refreshPodcasts();203250 if (e.key === '[' || e.key === ']') {204251 const i = lists.findIndex(([k]) => k === podUI.list);205252 podUI.list = lists[(i + (e.key === ']' ? 1 : lists.length - 1)) % lists.length][0];206253 return renderPodcasts();207254 }208255 if (podUI.zone === 'menu') return podMenuKey(e, lists);256+ if (podUI.zone === 'refresh') return podRefreshKey(e);209257 if (podUI.zone === 'actions' && podActionsKey(e)) return undefined;210258 const eps = podList()[2];211259 const sel = podUI.sel;@@ -234,15 +282,18 @@ function podcastsKey(e) {
234282 return undefined;235283 }236284 237-// podMenuKey: Up and Down pick a list, Right or OK goes into it, Left (or238-// Up from the first) to the dock.285+// podMenuKey: Up and Down pick a list (Down from the last to Refresh),286+// Right or OK goes into it, Left (or Up from the first) to the dock.239287 function podMenuKey(e, lists) {240288 const i = lists.findIndex(([k]) => k === podUI.list);241289 switch (e.key) {242290 case 'ArrowUp': case 'ArrowDown': {243291 const j = i + (e.key === 'ArrowUp' ? -1 : 1);244292 if (j < 0) return openDock();245- if (j >= lists.length) break;293+ if (j >= lists.length) {294+ if (isShown(f($('#podcasts'), 'refresh'))) podZone('refresh');295+ break;296+ }246297 podUI.list = lists[j][0];247298 renderPodcasts();248299 podZone('menu');@@ -258,6 +309,21 @@ function podMenuKey(e, lists) {
258309 return undefined;259310 }260311 312+// podRefreshKey: OK refreshes, Up goes back to the lists, Right into the313+// list picked, Left to the dock.314+function podRefreshKey(e) {315+ switch (e.key) {316+ case 'Enter': f($('#podcasts'), 'refresh').click(); break;317+ case 'ArrowUp': podZone('menu'); break;318+ case 'ArrowRight': if (podList()[2].length) podZone('grid'); break;319+ case 'ArrowLeft': openDock(); break;320+ case 'ArrowDown': break;321+ default: return undefined;322+ }323+ e.preventDefault();324+ return undefined;325+}326+261327 // podActionsKey: Left and Right along the actions, OK presses, Down (or262328 // Back) to the grid, Up to the dock.263329 function podActionsKey(e) {@@ -359,27 +425,41 @@ async function searchShows(q) {
359425 openSettingsList('podfound');360426 }361427 428+// subscribers is a channel's following as YouTube says it: "3.39M429+// subscribers".430+const subscribers = (n) => `${new Intl.NumberFormat([], { notation: 'compact', maximumSignificantDigits: 3 }).format(n)} subscriber${n === 1 ? '' : 's'}`;431+362432 function podFoundRows(add) {363433 const found = state.podFound || [];364434 for (const c of found) {365- const have = podShows().some((s) => s.url === c.url);435+ // A channel found is its videos as a show.436+ const have = podShows().some((s) => s.url === c.url || s.url === `${c.url}/videos`);366437 add({367438 id: c.url, kind: have ? 'info' : 'action', verb: 'adds it', label: c.title,368- sub: [c.subscribers, c.description].filter(Boolean).join('. ').slice(0, 140),439+ sub: [c.subscribers ? subscribers(c.subscribers) : '', c.description].filter(Boolean).join('. ').slice(0, 140),369440 value: have ? 'Added' : 'Add', run: () => addShow(c),370441 });371442 }372443 if (!found.length) add({ id: 'none', kind: 'info', label: 'Nothing found', sub: 'Try other words, or paste the show\'s address' });373444 }374445 446+// addShow adds a show found, one at a time: it's listed first, a few447+// seconds.448+let podAdding = '';375449 async function addShow(c) {376- toast(`Adding ${c.title}...`, 30000);450+ if (podAdding) return toast(`Still adding ${podAdding}...`, 2000);451+ podAdding = c.title;452+ toast(`Adding ${c.title}...`, 120_000);453+ let show;377454 try {378- await api().AddPodcast(c.url, 0);455+ show = await api().AddPodcast(c.url, 0);379456 } catch (e) {380457 return toast(String(e && e.message ? e.message : e), 6000);458+ } finally {459+ podAdding = '';381460 }382- toast(`Added ${c.title}`, 2000);461+ const n = (show.episodes || []).length;462+ toast(`Added ${show.title || c.title}, ${n} episode${n === 1 ? '' : 's'}`, 2500);383463 closeSettingsList();384464 await loadPodcasts();385465 return undefined;@@ -598,6 +598,12 @@ body:not(.mode-guide) .g-cell.rec-on::after { animation: none; } /* hidden, so s
598598 /* Recordings' library, for episodes: the one picked above a grid. */599599 .p-main { display: grid; grid-template-rows: auto minmax(0, 1fr); gap: 24px; min-height: 0; }600600 .p-menu { overflow: auto; min-height: 0; scrollbar-width: none; }601+/* Refresh, under the lists: every show listed again now. */602+.p-refresh { display: grid; gap: 8px; padding: 0 14px; }603+.p-refresh[hidden] { display: none; }604+.p-refresh .btn { justify-self: stretch; text-align: center; }605+.p-refresh .btn.busy:not(:focus) { color: var(--amber); box-shadow: inset 0 0 0 1px var(--amber-2); }606+.p-rnote { font-size: 12px; line-height: 1.35; color: var(--faint); }601607 .p-name { white-space: normal; display: -webkit-box; -webkit-line-clamp: 2; -webkit-box-orient: vertical; min-height: 2.6em; line-height: 1.3; }602608 .r-badge.p-len { color: var(--paper); }603609 @@ -285,6 +285,7 @@
285285 286286 // The Podcasts app.287287 Podcasts: () => call('GET', '/api/podcasts'),288+ RefreshPodcasts: () => call('POST', '/api/podcasts/refresh'),288289 SearchPodcasts: (q) => call('GET', `/api/podcasts/search?q=${enc(q)}`),289290 AddPodcast: (url, minMinutes) => call('POST', '/api/podcasts/shows', { url, minMinutes: minMinutes || 0 }),290291 RemovePodcast: (id) => call('DELETE', `/api/podcasts/shows/${enc(id)}`).then(() => undefined),@@ -43,6 +43,8 @@ export function Podcasts():Promise<podcast.State>;
4343 4444 export function Record(arg1:dvr.Request):Promise<dvr.Rule>;4545 46+export function RefreshPodcasts():Promise<podcast.State>;47+4648 export function RemovePodcast(arg1:string):Promise<void>;4749 4850 export function SaveProgress(arg1:string,arg2:number,arg3:number):Promise<void>;@@ -70,6 +70,10 @@ export function Record(arg1) {
7070 return window['go']['main']['App']['Record'](arg1);7171 }7272 73+export function RefreshPodcasts() {74+ return window['go']['main']['App']['RefreshPodcasts']();75+}76+7377 export function RemovePodcast(arg1) {7478 return window['go']['main']['App']['RemovePodcast'](arg1);7579 }@@ -905,6 +905,7 @@ export namespace podcast {
905905 export class State {906906 shows: Show[];907907 updating: boolean;908+ listed: time.Time;908909 909910 static createFrom(source: any = {}) {910911 return new State(source);@@ -914,6 +915,7 @@ export namespace podcast {
914915 if ('string' === typeof source) source = JSON.parse(source);915916 this.shows = this.convertValues(source["shows"], Show);916917 this.updating = source["updating"];918+ this.listed = this.convertValues(source["listed"], time.Time);917919 }918920 919921 convertValues(a: any, classs: any, asMap: boolean = false): any {@@ -61,6 +61,7 @@ type Backend interface {
6161 // to add, and an episode played from a point, with how far it was6262 // watched.6363 Podcasts(ctx context.Context) (*podcast.State, error)64+ RefreshPodcasts(ctx context.Context) (*podcast.State, error)6465 SearchPodcasts(ctx context.Context, query string) ([]podcast.Found, error)6566 AddPodcast(ctx context.Context, url string, minMinutes float64) (*podcast.Show, error)6667 RemovePodcast(ctx context.Context, id string) error@@ -290,6 +291,9 @@ func (s *Server) Handler() http.Handler {
290291 mux.HandleFunc("GET /api/podcasts", func(w http.ResponseWriter, r *http.Request) {291292 reply(w)(b.Podcasts(r.Context()))292293 })294+ mux.HandleFunc("POST /api/podcasts/refresh", func(w http.ResponseWriter, r *http.Request) {295+ reply(w)(b.RefreshPodcasts(r.Context()))296+ })293297 mux.HandleFunc("GET /api/podcasts/search", func(w http.ResponseWriter, r *http.Request) {294298 reply(w)(b.SearchPodcasts(r.Context(), r.URL.Query().Get("q")))295299 })@@ -116,6 +116,12 @@ func (f *fake) Podcasts(context.Context) (*podcast.State, error) {
116116 return &podcast.State{Shows: []podcast.Show{{ID: "s1", Title: "Show", Episodes: []podcast.Episode{{ID: "e1", ShowID: "s1", Duration: 3600}}}}}, nil117117 }118118 119+func (f *fake) RefreshPodcasts(ctx context.Context) (*podcast.State, error) {120+ st, _ := f.Podcasts(ctx)121+ st.Shows[0].Episodes = append([]podcast.Episode{{ID: "e2", ShowID: "s1", Duration: 1800}}, st.Shows[0].Episodes...)122+ return st, nil123+}124+119125 func (f *fake) SearchPodcasts(_ context.Context, q string) ([]podcast.Found, error) {120126 return []podcast.Found{{ID: "UCa", Title: q}}, nil121127 }@@ -155,6 +161,9 @@ func TestPodcasts(t *testing.T) {
155161 if st, err := c.Podcasts(ctx); err != nil || len(st.Shows) != 1 || st.Shows[0].Episodes[0].Duration != 3600 {156162 t.Errorf("podcasts: %+v, %v", st, err)157163 }164+ if st, err := c.RefreshPodcasts(ctx); err != nil || len(st.Shows[0].Episodes) != 2 || st.Shows[0].Episodes[0].ID != "e2" {165+ t.Errorf("refresh: %+v, %v", st, err)166+ }158167 if found, err := c.SearchPodcasts(ctx, "sad boyz"); err != nil || len(found) != 1 || found[0].Title != "sad boyz" {159168 t.Errorf("search: %+v, %v", found, err)160169 }@@ -228,6 +228,12 @@ func (c *Client) Podcasts(ctx context.Context) (*podcast.State, error) {
228228 return &s, c.call(ctx, http.MethodGet, "/api/podcasts", nil, &s)229229 }230230 231+// RefreshPodcasts implements Backend.232+func (c *Client) RefreshPodcasts(ctx context.Context) (*podcast.State, error) {233+ var s podcast.State234+ return &s, c.call(ctx, http.MethodPost, "/api/podcasts/refresh", nil, &s)235+}236+231237 // SearchPodcasts implements Backend.232238 func (c *Client) SearchPodcasts(ctx context.Context, query string) ([]podcast.Found, error) {233239 var out []podcast.Found@@ -101,11 +101,13 @@ type episode struct {
101101 Watched bool `json:"watched,omitempty"`102102 }103103 104-// State is the library for the app: its shows, as they were added, and105-// whether a show is being listed.104+// State is the library for the app: its shows, as they were added,105+// whether a show is being listed, and since when every show has been (the106+// longest ago of their listings; zero without shows).106107 type State struct {107- Shows []Show `json:"shows"`108- Updating bool `json:"updating"`108+ Shows []Show `json:"shows"`109+ Updating bool `json:"updating"`110+ Listed time.Time `json:"listed,omitzero"`109111 }110112 111113 // Show is a show for the app.@@ -191,8 +193,11 @@ func (l *Library) State() State {
191193 defer l.mu.Unlock()192194 l.load()193195 out := State{Shows: make([]Show, 0, len(l.st.Shows)), Updating: l.updating}194- for _, s := range l.st.Shows {196+ for i, s := range l.st.Shows {195197 out.Shows = append(out.Shows, s.view())198+ if i == 0 || s.Listed.Before(out.Listed) {199+ out.Listed = s.Listed200+ }196201 }197202 return out198203 }@@ -232,9 +237,9 @@ func (l *Library) Search(ctx context.Context, query string) ([]Found, error) {
232237 }233238 234239 // Add adds the show at raw (a channel, one of its tabs, or a playlist; see235-// source), listing it first, with minMinutes the shortest video that's an236-// episode (DefaultMinMinutes when 0). A show already there is returned237-// as it is.240+// source), listing it first, at once (see list), with minMinutes the241+// shortest video that's an episode (DefaultMinMinutes when 0). A show242+// already there is returned as it is.238243 func (l *Library) Add(ctx context.Context, raw string, minMinutes float64) (Show, error) {239244 src, err := source(raw)240245 if err != nil {@@ -29,8 +29,9 @@ func TestMain(m *testing.M) {
2929 3030 // fakeYtDlp answers yt-dlp's command line, noting each run's arguments in3131 // the file AIRWAVES_FAKE_YTDLP names. Show A (@showa) has three episodes,32-// a clip and a premiere to come in its videos, and a past and a current33-// stream in its streams; Show B is a playlist with a private video. Any32+// a clip and a premiere to come in its videos (and a fourth episode, new,33+// with AIRWAVES_FAKE_NEW set), and a past and a current stream in its34+// streams; Show B is a playlist with a private video. Any3435 // video's details give it an hour, but "feedclip" lasts 200 seconds; with3536 // a format asked for, an H.264 picture and AAC sound six hours long.3637 func fakeYtDlp(args []string) int {@@ -57,6 +58,10 @@ func fakeYtDlp(args []string) int {
5758 {"id": "up1", "title": "Premiere", "live_status": "is_upcoming"},5859 {"id": "ep1", "title": "Episode 1", "duration": 4000, "timestamp": now - 3*day},5960 }}61+ if os.Getenv("AIRWAVES_FAKE_NEW") != "" {62+ out.(map[string]any)["entries"] = append([]map[string]any{{"id": "ep4", "title": "Episode 4", "duration": 3600, "timestamp": now}},63+ out.(map[string]any)["entries"].([]map[string]any)...)64+ }6065 case u == "https://www.youtube.com/@showa/streams":6166 out = map[string]any{"channel": "Show A", "channel_id": "UCa", "entries": []map[string]any{6267 {"id": "now1", "title": "Live now", "live_status": "is_live"},@@ -252,6 +257,49 @@ func TestUpdate(t *testing.T) {
252257 }253258 }254259 260+// Refresh lists every show again at once, a new upload joining, and261+// neither it nor Add waits behind yt-dlp's other listing jobs, which can262+// take minutes.263+func TestRefresh(t *testing.T) {264+ l, _ := library(t)265+ if _, err := l.Add(context.Background(), "@showa", 0); err != nil {266+ t.Fatal(err)267+ }268+ if _, err := l.Add(context.Background(), "https://www.youtube.com/playlist?list=PLtest00000001", 0); err != nil {269+ t.Fatal(err)270+ }271+ before := l.State()272+ if before.Listed.IsZero() {273+ t.Fatal("never listed")274+ }275+ t.Setenv("AIRWAVES_FAKE_NEW", "1")276+ l.YtDlp.Jobs.Lock() // a channel's whole catalog, being listed277+ defer l.YtDlp.Jobs.Unlock()278+ done := make(chan State)279+ go func() {280+ st := l.Refresh(context.Background())281+ if _, err := l.Add(context.Background(), "https://www.youtube.com/playlist?list=PLtest00000002", 0); err != nil {282+ t.Error(err)283+ }284+ done <- st285+ }()286+ var st State287+ select {288+ case st = <-done:289+ case <-time.After(30 * time.Second):290+ t.Fatal("waited behind the listing job")291+ }292+ if got := ids(st.Shows[0].Episodes); !slices.Equal(got, []string{"ep4", "ep3", "ep2", "ep1"}) {293+ t.Errorf("episodes: %v", got)294+ }295+ if !st.Listed.After(before.Listed) || st.Updating {296+ t.Errorf("listed %v, was %v; updating %v", st.Listed, before.Listed, st.Updating)297+ }298+ if n := len(l.State().Shows); n != 3 {299+ t.Errorf("%d shows", n)300+ }301+}302+255303 func TestProgressPersists(t *testing.T) {256304 l, _ := library(t)257305 if _, err := l.Add(context.Background(), "@showa", 0); err != nil {@@ -13,6 +13,8 @@ import (
1313 "strings"1414 "time"1515 16+ "golang.org/x/sync/errgroup"17+1618 "airwaves/internal/web"1719 "airwaves/internal/youtube"1820 )@@ -92,11 +94,14 @@ func (l *Library) update(ctx context.Context) {
9294 }9395 l.updating = len(lists) > 09496 l.mu.Unlock()97+ tool := l.tool()9598 for _, s := range lists {9699 if ctx.Err() != nil {97100 return98101 }102+ tool.Jobs.Lock() // one listing job at a time across yt-dlp's users99103 l.relist(ctx, s)104+ tool.Jobs.Unlock()100105 }101106 l.mu.Lock()102107 l.updating = false@@ -119,16 +124,16 @@ func (l *Library) listDue(s *show, now time.Time) bool {
119124 return now.Sub(s.Listed) >= listEvery120125 }121126 122-// list lists a show's newest videos, one listing job at a time across123-// yt-dlp's users.127+// list lists a show's newest videos, a few seconds' work. It doesn't take128+// its turn with yt-dlp's other listing jobs, which can be a channel's whole129+// catalog and take minutes: the background updates take turns (see130+// update), while someone adding a show or refreshing doesn't wait behind131+// them.124132 func (l *Library) list(ctx context.Context, s src) (youtube.Listing, error) {125133 ctx, cancel := context.WithTimeout(ctx, 2*time.Minute)126134 defer cancel()127- tool := l.tool()128135 args := append(slices.Clone(youtube.ListingArgs), "-I", "1:"+strconv.Itoa(listLimit), s.url)129- tool.Jobs.Lock()130- raw, err := tool.Run(ctx, args...)131- tool.Jobs.Unlock()136+ raw, err := l.tool().Run(ctx, args...)132137 if err != nil {133138 return youtube.Listing{}, err134139 }@@ -138,9 +143,39 @@ func (l *Library) list(ctx context.Context, s src) (youtube.Listing, error) {
138143 return youtube.ParseListing(raw)139144 }140145 146+// refreshAtOnce is how many shows Refresh lists at the same time.147+const refreshAtOnce = 4148+149+// Refresh lists every show again now, rather than at its turn: for someone150+// at the screen, so a few at once, and not behind yt-dlp's other jobs (see151+// list). The new episodes are looked up after, in the background. It152+// returns the library as refreshed.153+func (l *Library) Refresh(ctx context.Context) State {154+ l.mu.Lock()155+ l.load()156+ shows := slices.Clone(l.st.Shows)157+ l.mu.Unlock()158+ var g errgroup.Group159+ g.SetLimit(refreshAtOnce)160+ for _, s := range shows {161+ g.Go(func() error {162+ l.relist(ctx, s)163+ return nil164+ })165+ }166+ g.Wait()167+ l.mu.Lock()168+ l.poke()169+ l.mu.Unlock()170+ return l.State()171+}172+141173 // relist lists a show again and takes in what's new.142174 func (l *Library) relist(ctx context.Context, s *show) {143175 lst, err := l.list(ctx, s.source())176+ if err != nil && ctx.Err() != nil {177+ return // stopped, not failed: it's listed at its turn178+ }144179 now := l.clock()145180 l.mu.Lock()146181 defer l.mu.Unlock()@@ -149,9 +184,6 @@ func (l *Library) relist(ctx context.Context, s *show) {
149184 }150185 s.Listed = now151186 if err != nil {152- if ctx.Err() != nil {153- return154- }155187 s.Error = err.Error()156188 log.Printf("podcasts: %s: %v", s.Title, err)157189 l.save()@@ -383,10 +415,10 @@ func (l *Library) lookups(ctx context.Context) {
383415 // lookup looks up a video's details, and drops it from its show when it384416 // isn't an episode after all.385417 func (l *Library) lookup(ctx context.Context, s *show, id string) {418+ tool := l.tool()419+ tool.Jobs.Lock() // its turn first: its time starts when it runs386420 ctx, cancel := context.WithTimeout(ctx, resolveTime)387421 defer cancel()388- tool := l.tool()389- tool.Jobs.Lock()390422 raw, err := tool.Run(ctx, "-J", "--skip-download", "--no-playlist", "--ignore-no-formats-error", youtube.WatchURL(id))391423 tool.Jobs.Unlock()392424 var info youtube.Info@@ -21,6 +21,16 @@ func (s *Service) Podcasts(context.Context) (*podcast.State, error) {
2121 return &st, nil2222 }2323 24+// RefreshPodcasts lists every show again now, and returns the library25+// as refreshed.26+func (s *Service) RefreshPodcasts(ctx context.Context) (*podcast.State, error) {27+ if s.podcasts == nil {28+ return nil, ErrNoPodcasts29+ }30+ st := s.podcasts.Refresh(ctx)31+ return &st, nil32+}33+2434 // SearchPodcasts finds YouTube channels to add as shows.2535 func (s *Service) SearchPodcasts(ctx context.Context, query string) ([]podcast.Found, error) {2636 if s.podcasts == nil {@@ -99,7 +99,7 @@ func ParseListing(raw []byte) (Listing, error) {
9999 if p.ChannelID == "" {100100 return Listing{}, errors.New("not a YouTube channel")101101 }102- l := Listing{ID: p.ChannelID, Name: p.Channel, Image: p.channel().Image}102+ l := Listing{ID: p.ChannelID, Name: strings.TrimSpace(p.Channel), Image: p.channel().Image}103103 for _, e := range p.Entries {104104 if e.ID == "" {105105 continue@@ -155,7 +155,7 @@ func ParsePlaylist(raw []byte) (Listing, error) {
155155 if p.Type != "playlist" || p.ID == "" || p.ID == p.ChannelID { // not a channel's tab156156 return Listing{}, errors.New("not a YouTube playlist")157157 }158- l := Listing{ID: p.ID, Name: p.Title, Owner: cmp.Or(p.Channel, p.Uploader), OwnerID: p.ChannelID}158+ l := Listing{ID: p.ID, Name: strings.TrimSpace(p.Title), Owner: strings.TrimSpace(cmp.Or(p.Channel, p.Uploader)), OwnerID: p.ChannelID}159159 seen := map[string]bool{}160160 for _, e := range p.Entries {161161 if e.ID == "" || seen[e.ID] || hidden[e.Title] || locked[e.Availability] {@@ -381,7 +381,7 @@ type channelInfo struct {
381381 }382382 383383 func (e channelInfo) channel() Channel {384- c := Channel{ID: e.ChannelID, Name: cmp.Or(e.Channel, e.Title), Subscribers: int64(e.Followers),384+ c := Channel{ID: e.ChannelID, Name: strings.TrimSpace(cmp.Or(e.Channel, e.Title)), Subscribers: int64(e.Followers),385385 Description: Blurb(e.Description)}386386 switch {387387 case strings.HasPrefix(e.UploaderID, "@"):