youtube: read through InnerTube with a pasted cookie
Advanced Protection accounts cannot grant OAuth to an unverified client, and the Data API never returns Watch Later. The cookie is pasted for each fetch so rank-king never holds a full-session credential. Assisted-by: Claude Opus 5.5 via Claude Code
diff --git a/AGENTS.md b/AGENTS.md
index 1358e0a..d631527 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -24,8 +24,8 @@ db/migrate/ — Sequel migrations
lib/rank_king/ — app code
config.rb — Config data class, reads from ENV
db.rb — Sequel connection + migration runner
- models.rb — Sequel models (User, GoogleAccount, List, Item, Axis, Result)
- google.rb — Google client: OAuth and the YouTube Data API
+ models.rb — Sequel models (User, List, Item, Axis, Result)
+ innertube.rb — YouTube client: InnerTube, signed with a pasted browser cookie
sources.rb — Sources.fetch: a list's source_kind names a class in Sources
sources/ — one class per source kind
sync.rb — reconciles a list's items with a fresh snapshot from a fetching source
@@ -41,7 +41,7 @@ Steepfile — type checking config
public/ — static assets
vendor/gloss/ — vendored Gloss CSS and JS (refresh with rake gloss:vendor)
css/app.css — Rank King's layout over Gloss
- js/app.js — axis tags, confirmations, busy buttons, new-list source fields
+ js/app.js — axis tags, confirmations, busy buttons, new-list source fields, YouTube playlist picker
.github/workflows/ — CI, PR previews, preview reaper (GitHub Actions)
Dockerfile — production image
fly.toml — Fly config for the main demo and PR previews
@@ -50,9 +50,9 @@ fly.toml — Fly config for the main demo and PR previews
## Domain model
-- **User** has many Lists; identified by the login on the `Remote-User` header. Their settings page (`/settings`) connects a Google account (**GoogleAccount**, which keeps only the refresh token)
+- **User** has many Lists; identified by the login on the `Remote-User` header
- **List** belongs to a User and has many Items and Axes, and one source (`source_kind` names a class in `Sources`). Each source renders its own sync dialog on the list page and applies what it posts: `Text`'s edits items by id, and a kind that fetches calls `Sync.call`. A fetching kind keeps its config in its own table keyed by `list_id` and its per-item details in a table keyed by `item_id`
-- Sources: `Text` (typed in, edited by id) and `YouTube` (follows one of the owner's playlists through their Google account; `youtube_sources` by `list_id`, `youtube_items` by `item_id`)
+- Sources: `Text` (typed in, edited by id) and `YouTube` (follows one of the owner's playlists, Watch Later included, with a youtube.com cookie pasted for each fetch and never kept; `youtube_sources` by `list_id`, `youtube_items` by `item_id`)
- **Item** belongs to a List; removed items, and items missing from the latest sync, are archived (`archived_at`) and hidden everywhere
- **Axis** belongs to a List, has many Results
- **Result** belongs to an Axis, a winner (Item), and a loser (Item)
@@ -85,18 +85,30 @@ fly.toml — Fly config for the main demo and PR previews
- `DATABASE_URL` — SQLite path (default: `:memory:`)
- `SESSION_SECRET` — session signing key (auto-generated if unset)
-- `GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET_COMMAND` (required) — a
- Google OAuth web client with the YouTube Data API enabled, for
- connecting Google. The secret is whatever the command prints, run once
- at boot (e.g. `op read op://…` or `cat ~/.config/rank-king/secret`).
- Its redirect URI is `<app URL>/settings/google/callback`. A consent
- screen left in Testing mode expires refresh tokens after 7 days. The
- Fly demos set placeholders in `fly.toml`, since anyone can open them.
-
-Tests set `DATABASE_URL=:memory:` and a dummy Google client in
-`test/test_helper.rb`. The app hands its `google` client to every
-source; the YouTube tests subclass the app to return a `Google`
-subclass whose `request` answers without the network.
+
+Tests set `DATABASE_URL=:memory:` in `test/test_helper.rb`. The app
+hands its `innertube` client to every source; the YouTube tests subclass
+the app to return an `InnerTube` subclass whose `request` answers
+without the network.
+
+## YouTube
+
+YouTube is read through InnerTube (`youtubei/v1/browse`), the private
+API youtube.com's own pages call, not OAuth and the Data API: an
+Advanced Protection account cannot grant OAuth to an unverified client,
+and the Data API never returns Watch Later. Requests carry the user's
+pasted Cookie header and a `SAPISIDHASH` Authorization signed from it.
+The cookie is pasted on the new-list form, where app.js posts it to
+`/youtube/playlists` for the picker, and again in each sync dialog;
+rank-king never stores it.
+
+- `FEplaylist_aggregation` lists playlists (Watch Later and Liked
+ videos included); `VL<playlist id>` browses one.
+- A signed-out cookie still gets `200`, with `logged_in` `0` and an
+ error alert, so `InnerTube#check` raises rather than let a sync
+ archive the whole list.
+- `InnerTube::CLIENT`'s `clientVersion` may need bumping if YouTube
+ starts refusing it.
## Conventions
diff --git a/Steepfile b/Steepfile
index c928741..ef1fdbf 100644
--- a/Steepfile
+++ b/Steepfile
@@ -10,5 +10,5 @@ target :lib do
ignore "lib/rank_king/views", inline: true
signature "sig"
- library "securerandom", "open3", "net-http", "json", "uri"
+ library "securerandom", "digest", "net-http", "json", "uri"
end
diff --git a/db/migrate/005_add_youtube.rb b/db/migrate/005_add_youtube.rb
index b6bf69f..f102690 100644
--- a/db/migrate/005_add_youtube.rb
+++ b/db/migrate/005_add_youtube.rb
@@ -2,17 +2,6 @@
Sequel.migration do
change do
- # A user's connection to Google, for reading their YouTube playlists.
- # Only the refresh token is kept: each sync trades it for an access
- # token.
- create_table(:google_accounts) do
- primary_key :id
- foreign_key :user_id, :users, null: false, unique: true, on_delete: :cascade
- String :refresh_token, null: false
- DateTime :created_at, null: false
- DateTime :updated_at, null: false
- end
-
# The YouTube source's config: which playlist a list follows.
create_table(:youtube_sources) do
foreign_key :list_id, :lists, primary_key: true, on_delete: :cascade
diff --git a/fly.toml b/fly.toml
index 4cb066b..f8e8383 100644
--- a/fly.toml
+++ b/fly.toml
@@ -16,14 +16,6 @@ primary_region = 'sjc'
[build]
-# The app requires a Google client. The demos get a placeholder rather
-# than the real one: anyone who opens them is the one demo user, so a
-# Google account connected there would be everyone's. Connect Google on
-# a demo fails at Google's consent screen.
-[env]
- GOOGLE_CLIENT_ID = 'demo'
- GOOGLE_CLIENT_SECRET_COMMAND = 'echo demo'
-
[processes]
app = 'rackup -o 0.0.0.0 demo.ru'
diff --git a/lib/rank_king/config.rb b/lib/rank_king/config.rb
index 3a4204f..efbcda8 100644
--- a/lib/rank_king/config.rb
+++ b/lib/rank_king/config.rb
@@ -1,6 +1,5 @@
# frozen_string_literal: true
-require "open3"
require "securerandom"
module RankKing
@@ -8,8 +7,6 @@ module RankKing
Config = Data.define(
:database_url,
:session_secret,
- :google_client_id,
- :google_client_secret,
)
class Config
@@ -23,23 +20,8 @@ module RankKing
new(
database_url: env.fetch("DATABASE_URL", ":memory:"),
session_secret: env.fetch("SESSION_SECRET") { SecureRandom.hex(64) },
- # A Google OAuth client, for connecting YouTube. The secret comes
- # from a command, so it can live in a file or a password manager
- # rather than the environment.
- google_client_id: env.fetch("GOOGLE_CLIENT_ID"),
- google_client_secret: secret_from(env.fetch("GOOGLE_CLIENT_SECRET_COMMAND")),
)
end
-
- # Runs a shell command, such as `op read op://…` or `cat path`, and
- # returns what it prints.
- #: (String command) -> String
- def self.secret_from(command)
- output, status = Open3.capture2(command)
- raise "#{command} exited with #{status.exitstatus}" unless status.success?
-
- output.strip
- end
end
CONFIG = Config.from_env
diff --git a/lib/rank_king/google.rb b/lib/rank_king/google.rb
deleted file mode 100644
index 2df93f8..0000000
--- a/lib/rank_king/google.rb
+++ /dev/null
@@ -1,141 +0,0 @@
-# frozen_string_literal: true
-
-require "json"
-require "net/http"
-require "uri"
-
-module RankKing
- # The Google APIs rank-king calls: OAuth, to connect a user's account,
- # and the YouTube Data API, to read their playlists. Every call goes
- # through #request, which a subclass overrides to answer without the
- # network, as the tests' does.
- class Google
- # Google refused a request, or answered with something unexpected.
- class Error < StandardError; end
-
- # A video in a playlist, as YouTube describes it.
- # Signed in sig/rank_king/google.rbs: a Data class.
- # @rbs skip
- Video = Data.define(:id, :title, :channel_title, :thumbnail_url)
-
- AUTH_URL = "https://accounts.google.com/o/oauth2/v2/auth"
- TOKEN_URL = "https://oauth2.googleapis.com/token"
- PLAYLISTS_URL = "https://www.googleapis.com/youtube/v3/playlists"
- PLAYLIST_ITEMS_URL = "https://www.googleapis.com/youtube/v3/playlistItems"
- SCOPE = "https://www.googleapis.com/auth/youtube.readonly"
-
- #: (client_id: String, client_secret: String) -> void
- def initialize(client_id:, client_secret:)
- @client_id = client_id
- @client_secret = client_secret
- end
-
- # Where to send the user to grant read access to their YouTube
- # account. prompt=consent makes Google send a refresh token every time.
- #: (redirect_uri: String, state: String) -> String
- def authorize_url(redirect_uri:, state:)
- query = URI.encode_www_form(
- client_id: @client_id, redirect_uri:, state:, scope: SCOPE,
- response_type: "code", access_type: "offline", prompt: "consent",
- )
- "#{AUTH_URL}?#{query}"
- end
-
- # Trades the code Google sent back for a refresh token.
- #: (String code, redirect_uri: String) -> String
- def refresh_token(code, redirect_uri:)
- body = request(:post, TOKEN_URL, { **client, code:, redirect_uri:, grant_type: "authorization_code" }, nil)
- body.fetch("refresh_token") { raise Error, "Google sent no refresh token" }
- end
-
- #: (String refresh_token) -> String
- def access_token(refresh_token)
- body = request(:post, TOKEN_URL, { **client, refresh_token:, grant_type: "refresh_token" }, nil)
- body.fetch("access_token") { raise Error, "Google sent no access token" }
- end
-
- # The user's own playlists, as [playlist id, title]. YouTube leaves
- # out Watch Later and Liked videos.
- #: (String access_token) -> Array[[String, String]]
- def playlists(access_token)
- pages(access_token, PLAYLISTS_URL, { part: "snippet", mine: "true" }).map { |item|
- [item.fetch("id"), item.dig("snippet", "title").to_s] #: [String, String]
- }
- end
-
- # The videos in a playlist, in playlist order. Deleted videos, and
- # private ones the user cannot see, carry no owner and are left out.
- #: (String access_token, String playlist_id) -> Array[Video]
- def playlist_videos(access_token, playlist_id)
- items = pages(access_token, PLAYLIST_ITEMS_URL, { part: "snippet", playlistId: playlist_id })
- items.filter_map { |item|
- snippet = item.fetch("snippet")
- channel_title = snippet["videoOwnerChannelTitle"]
- next unless channel_title
-
- Video.new(
- id: snippet.dig("resourceId", "videoId"),
- title: snippet.fetch("title"),
- channel_title:,
- thumbnail_url: thumbnail_url(snippet["thumbnails"] || {}),
- )
- }
- end
-
- private
-
- # The medium thumbnail (320×180), or the nearest size there is.
- #: (Hash[String, untyped] thumbnails) -> String?
- def thumbnail_url(thumbnails)
- size = %w[medium high default standard maxres].find { |name| thumbnails.key?(name) }
- size && thumbnails.dig(size, "url")
- end
-
- # Every item of a YouTube list endpoint, following its pages.
- #: (String access_token, String url, Hash[Symbol, String] query) -> Array[Hash[String, untyped]]
- def pages(access_token, url, query)
- items = [] #: Array[Hash[String, untyped]]
- page = nil #: String?
- loop do
- params = { **query, maxResults: "50" } #: Hash[Symbol, String]
- params[:pageToken] = page if page
- body = request(:get, url, params, access_token)
-
- items.concat(body["items"] || [])
- page = body["nextPageToken"]
- break unless page
- end
- items
- end
-
- #: () -> Hash[Symbol, String]
- def client = { client_id: @client_id, client_secret: @client_secret }
-
- # Makes one request and returns its JSON body: a POST of a form, or
- # a GET with a query, authorized by the access token when there is
- # one.
- #: (Symbol method, String url, Hash[Symbol, String] params, String? access_token) -> Hash[String, untyped]
- def request(method, url, params, access_token)
- uri = URI(url)
- request = if method == :post
- Net::HTTP::Post.new(uri).tap do |post|
- post.set_form_data(params)
- end
- else
- uri.query = URI.encode_www_form(params)
- Net::HTTP::Get.new(uri)
- end
- request["Authorization"] = "Bearer #{access_token}" if access_token
-
- response = Net::HTTP.start(uri.hostname.to_s, uri.port, use_ssl: true) { |http| http.request(request) }
- body = JSON.parse(response.body.to_s)
- return body if response.is_a?(Net::HTTPSuccess)
-
- error = body["error"]
- message = error.is_a?(Hash) ? error["message"] : body["error_description"] || error
- raise Error, message || "HTTP #{response.code}"
- rescue JSON::ParserError
- raise Error, "Google answered HTTP #{response&.code} with no JSON"
- end
- end
-end
diff --git a/lib/rank_king/innertube.rb b/lib/rank_king/innertube.rb
new file mode 100644
index 0000000..8c0d712
--- /dev/null
+++ b/lib/rank_king/innertube.rb
@@ -0,0 +1,175 @@
+# frozen_string_literal: true
+
+require "digest"
+require "json"
+require "net/http"
+require "uri"
+
+module RankKing
+ # InnerTube, the API youtube.com's own pages call, signed in with the
+ # user's browser cookie. It stands in for OAuth and the YouTube Data
+ # API, which an Advanced Protection account cannot grant to an
+ # unverified client and which never shares Watch Later. Every call goes
+ # through #request, which a subclass overrides to answer without the
+ # network, as the tests' does.
+ class InnerTube
+ # YouTube refused a request, or answered with something unexpected.
+ class Error < StandardError; end
+
+ # A video in a playlist, as YouTube describes it.
+ # Signed in sig/rank_king/innertube.rbs: a Data class.
+ # @rbs skip
+ Video = Data.define(:id, :title, :channel_title, :thumbnail_url)
+
+ ORIGIN = "https://www.youtube.com"
+ BROWSE_URL = "#{ORIGIN}/youtubei/v1/browse?prettyPrint=false"
+ # The web client InnerTube is told it is serving. YouTube may refuse
+ # a version once it is old enough.
+ CLIENT = { clientName: "WEB", clientVersion: "2.20261020.00.00", hl: "en", gl: "US" }.freeze
+ # Each cookie a request is signed with, and the scheme its signature
+ # goes under in the Authorization header.
+ API_COOKIES = {
+ "SAPISID" => "SAPISIDHASH",
+ "__Secure-1PAPISID" => "SAPISID1PHASH",
+ "__Secure-3PAPISID" => "SAPISID3PHASH",
+ }.freeze #: Hash[String, String]
+
+ # A Cookie header as pasted, without the "Cookie:" it may have been
+ # copied with, or nil when it has no cookie to sign requests with.
+ #: (String text) -> String?
+ def self.cookie(text)
+ cookie = text.strip.delete_prefix("Cookie:").strip.gsub(/\s*\n\s*/, " ")
+ cookie if API_COOKIES.keys.any? { |name| jar(cookie)[name] }
+ end
+
+ #: (String cookie) -> Hash[String, String]
+ def self.jar(cookie)
+ jar = {} #: Hash[String, String]
+ cookie.split(/;\s*/).each do |pair|
+ name, value = pair.split("=", 2)
+ jar[name.to_s] = value if value
+ end
+ jar
+ end
+
+ # Google's signature for a request made with a cookie: for each API
+ # cookie, a SHA-1 of the time, its value, and the origin.
+ #: (String cookie, ?at: Integer) -> String
+ def self.authorization(cookie, at: Time.now.to_i)
+ jar = jar(cookie)
+ API_COOKIES.filter_map { |name, scheme|
+ value = jar[name]
+ "#{scheme} #{at}_#{Digest::SHA1.hexdigest("#{at} #{value} #{ORIGIN}")}" if value
+ }.join(" ")
+ end
+
+ # The user's playlists, as [playlist id, title], in YouTube's order:
+ # Watch Later and Liked videos among their own and the ones they saved.
+ #: (String cookie) -> Array[[String, String]]
+ def playlists(cookie)
+ lockups = pages(cookie, { browseId: "FEplaylist_aggregation" }).flat_map { |page| find_all(page, "lockupViewModel") }
+ lockups.filter_map { |lockup|
+ next unless lockup["contentType"] == "LOCKUP_CONTENT_TYPE_PLAYLIST"
+
+ title = lockup.dig("metadata", "lockupMetadataViewModel", "title", "content").to_s
+ [lockup.fetch("contentId"), title] #: [String, String]
+ }
+ end
+
+ # The videos in a playlist, in playlist order. Deleted videos, and
+ # private ones the user cannot see, are not playable and are left out.
+ # Some videos come with no channel.
+ #: (String cookie, String playlist_id) -> Array[Video]
+ def playlist_videos(cookie, playlist_id)
+ renderers = pages(cookie, { browseId: "VL#{playlist_id}" }).flat_map { |page| find_all(page, "playlistVideoRenderer") }
+ renderers.filter_map { |renderer|
+ next unless renderer["isPlayable"]
+
+ id = renderer.fetch("videoId")
+ Video.new(
+ id:,
+ title: text(renderer["title"]).to_s,
+ channel_title: text(renderer["shortBylineText"]),
+ thumbnail_url: "https://i.ytimg.com/vi/#{id}/mqdefault.jpg",
+ )
+ }
+ end
+
+ private
+
+ # Every page of a browse, following its continuations.
+ #: (String cookie, Hash[Symbol, String] body) -> Array[Hash[String, untyped]]
+ def pages(cookie, body)
+ pages = [] #: Array[Hash[String, untyped]]
+ loop do
+ page = request(cookie, body)
+ check(page) if pages.empty?
+ pages << page
+
+ token = find_all(page, "continuationItemRenderer")
+ .flat_map { |item| find_all(item, "continuationCommand") }
+ .first&.dig("token")
+ break unless token
+
+ body = { continuation: token }
+ end
+ pages
+ end
+
+ # A cookie YouTube has signed out still gets answers, as anyone would,
+ # so a sync must not take them for an empty playlist.
+ #: (Hash[String, untyped] page) -> void
+ def check(page)
+ params = page.dig("responseContext", "serviceTrackingParams").to_a.flat_map { |service| service["params"].to_a }
+ unless params.any? { |param| param["key"] == "logged_in" && param["value"] == "1" }
+ raise Error, "YouTube signed the cookie out. Paste a fresh one in settings."
+ end
+
+ alert = find_all(page, "alertRenderer").find { |a| a["type"] == "ERROR" }
+ raise Error, text(alert["text"]) || "YouTube sent an error" if alert
+ end
+
+ # The words of an InnerTube text, which comes either whole or in runs.
+ #: (untyped node) -> String?
+ def text(node)
+ return unless node.is_a?(Hash)
+
+ node["simpleText"] || node["runs"]&.map { |run| run["text"] }&.join
+ end
+
+ # Every value under key, however deep: InnerTube nests the same
+ # renderers differently from page to page.
+ #: (untyped node, String key, ?Array[Hash[String, untyped]] found) -> Array[Hash[String, untyped]]
+ def find_all(node, key, found = [])
+ case node
+ when Hash
+ node.each { |k, v| k == key ? found << v : find_all(v, key, found) }
+ when Array
+ node.each { |v| find_all(v, key, found) }
+ end
+ found
+ end
+
+ # Makes one browse request and returns its JSON body.
+ #: (String cookie, Hash[Symbol, String] body) -> Hash[String, untyped]
+ def request(cookie, body)
+ uri = URI(BROWSE_URL)
+ post = Net::HTTP::Post.new(uri)
+ post["Cookie"] = cookie
+ post["Authorization"] = self.class.authorization(cookie)
+ post["Origin"] = ORIGIN
+ post["X-Origin"] = ORIGIN
+ post["X-Goog-AuthUser"] = "0"
+ post["Content-Type"] = "application/json"
+ post.body = JSON.generate({ context: { client: CLIENT }, **body })
+
+ response = Net::HTTP.start(uri.hostname.to_s, uri.port, use_ssl: true) { |http| http.request(post) }
+ json = JSON.parse(response.body.to_s)
+ return json if response.is_a?(Net::HTTPSuccess)
+
+ raise Error, json.dig("error", "message") || "HTTP #{response.code}"
+ rescue JSON::ParserError
+ raise Error, "YouTube answered HTTP #{response&.code} with no JSON"
+ end
+ end
+end
diff --git a/lib/rank_king/models.rb b/lib/rank_king/models.rb
index 38a89e8..8e3f8ae 100644
--- a/lib/rank_king/models.rb
+++ b/lib/rank_king/models.rb
@@ -10,12 +10,6 @@ module RankKing
# @rbs skip
class User < Sequel::Model(DB[:users])
one_to_many :lists, class: "RankKing::List"
- one_to_one :google_account, class: "RankKing::GoogleAccount"
- end
-
- # @rbs skip
- class GoogleAccount < Sequel::Model(DB[:google_accounts])
- many_to_one :user, class: "RankKing::User"
end
# @rbs skip
diff --git a/lib/rank_king/sources/text.rb b/lib/rank_king/sources/text.rb
index 2262dd3..a32ad40 100644
--- a/lib/rank_king/sources/text.rb
+++ b/lib/rank_king/sources/text.rb
@@ -8,19 +8,19 @@ module RankKing
# over the list's items, which matches them by id, so there is nothing
# to fetch and no table of its own.
class Text
- #: (List list, google: Google) -> Text
- def self.for(list, google:) = new(list)
+ #: (List list, innertube: InnerTube) -> Text
+ def self.for(list, innertube:) = new(list)
#: () -> String
def self.title = "Typed list"
- #: (User user, google: Google) -> _Component
- def self.fields(user, google:) = Views::Sources::TextFields.new
+ #: (csrf: Method) -> _Component
+ def self.fields(csrf:) = Views::Sources::TextFields.new
# Adds the items typed into the new-list form, of which there must
# be at least two.
- #: (List list, Hash[String, untyped] params, google: Google) -> bool
- def self.create(list, params, google:)
+ #: (List list, Hash[String, untyped] params, innertube: InnerTube) -> bool
+ def self.create(list, params, innertube:)
names = params["items"].to_s.lines.map(&:strip).reject(&:empty?)
return false if names.size < 2
diff --git a/lib/rank_king/sources/youtube.rb b/lib/rank_king/sources/youtube.rb
index bbc3b31..5acdf27 100644
--- a/lib/rank_king/sources/youtube.rb
+++ b/lib/rank_king/sources/youtube.rb
@@ -1,53 +1,64 @@
# frozen_string_literal: true
-require_relative "../google"
+require_relative "../innertube"
require_relative "../sync"
require_relative "../views/sources/youtube"
module RankKing
module Sources
# A list that follows one of the owner's YouTube playlists, read
- # through their connected Google account, so private ones work too.
- # Each sync fetches the whole playlist again.
+ # through InnerTube, so private ones and Watch Later work too. Nothing
+ # signs rank-king in for good: each fetch takes a youtube.com cookie
+ # pasted in for it, and the whole playlist is fetched again.
class YouTube
# Which playlist each list follows, keyed by list_id.
TABLE = :youtube_sources
# Which video each item is, keyed by item_id.
ITEMS = :youtube_items
+ # Where the new-list form sends a pasted cookie for its choice of
+ # playlists.
+ PLAYLISTS_PATH = "/youtube/playlists"
#: () -> String
def self.title = "YouTube playlist"
- #: (List list, google: Google) -> YouTube
- def self.for(list, google:) = new(list, google:)
-
- # A choice of the user's playlists, fetched as the form is drawn. A
- # Google error says so in place of the choice rather than failing
- # the whole form.
- #: (User user, google: Google) -> _Component
- def self.fields(user, google:)
- account = user.google_account
- playlists = account && google.playlists(google.access_token(account.refresh_token))
- Views::Sources::YouTubeFields.new(playlists:)
- rescue Google::Error => e
- Views::Sources::YouTubeFields.new(playlists: nil, error: e.message)
+ #: (List list, innertube: InnerTube) -> YouTube
+ def self.for(list, innertube:) = new(list, innertube:)
+
+ #: (csrf: Method) -> _Component
+ def self.fields(csrf:) = Views::Sources::YouTubeFields.new(csrf:)
+
+ # The choice of playlists a pasted cookie opens. Whatever stops it
+ # is said in the choice's place.
+ #: (String text, innertube: InnerTube) -> _Component
+ def self.playlists(text, innertube:)
+ Views::Sources::YouTubePlaylists.new(playlists: innertube.playlists(cookie(text)))
+ rescue InnerTube::Error => e
+ Views::Sources::YouTubePlaylists.new(playlists: nil, error: e.message)
end
- # Follows the playlist chosen on the new-list form and fetches it.
- #: (List list, Hash[String, untyped] params, google: Google) -> bool
- def self.create(list, params, google:)
+ #: (untyped text) -> String
+ def self.cookie(text)
+ InnerTube.cookie(text.to_s) or raise InnerTube::Error, "That cookie has no SAPISID to sign requests with."
+ end
+
+ # Follows the playlist chosen on the new-list form and fetches it
+ # with the cookie pasted there.
+ #: (List list, Hash[String, untyped] params, innertube: InnerTube) -> bool
+ def self.create(list, params, innertube:)
playlist_id = params["playlist"].to_s[/\A[\w-]+\z/]
- return false unless playlist_id && list.user.google_account
+ cookie = InnerTube.cookie(params["cookie"].to_s)
+ return false unless playlist_id && cookie
DB[TABLE].insert(list_id: list.id, playlist_id:)
- new(list, google:).fetch
+ new(list, innertube:).fetch(cookie)
true
end
- #: (List list, google: Google) -> void
- def initialize(list, google:)
+ #: (List list, innertube: InnerTube) -> void
+ def initialize(list, innertube:)
@list = list
- @google = google
+ @innertube = innertube
end
#: () -> String
@@ -56,21 +67,17 @@ module RankKing
#: (csrf: Method) -> _Component
def form(csrf:) = Views::Sources::YouTube.new(list: @list, csrf:)
- # The dialog posts nothing to apply: syncing is fetching.
+ # The dialog posts the cookie to fetch with.
#: (Hash[String, untyped] params) -> void
def sync(params)
- fetch
+ fetch(self.class.cookie(params["cookie"]))
end
- #: () -> Sync::Summary
- def fetch
- account = @list.user.google_account
- raise Google::Error, "Connect Google in settings to sync this list" unless account
-
- access_token = @google.access_token(account.refresh_token)
+ #: (String cookie) -> Sync::Summary
+ def fetch(cookie)
# Kept for saved, which caches each video's details by its item.
- @videos = @google.playlist_videos(access_token, playlist_id).uniq(&:id).to_h { |video|
- [video.id, video] #: [String, Google::Video]
+ @videos = @innertube.playlist_videos(cookie, playlist_id).uniq(&:id).to_h { |video|
+ [video.id, video] #: [String, InnerTube::Video]
}
entries = @videos.each_value.map { |video| Entry.new(key: video.id, name: video.title) }
Sync.call(@list, entries, source: self)
diff --git a/lib/rank_king/views/home.rb b/lib/rank_king/views/home.rb
index 2232771..1a9d8eb 100644
--- a/lib/rank_king/views/home.rb
+++ b/lib/rank_king/views/home.rb
@@ -14,8 +14,6 @@ module RankKing
def view_template
render Layout.new do
render PageHeader.new(title: "Rank King") do
- a(href: "/settings", class: "icon-button", data_size: "lg", aria_label: "Settings",
- style: "--gl-icon: var(--rk-icon-settings)")
a(href: "/lists/new", class: "btn", data_variant: "primary", data_size: "sm",
style: "--gl-icon: var(--gl-icon-plus)") { "New list" }
end
diff --git a/lib/rank_king/views/settings.rb b/lib/rank_king/views/settings.rb
deleted file mode 100644
index a5c63e8..0000000
--- a/lib/rank_king/views/settings.rb
+++ /dev/null
@@ -1,45 +0,0 @@
-# frozen_string_literal: true
-
-require "phlex"
-
-require_relative "layout"
-
-module RankKing
- module Views
- # The user's own settings, as against a list's. A future setting is a
- # new section here.
- class Settings < Phlex::HTML
- def initialize(user:, csrf:)
- @user = user
- @csrf = csrf
- end
-
- def view_template
- render Layout.new(title: "Settings — Rank King") do
- render PageHeader.new(title: "Settings", back: "/", back_label: "Lists")
-
- section do
- h2(class: "type-label") { "Google" }
- if @user.google_account
- render_connected
- else
- p { "Connect Google to make lists from your YouTube playlists, private ones included." }
- a(href: "/settings/google/connect", class: "btn", data_variant: "primary") { "Connect Google" }
- end
- end
- end
- end
-
- private
-
- def render_connected
- action = "/settings/google/disconnect"
- p { "Connected. Lists can sync from your YouTube playlists." }
- form(method: "post", action:, data_confirm: "Disconnect Google? YouTube lists will stop syncing.") do
- input(type: "hidden", name: "_csrf", value: @csrf.call(action))
- button(type: "submit", data_variant: "danger", data_size: "sm") { "Disconnect" }
- end
- end
- end
- end
-end
diff --git a/lib/rank_king/views/sources/youtube.rb b/lib/rank_king/views/sources/youtube.rb
index 675ced5..f08612d 100644
--- a/lib/rank_king/views/sources/youtube.rb
+++ b/lib/rank_king/views/sources/youtube.rb
@@ -5,10 +5,38 @@ require "phlex"
module RankKing
module Views
module Sources
- # The YouTube playlist source's fields on the new-list form: a choice
- # of the user's playlists, once they have connected Google. playlists
- # is nil until then.
+ # Where a youtube.com cookie is pasted, on the new-list form and in
+ # the sync dialog. rank-king keeps it only for the one fetch.
+ class YouTubeCookie < Phlex::HTML
+ def view_template
+ label(class: "field") do
+ plain "YouTube cookie"
+ textarea(name: "cookie", rows: 3, required: true, autocomplete: "off", spellcheck: "false")
+ small { "The Cookie request header from a signed-in youtube.com tab, copied from your browser's developer tools. It is used for this fetch and not kept." }
+ end
+ end
+ end
+
+ # The YouTube playlist source's fields on the new-list form: a cookie,
+ # then a choice of the playlists it opens, which app.js fetches into
+ # [data-playlists] once the cookie is pasted.
class YouTubeFields < Phlex::HTML
+ def initialize(csrf:)
+ @csrf = csrf
+ end
+
+ def view_template
+ path = RankKing::Sources::YouTube::PLAYLISTS_PATH
+ render YouTubeCookie.new
+ div(data_playlists: path, data_csrf: @csrf.call(path)) do
+ render YouTubePlaylists.new(playlists: nil)
+ end
+ end
+ end
+
+ # The choice of playlists, or why there is none yet. playlists is nil
+ # until a cookie is pasted.
+ class YouTubePlaylists < Phlex::HTML
def initialize(playlists:, error: nil)
@playlists = playlists
@error = error
@@ -16,10 +44,12 @@ module RankKing
def view_template
if @error
- p { "Google would not list your playlists: #{@error}" }
- elsif @playlists&.empty?
+ p { "YouTube would not list your playlists: #{@error}" }
+ elsif @playlists.nil?
+ p { "Paste a cookie to choose one of your playlists." }
+ elsif @playlists.empty?
p { "Your YouTube account has no playlists yet." }
- elsif @playlists
+ else
label(class: "field") do
plain "Playlist"
select(name: "playlist", required: true) do
@@ -27,20 +57,14 @@ module RankKing
option(value: id) { title }
end
end
- small { "Private playlists are listed too. YouTube does not share Watch Later." }
- end
- else
- p do
- plain "Connect Google in "
- a(href: "/settings") { "your settings" }
- plain " to make a list from a YouTube playlist."
+ small { "Watch Later and private playlists are listed too." }
end
end
end
end
- # The YouTube playlist source's sync dialog: there is nothing to
- # fill in, so it says what syncing does and shows it working.
+ # The YouTube playlist source's sync dialog: a fresh cookie to fetch
+ # with, and what syncing does, shown working.
class YouTube < Phlex::HTML
def initialize(list:, csrf:)
@list = list
@@ -54,6 +78,7 @@ module RankKing
form(id: "sync-form", method: "post", action:, data_busy: "Syncing…") do
input(type: "hidden", name: "_csrf", value: @csrf.call(action))
p { "Fetches the playlist again. New videos are added, and videos that left it are hidden." }
+ render YouTubeCookie.new
if (synced_at = @list.synced_at)
small { "Last synced #{synced_at.strftime("%b %-d, %Y at %-I:%M %p")}." }
end
diff --git a/lib/rank_king/web.rb b/lib/rank_king/web.rb
index 1d687e9..a51aa7a 100644
--- a/lib/rank_king/web.rb
+++ b/lib/rank_king/web.rb
@@ -1,13 +1,11 @@
# frozen_string_literal: true
require "roda"
-require "securerandom"
require_relative "models"
require_relative "proxy_auth"
-require_relative "google"
+require_relative "innertube"
require_relative "views/home"
-require_relative "views/settings"
require_relative "views/list/new"
require_relative "views/list/show"
require_relative "views/list/compare"
@@ -19,11 +17,11 @@ module RankKing
plugin :static, %w[ /css /js /vendor ]
plugin :sessions, secret: CONFIG.session_secret
plugin :route_csrf, csrf_failure: :empty_403, check_request_methods: %w[POST]
- plugin :error_handler, classes: [ProxyAuth::MissingLogin, Google::Error] do |e|
+ plugin :error_handler, classes: [ProxyAuth::MissingLogin, InnerTube::Error] do |e|
response["Content-Type"] = "text/plain"
- if e.is_a?(Google::Error)
+ if e.is_a?(InnerTube::Error)
response.status = 502
- "Google: #{e.message}\n"
+ "YouTube: #{e.message}\n"
else
response.status = 401
"Unauthorized: #{e.message}.\n"
@@ -39,42 +37,10 @@ module RankKing
Views::Home.new(lists: lists).call
end
- r.on "settings" do
- r.get true do
- Views::Settings.new(user:, csrf: method(:csrf_token)).call
- end
-
- # Connecting Google is the OAuth dance: off to Google with a state
- # only this session knows, and back with a code for a refresh token.
- r.on "google" do
- redirect_uri = "#{r.base_url}/settings/google/callback"
-
- r.get "connect" do
- state = SecureRandom.hex(16)
- session["google_state"] = state
- r.redirect google.authorize_url(redirect_uri:, state:)
- end
-
- r.get "callback" do
- state = session.delete("google_state").to_s
- r.halt 403 if state.empty? || !Rack::Utils.secure_compare(state, r.params["state"].to_s)
- # The user said no on Google's side.
- r.redirect "/settings" if r.params["error"]
-
- refresh_token = google.refresh_token(r.params["code"].to_s, redirect_uri:)
- if (account = user.google_account)
- account.update(refresh_token:)
- else
- GoogleAccount.create(user_id: user.id, refresh_token:)
- end
- r.redirect "/settings"
- end
-
- r.post "disconnect" do
- user.google_account&.destroy
- r.redirect "/settings"
- end
- end
+ # A fragment for the new-list form: the playlists a pasted cookie
+ # opens. It is a POST so the cookie stays out of URLs and logs.
+ r.post "youtube", "playlists" do
+ Sources::YouTube.playlists(r.params["cookie"], innertube:).call
end
r.on "lists" do
@@ -92,7 +58,7 @@ module RankKing
# down and nothing is made.
list = DB.transaction {
l = List.create(name: name, user_id: user.id, source_kind: source_kind)
- raise Sequel::Rollback unless Sources.fetch(source_kind).create(l, r.params, google:)
+ raise Sequel::Rollback unless Sources.fetch(source_kind).create(l, r.params, innertube:)
axis_names.each do |n|
Axis.create(list_id: l.id, name: n)
@@ -108,7 +74,7 @@ module RankKing
r.get "new" do
kinds = Sources::KINDS.map { |kind|
source = Sources.fetch(kind)
- [kind, source.title, source.fields(user, google:)] #: [String, String, Sources::_Component]
+ [kind, source.title, source.fields(csrf: method(:csrf_token))] #: [String, String, Sources::_Component]
}
Views::ListViews::New.new(sources: kinds, csrf: method(:csrf_token)).call
end
@@ -200,13 +166,13 @@ module RankKing
private
- # The Google client every route and source goes through. Tests
+ # The InnerTube client every route and source goes through. Tests
# subclass the app to hand back a fake one.
- #: () -> Google
- def google = Google.new(client_id: CONFIG.google_client_id, client_secret: CONFIG.google_client_secret)
+ #: () -> InnerTube
+ def innertube = InnerTube.new
#: (List list) -> Sources::_Source
- def source(list) = Sources.fetch(list.source_kind).for(list, google:)
+ def source(list) = Sources.fetch(list.source_kind).for(list, innertube:)
# Names from a repeated field (`axes[]`), one per tag.
#: (untyped) -> Array[String]
diff --git a/public/css/app.css b/public/css/app.css
index 0df38f8..d8fe6b2 100644
--- a/public/css/app.css
+++ b/public/css/app.css
@@ -6,10 +6,6 @@
/* Lucide "ellipsis", drawn the way Gloss draws its own icon tokens.
Gloss has no "more" icon yet; this belongs upstream when it does. */
--rk-icon-more: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 24 24' fill='none' stroke='black' stroke-width='1.9' stroke-linecap='round' stroke-linejoin='round'%3E%3Ccircle cx='12' cy='12' r='1'/%3E%3Ccircle cx='19' cy='12' r='1'/%3E%3Ccircle cx='5' cy='12' r='1'/%3E%3C/svg%3E");
-
- /* Lucide "settings", drawn the same way. Gloss' own settings icon is a
- ring of loose spokes that reads as a sun at this size, not a gear. */
- --rk-icon-settings: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 24 24' fill='none' stroke='black' stroke-width='1.75' stroke-linecap='round' stroke-linejoin='round'%3E%3Cpath d='M12.22 2h-.44a2 2 0 0 0-2 2v.18a2 2 0 0 1-1 1.73l-.43.25a2 2 0 0 1-2 0l-.15-.08a2 2 0 0 0-2.73.73l-.22.38a2 2 0 0 0 .73 2.73l.15.1a2 2 0 0 1 1 1.72v.51a2 2 0 0 1-1 1.74l-.15.09a2 2 0 0 0-.73 2.73l.22.38a2 2 0 0 0 2.73.73l.15-.08a2 2 0 0 1 2 0l.43.25a2 2 0 0 1 1 1.73V20a2 2 0 0 0 2 2h.44a2 2 0 0 0 2-2v-.18a2 2 0 0 1 1-1.73l.43-.25a2 2 0 0 1 2 0l.15.08a2 2 0 0 0 2.73-.73l.22-.39a2 2 0 0 0-.73-2.73l-.15-.08a2 2 0 0 1-1-1.74v-.5a2 2 0 0 1 1-1.74l.15-.09a2 2 0 0 0 .73-2.73l-.22-.38a2 2 0 0 0-2.73-.73l-.15.08a2 2 0 0 1-2 0l-.43-.25a2 2 0 0 1-1-1.73V4a2 2 0 0 0-2-2z'/%3E%3Ccircle cx='12' cy='12' r='3'/%3E%3C/svg%3E");
}
/* -------------------- */
@@ -81,9 +77,6 @@ section {
gap: var(--gl-space-2xs);
}
-/* A section's own action keeps its button's width. */
-section > :is(a.btn, form) { align-self: flex-start; }
-
/* Actions at the foot of a form: one on the right, or two pushed apart. */
form footer {
display: flex;
diff --git a/public/js/app.js b/public/js/app.js
index 66edbce..ba312fd 100644
--- a/public/js/app.js
+++ b/public/js/app.js
@@ -67,6 +67,31 @@
if (event.target.matches("select[name=source]")) showSource(event.target);
});
+ // A pasted YouTube cookie fetches its playlists into the form's
+ // [data-playlists], which the server draws as a fragment.
+ var playlistsTimer;
+ document.addEventListener("input", function (event) {
+ var cookie = event.target;
+ var fieldset = cookie.name === "cookie" && cookie.closest("fieldset");
+ var target = fieldset && fieldset.querySelector("[data-playlists]");
+ if (!target) return;
+
+ clearTimeout(playlistsTimer);
+ playlistsTimer = setTimeout(function () {
+ if (!cookie.value.trim()) return;
+
+ var body = new FormData();
+ body.append("cookie", cookie.value);
+ body.append("_csrf", target.dataset.csrf);
+ target.setAttribute("aria-busy", "true");
+ fetch(target.dataset.playlists, { method: "POST", body: body })
+ .then(function (response) { return response.text(); })
+ .then(function (html) { target.innerHTML = html; })
+ .catch(function () { target.textContent = "Could not reach rank-king to list your playlists."; })
+ .finally(function () { target.removeAttribute("aria-busy"); });
+ }, 300);
+ });
+
document.addEventListener("DOMContentLoaded", function () {
document.querySelectorAll("select[name=source]").forEach(showSource);
});
diff --git a/sig/rank_king/config.rbs b/sig/rank_king/config.rbs
index 672805c..f359a38 100644
--- a/sig/rank_king/config.rbs
+++ b/sig/rank_king/config.rbs
@@ -5,9 +5,7 @@ module RankKing
class Config
attr_reader database_url: String
attr_reader session_secret: String
- attr_reader google_client_id: String
- attr_reader google_client_secret: String
- def self.new: (database_url: String, session_secret: String, google_client_id: String, google_client_secret: String) -> instance
+ def self.new: (database_url: String, session_secret: String) -> instance
end
end
diff --git a/sig/rank_king/google.rbs b/sig/rank_king/innertube.rbs
similarity index 63%
rename from sig/rank_king/google.rbs
rename to sig/rank_king/innertube.rbs
index 4dd1d82..99336d6 100644
--- a/sig/rank_king/google.rbs
+++ b/sig/rank_king/innertube.rbs
@@ -1,13 +1,13 @@
# Video is a Data class, whose members the inline parser cannot read.
module RankKing
- class Google
+ class InnerTube
class Video
attr_reader id: String
attr_reader title: String
- attr_reader channel_title: String
- attr_reader thumbnail_url: String?
+ attr_reader channel_title: String?
+ attr_reader thumbnail_url: String
- def self.new: (id: String, title: String, channel_title: String, thumbnail_url: String?) -> instance
+ def self.new: (id: String, title: String, channel_title: String?, thumbnail_url: String) -> instance
end
end
end
diff --git a/sig/rank_king/models.rbs b/sig/rank_king/models.rbs
index cf0aafc..caac5e6 100644
--- a/sig/rank_king/models.rbs
+++ b/sig/rank_king/models.rbs
@@ -6,12 +6,6 @@ module RankKing
def login: () -> String
def lists: () -> Array[List]
def lists_dataset: () -> Sequel::Dataset[List]
- def google_account: () -> GoogleAccount?
- end
-
- class GoogleAccount < Sequel::Model
- def user_id: () -> Integer
- def refresh_token: () -> String
end
class List < Sequel::Model
diff --git a/sig/rank_king/sources.rbs b/sig/rank_king/sources.rbs
index 4374991..fb54a16 100644
--- a/sig/rank_king/sources.rbs
+++ b/sig/rank_king/sources.rbs
@@ -36,18 +36,19 @@ module RankKing
# A source kind: a class in Sources, named by a list's source_kind.
interface _SourceClass
# The source for a list, configured from the kind's own table. Every
- # kind is handed the app's Google client, which only YouTube uses.
- def for: (List list, google: Google) -> _Source
+ # kind is handed the app's InnerTube client, which only YouTube uses.
+ def for: (List list, innertube: InnerTube) -> _Source
# What the kind is called where a list's source is chosen or shown.
def title: () -> String
- # The kind's own fields on the new-list form.
- def fields: (User user, google: Google) -> _Component
+ # The kind's own fields on the new-list form, which may post to
+ # routes of their own as they are filled in.
+ def fields: (csrf: Method) -> _Component
# Sets a new list up from what those fields posted: false, and the
# list is not made, when they are not enough to go on.
- def create: (List list, Hash[String, untyped] params, google: Google) -> bool
+ def create: (List list, Hash[String, untyped] params, innertube: InnerTube) -> bool
end
# A view component, as a source's form is.
diff --git a/sig/rank_king/views.rbs b/sig/rank_king/views.rbs
index ece8ee0..d7612a7 100644
--- a/sig/rank_king/views.rbs
+++ b/sig/rank_king/views.rbs
@@ -8,11 +8,6 @@ module RankKing
def call: () -> String
end
- class Settings
- def initialize: (user: User, csrf: Method) -> void
- def call: () -> String
- end
-
module Sources
class Text
def initialize: (list: List, csrf: Method) -> void
@@ -30,6 +25,11 @@ module RankKing
end
class YouTubeFields
+ def initialize: (csrf: Method) -> void
+ def call: () -> String
+ end
+
+ class YouTubePlaylists
def initialize: (playlists: Array[[String, String]]?, ?error: String?) -> void
def call: () -> String
end
diff --git a/test/test_helper.rb b/test/test_helper.rb
index baf903d..29efc4c 100644
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -1,7 +1,3 @@
# frozen_string_literal: true
ENV["DATABASE_URL"] = ":memory:"
-# The Google client the app requires. No request reaches Google: the
-# YouTube tests' app subclass hands out a fake one.
-ENV["GOOGLE_CLIENT_ID"] = "client-id"
-ENV["GOOGLE_CLIENT_SECRET_COMMAND"] = "echo client-secret"
diff --git a/test/test_youtube.rb b/test/test_youtube.rb
index 0252a5e..54971d8 100644
--- a/test/test_youtube.rb
+++ b/test/test_youtube.rb
@@ -4,73 +4,93 @@ require_relative "test_helper"
require "minitest/autorun"
require "rack/test"
-require "uri"
require_relative "../lib/rank_king/web"
class TestYouTube < Minitest::Test
include Rack::Test::Methods
- # The app, with the fake Google in place of the real one.
+ # The app, with the fake InnerTube in place of the real one.
class App < RankKing::Web
- singleton_class.attr_accessor :google
+ singleton_class.attr_accessor :innertube
- private def google = self.class.google
+ private def innertube = self.class.innertube
end
def app = App.app
LOGIN = "alpha@example.com"
PLAYLIST = "PLabc123"
+ COOKIE = "SID=sid; SAPISID=sapisid; __Secure-3PAPISID=3papisid"
- # Google, as far as rank-king talks to it: the real client with its
- # requests answered here. The playlist pages two videos at a time, so
- # fetching it has to follow nextPageToken.
- class FakeGoogle < RankKing::Google
+ # InnerTube, as far as rank-king talks to it: the real client with its
+ # requests answered here, in the shapes youtube.com gets. A playlist
+ # pages two videos at a time, so fetching it has to follow its
+ # continuations.
+ class FakeInnerTube < RankKing::InnerTube
attr_reader :calls
- attr_accessor :videos, :user_playlists, :error
+ attr_accessor :videos, :user_playlists, :error, :signed_out
def initialize
- super(client_id: "client-id", client_secret: "client-secret")
+ super
@calls = []
@videos = []
- @user_playlists = [["PLabc123", "Music"], ["PLdef456", "Talks"]]
+ @user_playlists = [["WL", "Watch later"], ["PLabc123", "Music"]]
end
private
- def request(method, url, params, access_token)
- @calls << [method, url, params, access_token]
- raise RankKing::Google::Error, error if error
-
- case url
- when RankKing::Google::TOKEN_URL
- if params[:grant_type] == "authorization_code"
- { "refresh_token" => "refresh-#{params[:code]}" }
- else
- { "access_token" => "access-#{params[:refresh_token]}" }
- end
- when RankKing::Google::PLAYLISTS_URL
- { "items" => user_playlists.map { |id, title| { "id" => id, "snippet" => { "title" => title } } } }
- when RankKing::Google::PLAYLIST_ITEMS_URL
- page = params[:pageToken].to_i
- body = { "items" => videos[page * 2, 2].to_a.map { |id, title| item(id, title) } }
- body["nextPageToken"] = (page + 1).to_s if videos.size > (page + 1) * 2
- body
- end
+ def request(cookie, body)
+ @calls << [cookie, body]
+ raise RankKing::InnerTube::Error, error if error
+ return signed_out_page if signed_out
+ return playlists_page if body[:browseId] == "FEplaylist_aggregation"
+
+ page = body[:continuation].to_i
+ items = videos[page * 2, 2].to_a.map { |id, title| video(id, title) }
+ items << continuation(page + 1, first: page.zero?) if videos.size > (page + 1) * 2
+ return signed_in("onResponseReceivedActions" => [{ "appendContinuationItemsAction" => { "continuationItems" => items } }]) if page.positive?
+
+ list = { "playlistVideoListRenderer" => { "contents" => items } }
+ signed_in("contents" => { "twoColumnBrowseResultsRenderer" => { "tabs" => [
+ { "tabRenderer" => { "content" => { "sectionListRenderer" => { "contents" => [
+ { "itemSectionRenderer" => { "contents" => [list] } },
+ ] } } } },
+ ] } })
end
- # A nil title stands for a deleted video, which has no owner.
- def item(id, title)
- snippet = { "title" => title || "Deleted video", "resourceId" => { "videoId" => id }, "thumbnails" => {} }
- if title
- snippet["videoOwnerChannelTitle"] = "A channel"
- snippet["thumbnails"] = {
- "default" => { "url" => "https://i.ytimg.com/vi/#{id}/default.jpg" },
- "medium" => { "url" => "https://i.ytimg.com/vi/#{id}/mqdefault.jpg" },
- }
- end
- { "snippet" => snippet }
+ def signed_in(body) = { "responseContext" => tracking("1"), **body }
+
+ def tracking(logged_in) = { "serviceTrackingParams" => [{ "params" => [{ "key" => "logged_in", "value" => logged_in }] }] }
+
+ def signed_out_page
+ { "responseContext" => tracking("0"),
+ "alerts" => [{ "alertRenderer" => { "type" => "ERROR", "text" => { "runs" => [{ "text" => "The playlist does not exist." }] } } }] }
+ end
+
+ def playlists_page
+ lockups = user_playlists.map { |id, title|
+ { "lockupViewModel" => { "contentId" => id, "contentType" => "LOCKUP_CONTENT_TYPE_PLAYLIST",
+ "metadata" => { "lockupMetadataViewModel" => { "title" => { "content" => title } } } } }
+ }
+ signed_in("contents" => { "richGridRenderer" => { "contents" => lockups.map { { "richItemRenderer" => { "content" => _1 } } } } })
+ end
+
+ # A nil title stands for a deleted video, which is not playable.
+ def video(id, title)
+ { "playlistVideoRenderer" => {
+ "videoId" => id, "isPlayable" => !title.nil?,
+ "title" => { "runs" => [{ "text" => title || "[Deleted video]" }] },
+ "shortBylineText" => { "runs" => [{ "text" => "A channel" }] },
+ } }
+ end
+
+ # The first page wraps its continuation in a command executor; later
+ # ones carry it bare.
+ def continuation(token, first:)
+ command = { "continuationCommand" => { "token" => token.to_s } }
+ endpoint = first ? { "commandExecutorCommand" => { "commands" => [{}, command] } } : command
+ { "continuationItemRenderer" => { "continuationEndpoint" => endpoint } }
end
end
@@ -82,105 +102,64 @@ class TestYouTube < Minitest::Test
RankKing::DB[:lists].delete
RankKing::DB[:users].delete
- @google = FakeGoogle.new
- App.google = @google
+ @innertube = FakeInnerTube.new
+ App.innertube = @innertube
end
- def test_settings_offers_to_connect_google
- get "/settings"
- assert last_response.ok?
- assert_includes last_response.body, %(href="/settings/google/connect")
- end
-
- def test_connecting_google_keeps_a_refresh_token
- get "/settings/google/connect"
- assert last_response.redirect?
- location = URI(last_response.location)
- query = URI.decode_www_form(location.query).to_h
- assert_equal "accounts.google.com", location.host
- assert_equal "client-id", query["client_id"]
- assert_equal "http://example.org/settings/google/callback", query["redirect_uri"]
- assert_equal RankKing::Google::SCOPE, query["scope"]
- assert_equal "offline", query["access_type"]
-
- get "/settings/google/callback", { state: query["state"], code: "the-code" }
- assert_equal "/settings", URI(last_response.location).path
- assert_equal "refresh-the-code", user.google_account.refresh_token
-
- get "/settings"
- assert_includes last_response.body, %(action="/settings/google/disconnect")
- end
-
- def test_callback_needs_the_state_it_was_sent_with
- get "/settings/google/connect"
- get "/settings/google/callback", { state: "forged", code: "the-code" }
-
- assert_equal 403, last_response.status
- assert_nil user.google_account
- end
-
- def test_callback_without_a_connect_is_refused
- get "/settings/google/callback", { state: "", code: "the-code" }
- assert_equal 403, last_response.status
- end
-
- def test_declining_on_google_connects_nothing
- get "/settings/google/connect"
- state = URI.decode_www_form(URI(last_response.location).query).to_h["state"]
- get "/settings/google/callback", { state:, error: "access_denied" }
-
- assert_equal "/settings", URI(last_response.location).path
- assert_nil user.google_account
- assert_empty @google.calls
- end
-
- def test_disconnecting_google
- connect_google
- csrf_post "/settings/google/disconnect", {}, from: "/settings"
-
- assert last_response.redirect?
- assert_nil user.google_account
- end
-
- def test_new_list_asks_to_connect_google_first
+ def test_new_list_asks_for_a_cookie_before_the_playlists
get "/lists/new"
assert_includes last_response.body, %(<option value="YouTube">YouTube playlist</option>)
- assert_includes last_response.body, %(<a href="/settings">your settings</a>)
+ assert_includes last_response.body, %(<textarea name="cookie")
+ assert_includes last_response.body, %(data-playlists="/youtube/playlists")
refute_includes last_response.body, %(name="playlist")
+ assert_empty @innertube.calls
end
- def test_new_list_offers_the_users_playlists
- connect_google
- get "/lists/new"
+ def test_a_pasted_cookie_lists_the_playlists
+ playlists("Cookie: #{COOKIE}\n")
+ assert last_response.ok?
assert_includes last_response.body, %(<select name="playlist" required>)
+ assert_includes last_response.body, %(<option value="WL">Watch later</option>)
assert_includes last_response.body, %(<option value="PLabc123">Music</option>)
- assert_includes last_response.body, %(<option value="PLdef456">Talks</option>)
+ refute_includes last_response.body, "<html"
+ assert_equal [COOKIE, { browseId: "FEplaylist_aggregation" }], @innertube.calls.last
+ end
- _, url, params, access_token = @google.calls.last
- assert_equal RankKing::Google::PLAYLISTS_URL, url
- assert_equal "true", params[:mine]
- assert_equal "access-refresh-the-code", access_token
+ def test_listing_playlists_needs_the_form_token
+ post "/youtube/playlists", { cookie: COOKIE }
+ assert_equal 403, last_response.status
+ assert_empty @innertube.calls
end
- def test_new_list_says_when_there_are_no_playlists
- connect_google
- @google.user_playlists = []
- get "/lists/new"
+ def test_a_cookie_with_nothing_to_sign_with_lists_nothing
+ playlists("SID=sid; HSID=hsid")
+ assert_includes last_response.body, "no SAPISID"
+ assert_empty @innertube.calls
+ end
+
+ def test_says_when_there_are_no_playlists
+ @innertube.user_playlists = []
+ playlists(COOKIE)
assert_includes last_response.body, "no playlists yet"
end
- def test_new_list_still_works_when_google_fails
- connect_google
- @google.error = "Token has been expired or revoked."
- get "/lists/new"
+ def test_says_when_youtube_signed_the_cookie_out
+ @innertube.signed_out = true
+ playlists(COOKIE)
assert last_response.ok?
- assert_includes last_response.body, "Token has been expired or revoked."
- assert_includes last_response.body, %(name="items")
+ assert_includes last_response.body, "YouTube signed the cookie out."
+ end
+
+ def test_authorization_signs_each_api_cookie
+ at = 1_700_000_000
+ sign = ->(value) { Digest::SHA1.hexdigest("#{at} #{value} https://www.youtube.com") }
+
+ assert_equal "SAPISIDHASH #{at}_#{sign.("sapisid")} SAPISID3PHASH #{at}_#{sign.("3papisid")}",
+ RankKing::InnerTube.authorization(COOKIE, at:)
end
def test_create_a_list_from_a_playlist
- connect_google
- @google.videos = [["v1", "First"], ["v2", nil], ["v3", "Third"], ["v4", "Fourth"], ["v1", "First again"]]
+ @innertube.videos = [["v1", "First"], ["v2", nil], ["v3", "Third"], ["v4", "Fourth"], ["v1", "First again"]]
create_list(playlist: PLAYLIST)
assert last_response.redirect?
@@ -196,73 +175,100 @@ class TestYouTube < Minitest::Test
assert_equal "A channel", first[:channel_title]
assert_equal "https://i.ytimg.com/vi/v1/mqdefault.jpg", first[:thumbnail_url]
- tokens = @google.calls.select { |_, url, _, _| url == RankKing::Google::TOKEN_URL }
- assert_equal "refresh-the-code", tokens.last[2][:refresh_token]
- pages = @google.calls.select { |_, url, _, _| url == RankKing::Google::PLAYLIST_ITEMS_URL }
- assert_equal 3, pages.size
- assert_equal ["access-refresh-the-code"], pages.map(&:last).uniq
+ assert_equal [{ browseId: "VLPLabc123" }, { continuation: "1" }, { continuation: "2" }], @innertube.calls.map(&:last)
+ assert_equal [COOKIE], @innertube.calls.map(&:first).uniq
end
- def test_create_needs_google
- create_list(playlist: PLAYLIST)
+ def test_create_a_list_from_watch_later
+ @innertube.videos = [["v1", "First"], ["v2", "Second"]]
+ create_list(playlist: "WL")
+
+ assert_equal %w[First Second], names(RankKing::List.first)
+ assert_equal({ browseId: "VLWL" }, @innertube.calls.last.last)
+ end
+
+ def test_create_needs_a_cookie
+ create_list(playlist: PLAYLIST, cookie: "SID=sid")
assert_equal 422, last_response.status
assert_equal 0, RankKing::List.count
+ assert_empty @innertube.calls
end
def test_create_needs_a_playlist
- connect_google
create_list(playlist: "not a playlist")
assert_equal 422, last_response.status
assert_equal 0, RankKing::List.count
end
def test_sync_adds_archives_and_restores_by_video
- connect_google
- @google.videos = [["v1", "First"], ["v2", "Second"]]
+ @innertube.videos = [["v1", "First"], ["v2", "Second"]]
create_list(playlist: PLAYLIST)
list = RankKing::List.first
first, second = list.items_dataset.order(:id).all
- @google.videos = [["v1", "First, renamed"], ["v3", "Third"]]
- sync(list)
+ @innertube.videos = [["v1", "First, renamed"], ["v3", "Third"]]
+ sync(list, cookie: "SAPISID=fresh")
assert last_response.redirect?
assert_equal ["First, renamed", "Third"], names(list)
refute_nil second.reload.archived_at
assert_equal "First, renamed", first.reload.name
assert_equal "First, renamed", RankKing::DB[:youtube_items].first(item_id: first.id)[:title]
+ assert_equal "SAPISID=fresh", @innertube.calls.last.first
- @google.videos = [["v2", "Second"]]
+ @innertube.videos = [["v2", "Second"]]
sync(list)
assert_equal %w[Second], names(list)
assert_nil second.reload.archived_at
end
- def test_sync_dialog_shows_progress
- connect_google
- @google.videos = [["v1", "First"], ["v2", "Second"]]
+ def test_sync_dialog_asks_for_a_cookie_and_shows_progress
+ @innertube.videos = [["v1", "First"], ["v2", "Second"]]
create_list(playlist: PLAYLIST)
get "/lists/#{RankKing::List.first.id}"
assert_includes last_response.body, "Sync from YouTube"
+ assert_includes last_response.body, %(<textarea name="cookie")
assert_includes last_response.body, %(data-busy="Syncing…")
end
- def test_google_errors_are_reported
- connect_google
- @google.videos = [["v1", "First"], ["v2", "Second"]]
+ def test_sync_needs_a_cookie
+ @innertube.videos = [["v1", "First"], ["v2", "Second"]]
create_list(playlist: PLAYLIST)
list = RankKing::List.first
+ @innertube.calls.clear
+
+ sync(list, cookie: "")
+ assert_equal 502, last_response.status
+ assert_includes last_response.body, "no SAPISID"
+ assert_empty @innertube.calls
+ end
- @google.error = "Token has been expired or revoked."
+ def test_a_signed_out_cookie_archives_nothing
+ @innertube.videos = [["v1", "First"], ["v2", "Second"]]
+ create_list(playlist: PLAYLIST)
+ list = RankKing::List.first
+
+ @innertube.signed_out = true
sync(list)
assert_equal 502, last_response.status
- assert_includes last_response.body, "Token has been expired or revoked."
+ assert_includes last_response.body, "YouTube: YouTube signed the cookie out."
+ assert_equal %w[First Second], names(list)
+ end
+
+ def test_youtube_errors_are_reported
+ @innertube.videos = [["v1", "First"], ["v2", "Second"]]
+ create_list(playlist: PLAYLIST)
+ list = RankKing::List.first
+
+ @innertube.error = "Request contains an invalid argument."
+ sync(list)
+ assert_equal 502, last_response.status
+ assert_includes last_response.body, "Request contains an invalid argument."
assert_equal %w[First Second], names(list)
end
def test_deleting_a_youtube_list
- connect_google
- @google.videos = [["v1", "First"], ["v2", "Second"]]
+ @innertube.videos = [["v1", "First"], ["v2", "Second"]]
create_list(playlist: PLAYLIST)
list = RankKing::List.first
@@ -274,23 +280,22 @@ class TestYouTube < Minitest::Test
private
- def user = RankKing::User.first(login: LOGIN)
+ def names(list) = list.items_dataset.order(:id).map(:name)
- def connect_google
- get "/settings/google/connect"
- state = URI.decode_www_form(URI(last_response.location).query).to_h["state"]
- get "/settings/google/callback", { state:, code: "the-code" }
- @google.calls.clear
+ # Posts the cookie as app.js does, with the token the new-list form
+ # carries for it.
+ def playlists(cookie)
+ get "/lists/new"
+ token = last_response.body[/data-playlists="\/youtube\/playlists" data-csrf="([^"]+)"/, 1]
+ post "/youtube/playlists", { cookie:, "_csrf" => token }
end
- def names(list) = list.items_dataset.order(:id).map(:name)
-
- def create_list(playlist:)
- csrf_post "/lists", { name: "Videos", source: "YouTube", playlist: }, from: "/lists/new"
+ def create_list(playlist:, cookie: COOKIE)
+ csrf_post "/lists", { name: "Videos", source: "YouTube", playlist:, cookie: }, from: "/lists/new"
end
- def sync(list)
- csrf_post "/lists/#{list.id}/sync", {}, from: "/lists/#{list.id}"
+ def sync(list, cookie: COOKIE)
+ csrf_post "/lists/#{list.id}/sync", { cookie: }, from: "/lists/#{list.id}"
end
def csrf_post(path, params, from:)