Remove email ingest
Assisted-by: Claude Sonnet 4.6 via Claude Code
diff --git a/README.md b/README.md
index c7be079..dfc5ca1 100644
--- a/README.md
+++ b/README.md
@@ -31,7 +31,7 @@ Purchase dates are fuzzy. Some receipts are decades old, and forcing an exact da
### Documents
-A **document** is any file the household wants to keep findable — tax returns, school records, contracts, warranties, manuals, medical paperwork. Supported file types include PDFs, images, and emails. Documents don't have to relate to the home itself; the inventory is one source of documents, not the only one. Documents have tags. OCR comes later.
+A **document** is any file the household wants to keep findable — tax returns, school records, contracts, warranties, manuals, medical paperwork. Supported file types include PDFs and images. Documents don't have to relate to the home itself; the inventory is one source of documents, not the only one. Documents have tags. OCR comes later.
Where a document does relate to an asset (a receipt, a manual), the two cross-reference each other.
diff --git a/db/migrate/001_create_documents.rb b/db/migrate/001_create_documents.rb
new file mode 100644
index 0000000..29268a3
--- /dev/null
+++ b/db/migrate/001_create_documents.rb
@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+ change do
+ create_table(:documents) do
+ primary_key :id
+ String :path, null: false
+ String :kind, null: false
+ DateTime :received_at, null: false
+ DateTime :created_at, null: false
+ end
+ end
+end
diff --git a/db/migrate/001_create_emails_and_attachments.rb b/db/migrate/001_create_emails_and_attachments.rb
deleted file mode 100644
index acd26fc..0000000
--- a/db/migrate/001_create_emails_and_attachments.rb
+++ /dev/null
@@ -1,30 +0,0 @@
-# frozen_string_literal: true
-
-Sequel.migration do
- change do
- create_table(:documents) do
- primary_key :id
- String :path, null: false
- String :kind, null: false # "email", "pdf", "image", "unknown"
- DateTime :received_at, null: false
- DateTime :created_at, null: false
- end
-
- create_table(:email_attachments) do
- primary_key :id
- foreign_key :email_id, :documents, null: false, index: true
- foreign_key :document_id, :documents, null: false, index: true
- String :filename, null: false
- DateTime :created_at, null: false
- end
-
- run <<~SQL
- CREATE TRIGGER enforce_email_attachment_kind
- BEFORE INSERT ON email_attachments
- BEGIN
- SELECT RAISE(ABORT, 'email_id must reference a document with kind=''email''')
- WHERE (SELECT kind FROM documents WHERE id = NEW.email_id) != 'email';
- END;
- SQL
- end
-end
diff --git a/docs/plans/2026-06-02-email-ingestion-design.md b/docs/plans/2026-06-02-email-ingestion-design.md
deleted file mode 100644
index 8a388a1..0000000
--- a/docs/plans/2026-06-02-email-ingestion-design.md
+++ /dev/null
@@ -1,80 +0,0 @@
-# Email ingestion design
-
-Domus receives emails at a fixed address and saves them as searchable
-records. Each email becomes an `email` record; each attachment becomes
-a linked `attachment` record. This document covers the ingestion
-pipeline only — storage, parsing, and persistence. It does not cover
-display, search, or tagging.
-
-## Schema
-
-Two new tables:
-
-**`emails`**
-
-| column | type | notes |
-|---|---|---|
-| `id` | integer PK | |
-| `path` | text not null | path to `.eml` file, relative to `storage_path` |
-| `subject` | text | |
-| `from` | text | |
-| `received_at` | datetime | |
-| `created_at` | datetime | |
-
-**`attachments`**
-
-| column | type | notes |
-|---|---|---|
-| `id` | integer PK | |
-| `email_id` | integer not null | FK → `emails.id` |
-| `path` | text not null | path to attachment file, relative to `storage_path` |
-| `filename` | text not null | original filename from the email |
-| `content_type` | text | |
-| `created_at` | datetime | |
-
-## File storage
-
-`Config` gains a `storage_path` field, defaulting to `"storage"` relative to
-the app root. Files land under two subdirectories:
-
-```
-storage/
- emails/
- 2026-06-02-<uuid>.eml
- attachments/
- 2026-06-02-<uuid>-<original-filename>
-```
-
-The date prefix keeps files browsable on disk. The UUID prevents collisions.
-The `path` columns store paths relative to `storage_path`, so moving the
-storage root requires no database updates.
-
-SendGrid Inbound Parse must be configured in **raw mode**, which delivers the
-full RFC 822 message as a single field. The stored `.eml` file is then a
-valid, openable email.
-
-## Webhook endpoint
-
-`POST /inbound/email` receives the SendGrid payload.
-
-### IP allowlisting
-
-A middleware checks `REMOTE_ADDR` against SendGrid's published Inbound Parse
-IP ranges before the request reaches the route handler. Requests from
-unlisted IPs receive a 403 response. The middleware accepts a config override
-for local development and testing.
-
-### Processing
-
-When a request passes the IP check, the handler:
-
-1. Parses the multipart form — raw email in the `email` field, attachments as
- `attachment1`, `attachment2`, etc.
-2. Writes the `.eml` to `storage/emails/`
-3. Inserts an `emails` row
-4. For each attachment: writes the file to `storage/attachments/` and inserts
- an `attachments` row linked to the email
-5. Returns 200
-
-Steps 2–4 run in a single database transaction. If any step fails, no partial
-records are written and no files are kept.
diff --git a/lib/ingest_email.rb b/lib/ingest_email.rb
deleted file mode 100644
index 1bc819a..0000000
--- a/lib/ingest_email.rb
+++ /dev/null
@@ -1,143 +0,0 @@
-# frozen_string_literal: true
-
-require "securerandom"
-require "fileutils"
-
-module Domus
- class IngestEmail
- def initialize(app)
- @config = app.config
- @db = app.db
- @storage = app.config.storage_path
- end
-
- def call(email_field:, attachments: [])
- now = Time.now
- eml_path = nil
- written_files = []
- committed = false
-
- db.transaction do
- eml_path = write_eml(email_field, now)
- written_files << eml_path
-
- email_doc_id = db[:documents].insert(
- path: eml_path,
- kind: "email",
- received_at: now,
- created_at: now
- )
-
- attachments.each do |att|
- kind = classify(att[:filename])
- content = att[:tempfile].read
- relative = write_file(storage_subdir(kind), att[:filename], content, now)
- written_files << relative
-
- doc_id = db[:documents].insert(
- path: relative,
- kind: kind.to_s,
- received_at: now,
- created_at: now
- )
-
- db[:email_attachments].insert(
- email_id: email_doc_id,
- document_id: doc_id,
- filename: att[:filename],
- created_at: now
- )
- end
-
- committed = true
- end
-
- { email_path: eml_path }
- ensure
- written_files.each { |rel| FileUtils.rm_f(File.join(@storage, rel)) } unless committed
- end
-
- class << self
- def handle(roda)
- app = roda.opts[:app]
- email_field, attachments = parse_multipart(roda.request)
- unless email_field
- roda.response.status = 400
- roda.response.write "Bad Request: missing email field"
- return
- end
-
- new(app).call(email_field: email_field, attachments: attachments)
- roda.response.status = 200
- roda.response.write "OK"
- rescue StandardError => e
- roda.response.status = 500
- roda.response.write "Error: #{e.message}"
- end
-
- def parse_multipart(request)
- email_field = request.params["email"]
- attachments = []
-
- request.params.each do |key, value|
- next unless key.to_s.match?(/^attachment\d+$/)
-
- att = attachment_from_param(value)
- attachments << att if att
- end
-
- [email_field, attachments]
- end
-
- def attachment_from_param(value)
- if value.respond_to?(:tempfile) && value.respond_to?(:original_filename)
- { filename: value.original_filename, tempfile: value }
- elsif value.is_a?(Hash) && value[:tempfile].is_a?(Tempfile)
- { filename: value[:filename], tempfile: value[:tempfile] }
- end
- end
- end
-
- private
-
- attr_reader :db
-
- def classify(filename)
- ext = File.extname(filename).downcase
- return :pdf if ext == ".pdf"
- return :image if %w[.jpg .jpeg .png .gif .webp .bmp .tiff .tif .svg].include?(ext)
-
- :unknown
- end
-
- def storage_subdir(kind)
- case kind
- when :pdf then "pdfs"
- when :image then "images"
- else "files"
- end
- end
-
- def write_eml(content, now)
- write_file("emails", nil, content, now)
- end
-
- def write_file(subdir, original_filename, content, now)
- relative = File.join(subdir, filename(original_filename, now))
- full = File.join(@storage, relative)
- FileUtils.mkdir_p(File.dirname(full))
- mode = content.encoding == Encoding::ASCII_8BIT ? "wb" : "w"
- File.write(full, content, mode: mode)
- relative
- end
-
- def filename(original = nil, now = Time.now)
- date = now.strftime("%Y-%m-%d")
- uuid = SecureRandom.uuid
- base = "#{date}-#{uuid}"
- return "#{base}.eml" unless original
-
- "#{base}-#{File.basename(original)}"
- end
- end
-end
diff --git a/lib/middleware/ip_allowlist.rb b/lib/middleware/ip_allowlist.rb
deleted file mode 100644
index bb45b33..0000000
--- a/lib/middleware/ip_allowlist.rb
+++ /dev/null
@@ -1,43 +0,0 @@
-# frozen_string_literal: true
-
-require "ipaddr"
-
-module Domus
- module Middleware
- class IpAllowlist
- # Source: https://sendgrid.com/en-us/blog/smtp-ip-range-sendgrid-2024
- DEFAULT_SENDGRID_IPS = %w[
- 167.89.115.0/24
- 167.89.114.0/24
- 159.183.224.0/20
- 159.183.240.0/20
- ].freeze
-
- def self.allowed?(addr, ips = DEFAULT_SENDGRID_IPS)
- ips.map { |ip| IPAddr.new(ip) }.any? { |range| range.include?(addr) }
- end
-
- def initialize(app, allowed: nil)
- @app = app
- @allowed = build_allowed(allowed || DEFAULT_SENDGRID_IPS)
- end
-
- def call(env)
- addr = env["REMOTE_ADDR"]
- return [403, {}, ["Forbidden"]] unless addr && allowed?(addr)
-
- @app.call(env)
- end
-
- private
-
- def build_allowed(ips)
- ips.map { |ip| IPAddr.new(ip) }
- end
-
- def allowed?(addr)
- @allowed.any? { |range| range.include?(addr) }
- end
- end
- end
-end
diff --git a/lib/models.rb b/lib/models.rb
index fd06e18..dff4000 100644
--- a/lib/models.rb
+++ b/lib/models.rb
@@ -2,11 +2,5 @@
module Domus
class Document < Sequel::Model
- one_to_many :email_attachments, key: :email_id, class: :EmailAttachment
- end
-
- class EmailAttachment < Sequel::Model
- many_to_one :email, class: :Document, key: :email_id
- many_to_one :document, class: :Document, key: :document_id
end
end
diff --git a/lib/web.rb b/lib/web.rb
index 41750ff..d4d1f86 100644
--- a/lib/web.rb
+++ b/lib/web.rb
@@ -2,8 +2,6 @@
require "roda"
require_relative "views/layout"
-require_relative "ingest_email"
-require_relative "middleware/ip_allowlist"
module Domus
class Web < Roda
@@ -11,15 +9,6 @@ module Domus
r.root do
render_with_layout { "ok" }
end
-
- r.post "inbound/email" do
- ip = request.env["REMOTE_ADDR"]
- unless ip && Middleware::IpAllowlist.allowed?(ip)
- response.status = 403
- next "Forbidden"
- end
- IngestEmail.handle(self)
- end
end
private
diff --git a/test/test_ingest_email.rb b/test/test_ingest_email.rb
deleted file mode 100644
index c98de55..0000000
--- a/test/test_ingest_email.rb
+++ /dev/null
@@ -1,139 +0,0 @@
-# frozen_string_literal: true
-
-require_relative "test_helper"
-require "rack/test"
-require "tempfile"
-require "fileutils"
-require "sequel"
-require_relative "../lib/ingest_email"
-require_relative "../lib/config"
-
-class TestIngestEmail < Minitest::Test
- include Rack::Test::Methods
-
- def app
- Domus::Web
- end
-
- def setup
- @test_storage = Dir.mktmpdir("domus-email-test")
- @config = Domus::Config.new(database_url: ":memory:", storage_path: @test_storage)
- @app = Domus::App.new(@config)
- Sequel::Migrator.run(@app.db, migrate_dir)
-
- @old_app = Domus::Web.opts[:app]
- Domus::Web.opts[:app] = @app
- end
-
- def teardown
- Domus::Web.opts[:app] = @old_app
- FileUtils.rm_rf(@test_storage)
- end
-
- def test_rejects_missing_email_field
- post "/inbound/email", {}, {"REMOTE_ADDR" => "167.89.115.100"}
- assert_equal 400, last_response.status
- end
-
- def test_accepts_email_without_attachments
- post "/inbound/email", {email: sample_email}, {"REMOTE_ADDR" => "167.89.115.100"}
- assert_equal 200, last_response.status
-
- email_doc = @app.db[:documents].where(kind: "email").first
- refute_nil email_doc
- assert_match %r{emails/\d{4}-\d{2}-\d{2}-.+\.eml}, email_doc[:path]
- end
-
- def test_stores_eml_file
- post "/inbound/email", {email: sample_email}, {"REMOTE_ADDR" => "167.89.115.100"}
- assert_equal 200, last_response.status
-
- email_doc = @app.db[:documents].where(kind: "email").first
- full_path = File.join(@test_storage, email_doc[:path])
- assert File.exist?(full_path)
- assert_equal sample_email, File.read(full_path)
- end
-
- def test_stores_pdf_attachment
- post "/inbound/email",
- {email: sample_email,
- attachment1: attachment_param("report.pdf", "application/pdf", "pdf content")},
- {"REMOTE_ADDR" => "167.89.115.100"}
-
- assert_equal 200, last_response.status
-
- ea = @app.db[:email_attachments].first
- refute_nil ea
- assert_match(/report\.pdf/, ea[:filename])
-
- email_doc = @app.db[:documents].where(kind: "email").first
- assert_equal email_doc[:id], ea[:email_id]
-
- att_doc = @app.db[:documents].where(id: ea[:document_id]).first
- full_path = File.join(@test_storage, att_doc[:path])
- assert File.exist?(full_path)
- assert_equal "pdf content", File.read(full_path)
- end
-
- def test_stores_multiple_attachments
- post "/inbound/email",
- {email: sample_email,
- attachment1: attachment_param("a.txt", "text/plain", "aaa"),
- attachment2: attachment_param("b.txt", "text/plain", "bbb")},
- {"REMOTE_ADDR" => "167.89.115.100"}
-
- assert_equal 200, last_response.status
- assert_equal 2, @app.db[:email_attachments].count
- assert_equal 3, @app.db[:documents].count
- end
-
- def test_attachment_linked_to_correct_email
- post "/inbound/email",
- {email: sample_email,
- attachment1: attachment_param("doc.pdf", "application/pdf", "data")},
- {"REMOTE_ADDR" => "167.89.115.100"}
-
- email_doc = @app.db[:documents].where(kind: "email").first
- ea = @app.db[:email_attachments].first
- assert_equal email_doc[:id], ea[:email_id]
- end
-
- def test_pdf_attachment_has_correct_kind
- post "/inbound/email",
- {email: sample_email,
- attachment1: attachment_param("file.pdf", "application/pdf", "data")},
- {"REMOTE_ADDR" => "167.89.115.100"}
-
- ea = @app.db[:email_attachments].first
- doc = @app.db[:documents].where(id: ea[:document_id]).first
- assert_equal "pdf", doc[:kind]
- end
-
- def test_attachment_path_includes_random_uuid
- post "/inbound/email",
- {email: sample_email,
- attachment1: attachment_param("file.pdf", "application/pdf", "data")},
- {"REMOTE_ADDR" => "167.89.115.100"}
-
- ea = @app.db[:email_attachments].first
- doc = @app.db[:documents].where(id: ea[:document_id]).first
- assert_match %r{pdfs/\d{4}-\d{2}-\d{2}-.+file\.pdf}, doc[:path]
- end
-
- private
-
- def migrate_dir
- File.expand_path("../db/migrate", __dir__)
- end
-
- def sample_email
- "From: sender@example.com\nSubject: Test Subject\n\nBody"
- end
-
- def attachment_param(filename, type, content)
- tempfile = Tempfile.new(filename)
- tempfile.write(content)
- tempfile.rewind
- Rack::Test::UploadedFile.new(tempfile.path, type, original_filename: filename)
- end
-end
diff --git a/test/test_ip_allowlist.rb b/test/test_ip_allowlist.rb
deleted file mode 100644
index 5f15f09..0000000
--- a/test/test_ip_allowlist.rb
+++ /dev/null
@@ -1,31 +0,0 @@
-# frozen_string_literal: true
-
-require_relative "test_helper"
-require "rack/test"
-require_relative "../lib/middleware/ip_allowlist"
-
-class TestIpAllowlist < Minitest::Test
- include Rack::Test::Methods
-
- def app
- inner = proc { |env| [200, {}, ["ok"]] }
- Domus::Middleware::IpAllowlist.new(inner, allowed: ["167.89.115.0/24"])
- end
-
- def test_allows_request_from_listed_ip
- get "/", {}, { "REMOTE_ADDR" => "167.89.115.100" }
- assert_equal 200, last_response.status
- end
-
- def test_blocks_request_from_unlisted_ip
- get "/", {}, { "REMOTE_ADDR" => "10.0.0.1" }
- assert_equal 403, last_response.status
- end
-
- def test_blocks_request_without_remote_addr
- get "/"
- assert_equal 403, last_response.status
- end
-
-
-end