Notes

by Matt Stubbs

Notes on deploying Phoenix application to fly.io with Crunchy Data postgres

15 October 2024

Deploying an Elixir Phoenix app to fly.io with Postgres running on Crunchy Data works well but there a couple of steps to get it all working. I’m making a note of it here for future reference.

Crunchy Bridge requires IPv4

Once you’ve set the DATABASE_URL with the connection string from Crunchy Bridge you might see NXdomain errors: Postgres can’t connect to the host.

[error] Postgrex.Protocol (#PID<0.2494.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (p.56vnpfhikjfn5k54wdvvweir44.db.postgresbridge.com:5432): non-existing domain - :nxdomain

Fly uses IPv6, but the connection string Crunchy Bridge supplies is IPv4. The default config/runtime.exs includes an IPv6 configuration option for Ecto which can be removed:

  config :myapp, MyApp.Repo,
    url: database_url,
    socket_options: maybe_ipv6, # <-- Remove this line
    pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10")

Crunchy Bridge enforces SSL connections

The app can now reach the database host but another error appeared:

[error] Postgrex.Protocol (#PID<0.2499.0>) failed to connect: ** (DBConnection.ConnectionError) ssl connect: Options (or their values) can not be combined: [{verify,verify_peer},
2024-10-14T23:31:34Z app[6e82541eb56648] syd [info]                                                {cacerts,undefined}] - {:options, :incompatible, [verify: :verify_peer, cacerts: :undefined]}

This one is to do with SSL verification. Crunchy Bridge quite reasonably requires all connections to use SSL. Their documentation instructs you to enable SSL by setting the Repo configuration to ssl: true. This should work unless you’re on OTP 26+. According to a forum post (How to set SSL options correctly when connecting to Heroku postgres db?) OTP 26 changed the default SSL verification setting from verify_none to verify_peer. Ecto is now trying to connect over SSL and verify the certificate.

The simple solution here is to switch it back to the old setting in runtime.exs

  config :my_app, MyApp.Repo,
    ssl: true,
    ssl_opts: [verify: :verify_none],
    url: database_url

If you want to verify the server certificate it requires a little more configuration:

  1. Download the public certificate from Crunchy Bridge. Certificates are issued per team, so get the correct certificate for the team the database belongs to. The certificate is available through the dashboard or API.
  2. Save the certificate in priv/repo/team.pem
  3. Update the Ecto SSL config:
  database_host_name = URI.parse(database_url).host

  certfile_path =
    :code.priv_dir(:my_app)
    |> Path.join("repo/team.pem")

  config :my_app, MyApp.Repo,
    ssl: true,
    ssl_opts: [
      verify: :verify_peer,
      cacertfile: certfile_path,
      server_name_indication: String.to_charlist(database_host_name)
    ],
    url: database_url,

Deploy and Ecto should now be able to connect.

Migrations won’t run without ssl started

Attempting to run migrations in production with the Release module may result in this error:

 ** (RuntimeError) SSL connection can not be established because `:ssl` application is not started,
you can add it to `extra_applications` in your `mix.exs`:

Add :ssl.start() to MyApp.Release.load_app/0 :

  defp load_app do
    Application.load(@app)
    :ssl.start()
  end

Canonical log lines

17 September 2024

Canonical log lines are a concept introduced in the Stripe blog post Fast and flexible observability with canonical log lines :

We’ve found using a slight augmentation to traditional logging immensely useful at Stripe—an idea that we call canonical log lines. It’s quite a simple technique: in addition to their normal log traces, requests also emit one long log line at the end that includes many of their key characteristics. Having that data colocated in single information-dense lines makes queries and aggregations over it faster to write, and faster to run.

I’ve often included some structured logging in my log files, by prepending log lines with useful context:

[2020-01-01 23:00:00] [user_id=32] User successfully logged in.

This makes it easy to filter all log lines related to the user account 32. Using the = keeps it readable and makes it machine parseable.

But Stripe found stitching lots of data together can be difficult:

Although logs offer additional flexibility in the examples above, we’re still left in a difficult situation if we want to query information across the lines in a trace. For example, if we notice there’s a lot of rate limiting occurring in the API, we might ask ourselves the question, “Which users are being rate limited the most?”

Stripe introduced canonical log lines in addition to the regular log statements. The canonical lines don’t prioritise human readability. They are dense, are emitted exactly once per event (web request, API call, background job, etc) with all the relevant data. They are canonical because “it’s the authoritative line for a particular request”.

In Metrics For Your Web Application's Dashboards Simon Hørup Eskildsen describes a JSON version of the canonical log line used by Readwise:

{
  "severity": "info",
  "processing_time": 0.273,
  "enqueued_to_processed": 1532.069,
  "finished_at": 1647783951.0888088,
  "time_in_queue": 1531.796,
  "worker_name": "worker-h@4877ad9e-2884-4973-8f8a-f4e09934e0ad",
  "task_class": "main.tasks.send_daily_review_push_notification",
  "task_args": [],
  "task_id": "68ba3676-ef44-4515-a154-46d70748d9a0",
  "retries": 0,
  "task_kwargs": {
    "profile_id": "[redacted]"
  },
  "hostname": "4877ad9e-2884-4973-8f8a-f4e09934e0ad",
  "enqueued_at": 1647782419.0191329,
  "started_processing_at": 1647783950.8150532,
  "parent_id": "b6168daa-2edf-4e4e-8b15-a4b72a35fa02",
  "root_id": "b6168daa-2edf-4e4e-8b15-a4b72a35fa02",
  "state": "SUCCESS",
  "category": "celery",
  "queue": "t2h-c1",
  "memory_mib": 213.3
}

His version uses JSON, which probably makes it a little more readable.

electric: sync postgres to the browser

19 August 2024

Electric provides an HTTP API for syncing Shapes of data from Postgres.

Replicates changes from postgres. Use the electric typescript client to sync data to your browser application. It uses a concept called Shapes to specify what data should be synced. The sync service is written in Elixir.

Looks like a good option for "I need Postgres on the server, but something like Firebase in the web client"

All notes →