Notes
by Matt Stubbs
Notes on deploying Phoenix application to fly.io with Crunchy Data postgres
15 October 2024
Deploying an Elixir Phoenix app to fly.io with Postgres running on Crunchy Data works well but there a couple of steps to get it all working. I’m making a note of it here for future reference.
Crunchy Bridge requires IPv4
Once you’ve set the DATABASE_URL
with the connection string from Crunchy Bridge you might see NXdomain
errors: Postgres can’t connect to the host.
[error] Postgrex.Protocol (#PID<0.2494.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (p.56vnpfhikjfn5k54wdvvweir44.db.postgresbridge.com:5432): non-existing domain - :nxdomain
Fly uses IPv6, but the connection string Crunchy Bridge supplies is IPv4. The default config/runtime.exs
includes an IPv6 configuration option for Ecto which can be removed:
config :myapp, MyApp.Repo,
url: database_url,
socket_options: maybe_ipv6, # <-- Remove this line
pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10")
Crunchy Bridge enforces SSL connections
The app can now reach the database host but another error appeared:
[error] Postgrex.Protocol (#PID<0.2499.0>) failed to connect: ** (DBConnection.ConnectionError) ssl connect: Options (or their values) can not be combined: [{verify,verify_peer},
2024-10-14T23:31:34Z app[6e82541eb56648] syd [info] {cacerts,undefined}] - {:options, :incompatible, [verify: :verify_peer, cacerts: :undefined]}
This one is to do with SSL verification. Crunchy Bridge quite reasonably requires all connections to use SSL. Their documentation instructs you to enable SSL by setting the Repo configuration to ssl: true
. This should work unless you’re on OTP 26+. According to a forum post (How to set SSL options correctly when connecting to Heroku postgres db?) OTP 26 changed the default SSL verification setting from verify_none
to verify_peer
. Ecto is now trying to connect over SSL and verify the certificate.
The simple solution here is to switch it back to the old setting in runtime.exs
config :my_app, MyApp.Repo,
ssl: true,
ssl_opts: [verify: :verify_none],
url: database_url
If you want to verify the server certificate it requires a little more configuration:
- Download the public certificate from Crunchy Bridge. Certificates are issued per team, so get the correct certificate for the team the database belongs to. The certificate is available through the dashboard or API.
- Save the certificate in
priv/repo/team.pem
- Update the Ecto SSL config:
database_host_name = URI.parse(database_url).host
certfile_path =
:code.priv_dir(:my_app)
|> Path.join("repo/team.pem")
config :my_app, MyApp.Repo,
ssl: true,
ssl_opts: [
verify: :verify_peer,
cacertfile: certfile_path,
server_name_indication: String.to_charlist(database_host_name)
],
url: database_url,
Deploy and Ecto should now be able to connect.
Migrations won’t run without ssl
started
Attempting to run migrations in production with the Release module may result in this error:
** (RuntimeError) SSL connection can not be established because `:ssl` application is not started,
you can add it to `extra_applications` in your `mix.exs`:
Add :ssl.start()
to MyApp.Release.load_app/0
:
defp load_app do
Application.load(@app)
:ssl.start()
end
Canonical log lines
17 September 2024
Canonical log lines are a concept introduced in the Stripe blog post Fast and flexible observability with canonical log lines :
We’ve found using a slight augmentation to traditional logging immensely useful at Stripe—an idea that we call canonical log lines. It’s quite a simple technique: in addition to their normal log traces, requests also emit one long log line at the end that includes many of their key characteristics. Having that data colocated in single information-dense lines makes queries and aggregations over it faster to write, and faster to run.
I’ve often included some structured logging in my log files, by prepending log lines with useful context:
[2020-01-01 23:00:00] [user_id=32] User successfully logged in.
This makes it easy to filter all log lines related to the user account 32. Using the =
keeps it readable and makes it machine parseable.
But Stripe found stitching lots of data together can be difficult:
Although logs offer additional flexibility in the examples above, we’re still left in a difficult situation if we want to query information across the lines in a trace. For example, if we notice there’s a lot of rate limiting occurring in the API, we might ask ourselves the question, “Which users are being rate limited the most?”
Stripe introduced canonical log lines in addition to the regular log statements. The canonical lines don’t prioritise human readability. They are dense, are emitted exactly once per event (web request, API call, background job, etc) with all the relevant data. They are canonical because “it’s the authoritative line for a particular request”.
In Metrics For Your Web Application's Dashboards Simon Hørup Eskildsen describes a JSON version of the canonical log line used by Readwise:
{
"severity": "info",
"processing_time": 0.273,
"enqueued_to_processed": 1532.069,
"finished_at": 1647783951.0888088,
"time_in_queue": 1531.796,
"worker_name": "worker-h@4877ad9e-2884-4973-8f8a-f4e09934e0ad",
"task_class": "main.tasks.send_daily_review_push_notification",
"task_args": [],
"task_id": "68ba3676-ef44-4515-a154-46d70748d9a0",
"retries": 0,
"task_kwargs": {
"profile_id": "[redacted]"
},
"hostname": "4877ad9e-2884-4973-8f8a-f4e09934e0ad",
"enqueued_at": 1647782419.0191329,
"started_processing_at": 1647783950.8150532,
"parent_id": "b6168daa-2edf-4e4e-8b15-a4b72a35fa02",
"root_id": "b6168daa-2edf-4e4e-8b15-a4b72a35fa02",
"state": "SUCCESS",
"category": "celery",
"queue": "t2h-c1",
"memory_mib": 213.3
}
His version uses JSON, which probably makes it a little more readable.
electric: sync postgres to the browser
19 August 2024
Electric provides an HTTP API for syncing Shapes of data from Postgres.
Replicates changes from postgres. Use the electric typescript client to sync data to your browser application. It uses a concept called Shapes to specify what data should be synced. The sync service is written in Elixir.
Looks like a good option for "I need Postgres on the server, but something like Firebase in the web client"