lopcode.com

Switching from Spotify to Apple Music

2025-02-01T00:00:00+01:00

Sometimes I want to write about something in a slightly longer form than for social media like Bluesky, but don’t want it to be so long and detailed that I have to spend days curating the post. I’m calling this style of post “casual” - something that’s been on my mind, of varying length (maybe even quite short), with less editing.

I recently switched from Spotify to Apple Music, which was a big deal for me. I listen to quite a lot of music, and it’s an important part of my life. Here’s why, how I did it, and how I feel about both platforms after a month.

Why

I used to really like Spotify, and used it since ~2010 - well over a decade. The radio was really unbeatable for a long time, and they popularised the “algorithmic playlist” which helped me discover a lot of new music. The direct integration with Sonos is also fantastic.

However, over the last ~5 years I feel there’s been a consistent degradation of the user experience for actually listening to music. The “Home” screen gained features pushing podcasts, and audiobooks, up over the top of your own library.

The prospect of a higher quality tier leaked out multiple times, with the most recent from July 2024 about a “deluxe”¹ tier. I’ve patiently waited multiple years for this feature, and I’m fed up with getting baited by it. I know, most folks can’t tell the difference between “high quality” (320 kbps MP3) and lossless. For the right track, I can probably be fooled. But I have a really nice pair of planar magnetic headphones (Hifiman Ananda²), and I would like to enjoy lossless music with them. I’m particularly annoyed at being told that I don’t actually want it.

Wrapped is a great concept, and I have enjoyed it for a few years, but my 2024 wrapped felt “off” somehow? I saw other people saying the same thing on social media at the time. Probably something to do with firing people who worked on it the year before³.

I don’t understand the “AI DJ” concept. I’m really not sure why the idea of pseudo-human voice interjecting between tracks is appealing. Perhaps to emulate the radio? But at least the people on the radio are… people, and not approximations. I guess I could just not use it, but it indicates a direction that I don’t vibe with.

I’m deeply skeptical of any desire to use AI to replace media created by people. I simply don’t want it and feel that kind of future is bleak, and worth fighting against. I don’t doubt that it could be a useful tool for exploration as part of the creative process, but that’s not what CEOs are pitching.

All that is to say, I was quite annoyed, and ready to explore alternatives.

Alternatives

Won’t go in to too much detail here, but there’s absolutely loads of platforms to choose from. Helpfully collated⁴ by The Verge recently (focused on platforms that make sense in the UK):

Amazon Music
Apple Music
Bandcamp
Soundcloud
Tidal

The aspects that matter to me most are:

Music as the focus, with less UI bullshit getting in the way
Higher quality offerings, including lossless
Ideally a native UI on each platform
iOS and macOS as primary platforms, with the option of Windows and web
Price doesn’t matter as much as the other points - if it’s good, I’ll pay for it
Artists deserve to be paid more on all platforms, and I’m not sure any of them are better at it than others

I’m deep in the Apple ecosystem already, so Apple Music was an obvious first choice. If that hadn’t been available I would have given Tidal a try.

Migrating

I exclusively used a really helpful app called “Playlisty for Apple Music”⁵. There’s a free trial for a handful of tracks to get you started. After I saw that it worked, I paid for it, and used it to migrate a huge number of songs and playlists. It self-reported a 99% success rate, which I was a little skeptical of - but happy to confirm that I only found one dubious song match in thousands.

Good stuff

I did actually try Apple Music a few years ago, but bounced off of it because the radio was really, really bad for me at the time.

Gladly that has vastly improved in the time since, and I think now I actually prefer it to Spotify’s radio, which feels wild to say given what a head start Spotify had. Spotify recently started getting “stuck” in my playlist generation, and felt like I’d hear the same songs repeatedly. One “Daily Mix” playlist barely changed every week and for some reason always started with the same Bonobo track? Weird.

The Apple Music playlists update less, sometimes once every couple of weeks, or a month, but they feel higher quality - more varied selections, fewer repetitions, and sometimes offering a track that expands your boundaries a bit, which I like.

It feels so nice to have desktop and mobile apps that look and feel native. They’re fast to load, the interaction is snappy, and it doesn’t feel like you have the weight of a browser waking up every time you foreground them.

Lossless is a really nice choice to have. Apple Music gives you two options for it - ALAC 24-bit/48 kHz, or ALAC hi-res 24-bit/192 kHz. The former is enough for me.

There’s a classical music app which I haven’t tried yet but love the idea of.

Room for improvement

I only have one negative comment about Apple Music, and that’s the lack of LastFM integration. You can tell when I made the switch from my profile⁶ - there’s no first-party way of scrobbling, which I miss a lot. It was an easy way to show folks what I’ve been listening to.

I haven’t checked if there’s an Apple Music API or not, but I’d definitely play around with it to connect the two platforms. If you know something about that, do let me know on Bluesky.

Conclusion

And that’s it! Hope this is helpful for anyone considering a switch. I encourage you to not feel trapped by your tech choices, regardless of the domain. If Spotify improved, I’m open to returning - I don’t think it’s irredeemable. I also understand that music is deeply personal, so use what makes you happy.

Sometimes giving alternatives a try can even help you rediscover why you liked the original thing. But for now, I’m very happy with my new music platform of choice.

Spotify CEO confirms a ‘deluxe’ version with hi-fi audio is coming soon - https://www.theverge.com/2024/7/23/24204520/spotify-ceo-hifi-audio-deluxe-plan-confirmed ↩
Hifiman Ananda - https://www.richersounds.com/hifiman-ananda/ ↩
(2023) Spotify to Cut 17 Percent of Global Workforce in Third Round of Layoffs This Year - https://www.rollingstone.com/music/music-news/spotify-global-workforce-cost-cutting-layoffs-1234908996/ ↩
The best alternatives to Spotify for listening to music: https://www.theverge.com/22910685/music-listening-service-spotify-apple-youtube-amazon ↩
Playlist for Apple Music - https://www.obdura.com/playlisty/ ↩
lopcode on LastFM: https://www.last.fm/user/lopcode ↩

Releasing vips-ffm v1.0 - libvips Bindings for Java / JVM Systems

2024-10-02T00:00:00+02:00

I’m very happy to announce that, after lots of testing and improvement, vips-ffm is now version 1.0 🎉!

In case you missed the last post, vips-ffm is a set of bindings, designed for Java / the JVM, that lets you use the popular libvips¹ library for image manipulation - think image thumbnails, cropping, resizing, and such. JNI-based alternatives like JVips exist, but have not been updated for a couple of years, and Java now offers safer, faster native library access via the Foreign Function and Memory (FFM) API².

Improvements since the last post include (around a hundred commits):

Production users giving really helpful feedback to test and shape the API - thank you!
Many more samples testing all sorts of functionality on macOS, Windows, and Linux (including arm64/amd64 architectures)
Comprehensive Javadocs for all libvips operations and enums, also hosted on a website
Rewritten using the libvips operations API to make the bindings even safer
100% operation and enum coverage for libvips, thanks to automation, with no custom JNI code
Preliminary benchmarks showing speed improvements of 10-35% compared to JVips, and 50-70% compared to AWT

I really wanted to make this library the best way to use libvips with JVM systems, including Kotlin, and I’m super happy with how it’s shaped up. I hope it’s useful for others! Go check the repo out, and give it a star to lend me some dopamine 🌟!

Here’s what a Kotlin sample (using the Java bindings) looks like in the 1.0 release:

import app.photofox.vipsffm.Vips
import app.photofox.vipsffm.VImage
import app.photofox.vipsffm.VipsOption
import app.photofox.vipsffm.enums.VipsAccess

// ...

// Call once to initialise libvips when your program starts, from any thread
Vips.init()

// Use `Vips.run` to wrap your usage of the API, and get an arena with an appropriate lifetime to use
// Usage of the API, arena, and resulting V-Objects must be done from the thread that called `Vips.run`
Vips.run { arena ->
    val sourceImage = VImage.newFromFile(
      arena,
      "sample/src/main/resources/sample_images/rabbit.jpg",
      VipsOption.Enum("access", VipsAccess.ACCESS_SEQUENTIAL)
    )
    val sourceWidth = sourceImage.width
    val sourceHeight = sourceImage.height
    logger.info("source image size: $sourceWidth x $sourceHeight")

    val outputPath = workingDirectory.resolve("rabbit_copy.jpg")
    sourceImage.writeToFile(outputPath.absolutePathString())

    val thumbnail = sourceImage.thumbnail(
      "sample/src/main/resources/sample_images/rabbit.jpg",
      400
    )
    val thumbnailWidth = thumbnail.width
    val thumbnailHeight = thumbnail.height
    logger.info("thumbnail image size: $thumbnailWidth x $thumbnailHeight")
}

// Optionally call at the end of your program, for memory leak detection, from any thread
Vips.shutdown()

Releasing version 1.0 is an indication that I feel it’s good enough for production usage. I’ll continue to make improvements and fix bugs as more people start using it, and give feedback.

libvips - https://www.libvips.org/ ↩
JEP 454 - Foreign Function & Memory API - https://openjdk.org/jeps/454 ↩

Announcing vips-ffm - libvips Bindings for Java / JVM Systems

2024-08-22T00:00:00+02:00

Edit: Version 1.0 has been released, check out the post!

This is short post announcing a new Java / JVM image-manipulation-related library I’ve shipped to Maven Central - vips-ffm 🎉.

When I started planning Photo Fox (a self-hosted photo management tool I’m working on), I knew I’d need some sort of image library. Photo Fox has a few requirements for image manipulation: creating high quality thumbnails, supporting modern file types to do transcodes, doing some basic edits like cropping, and reading image metadata like EXIF data. Although Java includes some image manipulation tools in the JDK, they don’t have the best reputation, and there’s no guarantee it’ll stay up to date with newer formats like HEIC and JXL, which I want to support.

My initial investigation took me to a long-standing library called ImageMagick¹, and then to a newer library called libvips². It turns out Mastodon recently switched from the former to the latter, and after having a look at the benchmarks, libvips seemed like a good choice. JVM bindings do seem to exist, but use JNI, which has a bad reputation and I wasn’t too keen to use, knowing that JDK 22 just shipped tools to replace it. Specifically, the “Foreign Function & Memory API” (JEP 454³) for calling C APIs, and the “Class-File API” (JEP 457⁴) for generating class files.

After a couple of weeks of work, I built something that could automatically generate JVM bindings from the libvips headers. This means there’s very little work involved in staying up to date with new upstream versions, there’s less human (rabbit?) error in the bindings, and mistakes can be corrected across the API in one go. On top of the raw bindings generated with jextract⁵ there’s another generated “helper API” to make usage safer and avoid mistakes with pointers and lifetimes, as the libvips documentation correctly warns about. The beauty of generating the bindings is that there is, in theory, close to 100% API coverage in the very first versions. Here’s an example of what some basic usage looks like right now, in Kotlin:

import app.photofox.vipsffm.generated.Vips
import java.lang.foreign.Arena

// ...

Arena.ofConfined().use { arena ->
    val vips = Vips(arena)

    val version = vips.versionString()
    logger.info("vips version: $version")
  
    val sourceImage = vips.imageNewFromFile(
        "sample/src/main/resources/sample_images/rabbit.jpg",
        VipsIntOption("access", VipsRaw.VIPS_ACCESS_SEQUENTIAL())
    )
    val sourceWidth = VipsImage.Xsize(sourceImage)
    val sourceHeight = VipsImage.Ysize(sourceImage)
    logger.info("source image size: $sourceWidth x $sourceHeight")

    val outputPath = workingDirectory.resolve("rabbit_copy.jpg")
    vips.imageWriteToFile(sourceImage, outputPath.absolutePathString())

    val outputImagePointer = VipsOutputPointer(arena)
    vips.thumbnail("sample/src/main/resources/sample_images/rabbit.jpg", outputImagePointer, 400)
    val thumbnailImage = outputImagePointer.dereferencedOrThrow()
    val thumbnailPath = workingDirectory.resolve("rabbit_thumbnail_400.jpg")
    vips.imageWriteToFile(thumbnailImage, thumbnailPath.absolutePathString())

    val thumbnailWidth = VipsImage.Xsize(thumbnailImage)
    val thumbnailHeight = VipsImage.Ysize(thumbnailImage)
    logger.info("thumbnail image size: $thumbnailWidth x $thumbnailHeight")
}

The code samples also function as correctness and memory leak tests during development, which has worked really well. I’m being conservative about version numbers and will promote the library to 1.0.0 when other users have a chance to test it out, but it should be ready for some real-world usage. Check it out, give feedback if you have any, and lend me some dopamine by giving the repo a star over on GitHub at github.com/lopcode/vips-ffm 🌟.

ImageMagick - https://imagemagick.org/ ↩
libvips - https://www.libvips.org/ ↩
JEP 454 - Foreign Function & Memory API - https://openjdk.org/jeps/454 ↩
JEP 457 - Class-File API - https://openjdk.org/jeps/457 ↩
jextract - https://github.com/openjdk/jextract ↩

Staying Organised With a Spicy Brain

2024-08-12T00:00:00+02:00

Keeping on top of things at work, and in my personal life, is a challenge. I’m neurodivergent, and struggle to form habits, even for things that others might find natural or easy. In this post, I share the framework that I use to stay organised, and offer some tips and tricks for getting things done.

Introduction

As a kid, I coasted through a lot of my education. I had a natural curiosity and could get away doing what I wanted to, when I felt like it, and still get labeled “gifted” (which I think is pretty toxic for a bunch of reasons I won’t elaborate on in this post). When I reached A-Levels (late high school for US folks), I hit a brick wall. There was too much stuff to do, and some material was much harder than my “natural ability” could cope with. I didn’t have any tools for managing my workload, and really struggled for several years. At university, I was fortunate to get help from a “mental health mentor”, who helped introduce me to some tools and techniques for staying on top of things.

That was over a decade ago, and I’ve read plenty of books on “self-organisation” since. This post is to share what worked and what stuck for me, and hopefully to inspire you to experiment in your own life. Remember to be kind whilst you do! If you’re struggling, it’s probably because the defaults that work for other people, and society in general, don’t work for you.

As you find things that work, and practice them and build your own processes, it gets easier 🧡.

A high-level process

There are, fundamentally, three things that I use to help myself get things done:

My notes
My calendar
My task list

Notes are for my day-to-day thinking, the calendar is for things that involve other people (like meetings, or sharing something with my husband), and my task list helps remind me and prioritise what I’m going to work on. Calendars are probably familiar to many — my only advice for the calendar is, do not use it like a task list. Your calendar is for recurring meetings, birthdays, anniversaries, and blocking out time to help you focus. Because the use of calendars is widespread, I’ll focus this post on notes, and tasks.

Taking notes

Writing notes helps me get thoughts out of my head, and forces me to articulate and organise what I’m thinking. My thoughts can feel like lightning bolts firing through connected topics, and quickly understanding connections between things is one of my strengths. But that means that writing things down is crucial, or they can get lost if my mind changes topic.

I use a reMarkable¹ tablet now, but in university I got through dozens of paper notebooks. I really like my digital notebook because I can move things around if thoughts end up written down in the wrong order (which is fairly regularly). It’s good to separate your work stuff from your personal stuff, so I use a separate (digital) notebook for work stuff, personal stuff, and larger projects.

A page from my digital notes

Every day gets a new page, and I write the date at the top, which is a nice way to start my day. If I start doing something, it gets no indentation, and as I write thoughts down, they get indented each time as they get more specific or detailed. Moving stuff to the right visually makes it easy to see if you’re getting too far in to detail on something. If there’s suddenly loads of space on the left, what you thought was a sub-topic might actually deserve more attention and become a left-aligned “major topic”. My major topics usually start as questions, spawning more questions, and end with either an answer, or a task to think more about it another time.

At the end of the day, it’s good to go back through and review your notes. I use a ⭐️ on the left-hand side of my notes if I want to add a task for myself. Do not rely on your notes alone to remind you to do things. Reviewing my notes helps me learn things, and make sure no stars/tasks got lost along the way.

Using a task list

I use Todoist² to manage my task list. I like it because it’s cross-platform, and has some natural language “scheduling” features that I’ll elaborate on shortly.

After starting my notebook page for the day, I’ll review my task list. It’s normal that I’ll end up with more tasks than one person could reasonably complete. Try not to get overwhelmed! Coping with a large task list is all about prioritisation ✨. I’ve developed a method I like to call the “four D”s:

Do
Delegate
Defer
Don’t

Do

Occasionally, a task is small enough that it’s better to get it done straight away. For example, this might be sending a message to someone to ask for their input, which you can review later. Or, it might be vital to complete quickly (like spotting a security-related problem at work) and therefore takes immediate priority over what you were doing.

If I do something straight away, I still like to write it down as a task, for the satisfaction of immediately marking it as done. If you notice that you’re doing many things straight away, you might be neglecting the other D’s.

Delegate

If you work in a team, and you’re increasing in seniority, it’s likely that there are countless things to do. The second thing to consider when you’re making a task is if someone else could/should do it. This could be an opportunity for someone else to learn, and if you have a habit of taking on all the “important” tasks, you might be accidentally starving the team of opportunities to grow.

The task might also not be important enough for you to tackle — at least not compared to your other tasks. In short, consider if this is a task for someone else, and can skip past your task list. I often make myself tasks to delegate something to someone else, so I can do a good handover.

Defer

If you decide that you’re the one who’s going to do the task, but you’re not going to do it right away (or in the short term, like today), then you can defer it.

I use Todoist’s scheduling feature to pick a date when the task should pop back up (usually using natural language — for example “Review X’s proposal in two days”). If, when the task comes back around, I still don’t think it takes priority, then I’ll defer it again.

Don’t

And finally, the art of simply not doing the thing. Folks who have had the pleasure of working any sort of team-based task list (like Jira) will know that task lists typically grow beyond your control. And tasks that you added a year ago probably don’t make sense any more because you either don’t understand the context any more, or the world has changed enough that the task doesn’t make sense.

If I’ve rescheduled a task beyond a couple of months, I’ll delete it. If it’s important enough, it will come back up. It’s funny how attached we can get to tasks if we write them down — deleting it can feel like giving up somehow. But I promise it’s OK, and you can even come to enjoy choosing to not doing things.

If you really want to keep something after this time, try approaching it differently — what you’re doing right now isn’t working. A helpful way of unsticking myself if a task keeps coming back up is to involve someone else — for example, if I’m writing a proposal, I’ll ask to talk it through with a colleague.

Conclusion

I hope this post inspires you to think about your process for getting things done, and to make one if you don’t have one yet. There are numerous existing frameworks to learn from, some evangelical in their confidence about being “the one true way”, but everybody’s brains are different — it can take a few tries. It certainly did for me.

Try not to get discouraged if something doesn’t stick — give it a solid go, and try something else if it isn’t working. There will be something that works for you 🧡.

reMarkable tablet - https://remarkable.com/ ↩
Todoist - https://todoist.com ↩

It’s Time to Ditch Twitter/X

2024-06-27T00:00:00+02:00

Prior to the platform’s buyout, I had a Twitter/X account since 2008 (nearly from the start!). Almost two years since Elon Musk bought it, it’s been long enough to make a judgement on the platform’s new direction. This post details why I feel it’s time to move on to bluer skies, and shares some practical tips on how to do so.

Why now?

I’ll preface this by saying It isn’t my intention to write a politically-oriented post. But I feel that regardless of your political sensibilities, the new owner’s recent behaviour towards his own daughter is blatantly unacceptable¹. There is a long pattern of such detestable behaviour, and it appears to be getting worse over time with the US election coming up.

I recognise that, for some, it might feel incredibly difficult to leave Twitter/X. But I’m hoping to share some tips and advice that might make the choice easier for you. And to encourage you that it is possible - I had a long-standing account, and originally found the prospect daunting, but months later, I genuinely don’t miss it. Although I am quite annoyed that the platform went so downhill so quickly.

Tips for leaving

The hardest part of quitting Twitter/X, for me, was the feeling that I was going to “miss out” on things happening in my communities. This is of course my own anecdotal experience, but after leaving I made an effort to be more deliberate about my social groups. In particular, I joined some smaller social groups on Telegram (as well as some more focused, tech-oriented channels), and it’s been much more enjoyable for me. It’s very likely that your existing friendship groups on Twitter/X know about these spaces - you could ask around. Or you could even make your own! Just remember that spaces need to be moderated to remain positive, safe, and inclusive.

It’s also a useful time to consider why you use social media in the first place. It’s likely that what you’re looking for exists, in a better form, elsewhere. Since Twitter/X imploded many other options popped up (and I list some options in the next section), giving you plenty of things to try.

I would caution against getting too attached to big numbers on your profile - consider if they really, actually bring you anything valuable. And remember that the culture on the platform now trends towards being “fickle” - large numbers of people watching you there are not necessarily a good thing. It’s possible that the benefit to your self-confidence is enough that you don’t want to go “cold turkey”, but you could still attempt to bring your followers over to other platforms, especially if they value your opinions.

A final difficult aspect of leaving was I felt like I was “losing” over a decade of posts that I’d made on the platform. To help with this, I followed a guide to download an archive of my data², and store it somewhere safe. I can honestly say that I’ve never opened this archive up since downloading it, but knowing it’s there helps somehow.

Why I like Bluesky

After the buyout, I joined several different platforms, including Cohost, Mastodon, Threads, and Bluesky. After a while of using each of them, I decided that I like Bluesky the most, with Mastodon a close second.

Bluesky has a strong technical foundation in ATProto³, and a good product roadmap⁴ - I get the impression that they know what their users want, and they’re taking the time to build it properly. The team is run very lean, which is a great way stay agile, and forces them to prioritise features. The culture in Bluesky so far has been open, queer-friendly, and manages (so far) to avoid the worst of the baseless “callouts” that plague Twitter/X.

I do have a couple of concerns about monetisation (including the scourge of targeted advertising on the internet), and about the technical feasibility of “private content” on the network, but the team has so far proven that they can tackle difficult problems and come up with considered solutions.

It likely won’t last forever, but I’m certainly enjoying my time there now. If you’d like to follow me, you’re very welcome to do so here: @lopcode.com 🧡.

Vivian’s interview with NBS - https://www.nbcnews.com/tech/tech-news/elon-musk-transgender-daughter-vivian-wilson-interview-rcna163665 ↩
The Verge - How to download an archive of your Twitter data - https://www.theverge.com/23453703/twitter-archive-download-how-to-tweets ↩
ATProto - https://atproto.com/ ↩
Bluesky 2024 Product Roadmap - https://bsky.social/about/blog/05-07-2024-product-roadmap ↩

Self-hosting Mastodon on AWS using Nomad

2023-01-23T00:00:00+01:00

With Twitter being a mess at the moment, I decided to try out Mastodon¹ as an alternative. Mastodon is a federated social media platform, built on top of a protocol called ActivityPub². It can be self-hosted, letting you own your data, and I wanted to do so using Nomad³ as a cluster orchestrator. This post shows how I did it, and will hopefully inspire you to give Mastodon a try if you haven’t already! You can find me on the fediverse at @lopcode@mastodon.social 🐘.

A screenshot of Nomad's web UI, showing it running the jobs that make up Mastodon

Getting started

The first thing to note is that I’m writing this up as inspiration, not as a step-by-step guide. It’s prove that it works, and offer an alternative to more complex setups that use Kubernetes (which Mastodon have a first-party Helm Chart⁴ for). Your mileage may vary - use your best judgement! If you know of improvements, feel free to send me a message at the link above.

This post is meant to be read with the repository that I’ve open-sourced, containing the actual Nomad task definitions and scripts discussed below: https://github.com/lopcode/nomad-mastodon

With that out of the way, let’s start with some goals and assumptions. I wanted to:

Self-host a Mastodon server, such that I could own my social media presence
Host the server infrastructure on AWS
Use Cloudflare in some capacity for caching
Use Nomad to do cluster orchestration and keep jobs running
Use Ansible and Terraform to manage the basic infrastructure (not in scope for this post)
Not spend more than around $50 a month doing so

It’s worth noting that my budget balances my financial situation, the tooling I’m already familiar with, and performance of the system. You can definitely do things cheaper if budget is a primary concern, for example by using cheaper cloud providers like DigitalOcean. If you’d rather have someone else host Mastodon for you, cheaper services like masto.host exist too.

Infrastructure

Mastodon itself is comprised of a number of systems:

The main web server (Ruby on Rails)
An event/message processor (Sidekiq, Ruby on Rails)
A front proxy to route web requests (nginx)
A streaming server, for real-time updates (Node)
A persistent database (Postgres)
A cache for “temporary” data (Redis)

On top of this, there are some other necessities to run it, and keep the system healthy after the initial setup:

Persistent storage, for user-generated content (S3-compatible)
A way of doing frequent cleanup (removing old media files, accounts, etc)
A way to run SQL migrations, for both the message processor and web server
A way to run tootctl - the administrative CLI for Mastodon⁵

As mentioned, I’m using Terraform and Ansible to manage my infrastructure. These tools have a lot of benefits, including making the act of setting up networking and servers repeatable, and documenting what exists in your cloud account. I strongly encourage you to use these tools (or alternatives) to manage your infrastructure, if you’re not already doing so.

Main components

There were some obvious choices on AWS for the main components - EC2 to run a server, RDS for a persistent database, and Elasticache for a Redis-compatible temporary data store. In researching S3 I also discovered that Cloudflare offer an S3-compatible service imaginatively named “R2”. It integrates nicely with their edge-caching service, and offers competitive pricing and performance, so I decided to give that a try as well.

There’s not too much to say about the Nomad jobs themselves - each primary job maps in to a Nomad job definition file (suffixed .nomad - for example, mastodon-web for the main web server). Mastodon have a first-party docker-compose.yml file⁶, to start up docker containers automatically, and you can use that to understand how to start each of the components. Secrets are passed in as Nomad variables, mapping to environment variables in the specific task. A future improvement noted below is to figure out a way to share these environment variables across job definitions, but for now, they’re copy/pasted between jobs via a template file.

Other components

Nomad offers “periodic” tasks with a cron-like interface, to let you run tasks on a frequent schedule, which is perfect for the “frequent cleanup” requirement. The cleanup job starts tootctl with some specific commands to remove media after a certain number of days, which I configured to 30 to start with. Media storage is something you should be aware of with Mastodon - it stores a “local” copy of media relevant for the users of your server. It can end up being quite a lot, so a good strategy to reduce your bill is to store discovered media for less time. The only downside being said media will have to be refetched when required again. Below is a graph of the storage usage for bunny.cloud - you can see media being cleaned up every day at 4am as it expires.

A graph of Cloudflare R2's storage

For running SQL migrations, I could see two choices:

Pick a job to run migrations in, as a “startup task” - for example, every time the message processor job starts
Add a “batch” job to run migrations as part of a deployment

I chose to use a batch job, because I wanted to control when I ran migrations, and decouple that process from running other critical components. This is because there are multiple jobs that require the migrations - at the very least, the main web server, and the message processor. Having a separate job means migrations can run before either of those jobs start. In practice I think the message processor would also be a decent choice.

`tootctl`

Another aspect of administrating a Mastodon server, is interacting with tootctl to do manual tasks. The tricky bit with a cluster orchestration system is “where do you run the command”? I think it’s overkill to spin up a dedicated entire job to run a single administrative command every now and then, so we have to pick a running job to run the command inside.

Nomad offers really handy CLI to inspect running jobs, and execute commands inside their tasks (called allocs). I’ve included a script, tootctl.sh, to let you run tootctl commands inside the message processing task on-demand. Usage is exactly the same as the original command, just through the script:

🥕 carrot 🗂 bunny-cloud-infra/infra/cluster 🐙 main $ ./tootctl.sh version
Discovering alloc of "mastodon-sidekiq" to run tootctl in...
Found alloc with ID: ef0fd901-d598-9fe6-b2c2-3c577c0aee2e
Executing: "tootctl version"
4.0.2

Future improvements

I’m a fan of shipping things and iterating, so there are a few obvious improvements that I plan on making:

Figure out a way to more easily share environment variables across jobs - it’s annoying having to copy a template to several files when they change, even if it’s rare
Migrate away from using “host networking” in the container definitions. Although the message processor task could reasonably autoscale in response to demand, using host networking means web server ports are statically assigned, making it impossible to run more than one replica
Investigate Nomad’s recently shipped “autoscaling” features, to spin up more message processing tasks in periods of high CPU load (or even detecting a message backlog)

Conclusion

Hopefully this inspired you to give self-hosting Mastodon a go, if you’re so inclined. I’ve also proven that you don’t need a complicated Kubernetes setup to get a lot cluster orchestration goodness. In total this setup costs around $45 a month, which is happily within my budget. That’s the cost for a single person on my server, so if there were more people who contributed to the running costs, it would rapidly get cheaper.

I also hope that more people give Mastodon a go - I’m really enjoying it so far, and feel that it’s a much healthier form of social media for me. I’ve been struggling with Twitter feeling toxic for a while, and having a space that I can share things to without being sold adverts, or served divisive content to push platform metrics, is something I value greatly.

Is there another aspect of this setup you’d like to know about in more detail? Feel free to ask over on the fediverse, at @lopcode@mastodon.social!

Footnotes

Mastodon website - https://joinmastodon.org/ ↩
ActivityPub specification - https://www.w3.org/TR/activitypub/ ↩
Nomad cluster orchestrator - https://www.nomadproject.io/ ↩
Mastodon Helm Chart - https://github.com/mastodon/chart ↩
tootctl - https://docs.joinmastodon.org/admin/tootctl/ ↩
Mastodon docker-compose.yml - https://github.com/mastodon/mastodon/blob/main/docker-compose.yml ↩

Using a Stream Deck with OBS 28

2022-08-17T00:00:00+02:00

This is a short post to say that I published an open-source project, which added Stream Deck support for OBS 28, including Qt 6, and native Apple Silicon (M1 / M2 processor) support. It’s since been archived because Elgato published their own. You can find the archived repository on GitHub ✨.

Edit 31/08/22: Elgato updated their own Stream Deck plugin for OBS 28 support 🎉. Please see this support document for instructions: https://help.elgato.com/hc/en-us/articles/8815141056013-Elgato-Stream-Deck-Plugin-Update-for-OBS-Studio-28

If the plugin helped you beta test OBS 28 since they added support last December, please consider sponsoring my open-source work, or follow me or on GitHub to hear about what I’m up to 🥰.

Previous post:

OBS 28 with a working Stream Deck plugin

OBS 28¹ is currently in beta at the time of writing this post, and is a huge release that includes such macOS goodies as:

ScreenCaptureKit integration for efficient, OS-native screen capture
Hardware CBR (constant bitrate) on Apple Silicon for significantly reduced CPU usage streaming to Twitch
Native Apple Silicon support

I wanted to be able to use this new version of OBS to stream, but plugins must be recompiled in order to support it. In my opinion, the best way to achieve this is to use the plugin templates provided by the OBS team themselves². I didn’t get any replies on the upstream plugin’s repository, and I had some spare time, so I did the work. You can grab a signed build of my plugin from the GitHub Releases page.

Please note that this plugin is not endorsed by Elgato or OBS, there’s no warranty when using it, and problems should be logged on my repository and not in either of their support channels.

Footnotes

OBS Studio 28 (beta 2) - https://github.com/obsproject/obs-studio/releases/tag/28.0.0-beta2 ↩
OBS Plugin Template - https://github.com/obsproject/obs-plugintemplate ↩

Building Pellet – Routing Requests

2022-05-08T00:00:00+02:00

An important part of web applications is how they route incoming requests to appropriate request handlers. This post details how I implemented Pellet’s first routing system, with room to grow in the future, whilst staying consistent with Pellet’s wider design goals of being fast, concise, and correct. Like the last post, it includes lots of code samples to help you follow along 🧑‍💻.

Requirements
Implementation
- Routing
- Route handlers
Benchmarking
Conclusion
Footnotes

Requirements

As with the structured logging feature in the previous post, the first thing I wanted to do, before writing any code, was define some terms. The definitions I came up with also form requirements of the new routing system:

The user should be able to define a number of routes, where a single route consists of a method, a path, and a reference to a route handler
A route handler will, given a particular context, return a single route response
A route response consists of an HTTP message in response to a request, including a status code, some headers, and an HTTP entity
In the context of a web framework, it’s probably worth decomposing a route path in to its path components (separated by a /)
Routes will sometimes require variable elements, and there should be a type-safe way to define these, and extract them in handlers - for example, to let API callers pass an ID, as a UUID, in the route path
Variable sized variables are likely to be much harder to implement well, so only fixed-size elements should be considered - for example, consider mapping route paths to system file paths as out of scope for the first iteration

With these requirements and definitions in mind, I could start diving in to the code.

Implementation

When I’m approaching a medium sized problem, I often like to think about the user-facing API first as a way of making sure the implementation ends up in a good place. When faced with complexity, a good way to break it down is to add constraints to make the problem smaller.

Routing

With that in mind, the first thing to think about was the router - how the user would define routes, and how to quickly match incoming requests to an appropriate handler (called route resolution). I’ve been enjoying defining both a regular Java-style Builder, and augmenting it with a Kotlin Builder (using Kotlin’s DSL feature¹), to give the user choice about style. So, the router building aspect looks something like this in code - I define a @DslMarker, a classic Builder, and a Kotlin DSL function:

@DslMarker
@Target(AnnotationTarget.CLASS, AnnotationTarget.TYPE)
annotation class PelletBuilderDslTag

@PelletBuilderDslTag
object PelletBuilder {

    suspend fun httpRouter(
        lambda: suspend (@PelletBuilderDslTag RouterBuilder).() -> Unit
    ): PelletHTTPRouter {
        val builder = RouterBuilder()
        lambda(builder)
        return builder.build()
    }
}

@PelletBuilderDslTag
class RouterBuilder {

    private val router = PelletHTTPRouter()

    fun get(
        routePath: PelletHTTPRoutePath,
        handler: PelletHTTPRouteHandling
    ) {
        router.add(PelletHTTPRoute(HTTPMethod.Get, routePath, handler))
    }

    // ... other methods

    internal fun build(): PelletHTTPRouter {
        return router
    }
}

This code sample references one of the concepts defined above - a “route path”. For familiarity, Pellet will include both string and type-safe options for “simple” paths, but if you want to include variables, you’ll have to build the path using another concept, called “descriptors”.

A descriptor in Pellet is a way to map a named variable to a type-safe deserialiser (to turn a String in to a specific type like a UUID), and can be used to handle both query parameters and path parameters. Both can be defined as a “route variable descriptor” as follows, with an example of adding a safe UUID descriptor:

data class RouteVariableDescriptor<T : Any?>(
    val name: String,
    val deserialiser: (String) -> T
)

fun uuidDescriptor(
    name: String
): RouteVariableDescriptor<UUID> {
    return RouteVariableDescriptor(name) {
        UUID.fromString(it)
    }
}

We can use these descriptors to build route paths that are automatically split in to components, like so:

val idDescriptor = uuidDescriptor("id")

// path: /v1/{id:UUID}/hello
val helloIdPath = PelletHTTPRoutePath.Builder()
    .addComponents("/v1")
    .addVariable(idDescriptor)
    .addComponents("/hello")
    .build()

val router = httpRouter {
	get(helloIdPath) { // context ->
        // ...
    }
}

Which leaves us with the concern of reusing this descriptor, in a handler, to extract the path variable we expect. Doing so requires some sort of “context” to operate on inside the handler - a context should have information about the incoming request in it, like the route path, HTTP entity, query parameters and such. The basic code definition of a route context is:

data class PelletHTTPRouteContext(
    val rawMessage: HTTPRequestMessage,
    val entity: EntityContext?,
    internal val pathValueMap: Map<String, String>,
    internal val queryParameters: QueryParameters
) {

    data class EntityContext(
        val rawEntity: HTTPEntity.Content,
        val contentType: ContentType
    )

    // ...
}

Having such a context lets us define functions to extract query parameter and path parameter values - for example:

fun <T : Any> PelletHTTPRouteContext.pathParameter(
    descriptor: RouteVariableDescriptor<T>
): Result<T> {
    val rawValue = pathValueMap[descriptor.name]
        ?: return Result.failure(
            RuntimeException("no such path parameter found")
        )

    return runCatching {
        descriptor.deserialiser.invoke(rawValue)
    }
}

fun <T : Any> PelletHTTPRouteContext.firstQueryParameter(
    descriptor: RouteVariableDescriptor<T>
): Result<T> {
    val rawValue = queryParameters[descriptor.name]
        ?.firstOrNull()
        ?: return Result.failure(
            RuntimeException("no such query parameter found")
        )

    return runCatching {
        descriptor.deserialiser.invoke(rawValue)
    }
}

Note that these methods make extensive use of Kotlin’s Result type, so that users get either a success of type T or a failure with a cause of type Throwable. Java frameworks (and indeed Java libraries in general) have a tendency to throw errors and let some system up the chain deal with it, but I think it’s better to deal with errors when you have enough context to do something useful, like recover.

The router itself is pretty straightforward - it builds an internal list of routes, and when a request comes in, filters that list to routes that are an exact match for the request, and returns the first one.

The only slightly tricky aspect of routing relates to path variables. Given an incoming HTTP request, the exact match must take in to account whether a component of a path can be “variable” or not. Thankfully, we can get a lot of this for free at runtime by doing some extra work “up front” when we define routes themselves - by “componentising” the route paths when they’re defined. Path components can either be Plain or a Variable, so the route path class ends up looking like this:

data class PelletHTTPRoutePath(
    internal val components: List<Component>
) {

    companion object {

        fun parse(rawPath: String): PelletHTTPRoutePath {
            return Builder()
                .addComponents(rawPath)
                .build()
        }
    }

    sealed class Component {

        data class Plain(val string: String) : Component()
        data class Variable(val name: String) : Component()
    }

    public class Builder {

        private var components = mutableListOf<Component>()

        fun addComponents(string: String): Builder {
            components += string
                .split("/")
                .mapNotNull {
                    val trimmedString = it
                        .removePrefix("/")
                        .removeSuffix("/")
                    if (trimmedString.isEmpty()) {
                        return@mapNotNull null
                    }
                    Component.Plain(trimmedString)
                }
            return this
        }

        fun addVariable(
            variableName: String
        ): Builder {
            components += Component.Variable(variableName)
            return this
        }

        fun addVariable(
            descriptor: RouteVariableDescriptor<*>
        ): Builder {
            components += Component.Variable(descriptor.name)
            return this
        }

        fun build(): PelletHTTPRoutePath {
            return PelletHTTPRoutePath(
                components
            )
        }
    }
}

Doing the extra work up front to split the route path means that, if you also split incoming request paths in to components, matching routes becomes very cheap indeed. Parsing variables out is trivial because if the route matches, then the contents of the incoming path component, at a given index, form the contents of the variable in a defined route.

In the future this routing system could be further improved by constructing a tree structure at startup, to avoid iterating through a list of routes one-by-one to see if they match. However, my intuition is that in practice the routing will be plenty fast until you have loads of routes defined.

Route handlers

Route handlers are where the user, given a context (discussed above), can reply to an incoming HTTP request message. In HTTP, one should send a single logical reply to a message. Note that I say logical reply, because a response could actually consist of several literal response sections - think of a chunked transfer, where you might want to send an HTTP status line and headers, then stream the contents of a large file whilst avoiding having it all in memory at once.

Replying is important enough that I felt it needed its own Builder to make things nice and easy for the user. I wanted to include methods to set status codes and response bodies, as well as helper methods that could affect multiple aspects of the response at the same time. For example, sending a JSON response implies a particular content type too, and “no content” responses should remove the Content-Type and Content-Length headers completely. With that in mind, the response class and builder ends up looking like this:

data class HTTPRouteResponse(
    val statusCode: Int,
    val headers: HTTPHeaders,
    val entity: HTTPEntity
) {

    class Builder {

        private var statusCode: Int = 0
        private val headers = HTTPHeaders()
        private var entity: HTTPEntity = HTTPEntity.NoContent

        fun statusCode(code: Int): Builder {
            statusCode = code
            return this
        }

        fun header(
            name: String,
            value: String
        ): Builder {
            headers.add(
                HTTPHeader(name, value)
            )
            return this
        }

        fun noContent(): Builder {
            statusCode = 204
            setNoContent()
            return this
        }

        // ... other methods

        fun entity(entity: HTTPEntity): Builder {
            this.entity = entity
            return this
        }

        fun entity(
            byteBuffer: ByteBuffer,
            contentType: ContentType
        ): Builder {
            this.entity = HTTPEntity.Content(PelletBuffer(byteBuffer))
            val contentTypeHeaderValue = ContentTypeSerialiser.serialise(contentType)
            this.headers[HTTPHeaderConstants.contentType] = contentTypeHeaderValue
            return this
        }

        fun entity(
            entity: String,
            contentType: ContentType
        ): Builder {
            val charset = contentType.charset() ?: Charsets.UTF_8
            val byteBuffer = charset.encode(entity)
            return this.entity(byteBuffer, contentType)
        }

        inline fun <reified T> jsonEntity(
            encoder: Json,
            value: T
        ): Builder {
            val encodedResponse = encoder.encodeToString(value)
            val contentType = ContentTypes.Application.JSON
            this.entity(encodedResponse, contentType)
            return this
        }

        fun build(): HTTPRouteResponse {
            val statusCode = this.statusCode

            val response = HTTPRouteResponse(
                statusCode = statusCode,
                headers = headers,
                entity = entity
            )
            return response
        }

        private fun setNoContent() {
            entity = HTTPEntity.NoContent
            this.headers -= HTTPHeaderConstants.contentType
            this.headers -= HTTPHeaderConstants.contentLength
        }
    }
}

That’s a lot of code samples so far! There’s one more important one, however, which is tying all of this together. The following sample defines a new server, with a single route at /v1/{id:UUID}/hello, that sends a friendly reply in a JSON response:

val idDescriptor = uuidDescriptor("id")

fun main() = runBlocking {
    val helloIdPath = PelletHTTPRoutePath.Builder()
        .addComponents("/v1")
        .addVariable(idDescriptor)
        .addComponents("/hello")
        .build()
    val pellet = pelletServer {
        httpConnector {
            endpoint = PelletConnector.Endpoint(
                hostname = "localhost",
                port = 8080
            )
            router = httpRouter {
                get(helloIdPath, ::handleHello)
            }
        }
    }
    pellet.start().join()
}

@Serializable
data class ResponseBody(
    val message: String
)

suspend fun handleHello(
    context: PelletHTTPRouteContext
): HTTPRouteResponse {
    val id = context.pathParameter(idDescriptor).getOrThrow()
    val responseBody = ResponseBody(message = "hello $id 👋")
    return HTTPRouteResponse.Builder()
        .statusCode(200)
        .jsonEntity(Json, responseBody)
        .build()
}

Sending a request looks like:

carrot 🥕 $ http localhost:8080/v1/06b39add-2b57-4d58-b084-40afeacab2e9/hello
HTTP/1.1 200 OK
Content-Length: 61
Content-Type: application/json

{
    "message": "hello 06b39add-2b57-4d58-b084-40afeacab2e9 👋"
}

In my opinion that’s a pretty nice looking demo - there’s a good balance of conciseness and correctness, and it’s really fast, which I’ll discuss next 🚀.

Benchmarking

In previous posts I’ve discussed simple approaches to benchmarking, but I had a suspicion that I wasn’t quite getting everything I could out of Pellet. I decided to give TechEmpower’s “Framework Benchmarks”² a try, as I knew it could handle sending a huge number of requests. As a bonus, I’ve submitted a pull request, and if it’s merged, Pellet will be automatically measured, and the results show up in the list of frameworks alongside others like Ktor.

I won’t go in to detail about adding a new benchmark to the framework, because their documentation was great, but Pellet serves around ~200k requests per second on my machine using the JSON benchmark, and ~800k using plaintext. At that point you’re likely starting to stress test the operating system’s network stack instead of the framework, and macOS isn’t especially well suited to these kinds of stress tests, so I think the JSON benchmark is more indicative of performance under real-world load.

Conclusion

Pellet now has a solid routing system that I think meets the wider design goals of being fast, concise, and correct ✨. There’s also room for improvement as the framework develops and caters to more use cases, especially relating to the route matching system.

Builds are now published to Maven Central to make it easy to try the framework out, under the dev.pellet coordinate! There are instructions on setting things up using Gradle in the README.

Please star the project on GitHub if you like where it’s going (), and share the post to social media - doing so helps more people discover the framework, and I’ve already had some really helpful feedback on Reddit. Thank you for reading 🙇!

Footnotes

Kotlin Builders - https://kotlinlang.org/docs/type-safe-builders.html ↩
TechEmpower Framework Benchmarks - https://github.com/TechEmpower/FrameworkBenchmarks ↩

Building Pellet – Structured Logging

2022-03-09T00:00:00+01:00

This post is about building a logging system for Pellet - an opinionated Kotlin web framework I’m working on, with the intention to build best-practices in from the start. I’ll discuss features I think a good logging system should have, introduce the concept of “structured logging”, and talk about implementing everything (with plenty of code examples along the way) 🚀.

Requirements
Implementation
Conclusion
Footnotes

Requirements

An important, but often overlooked, part of any web framework is how it logs things. From sense checking conditions at startup, to logging requests, and exceptions, the point of logging is to provide context about the state of the application at runtime - to aid in development, operation, and debugging.

With this in mind, I wrote down the following requirements for a logging system:

Pellet has a general goal of reducing external dependencies where sensible to do so - try not to add many, if any, dependencies
The overhead of logging should not be too much - my first guess was almost certainly no more than a millisecond per request, and tens of milliseconds at startup
Users of the logging system should be able to do the basics, like logging a message at different levels (think debug, info, error, etc)
Users should be able to add their own context to the log output, without trying to stuff it all in a single message string
The system should play nicely with SLF4J¹, and the associated MDC² (mapped diagnostic context) libraries - these frameworks are very popular in the JVM ecosystem, and ignoring them is likely to cause major headaches
At the same time, try not to be too limited by older frameworks - try to use the best of Kotlin
Be opinionated - choose sensible defaults, don’t worry too much about dynamic modification and configuration

Looking at all of these, it was obvious that doing “structured logging” out of the box would be the best thing to do - especially the part about being able to add your own context.

Structured logging means that the log output includes additional, usually keyed, information to add extra context to the message. For example, you might include the name of the thread that logged something, or the HTTP status code that a handler returned. Although this is theoretically possible with simple plaintext messages, doing so with a format like JSON makes it much easier to ingest log lines that might not have exactly the same structure in different contexts (for example, the “status code” structured log element doesn’t make any sense in an “application startup” context).

Implementation

With requirements sorted, I started thinking about the implementation.

A key component of a performant logging system is the avoidance of unnecessary work. During development, you’re likely to want to run the service with “debug logging” turned on, to see more of the application’s state and help you debug things. But after deploying to a production environment, these messages add unnecessary noise to the log output. Worse, if you’re not careful, constructing the debug messages themselves can add non-negligible overhead to your system. SLF4J and associated logging frameworks prevent this unnecessary overhead by letting you pass arguments along with your message, and uses “message formatting” to interpolate these at runtime, if that particular log level is enabled. For example:

val logger = LoggerFactory::getLogger(MyHandler::class.java)
val state = /* an expensive object to serialise */
logger.debug("application state: {}", state)

These frameworks are designed for Java, which does not currently have string interpolation built in. As such, the objects cannot easily be type-safe - you can pass any Object in to the logging functions, and you don’t know that they’ll serialise properly at runtime. Kotlin, however, has great string interpolation built in³, which uses StringBuilder under the hood 🤩! Combined with lambda expressions, we can avoid having to pass Objects around, keep things type safe, and lazily evaluate our output only when that particular log level is enabled - avoiding the overhead problem.

The structured logging requirement suggested that I’d need a JSON serialisation library - to turn logging models in to strings for output to the console. I knew that JetBrains were working on one called kotlinx.serialization⁴, which I’d heard good things about, so I investigated the state of the project. A couple of key things persuaded me to go with this framework to serialise log lines in to JSON - it was written and maintained by JetBrains (the same people who make Kotlin), and it allowed you to avoid reflection entirely. Instead, you can annotate classes with @Serializable, which an annotation processor picks up, and generates serialisation logic at compile time. Although JVM reflection can be fast at runtime, it takes time to “warm up”, and can be slow the first time you use it at a particular call site, causing unnecessary latency and load on the service until results are optimised and cached.

Next, I needed to think about adding context-relevant entries to the log output. Something new that I wanted to do, that isn’t available in older logging frameworks, is for everything to be type safe - if you can make a log element out of it, you’re guaranteed that it’ll serialise properly at runtime as you expect it to. To do so, we have to “box” log elements due to a limitation of Kotlin - in Swift, for example, you might add a protocol conformance of PelletLogElement to primitive classes like String, but you can’t do that with Kotlin. Instead, we can use helper methods to box based on type, and we do that for String, Number, Boolean, and PelletLoggable if you want to encapsulate your own serialisation logic somewhere else. Kotlin’s sealed class support means we can make a parent type, with a fixed set of supported boxed subtypes:

public sealed class PelletLogElement {

    object NullValue : PelletLogElement()
    data class StringValue(val value: String) : PelletLogElement()
    data class NumberValue(val value: Number) : PelletLogElement()
    data class BooleanValue(val value: Boolean) : PelletLogElement()
}

// ...

public class PelletLogElements(
    startingElements: Map<String, PelletLogElement> = mapOf()
) {
    private val elements = startingElements.toMutableMap()

    fun add(key: String, string: String?): PelletLogElements {
        elements += key to logElement(string)
        return this
    }

    // ...
}

// ...

public fun logElement(value: String?): PelletLogElement {
    if (value == null) {
        return PelletLogElement.NullValue
    }
    return PelletLogElement.StringValue(value)
}

Combining this with a Kotlin builder⁵ function, we can lazily create a map of structured log elements:

// ...

public fun logElements(
    builder: PelletLogElements.() -> Unit
): () -> PelletLogElements {
    return {
        PelletLogElements(mapOf()).apply(builder)
    }
}

// ...

val elements = logElements {
    add("hello", "world 🌎")
}
logger.info(elements) { "a message with lazily evaluated log elements" }

The features above suggested to me that it was worth making a “clean room” log interface, in pure Kotlin, with an SLF4J bridge to support the basics. Pure Java logging interfaces end up being pretty large⁶, because they have “helper methods” for each of the log levels. Kotlin offers us “extension functions”⁷ that mean the primary interface can be pretty lean, but users can still have pragmatic helper functions too:

public interface PelletLogging {

    fun log(
        level: PelletLogLevel,
        elementsBuilder: (() -> PelletLogElements)? = null,
        messageBuilder: () -> String
    )
    fun log(
        level: PelletLogLevel,
        throwable: Throwable? = null,
        elementsBuilder: (() -> PelletLogElements)? = null,
        messageBuilder: () -> String
    )
}

// ...

public fun PelletLogging.info(
    elementsBuilder: (() -> PelletLogElements)? = null,
    messageBuilder: () -> String
) {
    log(PelletLogLevel.INFO, elementsBuilder, messageBuilder)
}

public fun PelletLogging.info(
    throwable: Throwable? = null,
    elementsBuilder: (() -> PelletLogElements)? = null,
    messageBuilder: () -> String
) {
    log(PelletLogLevel.INFO, throwable, elementsBuilder, messageBuilder)
}

To create instances of the logger, Kotlin offers us a particularly nice feature called “reified generics”⁸. We can use this to make a function that automatically inlines itself, giving access to type-safe generic class information that isn’t ordinarily available in Java. Here are three examples of making a similarly named logger - the last one being my favourite and most concise way of doing it:

public fun pelletLogger(
    name: String
): PelletLogging {
    return PelletStructuredLogger(name) { PelletLogging.level }
}

public fun <T> pelletLogger(
    clazz: Class<T>
): PelletLogging {
    return pelletLogger(clazz.name)
}

public inline fun <reified T> pelletLogger(): PelletLogging {
    return pelletLogger(T::class.java)
}

// ...

val explicitLogger = pelletLogger(Main::class.java)
val namedLogger = pelletLogger("main")
val logger = pelletLogger<Main>()

The final requirement was making sure services built on Pellet still worked with SLF4J. I was comfortable with it being more of a fallback - features like the structured logging elements didn’t have to work with it. SLF4J searches for logging implementations at runtime, so I added a very basic SLF4J MarkerBase, similar to their slf4j-simple logger, that just does message formatting and then forwards the result on to a backing PelletStructuredLogger:

public class PelletSLF4JBridge(
    name: String,
    private val level: PelletLogLevel
) : MarkerIgnoringBase() {

    private val backingLogger = pelletLogger(name)

    override fun isTraceEnabled(): Boolean {
        return level.value >= PelletLogLevel.TRACE.value
    }

    override fun trace(format: String?, arg: Any?) {
        if (!isTraceEnabled) {
            return
        }
        val formattingTuple = MessageFormatter.format(format, arg)
        logFormattingTuple(PelletLogLevel.TRACE, formattingTuple)
    }

    // ...
}

Finally, putting everything together, here’s a code example from Pellet’s HTTP request handler. It adds structured log elements, like request method, and response duration; plus, it logs an “apache common log format” in the message text:

private val logger = pelletLogger<HTTPRequestHandler>()

// ...

private suspend fun respond(
    request: HTTPRequestMessage,
    response: HTTPRouteResponse,
    responder: PelletHTTPResponder,
    timer: PelletTimer
) {
    val message = mapRouteResponseToMessage(response)
    val requestDuration = timer.markAndReset()
    val elements = logElements {
        add(requestMethodKey, request.requestLine.method.toString())
        add(requestUriKey, request.requestLine.resourceUri.toString())
        add(responseCodeKey, message.statusLine.statusCode)
        add(responseDurationKey, requestDuration.toMillis())
    }
    logResponse(request, response, elements)
    responder.respond(message)
}

private fun logResponse(
    request: HTTPRequestMessage,
    response: HTTPRouteResponse,
    elements: () -> PelletLogElements
) {
    val now = Instant.now().atZone(UTC)
    val dateTime = commonDateFormat.format(now)
    val (method, uri, version) = request.requestLine
    val responseSize = response.entity.sizeBytes
    logger.info(elements) { "${client.remoteHostString} - - [$dateTime] \"$method $uri $version\" ${response.statusCode} $responseSize" }
}

Which outputs as follows:

{"level":"info","timestamp":"2022-03-07T00:32:17.765062Z","message":"127.0.0.1 - - [07/Mar/2022:00:32:17 0000] \"GET /v1/hello HTTP/1.1\" 200 22","name":"dev.pellet.server.codec.http.HTTPRequestHandler","thread":"DefaultDispatcher-worker-1","request.method":"GET","request.uri":"/v1/hello","response.code":200,"response.duration_ms":0}

Conclusion

I’m really happy with how the first prototype of Pellet logging has turned out. Users get fast, concise structured logging out of the box, without needing to faff around with configuration or log layouts. You can choose what log level you want to output at, and if something doesn’t end up getting logged, it doesn’t add needless overhead.

The API is clean, and hopefully familiar enough to not alienate users who are used to SLF4J - but offers an even better, incrementally improved experience to Kotlin-first users, using some of the best features the language has to offer ⭐️.

Footnotes

SLF4J - https://www.slf4j.org/ ↩
MDC - https://www.slf4j.org/api/org/slf4j/MDC.html ↩
Kotlin string templates - https://kotlinlang.org/docs/basic-types.html#string-templates ↩
kotlinx.serialization - https://github.com/Kotlin/kotlinx.serialization ↩
Kotlin functional literal with receiver - https://kotlinlang.org/docs/lambdas.html#function-literals-with-receiver ↩
SLF4J Logger - https://www.slf4j.org/api/org/slf4j/Logger.html ↩
Kotlin extension functions - https://kotlinlang.org/docs/extensions.html ↩
Kotlin reified type parameters - https://kotlinlang.org/docs/inline-functions.html#reified-type-parameters ↩

Building a Live-Updating List of Patreon Sponsors

2022-02-18T00:00:00+01:00

This post is an end-to-end description of how I built and shipped a new feature for this website - the live-updating sponsor list on the support page ✨. I wanted a way to recognise sponsors on the website, and I really wanted it to be live, so that someone could sponsor and almost immediately see their bunny show up on the page.

I’ll run through the entire process of building the feature - from gathering requirements, to planning, and implementing a solution. Although I won’t include all the code, I will include snippets where something is particularly interesting.

Requirements
Planning
Building
Conclusion
Footnotes

Requirements

As with any non-trivial technical project, I started by making a new project page on Notion¹, and drafting a list of requirements. I almost always start projects by listing requirements, and planning - I find it focuses the implementation, and helps prevent scope creep. With that in mind, this is what I wrote down:

There should be a bunny loaf for every Patreon sponsor, shown on the support page
The buns should be displayed in a responsive grid and look nice on mobile too
The component should include a visual display of the current sponsorship goal
The list should update as quickly as is reasonably possible - ideally live
Sponsor privacy should be respected - if a person hides their sponsorships, they should show up as “Anonymous”
If fancier browser features aren’t available, the component should fall back to using basic REST calls

Planning

With a list of requirements done, the next thing I needed to do was check what the capabilities of the Patreon API² were. I knew that there were definitely lists of sponsors available, but had no idea about how performant the API was, or whether it could support the fancier features like live updating, or sponsor privacy. I quickly spotted a message on their developer portal saying that, whilst endpoints would continue to function, they’re focusing on “the core product” - suggesting they don’t believe the API is part of that core. But it does clearly say that endpoints would continue to function, so I wasn’t too worried about the API going away.³

Browsing around the documentation suggested that there indeed were endpoints that could return sponsor data. Newer endpoints (version 2) use a standard called JSON API⁴ which I’m not a fan of - I find it over-complicates endpoints, adding quite a bit of structural overhead, and requires extra libraries to parse out information in the format that the API designer intended. It’s not a deal-breaker though, and just means the integration will be a little harder.

The main endpoint that I was interested in is the “campaign members” endpoint: GET /api/oauth2/v2/campaigns/{campaign_id}/members. I generated myself an API token on the developer portal, and used HTTPie⁵ to make some requests locally to validate the data, and “kick the tyres”, so to speak. Here’s an example of the kind of requests I was making in my terminal (noting that 725055 is the “campaign ID” for my Patreon page):

🥕 carrot 🗂 ~/git/carrot.blog 🐙 main $ 
http https://www.patreon.com/api/oauth2/v2/campaigns/725055/members\?include=currently_entitled_tiers,user\&fields\[member\]=full_name,patron_status\&fields\[tier\]=title,amount_cents,created_at,description\&fields\[user\]=vanity,full_name,hide_pledges Authorization:"Bearer <snip>"

I should also note that there is pagination on all the new endpoints, but that I’m nowhere near the page limit (which I think is 20 items), so I haven’t implemented support for it yet. In practice this means that if I got an influx of supporters, they might not all show up until I implemented this API feature, but I’m confident it would take about an hour to ship. A nice problem to have, for sure.

One of my requirements was having a visual indication of the current sponsor goal (like 3 / 20). The V2 API does not include a way of fetching “sponsor count” goals, only monetisation goals, so I fell back to the V1 API for this particular piece of information. That bit is included in the V1 campaigns endpoint: GET /api/oauth2/api/current_user/campaigns?include=patron_goals

Finally, I noticed that the API also supports webhooks⁶, which was great to see. That meant it was likely possible to avoid repeatedly polling the Patreon API, which can be slow at times, and I wasn’t sure about hitting rate limits (as there aren’t any hard guidelines noted on the developer docs). Instead, I could register a webhook and refresh the list of patrons, when notified that there was a change. The developer portal also supports sending test webhook messages which makes iterating on the feature much easier.

Now I knew my way around the API, I could start building, which would involve infrastructure, backend, and frontend work.

Building

With the planning and API investigation done, I could start building the actual service 🚀.

Infrastructure

This website is almost completely static, compiled using Jekyll, hosted on GitHub Pages⁷, and fronted using Cloudflare⁸, so the first thing to set up was infrastructure - I needed a place to run the API service. I have quite a bit of prior experience using DigitalOcean⁹, and the $6 “droplet” instances offer a really cheap, robust way to test out ideas, so I went with that platform.

Automating infrastructure, and deploying services in a repeatable way, are hard requirements for my projects nowadays. I have quite a number of things on the go, so it’s impossible to keep every project’s implementation details in my head. Coming back to a project in 6 months, and trying to unpick what you did, is a rubbish experience (and one I’m sure other software engineers can relate to). I went with what I knew - terraform¹⁰ to define the infrastructure, and ansible¹¹ to provision it. Both tools have good API integrations with DigitalOcean, which makes defining and provisioning infrastructure pretty straightforward.

Defining a droplet in terraform, with monitoring enabled, IPv6, and a floating IP, looks like this (noting that some of the backing information like a VPC for networking, or SSH key, is omitted):

resource "digitalocean_droplet" "carrot_blog_1" {
    name = "carrot-blog-1"

    image = "docker-20-04"
    region = "lon1"
    size = "s-1vcpu-1gb-intel"
    vpc_uuid = digitalocean_vpc.carrot_blog.id

    tags = ["carrot_blog"]

    monitoring = true
    ipv6 = true

    ssh_keys = [digitalocean_ssh_key.carrot.fingerprint]

    user_data = <<EOF
    		#! /bin/bash
        sudo hostnamectl set-hostname carrot-blog-1
EOF
}

resource "digitalocean_floating_ip" "carrot_blog_1" {
  droplet_id = digitalocean_droplet.carrot_blog_1.id
  region     = digitalocean_droplet.carrot_blog_1.region
}

DigitalOcean helpfully provide a “Docker” base image (the docker-20-04 bit above), so there isn’t actually that much to provision with ansible - just the basics like making sure the firewall is configured correctly, and changing Docker DNS settings. The only thing worth noting is that it’s best to use ansible’s “dynamic inventory” feature, where it queries the DigitalOcean API to figure out what the IP addresses of your droplets are. Otherwise, you’d have to change your “inventory file” every time the IP changed, which is a bit of a faff.

There’s a community ansible plugin called community.digitalocean.digitalocean which you can use for the dynamic inventory. I made a file called digital_ocean.yaml, containing:

plugin: community.digitalocean.digitalocean
api_token: '{{ lookup("pipe", "./get-do-token.sh") }}'
attributes:
  - id
  - name
  - memory
  - vcpus
  - disk
  - size
  - image
  - networks
  - volume_ids
  - tags
  - region
keyed_groups:
  - key: do_region.slug
    prefix: 'region'
    separator: '_'
  - key: do_tags | lower
    prefix: ''
    separator: ''
compose:
  ansible_host: do_networks.v4 | selectattr('type','eq','public')
    | map(attribute='ip_address') | first
  class: do_size.description | lower
  distro: do_image.distribution | lower
filters:
  - '"carrot_blog" in do_tags'

This file tells the plugin to filter by the carrot_blog tag, and maps the DigitalOcean API response in to attributes that can be used when running ansible playbooks. It references another script, get-do-token.sh, to get a valid API token from my local keychain:

#!/usr/bin/env bash
set -eou pipefail

security find-generic-password -a ${USER} -s CARROT_BLOG_DO_API_TOKEN -w

Running the playbook then looks like this (where infra.yaml is the ansible playbook I’ve defined to provision the server):

#!/usr/bin/env bash
set -eou pipefail

CARROT_BLOG_DO_API_TOKEN="$(./get-do-token.sh)"
CARROT_BLOG_AGENT_IDENTITY="~/.ssh/id_rsa_carrot_blog"
ANSIBLE_HOST_KEY_CHECKING=False

DO_API_TOKEN="${CARROT_BLOG_DO_API_TOKEN}" ansible-playbook -u root \
    --private-key "${CARROT_BLOG_AGENT_IDENTITY}" \
    --inventory-file=digital_ocean.yaml \
    -e 'ansible_python_interpreter=/usr/bin/python3' \
    infra.yaml "$@"

Backend service

With a place to put the API server, I could start building the backend service that would make calls to the Patreon API, and cache the results. As before, I went with what I know, and chose Ktor¹² for the framework. It’s a Kotlin web framework built by JetBrains (the same people who created Kotlin itself) and I’ve enjoyed using it for other projects in the past.

It’s good to build features incrementally instead of all at once, so I started with a very basic endpoint that used some hand-crafted “fetchers” to piece together bits of the Patreon API, in to a list of privacy-aware sponsors. As I mentioned before, the Patreon API uses the “JSON API” standard, and I didn’t want to complicate the project by depending on many third party libraries, so I decided to just piece these together in the fetchers themselves using the Kotlin standard library. It was a good excuse to try Kotlin’s kotlinx.serialization¹³ library, which can generate serialization code without using Java reflection. Instead, you annotate data classes in your code, and it uses an annotation processor to create the appropriate serialization logic during compilation. I’ve put an entire “fetcher” up in a GitHub Gist if you’d like to get a feeling for what those classes do¹⁴.

During the previous stage of planning I noted that sometimes the Patreon API could be quite slow - taking many seconds to return a list of sponsors. I knew that I would need to cache the results for a certain amount of time, so that my own API could be fast. The actual caching logic can be a bit complicated, to avoid the “thundering herd” problem when lots of clients cause a refresh at the same time, so I hid it behind a ReplayingCache implementation that I won’t go in to in too much detail about here. It both caches results, and handles refreshing the data in a thread-safe way, when a client requests it.

With the cache in place, the endpoint to fetch a list of sponsors looks pretty simple:

package sponsor.api.route

import io.ktor.application.ApplicationCall
import io.ktor.http.HttpStatusCode
import io.ktor.response.respond
import kotlinx.serialization.Serializable
import sponsor.api.cache.ReplayingCache
import sponsor.api.fetcher.SupportersFetching
import sponsor.api.serialization.OffsetDateTimeSerializer
import java.time.OffsetDateTime
import java.time.ZoneOffset

class V1SupportersRoute(
    private val supportersCache: ReplayingCache<SupportersFetching.SupportersDTO>
) {

    @Serializable
    data class SupportersResponse(
        val supporters: List<Supporter>,
        val goal_count: Int,
        val meta: Meta
    ) {

        @Serializable
        data class Supporter(
            val title: String?
        )

        @Serializable
        data class Meta(
            @Serializable(OffsetDateTimeSerializer::class)
            val updated_at: OffsetDateTime?
        )
    }

    suspend fun handle(call: ApplicationCall) {
        val cachedValue = supportersCache.fetch()
        if (cachedValue == null) {
            return call.respond(HttpStatusCode.NotFound)
        }

        val response = SupportersResponse(
            supporters = cachedValue.value.supporters,
            goal_count = cachedValue.value.goalCount,
            meta = SupportersResponse.Meta(
                updated_at = cachedValue.updatedAt.atOffset(ZoneOffset.UTC)
            )
        )
        return call.respond(
            HttpStatusCode.OK,
            response
        )
    }
}

At this point, almost all of the requirements from above are met 💪. We can hit an API route, which fetches information from Patreon, parses it, and caches the result. The final piece was to make it “as live as reasonably possible”, which I knew would require the use of webhooks and websockets.

A webhook is, in simple terms, when an API provider sends an event to a URL of your choosing, as something happens. You can see why this is helpful for live events - for example, as soon as a new sponsor signs up, Patreon could make a request to our API service saying as much, instead of us asking all the time.

I registered for the members:pledge:create, members:pledge:update, and members:pledge:delete Patreon events using their dashboard. When receiving one of those events, the backend services does a full refresh of the current sponsors, with a cooldown timer to prevent sending lots of requests if a few people sponsor at once. In theory you could use the contents of these events to update the list “piece by piece”, but I’ve often found that to be more hassle than it’s worth - sometimes data in eventually consistent systems doesn’t quite match up at a given point in time, so it’s better to request the entire state to avoid strange inconsistencies.

Finally, I used Ktor’s built-in websocket support to add an endpoint that the frontend could connect to. On connection, the backend would send the full sponsors state, and then monitor the cache to see when anything changed, at which point it sends another full state refresh.

The core of the websocket implementation looks like this (with some functions omitted) - a connection is opened, the full state is sent, and every 10 seconds the state of the cache is checked and sent if there are changes:

suspend fun handle(session: DefaultWebSocketServerSession) {
    val connection = trackConnection(session)
    var previousValue = supportersCache.fetch()
    if (previousValue != null) {
        sendFullState(previousValue, connection)
    }
    while (!session.incoming.isClosedForReceive) {
        val newValue = supportersCache.fetch()
        if (newValue != null && (previousValue == null || (newValue.value != previousValue.value))) {
            previousValue = newValue
            sendFullState(newValue, connection)
        }
        delay(5000L)
    }
    removeConnection(connection)
}

A trick worth noting here is that the “full state” event sent through the web socket uses exactly the same payload as the REST endpoint, wrapped in a very simple JSON frame:

An example `full_state` event in Firefox

Doing so makes both the backend and frontend implementations simpler, and they both need the same data anyway, so it’s a win-win.

Frontend

With the backend finished, we can do the final step of integrating it in to the frontend. I have to admit that frontend engineering isn’t my forte, so I chose to keep it as simple as possible - plain Javascript where possible, without any npm modules or build systems, in a single file, so that I can include it in pages in Jekyll as desired.

On page load, the script first checks whether websockets are supported in the browser or not. If not, it uses the REST endpoint described above, and doesn’t attempt to refresh anything live. If websockets are supported, it establishes a connection, listens for full_state events, and attempts to reconnect if the connection is lost (for example, if someone navigates to another browser tab for a while). There’s too much code to include inline, but here the main websocket pieces:

function connectWebsocket() {
  const socket = new WebSocket('{{ site.websocket_url }}/v1/supporters/websocket');
  socket.addEventListener('message', function (event) {
    const message = JSON.parse(event.data);
    if (message.type === 'full_state') {
      updateSupporters(message.data);
    }
  });
  socket.addEventListener('close', function (event) {
    const extraTimeMs = makeRandomInt(0, 4000);
    const waitTime = 1000 + extraTimeMs;
    console.log("Websocket closed, attempting reconnect in " + waitTime + "ms", event.reason);
    setTimeout(function() {
      connectWebsocket();
    }, waitTime);
  });
}

const supportsWebSockets = 'WebSocket' in window || 'MozWebSocket' in window;
if (supportsWebSockets) {
  connectWebsocket();
} else {
  fetch('{{ site.api_url }}/v1/supporters')
    .then(response => response.json())
    .then(data => {
      updateSupporters(data);
    })
    .catch((error) => {
      for (var div of supportersMessageDivs) {
        div.innerHTML = '<p>Failed to load sponsors data from Patreon 💔</p>';
      }
    });
}

An unexpectedly nice thing I discovered was that, by creating the JavaScript file as a Jekyll include, I could use variables just like you can in mustache templates (in this case, the REST and websocket URLs, {{ site.api_url }} and {{ site.websocket_url }}). Here’s an example of how I include the script on a given page - nice and easy:

## Supporters

These are the kind people who have already sponsored. Hover/click on a loaf to see their name, or add your own. Come back to the page after - it updates live!

{% include patreon_sponsors.html %}

Conclusion

I’m really pleased with how this feature turned out. I’m especially happy that the list updates live - usually within a few seconds of a patron sponsoring. It was fun to build, and I think it adds a little bit of extra sparkle to the website ✨.

I’ve included the component here, click/tap an empty slot to sponsor and try it out!

I hope this post was helpful, and if you have any questions, I’m happy to answer them on Patreon or Discord!

Footnotes

Notion - https://www.notion.so/ ↩
Patreon API documentation - https://docs.patreon.com/#introduction ↩
Patreon developer portal - https://www.patreon.com/portal ↩
JSON API standard - https://jsonapi.org/ ↩
HTTPie - https://httpie.io/ ↩
Patreon Webhooks - https://www.patreon.com/portal/registration/register-webhooks ↩
GitHub Pages - https://pages.github.com/ ↩
Cloudflare - https://www.cloudflare.com/ ↩
DigitalOcean (referral link) - https://m.do.co/c/6e8253961242 ↩
Terraform - https://www.terraform.io/ ↩
Ansible - https://www.ansible.com/ ↩
Ktor - https://ktor.io/ ↩
kotlinx.serialization - https://github.com/Kotlin/kotlinx.serialization ↩
Fetcher example - https://gist.github.com/lopcode/60457a994a3b0b0c7daa55e1f6e3227a ↩