✂️ snips.sh retrospective: 1000+ stars later

Finally released https://t.co/GEAONuN7Mt 🎉

Need to upload a code snippet?

echo "something" | ssh https://t.co/GEAONuN7Mt

It's a free, anonymous, open source, snippet hosting service pic.twitter.com/u2hUvl9gRO
— Rob Herley (@robherley) May 12, 2023

# What the snip?

A tad bit over a year ago, I released https://snips.sh, a passwordless, anonymous SSH-powered pastebin with a human-friendly TUI and web UI. No logins, no passwords, nothing to install. It’s ready-to-go on any machine that has SSH installed.

It’s a simple as:

echo 'this is amazing!' | ssh snips.sh

I wanted an easy utility to copy code snippets to/from machines, a dead-simple web UI to link to line numbers and something to just dump code snippets.

And the development community loved it. To my surprise, it rapidly gained popularity across social media. It even made the top of GitHub’s /trending under the Go language category for a couple days.

stargazers — Surpassed 1k stars just over a year!

Given I procrastinated for over a year making an original “release” blog post for snips, I figured a retrospective would be just as good.

# The philosophy

When designing snips, I wanted it to be as simple as possible. If I learned anything from maintaining open source libraries and supporting public APIs used by millions of people, it’s important to not bloat with verbose functionality that becomes a maintenance/compatibility nightmare.

I manifested my inner Ken and Dennis and kept the Unix Philosophy top of mind:

Write programs that do one thing and do it well.
Write programs to work together.
Write programs to handle text streams, because that is a universal interface.

And that’s exactly what snips is.

It’s a snippet store (with a UI and TUI), nothing more.
It works with other command line programs via pipes.
The “API” is just text over stdin/stdout.

pipe examples — Pipe into whatever you want

But, this isn’t just a command line utility. While I love the Unix Philosophy, it is not my creed. Just as much as I believe simplicity is key in software development, the user experience is just as important. This is often a very hard balance.

Under the covers, snips.sh is a stateful remote resource that requires functionality beyond the simple input/output. And that’s what the TUI is for. It’s a shell into the user’s snips. You can use the TUI to view snips syntax highlighted, edit attributes and delete them.

tui — A user can `ssh` into the TUI to view/manage snips

As a developer building tools for developers, I know how comfortable most are in the terminal, which is why I chose that as the entrypoint over a web UI. They don’t even need to lift their fingers off the keyboard.

I also wanted the onboarding experience to be as smooth as possible. Here’s how the upload works for a new user:

%%{init: {'theme': 'dark' } }%% flowchart TD ssh([ssh session]) fail((fail)) onboard((onboard)) a@{ shape: diamond, label: "auth" } pk@{ shape: diamond, label: "exists?" } wf((write snip)) ssh --> a a --"password"--> fail a --"pubkey"-->pk pk --"no"--> onboard pk --"yes"--> wf onboard --> wf

The usage of snips.sh does require public key auth in order to identify users. If a connection attempt is made with a password, it fails and sends a message to stdout. This also helps prevent against bots and other things that like to poke at port 22.

For any new users, if their public key doesn’t exist in the database, we’ll “onboard” which will create a new user record and associate a public key with it. A terms of service message is also printed for new users. Since we’re able to create a new identity and automatically onboard, there is literally zero friction to get started.

While it’s nice to keep things as simple as can be, like anything with software engineering “it depends” on your use case. Personally, I’m a fan of making easy (and fun!) to use software, which might need a few engineering tradeoffs for a better user experience.

As it turns out, people like easy to use software:

Bye bye pastebin !! This is something I am going to use forever.

What a fantastic project ✨
- Easy to use
- top-notch UX
- Works perfectly https://t.co/TGCjC4PId0
— Arnab Sen (@ArnabSen1729) May 13, 2023

This is going directly into my toolbelt. Incredibly cool idea and absolutely mindblowing execution 👏 https://t.co/2n6mamo7E1
— Silas Hansen (@silasmorkgard) May 13, 2023

# The technology

All of snips.sh is written in Go, from the SSH app to the web UI. While Go may not be as creative or fast as other languages, I do find beauty in the simplicity.

As luck might have it, there’s also an organization called Charm that builds amazing libraries for the command line, all built in Go. So surprise, snips.sh uses plenty of Charm libraries. They’re a great group of people, you should totally check them out.

There’s no real fancy frameworks in snips.sh. All the web-based routes use the standard library’s net/http server along with the html/template package for server side rendering. There is about ~120 lines of JavaScript and some old fashioned hand-rolled CSS to keep things as tiny as can be.

As for the backing storage, I went with the most deployed database in the world, SQLite. Why SQLite you ask?

It’s really fast.¹
It’s stupid simple to use. It’s embedded and doesn’t need extra resources/configuration.
The database is all stored in a single file, making it easy to manage (and backup).
Some people much smarter than me have been scaling it like crazy.²

While some S3-compatible storage might have been first choice for some, I considered it overkill. Having to run another program or worse (connect to a cloud provider!) I figured the 1MB file size limit would be absolutely fine in a blob column, especially since it’s compressed with zstd too.

For a lot of the fancy web UI rendering, I have to give credit to some amazing open source libraries:

alecthomas/chroma: Syntax Highlighter
yuin/goldmark: Markdown Parser
microcosm-cc/bluemonday: HTML Sanitizer
tdewolff/minify: Asset Minifier

And that’s pretty much it! Keeping the technology simple means it’s just as easy for someone else to run snips.sh on their own hardware. And that’s exactly why we have a self hosting guide and publish an multi-arch container image to GitHub Container Registry:

ghcr.io/robherley/snips.sh

Given the simple tech stack, it’s pretty easy to get going after a couple volume mounts and environment variables.

# The tensorflow-sized elephant in the room

So, one must have that I wanted for snips is to automatically detect the uploaded code language. To do this, I used a tensorflow model, yoeo/guesslang. This is actually the same model that Visual Studio Code uses, but they use Tensorflow.js, you can check it out at Microsoft/vscode-languagedetection.

But we do not have server side JS here, we’re in a compiled language. This was my first hurdle, and I ended up writing robherley/guesslang-go which uses some wrappers around libtensorflow’s C API.

Unfortunately, this means we lose the ability to make a static executable and need to sacrifice portability:

you@local$ docker run -it --entrypoint=ldd ghcr.io/robherley/snips.sh /usr/bin/snips.sh
	linux-vdso.so.1 (0x00007ffd219a8000)
	libtensorflow.so.2 => /usr/local/lib/libtensorflow.so.2 (0x00007f2922485000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f292225c000)
	libtensorflow_framework.so.2 => /usr/local/lib/libtensorflow_framework.so.2 (0x00007f292036d000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f2920368000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2920363000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f292027a000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2920275000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f2920049000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f2920029000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f2933a32000)

Even worse, look how big this is!

you@local$ docker run -it --entrypoint=ls ghcr.io/robherley/snips.sh -lah /usr/local/lib
total 416M
drwxr-xr-x 1 root root 4.0K Sep 13 15:53 .
drwxr-xr-x 1 root root 4.0K Aug  8 14:03 ..
-r-xr-xr-x 1 root root 375M Sep 13 15:51 libtensorflow.so.2
-r-xr-xr-x 1 root root  42M Sep 13 15:51 libtensorflow_framework.so.2

Yikes. Not ideal.

I did search for alternatives, like relying on chroma’s built in lexers to identify the language but it was not good enough for small snippets. Other language detection features of editors and other tools like GitHub’s linguist rely on file extensions, which we don’t have.

This is a prime example of making sacrifices for an extremely useful feature. It does put a smile on my face when I see the correct language detected on upload.

Another huge gotcha with libtensorflow is the lack of support for many architectures. Luckily this can be solved with some compiler flags (-tags noguesser) and a multiarch container image, but some users lose that critical functionality.

This is an area that I am not very strong in. I’d love any suggestions on this topic, feel free to open an issue.

# The ship

I started on this side project at the beginning of 2023³, and “released” it via social media in May of that same year. This was my first real side-hack that turned into a pretty useful tool, and I felt a warm welcome from the developer community.

My largest audience was Twitter, having over 120k views on my tweet. Some retweets from folks like @mxcl (creator of Homebrew) and @charmcli really helped get it to the right audience.

Surprisingly, even folks on reddit took it pretty well!

snips.sh: passwordless, anonymous SSH-powered pastebin

I was delighted to see snips.sh in all difference communities:

console.dev made a review: https://console.dev/tools/snips
@JeremiahSecrist published a nixpkg: https://mynixos.com/nixpkgs/package/snips-sh
@Sanix-Darker made a nvim extension: https://github.com/Sanix-Darker/snips.nvim

Shortly after release, we already had issues and contributions coming in too!

It truly was a great ship! 🚢

# The numbers

# Connections

Users can reach snips.sh via HTTP or SSH. The request metrics are emitted to DataDog but unfortunately I only have up to a little over a year’s worth of retention, so here’s since July 2023:

HTTP Requests

SSH Sessions

Note: the above SSH sessions are for successfully authenticated users. If we include all non-authenticated (anything hitting port 22), snips.sh has seen 2.148 million unique SSH sessions.

# App

This entire app is hosted on a Digital Ocean droplet to keep costs low. The database is still relatively small around ~24MB, which is not including backups on Digital Ocean Spaces via litestream.

Users

Files

Langs

After copying the sqlite database to my host machine and running aggregations, we can see a nice time series of usage:

Unsurprisingly, we had a huge burst of users during the “Twitter hype period” and gradually slowed down. While I would have loved to market this more, my goal wasn’t to make a disrupting product, just a fun developer tool. Plus, over the lifecycle of this release, I was busy planning an engagement and then my wedding!

Back to the metrics, we had the usual suspects of popular files.

files by type — Files by programming language

You can find the full language list… on snips!

# Open Source

We’ve had some great contributions like zstd compression, arm64 support, bug fixes and more. Dependabot is also carrying the weight a bit with ~65 Pull Requests alone.

Commits

Pull Requests

Issues

Stars

Contributors

# Thanks!

Appreciate all the contributions and kind words people have given me throughout this project! It gives me the motivation to keep on building, and you should too 💪