Let's Encrypt FS2! My experience

The Goal

To run a Scala-FS2 TCP server with Let's Encrypt's certificates.

The Motivation

This tutorial is developed as part of the Let's Encrypt Scala project as well as part of Scala Algorithms, where I wanted to collect live logs from Heroku's Logplex. For this, I chose to use FS2, because I had used it successfully on multiple other projects like syslog-ac. However, this time I needed SSL/TLS because the live logs are now going over the network and cannot go unencrypted! Being on the JVM, some things are not always so straightforward, especially when it comes to security.

This is a retrospective and not a tutorial, it is highly unlikely that anyone would need to develop a non-HTTP with SSL/TLS server from scratch/directly; if you want to do this for http4s, check out my detailed tutorial here: Let's Encrypt http4s :-). However, you might be one of the few working on these cool custom things, so this retro is for you :-)

First step: getting that connectivity going.

Incorporating certificates into the JVM

I had not found a standard automated way to use Let's Encrypt certificates on the JVM; thankfully there is a manual tutorial on Let's Encrypt's community forums: Tutorial - Java KeyStores (JKS) With Let's Encrypt. This was so manual and error-prone, that I decided to come up with the library Let's Encrypt Scala, which would take a Let's Encrypt certificate's directory, and turn it into an SSLContext, in one call.

Compositionality of FS2

Now that I finally built up that Java KeyStore, I can plug it into FS2. But where is an example of a TLS server? I can only find examples of clients! Well, guess I can just take the TLS client's example code, take a non-TLS server's code, and put it together - and that is all it took, thanks to the awesome compositionality of FS2. Of course you might not want to go through having to figure this out yourself, so I am sharing the source code for that below. Also, there are great examples in the unit test cases; if you ever cannot find an answer in the docs of any package, search for test cases; and if there are no test cases, search for a package that is at minimum unit-tested :-).

Nonetheless, now I have a TLS server parsing my syslog messages securely from Heroku. What a total treat! Until... it crashes in the middle of the night.

’Unrecognized SSL message, plaintext connection?’

This one really got me. Somebody connected to my SSL server without SSL. The connection died and took the server process with it.

screenshot of a crash message — Not the thing you want to see crash a live app.

I started furiously trying to figure out the error, trawling through fs2-io code and could not find anything to handle this error... if my server crashes, and there could also be other people doing the same thing... this could be a denial-of-service issue. I wrote up some test cases and sent them to a maintainer of FS2. At first glance, it seemed indeed that I was right, but after another check, he pointed out the error in my understanding. Thank you for your patience :-)

While I had handling of errors, it was at the wrong level: because it is a very compositional approach, you have the advantage of controlling the boundaries of your apps, and if you don't use them, the whole thing crashes. While it sounds bad at the outset, it is actually miles better than a silent failure, which could go on for days before anybody realises.

In the end, there was no CVE to file. But in the meanwhile, I found a very cool tool called Hercules ((chocolatey link on Hercules)) to play around with sockets on Windows; a much more pleasant UX compared to telnet. And to speak to a TLS server, use the gnutls-cli utility: gnutls-cli manpage.

A dual-mode TLS & plaintext echo server with FS2

I always want to go beyond proof-of-concept, and one of things that is important is making a distinction between development mode and production mode. When you are developing, you want to minimise the number of things you need to set up - download the source, set up the build tool, run the app. This app in particular will run without TLS locally, and upon deployment use TLS.

To actually use it, please borrow the instructions from the tutorial Let's Encrypt http4s!, as the underlying basis is exactly the same. I am not putting the contents here to avoid content duplication and divergence; and also to make this more of a retrospective than a tutorial.

The below code is model code that you can use to develop your own app because it considers the local unencrypted case and the production encrypted case. A link to the GitHub repository is at the end.

Without further ado, let's delve into the code; all the explanations are in the code comments.

The build.sbt definition

We include the necessary dependencies, and also include a configuration option for sbt-revolver. sbt-revolver is "a plugin for SBT enabling a super-fast development turnaround for your Scala applications". You can even combine it with TDD, where on every compile it checks if tests need to be re-run, runs them, and then restarts your app in the background: I posted how to do this in one line on my Twitter.

ThisBuild / resolvers += Resolver.sonatypeRepo("snapshots")

scalaVersion := "2.13.6"

libraryDependencies ++= Seq(
  "co.fs2"           %% "fs2-io"          % "3.0.1",
  "com.scalawilliam" %% "letsencrypt-ce3" % "0.0.5-SNAPSHOT"
)

reStartArgs += "--insecure"

SBT plug-ins

This is the list of plug-ins used, in the file project/build.sbt:

addSbtPlugin("io.spray"         % "sbt-revolver"        % "0.9.1")
addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.8.1")

We include sbt-native-packager for deployments.

The main app

Tip: it is worth checking out the project and having a look at the types in IntelliJ/Metals.

package com.scalawilliam

import cats.effect.kernel.{Async, Sync}
import cats.effect.{ExitCode, IO, IOApp, Resource}
import com.comcast.ip4s.Port
import com.scalawilliam.letsencrypt.LetsEncryptScala
import fs2.io.net.tls.TLSContext
import fs2.io.net.{Network, Socket}
import fs2.{INothing, Stream, text}
import cats.effect.Concurrent

object FS2LetsEncryptEchoApp extends IOApp {

  type SocketHandler[F[_]] = Socket[F] => Stream[F, INothing]

  /** Respond to a client socket with echoes - if they send a line, we return a line */
  private def handleEchoes[F[_]: Concurrent]: SocketHandler[F] =
    client =>
      client.reads
        .through(text.utf8Decode)
        .through(text.lines)
        .interleave(Stream.constant("\n"))
        .through(text.utf8Encode)
        .through(client.writes)
        /** This is VERY important; if you don't handle errors, the whole server will go down;
          * One example is if a plaintext client connects to an SSL server */
        .handleErrorWith(_ => Stream.empty)

  private def runTcpServer[F[_]: Concurrent: Network](
      forHandler: SocketHandler[F]
  ): F[Unit] =
    Network[F]
      .server(port = Port.fromInt(5555))
      .map(forHandler)
      .parJoin(100)
      .compile
      .drain

  /** Utility function to pipe a socket through SSL.
    * This is a prime example of composition in functional programming:
    * you merely 'secure' a socket handler, without having to
    * modify any of the plaintext server code.
    *
    * We have to use a 'Resource' here because the SSL Context
    * is something we should let go of once not needed any more.
    * */
  private def secureHandler[F[_]: Concurrent: Sync: Async](
      originalHandler: SocketHandler[F]
  ): Resource[F, SocketHandler[F]] =
    LetsEncryptScala
      .fromEnvironment[F]
      .flatMap(_.sslContextResource)
      .map(TLSContext.Builder.forAsync[F].fromSSLContext)
      .map { tlsContext => (clientSocket: Socket[F]) =>
        fs2.Stream
          .resource(tlsContext.server(clientSocket))
          .flatMap(originalHandler)
      }

  private def secureConditionally[F[_]: Concurrent: Sync: Async](
      enableSecurity: Boolean,
      handler: SocketHandler[F]
  ): Resource[F, SocketHandler[F]] =
    if (enableSecurity) secureHandler(handler)
    else Resource.pure(handler)

  override def run(args: List[String]): IO[ExitCode] =
    secureConditionally[IO](
      enableSecurity = !args.contains("--insecure"),
      handler = handleEchoes
    ).use(handler => runTcpServer(handler).as(ExitCode.Success))

}

The code is available at the Git repository.

Overview

We developed an FS2 app that echoes messages back to the user. Most likely you will not need this as most developers out there use HTTP, so if you would like to do that, head over to Let's Encrypt http4s! (tutorial).