Every now and then, in order to be more productive from then onwards, you have to bang your head against a wall in the now. This is a bit of what happened when my friend and mentor Matt Bishop set me thinking about writing ClojureScript lambda functions on AWS.
The goal here would be to avoid using the standard JVM-based compiler and instead use Lumo to interpret directly ClojureScript code in the cloud. This would allow us to capitalize on AWS tools like the integrated and collaborative IDE Cloud9.
I would like to share the journey that brought me to successfully compile the
lumo binary statically against
musl on the Amazon linux AMI so that I can go back to write Lisp.
One of the newest and coolest features of AWS Lambda is the possibility of providing your own runtime. The runtime layer is akin to a Docker layer and is shared across invocations. Each file you put in the layer will be visible to your running function.
This is already very useful for cross-cutting concerns: one could provide a layer for common dependencies that you upload once, or on update, instead of every time along with your function.
The killer feature though is that if one of the layers contains a shell script called
bootstrap, AWS will run that by convention on cold start.
Why is that a killer feature?
bootstrap can be used as the trampoline for running anything you like - including of course ClojureScript interpreters!
Everything looks shiny and bright on paper. Unless you find out that the paper is that bright because is burning.
In order to achieve that though,
lumo uses the Node.js snapshot feature, which effectively means that the
lumo binary is in reality a
lumo binary is built as dynamically linked binary for Ubuntu.
My guess is that by now you realize that this is not going to be fun. Do not worry, I have done the grunt work already and there is an happy ending.
Linking is actually a non-problem as long as we can build
lumo statically. No surprises there, a quick PR and voila, you can now compile
The whole idea also already existed in the world, as Mikkel Gravgaard had already worked something out in his grav/aws-lumo-cljs-runtime. Thank you Mikkel for showing me the path and answering to questions on GitHub.
So one day you demo your breakthrough to the aforementioned: "Hey Matt look! there is better way to run ClojureScript on AWS, without touching the JVM 1 !".
You are happy, you show your superior workflow at the REPL.
(require 'aws-sdk) (def s3-client (aws-sdk/S3.)) (.getObject s3-client ...) ...Segmentation fault (core dumped) !?!$%*...
Node.js segmentation fault, interesting. You try again, maybe it is a demo curse issue but inside you already know that you are no, the problem lies somewhere else.
The real problem here is
This is a weird one though. The
lumo binary that the
This is the time when you need Stack Overflow again 2.
It turns out that the standard
glibc included in most of the linux distributions is not very "flexible". Not that I really know what flexible technically means here but because
node needs to compile against it, we need something else.
Enter musl, pronounced like the word "mussel": an open source standalone implementation that can be used as drop in replacement to
Achievement unlocked, the plan is now to build
lumo inside the
amazonlinux docker image with static linking against
It is a smooth sail from now on, just build an image that allows the compilation steps to use
musl-gcc instead of the standard
gcc (again thanks Mikkel for the initial
I wish it was, sigh. The last iceberg to dodge is that you cannot really rely on
musl-gcc for compiling
node. You need the fully fledged cross-compiled toolchain for it. In particular
g++ is used as linker.
Fortunately, the rabbit hole ends here.
Use it in conjuction with
grav/aws-lumo-cljs-runtime for building the actual AWS Runtime layer.
We were trying to avoid the JVM cause of its higher startup time compared to tooling for other languages.↩
It is difficult to believe but I rarely open Stack Overflow when I develop Clojure. The language rarely gets in the way.↩
It goes without saying that I am immensely grateful to the maintainer and the contributors.↩