Towards a better Integration of the Julia REPL in Emacs

Published: July 18, 2017

It is not a secret that I am a fan of both Emacs and the Julia language. Even though I have not been working with Julia a lot recently, I still try to keep up to date with the language and use it in my research and other projects whenever it fits. It should also not be a secret that I am a big fan of the Emacs Way™ of working with interpreted languages, which involves the idea of controlling a REPL without leaving the editor. It was for those and other reasons that I initially hacked together julia-shell-mode (See my blog post about it here), but if you take a closer look at the open isses in GitHub and the overall lack of commits, you might sense that I have lost interest in this package. Of course, the reality behind my lack of enthusiasm for julia-shell-mode is a little more complicated.

Long story short, julia-shell-mode sucks. I threw it together out of necessity and basically ripped off most of matlab-mode's ancient Elisp code for parsing stdout of a MATLAB process. Among some of the disadvantages to this approach are:

In an ideal world, all interpreters would expose well-defined, versioned, and networked APIs which we can speak to from whatever editor or environment we like. This would mean that the interpreter would need to act as "kernel" which gets…

>> Hold on, I thought we already had…

Exactly. We already have all that we need in the Jupyter project. Different interpreters running behind different frontends communicating over a well-defined messaging protocol. This sounds pretty good! With a good interface in place, a seemingly well-maintained Julia kernel (IJulia.jl), and the flexibility of Emacs Lisp at our hands, what can stop us from building the perfect Julia REPL in Emacs?

Here is the problem: The Jupyter messaging protocol relies on ZeroMQ as its message transport mechanism. Unfortunately, there are (to the best of my knowledge!) no 0MQ bindings for Emacs Lisp… If there only was a different way of using the 0MQ bindings in Emacs…

>> Hey, have you heard of that new dynamic modules feature? I've heard that some people already used it to expose all kinds of DLLs to the emacs environment.

Ah. Sometimes, things just come together nicely. Using dynamic modules, we could expose 0MQ messaging functionality to Emacs and make steps towards interacting with Jupyter kernels!

Interestingly enough, I was not the only one that thought of this possibility over the weekend. John Kitchin at CMU is on the right track here. He wrote a shared library which exposes open, close, send, and receive of a 0MQ socket to Emacs. Oh how excited I was when I read his posts! His work inspired me to try my own quick hello world example that pings on a request/response socket. The difference between mine and John's implementation, however, is that I am using C++ to write the module instead of C. Mixing C and C++ can be a little difficult at times, but I think I figured out a nice and clean way to handle it. You can check out the repo for the whole thing, and I give a preview in the gist below.

Now, on first sight this looks basically like old-school C, but remember: this is only the thin layer between Emacs and your our program. Anything that does not require direct knowledge of the Emacs environment can be written in modern C++ (after casting a few void pointers of course…) For example, the "Connection object" to which I am returning a pointer to Emacs, looks like this:

Even though my lack of expert knowledge about C++ prevents me from declaring a piece of code as "modern" or not, what I am trying to get across is that, Yay!, we can write Emacs modules in C++! This means that we do not have to worry about memory management, we get a sweet standard library (lambdas!), and we get to use Boost!

Whoa. With all that out of the way, let's get back to where we started. Or better: what I started yesterday. I am now building a Jupyter client in C++ which is meant to interface with Emacs as a dynamic module. I am not sure when this thing will be usable, but I am inviting everyone interested to collaborate. Right now, we use CMake to build and we will try to be as cross-platform as possible. I don't have the architecture of this thing fully drawn out yet, so we're just going to have to design as we build (and then, thus, likely re-design). As of right now, I have some basic connectivity between Emacs and the kernel going, which means that I can read a connection file, open the relevant 0MQ sockets, and start a heartbeat thread. This is the easiest form of communicating with a kernel, but I guess you have to start somewhere ¯\_(ツ)_/¯.

It is nice to see a quick and early payoff: Here is a picture of Emacs (in --batch mode) talking to a running IJulia kernel, asking it whether it is alive:

jupyter_kernel_alive.png

It's pretty cool. Anyways. I'll be hacking on this in my free time. Maybe the dream of a good Julia REPL in Emacs will come true soon…