HTTP Basic Authentication for personal package.el repositories

Published: December 17, 2024

In my somewhat convoluted system to manage Emacs packages, I maintain a personal ELPA-compatible package repository (created using an ancient version of elpa-mirror) on this very server. Because these are my packages, I don't want to run this as a public-facing repository and I thus protect access to it using HTTP Basic authentication. For about five years, I was able to use this password-protected package repository by setting the package-archives variable to include the username and password in the URL, e.g., something like: 

(setq package-archives '(("my-repo" . "https://username:[email protected]")))

This special form of URL is now apparently being deprecated and I have noticed that recent versions of Emacs (I'm guessing 29+) do not support this URL scheme any more. I don't really know the details about these URLs, but some cursory googling produced some notes on the topic (cf. Dan Q, Neil Madden). Luckily, due to Emacs' flexibility, I was able to find a solution to this problem relatively quickly. As a bonus, this solution actually does HTTP Basic Auth right.

The basic idea here is that package.el and use-package use Emacs' URL library (manual), specifically url-retrieve to download the achive manifest and the individual packages via HTTP or HTTPS from the server. The right way to do HTTP Basic Auth is to include the username and password as an additional header in the HTTP request. The format of this header is explained in the Wikipedia article. All we have to do is concatenate the username and password, encode them using base64, and stick the result in the Authorization header. As the Wikipedia article explains, the Username Aladdin and password open sesame produce the HTTP header Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==.

The behavior of the url-retrieve function can be modified by dynamically binding variables like url-request-data, url-request-method and url-request-extra-headers around the function call. Here, the variable url-request-extra-headers is exactly what we need to inject our username and password into HTTP requests to our personal package repository. All we have to do is a) intercept every call to url-retrieve, b) check if the request is made to the URL of our repository, and c) if yes, add the HTTP Basic Auth header by dynamically binding url-request-extra-headers. The downsides to this approach are a) we have to advise url-retrieve, which comes with overhead and b) this method doesn't scale very well. For me, the overhead seems to not be a problem in my daily Emacs use so far and I do not need this method to scale anywhere past the single URL to my package repository. So let us take a look at how this is implemented.

First, I define the URL, the username, and the password in my init.el: 

(defconst do.bootstrap/archive-url "https://config.example.com/elpa/"
  "URL to my personal package repository")
(defconst do.bootstrap/archive-user "username"
  "Username for my personal package repository")
(defconst do.bootstrap/archive-password "password"
  "Password for my personal package repository")

Next, I use Emacs builtins to construct the Authorization header in the format that is expected for url-request-extra-headers: 

(defconst do.bootstrap/archive-auth-str
  (cons "Authorization"
        (concat "Basic " (base64-encode-string
                          (concat do.bootstrap/archive-user ":"
                                  do.bootstrap/archive-password))))
  "HTTPS Basic Auth string for my personal package repository")

Finally, I use Emacs' advice mechanism to intercept the calls to url-retrieve and inject the authorization header if necessary: 

(defun do.bootstrap/url-retrieve-check-for-archive (orig-fun &rest args)
  "Like `url-retrive', but check for `do.bootstrap/archive-url'
first. If URL starts with that, add the appropriate username and
password as extra HTTP headers."
  ;; the `url' argument is the first argument to the original function
  (if (string-prefix-p do.bootstrap/archive-url (car args))
      ;; if we are requesting from my archive, add the basic auth headers
      (let ((url-request-extra-headers (cons do.bootstrap/archive-auth-str url-request-extra-headers)))
        (apply orig-fun args))
    ;; else just call `url-retrieve'
    (apply orig-fun args)))

;; advise `url-retrieve'
(advice-add 'url-retrieve :around #'do.bootstrap/url-retrieve-check-for-archive)

This works, does HTTP Basic Auth right, and seems relatively futureproof. I don't know whether this is the "correct" way of solving this problem, so if anyone out there has a better method, please let me know.