Robot Testing With Emacs

I have written about assess previously (n.d.) it is a tool which provides predicates, macros and functions to support testing for Emacs. It is actually agnostic to the test environment, although has specialised support for ERT.

My new release of assess (v0.3.2) includes one significant change, and two new features. I have updated the call capture functionality — the first version stored all the call data in a global variable, which was quick and easy, but clearly not a log term solution. It now uses closures instead which means that several functions can be captured at once. This also allows the first new feature, which is the ability to capture calls to hooks, with the function assess-call-capture-hook, which takes a hook and a lambda, and returns any calls to the hook when the lambda is evaluated. As an example usage, from assess-call-tests.el:

(should
 (equal
  '(nil nil)
  (assess-call-capture-hook
   'assess-call-test-hook
   (lambda ()
     (run-hooks 'assess-call-test-hook)
     (run-hooks 'assess-call-test-hook)))))

This is good functionality and should be very useful. The API could be improved a bit; a macro version would avoid the explicit lambda, for example. And returning a list of nil means this function also works with hooks with args, but is a bit ugly for hooks without (which are the majority).

The second area that I wanted to address has come about because of my hacking into the Emacs undo system. This is hard to test automatically; I have often found myself writing things like this test from simple-test.el.

(should
 (with-temp-buffer
   (setq buffer-undo-list nil)
   (insert "hello")
   (member (current-buffer) undo-auto--undoably-changed-buffers)))

This is okay, but it’s painful to write; I am trying to robotize Emacs, and it’s not easy. Some times it’s hard to work out exactly what set of functions you need to call. It would be much easier just to type a key sequence and have Emacs run this for you.

Fortunately, Emacs has special support for this in the form of keyboard macros; you can remember, store and save any set of keypresses and run them, rerun them, automate them or, most importantly, save them to a file as a lisp expression. This example, for instance, comes from viper-test.el.

(kmacro-call-macro nil nil nil
                   [left
                    ;; Delete "c"
                    backspace
                    left left left
                    ;; Delete "a"
                    backspace
                    ;; C-/ or undo
                    67108911])

This is okay, but it’s still not ideal. I have had to add comments to make the test clear by hand. It’s not easy to read, and what is that 67108911 about? It comes from somewhere in Emacs and is stable into the future. But, you only have my word for it that this is undo. It would be all too easy to get this wrong, to have the wrong comment. Tests need to be readable.

Fortunately, Emacs provides a nice solution, in the form of edmacro — this is a major-mode for editing macros after they have been created. It also defines a human readable version of a macro. We can parse this and then execute it directly. This example comes from simple-test.el.

(execute-kbd-macro
  (read-kbd-macro
        "
C-c C-o         ;; latex-insert-block
RET         ;; newline
C-/                     ;; undo
"))

The advantage of this is that I didn’t actually write this string; I recorded the macro, the edited it and copied the contents of the edmacro buffer.

This is still not easy enough, though; I want an easier way of editing the macro as it appears in the test. This is, unfortunately, difficult as the edit-kbd-macro is not easy to invoke programmatically — it absolutely hard-codes user interaction (I did even try invoking edit-kbd-macro using a keyboard macro!). So, I have given up with that approach in the short term. Instead, I have written a function assess-robot-execute-macro that combines read-kbd-macro and execute-kbd-macro, but also sets the macro as the last macro, making it easy to edit. I’ve also added a keybinding to edmacro to copy the macro to the kill-ring. And here is a test using it:

(should
 (assess=
   "hello"
   (assess-robot-with-switched-buffer-string
     (assess-robot-execute-kmacro
"
hello           ;; self-insert-command * 5
"))))

This also demonstrates the final quirk. Keyboard macros work in which ever buffer is selected — not the one which is current. We cannot use with-temp-buffer to select on temporarily and run the macro in it. So I have added macros to display a buffer temporarily instead.

As with many parts of assess, the back end is quite convoluted and complex, as many parts of Emacs were not written with testing in mind (that is they predate the whole idea of unit testing by many years!). But, I hope that the API that assess provides is simple, clear and flexible.

Assess is available at github and MELPA.

Feedback is, as always, welcome.