Off The Main Loop: Async Actions within Neovim

    Neovim embeds the libuv library in the editor and exposes lua (and to some extent vimscript through the jobstart function) bindings for interacting with the library’s API. This allows actions in the editor to happen asynchronously, keeping the main editor loop clear for user input. What this means practically for users is that actions which take a long time such as grepping through large projects, generating ctags, or linting can now be done in the background without blocking the user’s ability to keep editing text.

    Let’s explore how to use the libuv lua bindings in Neovim to create useful asynchronous actions. I will cover two examples: using the pandoc program to convert a markdown file to html, and using ripgrep to search within a project. Both of these examples use the libuv bindings differently, but still illustrate well the capabilities of libuv inside Neovim.

    First Example: Convert Markdown Files with Pandoc

    Pandoc is a powerful tool for converting files into and different formats. I write many Markdown documents and it is useful to convert them into HTML for my blog. In order to do this in an asynchronous way, we will use vim.loop from Neovim’s lua library. I would recommend going through the documentation on vim.loop as you code along with this post. Let’s start by creating a basic lua function outline in our packagepath:

    -- in ~/.config/nvim/lua/markdown.lua
    local M = {}
    local loop = vim.loop
    local api = vim.api
    
    function M.convertFile()
      local shortname = vim.fn.expand('%:t:r')
      local fullname = api.nvim_buf_get_name(0)
     -- loop logic goes here
    end
    
    return M
    

    In this snippet, we first get the shortened name of the file (i.e. vimlooppost in vimlooppost.md), and the full name of the file (i.e. ~/blog/posts/vimlooppost.md). We will use both of these variables later when we spawn the pandoc process. In order to get a better understanding of what vim.loop.spawn does, let’s head over to the libuv documentation. Here we can see that it takes an options table, an onexit callback, and returns the processId and the handle of the spawned process. Using this information, let’s build out the rest of the convertFile function:

    
    -- in ~/.config/nvim/lua/markdown.lua
    local M = {}
    local loop = vim.loop
    local api = vim.api
    
    function M.convertFile()
      local shortname = vim.fn.expand('%:t:r')
      local fullname = api.nvim_buf_get_name(0)
      handle = vim.loop.spawn('pandoc', {
        args = {fullname, '--to=html5', '-o', string.format('%s.html', shortname), '-s', '--highlight-style', 'tango', '-c', '--css=pandoc.css'}
      },
      function()
        print('DOCUMENT CONVERSION COMPLETE')
        handle:close()
      end
      )
    end
    
    return M
    

    Pandoc receives as arguments the strings listed in the args table, and when the process finishes, we echo a success message and close the handle. Using this in our vimrc is as simple as:

    " in ftplug/markdown.vim
    
    nnoremap <leader>c :lua require'markdown'.convertFile()<CR>
    

    Second Example: Async Grep

    Much like in the first example, we will be relying on an external program to do the heavy lifting, ripgrep. Unlike in the first example, we don’t only want to kick off a background process, but also use the values generated in this process inside of Neovim. For that, we will need to pass a set of file descriptors to vim.loop.spawn:

    -- in ~/.config/nvim/lua/tools.lua
    local M = {}
    local loop = vim.loop
    local api = vim.api
    
    
    function M.asyncGrep(term)
      local stdout = loop.new_pipe(false) -- create file descriptor for stdout
      local stderr = loop.new_pipe(false) -- create file descriptor for stdout
      handle = loop.spawn('rg', {
        args = {term, '--vimgrep', '--smart-case'},
        stdio = {nil,stdout,stderr}
      },
      function()
        stdout:read_stop()
        stderr:read_stop()
        stdout:close()
        stderr:close()
        handle:close()
      end
      )
      loop.read_start(stdout, onread) -- TODO implement onread handler
      loop.read_start(stderr, onread)
    end
    return M
    

    Our function, asyncGrep will take a search term as an argument which it then passes to ripgrep in the loop.spawn call. After the process is spawned, we need to start reading the output into our file descriptors, which will then call the onread callback which we have yet to implement. Let’s implement the onread callback now:

    -- in tools.lua
    local results = {}
    local function onread(err, data)
      if err then
        -- print('ERROR: ', err)
        -- TODO handle err
      end
      if data then
        table.insert(results, data)
      end
    end
    

    The onread callback takes data written to our file descriptor by the ripgrep process and appends a table called results. We now want to add functionality to our asyncGrep function that will allow us to use these results to set the quickfix list:

    function M.asyncGrep(term)
      local stdout = loop.new_pipe(false) -- create file descriptor for stdout
      local stderr = loop.new_pipe(false) -- create file descriptor for stdout
      local function setQF()
        vim.fn.setqflist({}, 'r', {title = 'Search Results', lines = results})
        api.nvim_command('cwindow')
        local count = #results
        for i=0, count do results[i]=nil end -- clear the table for next search
      end
      handle = loop.spawn('rg', {
        args = {term, '--vimgrep', '--smart-case'},
        stdio = {nil,stdout,stderr}
      },
      function()
        stdout:read_stop()
        stderr:read_stop()
        stdout:close()
        stderr:close()
        handle:close()
        setQF()
      end
      )
      loop.read_start(stdout, onread) -- TODO implement onread handler
      loop.read_start(stderr, onread)
    end
    

    If you run this function, you will encounter this error message: lua/tools.lua:122: E5560: vimL function must not be called in a lua loop callback. In order for any vim functions to be called within a lua loop callback, they need to be wrapped in vim.schedule_wrap. Wrapping vim functions in vim.schedule_wrap is necessary since it schedules the callbacks to be invoked when it is safe, bridging the gap between the libuv event loop and the internal Neovim main loop. To learn more about vim.schedule_wrap, check out :h schedule_wrap. Let’s fix this error and see what our function looks like when it’s all put together:

    -- in ~/.config/nvim/lua/tools.lua
    local M = {}
    local loop = vim.loop
    local api = vim.api
    local results = {}
    
    local function onread(err, data)
      if err then
        -- print('ERROR: ', err)
        -- TODO handle err
      end
      if data then
        table.insert(results, data)
      end
    end
    
    function M.asyncGrep(term)
      local stdout = vim.loop.new_pipe(false)
      local stderr = vim.loop.new_pipe(false)
      local function setQF()
        vim.fn.setqflist({}, 'r', {title = 'Search Results', lines = results})
        api.nvim_command('cwindow')
        local count = #results
        for i=0, count do results[i]=nil end -- clear the table for the next search
      end
      handle = vim.loop.spawn('rg', {
        args = {term, '--vimgrep', '--smart-case'},
        stdio = {nil,stdout,stderr}
      },
      vim.schedule_wrap(function()
        stdout:read_stop()
        stderr:read_stop()
        stdout:close()
        stderr:close()
        handle:close()
        setQF()
      end
      )
      )
      vim.loop.read_start(stdout, onread)
      vim.loop.read_start(stderr, onread)
    end
    
    return M
    

    Let’s use this newly created async function in our vimrc:

    " in init.vim
    command! -nargs=+ -complete=dir -bar Grep lua require'tools'.asyncGrep(<q-args>)
    

    Now we can call :Grep searchTerm and get results without blocking the main editor loop!

    What’s Next?

    Having libuv bindings in Neovim unlocks a lot of potential for extending the functionality of your editor. The ability to asynchronously spawn other process can be used for linting, file watching, formatting, and much more. Free yourself from main loop blockage and try experimenting with vim.loop.

    See Also

    Mentioned around the web

    commented on Jan 14, 2020 see original

    This is a great write-up, thank you!

    commented on Jan 7, 2020 see original

    Nice! For those who don't know, there are some cool ready-to-go examples in the Nvim docs at :help lua-loop:

    • :help watch-file
    • :help tcp-server