Félix Saparelli

(about me)

a.k.a. passcod

INotify watch limit

Posted on Jun 25 ‘17

This documentation is also tagged as ,

On Linux, INotify might sometimes hit its internal watch limit. In applications like dropbox, guard, tail, my own notify, and tools depending on it: watchexec, cargo watch, alacritty… this error may manifest like this:

  • Unable to monitor filesystem Please run: echo 100000 | sudo tee /proc/sys/fs/inotify/max_user_watches and restart Dropbox to correct the problem.

  • No space left on device - Failed to watch “…”: The user limit on the total number of inotify watches was reached or the kernel failed to allocate a needed resource. (Errno::ENOSPC)

  • unable to create watcher: Io(Error { repr: Os { code: 28, message: “No space left on device” } })

  • tail: cannot watch ‘/path/to/dir’: No space left on device

  • Failed to watch /path/to/dir; upper limit on inotify watches reached!

What the manual says

As per inotify(7):

The following interfaces can be used to limit the amount of kernel memory consumed by inotify:

/proc/sys/fs/inotify/max_queued_events

The value in this file is used when an application calls inotify_init(2) to set an upper limit on the number of events that can be queued to the corresponding inotify instance. Events in excess of this limit are dropped, but an IN_Q_OVERFLOW event is always generated.

/proc/sys/fs/inotify/max_user_instances

This specifies an upper limit on the number of inotify instances that can be created per real user ID.

/proc/sys/fs/inotify/max_user_watches

This specifies an upper limit on the number of watches that can be created per real user ID.

Why this occurs

INotify watches have to be specified for each directory; there is no way to specify that a directory and all its subdirectories should be monitored using a single call or watch. Thus, libraries and applications implement their own recursive descent into directories and simply add watches to each folder they find. Some do that more efficiently than others.

Another issue is that the limit for those watches is specified per user, not per application, so the “pool” of available watches may be consumed by different applications at the same time, leading to an application that may consume a completely reasonable amount of watches failing because the rest of the pool was exhausted by something else running as the same user.

The default limit on most Linux distributions is 8192.

This is regularly blown up by watching dependency folders, like node_modules or vendor, or more extreme, by watching your entire $HOME folder.

Fix considerations

For most users, the default limit is much too low. 65536 is a good start on a more reasonable limit. Increasing to further than that is governed by more low-level details and your own concerns:

An INotify watch costs 540 bytes of kernel memory on 32-bit architectures, and 1080 bytes on 64-bit architectures. Kernel memory is unswappable. In addition to that, the library or application will use more memory to store the handle and other information (this time in swappable memory). Notify uses about 10 bytes plus the full path of the watched entry.

There are recorded instances of people using millions of watches. It just means they are okay with potentially spending gigabytes of kernel and userspace memory on that.

Adjusting the limit

You can either write directly to /proc or use sysctl (recommended):

  • echo 65536 | sudo tee /proc/sys/fs/inotify/max_user_watches

  • sudo sysctl fs.inotify.max_user_watches=65536

To do it permanently, you should put fs.inotify.max_user_watches=65536 in the sysctl configuration, which is either a new file (descriptively named) in /etc/sysctl.d/ (recommended) or a line in /etc/sysctl.conf.

Sources

Information was mostly taken from kernel manpages and sources, but this article was preceded by various blog posts and StackExchange QAs:

  • http://web.archive.org/web/20161106193425/http://blog.sorah.jp/2012/01/24/inotify-limitation
  • https://unix.stackexchange.com/questions/13751/kernel-inotify-watch-limit-reached
  • http://www.linuxjournal.com/article/8478
  • https://docs.rs/inotify/0.3.0/inotify/wrapper/type.Watch.html
  • https://stackoverflow.com/questions/535768/what-is-a-reasonable-amount-of-inotify-watches-with-linux
  • https://github.com/guard/listen/wiki/Increasing-the-amount-of-inotify-watchers

Note on dnotify, inotify, fanotify, fsnotify

Dnotify was the original Linux kernel implementation of filesystem watching. Prior to dnotify was “FAM”, which came from IRIX. Dnotify was replaced around 2008 by inotify. Then fanotify was initially implemented, but ultimately not merged and abandoned. Meanwhile, fsnotify replaced the entire backend kernel implementation, with dnotify and inotify (and then fanotify) re-written as frontends on fsnotify, eliminating duplication.

Thus, nowadays (i.e. on modern kernels) when speaking of all or any of these systems, one is talking about the same mechanism, but accessed through different kernel calls.