* InvokeFuse: A FUSE Filesystem for Browing InvokeAI Output

The [[https://github.com/invoke-ai/InvokeAI][InvokeAI User Interface]] is very nice and slick, but sometimes you want to access your outputs from the command-line or from another tool, and the trouble is that InvokeAI just throws all its outputs in one big directory.  There /is/ structure there, and indeed InvokeAI presents the pictures organized into "boards" which you create, and so on, but that structure is held in InvokeAI's database and not reflected in the actual directory structure.

InvokeFuse is a [[https://en.wikipedia.org/wiki/Filesystem_in_Userspace][Filesystem in Userspace]] (FUSE) script, written in Python, which attempts to give you access to your big =outputs/images= directory in a more organized form, broken down by "board" and also by what model or prompt you used to create the image, or the date you made it on.

It sort of accreted features here and there, and doesn't necessarily confirm that you're accessing things the way it expects, so it is fairly easy to wind up trying to access a path that gives you unexpected or erroneous results, but if you keep within its limits you should get expected results.  In /no case/ should InvokeFuse ever actually mess up your InvokeAI database; it does not write anything there.

Please /read these docs/.  At least the [[*Directory Structure][Directory Structure]] section.  They aren't that long, and things will be all confusing and won't work as you expect until you update your expectations with knowledge.

** Running InvokeFuse

It's probably simplest just to run it from your InvokeAI installation directory:

: $ python /path/to/invokefuse.py -o dbfile="$PWD"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/databases/invokeai.db ./mnt

(You should create a =mnt/= subdirectory there to mount it.)  Basically, you need to give it a path to the =invokeai.db= file in the =dbfile= option.  Make sure it is an /absolute/ path.  FUSE does not do so well with relative paths (but InvokeFuse will convert the =./mnt= mountpoint to absolute form.)  InvokeFuse presumes that the images are present where InvokeAI generally stores them relatively to the database, that is, in =../outputs/images= relative to the directory where the database is.

See also the [[*Options][Options]] section for how to proceed if your files are someplace else, and for other important behavior-changing options.

*** Unmounting

On Linux, use the =fusermount -u= command to unmount the directory.  Mac and WSL I think use the normal =umount= command (note spelling!)  You won't be able to unmount the directory if any process is still using it.  This includes having files open there or even having a directory in it as its current working directory.  The =fuser -m= command (may not exist on WSL) can tell you which processes are still using the mounted filesystem.

** Directory Structure

The basic directory structure as presented by InvokeFuse works like this (I started to draw a diagram but it just got ridiculous.)  Note that the structure is pretty rigid, and you are expected not to skip levels.  To see all the images in a board, for example, you need to look at something like

: mnt/<boardname>/models/ALL/ALL/

See also below, though, about [[*Unlisted Entries][Unlisted Entries]].

*** Level 1: Boards
The top level, directly under the mount point, consists of directories named for each of the "boards" you have created in InvokeAI.  There is also a special directory called =UNSORTED/= for the "Uncategorized" section, where InvokeAI puts images that you haven't assigned to any of the boards.  (See the [[*Caveats][Caveats]] section for warnings about board names.)

**** Level 2: =models= or =prompts=
The second level is fixed, and always contains the exact same two "subdirectories."  I was sort of hesitating between how to order things and wound up doing both.  The basic idea in my mind when I was starting out was that I wanted to access images by the (positive) prompt used and by the model used to generate them, but I wasn't sure which one of those was primary.  So I ended up doing both, and you can choose to have paths that look like

: <boardname>/models/<modelname>/<prompt>/003b9609-ce8d-4671-b59c-052859349a01.png

*or* ones that look like

: <boardname>/prompts/<prompt>/<modelname>/003b9609-ce8d-4671-b59c-052859349a01.png

So an image in board =myArt= generated with juggernaut with a prompt "happy people" appears both as

: myArt/models/juggernaut/'happy people-FS+YQH1pnb5W'/003b9609-ce8d-4671-b59c-052859349a01.png

as well as

: myArt/prompts/'happy people-FS+YQH1pnb5W'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/juggernaut/003b9609-ce8d-4671-b59c-052859349a01.png

***** Level 3: Modelname or Promptname subdirectories

At this level, one below choosing "models" or "prompts", you have subdirectories named for each model used to generate images in the given board, or subdirectories named for each prompt used to generate images in the given board (depending on which you chose in the parent directory.)

In the "models" case, it is pretty straightforward, as models usually have short and "well-behaved" names.  There will be a directory called =stable-diffusion-v1-5/= and one called =stable-diffusion-xl-base-1-0/= and one called =juggernaut/= and one called =realcartoonXL-v6/= and so on, for all the models that you've used to generate images in that board (so some of the examples I just gave might not be present.)  There will also be a special subdirectory named =NO MODEL/= which holds all images for which there is no model in the metadata (this happens for images generated by image-to-image processing, workflows, etc.)  The =NO MODEL/= subdirectory /always/ appears, whether or not there are actually any images in it.

In the "prompts" case, there is similarly a =NO PROMPT/= subdirectory for images which do not have a prompt in their metadata (this is usually caused by the same things that cause there not to be a model; most of the time, in fact, it seems that the set of "no model" images is the same as the set of "no prompt" images.)

But using prompts as directory names raises some awkwardness, because prompts can be very long and can contain funny characters and that can make them hard to read and type.  So I just truncate the prompt and hope that the /start/ of the prompt is meaningful to you.  Actually, first I preprocess the prompt: I strip out anything that isn't an ASCII letter, number, or underscore (=[A-Za-z0-9_]=), replacing any string of such characters with a single space, and then I strip off leading and trailing spaces.  Then I use the first 20 characters of the resulting string, hoping that is meaningful enough to you.  Because that preprocessing could easily result in distinct prompts boiling down to the same prefix, I also add a disambiguating suffix (after a =-= character), made of a 9-byte hash of the whole prompt (before processing) encoded into ASCII.  I use a short hash so that the suffix isn't too long, and because I figure even with a "small" hash space (2^{72}), it's very unlikely that there will be collisions and it isn't like we're protecting valuable secrets that need foolproof cryptographic hashing.  That's why the examples above showed prompts with that string of nonsense characters at the end: those were the hash.

****** Level 4: Promptname or Modelname subdirectories

This level is (almost) exactly the same as the one above it, except that it does the other one of the two initial choices.  So if you're in the =models/= branch and the level above this one was directories of model-names, this will be a directory of (truncated and hashed) prompts that were used in this board for that model.  And if you're in the =prompts/= branch and the level above this one was directories of prompts, this level will be directories of model-names that were used for the given board with the given model.  In the same way, there will be a =NO PROMPT/= or =NO MODEL/= subdirectory here.

There is one small difference, though.  Only in the case where you're in the =prompts/= branch and so the parent directory chose the prompt, in its abbreviated form, there will also be a file here called =PROMPT.TXT= which contains the actual text of the prompt, unprocessed.  This way you can check if you're in the right place, in case the preprocessing obscured the distinctions between prompts.  This is a normal file, you can read it with all normal tools.

******* Level 5: Image links

Finally, this is the bottom layer.  At this layer reside all the images you created in the given board, with the given model, with the given prompt.  The images exist here as /symbolic links/ to the actual images in the =outputs/images/= directory of InvokeAI, to the extent that matters, but for most intents and purposes they'll behave as the images, and you can display them etc.

If the =PROMPT.TXT= file was not in the parent of this directory (i.e., you were in =models/= subtree and the parent of this directory was the abbreviated prompt), it will be down in this one.

** Unlisted Entries

Here's where things get a little strange, if you're used to navigating a normal filesystem.  There are files and directories that you can access which are *not* listed when you do the listing of a directory.  They aren't hidden, just when you ask what's in a directory, they simply aren't there... and yet if you know their names you can access them anyway and things will happen.  This might limit their usability in disk-browing apps, unless you can type in the name directly.

*** =.META= Files

Down at the bottom level of the directory tree, where the symlinks to images reside, each image also has a not-shown ".META" file, which contains the image's InvokeAI metadata (that is, the metadata in the InvokeAI database, which is distinct from any metadata that may be stored in the image file itself).  The filename is the same as the image's filename, except with =.META= (all caps) appended.  So image =art/models/juggernaut/folks-RT85jDq9YdSl/003b9609-ce8d-4671-b59c-052859349a01.png= has an invisible companion file =art/models/juggernaut/folks-RT85jDq9YdSl/003b9609-ce8d-4671-b59c-052859349a01.png.META= which you can read to see its metadata (in JSON format).

*** =dates/= Subdirectories

The "bottom" of the tree structure described above isn't really the bottom.  There's an invisible directory called =dates/= down there which you can list.  So you can

: $ ls mnt/art/models/juggernaut/folks-RT85jDq9YdSl/dates/

and you'll see a bunch of subdirectories with names like =2024-02-29/=, for all the dates on which you generated images in the given board with the given model and prompt.  And in each of those subdirectories you'll find the symbolic links to the individual files (and their unlisted =.META=) files, as you might expect.

If you want to browse by date but not limit yourself to a particular board/model/prompt combination, you can use the [[*=ALL/=][ALL]] wildcard, below, to avoid limiting yourself at one or more of these levels.

*** =LIKE/*= Subdirectories

This feature perhaps has a little more potential for blowing up than some others.  It's meant to help with the problem of the truncated prompts not including the stuff you want.  At the =<board>/models/<model>/= level, instead of going into one of the truncated prompt subdirs (or =NO PROMPT/=), you can list a directory called =LIKE/%cute%/= or something like that.  This will give you a listing of prompt subdirs for all prompts that are "like" the name you gave for the sub-directory of =LIKE= (i.e., the =%cute%= part), using [[https://www.sqlite.org/lang_expr.html#like][the SQLite "LIKE" operator]], thus using =%= and =_= as wildcards, etc.

Unlike the usual prompt subdirectories, when =LIKE/<pattern>/= is used, the names of the subdirectories are *not* truncated and hashed, but are presented straight-up, as they are, full length.  So the directory names here can potentially be very long, or contain all sorts of punctuation or other characters.  See below under [[*Special Names and Naming Restrictions][Caveats]] for a potential bug that might come about.

Inside these subdirectories you'll find the usual =PROMPT.TXT= file and the symbolic links to the relevant images, etc.

Currently, there does not seem to be a way to combine =LIKE/= with =dates/=.

Using =LIKE/= without giving a pattern after it essentially ignores the =LIKE/= and acts like you're just getting the usual truncated prompts.

See also under [[*Lack of Error/Structure Checking][Caveats]]: using =LIKE/*= in other places in the directory tree may cause weird things to come about.

*** =ALL/=

Any place in a path where you are normally expected to have some value that limits to a particular board or model or prompt, you can substitute the special word =ALL= (all-caps) to sort of "skip" that level and take all of the options.  So =ALL/= at the top level means all of the boards, =ALL/models/ALL/= contains subdirectories for all the prompts you used for any board and any model, and so on.  (Note that the second level, "models" or "prompts", cannot be skipped with =ALL/=, because it would make no sense as the two trees are redundant to one another.)  It is important to use =ALL/= to skip levels (and not try just omitting them to put =dates/= or =LIKE/= someplace) because leaving it out can confuse InvokeFuse and you can wind up with image-names being listed as directories and so on.  See under [[*Lack of Error/Structure Checking][Caveats]].

** Options

As with any mount type command, options for InvokeFuse go in the =-o= option:

: python3 ./invokefuse.py -o dbfile="$PWD"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/databases/invokeai.db,foreground,imagesdir=/otherdisk/images,real_files ./mnt

(/not/ like most applications where options are given as =--option= or something).  The options follow the =-o= option (after a space), and are separated by commas /but not spaces/, otherwise the shell will consider it a separate command-line argument and InvokeFuse will probably try to interpret it as the mount-point.  Any spaces in the arguments (/e.g./, in pathnames) must be quoted, hence the use of double-quotes around =$PWD= in the example above.  (There isn't a good way to handle pathnames that contain commas; just avoid those.)  Options are either standalone or take a value with an ~=~ sign.

*** Options supported

 + =dbfile=PATH= :: The one mandatory option.  Provide the path to the =invokeai.db= file that you are using.  This must be an /absolute/ path (not a relative path).  Make sure you quote any spaces that might be in the pathname.

 + =foreground= :: Standalone FUSE option.  If provided, the FUSE system runs in the foreground and does not fork off immediately to become a background process.  This can be useful for debugging, since normal FUSE processes can't write to a visible standard output or error stream, so debug statements aren't helpful, and even exceptions don't show on the console.  But when running in "foreground" mode (even if you use =&= on the command to run it in the background like you can with any command), at least you can see the exceptions and any print statements, etc.

 + =allow_other= :: Standalone FUSE option.  Normally, a FUSE-mounted filesystem is accessible /only/ by the user who mounted it (not even the root user can access it.)  The =allow_other= option allows other users to see and read the filesystem.  This may be important especially if you are browsing over a network connection, etc.  For security reasons, FUSE can only do this if it is explicitly allowed in the system FUSE configuration file, =/etc/fuse.conf=.  You'll need to make sure a line =users_allow_other= is in that file.

 + =allow_root= :: Standalone FUSE option.  This is like =allow_other=, except it only allows the root (administrator) user access to the mounted file-system.

 + =rootdir=PATH= :: The directory that is the root of your InvokeAI installation.  If not provided, it is presumed to be one level up from the directory where the =dbfile= is.  That is, if your dbfile is =/mnt/d/invoke/databases/invokeai.db=, then rootdir is =/mnt/d/invoke=, which is what you would expect.  If you specify this, it should be an absolute path.

 + =imagesdir=PATH= :: The output directory where all your images are located.  If not provided, it is presumed to be =<rootdir>/outputs/images=, which is how InvokeAI organizes things normally.  This is useful if you store your images on a different disk or something for space purposes.  As with the other paths, this should be an absolute path.

 + =real_files= :: Standalone option.  Operate in "real files" mode.  Normally, the actual images are presented as symbolic links to the actual image files.  But it seems that Windows systems don't process symbolic links well, so the =real_files= option makes InvokeFuse try to present the images as actual files and handle reading from them, etc, so as to make it usable with Windows.  This may cause some small performance degradation, and there might also be other issues with it that we don't know about yet.

 + =incl_all= :: Standalone option.  Make the [[*=ALL/=][ALL]] directories show up in the directory listing.  So your list of boards or models or prompts will /show/ the =ALL= subdirectory, rather than it just being available without being listed.

 + =no_prompt= :: Standalone option.  Don't list the "PROMPT.TXT" file (though it is still available).  This way there'll be nothing but images in your lowest level, nothing but directories in your prompt level.

** Caveats

This project was sort of thrown together, and there is a lot of error-checking and due diligence with inputs that is /not/ being done.  The user expected to use things correctly and not to provide opportunities for mishaps.

*** Special Names and Naming Restrictions

There are several exceptional names (and a format) which InvokeFuse treats specially, and I'm counting on the user not to have boards or models with those names.  They are all defined as globals in the invokefuse.py file, so if you really need to you can just change their definition once.

 + =UNSORTED= :: This is the reserved name for the "Uncategorized" board, of files not in a board otherwise.  Don't use this name (all-caps) for another board.

 + =NO MODEL=, =NO PROMPT= :: These are reserved names for directories containing images with no listed model or prompt, respectively.  Using =NO PROMPT= as a prompt could probably only cause problems if it wound up being made into a directory under =LIKE/*=, and that is pretty unlikely.  It's also unlikely any model will be named =NO MODEL=, but if it ever happens, be careful!  Even so, be careful not to use these as board names.

 + =LIKE= :: This is reserved for the [[*=LIKE/*= Subdirectories][=LIKE/*= construction]], in the prompts subdirectory.  It's unlikely to conflict with anything else, since even if you used =LIKE= as a prompt, it would be replaced by its hash-appended replacement (=LIKE-DpmyJsCLx1DM=) in the directory, so it should not cause problems, except maybe if it occurs as a result of another =LIKE/*= construction.  And similarly, don't use this as a board name (you can use it if you don't use all-caps.)

 + =ALL= :: This is the reserved word for "all" models or boards or prompts.  Naming a board "ALL" (all-caps) would make it impossible to select just that board, since it would be confused for selecting all of them.

 + =dates= :: This name is reserved for the [[=dates/= Subdirectories][=dates/= subdirectory]], and using it as a board-name could be problematic.  Note that unlike the other reserved names, this is lowercase.

 + =/^\d{4}-\d{2}-\d{2}$/= :: This is what date subdirectories look like: four digits, a hyphen, two digits, a hyphen, and two more digits.  If you use board-names that look like this, it will cause problems.  (Because of the sloppy way in which things are done, the fact that it's way up at the top level and not where a date would be expected does not help, and any of these "special forms" can be a problem as a board name.)  This is actually a plausible way you might already be naming your boards; maybe this needs to be changed to have a prefix or suffix like =_= or something.  We'd have to change it both in the global variable and where the dates subdirs are generated.

 + The =/= Character :: Probably the most important remaining restriction is the use of the =/= character.  It's the only character (apart from the NUL character) which Linux forbids in a filename or directory name.  A board or a model with a =/= in the name will break things, and is not currently checked for.

   Ordinarily, prompts are stripped of bad characters, but if you use =LIKE/*= and one of the prompts selected has a =/=, that could cause problems too.  Right now, the =/= character is replaced by a space in all prompts, /everywhere/, so as to avoid the problem.  Especially if you're using auto-captioning, you might well have slashes in the prompts, and that causes I/O errors when reading the directory.  This opens the door to a different bug, namely that you could have two distinct prompts which differ only in slash versus space, which InvokeFuse would treat as identical, and you'd never see the slash even in the =PROMPT.TXT= file (though the images generated thereby should still be there.)  This seems to be a pretty remote possibility, so for now choosing to live with that bug, as being better than the limitations caused by leaving the slash in place.

There are some other globally-defined fixed strings, but I don't think they would cause problems even if you use the names elsewhere:

 + =models=, =prompts= :: These are the fixed names for the second level, but since they only matter at the second level, there should be no issues with them even if you name a board like that, etc.

 + =.META= :: This is recognized as the suffix for metadata files, but since that only matters when you're actually trying to read a file, it shouldn't be an issue elsewhere, where your names generally become directories.

*** Lack of Error/Structure Checking

The structure of a path to an image is =<board>/models/<model>/<prompt>/<image.png>= or =<board>/prompts/<prompt>/<model>/<image.png>=.  The special =dates/= subdir belongs at the bottom of either of those two structures, in place of the image-file, and it will have subdirs which will have images in them.  =LIKE/*= should only be used as =<board>/models/<model>/LIKE/<like_expr>= (and probably not in the =prompts/= subtree.)  Metadata files should be accessed with the same path as an image, plus the =.META= suffix.  =PROMPT.TXT= files should be accessed in the places explained above.  Trying to put special things like =dates/= anyplace else is not necessarily checked for or handled correctly and may result in [[http://www.catb.org/jargon/html/N/nasal-demons.html][nasal demons]].

*** Dates and Timezones

At the moment, the dating used by the =dates/= subdirectory is not timezone-aware, so the images which it says were created on =2024-01-05= are those which were created then /in UTC/, not local time.  So some of the ones you thought you made on that day might show up in the previous or next day's directory, depending on your timezone.

** Use with Windows
InvokeFuse was developed on a Linux system, using FUSE capabilities which Windows doesn't really support natively, so you might have some issues with using it.  I have heard that it /can/ be used with the [[https://learn.microsoft.com/en-us/windows/wsl/install][WSL environment]], which essentially provides you with a Linux-like environment in Windows.  You'll need to =sudo apt install fuse= in your WSL environment to make sure you have FUSE support installed.  When the docs here call for an "absolute path", you probably are going to need to use the WSL-style path (=/mnt/d/invokeai/databases/invokeai.db= or something).  To access the filesystem from outside WSL, you'll need to make sure that =allow_others= is specified, and that it's permitted in your =/etc/fuse.conf= file, so that external tools can see it.