BinFinder

For some time now I have been trying to solve the problem of organizing my computer files. I researched many options and techniques, and yet nothing really clicked with me. So, I finally decided to make my own system! The following is a description of the system I want to build. At the end of the article, I pose some yet unsolved questions and concerns. If you, the reader, would like to help with this project, reach out on HN! I would really appreciate it!

Problem

I generate a lot of files for various projects. Some of my projects are professional software and hardware development projects, or marketing and design projects, for ongoing or one-off customers. Others are personal video, welding, making, building, etc projects. Lot's of stuff to keep organized!

To maintain sanity I keep everything I generate, along with any long term reference materials and basically any files important to me, in a single root directory on my computer, currently in ~/Files/.

Inside my Files directory I strive to maintain a tidy hierarchal structure. However, retrievability is still an issue. Allow me to illustrate with an example. Let's say I vaguely remember doing a project 2 years ago that used ESP32 (Micro Controller with BLE and WiFi) and an OLED screen. I want to find that project for reference, but I don't remember its name or the name of the client it was for. My professional projects are organized in a /client/project hierarchy, which is of no help to me in this case. What I need is a way to search for projects that have the following criteria:

1. Started in 2019
2. Related to ESP32
3. Related to OLED Screen

Sadly, in a purely hierarchical structure these relationships are impossible to represent without either links or tags. A project folder can only have one relationship with the directory hierarchy, it can have a parent folder.

Tags or links would help, but both are sporadically supported on major OSs, are not well supported by older filesystems, and are not at all supported by the cloud storage providers.

What we need is a cross platform, cross UI (GUI and CLI interface), and cross file system solution. It has to work on thumb drives with ExFAT filesystem. It has to preserve the metadata even when copied to Dropbox or Google Drive. And, it has to support Windows, Mac, and Linux.

After much research I ended up coming up dry. So it's time to make my own system. Here is what I came up with.

Solution

BinFinder is a desktop file organization tool. It organizes files into Bins and combines the following 3 major file organization paradigms into one system:

1. Bin Tree - File Bins can be organized into a hierarchical tree structure, just like in any ordinary filesystem.
2. Bin Pointers - Pointers live in one part of the tree and point to a Bin in another part of the tree. Just like Aliases on MacOS, Shortcuts on Windows, or Soft Links on Linux, but cross platform and automatically self updating! If you move the target Bin, the Pointer will find where you moved it and update itself!
3. Bin Tags - Bins can be labeled with Tags for additional sortable metadata. A Tag can optionally store a value and groups of related tags can be logically organized into tag bundles. The tags are cross platform, and work with any filesystem.

File Bin

The base file organization unit in Bin Finder is a File Bin. A Bin is just a folder with a Bin Description File (file with .bfb extension) inside the folder. The .bfb file is a text file that stores all of the metadata about the Atom in JSON format. The metadata includes:

1. UID - A unique 256-bit number. Used to find the Bin if it is moved.
2. Name - Name of the Bin. Up to 250 characters, usually set to the name of the containing folder.
3. Description - Markdown encoded multiline description of the Bin.
4. Tags - A set of tags that store information in the Key:Value schema, where Key is the name of the tag and value is the value associated with that tag.

Because a File Bin is just a special type of folder, it can contain any number of files and sub-folders inside of it, including other Bins and Pointers to other Bins!

If you need to tag or point to an individual file, it can be wrapped individually in a Bin. Since most human endeavors of any substance produce multiple files, this use case is relatively uncommon.

Bin Trees

Bins are organized into a Bin Tree. Basically a Bin Tree is just a root Bin with a bunch of other Bins, sub-folders, and files inside of it. Basically just a normal hierarchical filesystem.

You can add multiple Bin Trees to the BinFinder and search and filter across all of them. For instance, you can have your active project Bins live on your laptop and move them to the "Archive" Bin Tree that lives on an external USB Drive once you are done. Any pointers to the moved Bins will be updated with the new location of the Bins.

Bin Pointers

Pointers are just a fancy shortcut / alias / link to a Bin. What makes Pointers special is there ability to locate the Bin they are pointing too even if it has been moved or renamed.

Let's imagine you have a Project Bin in you "Active" Tree on your laptop. After finishing that project, to free up space, you move the project from the "Active" Tree to the "Archived" Tree. The "Archived" Tree lives on an external USB hard drive.

When you click on a Pointer to the Project Bin you just moved, the system will first check where it has seen the bin last. If the system can't find the Bin in that location, it will quickly rescan all of your Trees. If the Bin is found in another location, the system will update the pointer and open the Bin. If the Bin CANNOT be found, the system will inform you accordingly with a message similar to: "Can't Find Bin X! It was last seen inside 'Archived' Tree, which is not available currently." That's a helpful reminder to you that perhaps you need to plug in your USB SSD.

In practice a Pointer is nothing more than just a JSON encoded text file with .bfp extension. It contains the following information:

1. UID - The UID of the Bin the pointer is pointing too.
2. Name - Cached from Target Bin.
3. Description - Cached from Target Bin.
4. Tags - Cached from Target Bin.
5. Path - Last system path the target Bin has been seen at.
6. RootUID - The UID of the root Bin under which the target bin last resided under.
7. RootName - The name of the root Bin under which the target bin last resided under.
8. Sync File Name - True or False. True means update the Pointer filename if the filename of the target Bin changes. False means keep the original Pointer name. This is useful if you want the pointer to have a different name from the target Bin. Default is True.

Bin Tags

BinFinder treats tags as basically additional metadata about the Bin. For instance, you can tag a bin with things like:

1. Client.Name : X Corp
2. Client.Contact : Dr. X from X Corp
3. Client.Email : drx@xcorp.io
4. Budget : 48,000
5. StartDate: 2019-03-12
6. Finished

In the list above you'll notice that the first 3 tags are prepended with 'Client'. Those tags are automatically organized into the 'Client' tag bundle, and are grouped together in the UI for convenience.

The last tag, Finished, has no value. To be more accurate, it has no value specified, so a default value of True is auto assigned. Absence of a given tag is equivalent to having a False value for that tag.

Example Bin

To bring it all together let's imagine a Bin for the project we used as an illustration in the 'Problem' section above. The bin lives at ~/Files/Clients/X Corp/Super BLE. Inside the bin path there is an INFO.bfb file. That's the file that turns a simple old directory into a fancy Bin! Here is what it looks like:

xxxxxxxxxx
{
  "uid": "4d9789224f246fd89270a88d40eaa151",
  "name": "Super BLE",
  "description": "A super cool project involving ESP32 and BLE and Magic!....",
  "attributes": [
    ["Project.Started", "2019-09-01"],
    ["Project.Ended", "2020-01-10"],
    ["Type", "Hardware"],
    ["Type", "Firmware"],
    ["Type", "ESP32"]
  ]
}

To make it easier to find this project in the future I may also drop some Pointers to the Bin in various other places:

1. ~/Files/Projects/Hardware/ESP32/Super BLE.bfp
2. ~/Files/Projects/Hardware/OLED/Super BLE.bfp
3. ~/Files/Projects/Hardware/Firmware/Super BLE.bfp
4. ~/Files/Projects/By Year/2019/Super BLE.bfp

And here is what we would find in one of those pointer files:

x
{
  "uid": "4d9789224f246fd89270a88d40eaa151",
  "name": "Super BLE",
  "description": "A super cool project involving ESP32 and BLE and Magic!....",
  "attributes": [
    ["Project.Started", "2019-09-01"],
    ["Project.Ended", "2020-01-10"],
    ["Type", "Hardware"],
    ["Type", "Firmware"],
    ["Type", "ESP32"]
  ],
  path: "~/Files/Clients/X Corp/Super BLE",
  rootUid: "3e0184a0de6ea1ed8ef94ac6de1a633b",
  rootName: "Files"
  syncFileName: True
}

When you double click on one of those .bfp files, they are automatically opened with BinFinder. BinFinder looks for the Bin and opens it in your default file manager. In practice, the whole process should function just as if you clicked a shortcut.

Unresolved Questions

I have most of the UI and workflows worked out in my head pretty well, but I find myself in a "paralysis by analysis" situation. Here are some of the questions I am struggling with in my head at the moment. Feel free to comment on HN and give me some ideas:

1. GUI vs. CLI - I want to support both. But should it be a CLI first application that's then wrapped with GUI? Or perhaps it's better to build it as a GUI first application, and add a CLI interface to it? Or just have a core library, and use it in independent CLI and GUI tools? I never wrote a cross-platform tool, so I am not sure what's the best approach.
2. Language / UI Library Selection - Here the choices appear to be Electron with JS vs. QT plus either Python, C++, or Go. I like the idea of Go for a good balance between speed of dev and speed of execution. Python might be ok, but given the nature of the app, Electron feels a little too heavy, especially for an app that you might be opening 30 times a day for a few seconds. If you have any guidance here, or perhaps know of a better alternative I have overlooked, please let me know in the HN comments. I am looking to learn some new tech while doing this side project, so anything goes, as long as it's a good idea :)
3. Use Cases - I have though this through in terms of my own use cases. But I am N of 1. Can you give me some ideas about how you would use this tool? What would you like to see in it?

Finally, if you would like to help on this project, that would be really cool! Please feel free to reach out if you think it's a good idea, and let's build it together!