[Feature] Meta FR for asset organization

[RobertHeim]

I have searched the existing feature requests to make sure this is not a duplicate request.

• Yes

The feature

Contents

• Introduction
• Requirements
  • Core Features
  • Open design decisions
  • UI / UX
  • Other
• Current Legacy Solution
• Alternative Solutions
  • digiKam
    • Hierarchical tags

Introduction

Hi!

Following up on the extended discussions from #1651 I chatted with Daniel in Discord to create this Meta Feature Request to move things forward in a more structured way.

This is my proposal for the next steps of which this post addresses the first two steps for an initial review by the immich team in this FR discussion.

1. Summarize FR from [Feature]: Discussion - Use tags in replacement (or additionally to) albums #1651
2. Review digiKam implementation as it is often mentioned to have a nice tagging system.
3. Review this FR and complete the list of requirements with things that we should think of when working on the design doc (4).
4. Formulate a design doc
   1. core vision of the tag system design and how it integrates with all the different FR (esp. data model, core API, and giving the different features a sane bounded context)
   2. while keeping out-of-the-box UX simple and sane
   3. define a roadmap to keep a first version as simple as possible but aligned with future goals
5. RFC on the design doc with a deadline
6. Start implementation on the next iteration/deliverable from the roadmap + review + deploy + get user feedback + repeat

Requirements

I reviewed #1651 and extracted the requirements/features and gave them some structure. However, I did not put every little detail but tried to keep some level of abstraction.

Core Features

• Tag definition
  • multi-word tags / allow spaces
  • source of tag (user, AI, import, ...)
  • type (user, album, face, location, cam-model, ...)
  • editable/lock
  • icon or thumbnail per tag (e.g. face)
  • shortcut
• Hierarchical tags/folders / multi-level tags
  • see digiKam
  • represent folder structures (e.g. of external libs?)
• Create tags
  • Manual tagging (user managed)
  • Smart tags / AI tagging (objects, faces, locations, ...)
    • smart-tag / suggest tags based on similar images (or faces, location, objects, ...)
      • do not suggest a tag if it formally has been denied on the asset
    • given a tag, find similar images that may also want the tag
    • location-specific tags
      • enable grouping photos without location data with such with location data
  • provide tag during mass upload
    • tag by folder/path name(s)
• Import/Export tags (embedded/EXIF/IPTC, sidecar, digikam, ...)
  • important for portability
  • support popular other tools (exif, digiKam)
  • maybe differentiate external libs and internal libs
  • maybe admin option to always save to files (cumbersome for bulk edits as we loose ACID guarantees)

Open design decisions

• define if albums are conceptually tags or keep existing as their own concept
  • tags = "smart albums"
  • tagging albums
    • auto-include the union of the tags of all images in the album
    • add/remove all images with a specific tag
    • add/remove tag from all images in the album
  • nested albums
    • grouping images within an album by tags

UI / UX

• search / filter by tags
  • filter logic ("a" and "b" and not "c")
    • syntactic sugar (e.g. "-a" for "not a")
  • timeline
  • album
• bulk edit tags
• URL per tag
• tag cloud (often used tags displayed bigger)
• Review / Inbox feature
  • auto-tag all new images as "inbox" to be reviewed and properly tagged by the user
  • integrate with NSFW feature
  • show images without "keep" tag, swipe left => delete, swipe right => tag "keep"
• "bookmark" icon over thumbnail to indicate if an asset is in any album
• undo
• shortcuts
• find tags not assigned to any image

Other

• permissions based on tags
  • share tag (permissions to view images with this tag)
  • enforce privacy (removes asset from all shared items)
  • NSFW / hide elements from default views with a specific tag / safe-browsing / allow-list/block-list
    • global default filter based on tags
  • enable / disable editing of an asset
• favorites/bookmarks based on tags
• quality rating based on tags (stars, color)
• Multi user (i.e. unique(userId, tag) and not unique(tag))

Current Legacy Solution

The current legacy solution consists of the TagEntity and related CRUD API without any UI. But we probably need to migrate existing data into the new design.

@Entity('tags')
@Unique('UQ_tag_name_userId', ['name', 'userId'])
export class TagEntity {
  @PrimaryGeneratedColumn('uuid')
  id!: string;

  @Column()
  type!: TagType;

  @Column()
  name!: string;

  @ManyToOne(() => UserEntity, (user) => user.tags)
  user!: UserEntity;

  @Column()
  userId!: string;

  @Column({ type: 'uuid', comment: 'The new renamed tagId', nullable: true })
  renameTagId!: string | null;

  @ManyToMany(() => AssetEntity, (asset) => asset.tags)
  assets!: AssetEntity[];
}

export enum TagType {
  /**
   * Tag that is detected by the ML model for object detection will use this type
   */
  OBJECT = 'OBJECT',

  /**
   * Face that is detected by the ML model for facial detection (TBD/NOT YET IMPLEMENTED) will use this type
   */
  FACE = 'FACE',

  /**
   * Tag that is created by the user will use this type
   */
  CUSTOM = 'CUSTOM',
}

Alternative Solutions

The current solution lacks a UI and any of the advanced features. This section explores alternative solutions.

digiKam

A lot of users seem to be very happy with the digiKam tagging system.
Docs: https://docs.digikam.org/en/main_window/tags_view.html
code: https://invent.kde.org/graphics/digikam/-/tree/master/core/libs/tags

Their system consists of a simple tag definition together with some reserved internal tags ( https://invent.kde.org/graphics/digikam/-/blob/master/core/libs/database/coredb/coredbconstants.cpp ) and accompany tables for the actual tag assignment, properties per tag and properties per assignment.

Tag
    * id (int, primary)
    * parentId (int, foreign?, 0 for toplevel)
    * name (text, name of the tag)
    * icon (int, foreign, optional image id of the image used as icon for the tag)
    * iconkde (text, optional name of the icon for the tag)
TagImage
    * imageId
    * tagId
TagProperties
    * tagId
    * property (text)
    * value (text)
ImageTagProperties (for marking the region in the image related to the tag, e.g. faces)
    * imageId
    * tagId
    * property (text, property of the region tagged in the image, e.g. `autodetectedFace`)
    * value (text, defines area/range of the region covered, e.g. `<rect x="1736" y="735" width="209" height="176"/>`)
TagAction
    * id
    * parentId (0 for toplevel)
    * type (AssignTag, CreateNewTag) 

digiKam export: They clean all tags from the image before writing the ones from DB. They warn before that action that user might loose tag data if they did not read tags before (by calling Read Tags from Image) and that the operation can take long (depending on collection size).

Hierarchical tags

digiKam uses "/" as delimiter for hierarchical tags. "," is used as delimiter for creating multiple tags.
Hierarchies come with the problem of categorization (an item only be able to live within the bounded context of the parents). This requires allowing non-unique tag names (e.g. Animal/Domestic/Cat and Animal/Wild/Cat) which can lead to problems when assigning (user does not know which "Cat" tag is used). The better approach is to get rid of hierarchical thinking as much as possible.

Platform

• Server
• Web
• Mobile

[briandipalma]

Animal/Domestic/Cat and Animal/Wild/Cat are two different tags. It's just that / allow a tree structure to navigate the tags but when you store and see the tag value you see all of it and it's a single value.

[platima]

Hey folks,

Half asleep but just jumping in with my points from the other discussion, and one I was having with the PhotoPrism team. Thanks heaps for doing this too Robert, greatly appreciated.

Aside from digiKam, which we tested and quite liked besides it's lack of multi-user and web UI, we also tested PhotoPrism and that was getting REALLY close, but they have some other priorities it appears.

For our use case, which like many others is purely all about managing external libraries, we have some wants/wishes for any platform we use. We're happy to throw some money at it to see it happen too. Here is a quick list of things it looks like you can consider for this FR definition. Apologies if some were in discussion 1651 and already pulled across:

• Viewable in multiple formats, eg in nav bar list when you expand 'Tags', in folder view, in list view, with details ,etc.
• Ability to turn off any auto tag if enabled eg YOLOv5. PhotoPrism has this with a manual YAML change plus also turning off geolocation, not as an option on it's own.
• Ability to script tags on import / ingest, eg IF [filename] [ends with] "Edit" THEN [add tag] "edited", or IF [meta.camera] [contains] "X2" THEN [add tag] "T".
• Case insensitive searching (PhotoPrism did not have this)
• Ability to select multiple tags at once for management, such as bulk removing tags no longern eeded, and bulk-adding to images
• Optional lazy string matching when searching tags, eg tag:dirt

I think your "Google Search" style logic idea is great too, such as a search of filetype:png AND tagged:ants AND (NOT tagged:bulk)
There's the optional format you hinted at though, whch would be something like tags["ants", -"bulk"] or similar. Depends how the logic is handled I guess.

Should all these features, once collated, perhaps go into a voting system for people to prioritise, then the final votes give us a good idea of what feature priority would appease the most?

BR

[cejote]

Paperless (https://github.com/paperless-ngx/) is a quite famous FOSS solution to manage all kinds of documents. They also have a very nice tagging system (adding tags to docs/groups of docs, removing tags, searching, etc.) which could be a helpful resource in addition to +digikam and +photoprism.

[bo0tzz]

Great overview, thank you! As I see it, there are two large decisions to be made:

• Do we support nested/hierarchical tags?
• Do we change albums to be tags under the hood?

[briandipalma]

What extra work is required for hierarchical tags? A UI that's a tree instead of a flat list? I can't think of anything else apart from that.

[bo0tzz]

The database schema for hierarchical tags is a fair bit more complex, with a higher chance of slow queries and such.

[AngelaDMerkel]

I would like to argue that folders and tags represent different things to the user and serve differing purposes. A folder is a quick way to access groups of images which might have nothing to do with any feature of the image. Lots of people with film photos categorise them into folders by their roll once developed, but this has nothing to do with a reasonable tag. The same can be said of more abstract ideas including folders for certain groups of friends. A tag can't neat fill that role, because the purpose of the folder isn't descriptive like the tag, but instead prescribed by the user.

[briandipalma]

I'm really struggling to see how tags can't be used for those purposes. You can have multiple tags on an image so you can have roll:56 and Berlin or however many tags you want on a picture.

[AngelaDMerkel]

@briandipalma You could use tags for the purposes of cataloguing rolls, but I think that tags ought to be descriptive to prevent diluting their meaning, and therefore don't fit neatly into arbitrary categorisations like "bullshit photos for the homies" or "those times we went drinking" and in such cases a folder suits better. It's purely my preference based on a typology of organisation.

[bo0tzz]

We had a discussion on this with the core team yesterday, and decided to move forward with basic hierarchical tagging support. We will write a design doc for a basic server/database implementation of hierarchical tagging. After that groundwork is implemented, everything else like UI, searching, sharing, etc can be implemented in followup PRs.

[RobertHeim]

@bo0tzz
Thanks for discussing it with the team. I agree that most of the features can easily build on top of the core in a decoupled way (i.e. unidirectional dependency).
I would vote for keeping the hierarchy explicit, i.e. make it a parent-child relation in the DB. This will be much easier to implement and maintain than implicit delimiters. Immich uses a relational DB, this is what it is made for.

@platima it's more complex, esp. for querying. The data model design will have a huge impact on more advanced features build on top of it. The more explicit relationships are defined, the easier it will be to maintain. Otherwise the cognitive load is higher and it is cumbersome to work on a higher level of abstraction, because the data model is unusual. Tree data structures and graphs in general are very well known in computer science. We are used to working with such.

[platima]

@RobertHeim very fair! Thank you for clarifying and not just dismissing.

[briandipalma]

@RobertHeim I feel this maybe over complicating hierarchical tags, I don't think there needs to be a parent/child relationship etc. The / delimiter is purely for the UI, when you click on each level of the tree you show only images that match that exact path, so that means the X/Y/C tag is one single tag and there is no need to worry about X/Y or X and linking them all together etc.

Clicking X/Y/C will only show images with that exact tag. No difference between that tag and a single tag like O to the database.

[iainhallam]

@briandipalma But what happens when you click the X/Y tag? Do you see images tagged with X/Y/C as well? I'd expect to, if the hierarchical tags are working correctly, and ditto for clicking X, which I'd expect to show any children as well, not just the parent. I've seen interfaces where you can just select the parent, and not have children included, but that's not been the default in my experience.

[RobertHeim]

@briandipalma It has proven effective to model the data as the semantics are defined. Having different semantics than the data structure is an anti-pattern as it requires implicit knowledge and increases cognitive load and thus would be hard to maintain by design.

[briandipalma]

But what happens when you click the X/Y tag? Do you see images tagged with X/Y/C as well? I'd expect to, if the hierarchical tags are working correctly, and ditto for clicking X

You don't in Digikam, it's never bothered me, makes perfect sense how it works. I click "December" I only see images with that tag. None of the images with a tag below.

Tags are lists of strings, it's an array of strings. The fact that some of those strings have / in them doesn't need to impact the data model.

[charles-997]

User-choice is great 👍

[briandipalma]

@xttlegendapi Being able to select only X/Y.

[OPerepadia]

I think that when you select a parent tag, you naturally expect all its child tags to be included. That is the point of hierarchical structure.

If you only want to select x/y without including z, perhaps your hierarchy is not quire right, and z should not be under x/y.

But I agree, having an option to control this behavior would be nice.

[briandipalma]

My hierarchy in the screenshot above is a perfectly reasonable one. It's also perfectly reasonable to say I want to see pictures from December that aren't from events. Random pictures of people, pets or whatever around the house for example. I'd have them as albums the house smart screens would show for example. I wouldn't want a year when I could have a far more specific tag applied to the picture that has event and location details for example

[iainhallam]

Keepass has a toggle, "Show entries of subgroups" for this sort of functionality. A tickbox could be exposed in the UI for this, and as long as it remembers the state from session to session, I don't think it matters much which is the default. User research could be done to find out which way the expectation runs in a majority of users.

[Bakadori]

Could "Select hierarchy" be a separate option to simply selecting a tag?

…

On Tue, 20 Aug 2024 at 06:30, Brian Di Palma ***@***.***> wrote: It makes perfect sense, how would you see the images only tagged with 2023/December otherwise? If you want to see more tags just select them. — Reply to this email directly, view it on GitHub <#11730 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKLXYWYD3SX2KHJZ7OGPU3LZSJPYDAVCNFSM6AAAAABMM3DEWSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMZYHA4DAMQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

-- Álvaro

[bo0tzz]

A first pass tag implementation is now being worked on in #11980.