On the other hand, a colorful sorting would only negatively affect you when you are actively looking for a book.

No, unfortunately, it would remind me every time that I looked at it that I had somehow married someone who thought this was acceptable.