> deduplication at the file level: every file is now stored under its BLAKE3 hash
Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing).
If I've got a file named:
DSC98731-b3-7b39197a22.JPG
then:
- if that file doesn't checksum back to 7b39197a22 there's a file integrity problem (amazing and it already helped me troubleshoot issues)
- if any other file has the same Blake3 7b39197a22 hash, it's a duplicate
- if that 7b39197a22 checksum is in my database, "things can happen".
For example my DB can say "any file with a Blake3 hash of 7b39197a22 can always be deleted" or "any file with a Blake3 hash of 887463c09e, if it's got a generic filename like "dscXXXXX" can always be renamed to "20260722jackJohnAtTheBeach-b3-778463c09e.jpg" (or whatever suits you).
It's really great (and I know several here independently made similar schemes) and Blake3 is an amazing hash for those kind of use.
People say uv is good because it's fast, but honestly I don't care about that. We switched to using it for distribution of our Python-based tools because it makes it very convenient to install directly from a git repository and then to subsequently update from same repository. No need to build a package.
I tried it out successfully for the first time the other day (had 1 false start some months ago). This is after 10 ish years of system wide installs or venvs. I didn't find it fast at all. Every time I went to run the script it spent multiple seconds checking dependencies. Then one time it updated one, which luckily didn't break anything but I did get concerned. I'm sure there're some flags I didn't know to use but uvx was not great. On the other hand, it did seem to install the packages faster than pip.
Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing).
If I've got a file named:
then: For example my DB can say "any file with a Blake3 hash of 7b39197a22 can always be deleted" or "any file with a Blake3 hash of 887463c09e, if it's got a generic filename like "dscXXXXX" can always be renamed to "20260722jackJohnAtTheBeach-b3-778463c09e.jpg" (or whatever suits you).It's really great (and I know several here independently made similar schemes) and Blake3 is an amazing hash for those kind of use.