STRONGLY CONSIDER OTHER ALGORITHMS than MD5
If you need deduplication: Look at murmur3-128 as it’s faster than MD5
If you need deduplication look at murmur3-128
Benchmark MD5 vs Murmur
Benchmark: guava md5-hash for string with [50000] chars
| Metric | (micros) | (millis) |
|---|---|---|
| Min | 48 | 0.048 |
| P50 (Median) | 49 | 0.049 |
| P90 | 50 | 0.050 |
| P95 | 51 | 0.051 |
| P99 | 54 | 0.054 |
| P99.9 | 143 | 0.143 |
| Max | 1,692 | 1.692 |
| Total | 2,466,909 | 2466.909 |
| Iterations | 50,000 |
Benchmark: guava-murmur3-128bit for string with [50000] chars
| Metric | (micros) | (millis) |
|---|---|---|
| Min | 7 | 0.007 |
| P50 (Median) | 8 | 0.008 |
| P90 | 8 | 0.008 |
| P95 | 9 | 0.009 |
| P99 | 16 | 0.016 |
| P99.9 | 43 | 0.043 |
| Max | 1,182 | 1.182 |
| Total | 407,266 | 407.266 |
| Iterations | 50,000 |
If you need cryptographic hash: look at sha-256 (as MD5 is broken for collision resistance)
Consider sha-256 if you want a greater level of collision avoidance or using it for security.
However, also note that the chance of collision with properly implemented MD5 hash is very small:
If you keep all the hashes then the probability [of collision] is a bit higher thanks to birthday paradox. To have a 50% chance of any hash colliding with any other hash you need 2^64 hashes. This means that to get a collision, on average, you’ll need to hash 6 billion files per second for 100 years. - ref