HN: Help make Golang more (resource|env) friendly

Hi, just discovered that the (sometimes excessive large) golang mod disk cache is - in opposite to the compiler cache - not (yet) properly deduplicated.

Try yourself: Made a little hack/tool to deduplicate data via hardlinks (unix/i-node-filesystems tested only, not windows yet).

https://github.com/paepckehh/fsdd

(less than 1kLOC, 100% golang stdlib, no external deps)

  go install paepcke.de/fsdd/cmd/fsdd@latest

  cd $GOMODCACHE && fsdd --hard-link .

  FSDD [start] [/usr/store/gomod][hash:MAPHASH]

  FSDD [_done] [time: 39.000221ms]

  FSDD [stats] [files:13329] [inode(s): 8680] [sym.valid: 0] [sym.invalid: 0] [data blocks: 277.1 Mbytes]

  FSDD [_info] [new deduplication savings][inode(s): 4649] [data blocks: 66.9 Mbytes]

If you can reproduce and has it shows no downsides (remove/add should auto-dereference) it should be part of src/cmd/go/internal/mod* pack?

If you can confirm/field test, please report back! I would love to write a quick stdlib/proposal.

At golang scale, it could make a real resource / enviromental dent!

It least for me, it cut my disk cache in half and reduced vfs fs mem-cache required as well, so should scale even on deduplicated fs like zfs/btrfs. Made my batch builds fast and snappy again!

Michael (edit:first-post-ever-style-fixes)