21:I[22016,["/_next/static/chunks/0__7n94dydo3p.js","/_next/static/chunks/3g3jn3-8mbixu.js","/_next/static/chunks/2oj925msqucms.js","/_next/static/chunks/1ymt1shyhmu-4.js"],""] 22:I[5014,["/_next/static/chunks/0__7n94dydo3p.js","/_next/static/chunks/3g3jn3-8mbixu.js","/_next/static/chunks/2oj925msqucms.js","/_next/static/chunks/1ymt1shyhmu-4.js"],"default"] 23:I[49308,["/_next/static/chunks/0__7n94dydo3p.js","/_next/static/chunks/3g3jn3-8mbixu.js","/_next/static/chunks/2oj925msqucms.js","/_next/static/chunks/1ymt1shyhmu-4.js"],"TimeAgo"] 24:I[82661,["/_next/static/chunks/0__7n94dydo3p.js","/_next/static/chunks/3g3jn3-8mbixu.js","/_next/static/chunks/2oj925msqucms.js","/_next/static/chunks/1ymt1shyhmu-4.js"],"CommentThread"] 25:T20a8,> For a while the only pytorch code that worked on newly released hopper GPUs we had was the Nvidia ngc container, not Pytorch nightly. The upstream ecosystem hadn't caught up yet and Nvidia were adding their special sauce in their image.

I'm sorry to come across as arrogant, but it's really just frustration, because being surrounded by this kind of cargo-culting "special sauce" talk, even from so-called principal engineers, is what drove me to burnout and out of the industry into the northwoods. Furthermore, you're completely wrong. There is no special sauce, you just didn't look at the list of ingredients. There never has been any special sauce.

NVIDIA builds their NGC base containers from open source scripts available on their gitlab instance: https://gitlab.com/nvidia/container-images/cuda

The build scripts for the base container are incredibly straightforward: they add the apt/yum repos and then install packages from that repo.

The pytorch containers are constructed atop these base containers. The specific pytorch commit they use in their NGC pytorch containers are directly linked in their release notes for the container: https://docs.nvidia.com/deeplearning/frameworks/pytorch-rele...

That is:

25.08: https://github.com/pytorch/pytorch/commit/5228986c395dc79f90...

25.06: https://github.com/pytorch/pytorch/commit/5228986c395dc79f90...

25.05: https://github.com/pytorch/pytorch/commit/5228986c395dc79f90...

25.04: https://github.com/pytorch/pytorch/commit/79aa17489c3fc5ed6d...

25.03: https://github.com/pytorch/pytorch/commit/7c8ec84dab7dc10d4e...

25.02: https://github.com/pytorch/pytorch/commit/6c54963f75e9dfdae3...

25.01: https://github.com/pytorch/pytorch/commit/ecf3bae40a6f2f0f3b...

24.12: https://github.com/pytorch/pytorch/commit/df5bbc09d191fff3bd...

24.11: https://github.com/pytorch/pytorch/commit/df5bbc09d191fff3bd...

24.10: https://github.com/pytorch/pytorch/commit/e000cf0ad980e5d140...

24.09: https://github.com/pytorch/pytorch/commit/b465a5843b92f33fe3...

24.08: https://github.com/pytorch/pytorch/commit/872d972e41596a9ac9...

24.07: https://github.com/pytorch/pytorch/commit/3bcc3cddb580bf0f0f...

24.06: https://github.com/pytorch/pytorch/commit/f70bd71a4883c4d624...

24.05: https://github.com/pytorch/pytorch/commit/07cecf4168503a5b3d...

24.04: https://github.com/pytorch/pytorch/commit/6ddf5cf85e3c27c596...

24.03: https://github.com/pytorch/pytorch/commit/40ec155e58ee1a1921...

24.02: https://github.com/pytorch/pytorch/commit/ebedce24ab578036dd...

24.01: https://github.com/pytorch/pytorch/commit/81ea7a489a85d6f6de...

23.12: https://github.com/pytorch/pytorch/commit/81ea7a489a85d6f6de...

23.11: https://github.com/pytorch/pytorch/commit/6a974bec5d779ec10f...

23.10: https://github.com/pytorch/pytorch/commit/32f93b1c689954aa55...

23.09: https://github.com/pytorch/pytorch/commit/32f93b1c689954aa55...

23.08: https://github.com/pytorch/pytorch/commit/29c30b1db8129b5716...

Do I need to keep going? Every single one of these commits is on pytorch/pytorch@main. So when you say:

> For a while the only pytorch code that worked on newly released hopper GPUs we had was the Nvidia ngc container, not Pytorch nightly.

That's provably false. Unless you're suggesting that upstream pytorch continually rebased (eg: force pushed, breaking the worktree of every pytorch developer) atop unmerged code from nvidia, the commit ishes would not match. Meaning all of these commits were merged into pytorch/pytorch@main, and were available in pytorch nightlies, prior to the release of those NGC pytorch containers. No secret sauce, no man behind the curtain, just pure cargo culting and superstition.19:[["$","article",null,{"className":"bg-white px-4 py-5 sm:mt-4 sm:rounded-xl sm:border sm:border-slate-200/70","children":[["$","div",null,{"className":"mb-3 flex flex-wrap gap-2","children":[["$","$L21",null,{"href":"/hacker-news/item/45400104","className":"inline-flex items-center gap-1.5 rounded-md border border-slate-200 bg-white px-3 py-1.5 text-sm text-slate-700 transition-colors hover:border-slate-300 hover:text-slate-900","children":[["$","$L22",null,{"ref":"$undefined","icon":{"name":"corner-left-up","size":24,"node":[["path",{"d":"M14 9 9 4 4 9","key":"1af5af"}],["path",{"d":"M20 20h-7a4 4 0 0 1-4-4V4","key":"1blwi3"}]]},"className":"h-4 w-4","strokeWidth":1.75}],"Parent"]}],["$","$L21",null,{"href":"/hacker-news/item/45404089#c45404089","className":"inline-flex items-center gap-1.5 rounded-md border border-slate-200 bg-white px-3 py-1.5 text-sm text-slate-700 transition-colors hover:border-slate-300 hover:text-slate-900","children":[["$","$L22",null,{"ref":"$undefined","icon":{"name":"list-tree","size":24,"node":[["path",{"d":"M8 5h13","key":"1pao27"}],["path",{"d":"M13 12h8","key":"h98zly"}],["path",{"d":"M13 19h8","key":"c3s6r1"}],["path",{"d":"M3 10a2 2 0 0 0 2 2h3","key":"1npucw"}],["path",{"d":"M3 5v12a2 2 0 0 0 2 2h3","key":"x1gjn2"}]]},"className":"h-4 w-4","strokeWidth":1.75}],"Full thread"]}]]}],null,null,["$","div",null,{"className":"mt-2 flex flex-wrap items-center gap-x-1.5 text-sm text-slate-500","children":[false,[["$","$L21",null,{"href":"/hacker-news/user/siliconpotato","className":"font-medium text-slate-700 hover:underline","children":"siliconpotato"}],["$","span",null,{"aria-hidden":true,"children":"·"}]],["$","$L23",null,{"time":1759065102}],false]}],["$","div",null,{"className":"break-words text-[15px] leading-relaxed text-slate-800 [&_a]:text-orange-700 [&_a]:underline [&_a]:decoration-orange-300 [&_a]:underline-offset-2 [&_a]:break-all hover:[&_a]:decoration-orange-600 [&_code]:font-mono [&_code]:text-[13px] [&_p]:mt-2.5 [&_pre]:mt-2.5 [&_pre]:overflow-x-auto [&_pre]:rounded-md [&_pre]:bg-slate-100 [&_pre]:p-3 [&_pre]:whitespace-pre mt-4","dangerouslySetInnerHTML":{"__html":"Still, it adds an air of arrogance to the whole post. For a while the only pytorch code that worked on newly released hopper GPUs we had was the Nvidia ngc container, not Pytorch nightly. The upstream ecosystem hadn't caught up yet and Nvidia were adding their special sauce in their image. Perhaps not stupidity but lack of docs from nvidia"}}],["$","div",null,{"className":"mt-4 flex flex-wrap gap-2","children":["$undefined",null,["$","a",null,{"href":"https://news.ycombinator.com/item?id=45404089","target":"_blank","rel":"noopener noreferrer","className":"inline-flex items-center gap-1.5 rounded-md border border-slate-200 bg-white px-3 py-1.5 text-sm text-slate-700 transition-colors hover:border-slate-300 hover:text-slate-900","children":[["$","$L22",null,{"ref":"$undefined","icon":{"name":"arrow-up-right","size":24,"node":[["path",{"d":"M7 7h10v10","key":"1tivn9"}],["path",{"d":"M7 17 17 7","key":"1vkiza"}]]},"className":"h-4 w-4","strokeWidth":1.75}],"View on HN"]}]]}]]}],["$","section",null,{"aria-label":"Comments","className":"bg-white px-4 pb-4 sm:mt-4 sm:rounded-xl sm:border sm:border-slate-200/70","children":["$","$L24",null,{"comments":[{"id":45405799,"by":"nickysielicki","html":"$25","time":1759077967,"descendants":0,"children":[]}],"op":null,"storyId":45404089,"commentCount":1,"track":false}]}]] 1c:null 20:[["$","title","0",{"children":"Comment by siliconpotato | Hacker News Reader"}],["$","meta","1",{"name":"description","content":"A fast, clean Hacker News reader. Top, new, best, Ask and Show HN, collapsible comment threads, new-comment highlights, search, and saved stories. No account needed."}],["$","link","2",{"rel":"manifest","href":"/manifest.webmanifest","crossOrigin":"$undefined"}],["$","meta","3",{"name":"robots","content":"noindex"}],["$","meta","4",{"property":"og:title","content":"Hacker News Reader"}],["$","meta","5",{"property":"og:description","content":"A fast, clean Hacker News reader. Top, new, best, Ask and Show HN, collapsible comment threads, new-comment highlights, search, and saved stories. No account needed."}],["$","meta","6",{"property":"og:url","content":"https://www.srizon.com/hacker-news"}],["$","meta","7",{"property":"og:type","content":"website"}],["$","meta","8",{"name":"twitter:card","content":"summary"}],["$","meta","9",{"name":"twitter:title","content":"Hacker News Reader"}],["$","meta","10",{"name":"twitter:description","content":"A fast, clean Hacker News reader. Top, new, best, Ask and Show HN, collapsible comment threads, new-comment highlights, search, and saved stories. No account needed."}]]