2,479 karma · joined June 1, 2009
Living in North Carolina, remote-working for San Mateo, CA software company.
The AI is enabling a lot more experimentation with software. The AI supplies the tomatoes (infrastructure / grunt work), grandma tries her recipes (the software / business idea).
https://en.wikipedia.org/wiki/2014%E2%80%932017_Brazilian_dr...
https://www.smartcitiesdive.com/ex/sustainablecitiescollecti...
EDIT: added timeframe
I had a couple of very specialized adapters for the first couple of environments that were very aware of the whole GPU conversion and deployment stack (ONNX, TensorRT, Triton, etc.), but even these were very fragile to version upgrades, etc., and required a lot of adaptation from one "stack" to the other. As soon as the frontier models got to be real good I took a more "agnostic" approach. My new framework's goal was to be "mealleable" and very hands-off w.r.t. the details of the conversion / deployment ... after all, these models are very well trained on the whole AI pipeline, including deployment. So now the basis is more or less (a) where are the model files, which Docker image do we want to use, which "conversion / deployment" method(s) do we want to try, what are the inference use cases addressed; (b) what is the conversion / deployment method used? (c) how do we assemble the smoke-test and basic performance test cases for multi-client, single- and batch-processing modes for all use cases? (d) how do we package the end results so that the DevOps person has all they need to make sure they have the proper files and that they can run Docker and check that the inference works? These questions and answers are all wrapped in some very flexible base classes. Every conversion is somewhat different, so each one involves extensive discussions with Claude Code (e.g., "hey, look at this other prior conversion, its raw files, compare with these raw files, develop a conversion / use-case-smoke-and-performance-test, and packaging). Rather than trying to be very hands-on at these lower level, Claude has almost free reign at the "low level" to suggest the best approach, and I so I "talk to an engineer with vast knowledge but (for now) a bit less judgment". With this method though I probably cut the total time to conversion-for-deployment by 80% to 90% now ... the choices and options are vast, and Claude knows a lot more than I do.
At times there is not a ready-made solution for the particular problem at hand, and so then it gets more interesting with how to shoehorn a solution into one of the available technologies. We prefer one of the various builds of the Triton Inference Server (or one of its hardened versions maintained by others).
QUOTE:
The National Rabies Management Program was established in recognition of the changing scope of rabies. The goal of the program is to prevent the further spread of wildlife rabies and eventually eliminate terrestrial rabies in the United States through an integrated program that involves the use of oral rabies vaccination targeting wild animals.
Since 1995, Wildlife Services (WS) has been working cooperatively with local, State, and Federal governments, universities and other partners to address this public health problem by distributing oral rabies vaccination (ORV) baits in targeted areas. This cooperative program targets the raccoon variant, canine variant in coyotes and a unique variant of gray fox rabies.
I had worked on some earlier deployment environments for a few models where we focused on one version of Triton and one runtime technology. Even upgrading to a newer Triton version was brittle, involving a lot of command-line changes at various phases. This was written mostly pre-Claude-getting-real-good. I decided that probably Claude had matured and was way better at understanding the particulars of AI-model-GPU-deployment-and-technologies. I worked with Claude to make the framework a much more lightweight wrapper, ignorant for the most part of a lot of the deployment internals.
After this refactoring and doing the first couple of models, it's quite amazing at how well Claude can figure things out. For any new model we now set up its "specialization" directory and its documentation subdirectory, point Claude at the proper AI-model files, point Claude to the sample non-optimized inference code and test data, and point Claude to a similar conversion we've done in the past. There a multi-layer class hierarchy dealing with various tiers. I ask Claude to explore the existing conversion/packaging, the model, any documentation that comes with it (a lot of times there are unexpected twists), the sample code, and the desired multiple use cases the model is meant to address, and the test data. Claude has been trained, I'm sure, on a lot of AI model conversion, so it's able to synthesize the full multi-stage conversion/deployment pipeline, come up with appropriate test cases for all the use cases involved. There are usually between 3 and 10 refinements after initial synthesis, fixing outright errors, refinemnts that the data-science team requests after playing with test deployments, etc. The options and pitfalls are vast, and without Claude each preparation likely would 10x or more longer. I just put most details in Claude's hands, and make sure the general framework is good enough to provide external uniformity. When all is working, it takes a couple of hours to make sure the documentation is good.
All this to say that at least for this domain, Claude / AI has been a game-changer and has sped up the process amazingly.
My biggest regret though is that I may never manage to play more than a few minutes of soccer at a time again. I got back to Latin America in early adolescence having missed some crucial soccer years. I was soon a couple of years younger than everyone else in my grade, and P.E. classes were not very fun, it was hard to compete and I rarely got to participate in real action on the soccer or rugby field. In my late teens I started to actually develop some soccer sense and got a bit better. But student/teacher political strikes during the dying years of a dictatorship and upcoming return of my family to the USA brought me to the USA for studies, and I didn't play much in college.
After a few years in SF Bay Area I started playing pickup soccer and eventually got to play quite well , especially during a particular two year stretch. Then marriage, busy jobs, having a kid meant I laid off the regular soccer for a while.
And now, with a bit more extra time I could maybe spend playing I no longer can. I've never been on a team, never been a specialist at a position, never trained regularly. The doctor said maybe with physical therapy and pain killers I could do it. I'll work toward that.
AVGO/Broadcom in some way acts like a big PE firm, rolling up other software companies, integrating them into their huge suite of offerings, ousting the new integrated offering's competing tools from the customers environments and selling the increment, and cutting off smaller customers not willing to subscribe to the huge suite.