Walmart is migrating the remaining F# code into Java | Hacker News Reader

Walmart is migrating the remaining F# code into Java

careers.walmart.com

87 pointspjmlp91 comments

That makes me sad, but it's not surprising.

I was part of Jet.com before and during the acquisition, and the main reason I joined was because I was (and am) a big functional programming geek, had a fair bit of Haskell and Erlang experience, and F# seemed like a neat language.

It turned out that it absolutely was a neat language, and I consider the two years I was at Jet to be a fun highlight of my career. I got to see where functional programming shined in real-world distributed applications, where it didn't shine, and how it integrated with other libraries.

I quit in 2018 to work for Apple, stayed there for about two years, and went back to Walmart in 2021 for another year. In that time, I worked on the chatbot; one of the very last things that was still F# in the Walmart ecosystems. I had less fun the second time around (not really because of anything Walmart did and not because of F#), so I only stayed for about a year, but I still found some enjoyment doing professional functional programming in a pure-ish language.

It doesn't surprise me that they're changing to JVM stuff. Even when I was there, there was a lot of external pressure to move our stuff to Kotlin. We stayed with F# because of a reliance on the Microsoft Bot Framework, which was predominantly .NET, but once ChatGPT started getting traction I figured it was only a matter of time before they migrated to a Transformers model and rewrite it into JVM.

It makes me sad. F# is a language that I really do not think gets its fair shake. It's a language that really does push functional purity, but manages to still be pragmatic. I found the integration with the core .NET libraries to actually be very good, and even when I had to do OOP, F#'s implementation was exceptionally pleasant compared to C#. Honestly, if I had any say in running a business, I'd consider F# for at least a few of the services.

Out of curiosity - what were the rough edges in the distributed application space?

Performance.

Generally speaking, F# was actually very fast, and had nice concurrency support, but there were times that wasn't the case.

For example, in 2016 I was part of the initiative to rewrite the ad feed. We had to read in several Kafka topics, do some joining on our end, and emit to a separate Kafka topic. This isn't terribly hard to write, but we were dealing on the order of about ~100gb of data being pushed into memory. This is hardly "big data" stuff, but it's enough to highlight some issues.

Specifically, the built F# persistent map structure was simply too slow to get the performance we wanted. I really like that structure, it's really handy and nice, but I ended up having to make heavy use of the ConcurrentDictionary that was built into .NET. This wasn't that hard or anything, but it made me a little sad that I had to move to a mutable store to get the performance I needed.

There was also the fact that the `async` monad, while generally very good and useful, had bizarre bottlenecks that were hard to measure. It was difficult to know when the async task was actually started, and when you tried to measure performance bottlenecks you were really only measuring the scheduler, not the actual performance. This isn't really F#'s fault, this is an issue with any kind of cooperative scheduling system, but occasionally to get the performance we needed we'd have to move to lower level threads instead of the pretty monadic stuff. Microsoft eventually released the Task monad which generally performed a bit better.

There were other things here and there; the Kafka client libraries for .NET simply aren't as good as the Java ones. Jet actually open-sourced their own (https://github.com/jet/kafunk) which did make it a bit more functional and nice, but it had performance issues as well, so a lot of us ended up using Confluent.

There were little annoyances specific to F# as well; there's no real concept of a monad transformer, so if you wanted to do something like, for example, combine an Option and an Async into generalized syntax, you'd have to write your own wrapper monad thing, which wasn't that hard but was sort of ad hoc.

The general rule of thumb was that the first draft of software, we would try and keep as functional and pretty. If that was too slow, we allowed mutation but only within a function. If that was too slow, we'd allow global mutation but only with thread-safe stuff.

> where it didn't shine

Care to expand on this for the curious?

Answered in sibling thread.

Jet.com was the original purchase by walmart to my understanding.

They contributed quite a bit of useful things to the ecosystem, and were one of the "F# companies that made-it".

Some of their projects live here: https://github.com/jet?language=f%23

An old HN thread on why they chose F# exists here: https://news.ycombinator.com/item?id=9971412 (though the link is broken, comments still exist).

As an F# enjoyer, it's not surprising that this happened. I've seen quite a few companies migrate to Java from various languages. Casting aside syntactical or language-feature niceties, there's just no substitution for the production-maturity of the Java ecosystem.

Plenty of open-source tested-at-scale libraries being put through their paces at large organizations taking in tons of traffic. Best-practices out-of-the-box adapter libraries for most infrastructure. First-party (actual first-party highest tier of coverage) libraries for communicating with the most common infrastructure (All of AWS, elasticsearch, kafka, cassandra, etc). Even with all of the open-source mindshare, most can have an extreme amount of information availability and support with the common core pieces like JUnit, Spring, Hibernate, and the accompanying set of libraries that usually join those. JFR is to my understanding still the only one of its kind.

I'm grateful for what everyone in the F# and dotnet community has done and continues to do, but even in dotnet (which is probably the closest competitor for production cloud web-services), you're spending a lot more time on writing and wiring up supporting libraries for things that the Java stack does for you out of the box. I prefer many small things in the dotnet world to the jvm world, but at the same time I can't deny that I'm much more able to focus on core business logic and concerns when working on a spring project than when I am on a .net 8 web project.

Does anyone have examples of people migrating from Go to Java? I suspect this has more to do with moving away from functional and/or reactive programming.

Go is a relatively young language. Building a product in Go, only to port it to Java has to be a rare phenomenon.

Go is 12 years old (1.0 in March 2012). Young compared to Java, but old enough for codebases to have gone through a few write-rewrite phases. Case in point - this thread is about code that Walmart got when it acquired a company founded in 2014.

Sure, but Walmart is moving from a niche language to a mainstream one. Significantly easier to make that business justification. Go to Java should require a lot more political capital to champion such a change.

For what it’s worth, the 2023 stack overflow survey said 14% of professional programmers used Go vs 1% for F#. (Java at 30%)

Given the description also describes moving from a custom event sourcing engine to Kafka it doesn't seem to be moving away from reactive programming. As much as anything it seems like something of an "Apache Effect" that some Enterprise Highly Paid Consultants came in, told them they were doing "everything wrong" and that everything would be better with off the shelf open source components and that the "best" language for working with off the shelf open source components is Java. Most of that isn't true in various ways, but the number of Enterprise HPCs that love Java despite that language's deficiencies is surprisingly high. (Though maybe not that surprising given Oracle owns Java today and has always directly and indirectly owned a lot of HPCs in the Enterprise space.)

There's a very strong "nobody ever got fired for choosing IBM" effect surrounding Java. I can list specific situations where I'd prefer another platform all day long. But there are only a few specific situations I can think of where I would say Java is not a defensible choice.

And giving advice that's optimally defensible is the Highly Paid Consultant's actual job. You don't bring a consultant like that in when you're playing to win; you do it when you're playing to not lose.

My team wants to do that. Only because the rest of the org uses Java.

Working link to the old Jet article: https://archive.is/lkDXg

> Plenty of open-source tested-at-scale libraries being put through their paces at large organizations taking in tons of traffic. Best-practices out-of-the-box adapter libraries for most infrastructure. First-party (actual first-party highest tier of coverage) libraries for communicating with the most common infrastructure (All of AWS, elasticsearch, kafka, cassandra, etc).

I doubt this is really it. It's probably more about the size of the labour pool, and unfamiliarity with the language and functional programming that extends onboarding.

This definitely affects my decision. Personally, I love functional programming, and will use it as much as possible. Professionally, I typically go with OOP or even procedural whenever it's my call to make. The vast majority of people I've worked with are much more comfortable in one of those two paradigms, and that makes onboarding new people onto the project much easier. Even people who express a desire to work in a functional language don't necessarily have the experience or training to know how best to do it in a professional setting.

It's to the point where I don't even like to advertise that I like to work in functional languages around the office. Too much risk of being called in to be the hero when the last person who understood the code for some pilot project to try out Scala or Clojure that grew to become mission critical decides to leave the company.

That's why I think C# is did the industry a great service by introducing LINQ. It really drove a movement to adopting some functional-style idioms in lots of languages, even Java in the end with it's stream API. It at least got functional programming's foot in the door among industry standard OOP languages. Still a long way to go though.

Though, Java's stream API is not suitable for use with typical functional programming idioms the way that LINQ is. In my opinion, no code that touches it should be considered functional.

That's a big part of why Scala and Kotlin code tends to avoid it.

Was actually driven by VB.NET initially, and Erik Meyer's contribution.

https://dl.acm.org/doi/abs/10.1145/1297105.1297078

Ironically LINQ is something that was already present in Smalltalk collections, yet kind of lost in OOP languages until closures came to be.

Reading this made me chuckle because there was Java developer in a "Ask HN" thread a few months back asking for advise after struggling to find work for several months.

The advice HN gave? Java is dead and he needed to reskill.

I don't have much familiarity with F# (although everything I've heard has been very positive), but your comment basically sums up why corporates use and will continue to use Java – the ecosystem is extremely mature and battle tested. While it might not be as fancy or fun to work with, you know you're in safe hands which is extremely important when you have billions of dollars resting on the stability and security of your software.

I'd even argue it's got little to do with the fact you have to spend time building things that comes out of the box with Java, and far more to do with the fact you can simply trust code that's been used by hundreds of fortune 500 orgs in production.

thats horrible advice. java is thriving.

Presumably all their F# runs on .NET--if their goal was simply transitioning to a language with a broader talent pool, wouldn't they "just" migrate the project to C# instead? I'm wondering if this is less about "the death of functional programming" and more about an internal platform unification under Java.

Non-zero chance Walmart's architects didn't even know .NET runs (and is easier to deploy to than Java) on Linux for good 5+ years already...

Edit: Okay, they know, I'm sorry :) Does not make it less sad and perhaps very unfortunate decision. Also, Hibernate sucks.

I'm sure they've been forced to endure mono as part of 'wonderful' security software

There's two kinds of Mono on Linux from what I understand...

The older one is packaged with a particular piece of security software, and the packages have more problems than I can count

edit: I got got! TIL that proper .NET on Linux exists. I'm more mad at Mono/this software now

What makes you think so? Mono has never been used for back-end workloads aside from projects led by enthusiasts.

I edited my post a bit to include a vague hint; it's not "backend workloads", assuming part of an application stack/service

It's security software - to be less vague, it's software made by Crowdstrike

Okay, now this evokes some bad (because of Crowdstrike) memories... But using Mono in any recent years where the blessed way to do so is just using .NET is a big disservice to what the thing can do, and it requires you to really go out of your way to make a container that uses Mono over vanilla (Core)CLR.

Also, not liking what is now MIT FOSS stack does not make sense. It is the unified set of repositories going forward, with Mono currently serving exotic targets like WASM and being a part of dotnet/runtime. Completely not surprising for it is the most standard myopic response you usually hear from Linux communities (which, checks notes, are supposed to advocate for OSS?) the moment .NET is mentioned.

I appreciate you making me current, being more on the architecture side than software... I get very little exposure to these happenings.

I mostly glue existing stuff together, not make new stuff. My frustration with .NET is dated, really due to Mono.

> which, checks notes, are supposed to advocate for OSS?

I'm not sure how much to unpack here; my complaints are more with what I was given than anything philosophical

You don't have to use Mono. People are talking about actually using .NET on Linux.

Well that makes me even more upset at this software

I'm sure they know .NET runs on Linux.

Changing the underlining OS for 1000's of hosts is not a trivial task.

I've worked with former Walmart technical staff and they're quite good, and there is indeed a zero-percent change that their team doesn't know this.

The Walmart folks are pretty bright in general. I did some consulting for them a while back so I saw this first-hand. Organizations aren't monoliths, of course, but my experience was that they definitely did know what they were doing.

I was on the Walmart teams that did F#. Since at least 2018 there was at least some services running on .NET Core on Linux, and by 2021 everything was on Linux. Most of us dev'd on MacOS using JetBrains Rider, running .NET Core.

My title wasn't "architect", but it was "Staff Software Engineer" by the end. Does that qualify?

I was on a call with one of the more senior Walmart (non-Jet) architects about Java and .NET back when I worked on the .NET team. He seemed dismissive of the idea that F# (and .NET) ran perfectly well on Linux, and didn't demonstrate even the slightest curiosity about that. We committed strong support for their workloads, talked about our staffing for that support, talked about how they'd flown me out to New Jersey twice for some in-person feedback and learning sessions, etc. Didn't seem to matter to this person.

Where my heart lies aside, I also think it's rather dumb. My understanding of the system talked about in this job post is that it's quite mature and works very well, and my default stance for any rewrite is "don't be dumb, don't do it", and that would apply here.

.NET really lacks a lot of first class open source libraries that the JVM ecosystem has. I think C# is a superior language to Java, and I think F# made some interesting trade offs that might be beneficial in the long run compared to Scala, but I wouldn't recommend companies go the .NET route these days.

It's too easy to run into a situation where the 'official' API supports Java/Python/Go as a first class citizen and the .NET library is a sad afterthought.

Could you give a couple examples of libs that are missing. Not disagreeing, just curious.

Mostly client SDKs for different RESTful HTTP APIs

Apache Beam, Kafka and Scylla at my previous job were both examples where I felt like the .NET offerings were either not official or severely lacking.

Dotnet and java usually have similar libraries for most things, but sometimes it comes down to how different they are or what they support.

Let's look at general distributed messaging abstractions.

The most modern-featureful one for dotnet I know of is MassTransit: https://masstransit.io/documentation/concepts. You can see it supports a handful of queues and some interesting patterns.

Compare this to the list of features in Apache Camel: https://camel.apache.org/components/4.4.x/eips/enterprise-in...

Additionally, the _support_ of the libraries for common open-source (and closed-source infra) is usually vastly different:

- https://pulsar.apache.org/client-feature-matrix/

- Small features being first party like https://github.com/awslabs/amazon-sqs-java-extended-client-l... vs https://github.com/raol/amazon-sqs-net-extended-client-lib

- Missing features or those that lag behind: https://github.com/confluentinc/confluent-kafka-dotnet/issue...

- Usually manually wired and configured vs the spring boot "starter" pattern of having libraries that automatically do some of the manual setup work for you: https://github.com/spring-projects/spring-boot/blob/main/spr...

I wish more client library sets had the feature-matrix that the pulsar one does, because in practice most end up being the same: Java supports everything because it's either built in the same codebase or is the most used client and gets the most support, while the dotnet client codebase has many feature-requests or performance improvement issues, often leading to a "third-party client" being created.

Isn't not having perfect support with Apache is, in a way, a criticism of any non-JVM language? Perhaps the problem is on the other end?

We need a couple more Apache Struts and Log4J incidents for the public to realize there may be better technologies and ecosystems. Spring is very slow anyways.

As for Pulsar, there is a nice and optimized client (written in F#!) https://github.com/fsprojects/pulsar-client-dotnet

Which .NET Foundation was supposed to solve, yet we observe more and more FOSS projects not happy with it, and moving away from it.

Avalonia being the most recent example.

How does that relate to Apache projects being somewhere between Java-centric and Java-exclusive depending on the feature? If anything, we need more decentralization, there are awesome products like NATS that offer rich and well-written client libraries in multitude of languages. More of that.

Oracle, IBM, Sun on its days, didn't set the tune of the music at Apache, as apparently Microsoft does at .NET Foundation, as per complaints of the projects that decided to withdraw from the Foundation.

It doesn't help that after the whole FOSS push, upper management has won and now Windows/Azure VS workloads are the ones being given focus, even the new VSCode plugin is under the same license scheme as VS.

Thus UNIX first shops rather focus on other ecosystems.

This was my take as well, especially with the intentional name dropping of Kafka.

Always a good sign when internal jargon leaks into the job description. It's like scam emails, they're intentionally letting people self-select out.

I remember interviewing for a job 20 years ago at an insurance company where they explicitly listed "must have experience with the XYZ123ABC system" where that was just a name they gave to one of their internal systems; no one outside of that company would have ever come across it. I tried explaining to the recruiter that unless they were limiting the position to internal applicants, they should ask for experience with whatever tech was used to build the system rather than the system itself. She was completely baffled by that.

Maybe they did want internal applicants only. Sometimes you can't state that publicly, etc.

That does sometimes intentionally happen if the company has a specific visa candidate or internal employee selected for the position but is required to post a public facing position - that's also where you'll see extremely specific and unrelated tech requirements like "15 years web development, 5 years embedded system driver development, maintainer of at least two major open source projects".

The posting is looking for someone with industry experience. It’s not a junior or entry position.

I recognize all the jargon here. None of it is exclusive to, or internal to Walmart, and the terms used are perfectly fine for the type of position.

Too bad, although as an F# fan, I expected this would happen eventually, somewhat surprised it didn't happen already. Jet.com seemed more of like an anticompetitive/acquihire purchase for walmart, I doubt they had any interest in the Jet tech.

F# is still pretty great though, I use it in many of my one-off personal projects. I think the jobs situation for it is still pretty poor in the US, but that seems true of any language outside the top 10. Even Rust devs, despite its popularity, apparently struggle to find jobs that use it. I suspect industry is very conservative with programming language adoption, maybe because there is a lot of risk in developing software in the first place. So there are winner-take-all dynamics in which ones get used in industry long term.

the tragedy of languages making an accent on functional programming is that CS education simply does not make a big enough focus on learning that paradigm, which takes a lot of effort to learn, and then there are simply not enough people to hire for reasonable money to seriously maintain these codebases

this self filters only the most committed and possibly eccentric programmers to keep using them, which again reinforces the loop. this is a shame because OOP, which has few of the theoretical and practical benefits of FP, is regularly taught

Yeah there was no mention of functional programming during my formal CS degree. I actually had started trying to learn more academic programming by taking a Coursera MOOC that I only realized was entirely based on functional programming after being a self-taught ad-hoc user developer. It was fascinating and intellectually stimulating but also challenging beyond my free time while at the time lacking more of the CS fundamentals. It looks like it was wisely broken up into multiple 3 week sections but I suspect this is what the course evolved into - https://www.coursera.org/learn/programming-languages

Of course when I took in person university courses we learned Java as the fundamental programming language.

I think almost everything useful and practical has made it into oo and modern languages already though or became best practice for writing in other languages.

I don't even really write very stateful or object oriented python anymore, and idk if anyone would really recommend that? That's been my experience though.

I thought most US universities taught some kind of Lisp? Over here in Europe many schools teach OCaml or Scala.

I don't really buy into the lack of talent/skill argument. Especially given that F# isn't exactly the most hardcore FP language and toolchain out there. I hire and onboard people from many kinds of background to a non-trivial Scala codebase. Even new grads coming from mostly Python notebooks do all right.

This is also supported by all modern OO languages going somewhat hybrid, as they adopt an increasing number of functional features.

If you're looking for an explanation as to why FP isn't more popular, I think it's mostly the fact that modern high-level languages have become good enough. And Go has proven that even an objectively bad language can succeed if the developer experience and productivity improve in other dimensions.

lol. most unis teach java, python or node these days. (i learned on java and C++ back in 2005-2007)

It's going to be a real struggle to hire F# developers to migrate things off of F#.

Most people working in niche langs arrived there looking for something better than the mainstream.

Perhaps AI/chatbot tools can help with code translation

I think it depends on the delivery during the interview/onboarding process.

I would definitely have a lot of questions around the goals and motivations for the decisions behind the port. If the thinking is in the right place (we're making a good trade given our situation) I could get up for it, despite my bias toward functional-style code.

You'd hope so, but the job ad specifies Java experience, so there's a good chance this will be converting code from FP to OOP. And the reason is most likely corporate homogeneity.

The best part is that they are migrating to JAVA, not Java.

It always amazes me how nearly every job posting somehow acts like Java is an acronym.

At my company I’ve asked that engineers proofread job descriptions so we don’t look stupid, but no, we still post for JAVA, NODE, and C SHARP.

I've always assumed that it is just because many of the companies asking for JAVA are still typing some of their job descriptions through mainframe terminals that haven't yet been able to afford upgrading to lower case letters and the job description work was a nice break between COBOL batches.

  Java
  As
  Viable
  Alernative

Let's not forget GIT and DOCKER.

Probably because they couldn't hire effectively?

Wouldn't C# be the most reasonable option if so? Maybe they did the math and the .NET ecosystem is much more expensive than Java? Even ignoring MS cost, most 3rd party libraries on .NET usually require a license.

What kind of libraries, given the tasks solved by Walmart, do you have in mind?

I dont know their needs, but last time I tried to do a computer-vision project on C#, all libraries in Nugets where shitty wrappers for cloud solutions or just paid libraries (last update to OpenCV is 7 years old).

Using bindings to C++ libraries is likely to yield better experience. C# has really good interop API (P/Invoke).

I don't have experience with machine vision but here's an example of a speech2text library that integrates whisper.cpp in an idiomatic way: https://github.com/sandrohanea/whisper.net

There are many good community projects with code quality way higher than your average enterprise SDK with layers upon layers of abstractions and allocations. Finding them is the same as with most other languages like Rust or TS.

Do they really? that seems difficult to navigate

>Ctrl-F "Remote", 0 results found

Looks like the position does not offer remote wfh unfortunately

Yup, first thing I did too. Shame!

Is Walmart "techy" again? I remember the 2015ish years where they pushed their new techy devops remote teams and then shut them down and fired everyone. It was so abrupt and out of left field. I don't know if their CTO changed or what but a lot of my peers were laid off.

I can't remember what they were called, they had some cool kubernetes (mesos?) stuff iirc.

This F# code is probably remaining from their Jet.com acquisition, which (IIRC) used it to run some box-packing algorithm.

They acquired Jet.com years ago. Not sure how much of that infra made it into Walmart, but they definitely made an attempt to be "more tech". Granted that the entire Jet project was an acquisition attempt to irritate Amazon - what the founder had done before with diapers.com

I’m pretty sure their online presence was rebuilt by the Jet.com guys.

The Jet.com CEO was almost immediately made the head of their digital transformation and took an end to end approach of digitizing everything from the website to their logistics and warehousing.

This is correct. I was there.

Oh so Jet was the techy one they brought in that brought that? I had never heard that, interesting. It's got to be terrifying working at a company that gets bought by Walmart.

edit: https://corporate.walmart.com/news/2016/08/08/walmart-agrees...

They were pretty "hardcore". I once went to a meetup there to learn about the Pony language.

what a shame. this was the coolest part of jet.com. never got to work for them, but i really wanted to since i knew f# well at the time (thanks, Jane Street; seriously) and they were the only place that was using it seriously.

they could've at least moved this codebase to rust. oh well. corporate's gonna corporate.

I suppose it was matter of time (and budget), since Walmart is known “Java shop”, and they want a singular platform?

Not all of it :) yet

Apparently this is due to acquired company using F# while Walmart stack is full Java. Though I do pity the developers who will have to now slog through a much worse language like that.

Which one is worse?

Of course it’s a job description and not an actual proper write up. Walmart’s engineering culture is _really bad_.

why would you get mad at a job description on a careers page for describing a job?

Would love to hear some evidence of this. I've never worked there (well, not true, I did in the 80's as a stock person, cashier, and later inputted sales data into the mainframe in their back office), but I know nothing of their modern engineering culture.

I’ve worked there. It’s pretty awful. Their hiring process is seemingly designed to acquire candidates that outright lie in the interview process. These people never get reprimanded, simply “moved around”. Lots of people in management positions without any engineering experience because they promote to tech from other parts of the organization. They have disdain for overly “academic” hires and opt for those coming from technical schools and boot camps. That on its own isn’t a problem, but when paired with the proclivity to hire fraudsters - it means they don’t hire people who have an eye to detect such things.

I went to a university they recruited out of heavily so took a tour of their campus and know quite a few people who ended up working there.

At the time (late 90's, early aughts) they were basically a sweat shop. I had one person working there tell me they got sued by one of their developers after said developer broke down their salary into hourly and were making under minimum wage due to all the overtime they were working.

The tech was pretty cool (back then, can't speak for today) but I would never work for them.

Reply on news.ycombinator.com