WalmartLabs open sources the application platform that powers Walmart.com | Hacker News Reader

WalmartLabs open sources the application platform that powers Walmart.com

techcrunch.com

220 pointsYhippa72 comments

My blog post that talks more about the release: https://medium.com/walmartlabs/introducing-electrode-an-open...

A lot of folks have debated whether there is a need or a place for a "Rails for Node" platform that brings together a set of blessed libraries that work well together and a set of tools for rapidly building software with that set of tools. In that sense at least, Electrode seems to have some structural similarities to Rails.

Do you see Electron as potentially having a Rails-like role of bringing together libraries and packages into a more "plug and play" development environment for people who want buy in to comprehensive Node development platform?

Internally, we do have a set of blessed libraries to keep the site and teams unified. To ensure we don't go stale, we usually have a team or so experimenting with something that we can bring into "Electrode Core".

Externally, for the "Electrode Core", we will continue to maintain a set of libraries that play well together, but I expect to see people doing their own thing. I see what we are doing with core as more of a jumping off point for your app and setting you on the right path for managing many apps/components (with our notion of archetypes that manage your dependencies instead of top level app) and than people will mix technologies that they want.

We will continue to maintain our list though and manage dependencies through archetype (I think archetype is a powerful idea).

"Electrode Modules and Electrode Tools" are independent to the rest of Electrode. You can use these modules/tools independently of Electrode Core.

I'll grant this is a subtle point, but why did you choose to use the word "archetype", borrowing a term to describe what most understand to be generated boilerplate code? Why not call it a framework, or an app generator? Why did you, collectively I will assume, choose to assign a new term to an existing concept?

The archetype terminology comes from the use of Builder; you can get a rundown of what an archetype really is here, independently of its usage in Electrode: https://formidable.com/open-source/builder/

Framework and app generator are really quite far off from what an archetype provides. An archetype is like an npm module template, but not one that is used to generate code files and then discarded, but more like a live "superclass" of a module; think inheritance but for npm modules.

The basic idea is that if you have dozens or hundreds of similar modules (for, say, React components), with similar build + testing needs, you can use the same archetype for all of them. A generator like yeoman could spit out boilerplate for you, but then what if your boilerplate changes? Your packages quickly get out of sync with your latest best practices. Having some components on Babel 5 and some upgraded to Babel 6, for example, was a real pain. You have to go through your dozens of packages and bring them up to date. Archetypes on the other hand are intended to be encapsulated boilerplate that packages basically "inherit" from at runtime.

Hope that helps!

That was really well crafted, thank you. Much easier to digest than some of the docs. Though, I'm still not convinced it warrants a new way of describing a collection of patterns for various concerns, e.g. a framework.

Thanks for the feedback about the documentation. We will work on improving it, especially around archetype.

Exogen, hope we can use what you wrote :D

Hey Alex,

I have a quick unrelated question. Why is Walmart.ca contracted outside if you have walmart labs? Well at least the front-end. Hope it's okay to ask you that.

Your question is completely fine to ask, but I unfortunately don't know the answer, sorry.

On a side note, just fun info, I know that Walmart.ca will be moving to electrode eventually, we don't have a date on the roadmap yet. We've built Electrode to be able to reuse the same components while maintaining a different brands styles, content, etc

i believe i had read in a medium blog post that the .ca backend was build on akka/scala

The backend is built with Oracle Commerce (ATG), the frontend with Play Framework (so Scala and Akka).

Source: http://www.nurun.com/en/news/nurun-launches-redesigned-trans... plus private conversations with people who have worked on the project.

Hi Alex. This follows on another announcement from Walmart around OneOps: https://github.com/oneops

However I don't see these projects getting any sizable traction from the open source community. What's the point of releasing them if WalmartLabs isn't putting any investment into marketing to developers in order to get these projects to grow in adoption?

Thank you for the question.

I think that WalmartLabs is investing in OneOps and investing into marketing it to developers. I know our OneOps folks have been at the OpenStack conferences recently and are doing their best to engage with the community.

Regarding Electrode, we will do our best to market to developers and grow adoption. I'll be at multiple conferences in the coming months, have talked in meet ups, writing blog posts to go more in-depth. I'm not in marketing or PR though, so any tips are always appreciated :).

Does this handle SEO well?

Because of our focus on server side render, we deliver the browser HTML - you should be good to go with SEO :).

Walmart.com uses it and we care deeply about SEO!

Clarifying because I was confused at first: this is the React/Node.js tech stack that the website is built on, not the ecommerce platform that drives it.

Yeah, I was hoping to see some very cool Clojure code.

Given Walmart's deep contributions and investments in the Backbone ecosystem [0], which were impressive in their own right, I'm even more impressed that they seem to have switched to a React-based front end stack in a few years. That seems pretty agile for a company the size of Walmart.

[0] https://github.com/walmartlabs/thorax

The switch of creating this platform, changing the walmart.com tech stack, and open sourcing it is less than a year :). Has been a wild ride!

Although it is great that this happened and I applaud open source contributions, especially by bigger companies, I wonder what the reasons for such a rewrite were. And why the online platform manager approved it. People often try to convince me of shiny-new-thing-39432 and sometimes we try on a non-risk project and usually we find out it's the same or (more often) worse than what we used before. I like React and theoretically (not practically unfortunately) also Node but if I already have a site built in something else the web team would have to come up with really over the top reasons (and prove them after the roll out!) to just dump that investment.

Probably a total redesign.

In fact design-wise, the React-powered site is identical to the previous one.

Great question, we definitely took a hard look on if we should transform our tech stack or not (and decided to do it) - to really answer this question though, you would need to understand our previous stack, what was holding us back, and where we wanted to go. I'm not sure if reading about our previous stack and why we moved is an interesting blog post or not :).

I believe it would be an interesting blog post - the amount of sheer "yeah we need xyz tech for more oomph" decisions made within business sector by managers who are not truly capable of judging the implications of said tech is quite something.

> I'm not sure if reading about our previous stack and why we moved is an interesting blog post or not.

It definitely is. There is a serious business cost to front end trendiness, so it's good to hear the legitimate business cases for these switches vs. "it's what the devs want to code on."

Well I think a blogpost like that would shine a light on this kind of issue from one of the bigger companies in the world. It will be good for the image as well as most believe companies like Walmart are a big layer of stodge, like many other enterprise, where MBA CIO's say no to everything and when you finally get approval, the Oracle head DBA makes sure it will not work anyway.

http://www.electrode.io/index.html

> "Electrode is a platform for building universal React/Node.js applications with standardized structure, best practices, and modern technologies baked in. Electrode focuses on performance, component reusability, and simple deployment to multiple cloud providers—so you can focus on what makes your app unique."

Sounds like the missing piece of the puzzle everyone's been waiting for?

Isn't this what Rails did/does?

Rails will load the above the fold content and render server-side, while allowing the rest to render client-side, do component level render caching on the server, and allow for client and server rendering with one codebase?

The complexity isn't likely completely free.. but could be incredibly useful for a lot of universal rendering scenarios... the above the fold content loading, and the component render caching server-side is particularly interesting.

Now if Walmart.com just didn't look quite so hideous... but that's another issue...

The tech stack for electrode is definitely interesting, and looking forward to playing around with it.

I hope you like it! I personally think that the above the fold content, server side caching, and redux-router-engine will make a positive impact on the server side rendering react community. We saw HUGE gains in performance using these.

We are working on a blog post that goes through the performance numbers we saw on the live site.

I also think Electrode Explorer will be popular for bigger companies that have many components.

http://www.electrode.io/docs/electrode_explorer.html

P.S. I'm from the Electrode team.

Thanks for the reply, yeah, from what I've read and seen so far it definitely looks interesting... And seems to resolve a lot of the issues I had very early on with ismorphic^wuniversal rendering back when Yahoo's fluxible app had just come out. The render caching seems very cool as well, making up for the general double-render for fetching data triggered from the component.

Right now at the day job I'm stuck in angular 1.x hell, though my contract is up in a couple weeks and looking for something in the React space again. Too much angular work in Phoenix to avoid completely though.

Aside: wish I could get a hold of a hiring manager there if you have space for new people, have never even gotten a callback from walmart labs, and would love to work for your org.

Your contact information is in your HN profile, I'll have our recruiters ping you with the opportunities we have available @WalmartLabs.

Added more contact info, thanks... I'm @tracker1 on github and npm as well.

This is somewhat surprising, in a good way. A lot of large companies are pretty risk averse, and this prevents them from releasing the source for their shopping platforms into the wild. I'm sure there are some differences between the open source version and what they actually run.

Is this the complete platform? What's held back? I would think the analytic tools for usage and sales, how they store user data, and perhaps whatever they use to build profiles about individual users' shopping behavior. But that's just speculation.

Electrode is just the front end platform. We also have OneOps for our app server lifecycle management. Test Armada for automation.

I don't know of anything lower in the stack that was open sourced than what I mentioned.

The stuff we didn't put into the Electrode platform that is integrated internally, off the top of my head:

1. Integrations to our global config manager, that allows us to turn flags on/off in real time.

2. Integrations to our A/B Testing Tools

3. Integrations to our content management

4. Logging/Analytics/Metrics Integrations

5. Our CSS styles and Icons

As you can see, almost everything is open sourced :)

For what it's worth, the walmart.com websites takes forever to load (> 10 seconds) on my single core laptop in Firefox.

edit: I should also state that my OS is Ubuntu.

Same here, 10ish on OSX with Chrome.

Antergos, i7, chrome, 6 seconds. I'm not thrilled, but I applaud them making a step forward for the community.

DOMContentLoaded: 1.48s document.ready: 1.88s

MBP Mid 2014, OSX 10.11, Chrome 53

It loads pretty quickly on my iPhone, about 1 second.

Check out the network timeline, this because there are images loading and analytics events firing. Totally independent of whatever front-end framework is used; even the most efficient vanilla JS won't speed up image loading...

tldr - http://www.electrode.io

I use Hapi.JS everyday and it's a bliss. Will check this out.

Looks like it's based off Hapi for the server component too. Hapi has been a dream to use, and this looks pretty well documented too.

(Seems Hapi.js is built in?)

http://www.electrode.io/docs/what_is_electrode.html#features

Electrode Boilerplate comes fully loaded with the best technologies available:

    • React is an awesome JavaScript library for building user interfaces, created by Facebook.

    • Redux a predictable state container for JavaScript apps. #game-changer.

    • React Router is a powerful routing library built on top of React.

    • CSS Modules a CSS file in which all class names and animation names are scoped locally by default. Fixes the problem of the global scope in CSS. #winning

    • Universal rendering built in.

    • Webpack a powerful module bundler.

    • Webpack Isomorphic Loader a powerful tool that makes NodeJS require understand files such as images for SSR.

    • Babel transpiles ES6 + 7.

    • ESLint a pluggable linting utility for Javascript.

    • Mocha a feature-rich Javascript testing framework.

    • Enzyme a Javascript testing utility for React, created by airbnb.

    • TravisCI a continuous integration service to build and test software projects.

    • Gulp a Javascript build tool that lets us automate tasks.

    • Yeoman a Scaffolding tool for modern webapps.

    • History a Javascript library for managing session history.

    • Bluebird a great Javascript promise library.

    • Electrode Confippet Confippet is a versatile and flexible utility for managing configurations of Node.js applications.

    • Electrode JWT CSRF Cross-Site Request Forgery (CSRF) protection with JWT.

    • Electrode-Redux-Router-Engine An Electrode routing and rendering engine using react-router and redux.

    • Component Caching Optimize React SSR with profiling and component caching.

    • Electrode-Server A configurable web server using Hapi.js on top of Node.js.

    • Electrify Tool for analyzing the module tree of webpack projects.

    • Electrode-Docgen A custom metadata extractor for the Electrode framework, automates component documentation.

That is correct, but please take a look at my other comment that explains where we are going with the hard tie to Hapi.

Yup, we are big fans of Hapi.js and that's what we use internally for walmart.com, but one of our goals in October is to disconnect the tieing of Electrode and Hapi.js - that way you can use whatever web framework you want with Electrode. Internally, we will continue to use Hapi though.

That is great to hear. I much prefer Express to Hapi, and I already have extensive APIs built on Express. I'm certainly not alone in this.

Yup! People have their preferences! We are working on it this month.

Until then, please check out Electrode Modules, they are independent of Hapi or Electrode Core:

http://www.electrode.io/docs/stand_alone_modules.html

So good. This looks like a great platform. Great work!

I have tried on multiple occasions to get started with react and every time the large amount of setup prevents me from getting anywhere useful. After looking over the docs I am somewhat hopeful that this might bridge the gap.

[Create React App](https://github.com/facebookincubator/create-react-app/blob/m...) simplifies setup greatly. I'd really recommend giving it Ila try.

https://github.com/Jon-Biz/simple-static-react

>The intention is to provide a basic boilerplate setup that doesnt inundate the new user with complexity, and stays out of the way of the more comfortable who just want an ES6/React environment to bang out an idea. It's ready to set up on your plain old web server, or your Firebase or Github site, or for building the front-end for your Wordpress API, with a minimum of fuss and configuration.

There are flavours that get you a simple router, a firebase backend, and Virtual Reality bindings, if you need those.

Hope you find the docs useful. We try to walk through building an app together.

We are working on breaking down everything inside our app for people who are starting off can get a jumping start, but than read through everything that's inside the app to understand better what's happening underneath the hood (such as, what's in the webpack config).

I second the recommendation for Create-React-App. It's explicitly intended to be a no-config, no-setup, Just Start Writing Code tool with a great developer experience.

Also, I keep a big list of links to high-quality tutorials, articles, and resources for React and related topics, at https://github.com/markerikson/react-redux-links . It's a great starting point for anyone trying to learn the ecosystem.

If you're interested, I am in the process of writing a react ebook, and I'm trying to focus on being very simple and explaining every step along the way. In fact, the whole first half of the book doesn't use a server at all (it uses script tags like https://facebook.github.io/react/docs/tutorial.html).

If you (or anyone else) is interested, I'd be happy to give you the chapters as I finish them, and would love to hear any feedback you have about it. Just send me an email (in my profile).

What happened to Clojure? I thought Walmart Labs was one of Clojure's flagships.

Clojure is used at Walmart Labs, but Walmart Labs is pretty big - there's a lot going on, and a lot of different tech.

I just tried out the Electrify [0] package on my codebase. It visualizes your webpack output. Very helpful!

[0] https://github.com/electrode-io/electrode-electrify

What are the server memory consumption numbers for this ?

How is this different from Hypernova [0] ?

[0] https://github.com/airbnb/hypernova

To any Walmart developers reading: thank you for open sourcing this!

However, there's something I'd like even more than your time spent on this project. Currently, walmart.com will indicate to customers that there is inventory available in-store that there sometimes isn't. This can lead to a terrible customer experience when you order something that runs out of stock after you order it, and your order is just cancelled. It seems like there should be the ability to have it backordered at the price at which you purchased. Could Walmart instead always honor the price at which an item was purchased online even if stock in-store runs out, and give the customer the option to cancel the order or get a 20% discount on any item if you don't want to get it on backorder? Right now this can be done, but it requires the store manager's approval, which can involve having to file a complaint, which seems completely unnecessary.

Very cool. I've watched a few of the videos that the Walmart guys did on various topics (really scaling, RUM, etc) and they clearly know their stuff.

Curious; is this stuff being developed at the location out in San Bruno (in front of Youtube)?

The Application Platform team (folks who created Electrode) is mostly based in the Sunnyvale office.

Archetypes? Gulp? Welcome to the age of vapidity, where Meaning is pastoral and abstraction sacred.

Ironic given that your comment is the most vapid in this entire thread.

Completely valid. I should have been more direct, but that wasn't the intent. Since you took the bait, I take issue with the seemingly flipant manner in which words are assigned new meanings, and existing concepts are relabeled. While the concept of archetypes is intellectually really fun to consider, it makes more difficult the ultimate project of speaking a common tongue. This is not an orthodox point of view, this is primarily economical. Far too much cognitive effort is required to learn new paradigms that turn out to be fresh takes on old ones, so best to identify them early and move on. But we should be critical as a community to self-regulate, as the marginal utility of each moment of learning is weighed against a million other potentialities.

Happy to receive feedback, do you have suggestions on what we should call it?

You might do just fine substituting archetype with template. The readme for https://github.com/electrode-io/electrode-archetype-react-ap... reads: "This "app archetype" provides for common patterns across all app projects so that each app project can standardize on common development behavior and patterns. Its essentially pre-made patterns for build scripts." "A pre-made pattern" sounds a lot like a template. The majority of the documentation focuses on applied concepts, with familiar terms. The only reference to archetypes is found in the first paragraph of What is Electrode, and I think it distracts and confuses more than it helps. If it is a core concept, then it might be good to link to https://github.com/FormidableLabs/builder, but I don't yet see where or if builder is included as part of electrode.

My experience is that Walmart's website is truly abysmally poor. Releasing their software is likely a scourge upon other commercial vendors who might inadvertently try to imitate them.

Prices take longer than the rest of the page to show up and occasionally you will get a Price: [NaN]

Reply on news.ycombinator.com