Cisco says 90% of internet traffic goes through Erlang-controlled nodes | Hacker News Reader

Cisco says 90% of internet traffic goes through Erlang-controlled nodes

twitter.com

304 pointsokket93 comments

It is used on the control plane.

Netconf (rfc6241) operations un most routers (not just Cisco) are handled from an Erlang application.

Some years ago the best netconf implementation was from tail-f systems. Cisco and Juniper licensed the software from them...

... And Cisco bought the company for the Netconf and orchestation/automation part.

"1.5B people rely on Hack every day [because Facebook runs on Hack]."

http://hacklang.org/

Even if I get your point, I think your comment is still an unfair comparison:the kind of processing activity is different in both processes, also the system and platform requirements etc also the requirements on the data, if some Facebook page is not loading it is okay to just reload but if an internet node collapses might have more consequences (all them are already foreseen and have alternative paths)

It’s about the same, really, both in severity and impact to users.

Routers fall over pretty regularly if you have enough of them so you already have things like MPLS FRR to deal with that.

Sounds like this is for netconf, which is a provisioning system. It failing is more akin to chef crashing on a server from a cosmic ray. Kinda sucks, but you restart it and probably no harm done.

Cisco IOS, NXOS, IOS-XE and IOS-XR are C/C++.

I wager there are some small python/bash/perl etc. scripts in use in Cisco firmware, tomorrow headlines - "World runs on Bash! Apocalypse is nigh!"

Cisco gear has enough problems regardless of programming language. They could start with not hard coding passwords. I will never trust them again.

Versions of IOS had embedded scripting available via TCL. It's likely that's the language used internally too.

Yeah, if you drop in a Linux shell on their various platforms you can find shell scripts dotted around. The boxes run Linux so that isn't surprising really.

Linux runs the control plane.

The programmed routes go into hardware on the bigger systems, and either way there is no Linux kernel routing involved at all.

I didn't say that Linux was involed in the forwarding plane or any routing. I was responding to Yihazi's comment that shell scripts are likely in use somewhere. This entire thread is in relation to the control-plane as that is where Cisco use Erlang. Scroll up much?

Edit: If I'm not mistaken I think Linux is actually used in some platforms for routing the out of band management interface, but not transit traffic.

100% of internet traffic goes through drivers written in C.

Traffic going through mainframes or Apple devices surely isn't through drivers written in C.

Mainframes, because they have saner systems programming languages than C.

Apple devices, because part of the network stack requires their Embedded C++ dialect.

> Mainframes, because they have saner systems programming languages than C.

Are you claiming that PL/X is saner than C? What is your reasoning?

Less UB, bounds checking, proper strings and arrays?

The PL/8 papers related to IBM's RISC research are quite interesting, before they decided to go with UNIX on RISC instead, as the best way to recover their money, thus switching to C.

Also not all mainframes were written in PL/X variants, there is also NEWP, which already had the notion of unsafe blocks in the late 60's.

PL/X and PL.8 may have various technical advantages over other languages, but it is hard to judge given IBM's long-standing refusal (going back decades) to publicly release compilers and other tools for them. From a software engineering viewpoint, developing your software in proprietary/in-house/unreleased programming languages – much of z/OS is written in PL/X, while some of the firmware/millicode is in PL.8 – seems to me a rather bizarre decision – not to mention other secret dialects of PL/I IBM has cooked up over the years, such as the PL/MP and PL/MI that were used on the AS/400s, and PL/DS on the IBM 8100. Why not release the languages, try to build some sort of community around them – wouldn't that be good for their long-term health, and the long-term health of the platforms written in them?

(Probably it is way too late for this, but if IBM had been willing to publicly release PL/S tools back in the 1970s, or at least not use their lawyers to stop others from independently doing that, PL/S and its descendants could have become much more significant systems programming languages.)

Those were the rules of the game back in those days, languages were tied to OS vendors.

UNIX and C only became widespread, because contrary to IBM, DEC, Wang, and many others, AT&T was forbidden to sell UNIX and they started offering it alongside source code for a symbolic price of $100, which was like being free given the prices of other systems.

In any case, that doesn't make C technically better than the alternatives outside AT&T walls, it was just cheaper to get hold of.

The folks calling for IBM to release a PL/S compiler were not demanding IBM hand it out for free. They would have been satisfied by IBM offering it for sale on usual commercial terms, for a similar price to IBM's other compilers (PL/I, COBOL, FORTRAN, etc).

But IBM wasn't willing to sell its customers a PL/S compiler.

One reason was that (up until the OCO announcement in 1983) IBM gave its customers the PL/S source code, but not giving them a PL/S compiler discouraged customers from changing the source code, since they couldn't recompile it. Customer modifications had to be made in assembler, which was a lot harder to maintain.

Another reason was to make life harder for competing vendors selling IBM-compatible mainframes, such as Amdahl, Fujitsu and Hitachi.

Now we are also starting to see national networks passing all their traffic through drivers written in Lua. Couple of talks about this at RIPE recently e.g. https://ripe76.ripe.net/presentations/105-lw4o6-ripe.pdf

Traffic doesn’t even reach the driver. It’s all ASIC.

It does on the endpoints.

Both of these statements are 100% true.

This isn’t very surprising, this is exactly the kind if stuff Erlang was created for. It’s the right tool for this particular job.

The fact that Erlang was created for this job does not imply it is the right tool.

Well, sure. The fact it was created for this job, by some really intelligent people, who understood the domain space very well, and has proven itself over the past 30 years, is what implies it's the right tool.

The parent's point was simply that it's not that surprising that it's effective in this space, when it was targeting that space to begin with.

How does Go compare to Erlang these days? When would you choose one over the other?

Not sure where I would choose Go, but Elixir + Rust is an extremely capable combo for BEAM (Erlang VM) projects.

Do you have some recommended reading on why and how to combine Elixir and Rust?

Try this: http://hansihe.com/2017/02/05/rustler-safe-erlang-elixir-nif...

Erlang is slower, and beam is not relevant anymore with kubernetes and the like which are much more powerful and language neutral.

I keep hearing stuff like that but it’s not even close to accurate.

BEAM and Erlang give you tooling that you can’t get from any other language, regardless of how it’s deployed. Using Kubernetes doesn’t make up for any of that...it just gives you a way to deploy containers.

What tooling please give an example.

strictly ordered message queues between processes

For a detailed example, just see the previous posted. There’s a lot.

https://news.ycombinator.com/item?id=13497505

I dont know whether this is a good thing or not.... The best nuggets of information on HN are at the mercy of a good soul digging up the link and dropping it on a new thread.

You'd think that this knowledge retrieval/storage problem could have a more elegant solution built by the HN folks. Example solution: Topic X stored in category, ranked to the top, its next 'commit' or successor is chained on kind of like how unroll works with twitter.. but with long HN threads instead. Think a 'git tree' of knowledge descending down, with merges too maybe...

Engh.. maybe thats part of the charm of HN... the old school style message board.

The machinery of the BEAM is in-process (and between nodes when you need it). Kubernetes and the BEAM VM address different levels of the architecture.

Apples to oranges.

I believe GoLang supports efficient sharing of large data-structures between cores of the same processor, whereas ErlangLang does not.

> I believe GoLang supports efficient sharing of large data-structures between cores of the same processor, whereas ErlangLang does not.

AKA Go does shared-memory multitasking, while Erlang does shared-nothing multitasking.

Yes, and when used in an immutable way, the GoLang approach is very powerful and efficient, because it allows one to pass large (parts of) immutable data-structures without copying.

How do I declare an immutable data structure in Go?

An unexported value type provided through an interface is immutable. The "context" object uses this. But it isn't really a solution for immutable value programming.

Its immutable everywhere, or just outside the package that declares it?

Outside the package that declares it, though if you have it as an interface value you'd have to do work to penetrate it.

Since it's not like I'm trying to sell this as a solution to all immutable data requirements, I didn't worry too much about that detail. In the end it's all about effort, anyhow; with the right invocations in Haskell I can mutate "immutable" data just fine. So even an interface in the package that declared it is still nontrivial protection against accidentally mutating it.

What about lists? Eg, an immutable list of packets.

I don't think thats really true about Haskell, unless you are talking about C code you've invoked through FFI in which case all bets are off in every language.

Isn't it a lot simpler to just say that Go has no feature support for immutable data? The fact that you can hide accessors in a package is small comfort if I have to read all the source code in that package to know the actual semantics.

AFAIAA, the only immutable core type in go is string.

"ErlangLang"?

While 'ErlangLang' is obviously wrong, I should point out that the '-lang' part of 'Erlang' is not related to the root word 'language'. Perhaps the name 'Erlang' was chosen because it already ended in '-lang', perhaps not, I don't know. But it was quite likely named after the Danish mathematician Agner Krarup Erlang.

IIRC "ERicsson LANGuage" because it was created for networking stuff at Ericsson.

EDIT: Yep, we're actually both right! From the Wikipedia page:

> The name "Erlang", attributed to Bjarne Däcker, has been presumed by those working on the telephony switches (for whom the language was designed) to be a reference to Danish mathematician and engineer Agner Krarup Erlang as well as a syllabic abbreviation of "Ericsson Language".

Previous discussion https://news.ycombinator.com/item?id=13497505

Thanks. Did a search before asking but couldn't find anything. Sorry to rehash!

Easy enough to miss. That one is Elixir and Go but for most purposes Elixir/Erlang/BEAM are fairly interchangeable.

No. Thank You. Go vs Erlang/Elixir back n' forths are the reason I read HN comments!

No change from when Go was released AFAIK: Go has no reliability/failover story (whether locally or in terms of distribution), uncaught panics will bring down the entire system, and the runtime has no support for routine hierarchies.

So I guess you'd choose Go if you want to build a local service — especially if it has some amount of concurrency (e.g. network daemons) — or CLI tools (fast startup). You'd use erlang if you want to build a system which really should not die, especially if it so should not die that you're willing to spread it over multiple physical machines.

Not an expert, nor to take anything from Erlang (was amazed at how robust RabbitMQ was first time I stress tested it) but fault tolerance seems to be a part of OTP. To me, k8s ~ OTP, GO ~ Erlang, and Erlang syntax is a bit easier than GoLang, but might just be the way I was brought up.

OTP is the standard distribution & library of Erlang, and most of the fault tolerance is either that or the core Erlang runtime e.g. supervisors are part of the standard library[0] and the underlying systems are basically built into the language[1].

Distribution is likewise built into the runtime and language, most of the distribution-related BIFs are not only part of the "erlang" module/namespace but also part of its "prelude" (auto-imported in all modules by default).

So no, it is absolutely nothing like kubernetes in relation to go.

[0] http://erlang.org/doc/man/supervisor.html

[1] http://erlang.org/doc/reference_manual/processes.html#id8861...

I think parent was referring to similarity of k8s to OTP from user's prospective..

People just want k8s to be that marvelous, to have strong supervision tree architecture. I want the same for systemd too. But sadly it's all mess.

> [...] but fault tolerance seems to be a part of OTP.

Yes, mostly, but the thing is that OTP was possible at all because of how the portions provided by the language and the runtime play together. You cannot replicate the OTP as it is in Go because Go's runtime is lacking and its communication model doesn't match.

The syntax you're looking for is k8S:Go::Erlang:OTP

Once you learn Erlang and OTP, you will start seeing everything as a poor re-implementation of the same concepts.

So now that we've established that massive chunks of the internet run on every significant language/platform, is everyone properly validated in their choice of language?

"Controlled" is the key word here. It means the network controller is running erlang. The actual switches and routers run C.

In core routers and switches, CPU is the slow path.

Brunt of the work is done in ASICs.

People who've never worked with a chassis-based router might find the distinction between control engine and the linecards/fabric hard to understand in an abstract sense.

It's more readily apparent if you've taken an empty Cisco ASR9010 and started from a blank slate configuration with a new pair of RSP440 route/service processors (the control plane) and then incrementally added linecards/interfaces.

Mid to large sized routers are designed in such a way that everything is either N+1 or 1+1 redundant and hot swap, it's like having a four engined airplane you can replace an engine on in mid flight.

You are correct that on core routers, the brunt of the work is done in ASICs, and CPU is a slower path.

People should know that Cisco takes this discrete fact and then extends and over-hypes it. The differentials between ASICs and CPU are not to the extent that Cisco make it out to be, where it seems like CPU is not even an option at all on most of the network. Also, Cisco has an overbroad notion of how necessary their particular routers and switches are in your network, when in many places in the network, off-the-shelf hardware, CPUs and software would do fine (for much cheaper).

For the past few years, people have been developing software solutions like quagga, BIRD, XORP and OpenBGPD/OpenOSPFD. They have seen some heavy production use ( http://www.openbgpd.org/users.html ). OpenBGPD even teased out problems in Cisco routers - some standards-compliant packets OpenBGPD sent out would crash Cisco routers.

People appreciate the openness, the flexibility and the extensibility and the price of these solutions.

ASICs are faster than CPU in the core of the core, but don't let that fact get you ensnared in Cisco marketing hype and FUD. People are paying a lot of money to Cisco on the low and medium end for equipment that could be replaced much cheaper with off-the-shelf hardware and free and open software.

It's more about forecasts and assurances. If you're financing the network that is currently doing X traffic, but expect it to grow by quite a bit soon, you're going to ask "is this solution going to cover everything?"

The Cisco guy is going to sure 'Sure! We can cover everything!'

The OSS off the shelf guy is going to ask 'Well that depends on the growth.'

At that point, most of the businesses just throw some money at Cisco so they don't have to worry about anything.

> The OSS off the shelf guy is going to ask 'Well that depends on the growth.'

Though this "depends" is a quite high bar in reality. I know a regional internet provider that runs purely on software routers on commodity x86_64 servers. You need to be a quite large company (or operate in a very specific domain) to ever hit this level of traffic.

The point is that the OSS off the shell guy is far inferior as a peddler than the Cisco guy... hmm.... I need to redo that sales presentation tomorrow....

I would argue that the "brunt" is borne by the thing with intelligence i.e the control plane.

The control plane which is some general purpose CPU running Linux/VxWorks(Cisco)/FreeBSD(Juniper)is what builds the FIB and actually populates those ASICs in the forwarding plane. The ASICs are basically just lookup tables that rely on data pushed to them by the control plane. In other words they rely on computation/preprocessing done elsewhere.

Yeah that's true. However the ASIC is programmed by the control plane, and that control plane exclusively runs on C.

> It means the network controller is running erlang.

network-controller is a recent phenomenon and it would be quite interesting to see if this is what was meant. fwiw, a bunch of ericsson h/w for cellular access ran erlang in the control-plane f.e. mme/sgsn etc.

Not network controller in the "SDN" sense of the word, but more the management plane of the devices. In essence, the former Tail-f company (acquired by Cisco) had written a daemon (ConfD) that network equipment vendors would OEM to integrate in their Network OS'es. This daemon is still available and used today and the lessons learned from using YANG data models to describe/abstract the underlying control and data plane settings have now been applied to abstract network services at a level higher in the NSO product (also Erlang).

D: I work at Cisco

afaik, conf-d is not in erlang though, fwiw, it does have erlang bindings. most of the config of these 'boxen' happen via cli's by network administrators.

what conf-d/tail-f brings to the table is the distributed nature of 2-phase commit to device configuration, as well representation of operational and runtime states of these devices. which has some appeal to it.

source: have worked for csco quite a number of times :)

edit: slight clarification on conf-d capabilities.

> afaik, conf-d is not in erlang though

Most of the actual Configuration Database (CDB) is written in Erlang. Which also explains the statement made Cisco during the conference.

> what conf-d/tail-f brings to the table is the distributed nature of 2-phase commit to device configuration, as well representation of operational and runtime states of these devices. which has some appeal to it.

Another thing that has quite some appeal, I think, is that by using a single YANG data model the daemon actually can synthesize the structures of the CLI, Web UI and NETCONF northbound API's.

Again, biased because of Cisco employment.

Which is the right way of divying up workloads with something like that. You use the high level language for 'command and control' and write specific bits in C or - in their case - maybe even have chips for specific tasks that get controlled by something higher up.

Wow! Major kudos to Erlang. Have found it an enjoyable language to code. Well Erlang and have been also enjoying learning Rust.

Meanwhile official Erlang website's certificate is invalid.

It's an old certificate no longer trusted by browsers. Feel free to email them and let them know. It's probably a better use of your time than leaving snarky HN comments on unrelated stories.

and erlangs BEAM is written in C? so... it all doesn't mean much. Though the wording here sort of implies erlang is playing a supervisory role?

It is used for control plane same as Ericsson

Everything eventually gets executed in the form of a machine code, what's your point exactly?

Erlang / Elixir give you a ton of safety guarantees inside the VM. Something many other languages were never bothered to offer.

> and erlangs BEAM is written in C? so... it all doesn't mean much.

Huh? That's an incredibly stupid comment.

Another pointless discussion about programming-languages. Can't believe people still don't understand that it is not about language. It is about the product.

The only thing that powers internet is smart people. specially those that don't spend their time on pointless discussions and instead they can focus on analyzing the problem and finding the right tools and solution for it.

Please don't call yourself software engineer or developer if you have a programming language bias.

What do you mean by bias here? Some languages are practical, some more/less efficient. The BEAM was BAD in 2005. The language is ugly and this has hindered adoption in a myriad of applicable areas for decades. Elixir was a wrapper for people who already knew Erlang. Languages don't live in a vacuum, they are for people and organizations (where decisions are made about non-existent people) to use. Erlang could be done without being so horribly ugly and with more syntactic specificity (give atoms an identifying symbol). You have to consider the pareto distribution when thinking about design, not just how intelligent some programmers naturally are.

Really? I'm writing Elixir just fine coming from Ruby...

I can use it just fine too. It's not unusable, it's a barrier. The defense of arbitrary syntax as no-consequence is misguided and ultimately irresponsible (in a lazy sense).

Jack, my comment was not against Erlang or Elixir, but SOME of these comments here.

People start comparing different languages in different areas and use cases and instead of discussing about the main subject, keep saying oh no my language is more popular or that language is written in C so C is the best...

Elixir was a wrapper for people who already knew Erlang

Elixir was a wrapper for people who already knew Ruby.

Which is why most Elixir users come to it from the Ruby/Rails community.

It was created by Jose Valim, a Rubyist and Rails committer, who was looking to improve Ruby/Rails concurrency story [0]:

Back in 2010, I was working on improving Rails performance when working with multi-core systems, as our machines and production systems are shipping with more and more cores. However, the whole experience was quite frustrating as Ruby does not provide the proper tool for solving concurrency problems. That’s when I started to look at other technologies and I eventually fell in love with the Erlang Virtual Machine.

I started using Erlang more and more and, with experience, I noticed that I was missing some constructs available in many other languages, including functional ones. That’s when I decided to create Elixir, as an attempt to bring different constructs and excellent tooling on top of the Erlang VM.

[0] https://www.sitepoint.com/an-interview-with-elixir-creator-j...

Given the experience I've had with Cisco's SDN nightmare in recent times, this is not a great advertisement. :(

I guess something has to move packets from interface A to interface B.

Ha! yeah that's done in hardware. They are talking about the controller that sets the route tabels for the asics.

forwarding tables to be correct.

Routing tables are in the control plane.

ASIC's really only do "layer 2" forwarding. (this is not an entirely proper abstract, but close enough anyway.

Reply on news.ycombinator.com