Winnti: Hackers attacking the heart of German industry | Hacker News Reader

Winnti: Hackers attacking the heart of German industry

br24.de

257 pointshakantan148 comments

As of today we still use shared network drives for everything in major German company. We don’t use slack/irc/Skype/zoom/whatever. Phone calls and conferences(!) from the middle of open office is normal. Asana/Trello/Jira are not known at all. GitHub is paid for, but never used. I am single weirdo in multi department project using github ticket system. The code with prefix is copied for colleagues to project’s shared folder. PostIt tickets on the table works good enough for others. Talking about bugs is impossible since nobody knows what’s fixed and what’s not, the bugs have date in the best case. Everything else is done in Excel sheets using in house written scripts. They usually end in a mess since some people use German regional settings and other English ones. That’s state-of-art situation in very rich and big company today. I don’t see any possible changes in future. Old boy club fights all the time against proposed improvements. You can forgot topics like information security, phishing, being silent about work topics outside the office. Hackers are known from the American movies only.

On the other hand I also worked in opposite unhealthy paranoid environment. I was hired to design Ethernet camera, but Wireshark usage in their office was prohibited. Packet analysis was seen as the worst thing in the company. I quit after few months trying to explain, that I need to analyze the packets during design phase. I think, it’s very normal, that other countries abuse illiteracy of German industry.

Germany's tech illiteracy is a self-inflicted consequence of its pitiful salaries in this field, treating IT like a cost center that has to be outsourced to wherever is cheaper and companies' tradition of rewarding management incompetence over technical competence.

Consequently, Germany's most brilliant tech minds leave for The Valley, Zurich or London.

You reap what you sow.

> treating IT like a cost center that has to be outsourced to wherever is cheaper

As if outsourcing wasn't a common trend across Western companies "thanks" to globalization and the utter dominance of US-american neoliberalism.

> and companies' tradition of rewarding management incompetence over technical competence

Again, quite common - the "old" Soziale Marktwirtschaft moguls with decades-long visions would not let today's next-quarter-only shit fly for long.

> Consequently, Germany's most brilliant tech minds leave for The Valley, Zurich or London.

Care to have a source for SV and London? People avoid SV/USA due to the current President and London due to the Brexit uncertainity - in fact, whoever can flees from the UK before Johnson drives everything into the ground. Only correct point is Switzerland but that's not surprising since their wages run way, way higher than Germanys across the board...

> People avoid SV/USA due to the current President

_some_ people do. Others allow themselves different opinions to this president and even to their prospects in SV

> and London due to the Brexit uncertainity

True

> before Johnson drives everything into the ground

... but also here there are different opinions concerning the nature of this uncertainty

Aaaand some people care more about their life than they do their country's politics, thankfully.

IMO, politics and stuff is necessary, but I really wish we could argue /debate respectfully without the damn flame wars everywhere of late. It seriously puts me off.

The point is that the current Trump administration has a massive problem with immigration - that may or may not have consequences if you want to emigrate to the US. Or Brexit - if Johnson really delivers a no deal Brexit then UK economy is getting fucked over hard plus the right of free movement for EU citizens vanishes so you can get deported overnight.

Politics at this scale are life changing matters, and unfortunately with both the US and the UK governments it is clear that they are not run by competent people with a plan but by people who literally don't have any fixed opinion except nationalism - and that means that, as an immigrant, you're rock bottom of the ladder.

You are certainly entitled to your opinion, but other people may make a difference in what "is clear" to them.

I'm a German soon to be CS graduate and right now my favourites are SV and London. I'll try to get into a company that can sponsor L1 visa for SV first and then I'll try London.

The German job market is precisely as described by GP: you are competing all the time with remote people in countries with much smaller cost of living and the wages are super low compared to aforementioned places. Germany has great schools, and medical system and I think when/if I want to raise a family I'll come back. But until then I want to make money and you do this while it's easy to move around.

You can say this is a cultural problem but I think the problem is different: there is just no big software industry in Germany. It's mostly SAP salespeople, development of various custom ISV software and a small startup scene in Berlin. Most German companies don't need good software for the stuff they are doing.

There are companies in Germany who pay really well, especially if you take into account cost of living. Look around a bit before leaving the country.

Do you have any names?

Twitch has some engineering in Berlin, Amazon pays quite well, Siemens pays quite well, as do the various car companies (basically any company with a strong IG Metall presence pays 70k+ for developers), Snowflake has some engineering in Berlin too (I work for them, you can figure out my email address fairly easily and drop me a mail if you like). Startups in Berlin generally pay 60-100k depending on seniority. Median income in Berlin is something like 18k after taxes, so maybe 30k before taxes.

Thanks!

Ah twitch is German explains the poor UI - they really really need some UX

The Berlin office is very small. Most development happens in San Francisco.

BMW, for example. They pay boatloads.

As Portuguese doing consultancy in Germany, the market is way better than in other European countries.

As for the software quality, it is everywhere the same when the main business is not selling software packages.

Companies just care that their use cases are covered, no matter how it looks under the hood.

Of course Germany is a better market than say portugal. But that doesn't mean that I'll try to check the globally competitive offers first, to which Germany doesn't seem to belong.

How is Portugal on the cost of living vs wage scale?

I was under the impression that the likes of Lisbon are great if you earn above average wage, and the surf is awesome.

Tech in Lisbon not comparable to Germany?

Fellow countrymen might jump in to set me straight, as I have been away for too long, so I lack proper information.

Lisbon is a great city, but expect to live in the suburbs due to the high cost of renting and enjoy about one hour commute time and salaries are still below what we used to enjoy during the first .com wave (1995 - 2002).

Yes, tech is comparable, but not everyone can live in Lisbon.

There are smaller tech hubs across Porto, Aveiro, Braga and Coimbra due to their universities, but pay is even less than in Lisbon, although you get to enjoy better quality life and might afford to live on the city center.

But expect to do lots of overtime without any kind of reward beyond a "thank you", while in Germany those kind of situations are regulated, and in when it happens you have the support to complain about them. If you want to actually take that route that is another matter.

Naturally there are a few unicorns that are great places to work and do reward the extra mile, but they are the exception, not the rule.

This. Italy here, working with Slack, GitHub, etc. But my customers don't always care about the technical details. Companies with an IT staff usually do, the others don't because they don't know anything about those problems, except some scare about cookies and GDPR.

I could write software on paper and deploy by magic, it would be ok for them. And why not, they make money selling other stuff. Then a customer I haven't been working for a very long time calls me asking if I got a copy of their production VM, even many years ago would be ok (obviously I didn't) because they got hit by ransomware years after having stopped to do backups, maybe because it was too inconvenient.

come to Berlin, nothing of what you say applies there imo. Salaries might be substantially lower than in SV, but it's also far far cheaper. Eg. as a Senior 70-90k EUR is realistic in Berlin (without management), allowing you a very good quality of live in a city that is nowhere near SV/London in terms of rent and many other expenses.

IDK I'm not a very consumption oriented person. In the phase of life following my graduation I plan to save my money so "how much can you save" is the metric I compare places by. Yes, SV has a higher cost of living and I'm not an extremist like this very frugal guy who lived in a van for his mountain view internship (cool trick though!). But from my assessment even if you rent a room/condo, SV/London are still more attractive than Germany.

If your lifestyle is to spend most of your paycheck (which is a fine lifestyle btw, I don't want to be judgemental), then I think Berlin is much greater than SV. The fewer money you get is at the same time more powerful in Berlin. If you have children, even more so.

But you know you maybe want to buy a flat or house, but for that you need money. It's hard and takes years to use your SV/London wage to buy SV/London flats. It's hard to use your Berlin wage to buy Berlin flats. It is comparatively easy though to use your SV/London wage to buy a Berlin flat.

Germany is not unattractive for me on a permanent basis, but I think in the current situation I'm in, other places are more attractive.

Also I might be wrong, I'll conduct a final assessment once I graduated and have concrete job and wage offers and can do cost of living calculations and how much I'll be able to save after deducing all costs I expend.

> The German job market is precisely as described by GP: you are competing all the time with remote people in countries with much smaller cost of living and the wages are super low compared to aforementioned places.

This is not something I ever encountered. Are you sure you are familiar with real job markets? If you're looking for an interesting job (in Munich), feel free to contact me.

> I'm a German soon to be CS graduate

As someone who was in that position ca. 20 years ago and then moved to SV for almost a decade: I don't think I would have dare to make such a sweeping claim as you:

> The German job market is precisely as described by GP

As a completely new entry into the job market, despite plenty of real work experience (during the last two years of my study the study was "secondary", I managed to make all major university projects about things I needed to do at work anyway), how could I possibly have know enough? I know I never had that feeling.

I actually did move to SV immediately so I can understand what draws you, no argument there. Still, I returned after almost a decade.

I think you are waayyyyyy overvaluing your own experience level. Where does such confidence come from? You cannot know even 1% of the German IT job market given how many medium sized of importance there are. Even in the US there is a large number of software businesses nobody has ever heard of because they are in a niche. For example, I once consulted for a company in Fort Worth (TX) where software for giants like Walmart was written. I think even here in this forum very few people would have heard of that company.

Occasionally someone, often magazines, ask about-to-be-finished students where they want to work. Inevitably the top ten are a handful of major names. That says a lot about students knowledge about potential employers, which seems to consist of only a few names of the already well-known few big companies. It says nothing at all about the reality of the far more diverse job market and the myriad of interesting options at thousands of other interesting companies.

Example: Top companies to work for,

- Students of business: https://www.arbeitgeber-ranking.de/rankings/studenten/bereic...

- Students of engineering: https://www.arbeitgeber-ranking.de/rankings/studenten/bereic...

If you just drive through Germany blindly, without a map, you'll find company after company that would never be listed here because they are not a mass-market brand.

So by all means, do go to SV, it's certainly a great experience. Just don't overvalue your own experience, and question why you are so confident in your claims and generalizing your own very limited experience.

secondign this. I'm 45 yo, and by the time I entered the job market, I was a real good developer (sold my first commercial program at 16, made 3D engines at 18, etc.). And you know what ? I didn't knew anything. That's because software is a just a part of the equation : knowing the business, the people, the rules, the companies, your own needs in life, working with others, with management, etc. All of that you don't learn at school and you learn as you go. So if you can land a job in SV, just do it, but don't think that's the end of it :-)

Now, for my part, I came to the conclusion software is just a tool and, as that tool, I want to be used for things that matter to me. Since I think that the only thing that matters now are fighting poverty, climate change, I really wonder what I'm gonna do...

And I can assure you, when I was at your age, all my life was oriented to make one and only one thing : 3D engines for video games, which I did.

So you see, life is full of surprise :-)

Thank you both for your input. I don't want to leave Germany on a permanent basis. I'm looking forward to the German job market in 5-7 years, maybe more, maybe less, depending on how much I like it.

As for well-known vs not well known: usually bigger companies have more employee protections, more career possibilities, pay better, etc. In small companies you have more power and control, and to some this makes them more interesting. I think larger companies are better but maybe I'll be annoyed by the bureocracy and switch to a smaller company, idk :).

> usually bigger companies have

And again you make sweeping generalizations...

I worked at what was a startup whose name you likely know, during the dot com boom, you could go from not-long-out-of-university to "very important head honcho of XYZ department" going (or flying) to very important meetings with really important executives of major companies in no time. Try that at some major company. It's actually rather unhealthy - for everybody, although the drug feels good to you at that time.

From what I've seen, salary vs cost of living is rather crap in London because London is very, very expensive. SV makes sense if that is what you want to do.

I don't think that's the case. I believe the problem is that large German companies tend to be hardware companies. They're used to development cycles that take years and extremely conservative in adopting new methods. Management and company culture is not used to dealing with fast changing development methods (where fast is anything that changes more often than once a decade). Rigid processes for compliance with external regulations over time where adopted for internal rules as well, making any change a bureaucratic nightmare.

Salary for developers really isn't that much of an issue. In Berlin for example a developer gets two or three times the median salary easily. That's enough for attracting people who are talented enough to choose, e.g. git and JIRA over whatever crusty system of shared folders and zip files or IBM crap you'd see for projects in many companies.

Pretty much agree with this sentiment regarding hardware/culture.

Also the salaries are acceptable (imo) compared to fairly high paying US jobs if you compare real working hours (vacation time, real 40h workweeks etc.). With a family there's even more benefits. Cost of living also tends to be fairly low (with a high quality of life) compared to higher paying places.

High paying American bigcorps mostly have perfectly fine work life balance.

Getting double the vacation time (6 weeks instead of 3) when I moved from the US to Germany was very nice, but going back I'll have 5 weeks, which isn't too bad.

> Getting double the vacation time (6 weeks instead of 3) when I moved from the US to Germany was very nice, but going back I'll have 5 weeks, which isn't too bad.

I keep hearing this. Yes, you will get 5 weeks. But what about your wife, your uncle, your friends? In Germany they're all guaranteed to have the same number of days off, access to healthcare, etc. When you have kids you have a bunch of weeks/months off before the child is born and more after.

In the US, as long as you're young, healthy, rich and selfish, life is grand :D

Nitpick: the number of vacation days in Germany does vary a bit, though by much less than in the US.

Agreed, I'd love it if all Americans were entitled to 5 or 6 weeks of vacation, plus the various other benefits that are standard in most of western Europe. Unfortunately, the GOP exists, so that's not happening anytime soon.

But that's the point of US, no? It's the best country for the best people. So, the best people go to the US... (well, many at least... I'm still holding out... for now, I value European way of life but I envy US political & legal systems, in particular their freedom of speech).

You can can't have 300 million best people. You can't even have 100 million.

And once someone acquires citizenship, their descendants can't lose it, provided it's their only citizenship.

This is the funniest thing I've ever read on HN

> Salary for developers really isn't that much of an issue.

Yes, it absolutely is. And Berlin is the best example; we're even underpaid by German standards.

Many (senior) developers I know in Berlin get 80000€/year gross income. That is in the top 6% of incomes in Germany. (Top 6% income bracket in the USA would be 150000$/year.)

I don't even get 80k€/year and have a very luxurious and high quality life in Berlin.

Resource: https://wid.world/simulator/

The median software developer - out of nearly 1.3 million developers - in the US makes ~$107,000 per year for 2019. A senior developer should be up near $150,000 even outside the primary major tech cities. The top 10% tier of developers starts over $160,000.

New grads can start at 200K+ in the right companies.

You are using absolute numbers to make it look more extreme than it is. 107000$ are 94000€ at the moment. Considering the living cost in the USA, it might still be higher in the USA but not that extremely more

No, the numbers I quoted are correct. If they seem extreme, it's because they are compared to everybody else not named Switzerland. I'm using absolute numbers (what?) because those are the dollar wage figures directly from the BLS for software developers. Living costs in the US are not higher than in Germany. That is only true in the most expensive places like eg NYC, SF, Sea.

If you're in Chicago, Atlanta, Dallas or several dozen other major cities, cost of living is very reasonable in the US. Your healthcare costs are typically either entirely, or mostly covered by your employer if you're an engineer making $100k+. In Atlanta your all-in effective tax rate is under 30% at $110,000; in Dallas it's under 24%. What's the problem?

What do say Lawyers, doctors and Traditional engineers Get ?

The source may be sketchy but is all I found.

Average lawyer: 50k€/year [1] Average doctor: 75k€/year [2] Average engineer: 60k€/year [3]

[1]https://www.gehaltsvergleich.com/gehalt/Rechtsanwalt-Rechtsa... [2]https://www.gehaltsvergleich.com/gehalt/Arzt-Aerztin-Uni [3]https://www.gehaltsvergleich.com/gehalt/search?jobname=Ingen...

In Germany, with an 80000€ salary, what would be the monthly take home in your pocket? And I'm assuming they would take out taxes for health system, pension, vacation, etc.

In the US in an average taxed state, compare a $150K salary: After federal and state taxes, our old person medical and retirement (i.e. FICA), maximum retirement investment of about $19K, health insurance, disability insurance, dental, etc, one will end up with about $7200 a month or $86K / year in their pocket, which is roughly 42% taxes and retirement. This includes 3 weeks of vacation and 10 Federal holidays.

80k€ would net you between 40-50k€ after all taxes, health insurances and a basic retirement fund. The exact number depends on you having a family or not and many other factors.

In addition your employer pays an amount equal to about 10-15% of your gross salary into your health and retirement insurance. And this pay is on top of your salary. (Arbeitgeberanteil)

Holiday is always included with a minimum of 21 days a year for a full time job. And Germany has paid parental leave included if you start a family.

Based on what I have heard from friends in London and Germany salaries are actually a lot better in Germany. Especially once you account for cost of living.

This. Salaries of 70-90k EUR for Senior devs in Berlin are absolutely achievable and combined with the quite low cost of living in Berlin you probably have more fun money than anywhere in Europe as a developer.

But not nearly as much as you'd have in the US, probably.

Rents are a lot cheaper in Berlin. You don't need a car either. It's also a nicer city in general, I feel a lot safer in Berlin than in San Francisco.

German cities themselves are definitely much nicer than American cities. In fact, "much nicer" is an understatement, American cities usually feel like they were designed by people who hate cities and want everyone in them to suffer.

That said, US lifestyle/culture has its own set of advantages compared to Germany. German culture is surprisingly backwards for consumer software/internet stuff, for example.

You don't need to live in the valley. I make twice that much in Chicago which is very comparable to Berlin price wise.

And how does salaries (and work environment) in Chicago compare to SF/Berlin?

(Weather is probably comparable to Berlin though)

Pretty much guaranteed. Even startups pay more than that and GAFAM pays 150,000 to new grads and 300,000-350,000 to senior engineers [0].

[0]: https://www.levels.fyi/

Salaries for devs in the US are insane compared to the rest of the world, I don't think they can be taken as a reference, especially as most people do not have an easy way to just choose to move there.

Anything above high 60's is really a pipe dream in this current market.

If you are looking at SMBs that treat you as a "IT guy", sure.

If you are looking at more "modern" tech companies in Berlin or well-funded startups it's definitely possible. The last 3 job offers I had were in the 90-100k range, for a "senior" position (~6 years of experience) in Berlin.

This is definitely an outlier and not the norm. Look at Glassdoor and LinkedIn.

Maybe Zalando offers more but few would want to work there.

Just from the top of my head, you could probably earn well at: Zalando, HERE, Mozilla, EyeEm, Blacklane, MongoDB, Zeitgold, Talon.one, Infarm (just raised a juicy funding round) + a boatload of blockchain startups

On top of that, a lot of the bigger SV startups have satellite offices in Berlin.

> EyeEm

Yep, personally know them. But they're in Köln last I remember.

Many of the companies you mentioned (not gonna named not to get sued) have reportedly terrible working culture and/or awful recruitment process (applied to most of them). So maybe the money is compensating.

> terrible working culture

Might be true for some of them, but for the ones on the list where I know people that work there, it sounded pretty good. In the end good working culture is one of many axis to optimize on.

Hard truth: It also heavily depends on how good you are. The salaries talked about are usually the top 10% maybe 20%. Like people throwing around Facebook and Amazon salaries. They only employ a small percentage of devs.

That leaves at least 80% that earn considerably less and 50% that earn below the mean.

Quick math lesson: 50% earn below the median. For salaries (in general, I've no idea what it's like for developers in particular) the right-hand tail (high performers with extremely high salaries) and the lack of left-hand tail (no salaries are negative) shifts the mean (arithmetic mean a.k.a. average) upwards so it's actually (usually) going to be more than 50% of population earning below the mean.

Well, it's more of a linguistic/english lesson. I meant the median and just forgot that average and mean are synonyms. But thanks for pointing that out.

I actually edited out exactly the same argument that you wrote, because I (and you) have nothing to actually back it up.

Counterexample: if 60% of people earn very close above the average and 35% earn 30% below average and 5% earn (on average) double the total average, then the majority earns above average even tho the right hand-tail gets high. This is just one possible scenario that is not even completely unrealistic.

Also remember that most crazy incomes are not salaries.

Its just from my personal experience, but i know a lot of people in the 70k+ range at different companies, all senior though. We also interview a lot and a considerable amount of candidates that get offered high sixties go somewhere else where they got a better offer.

> companies' tradition of rewarding management incompetence over technical competence

I witnessed this and moved out of the large rich corporate to a small startup to keep my sanity.

Care to share your experience?

To add to this: The CDU (conservative party) turned back the decision to role out a fiber network across Germany that would have been constructed from the 80ies until the millenium [1]. If you actively burn your fields, then there is nothing to reap.

[1] https://www.heise.de/newsticker/meldung/Missing-Link-Der-Kam...

I really don't get the argument about fibre to the home being good for business.

For working from home, VDSL is pretty much good enough for most working needs (from what I have seen).

Fibre really shines for the consumer - especially video.

We have fibre to the home in New Zealand, and it doesn't make us an IT nation, or meet any of the gushing political waffle about our IT future.

It suspect the costs have a reasonable timeframe for payback as a tax payer: the government "invested" about 500€ per household to cover 80% of the population (1000€ if only 40% of households use it). I would be interested to see numbers to justify it, but is smells ok on the surface.

I tend to agree and german political discourse about network development seems to generally be focused on broadband for rural areas with shitty if any internet available right now. Thus it's probably more addressing class divide than maximizing overall GDP or something like that.

>> Packet analysis was seen as the worst thing in the company

Wow, that is just insane... "We want you to put a nail in that board but, by god do not even think of using a hammer". That must be the worst case of "security" by obscurity I have ever heard. Was this one of the bigger companies or a smaller firm? I wonder what kind of decision making process leads to such policies.

A big company doing mainly nice white fridges and yellow construction machinery. But they also do parts for planes. After quitting I heard, they were having some military project and had strict rules for it... Same building, same strict rules for everybody.

If its the same building its not that secure I went for a job at hauntings (aerospace and weapons) years ago.

I wasn't even allowed inside the main plant for the interview - they had a room attached to the gate house they used.

I did internships at some top companies in the automotive industry. I can only confirm that.

I think there's a pretty big gap between "software first" companies and engineering driven ones (car companies, "Maschinenbau" etc.). Sometimes the cultural gap is pretty shocking to me. I made a career decision to avoid engineering driven companies for software related jobs because the situations are usually similar to your scenario.

The state of the German IT industry outside of a few select places (Berlin, mostly) is what makes me very pessimistic about the future of its car industry.

They (meaning the big 3: VW, BMW and Mercedes) apparently still think that building the best engines/transmissions and being the best at putting them all together is all it takes in order to make a modern car, unfortunately I think EVs will be more about software and the way said software can best manage the car's power resources. From far away Tesla looks like it's doing quite a nice job with its EV software, the Germans, not so much.

If it matters I've never worked in Germany but as an IT person/programmer living in Europe I've followed the German IT industry pretty constantly as Germany is one of the best countries in terms of quality of life (I know it's not perfect, but it sure beats my Eastern European country). Unfortunately for me its backwards IT industry (again, as seen from the outside) keeps me away from it.

This explains why one of the major tier 1 suppliers has an absolutely horrendous workflow in processing financial data. We do a lot of business with them and when processing invoices sent to them, almost every division outside of Germany accepts invoices by email, but for one of the central German plants, they only accept invoices by mail to a foreign PO Box, so we're effectively giving a discount due to international shipping per each shipment/invoice can get high. As far as I can tell this seems more like an IT issue rather than a financial one (very large firms are geniuses in pushing out which vendors to pay, but if this was a universal case then every international plant would only accept invoices by mail)

> The state of the German IT industry outside of a few select places (Berlin, mostly)

If Berlin IT industry is a golden standard I'm really not very hopeful for Germany.

It is. And yes, there is no reason to be hopeful.

I started freelancing in 2014 and somehow ended up only working for US companies after 4 years.

Even german startups feel to be a few years behind.

If you don't mind me asking, was it primarily a pay rate issue, or the number of freelancing opportunities that were available?

German IT companies just felt too conservative to me.

I work only remote.

Funfact: more than 30% value in a car is software, which is highly interconnected via various type of networks namely, LIN, CAN, MOST, FlexRay, Ethernet. Most current premium cars have more than 10 of those networks and around 100 to 130 ECUs that talk to each other.

So much about: they don't know about software.

BTW: The difference between IT and automotive aka embedded is that embedded should work even without DevOps. Well, IT we know.

I own a basically German-engineered car (the car badge itself is Spanish), and one of its main advantages is that it is old enough (well, by today's standards, the car is from 2007) that is not sophisticated enough in order to include that much electronics and software. As such I'm perfectly confident to hop right in and drive 400-500 miles, heck, I've driven almost 3000 miles a couple of summers ago all the way to Switzerland and back.

I would totally not have the same level of confidence driving a more modern German car with more electronics and software included. As such, how much of today's value of a car is comprised of its software is orthogonal to how good or bad that software actually is.

You wouldn't feel confident driving 400 miles in a 2019 German car because of software? What do you envision going wrong?

> you wouldn't feel confident driving 400 miles in a 2019 German car because of software?

Because of software and electronics combined, yes.

> What do you envision going wrong?

One of my closest friends purchased a VW Tiguan last year and not two weeks into his ownership the car left him stranded at the side of the read because of a coolant hose or something similar. If VW cannot make a simple hose not break after only two weeks in a 30,000 euros car I cannot trust their software verification processes and I'm not willing to spend that much money in order to find out. In other words, if they cannot properly verify a simple hose I cannot trust them with properly verifying software.

On new cars if a component is faulty they fail quickly. I had a tire blow out on a brand new car I was driving home from the dealers in the UK once. Tires and hoses are easy, it's the opaque systems that are a nightmare to troubleshoot, given lack of documentation and manufacturer rules forbidding removing plastic engine covers etc

"One of my closest friends purchased a Macbook last year and not two weeks into his ownership the notebook left him stranded at the side of the read because of a power cable breaking or something similar. If Apple cannot make a simple power cable not break after only two weeks in a 3,000 euros notebook I cannot trust their software verification processes and I'm not willing to spend that much money in order to find out. In other words, if they cannot properly verify a simple power cable I cannot trust them with properly verifying software."

I really tried following your logic but it still escapes me what a hose has to do with software.

You do know that there's a strong correlation between Apple's downward standards in how they physically manufacture their latest laptops and the quality (or lack of) of their latest OSes, don't you? I don't want to start an endless Apple-related flamewar but the links and discussions about this subject on this very website are countless.

Like I said, if you're not bothered anymore of doing basic quality testing for things like coolant hoses then how can I trust the same company with doing basic quality testing for their software?

> if they cannot properly verify a simple power cable I cannot trust them with properly verifying software.

Yes, one of the reasons why I told my boss that I don't want a 2,500 euros Apple laptop (I chose a Mini instead, they're more sturdy, less expensive and I gathered more difficult to fuck up) were articles like this one [1]. And yes, it does involve a cable fuck-up.

[1] https://www.theverge.com/2019/3/19/18271733/flexgate-display...

Always a good read on the subject:

https://www.yourmechanic.com/article/the-most-and-least-expe...

The challenge is troubleshooting at the roadside. Older vehicles have comprehensible shop manuals and systems, but most cars in the last 10 years have little guidance on owner repair, which is often formally discouraged. A 2019 car is unlikely to have systems wear and tear or short circuit issues but they are so complex if something does fail you are unlikely to be able to do anything about it

Probably more of an industry thing. I remember seeing a Twitter thread where someone was reporting about the state of software at Tesla. What was described was not much different from what I hear about other car manufacturers

Having worked at a German company (in Germany) that creates tooling for that kind of software development.

They had a huge smalltalk program that created embedded C code (AUTOSAR/MISRA if I remember correctly). Tests only ran manually and the coverage wasn't particularly high. I don't recall any automations that would have stopped you from saving new code that breaks tests to the smalltalk image.

Even though it 'worked', I wouldn't say this whole system excelled at what it did. Lots of manual QA.

So technically it worked, but it certainly was in poor shape from an automation perspective. It was my job as the mostly unsupervised intern to fix that. Granted, that was 10 years ago, but talking to some of my friends around Stuttgart, things move at a glacial pace :)

Seems to be true in the UK, too. I've heard bad things about e.g. Jaguar Land Rover. As another comment says, this is probably an issue with the embedded industry as a whole. There are great tech companies you can work for, which usually have a very high standard (ARM seems to be great).

Stuttgart/Baden-Württemberg is quite conservative. The real mystery for me was why their engineering pride of doing things correctly doesn't translate to the tech industry. I don't have experience in Berlin/Hamburg/Munich, but it'd be interesting to compare.

My five cents because the basic design philosophies are quite different. And also industry inherent arrogance, especially within the so called premium manufacturers.

Bit the the opposite is true as well isn't it? You cannot treat automotive manufacturing like software development. I'd say the first company to successfully blend the two has a huge advantage.

Technically, even though it does to the fev sides of things, isn't this tesla?

Not so sure about Tesla having figures out automotive mass production. Cannot comment on the software side of things, but I donhave the impression that Tesla is ahead of more traditional car makers there.

They tend to be the first to e.g. introduce OTA updates, map updates over wifi/lte, ...

They are also one of the very few manufacturers that maintain their cars and even do somewhat major UI upgrades for years after the original release.

I'd say as much as they mess up, they are on the forefront when it comes to car manufacturers

It is not 10 years ago but something around that time I was working on automated tests for car equipment. Quite a lot of it was still semi-manual because you had to change some things with physical buttons that you would have in car, but voltage changes, and quite a lot of sensor inputs we had automated. Was quite easy because you could fake CAN connected devices.

This seems high. Is it that 2019 model is worth 30% more than a 2000 model was at release? Or is it more a matter of refactoring mechanical linkages/interfaces into software?

Surely a car only has 1 ECU (Engine control unit). It may well have 100+ micro controllers on board.

ECU is abbreviation for Electronic Control Unit in automotive. (And a modern car may actually have multiple ECUs controlling the engine(s), especially if it is a hybrid=HEV, plugin HEV=PHEV or battery electric vehicle=BEV.)

That was maybe 15 years ago. These days cars have multiple CANbus networks running and there are control unit everywhere.

Below the details for an Alfa Romeo Giulia, which is interesting because it was a complete new design when introduced in 2016, not a rework of a previous platform:

3 networks run in parallel: These networks and their subnetworks are protected by a Security Gateway Module (SGW).

  - CAN-C (high-speed 500 kb/s)
  - CAN-CH (high-speed 500 kb/s)
  - CAN-IHS (medium-speed 125 kb/s)

Modules connect to 1st network:

  - Body Control Module (BCM)
  - Instrument Panel Cluster (IPC)
  - Occupant Restraint Control (ORC)
  - Radio Frequency Hub (RFH) Module
  - Antilock Brake System Module (ABS)
  - Adaptive Cruise Control Module (ACC)
  - Emergency Assistance Module (EAM)
  - Primary Powertrain Control Module 2.9 V6 (PCM - P)
  - Powertrain Control Module 2.0 I4 DI (PCM)

To 2nd network:

  - ABS Module
  - BCM
  - Electric Steering Lock (ESL)
  - Park Assist Module (PAM)
  - Electric Power Steering (EPS)
  - FFCM - Haptic Lane Feedback (FFCM - HALF)
  - Occupant Restraint Control (ORC)
  - Active Aerodynamic Module Left – Front left Splitter actuator (AAML)
  - Active Aerodynamic Module Right – Front right Splitter actuator (AAMR)
  - Adaptive Front Light System (AFLS)
  - Chassis Domain Control Module (CDCM)
  - Torque Vectoring Module (TVM)

To 3rd network:

  - Instrument Panel Cluster (IPC)
  - Comfort Seat Wheel Module (CSWM)
  - Left Blind Spot Sensor (LBSS)
  - Right Blind Spot Sensor (RBSS)
  - Amplifier (AMP)
  - Trailer Tow Module (TTM)
  - Heating Ventilation and Air Conditioning (HVAC)
  - Entertainment Telematic Module VP2, VP4 (ETM)
  - BCM
  - Entertainment Multimedia Control Module (EMCM)
  - Secondary Powertrain Control Module 2.9 V6 (PCM - S)

I worked for Mercedes Benz. They do a LOT of their software development in the US/silicon valley, and I was very impressed by their process and work environments.

> VW, BMW and Mercedes) apparently still think that building the best engines/transmissions and being the best at putting them all together

That stopped being true at least 10 years ago. In 2019 I would bet on a Renault engine every time (so does Mercedes btw :P)

In my major German company, mostly living off hardware, we seem to be at least living with a decent stack (though mostly using the MS copycats of Slack, Jira and Github). Interesting that it seems to be different at other companies. However, describing the automotive companies of being unaware of the need for more software devs is simply false - living in Munich, all the companies are doing is investing in software devs (car media, autonomous driving, electric cars, digital business models, apps, car sharing,...)

German car companies have absolutely recognized their problem with software several years ago (from what I can see 5-10 years ago), but they are still varying levels of bad at implementing the solution. One of the things they are doing is starting daughter companies that only do software... but still get infected by the awful development and management culture of the parent.

That's not just in Germany. The €3bn Dutch energy company I've worked for was glued together with a shared network drive. It was effectively their poor man's event bus. Pure anarchy, a house of cards of excel sheets and legacy software writing to and consuming from it. Controlling serious infrastructure like the major power plants of the 18th largest economy of the world in 2012 [1]. The result of IT managers who failed to see how the world was changing and came and go, to do things like "agile transformations" instead. I've learned valuable but painful lessons there.

[1]: https://en.wikipedia.org/wiki/Economy_of_the_Netherlands

This is a recurring flawed observation when young grads enter industry and expect continuous integration, test-driven development and $buzzword framework. What you describe is simply the reality in a vast majority of companies, Germany or elsewhere.

England checking in, that all sounds strangely familiar even down to the network shares.

Honestly security at this point is a myth, we can't close the door after the horse bolted as the barn is currently on fire and the horse has been gone so long it's settled down and raised a family.

Thanks for checking in England

We're hiring for software at Stenon, we make a portable soil analysis device for real time agricultural nutrient analysis [0][1]. Based in Potsdam, Brandenburg (30 minutes on the train from Berlin), and yes, we use git :)

[0] - https://stenon.io [1] - https://stenon.io/en/career/

Aside from localization issues this was how it worked at a very large tech (not software) US company I worked at

This group of german companies founded their own security group [German Cyber Security Organization (DCSO)]. That speaks a lot about their trust in their public services.

The only time I've been involved in a hacking attempt (it was ransomware) the company I work for contacted the CCN-CERT. I wonder if US companies contact NSA/Other gov agencies or deal with it themselves with security companies.

Also, while I understand the care and concern they put into securing their networks, many german companies basically gift their tech to china, like Deutsche Bahn, or being bought and transfered there, like it happend with Kuka. So be it by hacking into your network or "partnering", they'll copy your tech and kick you out of their market sooner or later.

Probably because when it comes to anything cyber-related, the German government, like most big companies there, is a dinosaur, greatly inferior to it's British and Swiss counterparts.

Germany at least has the CCC, which it sometimes asks for advice and, occasionally, listens to.

When they aren't trying to put them in jail that is.

I'm not sure what the social aspects in Britain and Switzerland are like, but in Germany, working for the BND or other governmental cyber security institutions is frowned upon in the cyber-security circles (at least that's my view from someone not too deep in the field).

Yeah, because the BND is the enemy. They are the ones who keep the vulnerabilities open instead of closing them. We have the CCC to trust in.

> This group of german companies founded their own security group [German Cyber Security Organization (DCSO)]. That speaks a lot about their trust in their public services.

> The only time I've been involved in a hacking attempt (it was ransomware) the company I work for contacted the CCN-CERT. I wonder if US companies contact NSA/Other gov agencies or deal with it themselves with security companies.

In the US, most large companies that have suffered breaches contact the FBI.

"Modern-day espionage operations have one big advantage: Instead of painstakingly planting agents in companies, digital spies are simply sending prepared emails."

We face this threat in my business - daily fishing attempts or schemes to get employees to open files. It never stops.

This is a primary reason when we started designing our new web app at bomquote.com a few years ago, we first focused on communication tools which reduce our use of email both internally and in our dealings with our customers.

Sure, there will be attempts to hack our app servers, but from my view we can deal with that easier than preventing our accounting admin from clicking on a well crafted email.

Using U2F security keys is one way to stop phishing if the app you're supposed to log into requires it.

https://krebsonsecurity.com/2018/07/google-security-keys-neu...

Unfortunately "phishing" nowadays is used to describe any kind of social engineering, including all variants of tricking the victim into executing malware on their machine.

U2F won't save you there, it will just make the attack a bit more annoying.

Is there any business out there that does not use email?

Is this a good way of tracking an APT? Just from bytecode? Isn't that easy to fake? What if they were tracking Russian hackers instead?

If I'm understanding the article correctly, the hackers are using a easily reversed cypher for storing configuration data for their malware, which was reversed by assuming the presence of the string "C:\Windows\System". In the following decrypted data the name of the respective company targeted was found.

Yes I suppose it would be easily faked if the faker had performed a similar analysis on the malware...

Good article. I am not so bothered by the effects, I think they complement the article well. I sure like that what may seem like decoration to some readers is actually real - I find it very interesting to find out that they find the malware using nmap.

Now where do I get that script? More detail would of course always be nice.

Git repo was linked by OP in this [dead] subthread: https://news.ycombinator.com/item?id=20514267

The nmap script was written by ThyssenKrupp's security division and can be found here: https://github.com/TKCERT/winnti-nmap-script/blob/master/win...

pretty decent tech write-up from the German NPR

Finally, somebody said it. The English also very good, nay, excellent.

> Winnti is a highly complex structure that is difficult to penetrate. The term denotes both a sophisticated malware and an actual group of hackers.

Hacking groups are corporations and spread risk away from indictable individuals just as efficiently, with a separation of liability and actions and knowledge

This needs to be understood

Reply on news.ycombinator.com