I was a senior VP of tech at Starwood: here’s my take on the guest data breach | Hacker News Reader

I was a senior VP of tech at Starwood: here’s my take on the guest data breach

phocuswire.com

93 pointsvaliant-comma57 comments

This article seems full of points that a lay person might nod along with, yet don't hold up to scrutiny.

> The fact is, if we accept Marriott’s statement that the breach began in 2014, the system would already have been operating securely for five years.

It does not mean that. It means that we don't know of any exploited vulnerabilities before that point.

> If the detection tool was used prior to this September, why hadn’t the breach been detected earlier? And if the tool was not used earlier, how can they be so sure the breach occurred in 2014?

It isn't unthinkable that the new tool alerted them to a problem, and during investigation discovered evidence that the vulnerability had been abused in the past.

> It is almost impossible to imagine a scenario in which an external hacker is able to gain access to the primary encryption keys.

Why? The argument seems to be: the primary encryption key is important, and thus will be most carefully guarded, so it is unthinkable that it would actually be exposed.

Ultimately the article strikes me as an article written by someone who has a beef with Marriott, and he ends noting that it's possible that the breach occurred not due to issues with design, but due to the layoffs of Starwood's technical staff.

> Ultimately the article strikes me as an article written by someone who has a beef with Marriott, and he ends noting that it's possible that the breach occurred not due to issues with design, but due to the layoffs of Starwood's technical staff.

I agree with your first several points, but a lay-off beef is unlikely since the author hasn’t worked for Starwood in over a decade.

Marriot replaced his system at Starwood with theirs (and he didn’t agree with that) so his technology was laid off :)

More like someone with "CTO of Starwood" on his resume at a time when "Starwood IT hacked" is the main headline.

I tend to agree with his conclusion: without further information, it's useless to speculate about how it happened. But his effort to spin an alternative narrative is, at best, self-serving.

> It is almost impossible to imagine a scenario in which an external hacker is able to gain access to the primary encryption keys.

I worked some place where lots data was encrypted with a key. The key hadn't changed in months - at least 6 months by the time I found it. I was told this same key would be used to encrypt web session data in a cookie. There were more than a dozen people who I knew had access to the key, and another 5 had come and gone (and had had access to the same key) in the previous 6 months.

I know not all companies are run like that, but I suspect it's closer to the norm. Even in places where they want to be more secure, enforcing security policies often becomes an after thought to more important tasks (in places I've seen/worked).

> The key hadn't changed in months - at least 6 months by the time I found it. I was told this same key would be used to encrypt web session data in a cookie. There were more than a dozen people who I knew had access to the key, and another 5 had come and gone (and had had access to the same key) in the previous 6 months.

What's so wrong with any of this?

Software requires operators and developers. If you can't trust them, you can't trust your service, period. It's normal for operators and developers to occasionally have access to sensitive data (e.g. when your service crashes, somebody has to look at the crash dump). Such access should be restricted (requiring approval) and logged, of course, but it's difficult to eliminate entirely at scale.

It's good to rotate keys on a regular basis - but an annual key rotation doesn't seem negligent to me for a key used to encrypt session data, which is necessarily long-lived.

> Such access should be restricted (requiring approval) and logged, of course, but it's difficult to eliminate entirely at scale.

It’s not clear how you could practically enforce this requirement if devs just have the raw key on their workstations.

Would be nice to use a multi-sig so the dev would need their key which they always have access to plus a key from an approver.

> What's so wrong with any of this?

A very basic principle of security is the principle of least authority. To implement that, you don't use one key for many different purposes over a long period of time and give it to anyone who needs it. You use different keys for different purposes, and replace the keys periodically to help ensure that they're only available to a limited group.

Another way to answer your question is "because that's how all these breaches happen."

The short (and sad) answer is that no one really cares about security; they only care about it when something hits the fan.

Security isn't a revenue driver, only a cost. It's commonly known that most companies (small or large) vastly prefer investing in things that increase revenue and profit.

There was no ability to tell who'd even seen the key. A colleague of mine 'got it', and when it was shown he had it, there was a "WTF? You're not supposed to have that? Where'd you get it?". It was checked in to version control. It was just in a project that only a few people were assigned to, but there was no restriction on who could view those projects.

I understand no one having access to the key, but if data is already encrypted using this key and you want to rotate to a new key, how do you decrypt the old data? re-encrypt all old data with the new key?

For keys that matter, good practice is that no one has direct access to the key. E.g. you have a process where there are ways to sign stuff with that key, and there are ways to gain access to that key if extraordinary circumstances arise, but in normal operation you should be able to make sure that you can revoke the ability to sign stuff from anyone, which requires you to be certain that they never ever had a chance to see, copy or write down private key material.

Isn't that the point of HSMs where nobody can get access to retrieve the private key - all you can do is sign stuff with the key?

Yup, but even in cases where the expense of a proper HSM isn't warranted, you can set up a software solution/process that works in a similar manner to a HSM, just with less tamper-resistance.

Of course even if you have HSMs there is still the issue of controlling access to the HSMs!

What was being encrypted was a sequential DB ID that identified your user ID in the system. Decrypting the data (with the key that multiple people had access to and there was no way of telling who'd taken it) would allow you to change your ID and re-encrypt and become someone else. (also, AFAICT, the same key was used for some internal data encryption too - same key used in multiple areas across different teams)

This was initially just a marketing site, but I was being told to use the same system for authentication around a financial service, and I objected. I suggested the ID be random, then mapped to something else. I had multiple 'senior' developers try to explain why "random isn't really random" and had a lot of timing attacks associated with it. I kept pushing back that timing attacks weren't as big a problem as "someone having the key can impersonate anyone, and we can't prove that people haven't taken the key out of the company". I "only" had 9 years experience at this point, but was the 'new guy' in the company, having only been there a few months, and apparently hadn't "paid my dues" in their ecosystem for long enough to be taken seriously.

It took weeks of me objecting before we had a big meeting, and their architecture was mapped out on a board, and defended. I then asked "how can you prove the key hasn't left the building with previous people?" Then my manager sort of "got it" and realized this was all kinda ... not something we should rely on for guarding financial info. Maddening that it took weeks of "deliberation" on this issue when it was patently insecure. I've not ever whiteboarded it out to anyone with more than a few years of experience who didn't stop me partway through to tell me that it was insecure.

Dejected through all of this, I asked to be put on another project, but I was seen as a troublemaker at that point, then left.

There are mature and well known techniques for mitigating insider threats. Key rotation capability is table stakes. Keys that are too expensive to rotate need to be in HSMs, and probably protected by operator quorum. There may be some short-lived, limited purpose keys that are more vulnerable, but the whole company’s master keys to everything should absolutely not be held by any one person.

I think for most companies convenience trumps security. Like the current workplace is first I worked @ were keys for signing are stored in a special secure location on air gaped systems.

Your first point lacks context of the next paragraph. Here’s the two paragraphs combined which changes the meaning.

>The fact is, if we accept Marriott’s statement that the breach began in 2014, the system would already have been operating securely for five years.

>It is difficult to imagine how an architectural or platform vulnerability would not have been discovered or exploited sooner.

He’s saying it’s most likely this exploit has been discovered earlier than 2014. This is even more aggressive than your point which was

>>It does not mean that. It means that we don't know of any exploited vulnerabilities before that point.

Honestly, after seeing this article upvoted so high and then reading it, I was relieved to see these comments. At it's root, security for always-on networked systems is extremely difficult, even at tech-first companies with an ingrained "security culture", nevermind a hospitality company like Starwood where "IT" is another department. And this guy comes forth with clueless statement after clueless statement about "The system was already operating securely for five years" and the one about the primary encryption keys. This whole article is incredibly self serving.

> This whole article is incredibly self serving.

Mmmmhmmmm. It's executive CYA, and a public article for a public clusterfuck.

5 years ago is when this stuff probably started going sideways and those failures manifest later as massive outages, breaches, etc. Could be that the author IS WHY a lot of these issues cropped up later, so take proactive steps to blame others.

Gotta keep that executive cachet high so that you can slide into a CTO role elsewhere.

"the article strikes me as an article written by someone who has a beef with Marriott"

Another comment here references a post he made in 2016: https://www.linkedin.com/pulse/marriottstarwood-back-future-...

Seems to be more of the same.

I'm amongst the most frequent guests at Starwood, spending >100 nights a year in their hotels. I wasn't thrilled when it was announced they'd be acquired by Marriott. This year, they began the switchover process to migrating to Marriott's technology, and the full switch officially happened in mid-August.

It was a complete and utter disaster.

Everything was buggy, points mysteriously disappeared, reservations disappeared. Inconsistent UI, a mix of old and new systems. A truly awful experience dealing with support agents who were incapable of comprehending what was happening. I'm still waiting for a handful of stays to be credited to my account months later and nobody can help me because the systems are broken.

I found myself staying mostly at Hyatt hotels while the dust settled. I'll end the year with another 100 nights with Starwood/Marriott, and 80 with Hyatt. But, given the direction the company has taken since the merger, that number will likely be going down on the Marriott side.

After hearing that Marriott laid off the majority of Starwood's technical staff before attempting this migration, I'm not surprised it went this way. I'm also very much inclined to believe that the data breach happened during this migration.

Agreed that their migration from SPG to Marriott has been painful. It ended up creating 3 new logins for me before finally consolidating everything under a new number.

Out of curiosity, do you find your Marriott / SPG Ambassador to be useful? I reached Platinum Premier Elite with Ambassador status in November but my ambassador hasn't been helpful at all. The Your24 perk also rarely works in practice. It's a nice marketing gimmick, but definitely not worth it so far.

Ambassador status is highly dependent on who you get. Overall, my ambassador is a nice guy who's professional, friendly, and helpful. I also really appreciate having a dedicated line of support considering how awful Marriott's customer support otherwise are. They're organized a few upgrades for me when they really mattered, and are my main point of contact for support, especially for some pretty complex issues which would've been much more painful through traditional channels. Beyond that, I can't say it's made that meaningful of a difference.

The quality of the service has gone markedly down since the August integration. My understanding is that ambassadors have been bogged down with technical issues, an influx of new customers (going from 50 guests per ambassador to >300), and a big drop in morale. A lot of the issues they can't help with aren't their fault, but rather problems with Marriott policies and technology.

And oh: on Your24, I've only tried to use it a couple times, and had it work about 50% of the time. I've checked in before 3pm many times and pretty much always get a room. (Except one frustrating time after a red-eye where I needed a shower desperately and the hotel had no rooms available or any showers in the gym. I was not a happy camper.)

You might be very interested in the following talk, where a Marriott tech lead describes how as recently as 2013 (and it's ambiguous, but it's likely still the case), much of the Marriott pricing engine was running on a single "green screen" (his words) IBM mainframe. He speaks about how only in ~2014 with a management shakeup did culture begin to shift towards acceptance of open source components. But, as we all know, these types of organizations don't turn on a dime, and sadly this breach is likely to make them more conservative (and increase reliance on those old-school systems) rather than less. I'm not surprised at all about your experience.

https://www.youtube.com/watch?v=wdFYEuWWpzo&t=5m30s

Disclosure: I'm working on a project (currently stealth) in the alternative accommodations (vacation rentals) space. Lots of people all over trying to reinvent these types of booking engines, and it's a tough challenge to get right. I don't blame them for not wanting to deal with distributed systems on top of those challenges. But IMO that's part and parcel of a modern approach to the problem.

The article he wrote about Marriott's choice to continue using its z/TPF based platform over migrating to Starwood is also telling.

https://www.linkedin.com/pulse/marriottstarwood-back-future-...

See this quote "To better understand the resulting Starwood’s technology compared to industry legacy systems, think Tesla Model S versus a gas-guzzling 1975 Buick Electra.

Then along came Marriott . . .

When Marriott announced its interest in acquiring Starwood, one would have believed that they factored in a $500 million Starwood IP technology value within their $13.6 Billion offer, and that they would have been salivating at the prospect of having their hands on the fruits of the multi-year transformation experience this IP represented. After all, while stable as a rock, Marriott’s own system today centers around 1970’s Mainframe TPF technology (MARSHA) suitably kept current via the judicious use of the scotch-tape and wires represented by a cornucopia of front-end gateways and the labor intense support of inflexible legacy code, eclectic data bases, hard-coded interfaces, and a veritable zoo of different property management systems crying for better integration. "

It reads as sour-grapes to me.

If you wanna read more about MARSHA - this seems to be a good source: http://ibmsystemsmag.com/mainframe/casestudies/miscellaneous...

> if [...] the breach began in 2014, the system would already have been operating securely for five years. It is difficult to imagine how an architectural or platform vulnerability would not have been discovered or exploited sooner.

I mean, that seems very easy to imagine? Just last year Wannacry exposed an RCE exploit in Windows that has been present since at least Windows XP (https://docs.microsoft.com/en-us/security-updates/securitybu...). And there are orders of magnitude more people looking for exploits in Windows than Marriott's internal systems. I don't find this article particularly credible.

This guy appears to have no clue what he is talking about, and is painfully ignorant of both security and technology.

I realize that’s not a very substantive comment, but wow.

”The Valhalla system was fully activated in 2009, and my understanding is that all best practices were followed in its design (firewalls, DMZs, encryption, etc.).”

”It is difficult to imagine how an architectural or platform vulnerability would not have been discovered or exploited sooner.”

”It is almost impossible to imagine a scenario in which an external hacker is able to gain access to the primary encryption keys.”

This is just painful to read from someone so senior on their soapbox. It’s probably exactly why they got hacked. Also note that this isn’t the first major starwood breach: https://www.starwoodhotels.com/html/HTML_Blocks/Corporate/Co...

You don't get to SVP of a big corporation by being an expert in technology or security. That high up, your skill set is on organization, process, prioritization/planning, and budgeting. Sometimes you do see technical CTOs.

This is an exceptionally self-serving take on the matter at hand. So much so that it’s frankly breathtaking that it’s been upvoted to #1.

Dear Israel del Rio,

As a Mariott and SPG member since history, kindly focus on not disclaiming responsibility in a public forum, since you almost assuredly aren’t as innocent as you claim.

> It is almost impossible to imagine a scenario in which an external hacker is able to gain access to the primary encryption keys.

I was reasonably sold on what was being said until that comment. Impossible is a strong word to use when it comes to computer security. It seems that everyone who has claimed that there system is unhackable, always ends up being hacked.

Indeed, the only time that "impossible" should be used when it comes to computer security is when stating that true security is impossible.

He said 'almost impossible'. Big difference.

He said almost impossible to imagine. It really was an extremely hyperbolic statement. I've never done security work but I've helped develop plans for when primary and secondary keys are stolen. It's something that should have been imagined, not "almost impossible to imagine".

I think it's actually pretty easy to imagine a scenario where private encryption keys were stolen.

Agreed. He seemed to make a decent argument that the quantity of records + types of data exposed didn’t add up (based on his knowledge of the system), but this statement (about primary encryption keys) undermined the credibility of the overall piece.

"The fact is, if we accept Marriott’s statement that the breach began in 2014, the system would already have been operating securely for five years.

It is difficult to imagine how an architectural or platform vulnerability would not have been discovered or exploited sooner."

Not really. There's been vulnerabilities that have been out in the wild for quite some time and took years to be found. Sometimes it just comes down to luck/what people are trying to exploit.

Don't disagree, but how much more insecure is Marriott to say, Motel 8?

Worth pointing out that at the time of this guy's tenure, and for many years afterwards, the way you authenticated yourself while booking a rewards reservation with SPG via the phone was to verbally tell the agent your online password. Like, WTF.

The SPG password for phone has been a different password from the web login password for as long as I can remember (2014?)

Yes, I remember them being the same from 2005-ish through to about 2011-2012 ish. This guy was at SPG until 2006.

This person has not worked at Starwood in 12 years. Their qualifications for making any kind of insightful analysis are basically nil at this point. As if this weren't weren't absurd enough he goes on to make the following laughable statements:

>"It is almost impossible to imagine a scenario in which an external hacker is able to gain access to the primary encryption keys."

>"The fact is, if we accept Marriott’s statement that the breach began in 2014, the system would already have been operating securely for five years."

>"Israel del Rio is executive technology consultant and CTO at Quilmach."

As someone who has family affected by this breach it's upsetting to see to see this individual using this incident for their own self-promotion. However at least his new company Quilmach now knows he is completely clueless about technology. So I guess he did everyone a favor here. Israel del Rio - Executive Idiot.

“It would be irresponsible to speculate at this time, unless I can point the finger at systems I wasn’t personally responsible for.”

There’s no meaningful security in travel companies. They share with everyone and controls are a joke.

Hell, Hilton allowed for 4-digit numeric passwords until a few years ago.

IHG (Holiday Inn, InterContinental) is still a 4 digit PIN and a numeric (or email) userid.

He seemed to say the database wouldn't have 500 million records in it at a time since they are deleted but that seems irrelevant with how the breach took place over 4 years. Anyway the article seems to be just speculation, which is disappointing.

Edit: this article has gotten a lot more upvotes than I would expect if something this quality, is there something about it I'm missing that makes it particularly insightful?

This whole article reads to me like an SVP who doesn't actually understand technology and security (which is definitely not uncommon).

Your expectations of HN users does not match reality.

Upvotes don't necessarily mean quality or correctness. Sometimes they just need newsworthy, or interesting.

This is a masterclass of CYA. He takes a lot of effort to try to prove that the system he was in charge of wasn't the cause, even though his arguments are absurd ("if we accept Marriott’s statement that the breach began in 2014, the system would already have been operating securely for five years." It was operating for 5 years but there's nothing to prove that it was operating securely for 5 years.) He also handwaves and leads the readers down some detective story ("ergo it must have been the data warehouse!") and says "We really don't know if it was Valhalla and may never know!"

No wonder he's an executive, he is an expert CYA-ers!

Also see extensive discussion of Marriott/Starwood situation within https://news.ycombinator.com/item?id=18651676

There's a lot of sour-grapes among Starwood employees in the lead-up to and after the merger. A lot of dedicated middle management folks got forced out and a lot of managers in well-performing hotels were forced to move. None of the grunt employees seemed happy with it on either side.

This is a surprisingly poor understanding of tech. Do hotels hand out titles like investment banks (there were hundreds of VPs at JP Morgan), or was this author actually in a position of responsibility? Is it common for non-tech companies to have such people in high up places?

2 factors I think:

* the CTO title is the new project manager. It seems as soon as someone is in a position to choose whether to use Postgre or MySql they break out the CTO moniker. You can now be the CTO of just about any organizational unit, no matter how small.

* CTO != CISO (quite the contrary usually). Being tech-savy doesn't mean you grok security, especially its operational, wetware/social aspects. Obligatory xkcd: https://www.xkcd.com/538/

I would tend to go with the second explanation in this case. When I read the quote "It is almost impossible to imagine a scenario in which an external hacker is able to gain access to the primary encryption keys", I thought it pointed to a lack of imagination (and probably a bit of Dunning-Kruger syndrome) rather than security expertise.

> Still, most commonly, breaches occur when someone obtains an administrative password via deceitful means (e.g., phishing attacks), enabling them to log into the system and install Trojan software to extract data or to manipulate the system.

> This is the method the Russians used to hack into the Democratic National Committee emails, for example.

AFAIK, phishing was the main attack for Podesta's emails, but I'm not aware that this was used on the DNC hack. I think the author is mixing scenarios.

Reply on news.ycombinator.com