Great Unsolved Problem In Computer Science | Hacker News Reader

Great Unsolved Problem In Computer Science

algeri-wong.com

155 pointsnopinsight55 comments

Other companies have solved this problem at the enterprise level with huge installations, most notably Lotus Notes and to a lesser extent Novell Groupware.

In the non-office space Facebook is making a strong run at the defacto group scheduler, at least among my network.

most notably Lotus Notes

Some people, when confronted with a calendaring problem, think "I know, I'll use Lotus Notes". Now they have two problems

Some people, when confronted with a calendaring problem, think "I know, I'll use Lotus Notes". Now their subordinates have two problems.

In my experience calendaring in Lotus Notes is excellent. As long as you are willing to leave your "but Outlook does it like that" attitude behind.

Sometimes I'll open a calendar notice in Notes, and it will tell me that this notice contains updates to a scheduled event that has unprocessed updates, and would I like to process those updates?

That's broken. Updating events I do not own, but am participating in, should be passive. Yes, it happens in my data in Notes, but even at the conceptual level, I don't own those events. If there is a change, my calendar should be automatically updated, and then I should be informed about it.

Instead, I get informed about it, and if I neglect to open that notice, the change does not get processed. Viewing the note in the preview pane is not enough - even thought I've checked the option to consider mails "read" when they appear in the preview pane.

I used to work for Lotus and was a Notes developer. But funny is funny.

Seriously I'd say it's no better or worse than any other group scheduling app like Outlook. TBH, email isn't really a source of competitive advantage for a company. It's on a par with a phone service or electricity. Just outsource it to google already.

I'd say it's no better or worse than any other group scheduling app like Outlook

I'm a Notes/Domino admin for a small company and I'd have to disagree. The number of bugs, and the way that they interact, is just ridiculous in Notes. The latest versions aren't nearly as appalling as the old ones, but they're still terrible.

Absolutely agree about the "competitive advantage" though. E-Mail/calendaring should just be outsourced, or simple drop-in software that doesn't need too much attention.

Much the same way Microsoft has Exchange as its server backend and Outlook as the client UI, Lotus has Domino as its backend and Notes as its frontend. Domino is a pretty good product. It's fast, secure, distributes well, was one of the first products to employ a document-based database, and it's backwards-compatible across something like fifteen years of its database format. Every sysadmin I know loves it.

Notes, in contrast, is now a wretched mess, having failed to correct for fifteen years' worth of poor UX decisions.

Taken as two independent entities, the calendaring in Domino is excellent. Notes makes it a pain to deal with, but Domino does a good job of solving the hard bits. So there's a good argument to be made that Lotus has made a successful go at this, but I wouldn't give Notes the credit.

There wasn't much in the way of meat in that post, unless I missed it. As far as I understood, the problem with calandering is:

- Synchronization: with potentially gazillions of users on a single network you've got a timeliness problem: User A may look free to User B, so he books User B, but actually in the interim User C books User B, so there's a conflict. Given the graph-like nature of such a network (all nodes potentially linked to each other) such conflicts could quickly ripple through the system.

Sounds interesting, I wish the author had been a little more specific about the problems.

I wouldn't think the cross-scheduling issue is actually a huge practical problem, as it requires fairly specific failure cases. OTOH if the users are disconnected, then you can't solve the problem anyhow, so you just write a consistency checker that runs while/after syncing and alerts users.

I think the real problem is sheer engineering complexity: having to support a large number of input/output formats, protocols, devices, locales, languages, countries, timezones, platforms, legacy features, backward-compatibility, tools, etc. and making it all look good on the desktop, web and mobile app.

I think the OP is saying if you multiply these together, then you're screwed, unless you're Microsoft or Google...

Right. That makes more sense. But none of these are CS issues - as you say they're engineering ones. Some self-healing conflict resolver otoh is probably closer to a cs problem.

What does this have to do with computer 'science'? Sounds to me like he's talking about 'engineering'.

Either none of us gets the irony or it's simple misleading. Personally I was expecting complexity/algorithm article starting with overview of P ?= NP followed by some other less known but interesting theoretical issues.

Just throwing an idea out there -

Is it possible to create a distributed calendar system based on git? Think about it, all the plumbing is there. Represent calendar semantics in a logical way, and you also have meeting collision (i.e. conflict resolution) support.

Git might address some of the technical requirements but it has nothing to offer for the functional requirements.

"Is it possible to create a cheesecake using moondust?"

To be fair the grandposter's idea does solve half the problems posed by the article.

Git? Graceful exception handling?

"Is it possible to create a cheesecake using moondust?"

How about stardust?

No one pointed out that this article is actually 2 years old, but that does make a difference to me.

I kept thinking of something the founders of Justin.tv said:

We were too easily distracted and hadn't really thought through the strategic implications of owning a standalone calendaring property (hint: no one wants a calendar without email).

My product involves scheduling of reoccurring reminders, deadlines, etc. Handling of timezones in particular has been an incredible productivity black hole.

Side note: Who do I need to pay off to get a Olson timezone header added to the HTTP spec?

Why? A few lines of Javascript can tell you which timezone the user's machine is set to. Add a timezone dropdown field to the form where they enter the meeting details, and default that field to their local timezone, but let them change it. Javascript can also tell you whether the meeting date that they picked occurs during DST or not (in the local timezone), store that information in a hidden field.

Now you have everything you need to schedule the meetings and make them appear correctly for everyone involved: The date and time of the meeting, the timezone of the meeting, whether or not DST is occurring there at that date and time. Store everything in the database separately, and store all datetimes at UTC.

This is the solution I built once for an "enterprise" client†, and it worked pretty well.

† a global lawfirm, who regularly needs to schedule meetings across timezones.

A few lines of Javascript can tell you which timezone the user's machine is set to.

That is simply not true. You can get the localized timezone as provided by the user's operating system. You would need a server side mapping of every time zone in language for both Olson and Windows. Even then, there are ambiguous timezone names, etc.

The best you can do reasonably is calculate GMT offset, presence of daylight savings time, and if DST is present, then which hemisphere the user is in. From there, you can guess at a city/region with the largest population.

See here: https://bitbucket.org/pellepim/jstimezonedetect/overview

Apologies for not being precise enough (I actually posted that same link in a response below).

With Javascript you can get the user's GMT offset and DST observance. You can't get the political name of the timezone or precise mapping to Olsen database. But for the purposes of scheduling meetings, what you get with Javascript is fine.

Were those recurring meetings or one-offs?

Part of a product I'm working on now is being able to schedule repeating tasks every day, every month, etc. Which means that "07:30 every Friday" has to happen at 07:30 in the author's timezone, whether or not DST is active. To my knowledge Javascript can only tell you the timezone offset from UTC for a given date/time. You can determine the DST-less offset that way, but there are multiple timezones sharing the same offset (e.g. PHP has 63 separate timezone records for Europe alone) and then simply matching them by offsets doesn't work ideally, you need the user to make a choice.

They were one-off meetings, but I think we could have done recurring meetings with the data that we were storing.

I see what you mean about the different timezones in PHP - these are based off the Olsen timezone database. I do think the best way to make users happy is to let them pick their default timezone on a personalization screen, and display it (but let them change it) along with each event that they create.

This Javascript code below can get you most of the way there, but it does come with the caveat that "Not all Olson tz keys are given by this script. If there are many timezones with UTC+6 but none of them use daylight savings, this script simply returns ONE of the timezone keys representing this type of timezones."

https://bitbucket.org/pellepim/jstimezonedetect/wiki/Home

I don't see it as a problem of CS, but management. It's a problem we've made all by ourselves.

Absolutely any problem involving humans where errors MUST be kept at ~0% in all cases (any you have just a bunch of variables that can combine and permute) becomes extremely hard.

If you add a ton of features to it, then it becomes intractable.

In my company, all meetings are set in GMT and a notification is sent as soon as the meeting is decided, which is never less than 3 days in advance. After that, it's not any software responsibility to attend to this meeting and it doesn't matter whether you received a reminder or not. You keep track of your local time. If your country decides to switch to the Mayan Calendar it's not my f*g problem, you keep track of GMT time. Period.

How is life at Initech?

I remember that Kiko was the first YC company I really cared about and rooted for. Their approach to this challenge was spectacularly pummeled from a timing perspective by the entry of Google (iirc), and from what I understand the guys have gone on to create some other, very excellent businesses (including Justin.tv).

I think it's interesting to note that five years on, the calendar problem remains.

(Happy for the Kiko team or those who actually know them to correct my memory; and for those with no such memory - http://techcrunch.com/2006/08/16/ajax-calendar-kikocom-goes-...)

Edit: While I was typing Larrik made his comment. So interesting to note that 2 years ago this problem still remained, and it's worth clarifying that Google's entry wasn't the only (or even main?) reason for Kiko's end.

Some more info on why handlist dates is hard (I didn't write it):

http://planetlotus.org/profiles/mikkel-heisterberg_87315_why...

Probably this comes from the fact that planning appointments across timezones (especially potentially shifting ones) while trying to speak local time is doomed.

Personally I use this simple rule: - if the meeting is local, use the meeting physical location local time. - if the meeting is global, use zulu time.

Now that doesn't help that calendar software actually fails to allow someone to do that properly and easily, but to me that's an artifact of the time notation system we use which grew overly complicated because we insist on tying time to the sun position (for various reasons, some being legitimate, some not really).

Although I despised the marketing aspect of it I actually like the Swatch internet time concept and .beat unit. It just makes sense, is deceptively simple and while it duplicates the role of zulu it removes any source of possible timezone confusion when you write a time (and don't want to sound like a military or an air traffic controller when appointing someone at 0900Z)

if the meeting is global, use zulu time

Do you really use this in conversations/emails etc?

I can't imagine many non-technical people being happy to deal with describing meeting times in this way.

Of course software can (and generally does) use something like this internally, but local software will then be expected to convert it back to local time. And to do that, you get the problems that the grandparent post describes.

This gets even worse as soon as there is one bug out there. I've seen problems where Outlook gets confused about daylight saving, or where someone travels with their Blackberry and changes the time rather than the timezone. As soon as people have seen one meeting with the 'wrong' time they'll start to distrust the ability of the software to manage things and then we're back to square one.

"Do you really use this in conversations/emails etc?"

Yup, and when (rare) people ask, I explain them the motivation. Might be due to the people I interact with though. As I said, this only pertains to remote meetings involving multiple individuals from across the globe, each staying at their own location.

If you're scheduling a worldwide meeting with someone in another timezone and you say 14:00 CEST or Eu/Paris summer time or whatever they already have to deal with timezones. If it's your time they'll have to convert and risk missing some peculiar rule of your timezone, and if it's their, they will have to recognize it's their, and you take a risk converting it with a potentially outdated rule. As soon as you're talking timezones, you might as well go with UTC: you know how to convert your time in UTC, and they know how to convert UTC to theirs (and back). This completely obviates the need to know where your correspondent is and how time is managed there, and none of you have to care about which side of the globe the other is living in.

Also, the thread-top post link misses a particularity of timezones. As mentioned above, one should probably mention timezones but e.g CET or CEST not by offsets like GMT+2 (where you don't know if it's CEST or EET). This way everyone knows what we're talking about: north/south and DST in effect or not.

What's more, a properly maintained system should receive updates about the various changes of timezone DST settings. IIRC Russians had one not so long ago. Of course, if your system is not up to date, it has no means of knowing when the DST change will occur.

"someone travels with their Blackberry and changes the time rather than the timezone" That's not a bug, that's a PEBKAC and no amount of software goodness will prevent that. Actually they probably do that because they usually manually enter their 4PM LA appointment when in NY Timezone, and if they changed the timezone the event would be offsetted to 2PM. They are essentially specifying time relative to them and not relative to UTC. Maybe they failed to notice they can do that or they did not bother to set the timezone of an event when creating it, and as soon as an event with the timezone set comes in, they're toast. Combined with the terrible pervasive expectation that software is always crap, they blame the software and fail to notice their mistake.

Edit: Russian timezone update on e.g Debian: https://groups.google.com/group/linux.debian.bugs.dist/brows...

Notice how the system timezone file describes every timezone update, so that your calendar software can use it for your years old events.

There is more than just two companies, there always has been.

* Apple released iCal in 2002.

* Mozilla announced their calendar project (Sunbird) in 2003.

* In 2003 Novell purchased Ximian and their calendar software: Evolution

Those are calendar clients. Now ask Mozilla to write a calendaring server that syncs with every mobile device on the market across 150 calendars some of which have many thousands of appointments each (with years of exceptions to recurrence) -- no mistakes allowed. Also meeting invitations.

This one issue might be what keeps Microsoft in the game.

As usual, Microsoft's wide compatibility is less a function of their effort and more a function of their marketshare; if your stuff doesn't work on Windows, you break it until it does because if you don't work with Windows/Outlook/MS, you're locking yourself out of 90%+ of the market. Hence, MS is always the standard unless you don't care about that market, and there aren't many for-profit companies willing to forgo such a large base of potential customers.

Uhm, there are Exchange clients for iOS, Android, OS X, and I if I wasn't lazy, would Google to see if there are clients for Linux.

Right, that's my point. I mentioned Windows, but I didn't mean to restrict my comments specifically to that (though everything MS does is ultimately a play to further entrench Windows and/or Office, including Exchange, IE, and .NET). MS has other properties whose presence is so ubiquitous that no person attempting to make money off of the market where they dominate can seriously consider leaving them out.

Everyone works with Exchange because Exchange is the de-facto standard calendaring and email server. Of course, Exchange mainly got to this point through association with Windows and the MS brand.

If you want to do something with email or scheduling for users who work at a company with > 100 people, you're almost definitely going to need Exchange compatibility, and there's a pretty good chance you'll run into Exchange in smaller groups too.

The ubiquity of Exchange, Office, and Windows make not supporting any of these infeasible unless your primary motive is not maximum profitability and/or wide impact. For most companies, maximum profitability and wide impact are the primary reasons to continue business. Ergo, support for Microsoft's mainstay suites is a forgone conclusion.

It's a chicken-and-egg problem, as they say. Until we can get business people off of Windows and Office, we are stuck. If Linux vendors were smart they'd target the business workstation with at least as much enthusiasm as they target servers or consumer workstations.

Actually, Apple makes an iCal server:

http://www.apple.com/server/macosx/features/ical-server.html

And Novell GroupWise is the server-side product from that company which does calendaring:

http://www.novell.com/products/groupwise/features/#calendar

There are definitely alternatives, depending on what functionality you need.

I also forgot to mention Yahoo! who bought Zimbra, which pretty much covers what you just mentioned through and through.

Also see: http://alternativeto.net/software/microsoft-exchange-server/

As guelo wrote: Lotus Domino and Groupwise. (Not tomention facebook)

Lotus Notes is a solution in the same sense that suicide is a solution to being sad.

tl;dr Calendaring is the only program that requires distributed coordination in the Microsoft office suite, so it is the hardest to make.

Can't argue with that. But all of the difficult problems mentioned in the post are difficulties with any distributed system, not specific to calendar at all!

Microsoft has probably been having those same issues with Word, Excel, even Powerpoint as they have been working on making them collaborative for a long time now...

One of the most annoying thing is getting timezones right when you try to exchange calendar invites across systems. (i.e., sending a Lotus Notes invite to Google Calendar, or an Evolution invite to Novell's Groupwise, etc.)

There is an IETF standard for the calendaring exchange, which is why it works at all (albeit badly), but unfortunately, when you are trying to coordinate a meeting schedule across multiple calendaring systems, you're very likely to have the global calendaring problem where time zones matter. :-(

I disagree. I believe MS Project is a much larger/more complicated animal - at least when using Project Server.

How do you handle exceptions when a task update is accepted by a timesheet manager, but rejected by the project manager, all while the data was written to the back-end reporting cube?

Have you ever tried to level a resource on four competing projects? This is much more complicated than scheduling meetings using a single-use resource like a meeting room.

The high profile startup Asana is attacking problems like this. One of their advisors is Mitch Kapor, the founder of Lotus and designer of Lotus 1-2-3.

Is that different to Chandler, or is he having another go at it?

Wow, I just looked at the company profile. That's some list of advisors. They must have more money than they know what to do with, and sit on very nice chairs.

"word processors are probably the easiest" hahahaha.

Microsoft has been working on Word longer than it has been working on calendaring, and it still sucks too. I think the problem is the underlying assumption that Microsoft is the world's greatest software company.

Apple pulled Pages out of its ass in nothing flat, and it's a better word processor than Word. (Framemaker is also a better word-processor than Word, but it's kind of expensive.)

I wish this person had been more specific with respect to the specific problems with calendaring. The ones raised in this article look reasonably straightforward to me. "Thousands of machines, each one run by a human, most of whom are probably illiterate in computer skills and pretty much randomly clicking here and there willy-nilly" is pretty much the internet, right?

> But we've been working on calendaring for maybe 20 years now, and Outlook/Exchange still sucks.

I'm still not sure whether it says something about the calendaring problem, or MS software design. As much as I'm happy with outlook/exchange in general, my outlook crashes ~4 times each day. I tend to blame MS being sloppy, not calendaring being hard in that situation.

> Why is calendaring hard? Well, [...] the product requirements [...] some sort of ridiculous amount of distributed systems hackery around timeliness of updates, heterogenous network, consistency, graceful exception handling, and node availability semantics.

I'm sure Facebook, Google, Apple or Twitter can do that.

fta: "How do you handle exceptions to recurring meetings?"

If the user says "No meeting this Thursday," don't show the meeting on Thursday? Did I miss something? /cocky youngster, question stands

http://en.wikipedia.org/wiki/List_of_unsolved_problems

Hasn't Google Calendar solved this?

I think the author is a fool for making claims about how comparatively hard large programs like Word and Excel are without having any such experience. Reminds me of the fool last week who was making claims about commercial game development practices without knowing anything about it.

Reply on news.ycombinator.com