ora600

Here's what I do in these situations (I'm an Oracle DBA, but this should apply to most loaded shards):

1) Use connection pooling at the application layer to prevent overloading the DB of any specific shard. This means that if a shard has 16 CPUs, having 16 connections sounds reasonable. Additional connections will not give you more performance. This means you need to queue and throttle requests at the application layer and with some thought you can probably figure out what to do with the waiting users - show partial results? show a nice whale? A "loading please wait" sign?

2) If you didn't do #1 and the DB is getting overloaded, my normal response is to start shooting down connections. Oracle has separate unix process per connection. MySQL has its own way of shooting connections down. Put up a small script that will kill the correct percentage of sessions to prevent overload on shared resources. This will generates lots of errors and will cause a percentage of the users to hate you, but you won't be down.

As a fair warning - going up Page Mill requires being in very good shape. Its 30-60 minutes of climbing, some of it steep.

Which in my experience is horribly broken.

Agree. I live in MountainView and still drive there for the coffee and ambiance.

Supporting evidence:

Two years back I tried Mint, Wesabe and the new Quicken online.

From those three sites, only Mint could retrieve data for my WellsFargo visa. So I stayed.

Quicken was the worse - they actually made changes to my account (for which WF notified me by email) and their support was basically useless when I asked them about it.

So, the name had almost nothing to do with my decision - if Wesabe had the same scrapper as Mint, things would have been different.

(I misspelled Wesabe twice while writing this post - so maybe the name does matter... but I still maintain that a working product matters more)

well, yeah - but I thought that the possible backstory due to the timing of the publication was exciting.

Maybe I'm missing something, but does anyone see what the tool actually does. I saw the constraint solver, but there are already many existing packages that do the same (Octave is open-source, I believe). What is all the excitement about?

Sometimes cluster systems are mistaken for high-availability solutions while they are actually load-balancing solutions and can decrease availability due to added complexity and dependencies.

I'm often encountering systems that are designed for very short queries processing small numbers of rows where the number of connections from the app servers is configured to be far greater than the number of CPUs.

Since those systems are doing very little IO, configuring connection pools to start with much more connections than the DB has CPUs and to add more if the connections are busy (i.e. DB is getting slower, probably because it is highly loaded), is guaranteed to cause resource contention on the DB and escalate the issue in case something goes wrong.

Its a total waste and yet the most common configuration in the world.

Daniel Abadi is one of the authors of the HStore paper that VoltDB is based on. It looks like the deterministic order system is using similar requirements and assumptions.

I'm not sure how to answer that. How do you know how much of Microsoft's time and money goes to developer support? How do you know that Oracle does any less? I don't have any real hard numbers.

Microsoft MSDN for developer support and Oracle has OTN. Both companies have developer oriented conferences in addition to business conferences. Both companies employ evangalists that work with startups to help them adopt their technologies. Where do you see the difference?

Don't get me wrong - Postgres is awesome and I would happily use it for any project. I'm not sure Postgres is easier to manage than Oracle and places any less maintenance overhead, but YMMV.

Its just that RAC is a special technology with no equivalent on other DBs. It is usually placed where you need load balancing on the DB level. It normally has very high availability that is a good fit for mission critical DBs. I hate seeing good technology wasted, but you are right that if Postgres is already there and works well, there is no point in moving anywhere.

Oracle used Sun's hardware to build the Exadata - which they very aggressively sell as a data warehouse solution. Competing with Netezza, Teradata, Greenplum and the likes.

I would love to hear why you made this decision.

Maybe I don't understand the concept of "developer mindshare" well enough but it seems to me that Oracle does engage with the user community and not just the business community.

They do this through user conferences such as Collaborate and ODTUG (which is developer specific) - those conferences are much more technical in nature than the marketing oriented OpenWorld.

Oracle also have the OTN and its ACE program for engaging with the technology people, including forums, beta programs, code sharing site, etc.

And of course, Oracle supports the local user groups which are also technology oriented - many of those user groups are highly critical of many Oracle decisions (and you should have heard the noise when Oracle Support site switched to flash) - but Oracle still supports them because they know that an active user community is essential.

So I was hoping you will accept one of my other examples of companies with large numbers of bad employees that show no intention of going out of business any time soon.

The cellphone market is highly regulated, but the large computer and software manufactures don't seem to depend on government manipulations.

Sigh.

HP? IBM? Oracle? Verizon?

Although "market manipulation" is broad enough to include any large company at all. Same with "anti-competitive barrier rising".

So, we can say that past a certain size companies become evil bastards and have many bad employees and they survive quite well.

They are still not government though, and consumers could choose to stop using their products and they could go out of business. Except they don't.

You will not believe how many bad employees a large organization can carry and still grow and prosper.

I mean, ever talked to an AT&T call center? Do you see them going out of business any time soon?

There was this guy I worked with on a previous job. He was simply unqualified to do his job, and displayed no interest in becoming more qualified. He did the bare minimum to survive, and very often others had to do his work for him so they can make progress in their own job. For various reasons his managers preferred to keep him even though he knew of his low performance. He was there for 3 years when I left. The organization is in no danger of dying, even though around 30% of the employees were of similar quality.

SLAs seem to have the effect of dramatically changing the culture of the team that is now measured by the SLA.

I remember two times when my managers announced an SLA. First time was "Every priority 1 ticket has to be answered within 24 hours". Before the SLA we were a cheerful and overworked IT team, doing our best to handle all tickets within reasonable times and with our best quality responses. Afterwards - We started negotiating priorities, replying "investigating" within 24 hours to make sure management doesn't get on our case, and we had lots of not-so-funny jokes about announcing "love every 38 hours" SLAs to our spouses.

The second time they actually went as far as saying "This SLA is between marketing, sales and the customer. You don't need to worry about it. Just keep on doing what you always did."

They had to say that, because with our current equipment and processes, offering 99.9% availability to our customers was impossible. Our real availability was an average of 99.5% with some services as low as 97%. Not because we were horrible IT - but because of trade-off decisions we reached in the past together with our management.

It didn't help much. Because we didn't trust management not to start holding us to those SLAs at a random moment. Which they did.

Again, from a team that did their best to keep servers up, we became expert negotiators of which downtimes "count" and which don't. Also, we because very defensive: "We can't risk upgrading this month, we were already down 13 minutes."

I understand the need to sell to corporates, but I would be very very careful communicating SLAs to my team.

I am actually talking about 4 different buildings in 3 different bay area towns. If there is a secret to picking buildings, obviously I'm not in on it.

But - I looked at apartment ranking website, and on every single building there will be some very negative comments about noise levels. So I still suspect a more systematic problem.

Agree, except the single family home.

Because the quality of life in US apartment buildings is appalling. I don't want to hear every step that my upstairs neighbors take. I don't need to know when they go to the restroom in the middle of the night. I don't need my downstairs neighbor ask me not to walk after 10pm.

This is ridiculous. I'll live in a US apartment once the sound insulation issue is resolved (hint to US architects: It was solved by 1930 in the rest of the world).

Wondering what you mean by "license to BigTable".

BigTable is using GFS, so it will not be much use to Oracle without GFS. Also, since Google nicely described some of the major ideas behind BigTable in a paper, there are few Open-Source implementations of it. I expect that like many database systems - getting a license is easier than building and maintaining a stable system :)

Actually, the customers that use OEL are normally customers of the Oracle database. They get the OS for free as part of the license deal and enjoy not having to play the finger-pointing game with Oracle and RH support any more.

You have no idea what a relief it is when your DB vendor can't blame the OS for bugs and performance problems, and actually have to solve them.

Thats why we love it when Oracle does storage, network and hardware (i.e. Exadata) - we love having one throat to choke.

Speaking as Oracle consultant, very often you have no clue how much work is going to be there until you are there. We are very lucky if management actually listens to what you need. Which is often very simple things such as fix reoccuring errors, upgrade components that keep crashing, put decent monitoring in place, make sure the backup strategy makes sense and test the backups.

I've been in situations where I was contracted for 30 hours to "just help us deploy X", discovered what the database looks like and have been able to upgrade my contract to 300 hours of upgrades, monitors, automation, backup verifications, etc.

I can see how this looks like a contractor trying to create lots of work, but all this work did have good reasons behind it.

I have to sell my rationale for extra work to the customer and it is not always trivial. Often it is easier to say "well, its your data, its not like its my problem that your backups are untested", but I wouldn't be in this business if I didn't care about data being safe.

I completely agree that contractors are more focused than employees on getting the job done well.

Once I've overheard two developers talk about me: "Well, of course Gwen is always cheerful and always happily does what we ask, she's a contractor - thats how they do their marketing.". I'm proud of cheerfully and competently doing my job. I'm always wondering why many full-time DBAs act like they are doing you a favor when they create indexes, clone production to development and generally do what they were hired to do.

No offense meant to the wonderful HR people who are no doubt out there, but from my experience HR people are good with:

* Processes, procedures and forms

* 401K, health insurance and other benefit programs

* Buying salary surveys

* Finding template bonus programs, performance evaluation programs and hire-a-friend programs. Adjusting them a bit to fit current organization.

* Employment and immigration laws

I've never found them to particularly understand people and relationships. I've worked for 4 years for a corporation where I never once managed to talk to an HR person. Everything was done through support tickets. But even in smaller companies, I found that HR folks are more of systems-people than people-people.

I guess the question is - does the customer know they don't exist yet but you have every intention of delivering?

The only way I can outline a good presentation is with a sharpie and sticky notes. One concept per note, you can rearrange them easily, have an idea of what the presentation will flow like, and then turn each concept to a slide.

I'll probably stop doing that once someone writes software that can replicate this workflow and integrates with presentation software.

Similar to promising potential customers that your product is doing things it doesn't actually do and hoping you can add the extra features before anyone finds out.

I view those as distasteful, but I've yet to see any company/salesperson who doesn't do it. I'm sure this looks different once you are in sales and everyone is doing it, but from my seat over at operations it sure looks dishonest. I hate it when vendors use these tactics on me - so how can I do this to anyone else?

Being the admin, I'm pretty sensitive to:

1) People trying to bump the priority of their own tickets by showing up in my cube. This is inevitably at the expense of other, more polite, users of the system. Prioritizing requests is one of the toughest parts of operations job - if you think I'm doing it all wrong, you can talk to my manager.

2) People who think that "open a ticket" means "go to hell". It actually means "We need to document all changes done to production and we need to prioritize requests. Please help us do our job by using the system designed to do so. The request will be done within 5 minutes if its urgent/important/really simple. But we still need a ticket to document it and follow up if it causes issues in the future."

On the other hand, if you boss is competent, he can raise the availability and priorities of admins in all kinds of management meetings. Hopefully resulting in better priority for developer requests and maybe even hiring few more admins.

Actually, I found more and better solutions in blogs than in official documentation and books. Especially the topic of edge cases or documentation that doesn't work as advertised or error messages that don't mean what they think they mean.

No official install guide will say "But if you are running this on Ubuntu 8.0.4, it will throw off a spurious error message about wrong xpong version. you can ignore it and everything will be fine or you can fix this by running the install with --xpong-compatible", but bloggers say things like that all the time.

I try to blog every time I run into misleading documentation and every time I run into an error and there are no useful results when googling for that error message. The error messages and install guides are probably the biggest traffic generators for my blog.