An UPDATE without a WHERE, or something close to it
rachelbythebay.com
rachelbythebay.com
Having run an `mkfs.ext3 /dev/sda` (note the missing partition number) by accident, I've learned to start potentially destructive commands by typing first whatever makes that command a comment (# in shell, -- in sql), then going back to the start of the line to remove the safety when I'm done.
That way ctrl+r won't bring back a destructive command.
However even with that, I agree with your second point. I always type
-- commit
in my SQL client and then highlight "commit" and execute only the highlighted text. That way I don't commit anything if I accidentally trigger the "execute all" action.It was a long time ago, but I think it was with SQLPlus on Oracle - and presumably staging or something non-prod since it's not a particularly vivid memory!
Start with ROLLBACK at the bottom (With a BEGIN TRANSACTION up top if the DB/env warrants it)
Only when I know I'm happy, change the ROLLBACK to a COMMIT.
You shouldn't get access to prod if you didn't have experience with Oracle. The database is just too fragile to survive inexperienced people changing it.
Well, this was back in the dark ages. Nowadays I'm sold on the notion that we shouldn't be running ad hoc stuff against the live prod database at all.
I'm sure there are exceptions, but I'll bet there are more temptations than solid reasons to do so!
Anyway, everybody should run their queries on a prod-like environment before running on prod, but on Oracle that's really not enough. Also the places that use expensive DBMSes tend not to have a lot of non-prod environments for people to test their scripts.
Nowadays Oracle supports you engineering your data so some of it is not on the bottleneck of anything and you can give some low amount of access to inexperienced people. But that's not the default situation.
Can't remember all the details. But something about adding a new nullable column with or without a default value of some kind. It had to lock the table, but multiple queries were already running. Those queries had some locks already, but then needed those rows the migration had already gotten hold of. Leading to this huge deadlock I had no idea how to solve.
Worst part is, it was one of the first times we tried deploying during working hours. At the time we normally only got to deploy 4 times a year, but we pushed for a more modern approach. Luckily the server guys were on our side, and a few years later that government agency is one of the best technical places I've seen after a complete revamp.
But our team learned the hard way that using transactions on the replica pg database actually locks it from getting updates globally during the duration of the transaction. And the whole idea of connecting to the replica was to not wreak havoc, oh well..
I went in there to adjust some data so that one of our dev users could re-publish some data which would then be synchronized to another database(update foo set published = false) and I couldn't understand that it wouldn't update in the UI.
I ended up telling him I'd have to get back to him, and the issue didn't dawn on me until the app logs started spitting out errors about not being able to retrieve a connection from the pool. YIL about pending transactions in DBeaver.
$ ls /
$ ls /tmp/junk; rm !$
It's taking "/" rather than "/tmp/junk"..EDIT: formatting
What you want is "ls /path; rm $_".
Even then, the above is fairly pointless. At the time you look at what you are deleting, it's gone.
1. ls /path
2. rm <esc><.>
(where "escape" "dot" brings up last argument) without the need to fiddle around with dollar signs underscores, exclamation marks, etc. to prevent further mistakes (e.g. "was it $! or !$ ?", shell expansion, etc)
Otherwise, simply having a mindset of, I'm doing dangerous things also goes a long way.
Never do things if you are in a panicked/frantic state.
Edit: I guess running galera is actually a blessing. I can't run any write/update methods on our qa/prod data...
I religiously write it out of order for this reason. The IDE complains for a bit, but it's better to deal with some squiggles for a few seconds until you've filled in the column assignments part.
At least for me, it’s safer to just… write the thing correctly in an environment incapable of running it.
This problem is fixed in query languages like EdgeQL.[0]
Not quite true, DISTINCT, ORDER BY and TOP/LIMIT happen afterwards.
WHERE foo=bar LIMIT 1;
Then ctrl-a, and fill in the rest: —- UPDATE name=“new name” WHERE foo=bar LIMIT 1;
Then take a moment, read over what I typed, and hit ctrl-a again and remove the comment string.Ideally I’m doing this in an editor (not the db shell) as well, and when I’m done pass the saved file in on the commandline. I try to type as little as possible in the db shell unless I’m logged in as a read only user.
I also set my db shell to display the current username and database so it’s always right in front of me. And I never, ever use command history in shells to construct new commands. I swear that bit me more frequently than I got it right when I used to. It’s like a footgun with an extra footgun attachment.
Given the annoyance of this and its ability to really, really ruin your day, I don't know why someone hasn't updated their parser to allow 'update X where [cond] set [blah]' as an alternate phrasing.
Of course if the update is not only important but critical, manually starting a transaction before is the first step ;)
BEGIN TRAN;
-- TODO: update statement
ROLLBACK;
Usually I'll run it with the ROLLBACK first, to confirm that it impacts the number of rows I'd expect, and only then change my ROLLBACK to a COMMIT.And yes, before that I had run an UPDATE without a WHERE in production..
Instead I do my dry run queries with a read only user and I select the affected data, often using CTEs (or temp tables where performance is an issue) to model any intermediate state. I don’t ever run any writes of this kind without review, backups, and automation.
Run code first, throw an error to force rollback the transaction...
Well, dangerous if you are using a simple terminal interface and not using a transaction when doing updates. The latter is generally a bad idea even if you aren't also doing the former.
Linq is closest to the a more natural querying language we have come up with. Too bad it doesn't do update.
But that doesn't mean it's ideal. It just has a large most that people wasn't able to disrupt.
Check out linq both the sql-like form and the lambda form. It feels much better for what it does.
UPDATE users SET password = '23r23r23rdsf';
Somehow he missed the "WHERE email = 'someone';" I forget exactly how many users had their password changed that day. For some reason (maybe MySQL didn't let you cancel an UPDATE like that way back in the old days?) to stop that query he ran into the other room and unplugged the MySQL server. The PROD server. It's funny to look back on that now, but at the time... oh boy, total panic.
If I just do 'SELECT *' and execute it, it does not rotate through all the tables. I did not specify it. Same with 'SELECT from xyz' if I can not have it empty and return all. Yet where is special somehow.
For example, I would like SELECT to be FROM, WHERE, SELECT. UPDATE could be UPDATE, WHERE, SET. But then DELETE would end up inconsistent...
> If I just do 'SELECT *' and execute it, it does not rotate through all the tables.
Because you did not specify what set you wanted to operate in, and the set of sets isn't meaningful because of schematic differences.
> Same with 'SELECT from xyz' if I can not have it empty and return all.
SELECT is asking what pieces of the subsets (rows) you want to display. If you don't ask for any, you don't get any. You are asking for the number of rows in xyz times zero. That's zero. You can write SELECT 1 FROM xyz and get 1 returned for each subset.
And yeah, thank you. Today i learned new thing.
Or at least (for now) a configurable option in the database config, so each site can switch it on as they like.
Adding "UPDATE xxx WHERE yyy SET z=42" to the grammar would be a nice addition too.
select *
--update <blah blah> [or delete]
from <blah blah>
where <blah blah>
which allowed me to execute the entire query as a select (or if I accidentally hit "run the whole buffer" it was safe), but then allowed me to highlight and run just the update (or delete) statement and ensure I had the same where clause as determined by my select statement pre-flighting.There's no signal you can do from the client directly. You can do a kill thread ID from another client, but that's only checked at some points, and I wouldn't expect it to stop an update in progress.
Kill -9 the unix process should work, but pulling the cord might mean less chance of changes persisting to disk.
You can't even easily ask "are you sure?" because if you're asking that for every little thing it ceases to be a useful guard. You need tools that detect if you're doing something stupid and dangerous and only ask then, but in the limit, that's strong-AI hard for ops people. That is, there are some obvious ones you can try to catch... "did you really mean to unassign all IP addresses?", but in general there's always something that will go wrong more cleverly than your detection code.
Hooking machines up to orchestration code is something I have to do. I operate at scales that Facebook would laugh at, but they're still well beyond what is practical to manually manage. In my opinion that scale taps out somewhere in the large single digits per ops person, which is nothing nowadays. But it always makes me nervous to do so, too, because I can see I'm putting all my eggs in one basket in the process, and the traditional "watch that basket really hard!" answer for when you're stuck in that situation is visibly not adequate.
I don't have a solution to propose. The tension seems fundamental to me. All I can suggest is that everyone sitting in front of any devops tool always be keeping the possibilities in mind, despite your brain's desires to say "hey, the last 1000 deploys went fine, I can stop being so vigilant about this one", and that any guard rails that can be added should be, even though they can never be 100% effective.
Continuing the SQL analogy from the OP, you absolutely should be able to UPDATE an entire table, but probably the syntax is at least somewhat to blame, because very rarely you want to run an update on every row. A simple change could be that an UPDATE without a WHERE clause is a syntax error (you could still add "WHERE TRUE", if that's what you mean to do).
Another example is how the React API uses funny method names like "dangerouslySetInnerHTML" for things you aren't usually supposed to do.
I'm a big believer in making invalid states unrepresentable, and a straightforward extension of that modus operandi could be "make unlikely states hard to reach".
With 20.000 servers, for example, an automated process could roll out updates to 1,000 every hour during 20 hours, check that response times, system load, etc. are within limits for the updated set during each hour, and send out alerts and pause updating when they do not.
So, the user still would press one button to do the update, but the change would slowly take effect, allowing both the system and humans to take action if needed.
Main problem there is to keep things flexible enough to allow somewhat out of the box updates. And of course, that requires that you can run with half your servers on a different version of your software.
you probably also will have to forget doing the entire update in a single transaction.
However, you still have things like "this update severs the machine from the management system due to unexpected XYZ", "this update is fine until it's rolled out to 80% of the world, at which point interactions with the other deployed systems hammer the system so hard the management interface can't get in properly", "this update looked fine because it was using almost entirely cached data but once the caches all expired it turned out to be a disaster, now restoring is a nightmare because we had to roll back the version, empty the cache, and regenerate everything", and all the other edge cases that no matter what you do, will cause cascading failures at a huge scale.
No matter what rule set you write, something's going to get past it.
Or, to put it another way, if you aren't yet on the Pareto frontier between power and safety, sure, by all means go get your free safety and power. But you will hit a limit on the two before you have all the power and all the safety, and the limit you will hit is going to be uncomfortable in at least one direction.
Somethings probably randomly fail, but because entropy and time is randomly selected we find ourselves in the successful ones.
If the number of pushes is small and the time to make the change is small, automating the change, but running it one at a time in a loop works ok. When things start breaking, you can stop the loop before too many servers fall over. If you have a lot of servers, you can split your hosts and run up to about 10 terminals doing loops before it gets really hard to supervise. Often, you can easily parallelize the prep part of the update, and leave only a quick change to be serialized.
I'll have to see if I can find it, but yinst-pw was opensourced somewhere and is really useful for sudo password prompts if you're doing it half-way like this. Edit: ahah, remembered it got renamed to autopw https://github.com/jschauma/sshscan/blob/master/src/autopw
Haven't used MySQL in a while, but when I was using I'd have this alias in my zshrc:
alias mysql="mysql --i-am-a-dummy"
It hasn't happened since. root@localhost [main]> update user set password = 'abc123';
ERROR 1175 (HY000): You are using safe update mode and you tried to update a table without a WHERE that uses a KEY column.When I thought I was done with the page for editing configurations, the UX person said it missed a popup confirming the config changes if it affected more than X items.
I'd prefer skipping it. It's more state to hold, data to be fetched up front etc. But it has probably saved us multiple times already from someone trying to change something used lots of places by accident (instead of making a new separate config for whatever they want to change).
When dealing with azure app services on the command line, you specify the slot with the `-s <slot>` flag.. but if you don't, it defaults to the production slot.
I'm really not sure what kind of moron though that was a good default, because the az CLI doesn't ask you for confirmation for anything. If you run `az webapp restart` you just restarted the production system.
If typing in an interactive session usually use a specific user account for it, with minimal set of privileges. And wouldn't have the DROP privilege or any other DDL statement privilege. DDL would be scripted out, tested and ran using another user account that had only privileges on specific databases it needed.
> TRUNCATE is not MVCC-safe. After truncation, the table will appear empty to concurrent transactions, if they are using a snapshot taken before the truncation occurred. See Section 13.5 for more details.
> TRUNCATE is transaction-safe with respect to the data in the tables: the truncation will be safely rolled back if the surrounding transaction does not commit.
In SSMS (the main query console tool thing for Sql Server) you can highlight a bit of code with the cursor (as if you want to copy/paste it) and press CTRL-E to execute it, its really handy when you've got a big sketchpad-like series of SELECT statements and you're doing exploratory tinkering.
But if your UPDATE statement is accross three lines, its a little bit too easy to accidentally select just the first two lines and not select the third line that has the WHERE clause. Then CTRL-E and you've footgunned yourself.
Its never actually happened to me, but I've often thought it probably has happened to some people.
I haven't built one so I'm not complaining!
Powershell FTW.
With SQL, that could be as simple as starting a transaction, doing your commando stuff, and committing it when you are satisfied.
But don't do that. Why are you doing anything like that in production. Why why why.
https://docs.ansible.com/ansible/latest/user_guide/intro_pat...
One can form a set by specifying just a table, or tune more by JOINs and further with WHERE clause.
Well, of course a JOIN is just one way to say WHERE. Still, when properly joined, the resulting set may not need a WHERE in UPDATE.
Edit: Gah, what have I become! Listen to me! I am so old. Fuck it, YOLO right? Oh, wait, I've got kids in college.
I think validation that a query is going to do what you want is up to the user of the database, not the database vendor.
OK, this isn't about breaking an entire network but small details can be super helpful. I have only found a small number of companies who seem to care about error messages.
Happened to me 18 years ago. :-)
A SQL statement without a WHERE looks too suspicious to me to make a mistake like that.