Google Drive issue affecting a significant subset of users
google.com
google.com
http://i.imgur.com/zAJVpAF.png
Notice the grammar and punctuation issues not to mention the email address to nowhere.
uploader-eng@ would presumably be uploader-eng@google.com
The error message also looks like it is also geared towards users with google.com accounts.
The whole thing smacks of someone pushing something internal from development to production.
They often do detailed postmortems so hopefully this one might get that treatment - though given the accidental exposure of the 'not for public release' UI and client they may well not go into much depth with this one.
Regarding spelling and grammar, it checks out fine for me. When you're typing a lot of code, it's easier to speak in short sentences and redact useless info (in this case, the @google.com email domain.)
I really think they have some policy to say "some" unless it's literally 100.00000% of the user base. Cause PR..
I wish they would improve on their communication accuracy in this aspect. I really do think it would make people trust them more in the long run. (Just tell us a percentage estimate! Is it 1% or 70%? Don't have an exact count? Just give us your best guess! Anything else than this really an insult. We're all in this together.)
What is odd ... I cannot find anyone who was upgraded to the new client that isn't getting the error message. I wonder how they define "some."
It would be interesting to see what criteria was used for upgrading. I've seen reports of Windows 10 and Windows 7 users being upgraded, but I have a Windows 7 machine that is still running the old Google Drive and hasn't been upgraded.
Well there's this one guy in Pacoima, and he's having no problems at all. "Some".
Do you have a .google.com email address?
Can you post a screenshot of the menu options for the new new Google Backup and Sync? I am interested in learning what is changing.
It seems like google released an early beta / alpha version accidentally.
Google Drive service has already been restored for some users, and we expect a resolution for all users within the next 12 hours. Please note this time frame is an estimate and may change. We are deploying an updated client now. The Windows client will automatically update with this fixed version within the next 12 hours. Alternatively, affected users can download the updated version of the client from https://www.google.com/drive/download/. Affected Windows 10 users may be required to sign in to their Google account within the client.
My organisation did a migration to Google Drive (from internal Windows server) in the last few months. I've had quite a few issues (I'm the administrator).
I'm posting this (abridged) laundry list of problems here...would love to have some feedback or pointers to what's wrong or what I'm doing wrong. When I've managed to get through to Google Drive tech support I don't get much in the way of answers.
For context, I'll confine this to just two central share accounts: fileshare@mydomain and records@mydomain. Each account has around 1.5 terabytes of files. Our org has 85 G-suite accounts (users). This is a small to mid sized business.
* I'll mostly skip the part of how difficult, slow and expensive a process it was to get around 3 terabytes into Google Drive. rclone was a good friend to us. It took months as we had to constantly re-sync as we migrate....we are running a business while we migrate. I'll also mostly skip the problems we had with the Drive sync client...its default behavior is to sync everything. I had to globally disable usage of this tool. Guess what happens to your company network when 50 users install the sync tool and click "add to drive" for the company fileshare. Good luck educating users with a wide range of PC skills on proper sync usage.
* Drive's file sharing operations are opaque and take an indeterminate amount of time to complete a share (could take days). If I need to restrict access to a large subset of files, I have no idea when the share is complete. I have found that sometimes it does not complete for all users and I'm left with some files not shared. I can't find a pattern. I have no tools to help me find what (under a large set of folders) didn't get shared. If a user or me as admin needs to unshare, this can take days. So there is no practical way to protect sensitive access files except put them in different root folders or different share account.
* No practical way to get a backup. I'm trying spanning backup at the moment. Its been running against my fileshare account all week. The spanning backup console tells me I have about 100 days to wait to complete the initial backup. Yeah, 100 days to backup 1.5 terabytes. What's the use of unlimited storage if I can't protect it? In this case, I'm suspecting Google Drive, not Spanning as I've become familiar with how performance variant the Google Drive API is.
* Google Drive allows an end user to easily do "destructive" things to the central file share. Last Friday I received a call from a staff. She knew something bad just happened and quickly reached out for support. She had accidentally used the move or delete operation in the Drive UI to cut out a huge subset of our fileshare. I say "move or delete" because she didn't know and as an admin I couldn't figure out what happened. I dug around both the fileshare account and her account and couldn't find the files. The Trash folder on both accounts was empty. This didn't make sense as the activity panel clearly showed her and other users deleting things. At first the Trash view returned a server error. I refreshed my browser and got it to show. Empty. I tracked through the activity list and found the operation of concern....it showed the folder had been removed. Fortunately, I was able to click on the folder in the activity panel and it took me to it. From what I could tell, the files were in the system, but not attached to anything...no parent folder, not marked as deleted or moved. I was able to move this unattached folder under a new temp folder at the root of the fileshare account. Then go to that folder and move it again back to its original place. Then we wait...for the removed share permissions to fall back into place. Good thing this happened late on Friday as we had the weekend to wait for the shares to come back.
* The Google Drive API is slow and unstable. I'm a programmer. We spent quite a bit of time writing a system to push files from our records management system to our Google Drive records account. Currently, we are adding less than 1000 files a day; mostly mid sized jpg's (1 to 3mb). So well below our API limit of 1 billion requests a day. Our server using the drive API tracks errors. Currently, around 1/3 of the uploads fail and need to be retried up to 10 times. Some never make it and we have to handle it manually. Good thing we wrote this in Elixir/Erlang as fault tolerance on the API client is our friend.
* The Google Drive API is "global". When you create a service account (Google's API method for server to server communication), you give access for the scope of the API but not the account. Our system needs read/write access to only the records account. No can do. You have to give your API keys full read/write access to every account. So yeah, your dev and test systems have full access to all production data. Better be a very careful programmer. If you mess something up in dev, you could mess up your production drive accounts.
* Combine these API problems with the inability to do a backup (and restore as well as restore to a test account) in any reasonable amount of time...and you can throw away your discipline of separating dev from production and being able to run repeatable tests before deployment.
I'm going to spend the rest of my day reviewing code with a programmer. The code is to refactor our folder structure in our records account. It shouldn't delete anything...just move lots of folders into a new structure. But if we have just one bug in our migration scripts, well who knows. So I'll be reviewing code today and on pins and needles Saturday while our migration script runs on our production drive.
BUT - have you tried this question in /r/sysadmin? This is the sort of question that would be great for that sub.
Google tech support recommends posting to Stack Overflow and tagging it so their support folks can take notice. So far in my reading of lots of SO posts, the Google tech support official answers are of little help.
The key reasons we made the decision to migrate from our internal file systems were:
* The old internal system is dying and out of space. So either reinvest or move to the cloud.
* We're a school. Half our teachers work remote as do around 70% of our students. The old system meant no students had direct access to their records and remote staff had to use a clunky and slow Remote Desktop over VPN solution.
* Our new records management solution for students and staff lives in the cloud. It seemed appropriate to integrate to the Google Docs/Drive solution in the cloud. Integrating into an internal file server would have been tough.
Anyways, something something about the best laid plans ;)
How the hell have you managed to stay sane dealing with it on that scale?
Nothing prepared me for the reality of production.
Google Docs is doing well. Its Drive that seems not "ready for business". And Google's tech support is very coy. That don't acknowledge these problems when I get them on the phone.
* Set user permissions really tight. I've had so many issues with users dragging folders around resulting in everyone re-syncing.
* Use a better desktop client. The Google one is really basic. We use SyncDocs which is more robust and has the features you need.
* Your API issues with jpg files are probably due to you tripping Google's anti-piracy system.
* Set up a separate Google Domain for testing stuff. You'll have to pay extra for it, but don't test on production data.
> * Your API issues with jpg files are probably due to you tripping Google's anti-piracy system.
Holy smokes! Didn't know this was baked in. Our school teaches art and creativity. The jpg's being uploaded are pictures of student's original work: painting, metal works, jewelry, glass blowing, hand written diaries and sketches. Our images are pretty darn unique (and some amazing art as well). Would love to know if Google's detectors are tripping up.
> * Set up a separate Google Domain for testing stuff. You'll have to pay extra for it, but don't test on production data.
Good idea. That solves part of the problem. But it doesn't enable me to have a set of test data based on near-current production data. And god forbid I someday make a mistake on production...no way to rollback.
No, you just don't see it mentioned much, perhaps as it is just seen as "un-cool" in some circles to paint anything Google does as being less than perfect.
Nice double entendre.
Which isn't to say I think your use case is unreasonable or that I disagree with your assessment. But regardless of right or wrong straying from the golden path with SaaS is dangerous. With any software really, but SaaS is by far the most precarious. If it's not designed with your use case in mind it's not uncommon to run into bugs or deprecations.
My not-so-practical advice:
My first suggestion would be to evaluate if you really need that command and control or if it's something you're used to having. Once you let go of that dealing with Drive as an administrator becomes dramatically easier. The worst thing you have to deal with is recovering deleted data (available for 25 days) and complaints about the Windows/macOS clients which most people end up ditching as they realize the golden path there is the web interface. Obviously this depends a lot on management style and culture. Sometimes letting go is so hard that it hurts more than it helps even if you commit to it.
My second suggestion would be to evaluate if you're using the right tool for the job. fileshare@mydomain seems obvious but records@mydomain makes me think you're using a file sharing system for something that would be better served by purpose built software. Like a CRM or a document management system. Even an SMB share is a terrifying place to store structured data built around human processes that require them to follow rules and not delete things (even at small scales).
More practical:
Sync client: If bandwidth is constrained roll out the client to users in comfortably small increments.
Drive API: I may be misunderstanding your use case, but I think this is just as simple as logging in as the records user and authorizing the application instead of granting it global scope to the GSuite account. Here you're going against the grain in a couple different ways: That kind of access is meant for things like an interview scheduling system that needs read access to all calendars. The expectation is that apps that need access to a single account would simply get that access by asking the user for it.
Backups: Besides using the sync client and taking snapshot based backups from the local copy (which is a half-assed fragile hack) the only thing that makes sense is trusting Google. That's only fair to ask of you if we're not talking about data that would sink your business if lost. But if all of the data is that critical are you using the right tool for the job? Even a local SMB server setup would need to be carefully designed to account for the myriad ways you can lose data. If it's only a subset do you need to back it all up? Maybe what you really need to backup is a manageable subset.
Good luck. I hope you get this sorted out–it sounds like a huge headache.
For those reading along or for my own purposes of tracking thoughts for a case study of this experience, I'll add some comments:
> Drive isn't (at least yet) designed with a central group share in mind.
Trouble is Google advertise it as being suited for this. Google provides well written docs to guide you through setting up a central file share for your organization. I just looked for the doc again. Its been moved/removed in the last few weeks. Now I see they have a new/improved feature called Team Drive which may solve some aspects of my troubles.
> Especially if they use one of those insane massive shares that the whole company has write access to.
Google advertises a 10 terabyte drive for $99 a month. The Business and Education plans advertise unlimited storage. This combined with a wealth of other docs Google provides for using G Suite for Business may lead a reasonable person to conclude it could handle a measly 1.5 terabyte drive account.
> Drive API: I may be misunderstanding your use case, but I think this is just as simple as logging in as the records user and authorizing the application instead of granting it global scope to the GSuite account.
Yeah, that's the next step for us to write some code against. Google Drive's API docs tell you that for server-to-server API usage, use the service account method. This method gives access (for a given scope) to all accounts. I've found no docs or example code for server to server API using the client/server OAuth method which would restrict access to a single account. rclone uses the client/server OAuth method. Its tricky stuff. To do this on a server we need to write a process so a server admin can manually bootstrap the OAuth access and cache the access tokens for reuse by the server. This method requires a bit of tricky work to refresh the server's access with a refresh token at times. No docs on how long this access is good for or how often you can refresh. So to do this for long running server access, we would need some method so the server can email me when the token expires so I can manually bootstrap the OAuth again.
> records@mydomain makes me think you're using a file sharing system for something that would be better served by purpose built software.
Sure. My use case wan't clear in my original post. The reason I picked this approach is Google Docs is great for our needs. Having the student's work evidence embedded in a shared drive (in read only folders) alongside other folders containing Google Docs of teachers assessment, feedback and admin docs is pretty slick. Actually this stuff is working great. Its the ops/admin aspects that aren't ready for business.
> Sync client: If bandwidth is constrained roll out the client to users in comfortably small increments.
When I checked about a month again, my only option as an admin was on for all users or off for all users. Hard to manage a roll out plan when Google pushes their sync tool on users and I can't control access on a per account basis. For now, its off. I'm happy to leave it that way ;)