AWS CloudShell
aws.amazon.com
aws.amazon.com
My impression is that the stuff that GCP does have tends to be more capable and production ready (although I'm curious if others would disagree with that).
The only one I can think of that might be in this category is SimpleDB. AWS recommends you use DynamoDB instead of SimpleDB for new applications.
However, I wouldn't call SimpleDB abandoned. SimpleDB continues to work as it has for may years.
One thing that AWS is amazingly good at is not breaking existing customers and their applications. Did you build an application 10 years ago based on SimpleDB? All the APIs you used 10 years ago are still there and available to your application today. Its really quite amazing how dedicated AWS is to not breaking existing customers.
Otherwise yeah, there might be too many ways to do the same thing, but AWS generally has a stellar track record of supporting everything else they have made.
I'm searching just now over *.amazon.com and nothing comes up. The product pages also bear zero indication of that.
Amazon are like the anti-Google: rather than killing products, they're happy to have them limp along forever.
https://aws.amazon.com/blogs/compute/implementing-dynamic-et...
Fair enough that services often continue to work the way they always have, but I'm thinking more of the case where you're actively developing something, and the AWS service has major bugs or is missing significant features that all the alternatives have which makes your job harder building on top of it.
Elasticsearch was a famous example, it went years without any updates during which time upstream Elasticsearch itself improved dramatically. Then they picked it back up again once the elastic.io hosted product got good enough that it was a much better alternative.
Another example is ECS, which was out in the wild for a couple of years with a very limited feature set while GKE was completely eating its lunch and upstream Kubernetes got a lot of major improvements. Then AWS released EKS which sort of seemed to replace ECS for a while, but they have gone back and forth now for a bit with ECS having some features that EKS didn't have (e.g. fargate for a long time) and vice versa.
There are all sorts of other support bugs I've stumbled across in their forums, too many to recall. Often years old threads that have never really been addressed.
Edit: another one that comes to mind is cloudwatch/the entire monitoring & logging stack. It's very basic, not really a viable alternative at all to something like splunk. As such an important thing, I always kind of expected it to get better but it just didn't, you had to export your own logs/events to a separate ES cluster, or to Redshift via S3, or something else, etc. Whereas GCP Stackdriver is a much better solution out of the box.
Last version was 10 years ago and this service is one of the core AWS services, so it's definitively not abandoned.
Unless I'm confusing your comment.
Its not obvious to me why supporting a newer wire format would be a high priority for AWS. I think I would rather they work on things like native JSON/semi-structured data support[1] than a new wire format.
[0]: https://aws.amazon.com/redshift/whats-new/ [1]: https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-re...
[1] https://docs.aws.amazon.com/cloudsearch/latest/developerguid...
I used CloudFormation for a few years and ran up against a lot of its limitations. Maximum file size. Maximum resource count. Automatic rollbacks of THE WHOLE STACK on any subsystem failure like unavailable instance. It has no templating built in, so doing ten things with minor differences means copying and pasting 10x or deploying your own templating solution to generate CF. And I did this back in the ROLLBACK_FAILED days, where if you did something that it couldn't automatically undo you were stuck - no way to roll forward or backward, just abandon innplace. The button for "Continue rollback" or whatever that came out 5-10 years ago was huge.
Contrast that with Terraform: all of your points addressed, plus all of mine are non-problems. It lets you do some awesome things that are no doubt inspired by CF, but it takes them to an entirely new level.
There are a few downsides too, but nothing compared to CF. You can build too-complex things much more easily in TF, so you have to be careful not to go overboard. It also bugs me that I can't spec a high-level thing like "compute with 4 cores and 32GB RAM, repeat 10x and put behind a load balancer with DNS name foo" and use it anywhere, I have to say google_compute or azure_load_balancer or aws_dns. That was the biggest disappointment coming out of CF and hearing how awesome this thing was, and then realizing it still left me vendor locked.
I totally can see where this is coming from :) CFN is best used as separate templates/stacks for parts of the solution, not the whole solution rolled in a single template. Reusable is the key word here, and Parameters. Let me try a city example. Have separate templates for a school, fire departnemt and house block. Build all Detroit schools using the same school.yml template, just supply different parameters for each. Don't copy-paste code from school.yml into detroit.yml. Actually, there should be no detroit.yml, leave city level to CI/CD job.
> I have to say google_compute or azure_load_balancer or aws_dns
Multicloud? It rarely makes sense. All you get is triple the infra code, triple monitoring tools, triple devops competence requirements. Properly designed solution with HA and AZ/regional redundancy is sufficient on a single cloud platform.
It tends to be badly implemented, but it makes a ton of sense as a strategy.
It would be limited to least-common denominator just because of the vagueness of objects that it would support, but the example I suggested would be incredibly useful.
Thats exactly why it is not useful. You design your solution using all features of the LB, solution using only basic ones will be meh.
Maybe CF has gotten better, but I'm not sure since I totally jumped ship for GCP.
Talk about damning with faint praise.
> It rolls back,
Sometimes.
> no leftovers on delete,
Well, on successful delete, maybe. DELETE_FAILED with partial stack deletion is a thing (and a thing CF could fairly trivially avoid in some common cases by simply querying resources for deletion protection.)
> full control of resource properties
Except the AWS resources it doesn't support, and properties of supported resources it doesn't support, because CF always lags the underlying services and their APIs.
That is until I realized that nothing worked. You had to use a version that was 5 versions behind the current version which is the opposite of what the documentation said which explicitly said not to use that version. Even then not everything worked out of the box.
https://docs.aws.amazon.com/streams/latest/dev/shared-throug...
I would classify this as a critical issue, and this is part of the reason why we stopped using Kinesis entirely.
However, those instances get used in more of a "pet" than "cattle" context.
The difference being that you have to launch & maintain the server yourself (and pay for its runtime).
From my understanding each service team within AWS is run pretty much as it's own little startup which sometimes make features across services, i.e. tagging, be inconsistent. It also leads to why the UI seems to be so fractured.
This probably abstract away the .aws profile? I can't see much reason to use it since I use aws cli just fine in the however many terminal tabs I have.
- You have to install CLI components piecemeal on GCP, and sometimes need to opt into beta features. With the Cloud Shell, it's all preinstalled for you.
- You have to "log into" the CLI if you're running it locally, which can be a minor annoyance. (I know the AWS CLI doesn't have this issue, as it doesn't use OAuth for authentication with the console.)
- All of the data transfers in Cloud Shell are happening between machines in Google's data center, so you get gigabit speed file transfers in the Cloud Shell. For example, this is super useful when you need to download a large bucket to a working directory to make edits to multiple files, or if you need to run scripts that pull and push to/from Cloud Storage.
I think a Cloud Shell for AWS is a net positive! It can make some workloads easier and reduce the amount of configuration you need to do.
https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-s3... + https://github.com/gaul/are-we-consistent-yet/blob/master/RE...
https://chrome.google.com/webstore/detail/console-recorder-f...
If AWS lacks anything, it is a "why the hell this API call failing exactly" feature. It is horrendous to debug a resource that is using other resources and you do not have any means to get what _exactly_ is missing. Usually you get an error like "s3 throw a 403, bye" message. The closest to a solution is CloudtTrail with giant amount of JSON entries to go through or try to load it to Athena or other database, and because you do not know what exactly you are looking for it is very hard. I usually just ask the support to debug it for me because they have internal tooling that can do that. Most of our support tickets fall into this category.
With increased distribution of systems, increased use of cloud proprietary infra software that you can't run locally and now these custom SOCs. Companies are just going to give up on local dev environments and force everyone to write code in a browser.
I haven't used the remote SSH feature, but I have been playing around with GitHub Codespaces through VSCode. Once you have the extension, you can select a repository and it will spin up a virtual workspace for you on their servers. It actually works surprisingly well- tasks like installing node packages is faster than on my local computer and it automatically handles things like setting up a proxy for local web environments.
It’s vscode in your web browser. Run it locally in your dev environment and setup your web server to proxy vscode.your domain.tld to it and boom, you have vscode running in your dev environment that anyone with a link can access.
The fastest way to set it up with tls+auth is probably something like https://hub.docker.com/r/linuxserver/code-server + caddy configured with basicauth.
Despite that I still tend to ssh to machines and use a local editor, because I typically don't want to just edit files, I also want to run some commands in-between editing files.
I use it for work sometimes, that + vterm and ssh makes for a fairly pleasant remote editing experience in emacs. LSP even works over tramp (kinda)
https://aws.amazon.com/cloud9/
It was more than half decent last time I used it.
The only downside is that the difference between extensions installed on the remote and locally is a little confusing, but the extension ecosystem in VS Code has satisfied all of my use cases. I've also run into some occasional ssh hangups on weak connections, but I haven't experienced that isuse in a few months.
For years I was a linux guy, and now I see no reason to go back, because I can just remotely access a linux environment from whatever system I'm already on. Less time spent swapping between operating systems to work on projects, and a consistent environment, are both huge features.
This way I don't have to spend so much money on expensive desktop replacement laptops.
I'm in a similar boat as the Macbook Air, but I was going to go for a Surface Book because I want the detachable tablet features
(and I know, GCP and Azure had this for quite some time now)
But it’s usually easier to trash a cloud IDE and create a fresh instance than it is to unknot a bad Python configuration on a local machine. (Although you can always use Docker or Vagrant)
I have worked in places which say, "no code on laptops! You get a remote machine, all code must live here"
Same security advantages, but you get way more customize-ability -- choose a terminal app, font, fullscreen or many windows, and so on.
That project ran in an enterprise environment, customer facing for 10 years with almost no maintenance.
10 years is a long time in security. Presumably that's what the maintenance was for?
For the individual: similar concerns of hardware failures losing work go away a bit (there are still ways to lose everything, but less of them), easy moving between environments (desktop, laptop, phone), ...
Though it depends how much is pushed to the "cloud". You may still need some meat on the local resource bones if not pushing any CPU crunching into the sky too.
Essentially we are reinventing the thin client from the 90s/00s, which in turn reinvented many mainframe concepts, not that either ever completely went away, with an eye on much the same benefits.
Almost every organization I've worked with has the policy of "if you get malware, we wipe your whole machine and reinstall the gold image" which is quite disruptive because you then have to reconfigure your settings and reinstall all your software packages and regenerate SSH keys etc. It can be a whole day of downtime and then a week of slowly getting back up and running full speed.
But if your hardware and local software are irrelevant, you can just swap your dumb terminal for another dumb terminal without skipping a beat. And with things like Chromebooks or iPads (actual real dumb terminals) the likelihood of getting to a "wipe it and start over" goes down a lot compared to machines running a full-fledged OS with a privileged user account.
If you drop your Chromebook in a lake, you could run to Best Buy and get a new one for $300 and you've only lost an hour or so, and if all your data is stored in OneDrive and your IDE is Codespaces you haven't lost anything of real value.
There are a lot of situations where people tolerate less sane practices because they are convenient, but this isn't a good strategy.
This feels like another solution a lot like Stadia. There have been countless other attempts at the same idea, but the problem is always the same. Latency and user interaction between the local and remote hosts always end up being overwhelming constraints.
None of that is something which can't be solved, of course, but this is a nice way to avoid having to deal with O&M yourself, which is the point to a lot of cloud services.
A local development environment should still be possible in many cases. One shouldn't need to call AWS services.
Can you help me understand what, precisely, this would do in your work that's so dreadful? Perhaps there's something I just don't know.
It's more painful.
I have little sympathy for a generation raised on such notions who see it all as theft. No such legal contract exists.
It makes me wonder how the next generations would feel about a stable environment being taken.
But we won’t be around and have our social orders so ...
Yes, provided you whitelist the IP range for Amazon's Instance Connect service. (They don't call it Cloud Shell.) From [0]:
> We recommend that your instance allows inbound SSH traffic from the recommended IP block published for the service.
You have to trawl the giant JSON document that it links to, to find the relevant IP range to permit, where the region matches yours and where you see "service": "EC2_INSTANCE_CONNECT". Then, whitelist the specified IP range (obviously for incoming traffic on TCP port 22).
[0] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-inst...
https://carriagereturn.nl/aws/ec2/ssh/connect/ssm/2019/07/26...
https://ystatit.medium.com/different-between-ec2-instance-co...
From EC2 Instance Connect docs: "[...] it generates a one-time-use SSH public key, pushes the key to the instance where it remains for 60 seconds, and connects the user to the instance. You can use basic SSH/SFTP commands with the Instance Connect CLI."
Disclaimer: I'm a SA at AWS.
I have zero problem doing this with the normal remote instance I use for this sort of thing in the past, but for whatever reason, walking junior engineers through this process is always one of the most painful things I deal with. Having a GUI way of doing this will make walking them through it easy.
That said, this environment doesn't deal with flaky connections well. A few toggles of my wifi, and now I have multiple bash orphans on my ECS container. I shouldn't be too surprised, looks like they've repurposed the SSH client from Cloud 9. It'd be nice if they brought in something like a Mosh client.
Which unfortunately means I can only access this from a browser window and can't start up a session from my own terminal. Sure would be nice to be able to launch a secure, remote CLI without all the limitations of a web client.
I would say no.
Alternatively, one can launch a custom AMI in EC2 to do whatever they want.
Multiple solutions already exist for the given problem, therefore yet another AWS service seems unnecessary.
Think "I'll run this arbitrary script to batch-process input-bucket X to output-bucket Y, enriching the data by calling out to internal service Foo and external service Bar." The kind of thing Google's Cloud Dataflow is for, but one-off and freeform.
—also, for a lot of people, just the fact that things are running in the cloud, means they're running more reliably. If you want to run something that's going to take four days to finish, you don't want to do it on your own workstation. What if the power cuts out in your house? (Just the fact that you can restart/OS-update your local computer and "keep your place" in the remote is nice, too.) You want a remote VM somewhere (preferably with live migration in case of host maintenance) running screen(1) or tmux(1), with your job inside it. Of course, you can just create a regular VM in your VPC, and do it on top of that; but a cloud shell abstracts that away, and "garbage collects" after itself if you leave it idle.
The browser profile is harder to exfiltrate, in part because modern OSes have ways to restrict access to particular processes, but that was also only part of the benefit: the main thing is the duration of the session. Tons of people leave AWS keys sitting around in ~/.aws for ages.
You can setup schemes with STS but not everyone remembers that and with this approach you have a very simple answer: it always uses STS, there's never a file sitting around for someone to accidentally save somewhere they shouldn't, etc.
Nothing here is something you couldn't do on your own — it's just a very easy option with safe defaults.
Or perhaps even entirely useless if you'd normally use it as part of a local build and test process.
https://docs.aws.amazon.com/systems-manager/latest/userguide...
I love to be able to use my terminal client than browser. This is neat because I don't want to maintain another ec2 myself even if its in the free tier.
But I have to confess I opened this article half hoping it would be about Lambda support for bare bash scripts. Horrifying, yes, but at the same time...
But seriously custom runtimes are real bastards to get working.
A shell is not something from the future, no need for fancy graphics.
It also has logical semantic meaning:
staright line = end if the content
"torn" line = content is "cut off"
It's just a picture of the corner of the page.
I ran workshops when I was at AWS, and using the Cloud 9 shell saved us a ton of time getting a room full of people set up with a functioning AWS CLI. Being able to just click a button to pull up a shell and then paste in a command is so much lower friction.
That should probably have been on the launch roadmap.
But before we could get started he had to:
- install the AWS CLI
- stop screen sharing while I walked him through creating an access key/secret key from the web console
- walk him through aws configure
start the screen share back
- install the SAM CLI
- install jq
If he had used this. He could have just run
git clone
aws s3 mb $artifactBucket
sam package....
sam deploy
Started a CloudShell session and ran:
ps aux
cat /proc/1/cgroup
echo cool :)
Also feels like now an EIP IPv4 has been assigned to my IAM user. Pros and Cons seem to equal right now in my head. Mmmmm
In our case, since we do development in a ChromeOS environment, and the browser is relatively isolated from the Linux VM, it also likely prevents classic SSH-hijacking.
But CloudShell is yet too narrow of a solution, I'm sure they will improve it over time, but a few problems with todays' release:
1) It only tracks bash commands. What if I write a quick Python one-off script and run it from a file? CloudTrail will never get the content of such script. This is script will get lost at the end of my session. What about Git for storing code?
2) Only works in the browser. The browser has it's good parts, but during incident resolution speed is critical. Getting a prompt without my local shell history, aliases, binaries, and many others, will make it slower to resolve incidents. One might say it's for a good reason, but we can do better.
3) Only works with AWS. This is a big problem as many companies are in the process of migrating to AWS, with services running within their own servers. Companies will use CloudShell to investigate edge cases, most of the time during incidents, engineers need fast access to all resources. Using a different solution for each type of resource won't help.
4) Hard to audit. If you ever tried using CloudTrail, you know what I'm talking about. And again, companies will need different solutions if they don't run only in AWS.
5) No review workflow support. If you only allow platform and SRE to access infrastructure, this is fine. But if you really want to bring ownership of problems to developers (DevOps), they need a way to get this level of access without risking production. This comes in the form of experts reviewing (instead of running) commands and scripts faster that the regular Github Pull Request workflow.
There are more, but I'm still happy with the product. AWS saying that you need one-off solutions no matter how much automation you have will help us move to a future where companies treat one-off scripts as first class citizens.
If you are interested in a solution that solves the problems I pointed out and many more, check out RunOps: https://www.loom.com/share/ea25027e73c94aa395f3e0ab70b71f0e
LOL, or run a remote VSCode session on it :D (I know that's not gonna happen, but would be kinda cool nonetheless)
(Note: I'm sure they would catch this and it would either be a policy violation that gets you shut down or they would just know how to bill you for it)
https://medium.com/google-cloud/how-to-run-visual-studio-cod...
And Hashicorp have one on using Terraform from GCP Cloud Shell:
https://www.hashicorp.com/blog/kickstart-terraform-on-gcp-wi...
Not quite logged in yet - had a "AWS CloudShell is temporarily unavailable because it's being activated" screen for a while now. Fingers crossed!
I hate mechanical voices.
Nonetheless, excited to see it -- it's something that I've complained about with AWS since using Google's CloudShell. It also continues us down the path to easy Ops-type work on an iPad (even though you can already have an EC2 instance and use Prompt to access it, being able to have a shell without needing to provision and EC2 instance is chefs kiss).
--
[1] https://aws.amazon.com/cloud9