jackgill

This is some of the most insightful high-level thinking about AWS services that I've seen. No taxonomy is perfect, but this is invaluable for gaining a panoramic perspective on the vast sprawl of AWS.

Thanks for sharing, I've heard many good things about Snowflake. In the past I've seen them as more of a Redshift competitor (data warehouse, as opposed to a data lake) but if they can simplify data ingest then I am definitely interested.

I'm interested in hearing opinions about this principle in the context of data engineering:

> This also means leaning heavily into all the service offerings and orchestration tooling that is afforded to you by your platform.

I've built a data lake and several ETL pipelines using AWS native services (Kinesis, Lambda, Athena). It works but it's a bit...fiddly. I spend a lot of time configuring these services and handling various failure modes. I've been wondering if I should be looking at third party vendors like Fivetran or Matillion for ETL.

Does anyone who's worked with AWS data engineering services have thoughts on the trade-off between AWS native services and third party vendors in this area?

Understanding the AWS bill is far harder than it should be. That being said, there are some resources that weren't mentioned in this blog post. The ultimate source of truth is the AWS Cost & Usage Report [1] which can be delivered in Parquet and queried with SQL via Athena.

Although the Cost & Usage Report alone can solve many billing mysteries, in some cases it's also necessary to go to CloudTrail logs to determine exactly which user or application incurred charges.

[1] https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2...

The writing in this post is superb. One of my favorite lines:

"Windows is the Superbowl Halftime Show of operating systems. Given what everyone got paid, and how many people were involved, you’d think it would be a lot more memorable."

As someone who spends a lot of time with Windows...yeah. But the post is really an amazing blend of technology and sentiment, kind of reminiscent of Neal Stephenson at his best. Makes me want to dust off my PDP-11 emulator and take another crack at Unix V6.

Also worth mentioning is the GnuWin32 project (http://gnuwin32.sourceforge.net/). It provides native Windows builds of many GNU utilities, including a few handy ones missing from the Git distribution like file and wget.

If you're serious about using Vim on Windows, check out gvim (http://www.vim.org/download.php#pc) which adds some nice Windows integration.

I would also echo the author's point about avoiding Cygwin. It gives you the worst of both worlds: you can't access Windows facilities, and many Linux programs don't quite work right. By far a better solution is to use a native Windows shell (Powershell) together with native Windows builds of various GNU utilities (from a Git install, GnuWin32, MinGW, etc). Powershell gives you fluent access to a wide variety of Windows-specific API's, and you don't have to sacrifice the convenience of a Linux command line environment.

Your zip example actually demonstrates why Powershell is significantly more powerful than Bash (or any other Unix shell). I presume that if asked "How do you zip a file in Bash?" you would reply with something like this:

  zip archive.zip folder

The equivalent Powershell code would look like this:

  7zip a archive.zip folder

Pretty much exactly the same, since in both cases the shell isn't doing anything other than invoking a standalone executable. So this doesn't really tell us anything at all about the relative merits of Bash and Powershell. A better comparison would be this: how do you zip a folder in Python? Again via Stackoverflow (http://stackoverflow.com/questions/1855095/how-to-create-a-z...):

  import os
  import zipfile

  def zipdir(path, zip):
      for root, dirs, files in os.walk(path):
          for file in files:
              zip.write(os.path.join(root, file))
  
  if __name__ == '__main__':
      zip = zipfile.ZipFile('Python.zip', 'w')
      zipdir('tmp/', zip)
      zip.close()

Compare that to your Powershell example, and I think you'll agree that the .NET API is nicer to use (heh, don't say that very often). But to return to the subject, how do you zip a folder in Bash? YOU CAN'T. Bash can't zip folders. It can't call libraries that can zip folders. It can only invoke programs that can zip folders. And any shell, even the wretched cmd.exe, can do that.

I have plenty of problems with Powershell but it is by far the best effort at a shell that the world has yet seen. I come from a Linux background and when asked what I like about working on Windows, I reply with "well, it's got a nice shell." Usually gets me a weird stare, but if we're really comparing shells to shells (and not shells to a wide variety of standalone utilities) then Powershell blows everything else out of the water.

I'll add another recommendation for MediaWiki. It is a good low-friction way to establish documentation for internal services and libraries. The markup syntax is rather idiosyncratic, but powerful enough for most of our needs. "Does it have a wiki page" is now a standard question for any new service or library in our shop.

I quite enjoyed this, it's a nice piece of "inside baseball" for high energy theoretical physics. For those who don't have time to read 20 pages of jargon from another specialty, here's an excerpt from the end of the paper that I think communicates the key point:

"I think that string theory is a wonderful theory. I have a tremendous admiration for the people that have been able to build it. Still, a theory can be awesome, and physically wrong. The history of science is full of beautiful ideas that turned out to be wrong. The awe for the math should not blind us. In spite of the tremendous mental power of the people working in it, in spite of the string revolutions and the excitement and the hype, years go by and the theory isn’t delivering physics. All the key problems remain wide open. The connection with reality becomes more and more remote. All physical predictions derived from the theory have been contradicted by the experiments. I don’t think that the old claim that string theory is such a successful quantum theory of gravity holds anymore. Today, if too many theoreticians do strings, there is the very concrete risk that all this tremendous mental power, the intelligence of a generation, is wasted following a beautiful but empty fantasy."

I spent a few years during my undergrad working in a high energy experiment lab, and this was a frequently debated subject. If the LHC does find supersymmetry, it will be interesting to see the impact on the current theoretical landscape.

That sounds like the "Second System Effect", described by Fred Brooks in "The Mythical Man Month":

http://en.wikipedia.org/wiki/Second-system_effect