Hacker News Reader

I've had the same frustration with S3, and the reasons you went over are what drove me to create this. It seems like they made S3 (10 years ago now?) and just forgot about it. There is no way to sort/filter or even list all the objects in a bucket without writing a recursive algorithm using one of their SDKs.

wellsjohnstonon Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

This does not have to do with AWS Lambda, I'm thinking about renaming it to "functional-s3", or something similar.

To answer your question, there isn't really a workaround for this yet, although indexing should be much quicker than "days". All the keys are listed recursively before running the lambda expression locally. If you have a huge number of files, this can take several minutes, maybe hours depending on the scope.

A workaround I've been considering is using a generator function to list the keys; that way, the lambda expression can start immediately, generating keys as it needs them.

wellsjohnstonon Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

Okay this seems like a good alternative. I just renamed the repo. Renaming it on npm...is a bit cumbersome :|

wellsjohnstonon Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

I did not know about this...looks like Amazon's version of BigQuery. Fantastic!

wellsjohnstonon Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

Sure. We have application logs that come in to s3 and are stored by date prefix. I have CRON jobs that run node scripts that do various counts/statistics.

wellsjohnstonon Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

Ah yeah, just realizing this...what would you recommend?

wellsjohnstonon Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

First, a list of keys is generated based on the set context (and modifier functions). "context" returns a Request object, allowing you to call a lambda function (each, forEach, map, reduce, filter). Each lambda function returns a Promise, allowing you to chain them together. They will operate over the same context, in sequence.

Edit: This is not related to aws lambda...sorry for the confusion

wellsjohnstonon Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter

Thank you :)

Writing this was a necessity for me, being a 1-person data team coming from a Node.js background.

wellsjohnstonon Why I'm Making Python 2.8

This is just wrong. For example, the "best" recording microphone (to many artists), the U67, can no longer be made because the parts aren't available anymore. Yet it is the most popular mic, and 100% not obsolete.

Similarly, Python 2 might not be made anymore, but it is used everywhere, and people are making new things with it. So...it's also not obsolete.

wellsjohnstonon Jid – Drill down JSON data incrementally

I made a gui for this https://github.com/wellsjo/JSON-Splora

wellsjohnstonon Jq – Work with JSON on the command line

Thanks :) https://github.com/wellsjo/JSON-Splora/issues/38

wellsjohnstonon Jq – Work with JSON on the command line

I made a gui for it, if you're interested https://github.com/wellsjo/JSON-Splora

wellsjohnstonon Show HN: JSON-Splora, a gui for editing, visualizing, and manipulating JSON data

Thank you for this! Accepted & merged!

wellsjohnstonon Show HN: JSON-Splora, a gui for editing, visualizing, and manipulating JSON data

Yes, in addition to JavaScript. The filter is first run through a JavaScript VM, then jq, allowing you to manipulate the object with either.

wellsjohnstonon Amazon QuickSight Now Generally Available

Preach- I'm going through the same process at my company. First I built a custom dashboard, but I haven't been able to maintain it with all the ad-hoc visualization requests I get. So I've been looking around for something else. It basically boils down to sketchy marketing sites (Chartio, Periscope, Mode, etc.) that end up being very expensive.

I'm very hopeful for quicksight, but unfortunately it feels very clunky and hard to use at the moment. Hopefully they improve the clunky UI and give it a SQL inerface.

wellsjohnstonon How AI based programming could work

The whole point of programming is to get exact results by solving all edge cases. Machine learning/Neural nets are only good at guessing results, and cannot solve all edge cases without specific direction.