twosdai

There are lies, there are damned lies, and then there are statistics.

I guess my question is that is it likely a person would get the outsized health benefits because they commit to the minor health change which then causes them to commit to more later.

There is a learning curve to do everything and i do occasionally switch to windows for some games. I am a senior engineer as well.

I think the biggest thing holding linux desktops back is not technical really it's a human element, like the services to talk on a forum, having things setup ahead of time for the user etc...

Windows has a learning curve which people went through if you want to switch to Linux you need to learn. Just as if you start on Linux and wanted to switch to windows you'd have a learning curve.

My examples would be communities situated near or around State/National parks. There is very much a just walk around in the integrated nature sort of feel there.

Specific example towns:

Keene Valley, NY Joshua Tree, California

To a lesser extent in the bay area east, and North along where the towns butt up against parks, and accessible areas.

Additionally I've found that towns/cities which have a major university that has open access for the public typically has the ability to wander and "lose yourself" feeling Berkley in CA has this feeling for me.

Big fan of pop os personally.

I could see people who are against major corporations being for paying inde groups a premium for less than perfect service, just as people who are against Microsoft and Apple will tolerate less than ideal user conditions in Linux.

Not to rain on your back of the napkin math. Although I think your sentiment is right, the cooling systems for computers in space would need to be bigger since you can use convection to move the heat away.

Not perfect but it's pretty damn good.

While it is a concern if we produce a lot of extra heat, I think the main issue is being able to expect the heat to leave into space. Which is why most of climate change science focuses on greenhouse gases and not heat produced by machines.

If you want to start a business it can definitely have an open-source element to it, but that's just it. The open-source part is just one piece to a business and you better be aware of all the other pieces that need to be done in order to make money which aren't always engineering/programming related. EG: marketing, sales, consulting, fundraising, accounting, etc...

A common approach that people do now is create the open source offering then run it as a service where they can charge a premium for the support and leg work for hosting it. See: mongodb or elasticsearch or influxdb.

I haven't found many pure open source solutions where all they do is make the software better all day and ignore all the elements of business growth needed. Maybe the Linux foundation is a good counter example but I don't think there is a clear path to be paid for just contributing solely to open-source while neglecting other business elements.

Source: I helped with a research project using Ge. Here's the paper if anyone is curious https://ieeexplore.ieee.org/abstract/document/8653561

So for aggregation we definitely do some of that work already, we enable clients to aggregate their raw data with a few different methods, like via running a total, average or count, over any arbitrary time frame.

For us doing the actual calculation of the dimensions at a specific time, we haven't thought about it, but we're always interested in building more so we might be able to prototype something.

Your specific example though is sounds like another way of metrics to be pushed into our platform, this really wouldn't be that big of a problem for us to implement, and we even floated the idea around of us exposing a stream that clients could push data to which we would process.

Given that the logs would be in an open standard format, we could definitely do that and it sounds like a good idea. :) Thanks for the suggestion.

That being said we still want to charge something to indicate a level of seriousness to the use.

For some context on how we arrived to the figure the ~1% figure is in the low end ballpark for how much we saw companies spend internally on their billing systems built in house, and its slightly higher but still in the same range as other billing providers.

Thanks for that callout, I'll update everything ASAP.

Yeah to add more color to this, we just observed the failure of a process and system we weren't a part of. Basically there were manual elements in the billing and aggregation of usage data on a SaaS platform and those were forgotten about over the course of many months which lead to the company under billing.

I walk through it all during the first 3 minutes of the youtube video if you wanted to follow along.

https://www.youtube.com/watch?v=T6J1Yh8GhdU

For creating the IAM role, we have some information under https://docs.paigo.tech/

If you get stuck or have issues feel free to ping me!

> Most SaaS providers would rather push data than have it pulled, is what I'd imagine. Are you hearing otherwise from folks you've been speaking with? For instance, in serverless environments (which is the poison of choice for me, at least), pull is much harder to accomplish, even where possible.

We totally offer Push based as well, all of our workers are just using the same API endpoint to push the data we collect to, its just not the strong highlight since other providers have push offered.

We went down the pull path since during our discovery process about 6 months ago, we were chatting with some DB and Infra companies who just built out and integrated with a billing provider. All of them mentioned being annoyed with the amount of engineering commitment it took for them to measure, persist, and then transmit the usage data to the provider for them to handle the rest. So we wanted to offer a pull based solution to help with this need.

You're totally right that its architecture dependent, and we don't want to cause a huge load (and cost) on a serverless platform. So for some dimensions, push is definitely an option.

> Apparently, a BigCloud, in perhaps a case of NIH, ended up creating a highly-parallel event-queue as a direct result of the scale it was dealing with: https://archive.is/IUKvT Curious to hear how you deal with the barrage of multi-dimensional events?

So to dive into more technical detail, we have an event queue where our workers drop data off to and then it gets persisted into our ledger, by workers reading from the queue. We have the queue hosted by a major cloud platform and its offered as a managed service, similar to kinesis.

For the different dimensions, we have a standard dataformat they need to be in before we can persist them, this transformation typically occurs on the client side, though in some cases we can transform the data from an open standard format (Prometheus https://prometheus.io/docs/concepts/data_model/) to our backend format.

At a criminally high level, this data format consists of a measurement, a value, a field, and a set of metadata tags. Our ledger is built on a schema-less Timeseries DB, so it doesn't matter if the same measurement has a different set of metadata from another. This gives us a boat ton of flexibility when it comes to how we want to query data.

For the different types of dimensions and their different data types, it becomes an issue when wanting to aggregate on them. For instance, a you may want the Total of one dimension, however Average and Count wouldn't make any sense.

To get by this, clients need to tell us what Aggregation method to use per dimension.

This really isn't present in the demo, since its a fairly simplistic version of the whole app, but its a requirement we have implemented into the API.

> 2 cents: Fly.io Machines is a tremendous platform

Thanks for the hot tip, looks awesome! I'll definitely check this out and how we can sneak it into our product.

For the security concerns, at a policy level we're currently going through getting SOC II compliance. In the future we will be progressing for fedramp complaince as well, but we haven't started on that.

  Regarding read access risks, the client has complete control in their account for giving us exactly what access they want/are able to via IAM. Additionally, right now our application is multi-tennant behind the scenes, however early on Matt and I saw that we will need to offer single tennant solution for increased data privacy, so we've designed the system with that in mind and its not a major lift for us to provide a single tenant and isolated environment within any region which data needs to reside in. It just hasn't come up as a concern for us yet. 

For GDPR, we don't store or process PII. Which sounds kind of insane saying it out loud but its true. We integrate with end payment providers like Stripe and AWS marketplace, where all we're reporting with is a UUID which is associated in their platform to the end customer's billing info, which we never need to see or touch.

Now it is possible that someone could manually enter client PII into the platform in which case we would need to deal with that, but it has yet to come up. If it did, we have API endpoints which can delete all data pertaining to specific clients by request.

I suspect in the future that this will change (we may start persisting PII), and we will need to have a more cohesive strategy for dealing with right to be forgotten, but in the near term it hasn't come up.

https://paigo.retool.com/embedded/public/99bd5e9a-af3c-4e9c-...

Here is the expanded link.

And here is a fun HN specific one linking to the same thing: https://hn.paigo.tech/

For the resiliency part, our workers / agents we're using a write-ahead log (WAL) https://en.wikipedia.org/wiki/Write-ahead_logging to track what data has been collected and sent, when agents or workers fail and need to be restarted they read from the log to make sure that the data was sent appropriately. For a good starting point I recommend looking at the Prometheus agent for agent design and construction they have implemented a lot of the resiliency into their agents, and if you're familiar with GO forking their work might be a good starting point.

Additionally for all the off the shelf products we use to process and transmit of the data we have "at least once" delivery guaranteed, which realistically under the hood is using a WAL.

For scalability, our agents are deployed into serverless and managed components by some SaaS providers, Confluent and AWS right now. These components have some auto scaling written in, like with AWS lambda.

But for some our measurement components we're utilizing a kubernetes cluster with our custom workers, right now we just statically provision them, however we have the ability to auto scale infra based on node resource utilization, (using AWS ASG's) and for our pods, we currently scale horizontally based on resource availability, like CPU consumption and memory consumption using k8s horizontal pod autoscaler: https://kubernetes.io/docs/tasks/run-application/horizontal-...

It was fairly half-baked but the general idea was to get a group of people together in the same physical location and work together like a commune to provide for each others basic needs, (food, water, shelter, clothing etc...) this is really no different than most other communes in the world.

The difference being that these would be considered chores. The main form of work which would need to be done that would actually bring in resources would be some form of technical software work, where everyone in the group is expected to create/contribute to software projects. These projects and work could and should be sold for money, however the group itself is not a profit driven organization, any proceeds just go back into increasing the total wealth of the group.

I realized a bit later that this is really an Non-for profit organization with a few extra steps and caveats.

But the basic idea being:

- Group provides basic needs for you to code

- Resources from the coding work done is reinvested into the group.

This is a much more advanced OSINT tool which can provide you better information if you really want to search for tweets within a specific area, or around a specific timeframe.

Here is an example command which would give you any tweet containing the word trump within a 50Km radius of DC.

twint -g="38.8935755,-77.0846155,50km" -s "Trump"

Live website: www.danielwasserlaufquicklinks.com

github if you want to manage your own: https://github.com/twosdai/contentGrabber

I don't come to hackernews that often so just open an issue on git if you have a question.