Pirate Weather
pirateweather.net
pirateweather.net
Actually we put the current post in the second-chance pool (https://news.ycombinator.com/pool, explained at https://news.ycombinator.com/item?id=26998308) — we try not to do that for dupes but sometimes we do anyhow. Sorry—it was just a good HN post!
> "This project started from two points: as part of my PhD, I had to become very familiar with working with NOAA forecast results. Separately, an old tablet set up as a "Magic Mirror,” and was using a weather module that relied on the Dark Sky API, as well as my Home Assistant setup. So when I heard that it was shutting down, I thought, "I wonder if I could do this.” Plus, I love learning new things, and I had been looking for a project to learn Python on, so this seemed like the perfect opportunity! Spoiler alert, but it was way more difficult than I thought, but learned a lot throughout the process, and I think the end result turned out really well!"
No need to worry about them selling out.
I'd have opted for a setup where basically every single "service" in that architecture diagram is one function in a monolith program. Easier to maintain, more performant, easier to deploy and basically better in every single way except you can't individually scale components. But at the scale that the project currently is, I don't think that's needed for a long time, nor is it hard to scale one machine if needed.
But again, if it's just a educational exercise in "web scale architecture", then it of course is OK. But I wouldn't want other students to see something like that and think that it's necessarily to create such a monster for a relatively simple "moving data around and wrangle it a bit" program.
Are you going to be maintaining it? Are you going to even be using it? Why does it matter to you what someone else did in their spare time for their own amusement? Or even to make a commercial product out of it? Either way, it’s not your architecture to backseat architect, let people do what they want.
I'm personally dealing with infrastructure at loads of different places, and the amount of engineers who over-complicate their infrastructure pains me, they all go for a distributed architecture before the products they are building even have any users, because they've read bunch of blog posts about building it that way.
It sucks to spend so much time and effort on maintaining and improving something when it's not really needed in the first place. Like using a Ferrari to deliver food in a traffic-heavy city, instead of just having a moped.
So basically, I want to save myself and others time.
If someone takes one look at one project and decides to copy it’s architecture word for word, then it’s on them. Not on the person who built an open source project, for whatever reason, and put it out there for the world.
It’s not your architecture to architect. If you want to build your own service to show best practices, then go for it. You weren’t there to make decisions, you don’t know of the reasoning behind them, and it seems ludicrous to backseat architect with context, let alone without any amount of it.
See it as wanting to advance my own trade by providing opinion on the current "everything should be a serverless function" trends in the hope of hopefully be able to move the trade into a different direction.
If I have no affiliation, or plan to affiliate with a project, why would I give a shit about what they do? It’s their project to figure out.
Did they ask you for an architecture review? No. Then don’t backseat architect. It’s as simple as that. Especially when your critique is “you’re doing it all wrong and you should feel bad if it’s not an educational exercise”.
We critique whatever gets shared on HN regardless of our affiliation with it.
In fact, why was this shared here, if not to ask for discussion and opinions?
I'm assuming we two have at least one thing in common, we're here on HN to discuss things we come across, both submission articles and comments, since we're having a conversation right now. That's how this works.
So I came across something that I have feeling and opinion about, and that led me to share my opinion about it, as I care about my trade. You're welcome to ignore things even if you have opinions about it, but to request others to do the same in a forum that is meant for discussion, feels a bit unproductive.
[EDIT]: Removed comment about user.
When disagreeing, please reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3."
Said differently, it's worrying that people decide to overengineer architectures like that before there's any real need. It's a sysops version of the early optimization problem.
Ten years ago, the student would have probably built something much simpler with a couple cron jobs on a simple EC2 instance, maybe celery/rabbitmq as a message queue if he was feeling fancy and called it a day
Explicitly,
> A bunch of people take information they come across on the internet and think that's the only way of doing things, so if someone comes across the post, looks at the architecture they might think building it with so many components is a required thing, not just optional for really big projects.
> So basically, I want to save myself and others time.
This seems like not only ridiculous critique, but also just plainly not the author’s problem what others choose to do with his open source project.
> Said differently, it's worrying that people decide to overengineer architectures like that before there's any real need. It's a sysops version of the early optimization problem
And why exactly is it yours or other commenter’s problem what people choose to do with their own time? How does it in any way shape or form impact you what this other person chose to do with their project that you will never maintain, contribute to, or likely even use?
You may or may not disagree, but it's arguably a worthwhile comment. Possibly more so than your comment on the merits of someone else's comment... which just derails the conversation into this meta-discussion
> And why exactly is it yours or other commenter’s problem what people choose to do with their own time? How does it in any way shape or form impact you what this other person chose to do with their project that you will never maintain, contribute to, or likely even use?
Critiquing technology referenced in submitted posts is what we do for fun around here...
I think the critique was not constructive, was dismissive about an entire project, backseat architected, and did all that with absolutely no context on the requirements, limitations, and decisions made.
In short, it’s a shit critique and I called it as such.
He's allowed an opinion especially one that's valid.
It was constructive and explained how he would want to see it structured.
He did it the right way.
Where as your comment is unconstructive. Doesn't foster discussion.
If the only point you're trying to make is to ask someone to stay quiet why not leave it to the mods or even silently downvote and move on?
There was no constructive criticism, no “you should do this differently” it was “this is worryingly large and you should feel bad if this isn’t an educational project”.
If this is “the right way” then I don’t want to be right, because it’s a shit way that lacks empathy or tact.
No he didn't. He had a caveat.
> no “you should do this differently”
That's not the case..he clearly says this is how I would do this differently.
None of this matters. Arguing about arguing is futile.
> This project started from two points: as part of my PhD, I had to become very familiar with working with NOAA forecast results (https://orcid.org/0000-0003-4725-3251). Separately, an old tablet set up as a "Magic Mirror,” and was using a weather module that relied on the Dark Sky API, as well as my Home Assistant setup. So when I heard that it was shutting down, I thought, "I wonder if I could do this.” Plus, I love learning new things (http://alexanderrey.ca/), and I had been looking for a project to learn Python on, so this seemed like the perfect opportunity! Spoiler alert, but it was way more difficult than I thought, but learned a lot throughout the process, and I think the end result turned out really well!
Seems perfectly fine to assume that this might be purely an exercise in order to learn as much as possible, rather than to build something that is trying to be as cost-effective and performant as possible.
As the adage goes: Suppose you have a problem, and you say I know, I'll use microservices! Now you have N problems.
I would venture as far as saying nearly every personal project built to scale is over engineered.
It sounds like you'd be well positioned to offer some feedback, but I'd recommend against that without your practicing some empathy for the stage he's in with his education.
Also, much of the complexity is from pulling from data sources. Lots of apps are just the web site and database, but weather app needs the complexity. I bet EventBridge has some important purpose.
[EDIT]: He's a Phd for chrissakes!
And yes, if the whole point is to utilize AWS resources, then of course a architecture like that is fine, as it's a learning exercise (to learn how to utilize resources). My comment includes that as well.
As far as I can tell, the author is studying at a university, that makes them a student in my mind. Maybe they do other things than just studying, I didn't look into it very far, hence the "maybes" and other caveats in my initial comment.
AWS is more reliable, and from this architecture, probably doesn't have to do anything. If the VPS dies, then the site is down until can build new VM. Or have to run two of them at twice the cost. More since he has to setup database or shared storage.
That might be overkill for this site, but for some upfront work, he gets something that needs less maintenance. If going to learn, then should do it the right way. He may be scaling for the future, but it is tradeoff to design now instead of later.
It's a pretty cool system, you have your roster of players and setup skates, and it automatically emails everyone x number of days before the skate. You can see who signs up, pay online, etc and if a goalie who's signed up cancels, the 'Skate Operator' is emailed immediately to take care of that (it's a big deal to not have two goalies). It also has an automatic waitlist, and if someone cancels before the skate, and you're the first person on the waitlist, it moves you to the end of the list and emails you to let them know you're in for the skate. It also sends out a reminder X hours before the skate, day of, etc.
After I started using it for my own skates, a few rink managers contacted me to discuss using it for their rink. I met with 2 different groups / municipalities (because most rinks are owned by the city they're in), and it never went any farther than those meetings. They didn't want to use a system by some guy that couldn't maintain it, wouldn't keep it going for more than a year or 3.
13 years later and I'm writing a new version and have run something like 11k skates in that time. There is one small, privately owned rink near my house, has used it the entire time without issue. I've got a few other friends/individuals that use it, too.
It is possible for a single person to create and maintain something purely because they need it.
To me, it is kind of expected. “Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away,” sure, but perfection – or even distantly approaching it – is the product of expertise and practice.
This is a kid that saw a need and went out and did it. His PHD is “currently completing my PhD in Water Resources Engineering”. So, this work is pretty amazing as a hobby, a side line to his PhD.
Isn’t that what Open Source is all about, jumping in and filling a need, not “waiting until I’m a grizzled 30 year veteran of the corporate IT wars, then deciding to do something with a perfect architecture”.
For the record I agree that his accomplishment with this project is impressive.
("Architecture astronauts" don't argue for simpler architectures, but instead want to build general, solves-everything kind of overengineered monsters. I don't see this in the critiques)
I think the critique is pretty mild and reasonable, regardless of whether you agree with it or not, and it's definitely not "hate".
and
They also have many (free) api's for whole world weather. If your life depends on it, they are my recommendation.
Maybe, but the Norwegian government does provide worldwide data via API: https://developer.yr.no/
https://api.met.no/weatherapi/locationforecast/2.0/compact?l...
https://github.com/alexander0042/pirateweather/raw/main/docs...
That looks like a pretty complex architecture.
I wonder if instead of financial support, the author could use help figuring out a more cost-effective solution?
The author discusses using AWS to get free in network transfers since the source material is published there by the guvment. In your analysis how would you offset that incresed cost by moving off network?
It would also be interesting to try this all on a $10/month commodity hosting and then open it up for user facing apps such as a darksky type service.
The architecture is complex from the standpoint of having a lot of icons and arrows, but if it was running on a single server would it be _that_ much less complex? At this level of architectural diagram you probably condense most of these service icons down to one or two icons. But drill down further and you still have storage, cron jobs, an API endpoint with throttling/quota logic, CDN, a shared filesystem, the need to access the S3 buckets with weather data that AWS is providing from its Open Data Registry, plus (ideally) fine-grained permissions to lock everything down. Could you run all of this on a single server in a VPS? Yes! Could you run it on a vendor-agnostic K8s cluster? Yes! Would those options automatically be less expensive? Not necessarily.
Examples: http://3.bp.blogspot.com/-B1LupTaXvSo/TcBNECP0A0I/AAAAAAAAEi...
https://flowingdata.com/wp-content/uploads/2011/03/Weathersp...
https://media.licdn.com/dms/image/C5112AQEuHVOGOd-5fQ/articl...
I am imagining a process that involves programmatic architectural diagrams, graphs, estimates of resources, DAGs, documentation, links to source code, etc. Almost like a sub-project, where the architecture itself becomes a project. The end result of that project could tell you rough estimates of cost, performance, complexity, and let you visualize changes. And of course it would become the "map" by which you discover pieces of the system, the place all new devs start to learn about the system or find documentation.
(I am not saying the author's system needs to change, I'm just speculating about how to organize a collaborative effort, if change was wanted)
The radar map allows you to scroll forward in time, but not the temperature map.
Anyone have suggestions for apps/sites that offer that view?
Update: neither their site nor their app allow scrubbing. Boo.
This has to be, by far, the most detailed “look at our cool app” page I’ve seen that doesn’t actually include a download link, or indeed any description at all of how to find it.
tried to sign in using the password I set, locked the account.
amazing