marklit

The JSON files in the S3 bucket can be downloaded in isolation from any other format and they're pretty small. I did some geo-enrichment of them in a previous blog post where I found the country and waters of Umbra's ship imagery. https://tech.marksblogg.com/yolo-umbra-sar-satellites-ship-d...

The above relies on network calls to OSM but with Capella, I found a way to get the smallest GeoFabrik partition for each image using a single GeoJSON file. The code could probably be modified to work on Umbra's feed with a bit of work. https://tech.marksblogg.com/capella-open-data-free-satellite...

Lastly, Umbra have ~24 locations they image frequently but there is an 'ad hoc' folder with a lot of subject names in English that give away the location and subject matter. This might be easier to look through for interesting imagery.

aws s3 --no-sign-request ls 's3://umbra-open-data-catalog/sar-data/tasks/ad hoc/' | grep -i 'tesla\|nvidia\|saudi'

DuckDB's file format is OLAP-based but supports CRUD operations. I've created DBs with it with 10s of millions of records and gotten amazing query speeds.

Efficient querying using indices over S3 for their format isn't something I've looked into just yet.

The GPKG files https://github.com/marklit/osm_split produces are SQLite files under the hood and come with an R-Tree Spatial index so they'll load into QGIS and other GIS software quickly.

Green Idea | Estonia | REMOTE | Full-time | Windows + DirectX C++ Developer

I'm looking for a C++ developer to build Geospatial / GIS visualisation and editing tools for Windows. These will work in both 2D and 3D. These will act more like desktop applications like AutoCAD and 3D Studio Max rather than full-screen video game applications.

The project is likely to be iterative so if you thrive on agency and being a part of the brain-storming process, you should feel at home.

Knowledge of the following is expected:

* C++17

* Computer Vision + Graphics, photogrammetry

* CUDA

* DirectX

* DuckDB and SQLite3

* GDAL and GEOS

* Packaging Windows Applications for distribution

* Python

* QGIS & ArcGIS Pro

* WinUI 3 and Windows Ribbon

We're based in Estonia but flexible with time zones between here and California.

Contact me via https://www.linkedin.com/in/marklitwintschik/ to apply for the role.

Yeah, you can. The file will be huge though and QGIS won't be performant. Somewhere like Tokyo will be ~400 MB in OSM's PBF format but will contain ~1,300 different types of features.

Seeing these listed in alphabetical order in a file explorer makes it easier to track down only what you need to build your map.

It could be you only end up using 150 MB of GPKG files and QGIS, even on an old laptop, would be performant.

Add them in https://rapideditor.org/rapid

There are usually several satellite image providers for anywhere that isn't a sea or ocean. You can trace over them.

I built osm_split to extract OSM features into named GPKG files. It makes it easy to pick features by name and drop them into QGIS. https://github.com/marklit/osm_split

Done :)

https://github.com/ClickHouse/ClickHouse/issues/45130

I'll take a look into it.

Last year I found ClickHouse did H3 enrichment much faster than PG or BQ so it has become a go-between for larger workloads with simple enrichment requirements.

The dataset in the post is an example anyone can download. I handle 30B+ records at work.

I began taking GIS seriously this year so QGIS, GDAL, Shapely + GEOS, PostGIS & Valhalla.

As for video editing, I'm using DiVinci Resolve for all my videos now.

As soon as you have a few of their IPs, look them up on ipinfo.io/1.2.3.4 and you'll find they probably belong to a handful of hosting firms. You can get each firm's entire IP list on that page and add all of those CIDRs to your block list. Saves you needing to make 30K web requests.

In most countries in the western world, there are 3-4 major ISPs and this is where 99% of your legit traffic comes from. Regular people don't browse the web proxying via hosting centres as Cloudflare will treat them with suspicion on all the websites they protect.

Given all the drawbacks of email and how successful Discord, Slack and WhatsApp have been, I'd be surprised if there is any further major push into email. Why bother when the current market has shown what it prefers?

I'll save you the trouble, ClickHouse will come up on top by a mile and is the only one listed that supports clustering and spill to disk.

Trello board.

Just this morning I finished watching a tutorial on setting up qgis, installing plugins and building a multi-layer map of Mumbai. Very much recommended for anyone looking into GIS https://youtu.be/jgbTosOPU-U

You're right. It's 2.1x faster doing it that way. Updated the post. Cheers for the heads up.

Yes, it would broaden the subject matter that hits the homepage. Also, encourage people to upvote more often. There appears to be a 150:1 click to upvote ratio on stories which seems very low.

It never got a lot of reads when I first posted and promoted it. I spent a good 16 hours writing it and it was useful for my own work. Let's see if HN is still into DBs.

The Parquet support, with the ability to handle row groups, skip over data not of interest to a query and aggregations that will likely be 3 orders of magnitude quicker are a pretty unique feature.

Also, there's no need to write any code here, it's a CLI app. There's much less that could go wrong versus rolling your own.

Feeding the fail2ban log to Papertrail or another logging service should give enough visibility.

I've seen fail2ban block the IPv4 of a company's headquarters because a new member of staff setup their SSH config wrong and made too many attempts to connect to a production system. In this sort of situation, having federated logs on external infrastructure is a real saviour.

Git supports extension-specific overrides which enables things like textual comparison of Office files. https://tech.marksblogg.com/git-track-changes-in-media-offic...

Long passwords can cause a DDOS attack as they need to be hashed. Longer the password, longer the hashing takes. Django had to patch this back in 2013: https://github.com/django/django/commit/5ecc0f828ebe270cfc92...

There are fewer steps to setting up Airflow locally compared to Prefect. There seems to be encouragement with Prefect to use remote servers instead of running things locally.

Airflow has been around since late 2014 and has ~800 developers contribute to it. It's hard to search for an issue with Airflow without coming across something helpful. That sort of community support will take time to develop with Prefect. Prefect's server repo will be celebrating its first birthday on Monday.

Prefect looks to be a project of a few developers aiming to make a hosting business out of the offering. I'm not certain if their ecosystem of developers and experts will grow to the size of Airflow's in the near future.

Prefect has taken advantage of recent language additions in Python 3 and its source is a joy to read through. They use Black to ensure the code formatting is consistent. They've decoupled the server, web UI and their answer to DAGs into 3 code bases.

Prefect seems aimed at those that want a managed solution to their data pipelines. The need for skilled engineers for data engineering has exploded in recent years and a lot of businesses struggle to find the senior staff they need to do things right the first time. I suspect Prefect will find some success removing certain steps and complexity in building data pipeline from less senior engineers which should help more businesses become data-oriented.

Very few Airflow competitors did a good job at decoupling from any one task runner, metadata backend, etc.. and brought few features to the table that Airflow lacked. Prefect's UI looks fantastic, the code base is really tidy and if they could ever turn the install process into a simple 'pip install prefect' with no need for Docker, K8s, npm, font awesome tokens or any external communication I'd be very excited to dig deeper into the project.

It's not about space reduction, it's about handling the small file problem. HDFS can handle up to 500M files without issue but the amount of RAM needed to store the files' metadata starts to go beyond what you'd typically find in a single server these days.

When you store multiple images and/or videos inside of a single PQ file, you'll end up keeping fewer files on your server.

I believe Uber store JPEG data in PQ files and Spotify store audio files in PQ or a similar format on their backend.

YouTube runs an in-house database called Procella. Its feature set is pretty amazing. Some of the devs behind it came from the Hadoop world. Google published a paper on its architecture.

Klepmann's book is still a good read. A lot of the concepts are evergreen.

I can't comment on native drivers. When I said ODBC-like interfaces I was trying to use a catch-all phrase for sources that look like a typical server-based data store (i.e. PostgreSQL, Hive, Kafka).

I've been using it for years with clients. It tends to sit in between source data and a final destination. Most data platforms are trying to take data from 10s if not 100s of sources and unify them. The variety of formats and sources is endless and I've often had to resort to using Python-based Airflow DAGs to collect data based on dates and store a cleaned up version of what was collected as a timestamped PQ file for Presto. Presto is then great for large scale transformations across the data and ad-hoc exploring.

Presto can be pointed at a lot of data stores but few external data providers offer ODBC-like interfaces. It seems to be either APIs or static file dumps for the most part. So Presto isn't going to be able to pull from these datasets alone.

In terms of security and maintenance, products like Redshift are easier to train traditional data warehouse people up on. The service is relatively cheap and has a nice UI for scaling.

The data world is extremely fragmented. Once firms have something in place changing it is going to be a struggle. Existing staff often gate keep and defend whatever technology they've staked their careers on. Once there are a lot of reports setup with any on data source migrating it could end up becoming a prolonged project which can be hard to sell.

It was quoted Snowflake had a $1M / day budget for sales and marketing. I'm not aware of any Presto consultancy spending that sort of money. Amazon does have Athena but they have countless other offerings which muddies the water.

Red Bull are a kdb shop: https://youtu.be/QxfdFWKo_pQ

I've seen Numpy code slightly blurred out of Mercedes videos on YouTube in the past.

Dexed, Odin2, VCV Rack, Audacity, Obxd, Helm and Surge are all open source and you can find their code on GitHub. JUCE seems to be the framework used to make most of the above compile across platforms.

VCV Rack does a good job at modelling analog synthesis, Dexed does a great job of modelling FM synthesis (this project is based off a 20% project at Google). It might be worth spending a week compiling the above, finding bugs and submitting pull requests to the projects. If this is the type of work you're looking for then doing it in a public fashion for free for a short period of time should help in your quest.

Audacity has a very large code base and there was a good CppConf talk on it a few years ago where one maintainer attempted to try and plot out its complexities. Many of the other code bases are reasonably concise and digestible.

Songs will often have 10s and in some cases 100s of instances of VSTs being used so there is a huge focus on using CPU cycles as efficiently as possible and moving GUI code off to the GPU to save as many cycles as possible. Code stability is also extremely important, treat your code like it's running a nuclear power plant.

It seems C++ rules the waves in the audio world from all code I've examined. I've dug through LinkedIn and it appears few audio tool manufacturers employ more than 100 staff. The revenue distribution among firms is a lot more spread out than you'd find in other tech industries as well. Boutique firms and even one-person shops seem to do well. I'm not sure if there are any real gatekeepers to this industry beyond your ability to market to cost-conscious creative types.

Aren't most developers in the world paid a salary or day rate and thus are paid for their time? What incentive is there for them to "hurry up" rather than sandbag?