sztanko

I agree it would be great to chose the location and I will try to come up with some solution.

Essentially, the main challenge here is that this is 99% statically hosted project. You download the full list of fuel stations and the app then shows you the closes ones.

We have around 7k fuel stations across the UK, which is 350kb compressed, so not a big issue.

However, full postcode list (1,768,727 postcodes) is 20MB compressed (62MB uncompressed), that won't be acceptable to be handled on the client device.

I am thinking of cutting off the last two characters off postcode, that will give us 11k rows and 286kb with around 1 mile precision in densely populated areas, that will do the trick.

In any case, I only started working on it on Sunday. Still have couple of ideas to implement, but need to do my day to day work :)

While not my most sophisticated project from technology perspective, it works, and is high-load proof.

It is almost entirely depends on github pages and actions, with a single exception being an AWS lambda which logs submitted data into Cloudwatch (which is the later collected by a cron github action).

Source code here: "https://github.com/sztanko/fuelfinder

This is very cool! Reminds me of https://phiresky.github.io/blog/2021/hosting-sqlite-database..., which is similarly cool project.

COMPASS Pathways | New York, London, San Francisco or Remote (UK/US) | Remote OK | https://compasspathways.com/

We want to transform the patient experience in mental health care. Our first programme is researching how psilocybin therapy could help people with treatment-resistant depression. A great opportunity to make an impact from day zero and promote best practices. We are a public company.

Tech Stack: Node/TypeScript, Python, R. Infra: Github, AWS (Lambdas, ECS), Infra as code tools.

* Data Scientist (US only): https://compasspathways.com/vacancy/data-scientist/

* Machine Learning Scientist: https://compasspathways.com/vacancy/machine-learning-scienti...

* Senior Full Stack Engineer: https://compasspathways.com/vacancy/full-stack-software-engi...

If you have any questions, email me at demeter.sztanko@compasspathways.com

It doesn't say it is just 6m rows. It is 6m patient, which only hints that one of the dimension tables is 6m. Facts gathered in patients might be significantly larger. Also, my experience is saying, if you have hundreds of queries running simultaneously, it is not the volume of data that can be a bottleneck. Depending on the system it can be anything, starting from acquiring lock on a record or initiating a transaction.

Apart from the genius of the author (conceiving a sequence of relatively understandable steps that unlock a potentially huge area), this highlights how efficient a SQLite can be in terms of slow resource usage.

COMPASS Pathways | New York, London, San Francisco or Remote (UK/US) | Remote OK | https://compasspathways.com/

We want to transform the patient experience in mental health care. Our first programme is researching how psilocybin therapy could help people with treatment-resistant depression. A great opportunity to make an impact from day zero and promote best practices. We are a public company.

Tech Stack: Python, R, Node/Typescript. Infra: Github, AWS (Lambdas, ECS), Infra as code tools.

* Data Scientist (US only): https://compasspathways.com/vacancy/data-scientist/

* Machine Learning Scientist: https://compasspathways.com/vacancy/machine-learning-scienti...

* Senior Full Stack Engineer: https://compasspathways.com/vacancy/full-stack-software-engi...

If you have any questions, email me at demeter.sztanko@compasspathways.com

Makes sense, thanks!

Data Engineer from Revolut here. Feel free to ask me anything. To comment on the topic, my team was never ever told to work late hours or weekends.

Opinions my own.

Main thread here: https://news.ycombinator.com/item?id=19309597

Just out of curiosity, what would have convinced you to consider a job with Revolut?

I felt like this article was describing a company that has nothing to do with Revolut. We have never been told to work late hours or weekends, for example.

In the context of engineering, the coolest thing is probably the absolute freedom in experimenting with new tools and technologies.

Hi, Data Engineer from Revolut here. Happy to answer questions on how it feels to work here, how things work, and technology stack we use.

Opinions my own.

I would continue working on my pet project, http://walks.io

Just try this out: https://github.com/EXASOL/docker-db and you will be impressed. This is an embryo of a real analytical database.

Pros:

- an 8 CPU installation with 64gb memory will probably be hundred times faster then postgres.

-it supports full sql

- It is super stable, even as docker container

Cons:

- it does not support nested data

- once you reach volumes of around 2Tb, you will probably have to switch to a paid version (I mean, you still can continue running on a 200gb ram box, but it will be suboptimal)

P.s. I am not affiliated with Exasol.

Revolut | Data Engineer | London, UK | Full-time | ONSITE

Revolut is a global money app, your digital banking alternative. Instant spending analytics, free money transfers, fee-free travel, crypto and other products. In two years, we’ve grown to over 350 people and we’re adding around 30 new additions each month. To put it bluntly, our culture is about getting shit done and owning what you do.

Currently, we are looking for Data Engineers in two functions: Core Infrastructure and Algorithms & ML

WHAT YOU’LL BE DOING - Enforcing consistent quality by incorporating tests and performing code reviews with data scientists and data engineers

- Exploring and experiment new tools, libraries, and technologies to improve our solutions

- Taking ownership of certain parts of the automation and abstraction framework dealing recurring ETL tasks, ensuring monitoring, reliability and scaling up of data both in volumes and variety.

- Creating and maintaining a company-wide repository of metadata and related artifacts

- Collaborating with product owners, engineers, and data scientists to implement a seamless data platform

- Helping out data scientists with data pipelines to ensure they have the best quality data available on time in the place they need

Tech Stack

- Primarily Python, SQL, Unix/bash scripting, but also some Javascript, Java and even Go.

- Cloud: GCP

- SQL: Exasol, PostgreSQL, MySQL, BigQuery

- Orchestration: Docker, K8s, Ansible/Puppet

Please apply here: https://revolut.workable.com/jobs/735131

Here is a photo of Denver taken from California (1400 miles away): https://www.reddit.com/r/Denver/comments/3xuc2t/denver_from_...

https://www.theatlantic.com/technology/archive/2015/12/a-new...

It is actually almost on the flight path between San Francisco and London:

http://www.gcmap.com/mapui?P=SFO-LHR

http://www.gcmap.com/mapui?P=SFO-LHR,+sfo-yeg,+yeg-lhr

Great project, thank you for your time!

(minor bug: Searching for "top 20%" results in "URIError: URI malformed /public/lecture_search.bundle.js?v=1.0.22:79346")

Optimal placement of labels is an NP-hard problem[1]. Since google maps transferred to vector and label placement is now done on the (mobile) client, I am not surprised by what happened. [1] https://en.m.wikipedia.org/wiki/Automatic_label_placement

Are you accepting pull requests? It doesn't work on amazon.co.uk and seems easy enough to fix without even bothering you.

Badoo, London

https://corp.badoo.com/jobs/?jvi=oHWj1fwH,Job

We are currently developing an in-house logging flow that traps hundred of thousands mobile client side events per second, aggregates, processes and stores them in HDFS using Python and Spark, then shows reports and visualisations on our React based portal using D3.

You, the Ideal Candidate, are a passionate and thorough individual who could contribute to the development of some components of this system. You must love exploring new tools, libraries and technologies in various programming languages, but also should convince us you are able to maintain larger and longer term projects. You can write easily understandable code that is so good that even your grandchildren will be using it. In the past you already worked with large volumes of data, experienced scalability problems and explored various data streaming and storage systems to understand their benefits and limitations.

You must have experience in JavaScript and Python, and you should be really good (better than us) in one of those. Experience in one of the JVM based languages is a huge plus. If you also have any side projects or open source contributions, that will be more than welcome.

We assume you know how to work with git, JIRA, regular expressions, etc. Apart from this, you should be really good at unix commands and SQL queries. Every day you will meet and talk with nice and friendly people who are using the system, so you should be good at explaining new features, showing new joiners how to start using it and explaining why things are not working in case of failure.

Badoo is a great company to work in. It gives you everything you need in order to enhance your skills. The problems we are solving are interesting and the tools we are choosing are cutting edge. We will be sending you to conferences and workshops, buying breakfasts and lunches, books and software for you.

Yaml is in fact a superset of JSON. You can parse a json with yaml parser. You can also mix json and yaml in the same document and use [..., ..] for annotating arrays and { "bar" : "foo"} for objects. Some yaml libraries allow you to generate yaml that has one style for top level and json style for nested elements / long arrays to improve readability.

However I have also learned the following:

- in python, parsing yaml file is hundred times slower than json (in fact, my benchmark was showing around 400x slowdown). Therefore you can't really use it in cases where performance matters at least a little bit. A yaml 1k line yaml can load more than half a second (yrmv)

- if your yaml doc is longer than two screens, it loses its readability benefits.

Therefore it is best to use a whole directory of yaml files, each describing a specific feature. E.g. Ansible is a good example of how to use yaml files.

Looks interesting, thanks!

"If you can't measure it, you can't prove you made it better" is a core value in our organisations' tech culture. We have written an interface in our framework for automatically creating new metrics and it is very easy for a developer to set up a new graph that monitors theirs code.

Now, we have another problem. There are over 2 million metrics in our monitoring system and no one knows what most of them mean. Some graphs have been set up for features that don't exist anymore, other graphs were set up by developers who have already quit, there are lot's of duplicated metrics and in general it is a mess. So we are currently working on this problem. I still would like to mention this is a better problem then not having metrics at all, but still a problem.

If you ever had a similar situation, I will be thankful if you could share your experience on how you solved it.

Things I have found worth digging into the history:

- Montreal: http://sztanko.github.io/solsticestreets/cities/montreal.htm...

- Riyadh: http://sztanko.github.io/solsticestreets/cities/riyadh.html

- New York http://sztanko.github.io/solsticestreets/cities/new-york.htm... (although most of the streets in Manhattan are aligned in a different way, see: http://en.wikipedia.org/wiki/Manhattanhenge )

- Old cemetery in Vienna: http://sztanko.github.io/solsticestreets/cities/vienna.html

If anyone knows if the alignment is intentional or that is just because a statistical artifact, please let me know.

Thank you for this map! Two questions: - what implementation of quadtree are you using? - will you ever consider opening the data for download as shapefiles/any other vector format? Thanks again, great job!

Project Lombok is your friend then: http://projectlombok.org/features/Data.html

Apologies, I totally forgot to mention it works for England only.

Just in case if you would like to plan your walks better/discover new places, let me introduce you http://walks.io, my weekend project that does this for you by solving an NP hard graph problem of finding an optimal circular route.