ellimilial

If it fits on a single machine - jq, flat files, JSON lines / avro if relatively flat. Change to a tabular format if when nesting not required.

Postgres JSONB works, but it requires maintaining a heavy server process. So does Lucene/elasticsearch.

I have been yearning for embeddable store (in line with SQLite the support that both works and also keeps the data compressed like JSONB). I know there were some attempts, tried some of it those, mostly monstrosities).

Try a nociception, drug design or pharmacy group at your local university. Chances are they’ve been doing this for years.

Great content, but incredibly difficult to trawl through.

It's always great fun compounding new, manual 'indexes' when you discover you need another query.

With a lot of fuzziness, some state and temporal stuff.

Yes please, it took me over 5 secs to figure out what to click, felt like ages.

Diffusione Urino

Sure it's delish, very appealing.

For https://biokeanos.com (biomedical data catalog and discovery tool), keeping things on disk and versioning by pushing to repo saved me months already on dubugging and maintenance only.

Isn’t this counting the lines with any G/C in them vs the total number of G/C literals?

A context might be useful.

From what I gather, the author is a researcher in bioinformatics related field. This may indicate that they tend to work either alone or in a relatively small group. The domain is small scope data processing/manipulation, research/exploratory code, ,likely short-lived or even one-off.

The progress in this context will possibly be governed by sheer processing speed (e.g. it’s unlikely anyone will delve deep into the code, a lot of iterations to ‘just get it done’ instead of testing etc.).

If this is more or less correct, the point that Nim might be more useful than Python for the author sounds very sensible to me. It’s a nice spot between command line tools and more functionality-loaded languages.

The team being 'all-British' for obvious security reasons. Which I imagine might have felt like and insult to an injury to the 'little people', who, despite cracking the code, were not permitted to continue working on it. Making them, you know, 'little people'.

The wording of the abstract indicates that this article itself has been auto-generated, right? :)

A fair amount of drugs inhibiting it had to be recalled after https://pubmed.ncbi.nlm.nih.gov/16554806 and subsequent regulators actions.

Quite possible. Nobody wants another hERG fiasco.

@setr explained it really well. A side note, NoSQL also includes graph databases, dedicated to this type of node/relationship traversal.

Hi Jason, thank you very much for the background and the explanation. It is fascinating to see the progress in this direction.

I started raising my eyebrow (in the best possible sense) upon seeing parts of tooling very similar to ours but simpler and more importantly - without moving parts. We operate in biomedical data space and deal with flat/static data a lot, for example we power https://biokeanos.com with data-in-repo, so Flat Data was immediately interesting.

It is really inspiring to see GitHub actions to having a foray in this direction, definitely something to keep an eye on.

Thank you for the response and clearing up the 'billion rows' / surly bonds confusion I had from reading project's Why Flat Data? section. I think I understand the target use case slightly better now.

One of the strong arguments for object-like storage (S3 etc) in the context of plain / flat data is scalability and availability for large scale processing frameworks. Databases are only occasionally relevant.

Very interesting how Github comes with more and more interesting 'actions' to turn repos into 'platforms' and moves us closer to serverless future.

@idan how does it scale with the size (including storage)? Is 'a billion rows' a goal or an actual tested use case?

[...] all currently have favorable relations with the Kremlin, including [...] Georgia, [...] Ukraine.

One might wonder how unfavourable relations with Kremlin look like then.

This really stuck a cord, thank you for sharing.

And they point that out: "There are other reasons for key logging, such as providing autocomplete functionality"

"There are other reasons for key logging, such as providing autocomplete functionality" - I don't think they're stretching anything.

A Polish job board https://nofluffjobs.com took over the market in several years simply by forcing employers to include the range when posting the offer. So there is a supply/demand argument here - at least for some markets.

The minority and underrepresented perspective is really important here.

But it takes a while to learn how to evaluate your skills against a particular company's needs, market at large and learn to negotiate. I know few people who were 'brought up [...] to stir the pot'. Talking to people, putting yourself out there, doing research and then finding a workable deal is not something that just comes as a given. But yes, privilege exists.

Having said that, in a competitive market a salary range simply makes a potential employer more trustworthy, and avoids wasting time for both parties.

You misspelt 'work in an office with a bad atmosphere', you're welcome.

The other way around. For most problems you don't need a DB. Only for the subset of those that do you need a dedicated process/container/machine.

We have learnt to use full-fledged RDBMSs as a default because they proved really flexible and powerful over the last 30+ years. But they do have limitations and cost.

It can’t run embedded. Having no live parts is a real paradigm shift for so many scenarios.

Hardware in this case being able to host relevant bacteria / archea earlier in their digestive tract.

It's on their roadmap https://pandas.pydata.org/docs/development/roadmap.html#apac...

I am actually quite surprised at the figure of 73% research-related code packages not being updated after the publication, was expecting it to be higher.