ellimilial

There certainly is quite a lot to be said about constant requirements drift. However, this is not something untypical to some of fast-paced product work or, even more closely, r&d effort within the industry.

What then drives the improvement of the code quality is the potential need for continuity and knowledge retention - either in the form of iterative cleaning of the debt or the re-write. This is reliant on the perceived value for the organisation. From this perspective it's more straightforward to get to author's reasons.

Neat!

I can imagine a periodic thread here where one could ask for, let's say 'unrestricted' feedback from the community.

Dear HN community, I have spent several years working in data operations for companies applying ML/AI to medical problems - e.g. pharmaceuticals / drug discovery and cancer diagnostics.

It's a truism that the data availability and quality are paramount to getting any useful results out. However, what I have seen and heard consistently is that the knowledge of biomedical datasets is a) siloed by specialisms b) has a very high barrier of entry (i.e. you need time and ideally some practice to figure out what's state of the art) c) the choice of databases in some domains can vary considerably between users be it companies or researchers - often without them being able to explain the reason.

To quickly understand the landscape our clients operate in, I started wondering if there's a way to systematise this information to know what datasets I should know well. I looked at aggregators like OpenPHACTS, OpenTargets, small curated lists (like Expasy or Nature recommended data repositories), but those were relatively small. I was aware of some larger aggregators (like NAR Database issue, FairSharing.org or Google database search). Still, it would not be easy to make sense of each of them - and they were not necessarily limited to biology/health data. The question slowly evolved into' what datasets are out there that might be useful for a particular ML/AI use case'.

Long story short - with plenty of help, I started building a discovery tool, mostly based on our needs but noticed it was useful for others. It cannot replace domain knowledge but provides a heuristic to be a starting point for new entrants (ahem ahem - hoping there might be a couple in this community). The needs evolved beyond a Google Sheet (that can solve like 80% of problems of this scale), so it's wrapped with a small search and recommendation engine.

It contains over 5000 datasets, periodically refreshed. They range from cell biology, chemistry, protein and molecule structure and classification through pathways, drug information, side and adverse effects, omics, to anonymised medical records and clinical standards. It merges the results from specialist sources with our private curated collection, reconciles the records, and augments them with information extracted from the scientific literature.

As far as we are aware - this is the most extensive such open collection of biomedical data - hoping this may be of some use to others, especially early in the journey!

'This is – to use a technical term – bullcrap.' It's moderately inspiring to read this kind of technical opinion from a technical leader.

I have heard similar objections/uneasiness concerning delivery-aligned reporting, coming especially from more junior specialists. More so within teams with broader sets of disciplines.

In addition to the need of improving their skillset and receiving tailored career guidance, there is sometimes a strong political component.

This won't answer this question in full but perhaps some part of it.

Projects like PostgREST (and PostGraphile) use schema introspection to generate the API. When the schema changes it's automatically reflected in the API. Sure one has to keep the changes synced on the frontend but the premise is you don't want to decouple.

This.

Alternatively look for an open-licensed dataset if one exists in your domain (e.g. using https://fairsharing.org or, shameless plug, https://biokeanos.com). With generated data you always add some assumptions, with more 'wild' data you have a chance to discover edge cases earlier.

Not exactly for the same intensity of the feeling but the first coherent script I wrote was to impress girl I liked in junior high.

It was hand delivered on a beautiful red floppy, printed out some ascii art and made noise. Who could resist? (spoiler: she somehow managed to).

Perhaps I should have been more precise. You're absolutely right about being apologetic for the atrocities committed.

I was referring to trying to justify Soviet Union's attacks on Poland and Finland as well as annexation of parts of Romania. E.g.'The Soviet Union was prepared to allow other nations to fight one another so that it could have a share of the spoils'. It was not 'other nations' fighting and Soviet-German alliance provided room for Soviet expansionism.

It's educational how Germany's East / West division reverberates in this speech with the special relation to the USSR. The apologetic tone, appeasement, justification.

One can only hope that this would no longer be necessary nowadays.

Not that suppression of said rights leads to violation of (what is a current consensus of)of human rights.

Ironically, those events were hosted for remembrance of those violations.

A fantastic approach meant at justifying non-interest, always present in the history of totalitarian regimes.

Yeah, the difference is at least some of the media can at least attempt to be truthful with limited chances of accidental novichok exposure, falling through the window or accidentally falling on a knife multiple times. etc.

Seconding. Text searching is a horribly hairy problem. I know 2 businesses for which the main source of income is tuning ES/Solr to particular user needs. Starting from performance, through templating case-specific queries to custom plugins.

Didn't mathematicians earn million-dollar rewards for their theorem - proving? :-)

Are there any examples you are aware of and are able to share?

Well, maybe not humour in the end.

If we can't distinguish between machine and human-generated ideas how does it affect the patent system?

There's also a conversation going on here: https://www.linkedin.com/feed/update/urn:li:activity:6620646... whether this is the good reason for it.

One opinion is that if a new discovery is made via automation (e.g. cognitive automation) it lack an inventive step.

And if generating new patents is the current state of art, things that were patentable before should no longer be patentable.

Baffling, the liberties taken by this article trying to link ‘current’ discrepancies using correlated variables from well over 500 years ago. Without mentioning trivialities such as evolution of religious beliefs, history or totalitarian regimes, industrial changes of 17th century or Industrial Revolution. Not to mention the generalisations and lack of definitions (unless I missed an appendix). It really shows that in order to perform interdisciplinary research you not only need competency in both but also understanding how those toolkits interconnect.

Just a nitpick that this is the approach that benefits mostly the founders, not yourself.

You are likely going to invest a lot of time and put yourself under an emotional strain for a very unlikely possibility that a) they will agree with you b) will consider this enough of an issue to invest time in resolving. If you and the founders had good enough relationships to warrant such an investment on your side, you would not be in this situation in the first place.

Yes, most of the time for the non-exploratory code that is not deeply ingrained with external framework.

When starting a new module / class I put a skeleton first, to establish an initial interface. Then I change it as I find, while writing tests, how it can be improved.

When dealing with bugs - red / green is incredibly helpful with pinpointing the conditions and pointing exactly where the fault lies.

When introducing new functionality I do most of the development as tests. Only double checking if it integrates once before committing.

Going test first pushes your code towards statelessness and immutability, nudging towards small, static blocks. As most of my work is with data, I find it to be a considerable advantage.

It provides little advantage if you already rely heavily on a well established framework that you need to hook to (e.g. testing if your repos provide right data in Spring or if Spark MRs your data correctly).

I tend to change/refactor a lot to minimise the maintenance effort in the long run. I would spend most of the time testing by hand after each iteration if not for the suite I could trust at least to some extent.

Seconding that. Several ‘AI’ startups I worked with, do not require a massive amount of data to create MVPs/first releases.

The do however invest early in sufficient data science functions to have a good liaison between the researchers and data acquisition.

Sure if your use case is an analysis of interconnected entities and you either work alone or have someone to help you maintain a shared server.

Otherwise please don’t. I earn my living (amongst other things) due to organisations that went with this as blanket approach.

Use a right tool for the job. SQLite is fantastic for small to medium size datasets, shines in immutable case. Plus its maintenance and sharing cost is close to zero(something you will cherish once docker comes to play or if you want to learn or test some SQL without going through a pain of setting local permissions for your schemas).

Sure, and get 350 mln a week to fund NHS.

Wow, this is the first time I actually want to get a pad!

If not in your plans: - Could use a text input, otherwise pretty hard on desktop. - In the current and past companies we'd use draw.io. Not sure if it's possible how difficult it would be to export it.

Would pay.

This is gold.

I've become a member of a team the company scrambled to deal with a `legacy` python/SQL - based ingestion/storage system in an effort to 'harden' it. Despite my best efforts, we are going for a full rewrite into java/spring/avro/mongo/es. We have internal users talking SQL and utilising the system at the moment, a fair amount of data is relational.

I have run out of ideas how to convince the team and stakeholders, will have a one-shot chance to talk to VP. Any ideas how to voice the concerns about the full re-design (perhaps I'm just being difficult)?

A bit orthogonal to the questions (apologies for that), but leaving the link just in case someone might find it useful: http://www.paulgraham.com/corpdev.html

Centralised authentication / SSO. A fancy obfuscation indeed.

Yup, makes sense, looks like it's a decomposition of it. Might have been the talk of installing agents, sharding etc that put me off track.

It sounds like a much easier solution for EC2 instances. However it's limited to AWS - hosted apps, if I'm not mistaken?

Thank you for the feedback.

I was considering Kinesis but my use case (multiple small apps not running on AWS sending medium-sized messages a few times per user session) values simple components and keeping the long-term maintenance effort to the minimum.

Sure, Kinesis is a clear candidate for a larger scale scenario, hence all the disclaimers.

This is aimed at small-scale operations, utilising hopefully simple components.