Ask HN: What's an unsolved problem in your field? | Hacker News Reader

Ask HN: What's an unsolved problem in your field?

56 pointsNirvash90 comments

There might be several different categories of problems, from the literal "unsolved" (i.e. mathematics / physics) to systemic (i.e. human resources / advertising).

Some examples that come to mind:

    Brand Influencer — "The Algorithm" prevents exposure, sales, etc.

    Customer Service — Explaining warranty status, other than "because it's the least we have to do legally".

    Mathematician — The length of pi is continually increasing, and there doesn't appear to be an end.

    Software Engineering — Thoroughly understanding a codebase in a reasonable amount of time.

Ideally, this would be less of a "here's why X field is bad" thought exercise, and more of a "that's interesting, I wonder if X problem could be solved" thought exercise.

View on HN

> The length of pi is continually increasing, and there doesn't appear to be an end.

Huh? How is this an unsolved problem? It's known that pi is an irrational number, so it doesn't have an "end". We can always compute more digits. Or did you mean something different?

Too much Aronofsky?

Not sure on the customer service one either. The same logic means calling privacy policies a problem because we legally have to follow privacy law?

Warranties are there for consumer protection. Most people know what they're there for. Even if some people didn't understand it, businesses that treat warranty as "something we legally have to do" probably don't consider the confusion about warranty a problem, because clarifying it would make more people exercise their warranty rights, and hence more costs for them.

Honestly, I don't see how any of the examples, except perhaps the software engineering one, are problems experienced by businesses that they wish to solve. Perhaps I'm greatly misunderstanding.

----

Edit: I think the OP may mean businesses that wish to advertise their warranty protection better explaining to users what that "end date" means, and what the warranty offers them. Which I think can be done fairly easily: look at what Apple does to advertise their warranty protection, or the warranty stickers on every fridge/washing machine you've ever bought.

Although, Apple tries to upsell AppleCare rather than advertise their legally mandated "Apple Limited Warranty" and consumer law warranties.

In regards to warranties, it can be difficult to explain that service will cost money after a year for instance.

Especially when it's an issue a customer didn't directly cause (i.e. spontaneous part failure).

Giving OP the benefit of the doubt here, it seems like a straw man example off the top of their head, just to throw some ideas out there.

Definitely a bad example on my part.

I'd love to say I meant something different, but I simply blanked on pi being irrational.

It happens :) You did pose a very interesting question!

Software development: creating web CRUD interfaces quickly. Here's the orders table, here's the customers table, here's the products table. Join them, present them with a nice UI that scales down to basic smartphones, make some fields editable, some sortable, some filterable. Now take a subset of the result and make that its own table with a search form, and so on.

Every solution I've tried is either too narrow (e.g. works on one table or kind of data only), too broad (too much boilerplate), hard to plug into existing data without massive ETL (SAP, Oracle APEX..) or cloud-based apps which are fine for a mom and pop store but basically useless for scenarios with millions of entries.

+1 to this.

Code has been written tens of thousands of times to take data from a webform, put it in a database, and display it on a webpage.

It's about time someone could just put that in a standard, simple, easy to use, yet extensive, library or service.

Simple things like defining a field on the frontend, in the server API, in the data model, and again in the database schema should be a thing of the past.

Well, there's Wordpress, isn't there? You can make a basic web application pretty quickly with no code.

And there's also MVC platforms that auto generate a lot of code. In ASP.NET MVC (and many other platforms) you can just define the data fields and it will generate views, models, controllers and DB scripts automatically.

Which is well and good until you have to process business logic, at which point you need a developer.

I have no personal experience with it but Retool looked nice, and probably covers some of your requirements.

Definitely an interesting problem to solve!

Out of interest what did you try and find most acceptable?

Well, the whole field of software development is geared towards solving that problem, isn't it?

You can have something highly customizable (e.g. write a web application in some common language), or you can try to create some out-of-the-box solution that works great for certain apps but quickly becomes a mess when you try to go beyond it.

The core problem is there's a lot of complexity that's not always obvious and it has to be dealt with somehow.

How would this work in practice though? Sorry if the question seems dumb.

For your own example : orders, customers and products. How would you generate a UI that matches design decisions of your product? Without specifying super detailed rules, in which case you could write it yourself? What am I missing?

Mathematics is the best field for this because a list of unsolved problems is kept, and many of them are relatively easy to explain: https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_m...

Chemistry/materials science: room-temperature superconductivity is probably the big one. There are a whole host of problems in the energy space which would benefit from improvement; while "electricity+CO2+water => fuel" is feasible at the moment it's uneconomic. Can it be done at close to the theoretical minimum energy input in a plant that's scalable and cheap to build?

IC design: is continually solving previously unsolved problems like EUV lithography, but has struggled for years with trying to go "3D" to overcome density issues. Also, is photonic computing feasible and would it achieve lower energy usage?

> electricity+CO2+water => fuel

Considering fuel => electricity+CO2+water is financially viable today, the reverse can never be financially viable unless the prices of the inputs and outputs shift quite a bit...

Hmm is the argument that you can generate infinite money otherwise? I think the two directions only look superficially the same because externalities and labor isn't taken into account?

This view ignores that energy can have a different cost in different locations, and that fuel is convenient due to its portable nature. Converting energy into fuel may be economically viable at a solar plant in the middle of a desert even if it wouldn't be so in the middle of a city.

Of course it can, it all depends on the actual proportions in each reaction.

Most notably of the share of CO2 in each, which aims to be drastically different.

It still depends on how expensive it is to get the fuel. It could be essentially free to go fuel => electricity + CO2 + water but that's no good if getting fuel is expensive.

Unless you include the price of obtaining fuel in that transformation arrow, but then the observation boils down to if extracting fuel is cheaper than synthesizing it then synthesizing it isn't financially viable.

The actual problem is that the downsides of turning (fossil) fuel into anything else are not priced in. If the damages done by emitting carbon, particular matter, nitrous gases, sulphur, ... had to be paid for by the producer, then fossil fuels would have a very bleak outlook.

There are times and places where electricity has a negative price, and potentially a carbon pricing scheme could subsidise taking it out of the air. But a big factor in this is the cost of the plant; nobody's going to spend £100m on a plant that runs 5% of the year. Hence the desire to simplify the process.

The US navy also has a use for this: at-sea refuelling of planes and other ships from the spare power from the nuclear aircraft carrier.

Software development: how to make realistic estimates, and deliver a solution on time.

Software Development: How to identify unknown unknowns and communicate the effects of scope creep?

This has been the worst for me. It seems near impossible to properly calculate edge-case development, alongside core functionality, while also building a good UX.

Only possible if you don't do anything novel. If you are building a website that's got all the same patterns as the previous 5 websites you've built, then you can provide a solid estimate and deliver on time.

If you're building something that nobody has ever made before, you've got unknown unknowns, and your best bet is intuition, which is inevitably going to miss the mark a bit.

And if you repeat something 5 times, you probably already started automating that anyway.

Make a wild guess then multiply it by a number between 2 (10+ years of experience) and 4 (newbie). Works pretty well.

Don't you want something simpler, like P=NP or the Riemann Hypothesis?

Just kidding. The problem with estimates in SwEng is complicated. The main problem is: every system is different from another.

Anything that's moderately complex will have a variable production time because the process keeps getting improved.

Now, you're not building 10 systems of the same type. Maybe if what you did all day was building, let's say, a blog with RoR you would be able to time it to the minute (but then again, that's not how software works, and once you built it once, it's just a matter of copying files).

In traditional "factories" the design takes some time, then putting together the production line, tools, then the production.

In SW, "real" design and production go together, and "paper sw design" is fragile.

Just stop paying devs by the hour.

I am a developer. My go-to answer for "What is your weakness" type interview questions is that I am bad at estimates.

interviewers want an example of a problem you've identified and are actively mitigating. they want to hear what solution you've created to deal with your own weakness. if you're giving the same answer you would've given a year ago, i have doubts that you're providing a compelling answer to the question.

Mechanical Engineering - Navier Stokes equation in 3D. This is the equation that governs fluid flow, so finding a solution in 3 dimensions should allow models & simulations to become far more accurate.

This is our version of P=NP, and similarly, there's a $1M prize for finding a solution.

https://en.m.wikipedia.org/wiki/Navier–Stokes_existence_and_...

Here is a problem that I've been thinking about for some time: suppose I express using some type of logic that I want the shortest path between two vertices of a graph. Given this logical description of the problem, is it possible for a computer program to emit Dijkstra's algorithm or some other efficient algorithm? One of the things that have interested me lately is logic programming, but I'm wondering if there has been any research done on using logic programming as a means of algorithmic discovery? What are the theoretical limitations of doing this? It took bright minds to come up with the various graph algorithms that we use today, and so I'm assuming there's a fundamental limitation that makes it difficult to convert declarative expressions of a problem into specific efficient algorithms.

Perhaps what you’re thinking of is inductive logic programming, although I’m not sure that field has advanced to the point where your particular example can be solved.

I remember minikanren being able to generate expressions that evaluate to a value.

In general, I guess the search space of all algorithms that might or might not solve a problem is simply extremely huge, so you'd need some good heuristics to find an interesting algorithm.

I can't think of any reason why there would be a theoretical limit on this, but I'm no expert in this.

CS Education - Virtually everything is unsolved. How do we teach people how to code? What is the progression? Do we teach them how to "think" algorithmically first or just dive into a specific language? How do we assess their learning? Which skills are transferable?

I find it interesting that out of the massive field that is CS you focused solely on a single area.

Yes, isn't the question "What's an unsolved problem in your field?"

Is CS education a problem to be solved or is it an art? I think techies get spoiled by how clean mathematics and technical solutions can be. But people are complex and perhaps the only solution is intuition developed over years of experience.

I'm not saying some ways of teaching aren't preferable to others -- obviously, lectures and hands on teaching is better than just throwing high school students a Dennis Ritchie book and telling them to come back in a year. Just that there's no "one size fits all" answer. (Unless I misunderstand what you're asking.)

I would say it is as solvable as most areas of psychology. The goal isn't to unveil a "one size fits all" answer, but rather to understand what works better in what situations and why. The progression of knowledge is generally from a craft to an evidence-based science.

Relatedly, I'm curious about different ways that CS education can be framed. Speaking to teachers, one of the main reasons why many kids (and, crucially, parents) aren't interested in CS is the misconception that you can't be creative with programming. When kids find out that this is not the case, engagement rises. (There's also the point that CS seems to invariably be folded into programming, but I'm not sure this is the biggest problem.)

I wonder what the impact on uptake would be if the focus was shifted towards CS as a venue for building things and being creative and away from lines of monowidth code and indecipherable errors. More to the point, I wonder how this might be done.

Physics: Quantum Gravity - Quantum Field Theory does not combine well with General Relativity

Physics/Astrophysics/Cosmology: Dark Matter and Dark Energy - are clearly observed, no satisfactory explanation exists.

Astrophysics (and Science in general?): very knowledgable specialized domain experts have issues learning from each other: too many different concepts, relations, methods. Even if universe is shared, representations are not easy to map on it at once. Astrophysics is especially bad in this.

Optics: Closing the computation to hardware gap. Yes, it is a 'solved' science, very much so, decades in fact. We still struggle mightily in making anything practical out of the equations. It's a bit backwards from most engineering. With bridges and milk-jugs we can get pretty close on paper to the actual product. With optics, we kinda get the theory after we've built it and made the light/photons go all wiggly. It's tough stuff, but would be very useful.

Mathematics: A generalized solution to n-order partial differential equations. Laplace, Chebyshev, Hamilton, Wave equations, sure, ok, yes. But I mean any n-order partial differential, not the 'lucky' cases. You get that, and you've solved a great deal of the outstanding problems in other fields (hydrodynamics, optics, economics, turbulence). I'm not sure up to date on this, and it may have been proved that a general solution is actually impossible, but I'm not sure.

Physics: Braver grad students. Quantum Gravity is a long was off, we need much bigger machines or a constellation of satellites that use the sky as the particle accelerator. As such, having these brilliant minds labor on obscure portions on some niche interpretation of physics, well, it's useless really. The data will make it all clear as day, whenever the data gets here. Until then, the grad students need to be braver and strike out on their own, leaving the golden age behind. We need them to work on other things. I know that's tough as nails, but I think it's what is needed. Great work has been done in neuroscience by physicists (despite their best efforts :P), and the quants on Wall Street are a meme now. More of that, but in, I dunno, weaving or something.

Information Security: getting people to classify data properly. Everything ends up as the default classification or set to public so they can send privileged data over email.

Also, having some agreement on code scanning. Every time security settles on a code scanning tool, engineering gets a million findings. This results in arguing about whether potential risks are actually vulnerabilities rather than improving security.

User Research - What's the best way to deal with loads of qualitative data? There are different approaches but all of them have problems.

I’ve recently started working on documenting (still very, very early) on “measuring” user feedback at https://datadriventeam.org after having 50+ conversations about it with various startup founders/leaders. It is backed by an open source GitHub repo and I hope people would contribute once I’ve gotten the initial version ironed out.

Right, what you're describing there is basically a framework to do content analysis ( with quotes as units of meaning, and tags instead of codes).

It's definitely an appropriate strategy in a lot of cases. But sometimes it just doesn't capture what matters (e.g. the pure counts of how often an issue comes up =/= the severity of the issues).

To have a better sense of how people feel, you could do thematic analysis. But then you sort of lose the counts. Or you can pick and choose the most important situations and start writing scenarios. I guess my point is that content analysis is just one of many strategies, in the same way that boiling potatoes is just one way to prepare them.

Anyways I'm not trying to put down your work, I think the website could help people feel more at ease with qualitative data. I'd be eager to contribute once you're ready for other contributors. Feel free to send me a message and I'll happily write some stuff up.

Programmer here, nothing is solved.

Getting non-technical people to pay for software.

That's partially on the community for making so much free (as in beer, often also as in freedom) software. The expectation is now there. Though how to reverse it remains an unsolved problem.

There's a solution to that: build something that clearly solves a real problem that they're facing.

Let's take a real example: a friend is reusing the same stupid & insecure password everywhere, tweaks it with random numbers and letters to pass password strength requirements and of course forgets it minutes later, so she has to go through the password recovery flow every single time.

1Password is ~30$ a year, and yet there's no way to convince her of paying for it. This is a person that's employed full time and that otherwise spends unreasonable amounts of money on hoarding clothing she's not even wearing so it's not like money is an issue.

I work in logistics building a transportation management system. Our unsolvable question is not technical - how do you get trucking companies to use technology? It is more unsolvable than the 7 Bridges of Koningsberg.

You show them that they can make more money by using technology. Out of several thousand trucking companies, a few will try it. If it pays off for them, then some more will try it.

And, really, if it doesn't pay off for the few that try it, why should the rest use it?

Here are some that come to mind:

* Professional Software Developer Certification. Software developers do not have an industry recognized certification or accreditation program. Every other professional industry has this. Truck drives have this. Here are some specialized subcategories.

    - Web Development
    - Security Remediation
    - Operating Systems and Systems Automation
    - Applied Mathematics Applications
    - Data Management

* Heat Energy as Electricity. We waste and expend so much energy in the form of heat that could, if captured and stored, be converted to electricity.

* Energy Efficient Hydrogen Capture from Water. Currently it takes more energy the shatter a water molecule than you would gain from burning the resulting hydrogen. Liquefied hydrogen is a wonder fuel whose energy efficient combustion yields water as its waste product and could power spacecraft deep in space.

* Obesity. Obesity is caused by a combination of 3 things: insufficient exercise, preference for carbs over fats as the primary energy source, and unhealthy fat sources. The third one can be solved with a combination of science, agriculture, and economics.

* Mental Health Therapy. There are a tremendous number of people who need mental healthy medicine but never get it (for many reasons). By tremendous I mean an utterly astonishingly significant percentage of the population. Those who do get medicine are often prescribed drugs instead of therapy when therapy is generally more effective and doesn't have negative side effects. Also the sheer quantity of mental health medications is detectable in the public water supply.

* Rapid Oil Metabolism. Oil is a necessary part of the modern economy. Crude oil is refined to make plastics, and so it will be with us well into the future. Oil spills are nasty though. It would be nice if there were micro-organisms that could consume oil so that oil pools could be removed organically in months instead of decades/centuries.

* Space Entry. We are currently limited to using rockets to enter space (or exit Earth). That is horribly fuel inefficient. Any alternative would most certainly be cleaner and more energy efficient, but there aren't alternatives yet.

Truck driving doesn't change a great deal. Software development still changes on an annual basis. As such, it feels to me that certification would go out of date extremely quickly.

Law and pharmaceuticals change substantially faster than technology does, so that strikes me as a weak counter-argument.

Contrarily the bed rock of most web technology standards were solidified 1998-2001 and yet most web developers have no idea how these things work without a mountain of abstraction to do it for them. This indicates speed of change isn't a considerable barrier in that example.

In Python education: Getting new language learners to a functional local development environment as quickly and painlessly as possible.

"Just use Anaconda/pipenv/the-Python-installer/Docker/etc" isn't a great answer, because they probably tried one of those six months ago, got into a weird state and can't remember what it did or where it put things.

Obligatory XKCD: https://xkcd.com/1987/

Noobie developers can install packages globally. Apt install python3-pygame would get the base, then use Geany as an ide connected to game.py.

Some terminal use will be needed, but it shouldn't amount to more than running "python3 game.py".

repl.it not a local environment, but if you're after the shortest path to learners being able to code, that's how I do it with the people I teach. Works well.

Yeah that's why I specified "local" - personally I'm keen on https://mybinder.org/ and https://colab.research.google.com/ and https://www.glitch.com/ as remote options.

Let their first lesson be that they are responsible for their own machine. If they installed crap without being able to fix it or even remembering what they did, then they'll have to reformat and start over. Suggest they keep notes this time around.

Learning to program is intimidating enough already without telling people they'll have to reformat!

I agree and I have never tried it on windows but I suspect it is worse than Linux because my brother almost threw his laptop out the window when he first started but got along after installing Linux.

I think Rust excels here with:

    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

And cargo is great to work with. Go seems to have similar problems from my experience.

I'm sure this will be an unpopular answer on HN but have you tried Azure notebooks? Microsoft's Eric Camplin has a series of edX courses[1] that do a great job getting total newbies up and programming with python as painlessly as possible.

[1] https://www.edx.org/bio/eric-camplin

Perhaps in software sales, why customers didn't choose to buy. It's a lot easier to poll customers on why they chose to buy once they did as you have their contact info.

Energy simulation: whether a convolution using kernel A on top of a thresholded convolution using kernel B can be represented using just one convolution using kernel C.

i.e. with H as Heaviside operator, T the threshold, and * the convolution operator, prove that the following can hold for some kernels A,B,C and signal D:

A * H[B * D-T] = C*D

Not C * (D-T)? If the idea is to combine the thresholding and the kernel wouldn't it be easier to start with combining B and T first? E.g. B * D-T = B' * D?

Once you have two kernels (no thresholding), combining them is fairly straightforward, I think (assuming no singularities). However, I primarily work with straight convolutions for signal processing, not thresholded convolutions, so maybe I am missing something. If so, I'd be happy to learn more!

The notation B * D - T is simply there to denote the threshold, but nevertheless combining B * D - T can done. A convolution with a constant yields another constant[1]. You can find another constant T' such that T' * D = T, and with the linearity of convolution operator, you get B' = B - T'.

Combining two kernels is straightforward indeed. You just convolve them together. This is possible because convolution is commutative.

The problem is when the thresholding operation is introduced. This makes the whole thing nonlinear. So far, the best way to calculate it (now I'm talking about 2D convolution) get a derivative form out of the kernel in order to apply Kelvin-Stokes theorem by tracing along the contour of the thresholded convolution.

[1] https://math.stackexchange.com/questions/1054165/convolution...

Fair, effective, and efficient technical interviews.

IMO, the best solution is probationary hiring (e.g. 3mo contract-to-hire), but that means you have to leave your current job first and then spend the next year bouncing through a number of "temp" jobs until you find a good fit.

Credit modelling: How to predict if someone will pay us back or not (probably an unsolvable problem)

I have some experience working on this problem.

These Kaggle kernels contain interesting feature engineering and tuning strategies https://www.kaggle.com/c/home-credit-default-risk.

This is nice if you have a single feature that rank-orders well and you would like to calibrate https://www.chrisstucchio.com/blog/2020/isotonic_python_pack...

As long as you have collected data on outcomes that contains more than 100 defaults, with covariates that would allow an informed expert to distinguish between higher and lower risk transactions, it's often possible to be more accurate than an informed expert on this problem.

Replacing the few human employees remaining with robots. Or possibly some sort of genetically engineered ape/dog hybrid. Whichever is cheaper, and has less legal red tape.

Statistics: How do we quantify evidence for a hypothesis from data?

Examples: p-values, Bayes Factors, credible regions, a likelihood ratio. These are all quite different!

IT - easy automated governance for non-technical folks

With so many data privacy regulations (GDPR, CCPA, etc) propping up across geographies, how do i make sure that my company is in legal compliance and doing right by our customers data.

There's a lot of GDPR consultants milking your company's cash for this.

Also probably a lot of articles on privacy law across the world, available free. For sales tax/VAT around the world, for example, avalara.com has comprehensive information and monthly newsletters for free.

Generally, you can't go wrong by just doing the right thing. GDPR is nothing excessive. Businesses that weren't being awful before GDPR are probably in the clear after GDPR too; they just need to update their privacy policies.

Not to mention: if you can't afford a consultant, you probably aren't large enough to be targeted by the regulators on tiny bits either. If you're acting in good faith, at least the European regulators will cut you a break and just tell you what to improve, without penalty, if you're ever flagged.

Can systematic compositionality be achieved by a connectionist approach?

How to earn money as a publisher with online advertising.

printers

Easier to make a whole society paperless than fix the printers.

Accounts receivable.

Reply on news.ycombinator.com