Ask HN: How big of a problem is unstructured data for companies?
Are there companies already in this space?
Trying to see if there's something here before I possibly create a MVP.
Are there companies already in this space?
Trying to see if there's something here before I possibly create a MVP.
Not just old and out of date. Plain wrong, like referring to internal systems, processes or intranet URL's that are (often poorly done) duplications of enterprise standard services, processes or pages, and should be deprecated instead of incorporated into training data.
Or the unstructured data sits within structured data. Some development teams stick giant strings with their own internal, proprietary formatting into a database blob column because...I dunno, they thought it was expedient or whatever for Very Good [Undocumented] Reasons. Invariably, these fields hold all sorts of nastiness you either want to tease out, or do not want to exist at all because they expose what you wish they wouldn't in cleartext. And even though your company pays the vendors of these monstrosities (oh yes, my PTSD suppressed memories of multiple vendors committing these same terrible, no good, very bad software sins) three commas a year of "support", they adamantly refuse to share their precious proprietary format with you, much less give you the courtesy to update you when they change the byzantine format that gives the three body problem a run for the money for a deterministic solution.
If you make a Generative AI sort sense out of that, then you can practically segment the market by "how much before the customer starts to put away the blank check" and you'll make bank. Everyone I know of in this Data Loss Prevention (DLP) or adjacent spaces are deer in the headlights when the conversation turns towards these operational realities and what we might do with their solutions to put a dent in these multiverses of pain. No one has roadmaps talking of iterative pipelines, search outcome behavior analysis and automated training thereof, or really much beyond various degrees of pattern matching, whether that be regex's or LLM's.
Where I see genAI fitting in - providing an incentive for teams to keep their docs up to date. Similar to how SEO was a driver for webmasters to clean up their info architecture. If I know that people are going to be chatting w/ an AI instead of bugging my team, I could justify more investment in documentation.
The workflow is by hand, though not quite all the way down, it damn near is.
> At what point in that workflow would a possible generative ai be of help?
Wherever the non-deterministic output of Generative AI still delivers benefits that exceed the transactional value of getting it wrong some X% of the time. Highly data and context dependent. That's a way of saying the "It Depends" cop-out, but I have a hunch we have to solve each industry segment individually, if not in finer gradations.
Most companies have internal training videos, recordings of meetings, PDFs of policies, and basically all of these are worthless within a year from a business perspective. Some things are useful for longer, or for reasons of historical interest, but the half life is short. The real thing that contains value is what those could potentially represent, like decisions or events that would benefit from action. If a meeting has a decision that some executive would want to know about, maybe a summary of the transcript could be useful?
Turning the random memorandum generated as part of business into valuable insights without process re-design is a pipe dream in most scenarios. Not all though.
https://stratechery.com/2024/enterprise-philosophy-and-the-f...