In particular, I have the following questions:
1. What was the product you were working on?
2. Were there any new software engineering challenges that came from working with GPT4 (e.g. output quality, testing, monitoring, etc.)?
1. I've observed multiple products across customers.
1.1 Correcting or filling missing information in structured data. For example a system to suggest corrections to products in a company catalogue (each product category has different schema). Unstructured data is pulled from various websites and optionally from categories retrieved from images. It is then compared against the data and most probable fixes are reported.
Most of the work is done by a few polite prompts to GPT3.5/4 (~5 English sentences in total)
1.2 Better search company data. E.g. a chat bot for internal documentation that can also access internal services in order to answer a question. Same ~5 English sentences to do bulk of the work.
1.3 (non commercial) Endangered language preservation. Building a smart agent that is accessible via chat/hardware (like Alexa/Homepod), that talks in native language can understand and helps to preserve the culture. This is a complex one.
2. Tech stack itself is rather simple. Mostly - GPT, LangChain/LlamaIndex, Vector database with embeddings for memory, plugins for external services and potentially agents to drive workflows.
Output quality, testing, monitoring, scalability etc also don't differ much from operating normal "old-school" ML models. If anything, it feels simpler.
The tricky part is that the entire notion of LLM-driven micro-services is new. Quality of the resulting product largely depends on knowing prompting tricks and following the latest news in an area.
Plus the biggest challenge that customers want to be solved: "How can I ran it on my hardware?"
- they may be limited in some cases - when you want to extract a deeper meaning from text - that is - gpt3/4 would figure out that an answer to a query is in a given document, but embeddings are way more shallow and will miss it
I know they are far better than previous search approaches, but do they have the limitations above?
My biggest challenge using the api so far was that the output is not reliable. Randomly from time to time it outputs notes and comments even tough I asked to only reply in a code block. Also if I rerun exactly the same promt, it can output something completely different (different content is fine, but I teach chatgpt to follow a structure - it works in 90% of the cases just fine). I’m using 3.5 not 4.
The api can be down regularly. This is annoying especially if you have longer conversations. I had a hard time to resume a conversation. I usually restart the whole process.
However, the overall capabilities are mind blowing. The system surprises me very often.
The system message is completely useless in 3.5 though and you’re better off putting your instructions in the first user message.
I haven’t used v4 in production yet though because it is so slow. I really hope they speed it up.
Challenge comes in pricing and getting a good result. Generally longer the prompt, better the results but you have to adjust accordingly.
Also, generally using only GPT-4 doesn’t make sense. Mix and match between 2 different models make sense. (e.g. data extraction can be done with GPT-3.5 but writing a good email should be done with GPT-4)
Is it a matter of everyone being in "don't think about it" mode and focusing on how the technology "enables" something and how happy your customers are? Something something engagement.
> Upload your leads into Nureply and instantly scrapes their personal public data to generate unique personalized first lines that you can use for your cold email icebreaker in seconds
Gross. I'm sure you're GDPR-compliant, yeah? And by GDPR-compliant, I don't just mean having a CYA paragraph in your privacy policy, but rather gathering informed consent from the "uploaded leads" about the processing of their personal data, and having THEM agree to the privacy policy, mmyea?
So, to reiterate my questions: How do you manage morale in a company that creates a not-only dubious/immoral but, in parts of the world, straight up illegal product?
Also, what is immoral? How are you so sure we are doing something immoral?
Employees are feeling great and more hyped than I am about the product. They see the value behind what we are doing. When a small business owner from a small country sends you a thank you message, we all want to work on our product more.
We are getting endless messages from small businesses around the world for showing their gratitude. Most of them have great product or service they want to sell and not good at copywriting. They don’t have enough money to hire someone to chase customers/clients but by using our product, they can finally sell their goods and start a business relationship.
We are not getting any contact information from the web. We are just checking their public social profile and website. Where in the GDPR or any similar law this is illegal?
Customer brings their own data. For example, one of my customers got this data from their embassy. They collected the data from companies who wants to buy specific type of material but couldn’t find someone to sell to them. They left their contact information and 100s of other companies followed this. As a result, my customer have 1000s of contact data, which are given with consent, waiting to be contacted.
This customer mentioned above just started their business and their product is awesome but have nobody to reach out those people. Our product fills this hole.
I understand your hate towards what you don’t understand but world is not about just EU or US. We are not breaking the law any means but if you think we do, I want you to prove it so we can fix it.
Oh dude, take the hint maybe.
> Where in the GDPR or any similar law this is illegal?
You're handling the personal information of, I'm guessing, ~4-6 magnitudes of people who haven't agreed to your privacy policy and data handling practices. You're a lawsuit waiting to happen.
Do you think more than 50%, or even more than 20%, of the contact details in that data want to be contacted in this manner?
It's stressful having to discard spam for at least an hour later on a Friday than in times past, when you would rather be with your family - I can't see a way this helps the world. I can assure you multiple side projects I run have never given consent to you or other similar services, yet are continuously spammed in this manner.
If you don't care about the receivers, think of the organic and legitimate senders that are not using such GPT-spam techniques. I now discard a higher proportion of people as false positives - perhaps local students keen to learn - who contact me genuinely, because of services like yours.
To truly reflect, imagine this scenario: Would you be up for going to a careers fair in Germany, entirely populated with young people, feeling sad, never having met or heard back a word of advice, because of your changes to human communication? People who never asked or heard of your US "tool".
PS What’s CYA?
> Consent isn’t required for B2B data processing and cold contact as far as I know, like it is for consumer data.
From what I can tell, the way this company ingests data uses personal channels and lookups. So although a company would be using their services and they are b2b, the targets of those cold emails are getting their personal data ingested by a company they have no knowledge of, and they are for sure not made aware of it in those cold emails.
If those targets are exclusively employees of a company, their professional email is being targeted, and only their professional profiles are being scraped, I would bet it's defensible. I am certain it's not the case though.
Do you guys have indie friendly pricing? I don't have $500/month to spend but could do $20/month.
2024 In a cafe
2025 In a posh supermarket
2026 In any supermarket
https://scrapestack.com/product
https://www.scrapingbee.com/#pricing
There's quite a few products in this segment around that price range.
How do you prevent this?
Parent commenter was talking about the latter. If you did the former, it will hallucinate like crazy, of course.
For those unaware DT produces a highlight-reel video after every software sales meeting.
Not sure if LLM will go on by default. The algorithmic version of DT is super strong, so just generating the scripts with GPT is MUCH worse.
For us the correct usage is to sprinkle in GPT, e.g. to also add a section to the output video which summarizes the user's goals
So far I've seen a ton of cool demos, but not much real life business use cases.
It mainly helps with 2 things:
- allowing engineers to develop their products much faster (especially doing good requirements engineering for now)
- allowing us to demo to/onboard users with data from their specific usecase (prepopulate their trial account)
Hardware engineering at first does not seem like an obvious choice for LLMs, but I think that it will be those vertical solutions that will still surprise us all the most.
Here are some more details, how hardware design gets concretely aided by LLMs: https://assistedeverything.substack.com/p/todays-ai-sucks-at...
https://www.intercom.com/ai-bot
Looks pretty cool from what I’ve seen
We were doing a workshop about LLMs for 110 people a week ago, and we spent 10 minutes explaining how you do stuff - as an example on how it should be done well.
2. One software engineering challenges is that ChatGPT often outputs code in markdown blocks. I've had to emphasize in prompts that it should explicitly mark the language. I then got inspired to make it possible to evaluate in place the code that appears in these blocks using a Jupyter kernel, and spent a week making that work (so, e.g., if you type a question into the chatgpt box on the landing page at https://cocalc.com, and code appears in the output, often you can just evaluate it right there). There seem to be endless surprises and challenges though. For example, a few minutes ago I realized that sometimes the giant tracebacks one gets when using Python in Jupyter notebooks are so big (even doing simple things with matplotlib) that they end up resulting in too much truncation: https://github.com/sagemathinc/cocalc/issues/6634
3. I'm mostly using GPT-3.5-turbo rather than GPT4, even though I have a GPT4 api key. Aside from costs, GPT4 takes about 4x as long, which often just feels too long for my use case. The average time for a complete response from GPT-3.5 for my application is about 8 seconds, versus over 30s for GPT4.
How did your software engineering workflow differ when using GPT4 (as compared standard software engineering workflows)?
How did you test the code that makes heavy use of generation?
We are ctm.app
Even though it's much slower, GPT 4 is way more consistent than 3.5. The OpenAI APIs have had lot of flakiness in the past couple of weeks, we retry requests up to 10 times to work around this