r_thambapillai

its extremely hard to tell what actually changed here

There's a really interesting analogy here with the LLM platforms that both are a vendor to companies that use their APIs, but also potentially compete with those companies as well as they evolve their offering over time. It would be interesting to see if this kind of regulation could be applied in that context as well.

this seems exactly backwards. AI will make good software engineers even more productive, and of course like any new technology those who learn to use it best will see more lift than those who cannot, but I suspect we should expect to see that the better the software engineer, the better they will be at learning how to use AI

hi! reach out to me @ ravin@credal.ai would love to hear from you! And yup, that's one of our verticals :)

Credal.ai | Full Stack | New York ONSITE

Credal is a security first data platform for enterprises building AI Apps. We have point & click connectors to your data sources (like GDrive, Slack, Confluence, O365 etc), but unlike other offerings we automatically enforce the permissions you setup in the source systems, mask any sensitive data like people's SSNs, audit log every interaction, and automatically enforce company acceptable use policies for AI tools like ChatGPT/Claude/LLama etc.

We serve enterprises from 100 person startups to Fortune 20 behemoths, and our team is comprised entirely of people who do truly exceptional things. You'll get to work at the cutting edge of making AI actually work in the real, messy but extremely impactful world of our customers: healthcare, sustainability, industrial robotics, finance and more.

It sucks, but the only thing to do is kick ass in the field. The more successful you are, the more companies will copy you. We had companies with 30 or even 100 people hard pivot into an exact clone of our company when it was clear we had traction. We just have to keep out executing

Python ended up 'specializing' in data contexts, thanks to Numpy / Pandas, and as a result, ended up becoming the first exposure to programming than anyone doing data stuff had. That was millions of people. In that space, it had no competitors.

Ruby ended up 'specializing' in web dev, because of Rails. But when Node and React came out, Ruby on Rails had to compete with Nodejs + React / MERN as a way of building a web app. Since people first learning programming to build a web app would usually start with javascript anyway (since a lot of very first projects might not even need a backend), it was a lot easier for the Nodejs/React route to become the default path. Whereas if you were a data scientist, you started on python, and as you got better, you basically just kept using python.

I think you have every single one of these risks exactly backwards:

Potential for Data Breaches: - The idea that third parties have inherently higher breach risk is pure FUD.

Data Sovereignty and Compliance Risks/Limited Overside: - Having every single company solve all of these problems for every single part of their software stack is obviously very inefficient. That's why we have SOC II, Pen Testing, etc, as other commenters point out. The process of reviewing vendor SOC II reports is way, way easier than having to manage the cost of managing compliance for the hundreds of little productivity apps your team want to use!

Shared Responsibility Model: I think its pretty typical at this point for companies to take ownership over the behaviour of their sub-processors from a risk perspective.

Thanks for the feedback! Going to make some changes to the website to reflect that later today! Right now we support connecting Google Doc, Google Sheet, PDFs from Google Drive, Slack channel, or Confluence space. O365, Notion and a couple other sources integrations are in beta. We don't technically have restrictions on volume, the biggest customers we have have around 100 GB of data with us total. If you were trying to connect a terrabyte worth of data, that might be a conversation about pricing! :)

There are great tools that do this already in a support-multiple-ecosystems kind of way! I'm actually the CEO of one of those tools: Credal.ai - which lets you point-and-click connect accounts like O365, Google Workspace, Slack, Confluence, e.t.c, and then you can use OpenAI, Anthropic etc to chat/slack/teams/build apps drawing on that contextual knowledge: all in a SOC 2 compliant way. It does use a Retrieval-Augmented-Generation approach (rather than fine tuning), but the core reason for that is just that this tends to actually offer better results for end users than fine tuning on the corpus of documents anyway! Link: https://www.credal.ai/

Got it, thanks! For context, Im at Credal.ai (an LLM governance platform), and we have a shared customer who I'm pretty sure does fall into that description with you (use Cody for all their devs), and keeps asking us about whether they can use Cody in conjunction with Credal.ai (bring your own LLM account is probably all that would be needed to support this). We'd also be open to contributing this for what its worth!

Is it possible to use our own LLM account if we have anthropic or open AI accounts?

How are you thinking about preventing customer PII making it to OpenAI?

We use both a lot. Their both great, they have different strengths and weaknesses. Right now, my default go to in GPT 4, but Claude for stuff where I need an answer to more reliably not be a hallucination

Its super unlikely to be a direct bribe, I think that much is fair to say. But I think viewed differently, the article is saying: Sunak's family has a tremendous interest in BP and Shell needing consulting services. Might this have an indirect effect on Sunak's willingness to legislate in ways that are highly likely to increase the amount of consulting services BP and Shell need?

Its not bribery, by any means, but I think its definitely noteworthy that these pressures / considerations may be involved

why is it racist to worry about a somewhat adversarial authoritarian nation state having the capability to disrupt life in the U.K.? I also cannot read the whole article, but you'll note that the article isn't about Taiwanese cars, or for that matter Japanese, Korean, Vietnamese cars. Its specifically about cars from a nation state with an arguably adversarial security relationship with the UK

Thanks for your encouragement and that is a totally fair criticism - when we say that, we mean two things:

1. We support using Credal with your own, open source LLM, which can of course be fully on prem in every sense

2. We also support using Credal with your own Azure OpenAI instance. As you say, OpenAI aren't giving us the weights, but many of our customers have procured Azure OpenAI from Microsoft and then we point GPT 4 usage at their Azure instance, meaning that the data never goes to Open AI at all.

One of the things that's going to be really interesting to see moving forward is whether the open source models are going to be able to compete with the blistering pace and funding that the closed source ones - Bard, Claude, and GPT-X are going to be able to attract (and maybe Mistral?). For the sake of the industry, I really hope that the OS models catch up but given the amount of funding (and now in OpenAI's case, revenue) the closed source models are generating, its hard to see how that happens

Stay tuned!! Right now no, but Foundry is definitely under consideration

Thanks so much! So about half of our enterprise customers use Credal in conjunction with either a self hosted LLM or Azure OpenAI (which you can debate, but most companies we've spoken to seem to treat their Azure OpenAI instance as equivalent to self hosted). In practice, you still need to:

1. Manage the permissions of making sure the self hosted LLM is only reading from the documents, slack channels, e.t.c. that the end user should actually have access to

2. Generate an audit log of exactly who did what when

So we actually see self hosted LLMs being a big part of how Credal is used! In the long term, we think Credal will actually become a tool for AI app developers to safely request access to data & embeddings from the enterprise on the fly, and make sure the data they get is appropriately controlled and the audit logs exist in a single place for the enterprise to see what data went to whom/when/why etc

Great questions and thanks for trying the product!!

Yup so a few thoughts here - we're exploring using embeddings to allow a description of what you want to hide, that will actually immediately show you what of the data you have synced already would be caught by that (or which previous requests).

On the API side: yes ABSOLUTELY! the API is already live and used intensely by some of our startup customers like Sourceful. The API docs for using the OpenAI models here: https://credalai.notion.site/OpenAI-Drop-in-API-0ef7cfd18a7c...

and the Anthropic models here: https://credalai.notion.site/Anthropic-Drop-In-API-ad298f6f7...

Thanks! :) It feels so surreal to be launching on Hacker News! When I was first discovering tech the people launching YC funded startups on HN seemed like wisened old Gods to me. Now I laugh about it because obviously I'm still learning so much, even the basics, every day. I hope we get to inspire someone else the way the early YC cos inspired me

Our AWS migration wound up taking about 4 weeks, getting all the policies in place took about 8 weeks (which overlapped with about 2 weeks of the migration), and then the audit itself was a couple weeks as well

Nice! Which AI model are you using for it? If you're using ChatGPT, you can actually use our ChatGPT API and get the PII redaction for free, with hopefully hardly any code changes

Thanks!! There are some fairly good OS models for the core stuff (PII, SSNs etc) out there already (Presidio, Spacey), so folks that need an OS option have one to start with. Detecting the more complex stuff can sometimes need a little iteration, but I could definitely imagine a world where we publish that in the future

On SOC 2, we used Drata, and spoke to Vanta, Laika and a few others. The price Vanta initially quoted us was waaaay higher than the other two, and between Laika and Drata we went with Drata mostly because there seemed to be more automation in Drata. In the end, the Drata live support was incredible and hard to imagine how we would have gotten the certification so fast without. We started our infra on DO, and so the most painful part of SOC 2 for us was the migration we did to AWS to take advantage of AWS' many security features. My main advice would be take full use of the Drata live support (I'd guess Vanta have something similar), but maybe on a deeper level - when you're doing SOC 2, don't focus on the certification: focus on the policies and technology that actually makes your company secure. In the end, that's what enterprises really care about, especially for the ones that have given us 300 question long questionnaires!

Tbf, the OP explicitly said "Preferably self-hosted" which is why I wanted to offer this along with the explicit disclaimer that I am the founder

if you're using it for business purposes you can use credal.ai (disclaimer: I am the founder)

there are many services that integrate with them that would allow you to self-serve signup

Breakdown of real life production usecases for LLMs at large enterprises today

Our biggest and most important lessons from YC

Totally agree, but these tools do provide real productivity boons! Full Disclosure: I am a founder of Credal.ai for just this reason, our mission is to help you get the productivity boosts of AI without trading off your infosecurity):

One thing I'm curious about is what you think of the recent OpenAI announcement about not training models on data submitted via OpenAI?

http://credal.ai/