We are really excited to open source YoBulk today.
YoBulk is an open source CSV importer for any SaaS application - It's a free alternative to https://flatfile.com/
Why are we building YoBulk:
In our previous startup, we were receiving CSV files from various billboard screen owners every day, following a specific template that we defined. Despite the well-defined template, the CSV files we received often contained manual errors, which was a challenge to fix with the data provider.
We were receiving around 500,000 billboard data updates each day, including price changes and creative info data. It was a difficult and time-consuming job to clean and format the data to fit our database schema and upload it into the database. As a result, we wanted to automate the entire CSV importing process. In our second startup, we encountered similar challenges when cleaning large CSV files with location and timestamp data.
We realised that more than 70% of business data is shared in CSV and Excel formats, and only a small percentage use API integrations for data exchange. As developers and product managers, we have experienced the difficulties of building a scalable CSV importer, and we know that many others face the same challenges. Our goal is to solve this problem by taking an open source AI and developer-centric approach
Who can use YoBulk:
YoBulk is a highly beneficial tool for a variety of professionals, such as Developers, Product Managers, Customer Success teams, and Marketers. It simplifies the process of onboarding and verifying customer data, making it an indispensable asset for those who deal with frequent CSV data uploads to a system with a predetermined schema or template.
This tool is particularly valuable for updating sales CRM or product catalog data, and it effectively solves the initial challenge of customer data ingestion.
The Problem:
Importing a CSV is a really hard problem to solve. Some of the key problems are:
1.Missing of Collaboration and Automation in CSV importing workflow:
In a usual situation, the customer success team responsible for receiving CSV data has to engage in extensive back-and-forth communication with the customer to address unintentional manual errors present in a CSV. This process requires a high level of collaboration and may even necessitate assistance from the customer's internal teams to correct the data. The entire workflow is currently manual and therefore needs to be automated.Being able to quickly see data errors and fix them on the spot in a collaborative way with the customer is the way forward.
2.Scale:CRM CSV files can sometimes reach sizes as large as 4 GB, making it nearly impossible to open them on a standalone machine for data correction. This presents a significant challenge for small businesses who cannot afford to invest in big data technologies such as EMR, Databrick, and ETL tools to address CSV import scaling problems.
3.Countless complex validation Types ::A single date format can have as many as 100 different variations, such as dd-mm-yyyy, mm-dd-yyyy, and dd.mm.yyyy. Manually setting validation rules for each of these formats is almost impossible, and correcting errors manually can also be difficult. Additionally, it can be challenging to comprehend errors without a human touch. Cross-validation between fields/columns is always a challenge in a specific CSV. For example, if a CSV contains two fields such as first name and age, creating custom validation to flag an error if the first name is missing and the age is greater than 50 can be really difficult.
4.Data mapping issues:In a typical scenario, the recipient of CSV data provides a template to the data donor and creates a CSV column to template mapping before importing. However, in many cases, the CSV column names do not match the corresponding template column names. For instance, the data receiver may provide a field labeled "EMP date of Joining," but the uploaded CSV may contain a field labeled "EMP DOJ." These mapping issues can significantly slow down the CSV importing process.
5.Data Security and Privacy:It is always risky to share your customer data with third-party companies for data cleaning purposes.
6.Non-availability of low code/No code tool: Product managers and customer success teams, who are typically no-code users, often rely on data analysts to create a programmed CSV template with validation rules, which must be shared with customers to receive CSV data in a specific format. However, in an ideal scenario, no-code users should be able to create a template independently, without depending on developers.
7.Vague error messages:Unclear error messages do not provide users with enough context to confidently resolve their issues before uploading their data. Without a specific explanation of the problem, users may have to try various fixes until they find one that works.Example:while uploading a CSV file to a portal, I had received an error like “baseID is null”.i was clueless:)
The Solution:
1. Smart Spreadsheet View: Designed to be a data exchange hub for any business that utilizes CSV files, YoBulk makes it easy to import and transform any CSV into a smart spreadsheet interface. This user-friendly interface highlights errors in a clear, concise manner, simplifying the task of cleaning data.
2. Bring your validation function: YoBulk offers a platform for Developers to create a custom CSV importer that includes personalized validation rules based on JSON schema. With this functionality, developers can design an importer that meets their specific needs and preferences.
3. AI first : YoBulk harnesses the power of OpenAI to provide advanced column matching, data cleaning and JSON schema generation features.
4. Build for Scale: YoBulk is designed for large-scale CSV validation, with the ability to process files in the gigabyte range without any glitches or errors.
5. Embeddable: Take advantage of YoBulk's customizable import button feature, which can be embedded on any SaaS or App. This allows you to receive CSV data in the exact format you require, streamlining your workflows.
Hosting and Deployment:
YoBulk can be self hosted and currently running on Mongo.
Github : git clone git@github.com:yobulkdev/yobulkdev.git
Getting started is really simple :
Please refer https://doc.yobulk.dev/GetStarted/Installation
Docker command: git clone https://github.com/yobulkdev/yobulkdev.git cd yobulkdev docker-compose up -d Or docker run --rm -it -p 5050:5050/tcp yobulk/yobulk Or git clone https://github.com/yobulkdev/yobulkdev cd yobulkdev yarn install yarn run dev
Also please join our community at :
- Github : https://github.com/yobulkdev/yobulkdev - Slack : https://join.slack.com/t/yobulkdev/signup. - Twitter : https://twitter.com/YoBulkDev - Reditt : https://reddit.com/r/YoBulk
Would love to hear your feedback & how we can make this better.
Thank you,
Team YoBulk