324 karma · joined December 7, 2014
Sandboxing isn't about having the perfect devex environment, most dev sandboxes (looking at codex cloud) don't actually have full verification available. Sometimes Github actions CICD is all you can get!
Use "instruction @web @Flask" to search the internet and Flask docs before answering, for example.
(1) The "baseline" comparison was to PyPDF + Naive RAG. For the LlamaParse evaluation, you appear to have used a different RAG pipeline, called "recursive retrieval." Why not use the same pipeline to demonstrate the improvement from LlamaParse? Can you share the code to your evaluation for LlamaParse?
(2) I ran the benchmark for the PyPDF + Naive RAG solution, directly copying the code on the linked LlamaIndex repo [1]
I got very different numbers: mean_correctness_score 3.941 mean_relevancy_score 0.826 mean_faithfulness_score 0.980
You reported: mean_correctness_score 3.874 mean_relevancy_score 0.844 mean_faithfulness_score 0.667
Notably, the faithfulness score I measured for the baseline solution was actually higher than that reported for your proprietary LlamaParse based solution.
[1] https://github.com/run-llama/llama-hub/tree/main/llama_hub/l...
If you're interested in cartography, Facebook created the Daylight Map Distribution [1] project. They analyzed satellite imagery, outlined buildings and roads, and published them for map makers. It significantly improves coverage in rural areas and non-Western countries.
Overture's sponsors seem pretty focused around big company needs, like coordinating efforts between mapping orgs and standardizing metadata. They're each pledging $3M + 20 FTEs per year to accomplish it [1].
But I would argue it's more productive to coordinate enterprise+individual contributions through a central organization: the OSM foundation.
When io.js forked from Node.js in 2014, it was more of a call for attention than a desire to split the ecosystem in two. They merged the next year. (I joined the core dev team after the split.)
Maybe instead of asking what OSM will become after Overture, we should ask, why can't enterprises accomplish their goals within the framework of the OSM Foundation?
Disclaimer: my startup has a product built on OSM [2].
[1] https://overturemaps.org/become-a-member/
[2] https://buntinglabs.com/solutions/openstreetmap-extracts
[1] https://constructionphysics.substack.com/p/another-day-in-ka...
Some people simply don't understand that C is actually everywhere around them. They don't understand that either the language they use is built in C, or the language THAT was built on was built in / on C, etc etc. Simply the importance of the language in history may just be a powerful point on its own. If someone knows quite a few languages and doesn't yet know C, I kind of don't understand why - they do not yet understand the importance of lower level operations, like the functioning of a garbage collector or why their pre-mature optimizations shouldn't even be attempted.
Great article, over all - I really liked it.