Hacker News Reader

Comment by alexpotato | Hacker News Reader

I have always loved SQLite.

I have also heard that some firms ban its use.

Why?

Because it makes it SO easy to set up a database for your app that you end up with a super critical component of your application that looks exactly like a file. A file that can have any extension. And that file can be copied around to other servers. Even if there is PII in that file. Multiply this times the number of applications in your firm and you can see how this could get a little nuts.

DevOps and DBA teams would prefer that the database be a big, heavy iron thing that is very obviously a database server. And when you connect to it, that's also very obvious etc etc.

I still love SQLite though.

View on HN

The question is, do the same firms ban Excel? Excel spreadsheets often end up as shadow databases in unlikely places.

This might catch flak, but generalizing I would assume that the people banning things are the same people who would use excel for something where a database would be better, and if so, that is the reason Excel isn't banned on the same conditionals that would get sqlite banned.

The sane thing would be to ban Excel and promote SQLite. Excel is often used for tabulated text (issue tracking) not calculations. Perfect use case for a relational db

Excel is made for calculations. But if you make it hard to make a DB, people will abuse Excel as a DB.

I mean, it might have been at first, but Microsoft figured out that the majority of users for lists without formulas in 1993 and they've strategized around that. IMHO, the biggest concession to this was when they added Power Query to core Excel in 2016.

Excel has sheets for tables, columns and rows, primary keys (UNIQUE), foreign key references etc if you squint.

It doesn't require you use all of that properly, but it's there.

or reimplement excel with sqlite as a backend :-D

BTW sqlite can run SQL queries on CSV files with relatively simple one-liner command...

Well heck can't someone make an SQLite extension that is basically just a simplified Excel ?

and excel has gui for forms

Only where VBA is available. Not available for MacOs versions if I'm correct?

VBA is just there for backward compatibility.

The modern alternative is to use JavaScript/TypeScript, which makes such solutions cross platform (including MacOs, web etc.):

https://learn.microsoft.com/en-us/office/dev/add-ins/overvie...

IMO, almost any Excel more than a month old should become readonly.

You should consider knock-on effects of this brilliant idea. Now there would be copies of spreadsheets younger than a month that get replicated 47 billion times, exponentially compounding the problem you're trying to solve.

This sounds like how we pass so many stupid laws. Nobody thinks about 2nd order effects.

So you're saying they should further auto-delete after two or three months?

3rd order effect, people copy and paste the old sheet into a new sheet, now we have worse exponential. You’re not very good at this huh.

Which is very annoying and people will complain. People complaining can be then directed towards a better solution. As a bonus, mistakes will also rise, leading to further complaints, especially ones that reach higher. All this making the dogshit practice, and the idiots committing them, infinitely more visible and thus fixable.

The sheer volume of data that needs tending to may even grind certain departments to a halt! What a great opportunity! It'd appear I'm positively stellar at this!

Sorry for the snark, that was shitty of me.

No worries, was a bit of a gamble of a joke from me (sarcasm frequently doesn't translate in text, or can be inopportune), so I tried taking it accordingly.

For clarity, while I did have some rather perverse fun toying with the idea, I do not actually think it should be implemented, or at least certainly not in one fell swoop and as-is. Mostly for the aforelisted reasons. This is what I actually intended to convey.

Though for better or for worse, that doesn't mean I think the notion is completely meritless either, so I might still be deserving of at least some of your snark. But a lot of it was in jest from my side indeed.

This whole keeping an inventory, disabling items, and later taking them out completely is a chore I already do in other contexts. While it does work, it is anxiety inducing, doesn't really scale well, and it's quite miserable to go through with. It's the cost of keeping things organized as far as I'm concerned, with no real way around it in the general case. The best one can do is try and mitigate it, by monitoring for patterns and building out systems, to automate and streamline the tedium away.

I do sincerely not know of other ways to keep things in check though, in lieu of which you do get the makeshift shadow ops with all of its pitfalls. It's kind of just life.

Doesn't this happen anyway with "final2.really.final" ?

PII sniffers are pretty good at dealing with excel files. Excel is seen more as an analyst tool than a dev tool. Any place that bans Excel needs to either let analysts use some other turing complete data tools, like python or R or something, or they'll have trouble attracting analyst talent. They'll have devs and data entry users and that's it.

The only way that works is if the dev team is large enough to be responsive to business needs, which almost never happens because devs are expensive. The juniors who are tweaking business logic every day are functionally doing a role analysts can do if you just give them a sane API and data tools.

They generally cannot. But they do banish Access.

Now that is different.

Access gets used for a shared DB and that is quite easy to corrupt. It is much more cost effective to have that in a proper central database (I supse SQLLite is better here as well)

Excel is also a shared DB: it has supported multiple concurrent users accessing and modifying the same spreadsheet for decades.

You can enforce classification and privacy labels (or something similar) in Excel and other document files, at least in a closed corporate environment. Azure also supports this. Also, everyone has Office installed (in a corporate environment), anyone can open and work with an Excel file.

I don't have Office installed, nor do a significant majority of my peers. Given that sqlite is installed by default on Macs, a sqlite file is far more portable than an Excel file.

I’ve worked at some organisations that have strict rules (not always strictly followed) about what can go in Excel spreadsheets, and where they have to be stored. The C drive is verboten. Some also have standards about classification and labelling of PII and sensitive data.

Don't get me started on Access...

Man, Access could've been so good if they just made an app around SQLite. Or since it's Microsoft and they need to do everything their own way, it would've been so good if they made a flat file DB à la SQLite, but with T-SQL (or a subset thereof) instead of JET-SQL.

Increase interoperability. Funnel data people from Excel into real DB technologies.

And if they did more to blur the lines between spreadsheets and databases, and make it seamless to work out of both Excel and Access, add more spreadsheet features to the data views, etc.

Do companies ban text files? Text files are used to store data.

That's why you store them on unsaved tabs instead.

Do companies ban data centers? It's crazy to send PII to other computers on the line.

Do companies ban brains? Brains are used to store data.

Brains do not exhibit ACID properties…

There are interesting uses for sqlite, like this one: https://sqlite.org/sqlar.html

Some firms don't understand how to do data management, and if we draw the venn diagram of those and the ones that ban sqlite, it'd be pretty close to a circle.

Yes, databases could have any extension. No sane dev team would accept code that doesn't use an object extension for a sqlite database.

Yes, databases can contain PII but no sane product manager would go "yes, that's a good use of sqlite".

Yes, you can trivially copy database files, but no sane product needs to in the same way that no sane product should require folks to just clone the db just to do some work.

Pretty much every reason a company has for banning sqlite is a red flag for working there.

Required reading for “anything can become a mission critical database” conversations:

https://www.reddit.com/r/sysadmin/comments/eaphr8/a_dropbox_...

>DevOps and DBA teams would prefer that the database be a big, heavy iron thing that is very obviously a database server. And when you connect to it, that's also very obvious etc etc.

If I was their CTO and they told me this, and it is not a joke, I'd fire them on the spot.

> a file that can have any extension

So read the magic number, you shouldn't trust file extensions anyway

> that file can be copied around to other servers

So can spreadsheets

I'm not discounting that having centralized data access is desirable but it doesn't sound like that particular reasoning is well thought out

This "shadow IT DBA" issue has always been a classic problem with Access databases, too.

A production app with customer data needs a data backup/restore strategy. I'm guessing a random app server writing to a local sqlite file isn't doing that either.

This is why I put configs like that into AppData or dotfile directories, or the equivalent for MacOS (I forget which one it is inside of the ~/Library directory).

I recently watched a YT video about this subject: https://www.youtube.com/watch?v=lSVgeMoXJTs

In summary, companies use the bus-metric to see how viable a project is. Bus, as in, how many people can be hit by a bus before there is no one left to maintain the project.

Despite its ubiquity, SQLite is maintained by only 3 people. That bus-metric for SQLite is 3, which is way too low for some companies.

Give the link a watch; it was really interesting.

At least with SQLite, it is really stable so if development did cease, you'd probably be fine indefinitely.

and anyone that considers this to be the case for sqlite, should probably have their reasoning skills examined.

if the unfortunate bus incident happens to sqlite developers, there is exactly ZERO chance that it will not be very well maintained on the count of all the users, many of whom already have support contracts going for decades, and which would require the same level of support they have already enjoyed.

DevOPs and DBAs must hate RAM and caches. We

That's so dumb

> DevOps and DBA teams

Ah so two teams nobody should listen to.

At least would take it with a grain of salt when the DBA wants you to depend more on the DBA.

Same with devops tbh.

"Hey everyone, we need to chose the option that involves us the most and provides us the most job security"

Well... eventually the company learns the lesson the hard way, either because a site goes down or gets 0wned. Then everyone will cry about "how this could happen", and the ops people will tell you in response "we warned you that this would happen, here's the receipts, now GTFO".

Preparing to say “I told you so” is a fairly obvious incentive for someone to act like Chicken Little. And of course sometimes they are right, but not always.

Lots of great people in both devops and security. But when teams position themselves as the conscience of the org and the gatekeepers of production, the defensive victim mentality can get pretty strong.

Reply on news.ycombinator.com