mattewong

> One gets the sense that [the authors] think the consultants they write about are rotten apples, but [they are wrong]

OK I paraphrased the rest of the sentence into the last 3 words but if you bothered to read to the end, you see that the reviewer completely lost her way. Purporting to review a book, she concluded by the end that its authors are misguided because the real culprit is not the consultants, but rather, every pursuit of efficiency, including all "software development".

> software development [is] in fact focused on enabling capitalists to enrich themselves further without the inconvenient interference of workers, taxpayers or regulation

> to beat 99% of its long-short equity fund peers this year

Titling an article on a fund's relative performance from a mere 11 months is a pretty amateur click-bait move. For obvious reasons, no fund would ever be legally allowed to put that in marketing materials (without a ton of up-front in-your-face disclaimers and caveats most likely also requiring disclosure of the same figure over longer time horizons); it's unfortunate that Bberg, whose journalism imho is generally pretty good, allowed this to be printed in this manner. Not saying the fund doesn't deserve the credit that the title suggests, but only that the evidence presented in the article is very insufficient to support that suggestion.

Thank you, had not heard of him and found some useful info in various places including: https://usesthis.com/interviews/andreas.kling/

interesting, had no idea of those limitations / considerations in competitive programming

Great resource, thank you!

That's another one I hadn't seen before. thanks!

From what I gather, Zuckerberg used emacs, so unless that information is wrong, then he would not be a responsive answer to this question. Thanks anyway

Had not seen that before thank you!

Many reasons it is true, including i) for anyone with the talent to do so, the "rational" thing to do, generally, is not to become an entrepreneur given the risk/reward/effort, and ii) if they are not contrarian, then the ideas they pursue are the ones that everyone else generally agrees with, and therefore the opportunity is small. Back in the day, only contrarians thought that online shopping could amount to meaningful dollars. Only contrarians thought that Amazon could take on Barnes & Noble. Only contrarians thought that licensing an operating system for a few bucks per PC-- on every single PC out there-- was a good business strategy and could ultimately grant leverage to a tiny company called Microsoft over a behemoth called IBM. Hindsight makes these "contrarian" views look obviously right, but back then, most people did not believe them.

Thank you, my bad. I thought I had read the guidelines but apparently missed that part. Apparently I can no longer change the title-- otherwise I would now-- but will to properly do so going forward.

Is it discouraged here to title the submission based on a statement within the linked article rather than its actual title?

I ask in all seriousness-- I'm still kind of new to this site.

The point that I think is worth emphasizing is not their funding but rather their assertion / premise that "The fact is, ‘Big Data’ is dead". Obviously an overstatement but the general point is: companies are being supported by reputable backers on the premise that Big Data is overhyped and other parts of the data eco-system are underdeveloped.

I considered giving it the same title as the article, and then just mentioning the "Big Data is dead" point in the first comment, but that seemed kind of roundabout and distracting to the point that I think is worth paying more attention to.

Happy to hear further suggestions given that background

> The fact is, ‘Big Data’ is dead [...] Cloud data vendors are focused on performance of 100TB queries, which is [...] irrelevant for the vast majority of users

>around 70% of our serious security bugs are memory safety problems

Is there any information available to assess what portion of that 70% was, or could have been, identified through static code analysis, sanitizers and other memory-safety techniques that can be applied in a bulk / automated manner?

"The hubris is that PHBs and spreadsheet jockeys think they understand my job better than I do": where are you imagining these things from? Who ever said they understand YOUR job better than you? I think it's safe to say, the article was probably not written with you personally in mind. And I think it's also safe to say, you personally are not the same as every engineer out there.

Someone simply suggested that VC-backed companies (presumably, often of the startup kind), the company can achieve more economic success when engineers spend some time with customers (and let's not forget that the customer need is the SOLE reason that the engineer's job exists at all).

The hubris is in the failure to acknowledge that you do not represent the entire world of engineers, and the unwillingness to contemplate a scenario where just maybe, an engineer could be better at their job, if they took more time to understand that which created its reason to exist.

imho these comments demonstrate an insane level of hubris that is difficult to find in any truly talented engineer

TL/DR: xsv is probably what you want, or maybe zsv and/or awk

awk can do this super easily. Here's an example snippet that not only shards, but compresses your shards. ``` (NR - 1) % shard_size == 0 { # ready to start a new shard current_n = current_n + 1 output_file = sprintf("%s%04d.%s.bz2", target_prefix, current_n, file_type) print "writing to " output_file > "/dev/stderr"

    # close any prior-opened output_command (else will err on too many open files)
    if(output_command != "")
        close(output_command)
    output_command = "bzip2 > " output_file

    # print header
    print headrow | output_command

}

NR != 1 { print $0 | output_command }

```

This of course assumes that each line is a single record, so you'll need some preprocessing if your CSV might contain embedded line-ends. For the preprocessing, you can use something like the `2tsv` or `select -e ...` command of https://github.com/liquidaty/zsv (disclaimer: I'm its author) to ensure each record is a single line.

You can also use something like `xsv split` (see https://lib.rs/crates/xsv) which frankly is probably your best option as of today (though zsv will be getting its own shard command soon)

A bit weird that suddenly there a lot of sentiment around "how [MDS company here] fails" when 12 months ago it was "MDS is the only future"...

I'm having a hard time clearly seeing what the problems were with other approaches that ASDF is intended to solve.

Are the other solutions not fast enough? If so what is the performance delta using ASDF? Are the other solutions not scalable enough? Likely not, given that ASDF mentions certain scale limitations-- but maybe those limitations are still less limiting than limitations of other solutions?

Whatever the reasons for making ADFS, it would be helpful to describe in the intro "here is a problem that ADFS solves" and "here is a graph (or similar) showing how it solves it better than alternatives".

Without that context, the description of Features, to my naive outsider's view, is not very valuable and I'm left still asking, "why use this instead of other solutions that many engineers already are familiar with?"

another shameless plug: https://github.com/liquidaty/zsv

Ah, got it, thank you!

Just one minor suggestions/feedback point, in case you find helpful, which is that I had to also add the `-d` flag with a comma value. Otherwise with just -H, I get the error "Bad header row" even though my header was simply "Country,City,AccentCity,Region,Population,Latitude,Longitude".

This suggests to me that `q` is not assuming the input to be a CSV file, but that seems at odds with the first example in the manual, which is `q "select * from myfile.csv"`, with no `-d` flag. Or perhaps the first example also isn't using a csv delimiter, but it doesn't matter because no specific column is being selected?

In addition, given that, from what I gather, a significant convenience of `q` is its auto-detection, then I think it would make sense for it to notice when the input table name ends in ".csv" and based on that, to assume a comma delimiter.

Just my 2 cents. Great job!

Nice work! I am a fan of tools like this.

However, in my first attempted query (version 3.1.6 on MacOS), I ran into significant performance limitations and more importantly, it did not give correct output.

In particular, running on a narrow table with 1mm rows (the same one used in the xsv examples) using the command "select country, count(1) from worldcitiespop_mil.csv group by country" takes 12 seconds just to get an incorrect error 'no such column: country'.

using sqlite3, it takes two seconds or so to load, and less than a second to run, and gives me the correct result.

Using https://github.com/liquidaty/zsv (disclaimer, I'm one of its authors), I get the correct results in 0.95 seconds with the one-liner `zsv sql 'select country, count(1) from data group by country' worldcitiespop_mil.csv`.

Second the support for vectorscan. And FWIW, I have posted issues in both hyperscan and vectorscan repos, and the only responses came from vectorscan (and quite quickly).

All this banter arguing over CSV, JSON, sqlite seems unnecessary when you can just push format X through a pipe and get whichever format Y you want back out: https://github.com/liquidaty/zsv/blob/main/docs/csv_json_sql...

(disclaimer: I'm one of the zsv authors)

There is only upside to using foreign keys. When enabled, the benefits are obvious, so I won't discuss, and as for the drawbacks, the only drawback is performance. But if that's a concern, then guess what, on most databases, you can disable foreign keys.

Well, one might ask, what is the point of having foreign keys if they are disabled? And the answer is, there are several benefits. Here are a couple:

1. foreign keys, disabled or not, create a record of your data design that itself serves as documentation, and that can be programmatically queried, extracted, copied, modified etc, all of which reduces technical debt and is useful for other users or developers to understand your schema and work more efficiently.

2. You still have the option to enable the foreign keys. Furthermore, you can pass this job over to someone in your organization who may not have authority to create or modify foreign keys, but does have the authority to enable them and/or to fix the data however required to enable them

You might have seen my post for https://hub.liquidaty.com/playground/ (make sure to include the trailing slash), which I posted a couple months ago with temporary public access (I am one of its founders). I'm not very good at the art of hackernews posts so not surprised if it got lost.

Anyway, from that site, you can:

- upload CSV, TXT, XLSX in local browser - run SQL queries on one or more files - run validation reports (like, real validation reports, with drill-down, breakouts, etc, not the BS ones that most products tout but are just an SQL query with a where clause) - convert data to JSON, sqlite3, etc - create and share your own domains that allow collaboration and reuse of common business processes, and also enable trainable smart mapping to whatever dictionary you define (e.g. map "Name, First" to "First Name"), automated rescaling (e.g. detect if percentages are x100 and automatically divide by 100 if so), automated derivations (e.g. A + B = C, my validation tests C. System figures out how to get C, regardless of whether data contains C, or only contains A and B). Automatically convert your data into schema Y. Etc.

It is now in private beta but from the website, you'll see a link to request access (by emailing support@liquidaty.com) and we'll give you a free beta account (you must have a legit linked-in account for federated login). Or feel free to send regular questions to that email address

Good luck. For me it was good to get at least a few yrs of tech experience before pivoting. it definitely was helpful later, some for the technical experience and maybe more for understanding the dynamics of how decisions get made by technical teams (including how politics comes into play). When you are part of the "business" that is the "customer" of the technical team, that experience can give you huge value that your peers may not be able to offer.

Glad to hear it was helpful!

Yes. I was a CS major. My resume wasn't too different from yours. I got tired of being narrowly focused. Went back to b-school for various reasons, among them, to get a "clean slate" in terms of job opportunities-- ie to avoid being pigeon-holed. It worked, and it was expensive, but the jobs on the other side made it worthwhile. So my journey was: tech, business school, wall street.

From what I can tell there are many opportunities if you are "tech + X", where X is real experience in a (non-tech) industry. In my case it was finance, and once I had a few yrs experience with that, I had lots of choices because every firm wanted someone who both understood their business and how to use technology to supercharge it. Funny enough, 15 years later, I decided to do something different again... and went back to tech.

Why would "no-code" be worse for the part where a non-admin user is just assembling together some actions and triggers? Will check out Temporal. Thank you!

Having sort of done this myself (self-funded for 18 months, then raised some friends-n-family and now cashflow positive), I would recommend:

- stay with your job (like others have said) as long as you can. I probably could have done this longer.

- control your cash burn, not just current but also the time horizon of the commitment. I have found that contract-based work can be very efficient for engineering work, IF you are good at specifying the technical job details.

- Before you get into a situation where you are spending more than you make (either because you kept your job but are spending more than your job pays, or because you left your job to do this before there is enough revenue to pay yourself living expenses), decide (with your partner if you have one) on some specific levels of your savings at which you will a) seek outside funding and b) close up shop and go back to finding a job. This will not only give you concrete goals to work towards (I need to get that customer to keep my bank account above $x!) but also help ensure that you don't blow yourself and your relationships up chasing a moving target.

- pitch to some institutional investors. "But I said self-funded!!". I get it. But recognize that: a) the process of distilling a 10-15 slide pitch is very, very helpful in refining and focusing your business plan, b) investors will have useful feedback, c) the experience is helpful (even if to understand how often very silly the investor courting process is), and d) having deep familiarity with any concerns that prospective investors have will help you to either address them, if you decide to do so, or alternatively, regardless of whether you agree with the concerns, to have thought ahead of time about how you will speak to them when other people have those same concerns such as prospective key employees and certain customers (especially large ones who may be about to sign a big check)

- find 1-2 other cofounders (who are not your relatives or spouse). you will have a ton of work to do, especially if you keep your day job, and you will want others to lean on, not just for the work but also emotionally (do not underestimate how physically and emotionally draining this is going to be). At least as importantly, just like pitching to investors, in the process of recruiting co-founders you will receive valuable feedback that will help you to refine the business

- Being fundable is a worthwhile option to keep open. For this purpose:

  a) get a lawyer at least for certain key items. Ideally, a lawyer with prior experience at a startup-focused firm (Wilson Sonsini, etc) who has left and started their own practice. You'll probably pay 350-550/hr, but the person will be experienced and in my experience, this kind of lawyer will not excessively bill. I would advise against using any medium or large law firm at this stage unless you are ready to pay way, way more. Key items below.

  b) make sure your IP is clean. Many states enforce employer's right to own the work you do outside of work hours. It is uncommon that your normal job will have absolutely nothing to do with your side project, which muddies the water even further. you should *definitely* review your employer handbook, policies etc, and research this topic. Keep in mind: you're not trying to be right in the case of a lawsuit, you are trying to avoid any possibility of any lawsuit. If there is any legal dispute with your employer, then regardless of whether you legally win, you automatically lose from a business perspective. Note: in many cases, it can be beneficial to be open with your employer about this side project. To decide, ask others in your firm who have gone thru this kind of experience (if you can find any such folks)

  c)  get the right corporate structure. An LLC can give you significant tax benefits when you are bootstrapping. A C-corp will be needed if you ever want to raise VC. I can imagine no reason to use an S-corp-- but that's what you want to ask your lawyer

Admittedly, you can still succeed doing none of this, and still fail doing all of this, but I hope these suggestions might help to maximize the probabilities in your favor