Tools of the Trade, from Hacker News | Hacker News Reader

Tools of the Trade, from Hacker News

github.com

581 points0x454298 comments

I find that my problem isn't finding services that match a requirement I have, but picking which one to use. It's totally overwhelming how many services do pretty much exactly the same thing these days.

I love that it's like this in the sense that there's healthy competition and they're all pushing each other forward, giving each other ideas to improve, etc... but as a pure consumer of these services, it's incredibly daunting to actually choose which to use.

I very much feel like this as well (on both counts), especially about CI.

I'd love to hear any strategies folksight have for coping with this and actually making decisions without looking into every single option, if that's a thing.

What I like to do sometimes is contact the support of the apps I'm hesitating on. Whichever is most responsive/helpful wins, as it's a good sign they actually care.

I hadn't thought of that, but it sounds like a useful thing to test out. Thanks!

yep that is a Massive data-point i use as well.

One tactic is to try one for 30m; if you can't get it working in that time, try another. Obviously biases toward good "getting started" tutorial and ease of adoption.

We from https://saasmark.com are aiming to solve this exact problem. We are currently in Y Combinator Startup School and are launching our beta in a couple of weeks. In case you are interested to check our service and give us some useful feedback, please let us know.

Fast forward to the future, where I will have to choose and pick the right saas evaluation service to help me pick and choose my saas stack.

I hope somebody is working on something already :^)

If you are really good you should have SaaS product evaluation results in your front page with links to competitors :)

.. where the one giving the best kickback wins? :)

I like to look for tools and services that have the most integrations with other products as well as a feature rich API so I can integrate my tooling with it.

Alternatively, trying to decide to pay for an off the shelf solutions or build it yourself (medium sized company).

Only build it yourself if you can really afford it. Usually it's faster to use a third party tool

Doesn't anybody do shit on their laptop anymore or on their own server? Why does it have to be 10 thousand services, why the dependency on the network and other entities? How many of these will go bankrupt by the end of the year, how many will change their terms in an incompatible way with your company project or your personal one?

When you have your own server, you're also the server admin. When that server craps out, so do ALL of the services on it and it's you who will spend nights and weekends fixing it.

Or you can pay a few bucks here and few bucks there and the uptime is Somebody Else's Problem.

And you now have to trust this random 'someone else' to have done a better job than you can. My intuition says there's more FUD around uptime and availability than it deserves. If anything, the hard work of maintaining uptime and security is a great learning path that should not be missed by beginners and small teams alike.

Related to this, using: https://hnprofile.com/

You can identify who is an expert related to each tool or any topic in general

Can you share more about how the "mood" is determined? Definitely will be controversial and is deserving of a much more clear explanation.

Why am I tense? Is it the sentences I formulate? Upvote vs. downvote patterns? RNG?

I pretty strongly believe that you should take that piece rather seriously, even if you don't previously consider it a big deal. People generally don't like when a stranger tells the world what their mood will be. Especially bad when someone believes it's quite wrong.

Also I love the opportunity for introspection. I'm trying to become a more whole, happy person who isn't defined whatsoever by his career. I would love to learn how some algorithms perceive me and why.

EDIT: Fixed the UI: I'm assuming you were looking at a topic, the "probable mood" used to render as it relates to the topic being searched, not current mood.

Highly recommend reading the "how it works" section (I assume you already have):

https://hnprofile.com/learn-more

That section has a few examples and details. I'll add further details below.

The system ONLY looks at comments sent to it. It, kind of looks like:

    { Author, Comment }

From there, everything else is derived. Nothing else, no profiles, no voting, nothing. That's it.

The system identifies:

  - Expertise
  - Knowledge
  - Interests
  - Relevant content
  - Promoter score (an improved net promoter score)
  - Trends
  - Mood
  - Related Topics
  - etc.

To your question, there are multiple algorithms at work determining mood, most are pretty standard models. These are ones you can find in NLTK, spaCy, CoreNLP, etc. There's also a couple home-baked ones. These models than "vote" for the mood, with an averaged score. Typically these scores are 3 - 7 models voting at any one time. The aggregate score is roughly 80% accurate (based on my tags), and it tags the topics, sentence, and overall comment all separately. Finally, the models are much more accurate at the extremes: Very Positive and Very Negative, but in the middle, it's much fuzzier. This is in part, just how language works, it's subjective.

All NLP models are based on a combination of open datasets with the tagged sentiment, as well as my own manually created dataset, and finally, a dataset that was manually created via looking at following comments (for instance, "no need to be negative", is used to tag the prior comment.

The probable current mood is a prediction of your prior moods. I believe right now, it looks at the last time you discussed the given topic you searched (if you searched for a user, it is the most recent mood). If there is enough data, it'll also predict your mood, based on the time of day + topic being discussed.

Finally, I should add - there are definitely improvements to be made, and I'm currently working on mood. Primarily, the focus was on expertise, knowledge, and interests. That is what enables the "related content" section and is now very accurate (given a decent number of data samples). Mood, as stated, is ~80% accurate based on my tags. I think I can do better (probably >90%), especially with more labeled data for this particular dataset.

The goal of this is it can drop into any company and instantly can search for who and what is relevant to your company.

Thanks for sharing these details. I'd like to chat more about it if you'll bear with me:

Last I checked I was "tense". Is it theoretically possible to explain why your models came to that conclusion? Is there a theoretical data view that could say "here's the comments that contributed to you being considered "tense" (or even better: specific parts of the comments).

From my perspective, without a way to concretely back up any conclusion made, I simply cannot consider the complex-sounding systems to be any better than RNG.

I alluded to this in my first comment, but I think it's imperative that if you're going to explicitly call out what mood you think someone was/is/will be, you need to back it up with "why" so that, at the very least, someone can say, "that's clearly an unfair representation of my mood and here's where the algorithms messed up."

I'm going to go on a bit of a limb and assume that you're just having some fun and exploring an interesting feature. And I think that's really quite cool. But I think we're getting into a software engineering ethics realm that I'm ill-equipped to dig in to. You've got a site that applies "moods" to individuals in a way that doesn't appear to be verifiable. And even if they are, and are accurate, is that even the right thing to do?

I don't want to ruin your fun or twist your arm into changing this. I'm just responding to my red-alert alarm that lives in my stomach. It's set off by the discovery that one of my few online accounts where I don't hide my identity has been assigned a mood label.

First, this conversation is not going to be comforting and your red-alert alarm should be going off. I know this is building toward the dystopian future. I personally use Signal, put my cell phone in the microwave when I come home, and am writing this through a VPN.

That being said, I'm 100% confident that Google, Apple, Amazon, and more know WAY more about you than I do. This system only looks at public data and only links back to usernames. Strangely enough, it can identify users across sites (even without knowing your identity) - say Reddit and Hacker News. It was originally built for https://projectpiglet.com/ to find company insiders, and make trades (which works ridiculously well btw).

There should be no expectation of privacy.

> You've got a site that applies "moods" to individuals in a way that doesn't appear to be verifiable. And even if they are, and are accurate, is that even the right thing to do?

The short answer is, I don't know. I don't want this to exist, but it does. I also know that I have already built systems for companies doing this and more. Albeit the system here is much more of a "turn-key" solution and its patent-pending for what it's worth (I'm also against patents and donate to the EFF, but feel I can potentially segment off this market). Still, lots of people are looking to build systems such as this. Personally, I'd rather be the one guiding the ship, because I do ask myself the same questions you're asking.

I'll be honest... I've used this system to show my friends that even when they change usernames I can still identify them. That particular system (which looks at speech patterns) is turned off here. That's in part because of those questions. However, I'm 100% confident, others are doing that right now, with this conversation. We know that because Snowden shared it with us.

> Strangely enough, it can identify users across sites (even without knowing your identity) - say Reddit and Hacker News.

I think a larger demonstration could make for a very meaningful "Show HN" with hopefully a large impact on privacy consciousness.

I'm sure some of us expected this, but to have it demonstrated by a single person in a casual "Oh, it could do this as well, but I don't have it turned on for ethical reasons" manner is quite effective.

Maybe it will raise awareness that "big data" isn't just trying to correlate cat-pictures one posts on their communication medium to cat food advertising, but everything you write anywhere on the Internet, even on different media under different (or no) accounts, into permanent and guarded profiles with no recourse of opting-out of the machine, effectively rendering informal discourse over the Internet dead.

This is very odd. How did it happen that you're the ship-steerer for this highly cutting-edge tech? Where did the research come from? Usually, this kind of stuff doesn't just pop out of the ether, and I've never seen anything quite so promising in this domain before. (Maybe I just haven't been paying attention.)

> Personally, I'd rather be the one guiding the ship, because I do ask myself the same questions you're asking.

I'm not necessarily the one guiding the ship, but I'd like to be (by being the largest on the market), and I'm just working to launch this as a business (called Metacortex). Fact is, I've spent years building this out for my trading platform. Most of the research is my own, with some contributions from open source and academia as it relates to NLP. That's why I submitted a patent on it, I want to corner this particular (and seemingly highly effective) method.

That being said, I've worked on products with similar goals, and have done technical reviews of WAY creepier products. Luckily, those other products rarely work at all. IMO the technology isn't quite there yet (outside of some of what I demoed), however, it's uncomfortably close.

Snowden told us this was happening, years ago. I'm sure they've improved since.

> That's why I submitted a patent on it, I want to corner this particular (and seemingly highly effective) method.

Which (turns out one of my areas of interest is "which") really only gives you a monopoly on selling the method but not on internal usage by less scrupulous actors...at least that's my opinion, it would be near impossible to enforce a patent on non-public facing code used to track people in BigCorp.

I do like the idea though, it's not like anyone (reasonably) should believe they have any semblance of anonymity on the internets and this just goes to show how easy it is to follow you around on the webs. If an individual can pull this off then the sky's the limit for the TLAs with massive budgets and computing horsepower.

Till recently, my primary account was a certain prominent account on HN. I was wondering if you'd be willing to email me the name of it.

I would be shocked and impressed, and it would serve as a great demo.

(I totally believe you. I'm just fascinated.)

Basically, I'm volunteering to be a lab rat.

Sent

Wow. Cheers for one of the coolest things I've ever seen on HN.

(The algorithm identified not one, but both of my former alts.)

Edited to tone down the reaction, but that's amazing.

lol And now, I'll promptly shut down that system again...

As I said in the previous comment, that IMO is going too far. Even though I'm 100% sure others are using similar systems, I don't feel comfortable in that business -_-

Kinda highlights, although the hnprofile.com demo may have it's current faults - there's a lot you can do. The cool part, is the platform (called Metacortex) is easy to build apps on top of.

Fascinating! Have you considered using the tool to try identify who wrote the recent NY Times anonymous piece? (Assuming you can find enough source material from whitehouse staff)

I don’t know what for yet, but I’m absolutely certain I will need to pay for your product. Quite the demo!

If you’re ever in the Chicagoland area, happy to buy you a beer!

Hey, I’m in Chi too! Right by North and Clybourn’s redline stop. If you ever want to reverse engineer this together or figure out how to train a drone to fly itself, drop an email / keybase.

If you ever need to work on insider threat detection, let me know

Ok I've got to ask - can you PM me any of my old or alternative usernames across the web?

...so...you feel tense? I feel tense reading this. But yes, I wouldn't like to read that either.

Ok, now I've waded in, I should do something to help. Reads the first few pages of your comments.. hmm Nah, you don't seem tense. All kinds of emotions. It's not you, it's HNProfile. Tries to think why it might say that.. There were a few with emotional words, frustration with situations..probably slightly more negative-emotion words than positive.. hmm I think maybe you just write better than most people, more vividly, mostly about serious topics. And are engaged with them. I don't know if it helps, but I say That's clearly an unfair representation of your mood. (I don't know you, wasn't paid by you)

The mood detection may as well be RNG.

I have an even better way to identify who is an expert on HN. Say "It's silly that X still isn't a thing." If X is a thing, you'll get ten people telling you why you're wrong within ten minutes. Example: https://news.ycombinator.com/item?id=17941772

You'll have to trade some karma for the answer though.

What else is karma for? :D I have seen this technique used to great effect inside large companies too. People will spend a lot of effort telling you why you are dead wrong.

https://www.xkcd.com/386/

Just to raise the visibility of this: lettergram's HNProfile project is awesome. It identified both of my alt accounts: https://news.ycombinator.com/item?id=17944348

The probable mood is very funny.

  > dang
  > Probable Mood: Tense

The eternal life of an admin I suppose!

Just now it was "Happy". Probably because https://news.ycombinator.com/item?id=17944992 contains "grateful".

Sentiment analysis is a finicky thing. (Maybe categories like "polite" or "argumentative" would be easier to detect?)

Conclusion: this user is emotionally not stable :D

This is neat! But, it seems a bit disingenuous to say I'm looking at experts, as opposed to people who like to talk about something. My expertise is industrial automation and connectivity for example, but I come up when searching for vue, a javascript framework, just because it's a tool I like to promote since I think it could use some noise given react's popularity. Step 2 seems like having someone or something reading through comments and assigning expert status to profiles.

Well I had to go look up myself out of curiosity

"youtube, everything, video, times, data, issues, software, experience, companies, point"

Its not entirely wrong, but I don't know what "point", or "everything" means honestly

wife, example, data, service, github, phone, bit, sort, hah, area

I should show this to my wife (:

You're an expert at everything!

haha I wish

This is cool. After skimming a few profiles of folks I know and read mood detection is questionable but the "Topics of interest" is spot on.

For reference, this started as a "different kind of search engine", utilizing conversation.

Mood wasn't exactly the focus, so it's robustness is limited - i.e. lots of room for improvement. The mood detection (based on my manual tagging) is roughly 80% accurate, which IMO was "good enough". It's more accurate as you get to the edges (very positive or very negative), the middle is kinda fuzzy, but that's kind of the nature of language.

Funny, I guess I am an expert in "heck". There is a certain rightness to that given the last couple of weeks. Is there a way to see the comments it deems 'happy'?

[edit] I do wonder if the analysis of topics is affected by both the frequency something shows up on HN[1] and not repeating the keyword in a reply post.

1) less examples and more likely to miss a topic a person is expert on

That's because it doesn't blacklist words which are commonly used in English without them being technical terms, nor doesn't it whitelist terms which are topics of interest. There's also no difference being made between referring to Wikipedia and being an expert about Wikipedia. If you regularly reference to Wikipedia (something I do) you then become an expert on the topic "wikipedia". As an example, check the expert topics from user tptacek [1] ("security, argument, fact, hn, thread, software, comment, story, google, sense").

The probable mood was rather... entertaining.

The notion this can be used to find sockpuppets or throwaways does not surprise me. Without sufficient effort you still use certain words, in certain order, consistently. Such can even be used to check out what your native language is. So if you are e.g. a blackhat who wrote ransomware it makes sense to have someone who never wrote on the internet to paraphrase or deal with the communication, or perhaps use tools such as translators.

Also, if you use delay (e.g. delay 10) it is going to appear on HN after you wrote it meaning it could appear in the next hour despite being written in the previous.

Despite the above, I did find this an interesting project; I just very much doubt its accuracy and usefulness.

[1] https://hnprofile.com/author_profiles?utf8=%E2%9C%93&search=...

Commenting here to boost my mood, or at least see how this works. I'm super happy at the moment, woohoo!

Happy. Happy. Happy...

How do I opt out, I don't expect such an invasion of my privacy from posting on HN.

The opt out currently, for privacy moving forward, is to cease contribution to the digital noosphere.

I suppose if you desired you could consolidate your doxx point to Google and ISP's by running all your comments through a couple rounds of Google translate.

There is no privacy possible in the global village. Expectation of such is technically naive.

I see there's no privacy policy, and no clear guidelines on opting out. Why? Privacy was the focal point of 2018, but you're somehow ignoring that altogether?

Ha, that's neet. I particularly like the "Probable Mood" addition.

I went from Happy to Unhappy while looking at it. I am so confused.

Needs to denote difference between price per month and price per user per month.

Especially on items like support and HR. Apps that charge per user in “flat hierarchy” companies like mine where any employee can do anything is death to your service because I’m not paying for all my employees when we equally share the workloads depending on schedule.

This is something I'd definitely be interested in adding to this list. Trouble is that services are changing their pricing very frequently.

Time to remove https://www.blitz.io/ from the list or add them to a dead-pool. Looks like they are shutting down October 1st, 2018. :(

"It's time to say farewell... Blitz will be shutting down on October 1, 2018. We'll no longer be accepting new subscriptions. Subscriptions will not be renewed after June 30, 2017. Your account credits will still be accepted until October 1, 2018."

Not affiliated but gcp could use some more recognition... as quick example firebase is also "deployment" and "identity verification". It's only mentioned as "database" which really does not do it much justice.

The contributing guidelines page at https://github.com/davisonio/awesome-irc/blob/master/CONTRIB... gives a 404.

Wanted to mention mattermost.com/.org as a Slack alternative in the Group Communication/Chat Tools section.

The correct link is: https://github.com/cjbarber/ToolsOfTheTrade/blob/master/CONT...

I submitted PR to fix this: https://github.com/cjbarber/ToolsOfTheTrade/pull/325

By chance, has anyone made a curated list of self-hosted things?

I think this one covers self-hosted nicely: https://github.com/Kickball/awesome-selfhosted

I don't understand why is there not more competition / options in the BaaS area. User management is stable, almost boring functionality required by every single product.

When talking about user management, are you referring to StormPath which was bought by Okta?

I think it is just that they are the only ones on the list.

AuthO is a stand-alone service, but aws (Cognito) and Firebase are also in this space. I'm sure there are more I'm not familiar with.

Any thoughts, folks?

Is this a good opportunity?

Sure, everything is an opportunity. Make it happen!

On the other hand, the traditional wisdom is that a lack of competition isn't a good sign.

That is what I was thinking, no competitors = red flag.

Is it really that hard to roll your own and do you really want that being maintained by a 3rd party?

It's simply too hard to make money. You have very demanding users with infinite resource requirements and finite budgets. There's no margin in it.

And there are existing solutions that are satisfactory.

Especially since GDPR

Why are most of these services rather than open source projects that fill these roles?

We used to build startups by piecing together other open-source offerings with our own programmatic secret sauce.

Cash is plentiful and time is valuable, so it is now easier to patch together a bunch of SaaS tools.

I would add site24x7.com for server monitoring. We're monitoring all sites and services in our small startup for just $10 a month. Sends out all type of threshold alerts and supports scheduled maintenance. Many other features I haven't even scratched the surface of.

Nice idea, feel free to send over a pull request

How come all these awesome lists use free form markdown? This is the sort of data that should have some sort of structure, if only just a schema less json file. That would make it easy to run a script from time to time to, for instance, remove 404 entries.

Because writing markup is more fun than writing JSON. If you want to automate something, you just need a regular expression to get whatever you want.

But but but what about XSLT?

It's also a nice resource of company one liners!

Anyone knows about a team shifting saas around there?

A tool that could help me to design and automatize a shift schedule for security guards for instance.

Basically I am looking at the following assumptions: - 24/7 coverage - 8 hours shift - 40 annual leaves days per year - 14 days of certified sick and uncertified sick leaves per year. - need for 52 periods of 2 consecutive days off (week ends). - team composed of 3 personnel. - 15 working days of training in the year. - teams of 3 personnel for each shift.

So I could easily answer how many teams of 3 personnel do I need and what type of shift schedule works for that?

Would you have any idea about that? Thanks.

If you have a business school locally, ask a MBA student/professor.

It is a standard resource allocation problem in decision support model, supply/resource allocation and management courses in MBA program.

If you do little bit digging online for textbooks in this area and available supplemental materials, you should be able to find a pre-designed spreadsheet to solve your problem.

This might help: https://www.tracktik.com/

I've got to mention https://www.logrocket.com, for error tracking, APM and user testing combined. Great price point, great support and absolutely changed the way we understand and capture errors. I'll create a PR to add it.

Great to go fast, but know you're building on shaky grounds that will shift underneath you.

I keep forgetting about this and I am always surprised when I click through.

Great list! Dropbox Paper is missing in the notes section.

Great list! Thank you author!

Side note - does anybody tried clockify.me ? Looks awesome, and it's free! Wondering how they earn money and how reliable they are.

It's legit. There is a paid plan (buried in extra features section), plus big companies pay lots of money for self-hosting and custom development

Thank you :)

What a great (and overwhelming) list!

In the code hosting section keybase seems to be missing. It has git support.

Thank you! Maybe Keybase is worth adding but git support is not one of its stand out features IMO

Agreed, but Encrypted Git is... they don't see your repo.

I don't know anybody else who does it.

What does everyone use to pull credit reports? Is there a good, affordable vendor with an API?

Excellent list! But you should probably add AirTable ;)

Thank you :) Airtable is a good one!

Missing joplin in the note taking tools section. - Great list though!

Thank you

Reply on news.ycombinator.com