shayanjm

Nested querying is not something that is standard for regular grammars, amongst other aspects introduced in this paper that implicitly require things like memory (again, not standard for regular grammars)

At what point do we drop the term "regular" expressions altogether for stuff like this? This is going to sound pedantic since I know that most popularly-used regex implementations are themselves non-regular, but I feel like we're just piling more and more stuff on top of good-old-regexes and trying to turn the concept into a catch-all for anything that does pattern matching on text.

I guess it just feels icky that "regular expressions" has inherent meaning (i.e. can be represented entirely by a finite automaton) which has become completely diluted at this point.

That rant aside, cool paper. The idea of bridging formal language theory with modern computational tooling feels timely. I think I would've liked to see more exploration of oracle-based costs, for instance:

* What happens when oracle outputs are inconsistent/uncertain?

* What happens as oracle interactions become more computationally expensive?

Building off our last research post, we wanted to figure out ways to quantify "ambiguity" and "uncertainty" in prompts/responses to LLMs. We ended up discovering two useful forms of uncertainty: "Structural" and "Conceptual" uncertainty.

In a nutshell: Conceptual uncertainty is when the model isn't sure what to say, and Structural uncertainty is when the model isn't sure how to say it.

You can play around with this yourself in the demo!

I think the author of this post probably meant to caveat that what we call "prompt engineering" TODAY might tend towards snake oil, but prompt engineering _doesn't have to_ be snake oil, and it _doesn't have to_ promote black box mentalities[1]. What's more is that fine-tuning is certainly not a panacea - it's not particularly great an injecting net-new context into these foundation models. It's great when you want to "close the aperture" a bit in model outputs. Even suggesting that fine-tuning is somehow a replacement for crafting prompts is just incorrect.

[1] https://heatmap.demos.watchful.io/

Interestingly we initially thought that prompt length would play a big factor in the performance of this approach. In practice, though, we discovered that it's actually not as big a factor as we predicted. For instance, Prompt #3 was 410 tokens long, while Prompt #5 was only 88 tokens. The estimation for Prompt #3 aligned fairly well with the IG approach (0.746 cosine similarity, 0.643 pearson correlation), while the estimation for Prompt #5 seemed to underperform (0.55 cosine similarity, 0.295 pearson correlation). Meanwhile, Prompt #2 was 57 tokens long and performed quite well (0.852 cosine similarity, 0.789 pearson correlation).

Re: our definitions of average/long/short prompts -- we weren't really rigorous with those definitions. In general, we considered anything under 100 tokens "short", 100-300 average, and 300+ large.

Our intuition here is that the relationship between performance of the estimation and the prompt structure is less about length, and more about "ambiguity". Again, we don't really have a rigorous definition of that yet, but it's something we are working on. If you take a look at the prompts in the analysis notebook you might get a sense of what I mean: prompts 1-3 are pretty straight forward and mechanical. Prompts 4 & 5 are a bit more open to interpretation. We see performance of the estimation degrade as prompts become more and more open to interpretation.

Great question - there are currently (likely) tons of limitations to this approach as-is. We're planning on testing this on more capable models (e.g: integrated gradients on Llama2) to see how the relationship might change, but here are some initial thoughts:

1. The perturbation method could be improved to more directly capture long-range dependency information across tokens

2. The scoring method could _definitely_ be improved to capture more nuance across perturbations.

I think what we've found is that there does seem to be a relationship between the embedding space and attributions of LLMs, so the next step would be to figure out how to capture more nuance out of that relationship. This sort of side-steps the question you asked, because honestly we'd need to test a lot more to figure out the specific cases where an approach like this falls short.

Anecdotally - we've seen the greatest deviation between the estimation & integrated gradients as prompt "ambiguity" increases. We're thinking about ways to quantify & measure that ambiguity but that's its own can of worms.

I came here to ask that EXACT question - fighting the good fight

Off the back of a napkin - the key should never be stored anywhere first of all. In the absence of keyring/keychain/etc., it'd be trivial to introduce a masterpassword implementation in the browser client which is XOR'd with secret credentials and stored as such.

Obviously not a 'secure' system by any stretch of the imagination but it's an order of magnitude better than storing in plaintext.

As addressed in the post - there are no mitigating factors in the scenario of accidental exposure. The lowest hanging fruit would be a dumb hashing function which uses some master password.

If you've been hit with an OS compromise you're pretty much SOL, but it shouldn't be so easy to grab highly sensitive data from accidentally exposed profiles.

Yes! There are tons of accidentally-uploaded profiles on github, for instance. Search for the readme string and you'll see a number of very dangerous commits.

Shameless self-plug - I built a "pastebin scraper as a service" platform, and released it here: https://github.com/shayanjm/pasteye

Took much inspiration from dumpmon, but distributed it so users can choose their own sensitivity settings.

I remember chatting with you a while back as I was also working on a pastebin scraping project. http://github.com/shayanjm/pasteye

Glad to see dumpmon is still going strong :)

Absolutely - this wasn't meant to be a be-all-end-all to lead generation by geographical bounds. This was more of a "I had a cool idea, wanted to test it, and here's how I went about doing that" sort of post. There are a ton of things that don't get picked up by social signals, but it's a low enough hanging fruit that I figured it was worth looking into to see if it was even a viable source of data at scale.

I think a 'real' analysis of the data at a much larger scale (probably state-wide would be a good test) would yield some interesting data on the effectiveness of this implementation in specific.

As mentioned in the post - the circle packing implementation itself is general enough to be used in other applications. The point was to show off a cool potential application for a solution to a non-intuitive problem.

That's smart - coupling this with the 'min layers' equation could lead to a 100% iterative solution. We could swap this component in and remove the recursive triangle generation :)

As mentioned in the post - deduping is cheap and easy so introducing the third basis vector u3 could potentially be a great move away from the recursive logic that currently plagues the project.

Looks like something I'd use every day if executed properly.

Some things that I think would make this an awesome app:

1) IFTTT integration would be sweet. "If i get an e-mail about something, remind me to do it by putting it in my contextual note for an application"

2) iCloud integration would be sweeter. Unified notes for applications that exist in "desktop form" and "mobile form".

3) A "view all notes" option to be able to browse sans-context

4) The ability to clip media into a note easily

@shayanjm on twitter.

I have a half-baked contextual analysis implementation which I could probably spin into a high-volume twitter analysis tool. Was doing NLP analysis on unstructured data (like news articles) and extracting topics + extrapolating commonalities between sets. Could be used to pick up topics from tweets and determine if two unrelated tweets are actually talking about the same thing (without necessarily replicating the same syntax).

Agreed. I think generating a more filtered list is possible - but would take significantly more time than it took to build the sentiment analysis tool.

The results of the more-filtered-list-tool would be quite interesting, though, as you'd essentially be modeling a set of "ideal leads" and determining how close/far a set of tweets are to those models. Just figuring out an "ideal lead" model for the segments you're targeting would be an interesting intellectual pursuit.

I think I might end up building this...

Kudos on building something cool to solve a pain point, but unsure about the efficacy of this implementation. A list of negative-sentiment tweets about competitor products is certainly a good place to start, but is by no means a list of actionable leads. Still requires quite a bit of human interaction to figure out which tweets are actually solid leads, and is only truly useful if your competitors have only one product.

You also miss out on users asking for suggestions who aren't currently using a competitor product (which IMO is a more valuable segment).

A more interesting implementation is one that takes context into account, but that would require some homemade ML work and likely outside of the scope of quick & hacky solutions.

If you have your personal projects listed in your resume - you'd better make damn sure they are at least 80% polished and _SHOW OFF_ your skills. If they are not/do not, don't list them in your resume or make them private.

Not here to say that one style is better than another, although I've used gitflow almost exclusively for a few years and it's served well. Pushed features/hotfixes/etc. out the door in a timely manner, and never felt held back by the 'ceremony' necessary to push to prod. CI handles most of that and with the 'git flow' plugin you can easily do a two command deploy while not compromising good branching/repo maintenance practices.

Most of it is stuff he would have to unlearn before he could then learn it on the job. I would rather hire someone who has no habits than one who has bad ones.

No, I wouldn't hire you as-is. Probably not a first-choice intern either.

Some notes on why not:

+ Your web-sume looks rough. As pointed out by others, there are a number of typos (i.e: "and provide an opporunity") not to mention the design itself could use work. If you are GREAT at web design/UX you should spruce it up. Otherwise, kill it and move to a traditional resume. Knowing HTML5/CSS3 today is pretty meaningless, so showcasing that is sort of pointless.

+ There are tons of issues with your resume itself (i.e: "Excellent verbal and written communication skills." despite multiple typos and unclear flow) which need to be addressed. Cut the fluff, point to recent projects & address why they are cool/why anyone should care. Anything that you did 10+ years ago that isn't directly applicable to what you want to do in the near future has no place on the resume.

+ Your bitbucket projects are lackluster. You don't follow good git branching habits, your commits are non-atomic, your code is cumbersome and unfinished in many places. You also seem to use .py files as notes in non-standard ways, introducing weird artifacts and conventions to your projects.

Some notes on how to improve:

+ Learn how to use git productively in a team environment (this means no more working directly out of master). This is a good resource to that end: http://nvie.com/posts/a-successful-git-branching-model/

+ Learn better coding habits in whatever language(s) you are most comfortable with. Your bitbucket only has python code, so learn how to do things in more 'pythonic' ways. (i.e: Don't just stub notes inside .py files. Throw them inside a README.md or keep them in a secondary utility so you don't clutter the repo).

+ Sort of back to point #1 but deserves its own category: Learn how to use .gitignore. You have tons of artifacts in your repos that do not need to be/should not be there.

If you address all of the above, you'll be in a much better position to start qualifying for entry level dev openings.

Thought I'd give a contrarian perspective here.

I was hiring for a telehealth startup based in NYC and got some referrals to some recent General Assembly grads. I bit and went ahead and scheduled some interviews. It was a total joke, honestly. The graduates glossed over the entry-level interview questions with a lot of handwaving (I would ask them things like "How would you do x given y?") and quoted rates upwards of $100/hr even though their total experience was the 10 week course @ GA campus.

I was so turned off by that experience that I just never even considered hiring from a 'hacker bootcamp' again. I'll echo what others have been saying as well: You can hire them, and maybe they'll perform for a while - but the amount of time you'll need to spend to get them up to speed on CS basics will more than likely not be worth the investment. You're better off hiring a recent college grad whose only experience is working with Java - at least they have the fundamentals and can build on top of them instead of backtracking.

Build cool things and don't focus on the income. If you need cash, get contract gigs to pay the bills. Gun.io is a great resource for that (I've only got a few years on you and I've made a pretty penny doing consulting work for the last few years).

If you've used it for anything non-trivial, then you probably should've been using a private registry - in which case you get the decentralized benefits of npm sans npmjs.org availability issues.

source: a number of non-trivial node deploys

Thanks for the suggestion :) I'll probably throw an overlay in there and stick the blurb in that.

This is my attempt (and first webGL anything) at a stupid implementation of the observer effect in quantum physics. Essentially, by observing something we are inherently changing its nature. Here, I've replicated a sine wave propagating in two directions (so it can be loosely correlated with some generic electromagnetic wave) which randomly changes in real time for all viewers when someone else starts viewing. (Very messy) code can be found here: http://github.com/shayanjm/quantum

tl;dr - It's a wave that changes slightly when you look at it. Pretty cool, when you realize that every stage of the simulation is unique, temporary, and will never be re-rendered again :)

Pasteye - https://github.com/shayanjm/pasteye is Pastebin Monitoring as a service. Essentially, the more eyes we have on the repo, the better the service becomes. Looking for contributors to push towards v0.2.0 release - where the service becomes 'really' useful. The project roadmap is included in the repo!

Pasteye (Pastebin monitoring as a service) - was trending on the front page for a few hours 2 days ago.

http://pasteyebeta.herokuapp.com/ http://github.com/shayanjm/pasteye

Source code is available, and you can use _any_ password.

Inb4 not reading before posting