CorvusCrypto

>Allergies to common foods are increasing rapidly as well

Is there a list of stats for incidence of food allergies? It seems many friends I know (in EU) have food allergies to things like cucumber, hazel, peanuts, also gluten (celiax or something). One person I know has Crohn's disease. I would be curious to see the stats in both US and EU to see if it's really linked to a region's food. Or if it's just that common diets across the world are causing it.

Edit: or a third possibility is that rates haven't changed at all and it could be increase in either reporting or population making it seem like it's more commom now than before.

Horse meat, anything deemed unsanitary like haggis (edit: think like scottish-made haggis, meaning containing "unsanitary parts" like lung), unpasteurized milk (though some states override this). I think banana tree oil and sassafras oil are allowed in EU as well but are banned in US due to links to cancer.

Well my hope is that something can be learned about this difference. As I said I don't want to take away from the fact it's a problem in all cities no matter how large. Rather, I want to understand what is causing differences in perception and is there a difference in regulation or response that causes this.

Seems that I've seen shelter rates brought quite a bit in the last 5 minutes, and I'm currently investigating these differences. I hope homelessness gets solved for every city of course.

I understand your wariness.

I am using your population numbers and dividing by the numbers in my reference.

The per capita rates are:

SF: 6857/884363 = 0.00775360344

NY: 78676/8540000 = 0.00921264637

The reason I said this is not to lie with statistics, but to show that when normalizing for population the rates are still worse in NY.

Yeah this is all stuff I'd like to know. It could point to potential frameworks for solutions. Maybe they have implemented something that mitigates problems associated with homeless in SF.

This is kinda where my mind is taking me with this kind of interesting deviation in exposure.

Dividing by each yields a higher per capita for New York still

One if the things I don't understand is how San Francisco is the majority exposure of this problem. Apparently New York has the most homeless at a whole magnitude more[1] than San Francisco, but there seems to be less of a focus on the problem with the homeless population there. Los Angeles as well, news tends to SF in my experiences. It's also discussed more (for instance in Sweden SF and the west coast in general gets more focus by searching t.ex. "hemlösa USA")

That's not to take away from this and I hope they can do something for those folks, but it also strikes me as amazing this city in general has the largest focus and not even close to the largest homeless population.

1. https://www.statista.com/chart/6949/the-us-cities-with-the-m...

Well for training attention-cnn model I think it.would speed it up, but as they note in the paper, running the pipeline they experienced worker saturation on their dataset (see figure 12). Looks like saturation occurs after 7 parallel workers with their hardware.

> For a novice it looks a pretty simple job of using some Fuzzy string matching tools and get this done.

Nice.

That annoying quip aside, as with many things in data processing, it's case to case. In TF-IDF you lose ordering information by definition. This is probably fine for this use case but it does mean that if ordering does matter since a set of stores share the same words.in different ordering, this will fail to resolve the difference. The author says he did due diligence on the data but there are other ways this can fall short. For example ["Walmart", "5280"] compared to ["Store", "5280"] is going to not be so similar as one would want due to the down-weighting of the identifying number in TF-IDF. So imo the disadvantage mentioned for using BoW over TF-IDF is actually not a disadvantage sometimes. As with everything it depends on your problem and data.

To the author I would hope in the future you remove statements like "to a novice, it seems easy to use X". There is nothing novice about going into a problem with an idea and trying it if it seems to fit the use case.

That's exactly the trade-off. I'm not going to say it's a big enough negative to dismiss using the stateless signed token scheme because it depends on the needs of the application.

But either way, if you really can't afford a database or cache layer lookup to see if a token is still valid, then you accept that by using a bearer token, that is only validated by signature alone, that it is possible a user will have their session hijacked without possibility of revokation.

The usual way this is mitigated is by use of a small expiry time (I've commonly seen <=5 min) and a revokable refresh token. This still gives a hijacker a possible 5 minutes (assuming 5 minute expiry) if a user revoked the refresh token, but it does mitigate the damage while still reducing DB lookups since you only do a lookup in token refresh. Hope that clears things up. Again your application needs should drive these decisions.

Same goes for any signed token scheme. You can still revoke JWTs if you give them an ID and keep a revoke list somewhere. Though as you said most use these to avoid datastore lookups. It's a trade off. Either time-limit signed tokens that can't be revoked with benefit of no lookups or implement revokation.

I can elaborate a tiny bit. It's been mostly a rocky road in library development as well as some confusion in the jwt specification. Basically the JWT spec is poorly designed for lay-programmer use and some folks are implementing the spec wrongly or are just configuring their systems that use properly-implemented libraries in dangerous ways. For instance you need to choose the algorithm carefully and then be careful not to accept any other specified algos as it can cause some interesting attacks (specifying symmetric algorithm when the token was meant for asymmetric ones can lead to valid signing using the public key if the system allows it). Also Technically a user can specify a "None" algorithm that doesn't do payload verification, which tbh all backends SHOULD drop tokens specifying this.

JWTs as bearer tokens aren't bad in their own right, but if you aren't careful you can screw yourself and therefore many security experts avoid them for use in securing systems. Plus a lot of people mistake it for an encrypted token which it isn't. You can imagine how bad that can get.

Tbh I'm with the parent commenter. I avoid them, but if you avoid common pitfalls they should work for your system no problem.

I'm on mobile and can't be arsed to gather sources, but you can search the claims I made and you'll see several articles about these problems. There's even a defcon talk about a new proposed standard (called Paseto I think) that starts by highlighting the major issues with JOSE and JWT specifically.

Pretty much my hope. If it's so easy to snoop on cell traffic my hope then is that the app communication is encrypted using modern standards and airtight. Though I'm sure you're right since these apps are more under the microscope. It's probably fine.

Hmm interesting read. In Sweden there is a thing called BankID and basically you can use your mobile device as a universal authenticator. Of course, you need to have the device and enter a 6-digit pin, but I often wondered how dangerous it was to use this so much. And on top of that I know people that used it in local cafes on public WiFi.

I would love to do an examination of communication via BankIDs app to the internet to see what kind of security exists to protect the user. If you can get the person's social number (personnummer) and their 6-digit code, then spoof their device (probably the easier part) you can basically take over their life in Sweden.

I have a few questions after reading. Mostly I'd like to know how they built their dynamic queueing system.

How do they signal the workers to refresh their dynamic queue lists?

Are they using a homebuilt queueing system or piggybacking on top of something like celery?

What is the underlying message bus and how is it deployed?

Would love to know more. Also kudos to them for stressing idempotency in a message system. It's usually much easier to ensure idempotency than to ensure exactly once delivery.

In both the original example and this updated version, the nested loops (or foreach calls) create generators on each iteration. However, the first example is creating the generator chain more implicitly than the newer example. Since the original example also uses a coalescing expression (i.e. I mean that he is returning a dynamically-created generator chain as a variable) it means the use of "return" keywords at every subcall to foreach which can confuse programmers that aren't as seasoned with generators or functional generator patterns. It's actually, I would argue, not "bad code" but rather a different approach that procedural programmers probably aren't used to. The benefit of this style of doing things is that it comes with the benefits that usually come with creating lambda expressions. There are better examples of creating on-the-fly generators, and surely thats what the original author wanted to showcase, but to programmers not well-accustomed to common range-based generator patterns, the Pythagorean triple example for this looks and feels a mess when it's clear you don't need to generate the generator dynamically. This updated example I think is more generally useful for showcasing the range iterators to programmers.

Also this is just my opinion. Of course it could be that the original example is hated by all and a total POS, but to me it seemed quite clear in a functional context what it was getting at and what it meant for use of dynamically-created generators in higher order functions.

This is kind of a grey area it seems. Manitoba IS one province that specifies a legal minimum age of supervision (12) but it applies only to being left at home or in a car (edit: I was wrong about these specifics. They leave it more generalized in the actual law) and is still left with a clause about "unless reasonable situations exist". Meanwhile the age to allow kids to walk to school, while unofficial and no limit exists across canafa, seems to be 9 years by majority opinion agreements.

Sounds like this social worker stepped out of bounds. If Canadian government is by and large okay with kids walking to school themselves at 9, surely walking a block with a 9 year old is reasonable enough. It's clear the parent was still aware of where they were so I'd say it's an arguable case, but IANAL.

Tbh I don't know if this is just one isolated case, but it kinda sounds like it.

Edit: here is the law and I linked the subsection that is relevant. It more clearly states what I paraphrased above. https://web2.gov.mb.ca/laws/statutes/ccsm/c080e.php#17(2)

For the first question, we were looking at it to perform distributed matrix factorization in the immediate use case, but were planning to use the platform for more advanced techniques later on. We also tried to implement a quick and dirty distributed ALS algorithm and that's where we used the RDD API came in and my subsequent hatred for said API. Kafka was our streaming platform through which all our customer data was sent.

For starting out, the tutorials on the spark documentation pages are pretty good actually. However for streaming sources it seemed hacky to use spark with our kafka use since we had to basically scour maven/google for a package that did anything useful with avro-serialized data. Even after finding relevant packages we had to copy and paste some avro-to-sparkstruct lib code into an internal library since it didn't play nice with our versions of spark in it's public form. We were using structured streaming so that we could leverage the dataframe API which is much easier for data scientists to use and grasp coming from pandas/R. It's a lot nicer for a data engineer as well (imo).

JSON-serialized data worked okay actually, so we ended up using kstreams to deserialize the avro data and output it to another topic that was JSON-serialized since spark handled that better (read: more easily). Then we used the dataframe APIs to implement some of the higher order operations such as building interaction matrices and factoring them out, then outputing the resulting factors to cloud storage for collaborative filtering to use as a recommendation system we had elsewhere.

I guess to me it seemed wholly immature for Kafka-sourced data and it made me think that if we dumped into Hadoop and then operated on microbatches it would have been much easier.

About Scala vs Java vs python... We saw that python still wasn't the first-class language contrary to what they say I'm the docs and that to utilize any newer, fancier algorithms that would come up it would be beneficial to use Scala since that is the language most developed against (as of our use of spark). There are plenty of libraries to connect with Kafka in any language so Kafka didn't factor into the language decision at all.

I will leave out some details about managing the resources on a cluster as well as implementing CI/CD. That could be a whole blog post on it's own.

Hope this helps. It may be the best tool at the moment for what it offers, but it was painful to use and I hope someone makes something better with the knowledge gained from the usage of spark.

Learned Scala, Kafka, and spark (and spark streaming/structured streaming).

I will say that after using spark, the big data community badly needs better distributed computing tools. It was just awful to work with. Also good luck debugging anything easily when you create stuff with the RDD apis.

I did come to like Scala, and Kafka as well, though.

Really great talk! I think my favorite was his continuous emphasis on not being so vendor-favored. It's increasingly important to be reminded of this these days.

I think more and more it's a case of people realizing there is no definitive "microservice" architecture. Because of that architects around the industry are now focusing on the flow of data and using whichever tools are best to get the right behavior in their business.

What was called microservices imo was just a higher level modularization of code to allow easier parallel development and maintenance/scaling, but since it needs a server that meant some boilerplate to initialize and keep the app running. A lot of people went about this in different ways which made things a bit chaotic to those new to the idea I think. The next logical step would be to remove that server code by running on a serverless architecture to focus on the functions that operate on data rather than having to maintain the initialization code and other boilerplate as well.

Among all of this advancement, an architect thinking the way Michael Bryzek does would have no problem using either approach since the focus is on the data and how data is used.

Still, server-based microservices are still very prominent, and I'd argue they are also much more platform agnostic if done right whereas (if I'm not mistaken) much of serverless is done in specific platforms causing you to be vendor-locked due to different deployment processes for each platform. I think research is being done to make platform-agnostic serverless deployments easier, but I haven't read much on it yet and am unaware of current advancements in that area.

"Design Microservice Architectures the Right Way" by Michael Bryzek [1]

Really good discussion on event-collaborative microservice systems that scale well for performance and maintenance as well as a good argument and use case for server and client code generation.

Definitely my favorite so far.

[1] https://www.infoq.com/presentations/microservices-arch-infra...

This isn't really a fair statement to make. Sure some people will have struggles with using a bike all winter and it may not be possible for them, but it doesn't invalidate the ideas or arguments as a whole. To assume all people for the idea are just biased and privileged based on cherry picked negative situations is just not fair. If you can't cycle then you can't cycle. Just because a few can't do something doesnt mean that this idea as a whole is invalid.

"You gotta have quick keys to say 'Your Mamma' or.." "We're gonna have macros in there. You're gonna be able to set em up in the setup program... ...so you just hit it real quick and it will say 'Screw you faggot!' or somethin"

Man things never change. That is hilarious and what a great environment to work in for them having to work late hours like they were.

the original benchmarks the author likely used had a screenshot of some benchmark tool's output. It showed a 24 MHz processing speed which I also thought was weird. Could just be an anomaly in the benchmark tool.

Either way the tool they used (geekbench) apparently looks into chip-specific optimizations for tests and the rest of their process is not disclosed so it's possible all of their benchmarks are non-uniform (again based on an optimization ticket here: http://support.primatelabs.com/discussions/geekbench/18305-o...)

> Next year, your company will be a year more mature.

Yep and we will be looking at the cutting edge always to see if there are things we can use to improve our stack/development process.

> Your product will be a year more developed.

Well that is how software development works.

> You will have a year more customers.

Yes and we will therefore have at least some stable income too!

> You’ll have a year more tech debt to pay off.

Yes, but because we have kept up to date in trends and technology the debt is significantly less than one might expect. Plus we aren't laser beams (i.e. very narrowly focused) which means we are able to manage three primary goals: fix bugs, improve/add features, and maintain/improve our stack.

This post was not great imo.

Very well put. And yes I assumed using the same key. the way I like to think about it is that if two completely independent schemes didn't maintain their randomness when combined OTP would be broken and of course that's not the case (not the most accurate way to put it, I know). However I'm always cautious to just say "yes you can add layers" because the last thing I'd want is someone to then try and develop some hybrid scheme by just slapping stuff together that ends up being vulnerable because of key reuse or some other oversight.

apples and oranges. With Key exchange algorithms you can get away with this kind of reinforcement. If you start combining encryption algorithms however, you may accidentally reduce the overall entropy of your crypto system by combining algorithms.

I'd imagine that funding and investments are a way to pressure research to favor your stance as a company. While think tanks can operate under their own rules, as an investor you'd probably not want to spend more money if the result isn't helping you in some way.

Phrased a different way I think technically it's not a requirement to follow your investors' goals, but it's pretty much a necessity so your hands are tied.

vertical monitor is a blessing and a curse at times. For just coding, it's great because I have more lines to reference at once. However when I show the directory tree for my projects my width is small and sometimes hard to deal with. Same problem with having a mini-map sadly.

I actually am right handed and read left-to-right. My dominant screen is my left screen and it used to be my editor screen, but I switched my code to the right because nowadays more of my work is server monitoring, communication, and review of code. The wider screen space is nice for this.