minaguib

What is it with "search doesn't work/work well" being a universal gripe with all of Microsoft's products ?

It's not like there are no good examples of what modern user expectations are for searching a body of text....

If I understand things correctly, the modern way to to read-time masking in the AWS ecosystem is to introduce AWS Lake Formation ( https://aws.amazon.com/lake-formation/ ) as the abstraction layer between Athena and S3.

Good start but very simplistic

What happens when these threads or goroutines start doing real work (CPU, IO) concurrently ?

I'm hiring, in advertising technology:

* Architect with big-data/data-lake experience (Bay area)

* Architect with distributed systems experience (Montreal area)

* Site Reliability Engineers (Bay area AND Montreal area)

Please get in touch :) Let's chat and have a (virtual) coffee.

It looks like a more distributed alternative to lsyncd ( https://axkibe.github.io/lsyncd/ )

Yes

I often have to pluck out attributes from streams of json records (1 json object per line) - often millions/billions.

jq is almost always the bottleneck in the pipeline at 100% CPU - so much so that we often add an fgrep to the left side of the pipeline to minimize the input to jq as much as possible.

Welp, here it is - from https://www.garmin.com/en-CA/outage/ :

> Garmin has no indication that this outage has affected your data, including activity, payment or other personal information.

Forensics can sometimes prove "X happened", but not "Y didn't happen".

That's why sometimes companies will make statements like "we have no evidence that the hackers did Y1, Y2 or Y3". It doesn't mean anything really.

That is to say, once Garmin becomes communicative again, they may be prescriptive in answering questions like yours head-on, or due to lack of concrete proof, punt and obfuscate.

Suffice to say, it appears they've been owned through-and-through, so you may want to err on the side of caution.

Agreed, FWIW this should be a habit for any PS1-modifiers (like git repo status)

That would place the plugin authors in a potentially dicey position legally-speaking.

The publisher is trying to adhere to a law (for better or for worse) by giving the user visibility into technologies and data usage.

A plugin hiding that by default can lead to all sorts of nastiness.

I've maintained some high-level notes on this event, problems and fixes here: https://gist.github.com/minaguib/c8db186af450bceaaa7c452b76a...

Since their introduction, I've had no issues telling the agent "I'd like to opt out of the scanner". This adds ~10 minutes as I have to wait and perform a manual pat-down, but personally that's not something I mind. Others obviously may feel otherwise for varying reasons.

I've found this article (10 years old now, : https://www.health.harvard.edu/diseases-and-conditions/are-f... ) to be fairly well-rounded. I wonder if, since then, there has been a better analysis of actual/measured results of these technologies.

Yes, absolutely.

More fulfilling for me however is not the code itself, but significant results. I have captured/framed a few graphs in my career that look like a "sudden jump" or a "sudden drop", which were caused by deploying a more elegant solution to a problem or pain point.

With more modern databases, there are server-side resource constraints per bucket (query, connection, etc..). One of the resources is execution time that can be constrained. Sometimes it can even accept a hint, from the client, regarding the maximum runtime to enforce server-side.

Linux and fallocate() allows precisely this - we use this to "log locally" for applications, and a "tail this asap and publish to kafka then delete locally" helper.

See ( https://gist.github.com/minaguib/1cbe29922b06d50755a2f580b8c... ) for some test notes I took a couple of years ago.

I've written a quick tutorial on linux sparse files/fallocate and effect on different writers/readers:

https://gist.github.com/minaguib/1cbe29922b06d50755a2f580b8c...

So is youtube: https://downdetector.com/status/youtube

FWIW, past a certain scale, it makes a lot more sense to export dedicated telemetry out of your app into a TSDB.

The reason is, it's thousands of times cheaper to +1 a counter/record a gauge and flush it every once in a while, than to serialize per-request metrics into HTTP headers, log them, and re-parse them for analytics later.

Without spending too much time on it, I think a cursory check shows git doing "the smart thing" with ignore rules.

  $ mkdir foo
  $ cd foo
  $ mkdir $(seq 1 1000)
  $ git init .
  $ strace -c git status
    On branch master

    No commits yet

    nothing to commit (create/copy files and use "git add" to track)
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     75.97    0.058830          57      1035        15 openat
     16.48    0.012764           6      2002           getdents
      3.87    0.002999           3      1021           close
      2.91    0.002254           2      1022           fstat
      0.19    0.000147           8        18        14 lstat
      0.13    0.000100           4        24           read
      0.12    0.000092           6        16        12 stat
      0.09    0.000072           3        21        14 access
      0.06    0.000048          10         5           write
      0.04    0.000033          33         1           unlink
      0.03    0.000020           3         8           rt_sigaction
      0.02    0.000018           2        12           mprotect
      0.02    0.000015          15         1           munmap
      0.01    0.000010           3         4           getcwd
      0.01    0.000006           0        16           mmap
      0.01    0.000006           6         1           ioctl
      0.01    0.000005           2         3           brk
      0.01    0.000004           4         1           chdir
      0.00    0.000003           2         2           getpid
      0.00    0.000003           3         1         1 readlink
      0.00    0.000002           1         2           rt_sigprocmask
      0.00    0.000002           2         1           set_tid_address
      0.00    0.000002           2         1           set_robust_list
      0.00    0.000002           2         1           prlimit64
      0.00    0.000000           0         1           execve
      0.00    0.000000           0         1           arch_prctl
    ------ ----------- ----------- --------- --------- ----------------
    100.00    0.077437                  5221        56 total

Now let's add an ignore-all and check:

  $ echo "/*" > .gitignore
  $ strace -c git status
    On branch master

    No commits yet

    nothing to commit (create/copy files and use "git add" to track)
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     46.25    0.000568         284         2           getdents
     16.78    0.000206           6        35        14 openat
      7.41    0.000091           6        16        12 stat
      7.17    0.000088           4        21        14 access
      5.62    0.000069           3        25           read
      4.72    0.000058           3        18        14 lstat
      4.15    0.000051           2        22           close
      2.93    0.000036          36         1           unlink
      2.28    0.000028           1        23           fstat
      0.81    0.000010           3         4           getcwd
      0.65    0.000008           1         8           rt_sigaction
      0.41    0.000005           5         1           ioctl
      0.33    0.000004           4         1           chdir
      0.24    0.000003           2         2           getpid
      0.24    0.000003           3         1         1 readlink
      0.00    0.000000           0         5           write
      0.00    0.000000           0        16           mmap
      0.00    0.000000           0        12           mprotect
      0.00    0.000000           0         1           munmap
      0.00    0.000000           0         3           brk
      0.00    0.000000           0         2           rt_sigprocmask
      0.00    0.000000           0         1           execve
      0.00    0.000000           0         1           arch_prctl
      0.00    0.000000           0         1           set_tid_address
      0.00    0.000000           0         1           set_robust_list
      0.00    0.000000           0         1           prlimit64
    ------ ----------- ----------- --------- --------- ----------------
    100.00    0.001228                   224        55 total

I use something even simpler

My dotfiles git repo is meant to be cloned in my home directory. It comes with this .gitignore committed in the repo:

  /*
  !/.vim/
  /.vim/.netrwhist

Basically it ignores everything in my home directory, unless I explicitly `git add` it, which matches my workflow. For the few cases where I want to notice changes (like the entire ~/.vim/ subdirectory), I explicitly un-ignore it as you can see above.

The only downside I've experienced is my bash PS1 prompt shows the status of the dotfiles repo (branch/dirty/etc) in any directory I'm in that's inside my home dir - I've learnt to ignore it, and it doesn't interfere with CDing into an actual directory that's its own git repo.

Tangential, but since you mentioned Akamai.

I wrote a framework at work that maintains authoritative DNS configuration truth (static, and advanced GSLB variations) as text files going through git and regular PRs, auditing, etc.

The truth is then flushed out via APIs to multiple DNS providers (active-active), Akamai being one of them. Nobody's APIs are perfect, but I don't recall it being too difficult to push the state to their system.

I've had a similar problem but in a professional context. We use dual DNS providers (active-active), and in both providers we used advanced dynamic DNS services (not static records - but think things like DNS directing users to the closest healthy data center).

Both providers offered APIs, but, especially due to the higher-level dynamic DNS offerings we were using, were vastly different.

I ended up implementing a custom tool that uses, believe it or not, DNS zone files, all together versioned in git, as the truth for applying and provisioning changes.

I know it doesn't exactly answer your question, but FWIW when we needed it we had to write it from scratch. Ours is an internal tool, but more recently I've also come across something by Spotify that looks similar in spirit ( https://github.com/Shopify/record_store )

It doesn't have to come to this, at least from adtech's side.

Generally, your device is instructed via a publisher's site/app to reach out to ad tech servers either directly (firstparty), or indirectly (firstparty->thirdparty, firstparty->RTB exchange->thirdparty).

Due to the "chaining", GDPR is particularly onerous on the adtech industry. Granted all the data is keyed by semi-anonymous IDs (cookies, IDFAs, IPs), the concerns for consent, retrievals, deletions, in a cascading manner, are an industry-wide problem requiring collective action. The IAB proposed something for the RTB side, the publishers don't like it, and it'll be tense until and through May 25th :)

Having said that, nobody wants to shut-the-whole-thing-down. While all these servers may refuse service based on fuzzing the request as originating from the EU, they may also decide to serve as-best-as-possible and minimize logging of the sensitive fields - it may be better, for example, to lose some functionality for European devices (behavioural targeting, for example, the idea of showing you an ad for the Widget you just looked at over and over), than to serve nothing at all.

This one's on the easy side.

The company is the controller of the data, and Amazon is the processor.

Here's Amazon's declaration and stance, stating they are GDPR-compliant both as a company (when they are the controller - of their direct customers' data), as well as then they are a processor (infra for use by others who control private data): https://aws.amazon.com/compliance/gdpr-center/

There's generally no need for a controller who relays data to a processor to understand the intricacies of the implementation on the processor's side (is deleted data really deleted ?) - what's more important is the processor's self-declaration for GDPR compliance.

The above is my personal $0.02 as I've been spending quite some time getting into GDPR recently. IANAL

Depending on what they're reason is, it may be satisfactory to have the code looked at by a qualified 3rd party (with proper NDAs all-ways).

"Using electron" and "uses less battery" are too close to each other.

I work in online advertising (AdGear/Montreal/Canada) and a lot of the business is around RTB (Real Time Bidding). This is an area where millions of requests/second is the norm and latency is the name of the game.

We've been using Cassandra for many years. As the business has grown, we've refreshed/updated/resized/tweaked/hacked the cluster. Unfortunately past a certain point, there were no more performance wins to be had - further - we started experiencing stability issues.

We came across ScyllaDB. I had heard about the underlying SeaStar framework and was excited to test something based on it that was addressing a real pain point.

We are not (yet) customers of ScyllaDB - however we're pretty advanced into our POC. To summarize: * Extremely low latency compared to cassandra * During delivery valeys, peaks, and even burst batch loads * It delivers the above running on a fraction of the hardware Cassandra does * Very good no-bs responsive engineering support

Scylla's having a conference in SF later this month where AdGear will be presenting our use case more in depth. If it sounds like the product may address some pain points you're working on, might be worth attending.

/not affiliated with Scylla beyond the potential-customer-in-POC case described above

What exactly are people expecting when they order these things ? Are they unable to tell the texture is plasticky and highly processed, unlike normal "stringy" chicken meat you prepare at home ?

The demo page causes a crash and reload loop on Safari iPhone iOS 9.3.2

Not only is the handshake expensive, but TCP has something called "slow start" which effectively artificially limits the rate of data in the beginning of the connection till the network capacity is discovered.

An ideal N would be high enough where the cost of the TCP handshake and slow-start are minimal, but small enough not to trip the traffic shaper to downgrade the connection from the fast flow to the slower one.