lacksconfidence

> believe the MediaWiki parser itself is something like a single 5000 line PHP function!

It's not. I'm on mobile so not easiest to link, but the PHP versio of the parser is nothing like a single function. There is also a nodejs version of the parser under active development with the goal of replacing the php parser.

You could download the search indexes, also on the dumps site, that has the text content among other things.

https://paws.wmflabs.org/paws/hub/login is available to anyone with a wikimedia account

Rumor was he received more than 500m from the original Facebook buyout.

For a comparisson, in an SF based org with 2/3 remote engineers we have 14 regular holidays, most of which can be taken +- 2 weeks of official date, plus 5 days at the end of the year. Start with 12 days PTO (negotiable), 15 at 1 year, 20 at 5 years and 25 at 10 years. All employees acrue 9 sick days per year. Additionally can take 2 paid days off to volunteer at a non profit or educational institution. There is also a sabbatical program, but it's unpaid and for up to 2 months. This can be taken every 3 years. Parental leave is 12 weeks of 100% pay.

If taking all possible days, in the first year you have 42 days off, up to 55 by year 10

There are dumps available that already have lots of metadata extracted and the content available as plaintext without wikimarkup or html. Specifically these are the dumps of the search indices used at wikimedia:

https://dumps.wikimedia.org/other/cirrussearch/current/

These dumps contain a json line per article and are formatted roughly like so: https://en.wikipedia.org/wiki/Hacker_News?action=cirrusdump

What is being used to tune the search results, if user tracking isn't involved? I'm of course guessing here, but I would assuse no tracking means no log of queries and no log of clicked results?

Isn't safe harbor something else? safe harbor explicitly refers to the DMCA but this article seems to be about an older provision of the Communications Decency Act

I've also been building a learning to rank plugin for elasticsearch, we might want to sync up.

I hadn't set it up to sync to github as it was just internal development, but i've started the sync and it will show up at https://github.com/wikimedia/search-ltr soon.

It's got a bit more of the integration with elasticsearch put together, including storing models in cluster state and a rest interface for managing them. It's a bit more of a direct port of the solr plugin rather than a rewrite from the ground up so there are also some oddities that don't yet make sense. Refactors will certainly be done. It's also tied a little less directly to RankLib, such that i can convert and load in MART models trained by lightgbm or xgboost which have done pretty well in my offline tests and are able to utilize resources on my training machine much more efficiently than ranklib's LambdaMART (although in terms of results, the ranklib implementation is pretty good).

Being that this is a SVM, which is typically evaluated as a simple linear sum of weights, I imagine they reimplemented that in the application layer. Would be curious how they handled the normalization steps (reimplement that as well?)

Try a different quality? Best quality on amazon video is 7GB per hour, works out to about 150 hours a month. Sure this is a lot of video, but spread across a household with say 4 people with their own screens and viewing habits is under 1.5 hours per day per person

You could provide them manually, but then any schema upgrade becomes a big pain. Wikimedia, as one example, uses versioned schemas. As such each version is immutable and can be pulled from the cache. Each kafka message has a null byte, and then a long version number prefixed to indicate how it should be decoded.

https://github.com/wikimedia/analytics-refinery-source/blob/...

> I wonder what they're using to retrieve that data for analysis later on. I do something very similar to this, but having to sift through millions of messages for a given time period, to find a subset of said messages is kinda annoying.

It looks like hive or spark, depending on the use case. The data is also loaded into Druid when looking at statistics, rather than getting full data about individual messages.

> It's a good thing they didn't use Confleunt Camus. -shudder- It supports Avro-Over-Kafka out of the box, on the caveat that every single time it reads a message off kafka, it pings the schema registry to get the schema for it. That's great and all, until you've got thousands of messages per second.

They are using camus, much of the post is dedicated to it. It looks like they are also running avro over kafka+camus for some application logging, but at a lower volume (~10k messages/sec peak)

There is a range immediatly below the number, which seems pretty reasonable to me. For my place it shows a range from -8% to +10%

It perhaps depends on where you work, but here is one example of levels of promotion while still being a programmer and not a manager (at google):

Software Engineer 2

Software Engineer 3

Senior Engineer

Staff Engineer

Senior Staff Engineer

Principal Engineer

Distinguished Engineer

Google Fellow

Senior Google Fellow

Fwiw my 486dx2-66 had a fan, and when that fan died we noticed because of corrupted memory crashing the system. Compared to modern CPU coolers it was downright tiny, but required.

I looked at dbpedia, but it was non-obvious to me what statements to use. We can also use SPARQL with wikidata, although the coverage isn't particularly great.I threw together an example query for medalists and michael phelps doesn't make the list, because he doesn't have the appropriate participant of/award received statements:

https://query.wikidata.org/#SELECT%20%3Fhuman%20%3FhumanLabe...

  SELECT ?human ?humanLabel ?count WHERE {
    {
      SELECT ?human (COUNT(*) as ?count) WHERE {
        ?event wdt:P31 wd:Q18536594 . # All items that are instance of Olympic sporting event
  
        ?medal wdt:P279 wd:Q636830 .    # All items that are subclass of Olympic medal 

        ?human p:P1344 ?participantStat . # Humans with a participant of statement
        ?participantStat ps:P1344 ?event . # .. that has any of the values of ?event
        ?participantStat pq:P166 ?medal . # .. with the award received qualifier of any of the values of ?medal
      }
      GROUP BY ?human
    }
  
    SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
  }
  ORDER BY DESC(?count)
  LIMIT 100

EDIT: The dbpedia search should be something like: http://dbpedia.org/sparql?default-graph-uri=http%3A%2F%2Fdbp...

  SELECT ?human ?count WHERE
  {
    {
      SELECT ?human (count(*) as ?count) WHERE {
        ?event rdf:type dbo:OlympicEvent
        {
          ?event dbo:bronzeMedalist ?human .  
        } UNION {
          ?event dbo:silverMedalist ?human
        } UNION {
          ?event dbo:goldMedalist ?human
        }
      }
      GROUP BY ?human
    }
  }
  ORDER BY DESC(?count)
  LIMIT 100

Because i was randomly curious, can extract this data from the structured html with some dom selectors in a similarly haphazard way:

Start with: https://en.wikipedia.org/api/rest_v1/page/html/List_of_Olymp...

Run this js one-liner: [].slice.call(document.querySelectorAll('table[typeof="mw:Transclusion mw:ExpandedAttrs"] tr td:nth-child(n+2) > a:nth-child(1), table[typeof="mw:Transclusion mw:ExpandedAttrs"] tr:nth-child(3) td > a:nth-child(1)')).map(function(e) { return e.innerText; }).reduce(function(res,el) { res[el] = res[el] ? res[el] + 1 : 1; return res; }, {});

The result is an object with the medalists as keys, and the count as values. JS objects are unordered so sorting is left as an excercise for the reader.

see also the SPARQL search: https://query.wikidata.org/

Oculus DRM doesn't work like that at all. Anyone can program against the oculus API and release software for it. It's not like there is a standard VR api (OpenVR is anything but, it's controlled by a single company with their own intentions).

Whats the current price for a 1080 delivered by Friday?

php7

s f g x = f x (g x)

k x y = x

b f g x = f (g x)

c f g x = f x g

y f = f (y f)

cond p f g x = if p x then f x else g x

fac = y (b (cond ((==) 0) (k 1)) (b (s (*)) (c b pred)))

turns out that is the only thing this project is about. There is no web crawler, there is no external content. The grant and the money WMF is spending are being spent to improve internal search at wikipedia.

Compared to the licensing/production costs hosting the front ends is peanuts. At least that's what Adrian Cockcroft used to say before he left.

Any accounts of what happened? This used a whole lot of words to say almost nothing. It basically says it didn't work, but didn't go into any details of why beyond a high level complaint about beurocracy

This looks to be a single machine test, where this video and the poster above specifically talked about running against compute clusters. I don't think a single machine benchmark is going to be nearly as interesting.

Notice the date of last update on that status: Thursday, 23 July 2015, 11:46AM

It didn't read that way to me. It read as if Google had contractual obligations for how they should part ways if Google felt he wasn't performing. He wanted Google to fire him and keep their end of the contractual obligation.

  Well if a single primary school needs 1 bln or anything close to it to
  operate that won't be scale able.

Just to clarify, Zuckerburg has spent over a billion on education related grants and other gifts since 2010, but no one knows what he has spent on this one school. I can't see how he could have spent anywhere close to a billion dollars. Unfortunately the writing is fairly poor so I'm not even sure how many students this school will serve from which to guesstimate a cost.

FTA:

  Chan did not tell the Mercury News how much she and her husband are contributing
  to The Primary School, but when it is fully built, it will serve 50 students 
  in 14 grades (pre-K through 12) plus the families of their 700 students.