se4u

I am surprised that people aren't using google colab pro/pro+ in this context. You basically get access to multiple A100 for $10/month and with some simple javascript tricks, you can get a session to last for 24hrs at least.

Pro+ is more expensive at $50/mo but it allows for more simplified background execution. if you are only just getting started and don't expect to be training for multiple months, then colab or other cloud-notebook providers are really great to start.

Hi OP here. Yes you are right about the asymmetrical nature of the penalties. Actually I do handle this case by using an asymmetric `shrinkage` on the error. See the function called shrinkage in the notebook.

Good day :)

Hi OP here.

To answer why PID, basically somebody I follow asked this question on twitter, and I thought yeah why not seems like a reasonable thing to try :)

Actually I do conclude that PID is not quite the right thing for this problem. For me the learnings from making a PID sort of work for this problem were:

1. must use the right error function. like frequency not time. 2. must use shrinkage on the error to handle discrete number of server. 3. have to run controller at a multiple of server delay to avoid perturbations.

Also I discuss the basic assumptions that a PID controller makes that are suboptimal in the video.

Before someone asks about why can't this work be automated, here's the FAQ where she answers this question.

https://scienceintegritydigest.com/frequently-asked-question...

Yes, definitely. I think the app interface is really the innovation here. That and the fact that one can make human readable salts. So I can use "kortex@kortex.com" as a salt and now anyone can check just by reading that "kortex" must have made this hash.

So actually this wont be a problem. Basically if they do this then each account that they create will only have very few published predictions. So it will be like a coin which comes up head twice after two tosses. The public will not trust such accounts with low number of correct predictions.

To build trust you'd need to have a high count of revealed predictions in any given account.

That said people can try to game the system by uploading a very large number of predictions programmatically, so I think a rate throttling will have to be enforced. Any account can't make more than one prediction a day or something like that.

Also note that the issue you raised will also affect OP's implementation.

One alternative implementation of this which is very simple is as follows.

1. User makes a public declaration to always make a prediction using some platform. Like they release a tweet saying all future predictions will happen on futuu.re (or on a Blockchain)

2. They only release the sha hashes of their prediction to futuu.re, along with a date for that prediction, and users keep the true prediction with themselves. At any time before deadline they must reveal the content to the world and tell futuu.re about it. Futuu.re verifies that the sha matches the prediction and keeps an easy record. All predictions must start with a preamble.

3. futuu.re maintains three counters for a use, one showing the number of predictions made till that time, second showing predictions that should be revealed till that time, third showing the number of predictions that have been revealed . This establishes the long time accuracy of the user.

This way when someone reveals their pok then the public will know whether they tried to fool people because the users trust ratio will go down if they make a lot of predictions without revealing them. This is also more secure since the actual content is never stored on the server.

The only thing that needs to be done is to use a content formatter/validator at the time of hash creation for stylstic reasons and to make it convenient for public to verify the accuracy of the user.

@averagewall already explained what depth 3 is.

The explanation of poly time is a little bit lengthier. Let me briefly explain the PAC learning paradigm in the context of this paper.

Consider the set of all depth 3 neural networks with n dimensional input and k dimensional hidden layer whose weights have total norm 1. Let this set be called H. Note that this is a well defined set, parameterized by the weights of the neural networks.

Let ∊ be an arbitrary positive error value and let δ be an arbitrary prob value. We are interested in ∊ ≈ 0 and δ ≈ 0. The solution of the PAC learning problem is an algorithm, say A, which takes a dataset D as input, generated from a dsitribution Π, and which runs in time that is polynomial in the size of D, and spits out a specific neural network h that lies in H. Furthermore, no matter what Π was, the probability that the neural net h, that A produced, has error that is at max ∊ above the optimal error achievable by any member of H, should be more than 1-δ. This is the PAC learning paradigm.

The paper assumes that Π is a distribution over the n-dimensional unit sphere, so all examples have the same euclidean norm and constrains the weight vectors to have unit norm, AND WITH THESE ASSUMPTIONS, it gives a algorithm that requires |D| = poly(n, k, 1/∊) and runs in time poly(D).