Ask HN: 0.000068 MSE for TensorFlow Model? | Hacker News Reader

Ask HN: 0.000068 MSE for TensorFlow Model?

1 pointjzombie14 comments

Question, just for comparison purposes, is anyone getting an MSE like this in any of their models for validation metrics?

Mine is doing some predictions, and over the past few weeks of iterations I've gone from 0.2 down to this, and trying to push it lower.

I know this all depends on context and the number means nothing by itself, but would be curious to know if anyone has a really low number, much lower than this.

View on HN

This is almost a meaningless, the MSE depends on the model structure and can't be compared arbitrarily.

If you had a single perceptron/single layer model, its a fantastic MSE.

I thought MSE was tied to the predictions it makes and not the structure itself (i.e. if you get the same value out for the same input, using different structures, you should at least get the same MSE).

without more details, this sounds like overfitting or training data leakage

I can assure the data is not shuffled and is ordered in chronological order (time series model).

However at present I've only got training/validation splits and not an additional testing.

One thing you might try is creating some brownian noise and train on that. If you get really low errors it would be sus

Thanks for the feedback. Will try that. Been mostly using Keras Tuner and Tensorboard to slowly get to the point I'm at and have had some wild results in the process.

The best test will be to run it against the real market, but without actual money. With results like that, I would not wager my dollars

Yes, actually I was planning on training a separate model to determine buy/sell signals, using this prediction model as possibly part of a transfer learning process, and do paper trading w/ it to start w/.

Fun story, my first market model (2003) was a simple feed-forward network trained using an evolutionary algorithm. This led me to LSTM and backprop, which required calculus, and pushed me to go back to college.

I've been thinking about building something again, based in P/E ratio and some portfolio mgmt strats I used for a college paper. Got the best returns for the semester but only a C grade because the prof didn't like my mechanical approach. I'm all about removing the fear and greed from the decision making.

I started w/ an LSTM and wound up w/ a mixture of transformers, CNNs and skip connections.

I think my next attempt will be a handful of hand crafted rules on daily data. I'm definitely never going to try day trading scale strategies or try to pick good points intraday to buy. It should be resilient to the price difference over a day or two

> I think my next attempt will be a handful of hand crafted rules on daily data. I'm definitely never going to try day trading scale strategies or try to pick good points intraday to buy. It should be resilient to the price difference over a day or two

Not sure why HN won't let me reply directly w/o replying this way at times.

When threads get deep, there is additional delay in the normal place you find the reply link. From what I understand, this is done because most often this happens when people arguing.

If you click on the timestamp of the message you want to reply to, you can reply sooner, like I am here :]

Nice! Thanks for the tip.

Anyway, I appreciate this conversation and am going to make some coffee.

Reply on news.ycombinator.com