brahbrah

If you have a mortgage, it ain’t half bad

> I could not imagine this ever happening when dealing with the IRS.

Have you tried? I haven’t tried it with the IRS, but I have tried asking my states permitting and planning department about the legality of various rental schemes for a property (that I didn’t own) and they were happy to look up the permits and tell me that indeed what I was asking about was not legal for that property. And further they let me know that tons of people do it anyway and that they don’t really check, but it’s a risk.

> And is the only source I know to offer 2 weeks of hourly forecasts

Enjoy the data directly from the source producing them.

American weather agency: https://www.nco.ncep.noaa.gov/pmb/products/gfs/

European weather agency: https://www.ecmwf.int/en/forecasts/datasets/open-data

The data’s not necessarily east to work with, but it’s all there, and you get all the forecast ensembles (potential forecasted weather paths) too

Would you pay it off early if you have a 2.5% interest rate and could get a guaranteed 5+% on a CD?

> If they weren't, Delta would say so, since it makes them look better. That they won't say means the parts were used in service.

More likely they would just never comment whether it makes them look better or not, because otherwise for the reason you stated you could always gather the information they didn’t want you to have in the negative case.

> have to be accessed through python

It’s because most of the people doing these computations don’t have the capacity to become experts in multiple fields. They understand the math and analytics very well, and they expend all their time thinking about that, not about type systems, memory management, etc. Python lets them code without having to think about a lot of that stuff so they can focus on the things they care about. These aren’t computer scientists or programmers, they’re meteorologists, astronomers, oil and gas analysts, investment bankers etc. That’s why some truly great computer scientists and programmers invested their time into building these tools for python vs other languages.

The explanation I’ve always heard for why New Zealand was settled by Polynesians so late is that they preferred to start their voyages against the currents when they were fresh so that they wouldn’t have to fight the currents on their way back if they weren’t able to find any new lands to rest and recharge

Mining is not needed to keep the grid reliable or smooth imbalances. You can achieve this by dispatching or cutting off the marginal generators needed to serve the load in real time (the marginal generators serving the load of these facilities are already quick response generators that don’t benefit from already being on and switching from mining facility to demand bursts elsewhere). They increase demand and necessarily move dispatch up the supply stack, increasing the marginal power price which sets the clearing price for all megawatts in the iso auctions and consequently the power prices for all customers. They also increase congestion by requiring more megawatts to flow increasing the congestion price component of the nodal LMPs (locational power prices).

Here is how prices are set in an iso auction. This is from iso New England. But works the same way in all isos including ERCOT.

https://www.iso-ne.com/about/what-we-do/in-depth/how-resourc...

It doesn’t matter to me if the mining operations are running or not, but they’re not helping the grid. Citing a crypto company on this is comically biased.

Dask added an actor model after seeing it in Ray. I’ve used dasks in some computationally intensive applications (already had existing dask infrastructure which is why we didn’t go with rays)

This is why you see a lot of 1 year and 1 day sentences. Any federal sentence 1 year or less must be served 100% in full. If it’s 1 year and 1 day it’s eligible for the time reduction credit and you can serve less than 1 year.

Vermont also consumes ~3x more energy (all energy, not just electricity) than it produces and has the lowest energy consumption of any state.

https://www.eia.gov/state/?sid=VT

Just speaking it off my ass here, but it could be the scale of those companies that make those teams viable. Like let’s say you have some standardized systems that can cover the use cases of 20% of your teams. If you have 1000 teams vs 10 teams, covering 200 teams might make economic sense, but 2 teams wouldn’t

To be fair pandas was never meant to be a replacement for sql, it was meant to be a replacement for excel in financial models. Which it still excels at (pun intended).

So something like this?

    def add(df1, df2, meta_cols, val_cols=None):
        # join on meta cols
        # add val cols (default to all non meta cols if None)
        # return df with all meta and val cols selected

In theory I think that's fine. The problem is that in practice this will cause a lot of visual noise in your models, since for every operation you would need to specify, at least, your meta columns, and potentially value columns too. If you change the dimensionality of your data, you would need to update everywhere you've specified them. You could get around this a bit by defining the meta columns in a constant, but that's really only maintainable at a global module level. Once you start passing dfs around, you'll have to pass the specified columns as packaged data around with the df as well. There's also the problem that you'd need to use functions instead of standard operators.

One thing that would be nice to do is set an (and forgive me, I understand the aversion to the word "index") index on the polars dataframe. Not a real index, just a list of columns that are specified as "metadata columns". This wouldn't actually affect any internal state of the data, but what it would do is affect the path of certain operations. Like if an "index" is set, then `+` does the join from above, rather than the current standard `+` operation.

In any case I realize this is a major philosophical divergence from the polars way of thinking, so more just shooting shit than offering real suggestions.

(Taken from an old comment of mine)

If you were to say “pandas in long format only” then yes that would be correct, but the power of pandas comes in its ability to work in a long relational or wide ndarray style. Pandas was originally written to replace excel in financial/econometric modeling, not as a replacement for sql. Models written solely in the long relational style are near unmaintainable for constantly evolving models with hundreds of data sources and thousands of interactions being developed and tuned by teams of analysts and engineers. For example, this is how some basic operations would look.

Bump prices in March 2023 up 10%:

    # pandas
    prices_df.loc['2023-03'] *= 1.1

    # polars
    polars_df.with_column(
        pl.when(pl.col('timestamp').is_between(
            datetime('2023-03-01'),
            datetime('2023-03-31'),
            include_bounds=True
        )).then(pl.col('val') * 1.1)
        .otherwise(pl.col('val'))
        .alias('val')
    )

Add expected temperature offsets to base temperature forecast at the state county level:

    # pandas
    temp_df + offset_df

    # polars
    (
        temp_df
        .join(offset_df, on=['state', 'county', 'timestamp'], suffix='_r')
        .with_column(
           ( pl.col('val') + pl.col('val_r')).alias('val')
        )
        .select(['state', 'county', 'timestamp', 'val'])
    )

Now imagine thousands of such operations, and you can see the necessity of pandas in models like this.

The first issue I have with it is that they've now convinced a large portion of people that read this article that a very good tool is not as good as it actually is. This is a disservice to the great engineering that has gone into it.

The rest of my issue with it is hypothetical. I don't care what he does at work, but I would imagine if I was that dude's manager and he convinced me that he put in all this work and determined that the best path forward is to introduce a brand new language and tool chain into our environment to maintain (obviously not as big a deal if it was already well engrained in the team), and then I come to find out that he could have gotten even better results by changing a few lines with the existing tools, that I would have to reevaluate my view of said developer.

This is a very valid point that I can’t disagree with. I’ve gone through the pain of learning that subset of the language decently well, but also been lucky enough to work at places that compensate very well for that knowledge.

So in addition to what akasaka said (another thumbs up for line profiler from me, great tool) this isn’t a problem with linalg.norm being slow. It’s plenty fast, but calling it thousands of separate times in a Python loop will be slow. This is more just about learning how to vectorize properly. If you’re working in numpy land and you’re calling a numpy function in a loop that’s iterating over more than a handful of items, chances are you’re not vectorizing properly

No, their v1.5 is still calling norm on every polygon. They’re still using it wrong

On Google colab

    import numpy as np
    import time


    vals = np.random.randn(1000000, 2)
    point = np.array([.2, .3])
    s = time.time()
    for x in vals:
        np.linalg.norm(x - point) < 3
    a = time.time() - s

    s = time.time()
    np.linalg.norm(vals - point, axis=1) < 3
    b = time.time() - s

    print(a / b)

~296x faster, significantly faster than the solution in the article.

On Google colab

    import numpy as np
    import time


    vals = np.random.randn(1000000, 2)
    point = np.array([.2, .3])
    s = time.time()
    for x in vals:
        np.linalg.norm(x - point) < 3
    a = time.time() - s

    s = time.time()
    np.linalg.norm(vals - point, axis=1) < 3
    b = time.time() - s

    print(a / b)

~296x faster, significantly faster than the solution in the article. And my assertion was supported by nearly 20 years of numpy being a leading tool in various quantitative fields. It’s not hard to imagine that a ubiquitous tool that’s been used and optimized for almost 20 years is actually pretty good if used properly.

Yea sorry that was just pseudo code. You want it in this form:

    centers = np.array([
        [3, 3],
        [4, 4]
    ])
    point = np.array([3.5, 3.5])
    vals = centers - point
    np.linalg.norm(vals, axis=1)

Agreed it was well written, but kinda pointless though since they could have “solved” the problem using the existing tools in a couple lines of code without any new deps. All that content annd profiling and they missed the fact that they were using numpy wrong.

They would have gotten the same performance in python with numpy if they did it like this instead of calling norm for every polygon

centers = np.array([p.center for p in ps]) norm(centers - point, axis=1)

They were just using numpy wrong. You can be slow in any language if you use the tools wrong

This was a silly and unnecessary optimization. He’s just using numpy wrong.

Instead of:

for p in ps: norm(p.center - point)

You should do:

centers = np.array([p.center for p in ps]) norm(centers - point, axis=1)

You’ll get your same speed up in 2 lines without introducing a new dependency

Does it support multiindex columns?

fn_graph: https://github.com/BusinessOptics/fn_graph

(My favorite one ^)

pyungo: https://github.com/cedricleroy/pyungo

Loman: https://github.com/janushendersonassetallocation/loman

Tributary: https://github.com/timkpaine/tributary

Not to take away from anything you’ve done here, you guys have put a lot of great effort into this, but this paradigm is not “new”. It’s a common modeling paradigm in banks and hedge funds at least. I’ve built/worked on frameworks based on exactly this concept at 2 previous firms. Here’s some open source examples of the same concept: pyungo, fn_graph, Loman

I don’t think it’s so black and white. Depends on what you’re doing. If you’re doing lots of cross dataset operations like econometric and physical system modeling polars can get a bit too verbose. Eg how would you do the following two pandas operations in polars?

    capacity_df - outage_df

    prices.loc['2023-01'] *= 1

There’s many workflows and models that do thousands of these types of operations.

Fwiw I actually do prefer polars for standard long format relational operations. But sometimes it’s just more convenient to work with data in other ways. Another example:

Polars:

    polars_df.with_column(
        pl.when(pl.col('timestamp').is_between(
            datetime('2023-03-01'),
            datetime('2023-03-31'),
            include_bounds=True
        )).then(pl.col('val') * 1.1)
        .otherwise(pl.col('val'))
        .alias('val')
    )

Pandas:

     pandas_df.loc['2023-03'] *= 1.1

I see your:

    (
        polars_df1
        .join(polars_df2, on=['state', 'county', 'timestamp'], suffix='_r')
        .with_column(
           ( pl.col('val') + pl.col('val_r')).alias('val')
        )
        .select(['state', 'county', 'timestamp', 'val'])
    )

and raise you:

    pandas_df1 + pandas_df2