Hacker News Reader

Comment by dragonwriter | Hacker News Reader

Note that similar issues were raised with Ruby stdlib, which is being addressed in part with “Gemification” of stdlib, so that all of stdlib (targeted for 3.0, though it's been going on since 2.4)[0] is being moved out to externally-updatable packages that are included by default (default and bundled gems), so that it is still “batteries included” but the batteries are at least replaceable.

Amber's suggestion seems to be in the same direction (though perhaps not as extreme.)

[0] https://www.slideshare.net/mobile/hsbt/gemification-for-ruby...

View on HN

Python cannot be atomized effectively, and the issue is political.

The problem is that I cannot count on being able to install new software in many environments.

If I fight the battle to get centralized IT to install Python, I now have a guaranteed set of standard libraries as well. I'm never going to get permission to install anything other than default. Ever.

Consequently, the standard libraries need to be very complete and very useful.

And, while people seem to love the Rust approach to libraries, I'm not necessarily a fan. Far too many times I have pulled a library that is "obviously" something that a language should consider to be "standard library" and gotten bitten because it was broken. Only VERY core libraries in Rust are guaranteed to work across multiple architectures and OS's.

I think Rust is probably doing the right thing for Rust as "batteries included" is NOT one of its tenets. However, that doesn't make it right for everybody else.

> If I fight the battle to get centralized IT to install Python, I now have a guaranteed set of standard libraries as well. I'm never going to get permission to install anything other than default. Ever.

Can you explain this more? What kind of place do you work? I've had some experience with large, bureaucratic companies, but nothing ever so far as "you can't install any other libraries."

Maybe just me but sometimes after initial install l, I later find a weird non-standard library that I need to do something.

At which point you have to scale the IT wall all over again if you work at a Fortune 500 company.

> At which point you have to scale the IT wall all over again if you work at a Fortune 500 company.

I work at a Fortune 500 company, and the only wall I have to scale when I want to use a library that nobody at our company has ever used before, is to get someone to check and approve the license (typically takes 1-2 days), and import it into our code repos.

I mean I see your point, but not everywhere is as bad as you make it seem.

At least in some places, local install permission may be available, but cross company install permission is much more difficult to get.

As in, it is very difficult to get software installed in general images or on multiuser servers.

There are obviously good reasons for it to be conservative about this.

As far as I understand this. It’s something like “You can install whatever libraries you want, but we’ll only ever install default python on user PC’s”

>What kind of place do you work?

Not OP but same. I'm currently debating with myself whether I should attempt to install PUTTY. Given that port 22 is blocked and it's not needed for my core role it'll be dicey if I get challenged.

Pulling executable code off some repo...no way that is ever officially passing muster. People might do it anyway, but on a personal risk basis.

>Can you explain this more?

Place that are heavy on confidential financial info basically. Practically everything I touch is confidential client data. So employer is naturally jumpy about what's on my laptop software wise.

Ironically the above comes full circle...need putty to get onto a VM in cloud where there are no restrictions and crucially no client data. Nobody cares what I do there - hell they'll even pay for it thanks for MSDN enterprise

Nice! Too bad our desktops are still on Windows 7 at the office.

I've found the built-in Windows 10 ssh client has trouble with tunneling, so I still use the ssh client included with git.

netsh interface portproxy add v4tov4 listenport=8001 connectport=80 connectaddress=1.1.1.1

this isn't useful for tunneling through a remote machine (which is what ssh does)

If you can figure out how to reproduce the issue, I'm sure the team would accept a bug report at https://github.com/PowerShell/Win32-OpenSSH

I've been checking but I can't add the module. Either it's slow to get to Enterprise version or it's blocked. Not sure.

Found a work-around though - Google cloud shell being *nix works fine for SSHing about the place. Gets me around the port fw too

Where I work, there is currently a push to get python on the computers that manage physical equipment operation. These computers are not allowed to connect to the internet, and have extremely limited connectivity to the rest of the business network. Installing anything new on them requires risk assessments like you wouldn't believe, since the consequences of malicious code could easily hit 10s of millions of dollars and a nonzero number of lives.

Maybe a stupid question but can you ship code on the machine? If you can, what is stopping you from including the source of the library that you're trying to 'install'?

Many do, but you can get fired for it.

If your risk assessment says that the exact same tkinter outside of Python stdlib is riskier than in Python stdlib, maybe your risk evaluation process needs reevaluation.

I hear this sentiment frequently. Come on, one software engineer cannot steer the huge ship that is BigCo Risk Assessment. Well, they couldn't do that and the original task.

It might me more helpful to think of these types of external factors as fixed points that cannot be moved and just engineer around them.

You'll burn out if you try to boil the ocean on every business process that doesn't seem "logical" from your cursory examination.

On one hand, this is true. On the other hand, this is being put forth as a reason to not make a change in the entire Python ecosystem, and it's not really Python's job to bend over backwards for shops that have bad risk assessment either.

Yes but it won't get it. And at the end if the day, people need to be able to get work done.

The corporate world is full of stupid things that will never not change, or take years to change.

Where I work the solution was to use a proxy to pypi. Basically an internal pip repo (and docker, npm, maven, everything else...). All internal apps go through the internal repository that creates a local version of the package from pypi. That gives the security / compliance folks a way to block packages with security issues, etc. and at the same time provide the developers flexibility to get most of what is needed.

In a large company this gives the compliance folks a central place to blacklist packages - along with a trail of what systems have downloaded the package to target for upgrades.

Many technical solutiins exist, but the problem is political or organisational.

Agree. At this point it was more a case of executives saying they wanted internal dev teams to use and contribute to open source and supporting orgs to come up with solutions on how that can be possible with a 0-touch approach. That’s what tipped the balance.

As long as you cannot even prove that due to a lacking python code signing infrastructure for packages (wheels can do it, but it is far from wide spread).

And setup.py is a trainwreck, e.g. some packages compile download and compile huge dependencies (e.g. a full Apache httpd...), the default compiler flags may lack all the mandatory security flags (e.g. for using ASLR on python 2.x), or ship their own copy of openssl statically and break your FIPS-140 certification that way...

And since setup.py is a Python file, you can't express build time dependencies properly. Pyproject.toml let's you do that, but it's new, nobody knows about it, and older pip clients don't support it.

I disagree. tcl/tk is written in C and C can be compiled in very very badly indeed (from a security perspective).

Not who you asked, but I work for a large international company, a big 4 professional services firm. I wanted to install anaconda and Jupyter on my machine (data science is not a part of my 'official' job description, but I wanted to see how much of my data exploration workflow I could speed up or automate). I had to go up three separate hierarchy ladders to get sign off. First, my own team, then IT, then our risk and quality team (thanks to our audit practice and a few historical issues with whistleblowers and leaks, the risk guys are pretty much the final arbiter of... Everything)

After about 9 weeks of emails, meetings, and pitches, I finally got Anaconda up and running. A week later, I tried to upgrade the 3rd party packages.. No dice. Blocked by the corporate VPN. I'd need the sign off every time I wanted to `pip upgrade` anything

Needless to say, I do not bother anymore.

Hierarchies and the systems or checks that they serve at places like this exist only to keep some people employed. That has got to suck! For anyone who has a rogue or novel idea will get shot down because it’s too much of a burden in terms of overhead to get any decision made.

I can't imagine myself working in a place like this..I understand there should be a level of checks, however this is just crazy...

I also work at a big 4, and perhaps the same one as you, and I assure you, there is a way. There is always a procedure or some way you can follow. You just need to know how to look it up.

We have our own development team, our own servers, our own freedom to deliver to clients fast without the hassle of the main corporation. How? We talked to the right persons.

Big 4?

https://en.m.wikipedia.org/wiki/Big_Four_accounting_firms

I don't think your company deserves you.

I work for a major defense contractor, and while we have vanilla python we are categorically not allowed to download software off the internet and install it without authorization, even on the unclassified internet-connected corporate network.

Theoretically there's a process for requesting new software and getting it approved, but actually pushing it through requires getting one of my program architects to care enough to file the request (As a mere level 2 engineer all I can do is write it, can't submit), then potentially weeks of followup, for 1 specific version of 1 specific package.

In the case of python packages, perl and the perl packages we need are already approved because a few senior devs got together and pushed them through 10 years ago (was before my time, but I understand it was with quite a bit of arm twisting). It's more time-efficient to just code perl than to fight for python.

It's one of the many reasons I intend to get myself another job for Christmas. :)

As for why the system exists: Cost cutting, in the sense of "the less we invest in infrastructure the more we can divert to sexy hardware for the cameras and shareholder dividends. So long as it's theoretically possible for you to do your work, we don't care how many hoops you have to jump through to do it. And our competition is even worse than us, so we don't have to worry about anyone undercutting."

As a result all our infrastructure is centralized. Programs have to jockey with each other for everything from virtual servers to physical workstations and monitors. Hell the only reason my program has our primary test server is because one of our architects literally overheard a hallway conversation about a program that was spinning down and getting rid of some servers, so he jumped on it.

> Theoretically there's a process for requesting new software and getting it approved, but actually pushing it through requires getting one of my program architects to care enough to file the request

I worked for a much smaller government contractor, but before I left they were moving to a system where you needed approval from the customer in order to get new packages. (For those who don't work in this field, that means you are actually making a request to the contracting representative from the particular government agency for each package you want.) So it wasn't just in-house bureaucracy in the way of progress, and I generally just went without or wrote my own instead of trying to deal with it.

Last June I opened a ticket for our procurement department to have an open source license reviewed. It is still open. Our purchasing process is similar to [0].

[0] https://training.kalzumeus.com/newsletters/archive/enterpris...

If you must work with one hand tied behind your back you are fucked, and your company is even more fucked. Your priority should be supporting your managers who want to get things done in the struggle against centralized IT managers who want to repress aspirations to work.

I'm afraid the standard library has to be aligned with the needs of more normal users who, as already discussed, want to allow libraries to have their own release cycles and to be more "opinionated" and specialized than the standard library would permit.

> the needs of more normal users

I'm afraid users dealing with that sort of bureaucracy are much more normal than you think, if not the norm. They're just usually not the types of folks that are hanging around HN, or they're at least less vocal.

Putting code into the standard library doesn't magically create developer resources to maintain it. Indeed, Amber Brown is saying that many libraries in the Python standard library aren't properly maintained. So it's not clear that standard library policy is relevant to the Rust issues you have.

You could install packages with `pip install --user` to have them install under your home directory.

> > If I fight the battle to get centralized IT to install Python, I now have a guaranteed set of standard libraries as well. I'm never going to get permission to install anything other than default. Ever.

> You could install packages with `pip install --user` to have them install under your home directory.

You might find that the issue isn't necessarily always technical (ie permissions), but policy. A lot of places don't let you arbitrarily download software off the internet onto a system. Pp's point is that having a full feature stdlib let's you only need to crank the policy/approval process once, rather than once for every dependent lib.

As I understand, the Ruby model -- that is suggested for Python above -- is to include the libraries by default but to keep them as separate pypi packages.

In this case, even If you can't upgrade, you still get the same libraries but perhaps older versions.

> The problem is that I cannot count on being able to install new software in many environments.

The approach Ruby is taking with gemification and default and bundled gems for standard libraries is equivalent to the traditional standard library if you can't install updates, but superior in other cases.

This is close to what Rust is doing, and it's working pretty well, apart from shocking newcomers who expect libstd to be useful on its own.

In Rust, libstd is mainly for interfacing with the compiler and providing interoperability between packages (crates). The wider crate ecosystem is the real standard library, since external crates are as easy to use as the standard library.

For example, the libstd doesn't even have support for random number generation. There's a rand crate, which is now on 6th major breaking version. That's perfectly fine, because multiple versions can coexist in one program, and every user can upgrade (or not) at their own pace. And the crate was able to refine its interface six times, instead of being stuck with the first try forever.

This is interesting. Go seems to have the complete opposite stance. The stdlib are some of the most useful and well written packages you can use in the Go ecosystem, and then you have the "extended" standard lib which isn't 100% in the language yet, and even further sometimes concepts from useful community packages make it into the std lib.

As to why this is the case, I think maybe this is enabled by Go's backward's compatibility focus and encouragement to upgrade early and often, and the community's focus to utilize small interfaces sometimes from the stdlib itself, like io.Reader and io.Writer, or http.Handler. Added to that w.r.t. using the latest and greatest, most Go users frequently are using the latest version of Go even in production (per the go experience surveys).

I am sure it also helps that Google pays people to develop, maintain, and improve the stdlib.

This approach is useful for a while, but once something is in a stdlib, its interface is frozen forever.

(I am trying to tie the two concepts together from your reply, so I am not saying this to come off as combative but understand your objection)

- What would you define as a while? I think Go has done well with it, given that it is almost 10 years old now at this point, and in reality, was in development internally at Google years before that.

- Do you feel like an interface being frozen forever is particularly a bad thing in of itself? What if the interface does a really good job describing the thing, whatever it may be? For example, Go's io.Reader/io.Writer interfaces.

I agree sometimes an interface being frozen is bad, but for example when Go standardized the context package, it simply added "Context" to the existing functions that now take a context (e.g. in the database/sql package, you have the old, Exec, Query, QueryRow functions, and after 1.8 you have ExecContext, QueryContext, QueryRowContext, which some people may view as a reason to have method overloading, but I view as adding better clarity.

The way to answer your two questions is to combine them, the definition of “a while” depends on how good a job the interface does. If the interface is really good, then it can last a really long time (perhaps as long as the language lasts).

I read some discussion the other day about some ways in which the Any type in Rust isn’t as flexible as it could be, since it is frozen the only way to improve it would be to introduce a new name, such as Unknown. Similarly in C# along with adding async support, the standard library added async versions of many methods, eg Read now also has a ReadAsync partner.

It does seem that having multiple names for basically the same thing adds a small but tolerable level of overhead to a language. At least if as much as possible is moved out, then projects can choose to only use the latest versions, and live in a world as if past versions never existed.

Go was and is by design a "boring" language. The core designers didn't have much trouble looking at decades of prior art and getting it mostly right.

Rust libraries should be expected to take a few tries to get right, especially earlier in its lifecycle. There's more possibilities and less experience in the language.

You can see a similar effect in Haskell, which has iterated many basic bits of functionality many times over.

No doubt Go's stdlib is useful, and there's plenty of things it got right. However, it's not immune to making some mistakes and having to freeze them forever. The more functionality you add, the harder it gets to get it perfect on the first (and only) try. Search for "deprecated site:https://golang.org/pkg/" finds various issues ranging from cosmetic mistakes to entire packages being deprecated.

    CompressedSize     uint32 // Deprecated: Use CompressedSize64 instead.
    CompressedSize64   uint64 // Go 1.1

    // Deprecated: HeaderMap exists for historical compatibility
    // and should not be used.

Requirements may change over time, so even getting something perfect now is not a guarantee it will last (e.g. pre-UTF-8 languages froze byte-oriented or UCS-2 strings, even though these were good decisions at the time).

Sometimes improvements are not worth the cost of deprecation and replacement, so things are just left as they are. For example, an HTTP interface designed for request-response HTTP/1 works for stream-oriented HTTP/2, but support for push, prioritization and custom frames is bolted on. Packet-oriented HTTP/3 will add even more stuff that will have to be retrofitted somehow to the old model. Libraries can come and go, but std can't just throw away an old interface and start over.

But then you have six versions to have security fixes, how do you do that?

You release fixes for older versions according to semver.

Having incompatible libraries solving the same basic problems is absolutely not fine. Over time, this will become a huge issue for composability. C++, for instance, already is in this kind of mess with things like STL containers versus Qt comstainers vs. homegrown special case (optimized containers) and heaps of additional libraries building upon each. There are other good examples in other programming languages as well, mostly older ones.

The example of random number generators is a good one, too. There are a lot of applications that require (reproducable!) PRNG sequences and sometimes you have to share PRNGs between modules. Now, looking at the rand crate, I see that the prng part of it was recently mucked around with. If I have two 3rd party modules that I require to share a PRNG that I control (say, noise generators for procedural textures), I cannot compose them if one of them uses the old and one of them uses the new version of the library.

When a crate is part of a public interface, that's harder, indeed. Crates solve it in a few ways:

• crates that expect to be used for interoperability are often split into smaller crates (like API and back-end, or low-level API and high-level API), so that they can evolve some parts without breaking others.

• sometimes breaking changes are technically breaking, but easy to upgrade (e.g. methods renamed, args reordered). In that case most users catch up quickly.

• in desperate cases, a new version can import its own old version and re-export old structs and interfaces that haven't changed, so they're compatible across major versions.

• proper sharing and composition should be done via traits, so that you can implement a trait for any number generator, not just one version of one implementation.

I've never coded Rust - is there any distinction between a really important crate used by millions of people and something really obscure with 3 users? Are all the crates subject to security audit?

Anyone can upload to crates.io without review.

There is no technical distinction. The community is working on a WoT/review tool (cargo-crev), but in the meantime you can see who has published the crate and who uses it. The de-facto standard crates are maintained by Rust team members or well-known authors.

ok. So it's kind of informal at the moment.

Maybe in the future we'll see more hacking of libraries (people managing to deliberately sneak exploits in) and in response stronger lockdowns on important library code.

Yes and no. Until the Ruby maintainers give the gem versions of the internal modules different names you will get bizarro conflicts when using a gem after the "batteries included" version has already loaded.

I don't know how many hours I've spent battling Psych errors because of this very thing, but it's way too many. Calling the gem something, anything else, would solve the issue.

It's great that they're unbundling a lot of things, but there's still some serious friction between external and internalized versions of these gems.

For Ruby, EventMachine sub-universe is really in bad shape. EventMachine is creaky and old. Event-aware packages are in short supply and are usually woefully out of date, unmaintained.

This works in Perl, a language that almost everybody criticizes or hates. I've never had any issues in upgrading core packages.

Reply on news.ycombinator.com