I said this already 10 years ago: We don't need more cloud computing but need to empower users end devices again. Jupyter is typically operated on powerful notebooks and not on mobile devices.
If you’re working with data of any significant size at all it then it doesn’t matter how fast your user device is—it’s so much cheaper (time and network egress costs) to send the computations from a user device to the cloud than to pull tens-thousands of GB of data to your local machine. Moreover, I don’t know of many local machines with tens of CPU cores, hundreds or thousands of GB of RAM, or tens-hundreds of TB of SSD for handling that computation quickly.
User devices are great for very small data, but I don’t see the point for larger datasets.
As an anecdote, I work on a Jupyterhub managed service offering with customers in both the private and public sectors and our data sizes are pretty much all in this range.
That sounds like selection bias.
You want to train the next imagenet model? Analyse 100Gig database? Probably not the correct tradeoff.
You want 20 students to have a perfectly consistent instant-start-up python environment? Definitely the correct tradeoff.
You want to try some python methods, write tests, ... Very short latency is going to help you more than throughput is.
I don't agree that we don't need more of cloud, IMO we do, but we need to focus on personal computing much more than now, which is the general theme of the GP comment.
Not everything is big data.
If you're spending the time and effort to rent these big servers, why not spend the 5 percent of the effort and do it right?
Jupyter exists mostly because analytics/math guys are too lazy to spend a day learning software development practices. Must be some sort of us-vs-them point of misplaced pride.
Bahahahaha, yup those dang lazy mathematicians just shooting themselves in the foot and forcing us to deal with it! /s
Your preferences for software development are irrelevant. The value is in delivering the math to the end user. Using GCP, kubernetes or JavaScript to do that is an implementation detail. Sorry to tell you this, but you're a servant to those dang lazy analysts and without their insights, you're worthless.
The guy above correctly said that Jupyter is nice for ad-hoc analysis kind of work. The problem is that when you've reached terabytes of data and tens of cores you're not "ad-hoc" anymore.
Too often the math guys try to avoid responsibility by claiming they're doing "ad-hoc" work when they're clearly not anymore. It's convenient, yes, but leads to a bad place eventually.
There isn't some golden truth of software development that analysts are too lazy to learn and implement. There's the problem and then there's solutions. Complaining that Jupyter-based development doesn't adequately accomdate version control or some other whistle commonly used in software development is some peak developer entitlement.
At which point you'll dump the hot mess in somebody else's lap, and the whole thing will be rewritten from scratch.
If that's your thing, then go for it.
Not in any meaningful way, the code to explore a 50MB data set on a little machine looks the same as the code to explore a 50GB data set on a big machine, so of course one doesn’t need version control more than the other.
> Too often the math guys try to avoid responsibility by claiming they're doing "ad-hoc" work when they're clearly not anymore. It's convenient, yes, but leads to a bad place eventually.
This is a moralistic argument. The economic argument is that the primary artifact of research is insight, not code, so getting to that novel insight as fast as possible is paramount. Putting version control, tests, or other ceremony into the exploration loop is a pointless cost. Productionizing that insight can happen later in a more traditional software development workflow. It’s similar to how we write proofs of concept without the intensive testing effort that we would go into if we were writing production software.
Oh, yes you are. "Ad-hoc" doesn't imply small. It implies one-off exploration and creation. You have no less of a need to do that with terabytes of data than you do with megabytes.
It's easy to convert a notebook into a script with version control, reproducibility, and automation.
Version controlled Jupyter notebooks running in an automated environment (eg Kubernetes), with repeatable test data loading into a processing environment (eg. MinIO or Spark), is quite commonplace. Making it even easier with WASM makes sense.
What about Jupyter impedes good practices? That it empowers ad hoc exploration at all? It is merely an IDE tailored to sharing interactive text and code. To me it is one of the most exciting ecosystems for modern software development (and I’ve been developing software for 30 years).
It has a few quirks, but works quite good for daily use.
I try to work mostly with "py:percent" script files now, in the same jupyter notebook style but without saving outputs.
Then, we can use the on-demand nature of cloud services to reduce their power consumption. Simultaneously we can move that consumption into renewable-powered datacenters. This is literally better for everyone.
Think we've just got better at runtimes over time (JIT, intermediate formats etc.) and WASM was designed to be good for this rather than needed to work with an already exising bytecode
The reflection API has two issues, a security check each time you call a method and the arguments being transformed to objects. The code is still generated to assembly code but the assembly code is slower because of that overhead.
"Everything Old is New Again: Binary Security of WebAssembly"
https://www.usenix.org/conference/usenixsecurity20/presentat...
So, enjoy the 2nd coming of applets/flash,
https://dotnet.microsoft.com/apps/aspnet/web-apps/blazor
.... favourite stack compiled into WASM.
- Applets tried to render their own GUI, Wasm doesn't and defers to the browser.
- applets needed a big, slow to start and resource hungry VM. Wasm is running in the same thread your JS is also running in, it's light, and loads faster than JS
- Java and flash were plugins, which needed to be installed and kept up to date separately. Wasm is baked into your browser's JS engine
- Wasm code is very fast and can achieve near native execution speeds. It can make use of advanced optimisations. SIMD has shipped in Chrome, and will soon in Firefox
- The wasm spec is very, very good, and really quite small. This means that implementing it is comparatively cheap, and this should make it easy to see it implemented by different vendors.
- Java was just Java. Wasm can serve as a platform for any language. See my earlier point about the spec
So it's apples and oranges. The need to have something besides JS hasn't gone away, so their use cases might be similar. The two technologies couldn't be more distinct, though.
The browser renders it's own GUI too, it's not OS native
The browser uses lots of resources too.
The browser is kind of a plugin to the OS and must be updated separately.
Java nowadays is pretty fast too.
Java VM serves a platform for multiple languages like Scala, Kotlin, Clojure.
Let's face it, the browser is the new JVM and a soon it gets the same permissions like the JVM to access the file system and such, we get the same problems.
Like... never?
We get better systems as we get more experience. That's why C# was better than Java, Java today is better than Java was when C# launched. That's why we now have amazing languages like Rust and also that's why the same problems will never be the same given we have a ton of experience with VMs, docker, sandboxing in browsers etc.
> https://developer.mozilla.org/en-US/docs/Web/Security/Subres... : "Subresource Integrity (SRI) is a security feature that enables browsers to verify that resources they fetch (for example, from a CDN) are delivered without unexpected manipulation. It works by allowing you to provide a cryptographic hash that a fetched resource must match."
> There's a new Native Filesystem API: "The new Native File System API allows web apps to read or save changes directly to files and folders on the user's device." https://web.dev/native-file-system/
> We'll need a way to grant specific URLs specific, limited amounts of storage.
[...]
> https://github.com/deathbeds/jyve/issues/46 :
> Would [Micromamba] and conda-forge build a WASM architecture target?
“Web” assembly is a bit of a misnomer. It’s an IR at the end of the day and can be run without a browser[1]. But your other points could be true one day if the de facto WASM runtime becomes bloated or decides to ship with some GUI renderer.