Goodbye, Node.js Buffer

I had to convert a ReadableSteam to base64 recently, and I was shocked at how much boilerplate it required in 2023:

    if (isReadableStream<Uint8Array>(value)) {
      const chunks = [];

      for await (const chunk of value) {
        chunks.push(chunk);
      }

      if (chunks[0].byteLength) {
        const length = chunks.reduce(
          (agg, next) => agg + next.length, 0
        );

        // Make the same species TypedArray that ReadableStream gave us.
        value = new (chunks[0].constructor as Uint8ArrayConstructor)(length);

        for (let i = 0, offset = 0; i < chunks.length; i++) {
          value.set(chunks[i], offset);
          offset += chunks[i].length;
        }
      } else {
        throw new Error(`Unrecognized readable stream type: ${ chunks[0].constructor.name }`);
      }
    }

    return  `data:application/octet-stream;base64,${
      btoa(
        (value as Uint8Array).reduce(
          (result, charCode) => result + String.fromCharCode(charCode),
          ''
        )
      )
    }`;

https://wintercg.org/

If it's faster than node and the API to do what I need is better (Bun.FileSystemRouter comes to mind), I don't see why not to switch over.

Heck, I would accept even less retro-compatibility to get rid of transpiling, that completely oscene political mess they did with modules and treating TS as a 2nd class citizen.

The amount of time I'm still hit by ESM vs CJS is insane. Truly a Python 2vs3 moment for the node.js community.

it's always vendor lock-in. can never trust companies, mate!

- File: Like a Blob, but with additional file-specific properties (e.g., filename).

- ArrayBuffer: Fixed-length raw binary data in JavaScript, not directly accessible.

- Uint8Array: Interface for reading/writing binary data in ArrayBuffer, showing them as 8-bit unsigned integers.

- Buffer: Readable/writable raw binary data container in Node.js (subclass of Uint8Array)

Buffer is a subclass of Uint8Array.

High level languages will still want to do low level things

You can see it in action on this WebGL fluid simulator[0] by PavelDoGreat.

[0] https://github.com/PavelDoGreat/WebGL-Fluid-Simulation

So a list of floats? No! Let's not use floats everywhere just because JavaScript does.

This is what I feel the node people get right over the ES people, the ES ideology is so abstracted and pushes everything out into small utility classes so you have to create a handful of different objects with odd combinations of methods to get one useful conversion done.

The ES people also seem to have no love of the CLI or for type types of debugging and testing done there. As a result, I almost always choose the node created abstractions over the ES specified ones.

Not a security expert etc.

If the buffer doesn't have any cross-contamination with global state, there's no way one user could access another's data (because it's behind an object reference that never comes into the scope of the request logic for the other user). But if it did, and a malicious user found some other kind of vulnerability, they could potentially access data across-scopes

I also provide a package to make the transition easier: https://github.com/sindresorhus/uint8array-extras (Feel free to copy-paste the code if you don't want another dependency)

[0] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

Feel free to copy-paste the function to your own code base if you don't want the dependency:

``` const objectToString = Object.prototype.toString;

export function isUint8Array(value) { return value && objectToString.call(value) === '[object Uint8Array]'; } ```

As for the fastest way to serialize data to Pandas data to the browser, you should use Parquet; it's the fastest to write on the Python side and read on the JS side, while also being compressed. See https://github.com/kylebarron/parquet-wasm (full disclosure, I wrote this)

Not vetting the bedrock you chose to build upon; deciding to rely on the unreliable.

("Everything" is an overstatement.)

It's one of those names where I tend to raise an eyebrow when I hear it. This looks like something similar to the module/require changes, where the overall change is good, but a slower transition may help people more.

No, it also (from the article):

> introduces numerous methods that are not available in other JavaScript environments. Consequently, code leveraging Buffer-specific methods needs polyfilling, preventing many valuable packages from being browser-compatible.

I think you are missing the point. Mutable and copied slices both have their use, and maybe a method that sometimes copies and sometimes shares does too. The problem with Buffer is that it overrides a method with well-defined semantics, then violates those. Buffer is a subclass of UInt8Array, but not a subtype nor Liskov-substitutable.

In practical terms, when you get a UInt8Array and even check that it in fact is a UInt8Array, you no longer know that .slice() does even though UInt8Array's documentation explains how every UInt8Array behaves, because that documentation is now wrong. If the UInt8Array you get is a Buffer, then it is both a buffer and a UInt8Array but its .slice() behaves differently than expected.

Changing method behavior in subclasses is part of inheritance, but it shouldn't confuse or mislead. In the case of Buffer and Uint8Array, the altered `.slice()` functionality isn't a mere implementation detail; it's a significant deviation. This inconsistency can lead to unexpected bugs, especially for those who assume similar behavior based on the inheritance hierarchy. It's crucial for reliability that such fundamental behaviors remain predictable across subclasses.