Nice!

> However, for reasons unknown to me, they wrap these neatly separated rows with brackets ([ and ]) and add a comma to each line

Well, the reason (misguided or not) is as you say, I imagine:

> so it’s a valid, JSON array containing 100+ million items.

> We are not going to attempt to load a this massive array. Instead, we’re running this command:

    zcat ../latest-all.json.gz | sed 's/,$//' | split -l 100000 - wd_items_cw --filter='gzip > $FILE.gz'

That's one approach - I'm always a little wary of treating a rich format like JSON as <something> deliminated text - I'd be curious if using jq in streaming mode is much different in run-time. I believe this snippet, the core of which we lifted from stack overflow or somewhere does the same thing; split a valid JSON array into ndjson (with tweaks to hopefully generate similar splits:

    gunzip -c ../latest-all.json.gz \
     | jq -cn --stream \
       'fromstream(inputs|(.[0]  |= .[1:]) | select(. != [[]]) )' \
   | split -l 100000 - wd_items_cw --filter='gzip > $FILE.gz

Note on MacOS zcat might not be gunzip, hence the change.