Hacker News Reader

I was curious to see how this would work with bzip2 and zstd. The source is public at https://github.com/nathanrs/gzipt, and I asked MiMo-V2.6-Flash to fork and modify it in a straightforward way. The answer is that bzip2 produces sequences that don't resemble human language:

  gzipt \
      --corpus data/tinyshakespeare.txt \  
      --prompt $'MENENIUS:\n' \
      --length 200 \
      ;
  
  MENENIUS:
  MtLUMSeptuttyyyxyxyxyxyvyyyxyxyxyxyvyyyxyxyxyxywyvzyxyxyx
  yyxyyyxyxyxyxyxPlyxyxyxyxyxyxyxyxyxtoxzfTUS.zxzzzyzzzvzzz
  vzzzxvzyvyxyxyxyvyxyxyxyvy--,Vdvyxyxyxyxyxyxyxyxyxxy!zFlx
  zzyyxyxyxyvyxyxyxyvyySPffuyuy

Line breaks added. This looks roughly optimized for the most repetitive Burrows-Wheeler transform (https://en.wikipedia.org/wiki/Burrows%E2%80%93Wheeler_transf...). Why are they runs of alternating symbols and not one symbol?

Zstandard produces whitespace with the occasional letter thrown in. To quote MiMo: "As you can see, zstd does not speak Shakespeare. ... zstd encodes a run of one repeated byte as a near-free run-length sequence, and space and newline are the cheapest literals in the corpus: ten newlines cost about the same to append ten bytes of genuine corpus text and less than nonsense does."

View on HN

@@ -33,19 +34,16 @@ def candidate_lengths( level: int = 9, pool: ThreadPoolExecutor | None = None, ) -> list[int]: - """Compressed length of ``context + seq`` for each seq, sharing the context. + """Compressed length of ``context + seq`` for each seq. - Compresses ``context`` once into a ``compressobj``, then clones its encoder - state per candidate and feeds only that candidate. Identical to - ``len(zlib.compress(context + seq, level))`` for each seq, but the expensive - match search over ``context`` happens a single time. + Unlike ``zlib``'s ``compressobj``, Python's ``BZ2Compressor`` cannot be + snapshotted mid-stream, and bzip2's move-to-front + Huffman stages see the + whole block, so every candidate recompresses the full context. Threads + still scale because ``bz2`` releases the GIL. """ - base = zlib.compressobj(level) - head = len(base.compress(context)) def length_for(seq: bytes) -> int: - clone = base.copy() - return head + len(clone.compress(seq) + clone.flush(zlib.Z_FINISH)) + return len(bz2.compress(context + seq, level)) if pool is not None: return list(pool.map(length_for, sequences))

MENENIUS: 'Though all at once cannq MARCIUS: I'll fight 'Though all at once cannq MARCIUannq MARCIUS: I'll fight 'Though AUFIDIUS: If I fly, Marci AUFIDIUS: If I fly, Marci AUFID AUFIDIUS: If If I fly