m0shen

Ah I went through the same thing myself, but with Windows

https://moshen.net/posts/virtual_kvm/

I thought it depended on libc by default?

Nyancat terminal animation in perl: https://gist.github.com/moshen/1417991

On the slightly more useful side...

gotermimg to play gifs in the terminal: https://github.com/moshen/gotermimg

I see what you're saying. When I was writing scripts in `expect`, I didn't really ever try to automate tui programs. So, this could absolutely be a better way to script the the terminal as a ui as you said.

I'll certainly tuck it into my toolbox. Thanks :)

`expect` is absolutely geared towards scripting, as it's an extension of TCL. Though as far as getting a "current terminal view" `expect` has `term_expect`: https://core.tcl-lang.org/expect/file?name=example/term_expe...

Paperless supports OCR + full text indexing: https://docs.paperless-ngx.com/

As far as AI goes, not sure.

I love seeing how many people wrote one of these. I've written 2:

One to learn golang that plays gifs - https://github.com/moshen/gotermimg

And one in perl - https://github.com/moshen/Image-Term256Color

Yes, you're correct. I had in mind the 2022 date of the update. It also looks like you've updated the article again today! Thanks

This article was written when cosmopolitan: https://justine.lol/cosmopolitan/index.html was v2. v3 has since been released: https://justine.lol/cosmo3/ .

https://cosmo.zip/ Also contains a python build https://cosmo.zip/pub/cosmos/bin/python , though I'm not sure if the problems mentioned in the article are solved.

Anything that converts to PDF/A will strip Javascript

I've updated this script with some single-file cli numbers, which are (as expected) not good. Mostly just comparing python startup time for that.

    make
    sqlite3 < analyze.sql
    file_avg              python_avg         python_x_times_slower_single_cli
    --------------------  -----------------  --------------------------------
    0.000874874856301821  0.179884610224334  205.611818568799
    file_avg            python_avg     python_x_times_slower_bulk_cli
    ------------------  -------------  ------------------------------
    0.0231715865881818  0.69613745142  30.0427184289163

Absolutely, and honestly in a non-interactive ingestion workflow you're probably doing multiple checks anyway. I've worked with systems that call multiple libraries and hand-coded validation for each incoming file.

Maybe it's my general malaise, or disillusionment with the software industry, but when I wrote that I was really just expecting more.

Good to know! Thank you. I'll definitely be trying it out. Though, I might download and hardcode the model ;)

I also appreciate the use of ONNX here, as I'm already thinking about using another version of the runtime.

Do you think you'll open source your F1 benchmark?

My little script is trying to identify in bulk, at least by passing 165 file paths to `magika`, and `file`.

Though, I absolutely agree with you. I think realistically it's better to do this kind of thing in a library rather than shell out to it at all. I was just trying to get an idea on how it generally compares.

Another note, I was trying to be generous to `magicka` here because when it's single file identification, it's about 160-180ms on my machine vs <1ms for `file`. I realize that's going to be quite a bit of python startup in that number, which is why I didn't go with it when pushing that benchmark up earlier. I'll probably push an update to that gist to include the single file benchmark as well.

That doesn't appear to be a valid link. Try building `file` from source and using the provided default magic database.

Thank you for the release! I understand you're just getting it out the door. I just hope to see it delivered as a native library or something more reusable.

I did try the python cli, but it seems to be about 30x slower than `file` for the random bag of files I checked.

I'll probably take some time this weekend to make a couple of issues around misidentified files.

I'll definitely be adding this to my toolset!

At multiple points in my career I've been responsible for apis that accept PDFs. Many non-tech savvy people seeing this, will just change the extension of the file they're uploading to `.pdf`.

To make matters worse, there is some business software out there that will actually bastardize the PDF format and put garbage before the PDF file header. So for some things you end up writing custom validation and cleanup logic anyway.

This is false in every sense for https://www.darwinsys.com/file/ (probably the most used file version). It depends on the magic for a specific file, but it can check any part of your file. Many Linux distros are years out of date, you might be using a very old version.

FILE_45:

    ./src/file -m magic/magic.mgc ../../OpenCalc.v2.3.1.apk
    ../../OpenCalc.v2.3.1.apk: Android package (APK), with zipflinger virtual entry, with APK Signing Block

Yeah, there is this line:

    By open-sourcing Magika, we aim to help other software improve their file identification accuracy and offer researchers a reliable method for identifying file types at scale.

Which implies a production-ready release for general usage, as well as usage by security researchers.

Made a small test to try it out: https://gist.github.com/moshen/784ee4a38439f00b17855233617e9...

    hyperfine ./magika.bash ./file.bash
    Benchmark 1: ./magika.bash
      Time (mean ± σ):     706.2 ms ±  21.1 ms    [User: 10520.3 ms, System: 1604.6 ms]
      Range (min … max):   684.0 ms … 738.9 ms    10 runs
    
    Benchmark 2: ./file.bash
      Time (mean ± σ):      23.6 ms ±   1.1 ms    [User: 15.7 ms, System: 7.9 ms]
      Range (min … max):    22.4 ms …  29.0 ms    111 runs
    
    Summary
      './file.bash' ran
       29.88 ± 1.65 times faster than './magika.bash'

As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space.

Though I have to say when looking at the Node module, I don't understand why they released it.

Their docs say it's slow:

https://github.com/google/magika/blob/120205323e260dad4e5877...

It loads the model an runtime:

https://github.com/google/magika/blob/120205323e260dad4e5877...

They mark it as Experimental in the documentation, but it seems like it was just made for the web demo.

Also as others have mentioned. The model appears to only detect 116 file types:

https://github.com/google/magika/blob/120205323e260dad4e5877...

Where libmagic detects... a lot. Over 1600 last time I checked:

https://github.com/file/file/tree/4cbd5c8f0851201d203755b76c...

I guess I'm confused by this release. Sure it detected most of my list of sample files, but in a sample set of 4 zip files, it misidentified one.

Not all. Jet fuel doesn't have lead. AFAIK it's piston based small planes.

Have this game. Love it! It's a great pick up and play couch game.

The fitness apps looked exactly like what I need, but when going to the play store says they were made for an older version of Android. Visiting Github seems like they aren't actively maintained either.

Yeah margin per processor shipped is a different question, and Apple probably does very well here.

Number one in performance / watt? maybe? sales? Well no. According to the web[1] total chip sales in 2022 excluding arm amount to 374 million units. Server chips being 36.1 million units. Apple's biggest quarter in 2022 was 7 million macs in Q4[2].

[1]: https://www.networkworld.com/article/3688288/amd-gains-share...

[2]: https://www.statista.com/statistics/263444/sales-of-apple-ma...

It looks like that's auto generated

I just picked up a launchpad x for this purpose. Has a note mode and you can customize what the grid does. Focusrite is also having a sale at the moment: https://store.focusrite.com/en-gb

Xero | Multiple Roles | REMOTE (ON/AB/BC Canada), HYBRID Toronto | Full-Time | https://www.xero.com/

Xero is one of the fastest growing software-as a-service companies globally. We lead the New Zealand, Australian, and United Kingdom cloud accounting markets, employing a world-class team of 4,000+ people. The future of our growth is in the Americas and as the Product Development Hub for Xero North America we're continuing to rapidly grow and expand and have roles available, in both a hybrid and fully remote capacity, across almost every department. In terms of tech, we build in .NET and Node on the backend, React on the front-end and run everything in AWS. We're looking for engineers and engineering leads who are enthusiastic about solving problems at scale to come build the future of Xero in North America.

Selected open Remote roles:

  - Software Engineer
  - Machine Learning Engineer
  - Senior Android Engineer
  - Senior iOS Engineer

More information here:

Remote Roles: https://jobs.lever.co/xero?lever-via=XwiHpkoGYx&location=Rem...

Hybrid, Ontario Roles: https://jobs.lever.co/xero?lever-via=XwiHpkoGYx&location=Tor...

Please apply directly through Lever above or feel free to reach out directly at colin [dot] kennedy [at] xero [dot] com

I don't know about the preview issues, but I have also had some bizarre issues during encoding. Specifically h264 using the mp4 container will crash encoding almost 100% of the time. This started a couple of major versions ago, and continues to today. When encoding h264 in a mov container, no crashes, any other codec, no crashes. I would say simply try a different container and then use ffmpeg to copy / transcode into a format you want to deliver.