HNHacker News
TopNewBestAskShowJobs

kodablah

9,823 karma · joined February 17, 2011

https://github.com/cretz
submissionscomments
kodablah··on How Python asyncio works: recreating it from scratch
Like many static analyzers in ambiguous situations, you have to decide whether the possibility of false positives is worth the safety. In this case I'd flag it knowing that it could be marked ignored as needed (same with open() if it was used).

I have toyed with a MyPy plugin to do this, but the arbitrary state storage/cache is a bit limited to do good call graph and function coloring (metadata can be on types but not functions). And there's just not many other good _extensible_ static analysis libraries. Language like Go have static analysis libraries that let you store/cache arbitrary state/facts on all constructs.

kodablah··on How Python asyncio works: recreating it from scratch
Asyncio allows you to replace the event loop with an implementation of your own. For Temporal Python we represent workflows as a custom, durable asyncio event loops so things like asyncio.sleep are durable timers (i.e. code can resume on another machine, so you can sleep for weeks). Here is a post explaining how it's done: https://temporal.io/blog/durable-distributed-asyncio-event-l....

The biggest problem with asyncio is how easy/common it is in Python to be able to block the asyncio thread with synchronous calls, gumming up the whole system. Python sorely needs a static analysis tool that can build a call graph to help detect if a known thread-blocking call is called directly or directly from an async def.

kodablah··on Coroutines and effects
> Overall, coroutines strike me as the most promising way to handle many kinds of effectful functions

This is basically what we do at Temporal, which is essentially deterministic coroutines with externalized effects. This gives another nice property: using coroutines to wrap effects lets the non-effect logic replay/resume durably.

kodablah··on Pylyzer – A fast static code analyzer and language server for Python
Hopefully it can be used easily as a library. I have multiple use cases for programmatically operating on typed ASTs. MyPy plugins are very limited and most other analyzers have no such extensibility.

> Limitations [...] pylyzer (= Erg's type system) has its own type declarations for the Python standard APIs

I wonder why https://github.com/python/typeshed could not be used

kodablah··on Every Dunder Method in Python
They didn't have to be dunder methods though, they could have used any other naming scheme (e.g. just `__` at the beginning, EDIT: or something else) and remained just as important in the ecosystem.
kodablah··on Is something bugging you?
Has any thought been given to repurposing this deterministic computer for more than just autonomous testing/fuzzing? For example, given an ability to record/snapshot the state, resumable software (i.e. durable execution)?
kodablah··on Mariposa – A programming language with time-travel
This seems to have some parallels with durable execution engines like Temporal which have deterministic concepts of time and use event sourcing or snapshotting to ensure the past remains the past but allow you to evolve code for the future (disclaimer I work at Temporal).

Have you considered this approach to apply to more than time? There is all sorts of conditional-based divergence code may take beyond time.

kodablah··on An app can be a home-cooked meal (2020)
Unfortunately you almost have to do this in secrecy with a homemade scraping app lest its popularity causes the site owner to take action
kodablah··on LLMs and Programming in the first days of 2024
I had a project recently: building an advanced Java class file analyzer. I knew a lot about ow2 asm libraries, but it saved me a lot of digging time remembering exact descriptor formats. Also it helped me understand why other static analysis libraries weren't good enough for me for stateful reasons.

For me, ChatGPT is doing two things: 1) saving trivial StackOverflow and library code walking to answer specific question, and 2) helping the initial project research stage to grasp feasibility of approaches I may take before starting.

kodablah··on What's Going on with Language Rankings?
> the data [...] showed a roughly 25% decline in pull requests in 1H2023 as compared to 2H2022 PRs [...] largely lacking an explanation

May be obvious, but would rather know if you saw PRs-per-repo decline. Otherwise, I wonder if some very frequent-PR repos (automated or otherwise) had previously skewed numbers. Also curious if early numbers for this half can be obtained/compared.

kodablah··on Servo announces grant from the NLnet Foundation
> I also love the idea of a web engine as a component. This for some reason seems to have disappeared in modern times.

I have used CEF, WebKit, and WebView2. Only the former is easily embeddable cross platform, and as soon as it became popular, Google put a lot of resources into disallowing Google login from it or any other embedded browser[0]. I had to abandon my own CEF-based browser because of this. While I abhor the business practice, there's nothing I can do. If my browser can't login to Google products, it can't see much adoption and I don't have the time to try and win a cat-and-mouse game against their detection methods.

If we want embedded browser components to be more mainstream where anyone can develop a browser, we're going to first have to address that large companies block them if they get too popular but the embedding company is not popular. Ideally an embeddable non-Android Gecko would come about because FF is too big to block (lots of talk in the past, and I even PoC'd stealing the window handle[1]). But there's no money in it.

0 - https://developers.googleblog.com/2016/08/modernizing-oauth-...

1 - https://github.com/cretz/ffembedpoc

kodablah··on Bringing garbage collected programming languages efficiently to WebAssembly
> I was curious if there was a WASM to JVM and it seems there is one on GitHub [...] https://github.com/cretz/asmble

While it works well, this was mostly a fun project for me and I no longer really maintain it. I hope that the ideas and explanations of how I mapped WASM IR to JVM bytecodes helps whoever does build this in a more official capacity. I don't have any plans to support WASM GC currently.

kodablah··on Flawless – Durable execution engine for Rust
When I implemented a WASM compiler, the only source of float-based non-determinism I found was in the exact byte representation of NaN. Floating point math is deterministic. See https://webassembly.org/docs/faq/#why-is-there-no-fast-math-... and https://github.com/WebAssembly/design/blob/main/Nondetermini....
kodablah··on Flawless – Durable execution engine for Rust
https://temporal.io/blog/durable-distributed-asyncio-event-l... explains how it works
kodablah··on Replit's new AI Model now available on Hugging Face
So many code models seem to be used for code generation purposes, but is there a general effort to apply these as static/code analysis tooling? It'd be nice to write my rules in English and have somewhat predictable behavior when analyzing small bits of code. I have had success with GPT4 and a bit less with StarCoderPlus, but I have to build the engine to chop up code, send pieces to remote as needed, cache results for same-hashed snippets, etc.

Surely someone is working on general AI powered code analysis tooling?

kodablah··on Show HN: Standalone Jdk.compiler, Java Compiler Framework and Tree API
A neat use case would be to replace Error Prone's use of the compiler tree API. If this gets a Java 21 version soon, I may be able to use it for a new project I was just going to make an Error Prone plugin for.
kodablah··on The Workflow Pattern
> I always thought it was a bit silly that the async mechanism in modern languages is so myopic and single-purpose. These kinds of high level language transformations ought to be extensible and pluggable so that we could write workflows in a proper programming language and have it look like normal code except for the occasional “await” keyword.

This is how Temporal works. For example, in Python the async event loop is replaced by a durable event loop [0], the JS promises become durable promises, .NET tasks become durable via a custom scheduler, etc. Granted it doesn't serialize the stack, it uses event sourcing almost exactly like the article describes, and therefore requires deterministic code for replaying. From the dev POV, it looks like any code can just be frozen in the middle of a function and magically resumed elsewhere.

0 - https://temporal.io/blog/durable-distributed-asyncio-event-l...

(disclaimer, I work at Temporal and have written some of these distributed coroutine impls)

kodablah··on HashiCorp did it backwards
> if you want to start a business building developer tooling and you’re concerned about competitors using your work, you should probably start source-available (BSL) and then go for a license that is OSI-approved

To put another way, if you want to start a business, make sure what you're selling you can do better than others. If your business is hosting open source software and you can't do that better than others, then don't make that your business. Maybe you should have been selling non-open-source instead (which they pivoted to now anyways).

kodablah··on Asyncio, twisted, tornado, gevent walk into a bar
> The only danger is if you call gevent.monkey.patch_all() too late.

Gevent takes over, often making advanced asyncio not work on gevent programs. Patching things globally should obviously be seen as dangerous in itself.

kodablah··on GIL removal and the Faster CPython project
I have a use case for sandboxing (without a secure-sandboxing requirement) that it would help solve. But at last read, they discourage/disallow sharing modules across the subinterpreters and reloading all modules in each subinterpreter is not acceptable for my use case.
kodablah··on HashiCorp adopts Business Source License
I don't think the loss is due to bad actors. These actors aren't bad, they're rational. I think the loss is due to giving away things unconditionally without thinking about what that means. Many companies aren't reneging on open source. The ones that are reneging can't compete on what they are selling, so they have to compete on what they're not selling.
kodablah··on HashiCorp adopts Business Source License
> abusing the spirit of open source for their own monetary gain

For many, the spirit of open source is to allow it to be used in any way, commercially or otherwise. I don't consider it acting in bad faith at all to use what's out there and made available, and I am happy to give things away as in beer. Otherwise it's not open source (which is also fine, not all software has to be open source).

To abuse the spirit of open source, make arbitrary rules about its use.

kodablah··on HashiCorp adopts Business Source License
I also work on open source full time. I personally am happy to give my work away unconditionally to both freeloaders and contributors and fortunately my employer feels the same way (I am often a freeloader myself on unrelated projects).

Ideally your moneymaker can compete on its own without being dependent upon gatekeeping how the open source part is used. But with project ownership means you can change the rules for your competitors, but it's also a sign that you fear your offering is not competitive enough without the rule change.

kodablah··on WebAssembly and Replayable Functions
> why workflows would rely on code that doesn't remove the sources of non-determinism?

We can't always sandbox every environment to remove these sources at a language level. Sure in JS/TypeScript we can use V8, but in Go there aren't many good ways to do this (all options, e.g. interpreters, have tradeoffs). So we offer deterministic alternatives for non-deterministic language features/stdlib and rely on authors and static analysis for catching misuse.

> Or is the issue that the evolution of a language might introduce additional sources of non-determinism?

This is less concerning, but still can be a concern.

> So merely upgrading without checking for these wouldn't be safe if code execution depends on the behavior of a specific runtime?

Technically true, though the bigger concern is updating code you think wouldn't cause a change in the order of upstream operations (what we call "commands") but does.

kodablah··on WebAssembly and Replayable Functions
At Temporal we've done a lot of research on WASM-based deterministic workflows. While they make sense in a general way, they suffer a similar problem as deterministic OS environments (e.g. hermit) - safe updates. People often need to apply code changes and know they will be deterministic changes with regards to past executions.

When determinism is clearly defined at the language level, you can easily see when an alteration would be compatible with existing long running executions when replaying old history. However when it's at a low level, it's less clear.

For example, Go WASM includes its coroutine scheduler inside the WASM blob. A slight change in an unrelated part of the code can change the order of coroutines causing it to be an unsafe non-deterministic change when deployed for running workflows. Similarly, Go may make an external call to a seeded deterministic random for map iteration (yes, Go explicitly randomizes map iteration order). One other call to that random will make that map iteration order possibly different breaking replay.

So WASM is great unless you need to alter code for existing workflows that run for years. Then it becomes less clear, whereas language level determinism can have clear semantics to code authors and static analyzers with explicit versioning/patching if needed. Of course using lower-level languages can help here too since they are more predicably compiled with a smaller runtime, but they still require authors to be careful on code changes.

kodablah··on Theseus DHT Protocol (2018)
I once combined libp2p/ipfs's DHT impl with Tor many moons ago: https://github.com/cretz/tor-dht-poc
kodablah··on CheerpJ 3.0: a JVM replacement in HTML5 and WASM to run Java on modern browsers
I wrote https://github.com/cretz/asmble which has an AOT compiler and interpreter for WASM. Haven't worked on it for a while though.
kodablah··on Temporal .NET – Deterministic Workflow Authoring in .NET
After looking at it, I am concerned it is not possible for a clean experience. We have to give guidance and samples and we have to choose a way of referencing methods in those. Having ambiguous approaches is a bit rough. We may have to just move to expressions.
kodablah··on Temporal .NET – Deterministic Workflow Authoring in .NET
That framework was created by the cofounder of Temporal
kodablah··on Temporal .NET – Deterministic Workflow Authoring in .NET
From my limited understanding, Dagster and Airflow are limited DAG-based tools compared to Temporal that lets you write your logic in the programming language directly (loops, conditionals, coroutines, other libraries, etc) so long as you remain deterministic. Prefect I am also not that familiar with so I can't tell if their "flows" have limited logic they can do from Python, but they do not seem to support things like signals/queries at first glance.

One big difference is those three are Python only, and Temporal is Go, Java, JS/TypeScript, Python, PHP, and .NET. Usually another difference between most workflow tooling and Temporal (granted I have not checked with these) is the care they take in handling errors, retry, replay, cancel, etc so that'd be worth checking as well.

← PreviousPage 4 of 34Next →