samsquire

Thanks for the interesting article.

Is anyone not just me hesitant to run their own servers? I think about patching and it becomes a full on devops job to set up something safe and stay on top of upgrades etc.

One idea Ive had is that if the web facing part of the software is just a form saver. It persists the form to disk for retrieval by the local-first server that I run on my laptop after pulling all records.

Would like to talk to persons about this topic: local first and low-maintenance service design SAAS

I wrote about an imaginary cloud service that would be about hosting forms for batch processing later this is similar to the workspace.zip in this article but this is used for the app itself.

https://www.halfbakery.com/idea/Secure_20Form_20Endpoint_20S...

I agree with you, I perceive it to be really difficult to change existing code to fit new things in. There's all interactions between things.

The source code of the widgets we use everyday - such as text editing or graphical editors are or your web browser's Javascript engine, are enormous.

'Just do X, too' is really difficult with what is already there.

If you have a large body of existing code and you want to change its behaviour, you have to work out where to add your change, without breaking what already is there.

My thoughts in this is to somehow create a system where additional rules or changes to behaviour have marginal cost.

I am interested in the Rete algorithm, a rule engine algorithm. But we could run this sort of thing at compile time to wire up system architecture.

Boilerplate or configuration is an enormous part of programming and I feel there really could be more tools to transform software architecture.

Thank you for this post, I enjoyed it.

I like these kinds of posts, about wholesome satisfying work

I found a similar feed to HN on https://bearblog.dev/ at https://bearblog.dev/discover/ that might be helpful for persons to find new posts that are interesting.

Thanks for your reply.

The JVM is a reliable stack. I used it at work on most projects.

Can someone reply me a mature stack that is extremely reliable and won't go away? That they're using.

I had frequently thought that I would prefer to maintain a batch system than an online system.

where can I go to see the awesome things you're working on in your home personal computing?

I like the GitHub collected archives of "awesome" that is lists of things that persons have created or done

https://github.com/sindresorhus/awesome

In early 2000-2007 I felt technology optimism (things like Digg, slashdot) about new websites and there was a hopefulness about new technology (file sharing) The spirit of new technology that "there is something new" and the "this is how things work from now on" (WAP websites, floppy disks, guest books, simple 1megabyte web hosting, geocities, fan sites, myspace, WhatsApp on cheap phones).

In other words, every new thing was something that may have been before but it was "this is how things work from now on". The platform defines and upholds the character of interaction. Twitter and Reddit do that and as pg highlights how twitter recipients is by algorithm. (From OP: "where you don't specify the recipients.")

I have fond memories of writing HTML from magazines and in the eras before me it was handwriting text games into BASIC interpreters.

I would like to benefit from big tech's security teams by hosting web forms and those various different kinds of site you suggest behind them and their teams! WAF + captchas + defence against bots I would rather not do the server handholding and hardening myself.

If you're looking for casual ideas what persons have ideas on then have a scroll through halfbakery.com they are inspired by problems that people foresaw.

Thanks for such a detailed article.

In my spare time working with C as a hobby I am usually in "vertical mode" which is different to how I would work (carefully) at work, which is just getting things done end-to-end as fast as possible, not careful at every step that we have no memory errors. So I am just trying to get something working end-to-end so I do not actually worry about memory management when writing C. So I let the operating system handle memory freeing. I am trying to get the algorithm working in my hobby time.

And since I wrote everything in Python or Javascript initially, I am usually porting from Python to C.

If I were using Rust, it would force me to be careful in the same way, due to the borrow checker.

I am curious: we have reference counting and we have Profile guided optimisation.

Could "reference counting" be compiled into a debug/profiled build and then detect which regions of time we free things in before or after (there is a happens before relation with dropping out of scopes that reference counting needs to run) to detect where to insert frees? (We Write timing metadata from the RC build, that encapsulates the happens before relationships)

Then we could recompile with a happens-before relation file that has correlations where things should be freed to be safe.

EDIT: Any discussion about those stack diagrams and alignment should include a link to this wikipedia page;

https://en.wikipedia.org/wiki/Data_structure_alignment

So one shot transactions can check if every timestamp in every write and item inside the transaction packet depends on data that is before the timestamp of that particular monotonic transaction timestamp?

And the pattern of including "check" transaction item is how we manually maintain data integrity (characteristic of Atomic in DBMS)

And we know which transactions are writing because they told us they wanted to write in the prepare phase (the part that the transaction manager handles separate from the one shot transaction information perspective from the client with its own communication between the transaction manager and storage nodes)

I implemented a toy dynamodb that is a trie in front of a hash map, it handles the "begins with" query style.

I enjoy writing tools.

How do you feel about Kubernetes?

It would be good to have some interesting tasks to do?

I think the tools should do also much of the work. I actually prefer batch systems that are a simple execution of a program against a dump which are just process all the data and generate data with the new states than a networked online system that breaks all the time and due to DNS

Micro services keep me awake but a simple CSV processing I can fix in my own time.

We can all try make good content and share it but how would it be discovered?

One thing I have been doing is to create a directory called "debug" from the software and write lots of different files when the main program has executed to add debugging information but only write files outside of hot loops for debugging and then visually inspect the logs when the program is exited.

For intermediate representations this is better than printf to stdout

Thanks for sharing your thoughts.

I am also agreeing that relational approach to in-memory data is a good, efffective thought.

I recently compiled some of my C code with the sqlite database and I'm preparing to think how the SQL model of my standard code could be used as the actual implementation language of in memory operations.

Instead of writing the hundredth loop through objects I just write a SQL query instead with joining with seeing the internal data representation of the software as an information system instead of bespoke code.

I was hoping to make it possible to handle batches of data and add parallelism because arrays are useful when you want to parallelise.

I was thinking, wouldn't it be good if you could write your SQL queries in advance of the software and then parse them and then compile them to C code (using an unrolled loop of the SQLite VM) so they're performant. (For example, instead of a btree for a regular system operation, you can just use a materialised array a bit like a filesystem so you're not rejoining the same data all the time)

I was thinking of ways of representing actors somehow communicating by tables but I do not have anything concrete for that.

Thanks for putting the work into this.

These kinds of software pipelines can be strangely satisfying to implement.

The development style of thinking of stream processing and online algorithms

It's also inspired by mapping and filtering and functional programming with flatmap.

It reminds me of factorio

Thanks for sharing.

I am curious: how low maintenance is this?

Is this something that can be hosted for long periods of time without security interventions and updates?

Is it fire and forget?

This document helped me learn the io_uring API.

You can use io_uring with epoll to monitor eventfd to wake up your sleeping with io_uring wait for completions.

I have implemented a barrier and thread safe techniques that I am trying to turn into a command line tool

My goal is that thread safe performant servers are easy to write.

I am using bloom filters for fast set intersection. I intend to use Simd instructions with the bloom hashes.

CPUs and railway signalling come to mind.

The low level uops (microops) of a CPU seem to be relevant.

Openttd is another example of a game where you can implement train signalling.

I implemented split IO with io_uring, epoll and eventfd and my own thread safe comm pattern

You can write to an eventfd to wake up io_uring listening on an epoll instance.

A word cloud could be a useful navigation technique if the words were clickable and the sizes of words could be customized.

They could also link to other word clouds

Generated and updated automatically

We need tools that process command line output and large files and create tables of contents dynamically based on rules of architecture or comments and allow efficient navigation. I need overlapping buckets or tags.

I've been thinking about it lately but what I've seen in IDEs is not what I want

I liked OpenGrok

I greatly enjoyed this comment of yours daltonpinto. Thank you.

I do not rejoice with code bases where every file has no logic or code in it but there are hundreds of methods and files with everything spread out. I have no idea how those projects fit together because the actual logic is spread out.

For my personal side project hobby work it's all in one file.

Thank you for sharing.

I remember working with Html applications (HTA) on Windows back with JScript or VBScript.

I am thinking of tools that would make navigating plain text long files useful such as simple table of contents generation or indexes.

My side projects from 2012-2017 cannot be built or ran because of dependencies. My jsbin repo with lots of experiments cannot be ran anymore. But I have the sqlite database.

I forgot to pin dependencies when I was working. It would take a lot of trial and error and effort to get back to where I was. Or I have to rewrite the experiments.

Take lots of screenshots and screen casts to preserve your software!

A screenshots of multiplayer editor that allows transclusion that I can't build anymore.

https://github.com/samsquire/liveinterface?tab=readme-ov-fil...

Thanks for your reply.

I think I am contradictory when it comes to software : I don't enjoy maintaining something that breaks all the time: dependencies, system upgrades, deployment scripts and things that aren't 100% working reliably every time.

So my ideal system is to run a simple binary against a file or SQLite database and have it work reliably everytime. Not a complicated micro service architecture with lots of indirection and keep things running on network.

But balancing this with my hobby of designing multithreaded software.

I understand software better from words and diagrams than reading the code.

Complexity is aided by having sturdy mental models - what you intuitively understand. And to see the truth clearly.

To have humility: my code is probably ugly to other people, but I understand my code faster than reading yours.

I can't pretend to know that the system I build would be better than yours.

Be wary of pride.

This is really helpful. Thank you.

I would like to see some kind of query AST for this stuff in a query engine for semantics that its ops can be fused together for efficiency. For example, like a Clojure transducer.

Thank you for this post Janet.

I think my technical perspectives have moved in a similar direction: keep things extremely simple with minimum moving parts.

I maintained (automated upgrades) a RabbitMQ cluster and while it is powerful software it is operationally expensive. For a side project you probably just batch process in a cron.

If I were to take the approach in this blog post I would want everyone on the team to be extremely familiar with the model of task running: stuck jobs, timeouts, duplicate jobs, client disconnects and retries, stuck "poison" jobs seem like issues you might face.