Resizable structs in Zig
tristanpemble.com
tristanpemble.com
Too bad, aligned byte-typed VLAs (and a license to retype them as a struct) are what you need to get stack allocation across ABI boundaries the way Swift does it. (A long long time ago, SOM, IBM’s answer to Microsoft’s COM, did this in C with alloca instead of VLAs, but that’s the same thing.) I guess I’ll have to use something else instead.
At a fundamental level, runtime-known stack allocation harms code reusability.
Edit: commenters identified 2 more puzzle pieces below, but there's still one that didn't get asked about yet :P
@alloca(T: type, count: usize, upper_bound_count: comptime_int)
when I had been only thinking about zig for 2 years, I thought the same.
> It's too tempting to use incorrectly.
A compile-time-determined upper bound would solve this.
> The stack is allocated based on a compile-time-determined upper bound.
A compile-time-determined upper bound would solve this too.
Shouldn't a performance-oriented language give the programmer tools to improve memory locality? And what's wrong with spexguy's idea?
there are no good choices in the case where you really need that thing you claim to need. recognizing that fact and picking different strategy is good engineering.
> what are you going to do with the rest of the stack?
I'll leave it for the rest of the system. My app will use less memory, and since memory locality is improved, there will be fewer cache misses, meaning it runs faster too.
> let's say you take a function call that is about to overflow the stack
Stack overflows are impossible thanks to the comptime upper_bound parameter. That's the entire premise of this thread.
I thought Zig was all about maximum performance. Sometimes I just want a little bit of stack memory, which will often already be in L1 cache.
Sigh. So I have to choose between something I think might be useful, for something that too many languages have already soiled themselves with. Hopes that Zig has a better solution, but not optimistic.
Our stack compels me to work in Swift, Kotlin, Elixir, and Python. I use the async feature of Swift and Kotlin when some library forces me to. I actually preferred working with GCD before Swift had to join the async crowd. Elixir of course just has this problem solved already.
I frequently ask others who work in these languages how often they themselves reach for the async abilities of their languages, and the best I ever get from the more adventurous type is “I did a play thing to experiment with what I could do with it”.
C libraries?
Even on languages without VLAs one can implement a simulacra of them with recursion.
So it will be the same thing but with more (error handling) steps.
This annoyance can be avoided by avoiding recursion. Where recursion is useful, it can be done, you just have to handle failure properly, and then you'll have safety against stack overflow.
> Where recursion is useful, [...]
Recursion is so useful, most imperative languages even have special syntax constructs very specific special cases of recursion they call 'loops'.
Yes[1]. You can use the @call builtin with the .always_tail modifier.
@call(.always_tail, foo, { arg1, arg2, ... });
How do incremental compilation and distributed compilation work?
That’s a genuinely interesting point. I don’t think known sizes for locals are a hard requirement here, though threading this needle in a lower-level fashion than Swift would need some subtle language design.
Fundamentally, what you want to do is construct an (inevitably) runtime-sized type (the coroutine) out of (by problem statement) runtime-sized pieces (the activation frames, itself composed out of individual, possibly runtime-sized locals). It’s true that you can’t then allow the activations to perform arbitrary allocas. You can, however, allow them to do allocas whose sizes (and alignments) are known at the time the coroutine is constructed, with some bookkeeping burden morally equivalent to maintaining a frame pointer, which seems fair. (In Swift terms, you can construct a generic type if you know what type arguments are passed to it.) And that’s enough to have a local of type of unknown size pulled in from a dynamic library, for example.
Again, I’m not sure how a language could express this constraint on allocas without being Swift (and hiding the whole thing from the user completely) or C (and forcing the user to maintain the frames by hand), so thank you for drawing my attention to this question. But I’m not ready to give up on it just yet.
> At a fundamental level, runtime-known stack allocation harms code reusability.
This is an assertion, not an argument, so it doesn’t really have any points I could respond to. I guess my view is this: there are programs that can be written with alloca and can’t be written without (unless you introduce a fully general allocator, which brings fragmentation problems, or a parallel stack, which is silly but was in fact used to implement alloca historically). One other example I can give in addition to locals of dynamically-linked types is a bytecode interpreter that allocates virtual frames on the host stack. So I guess that’s the other side of being opinionated—those whose opinions don’t match are turned away.
Frankly, I don’t even know why I’m defending alloca this hard. I’m not actually happy with the status quo of just yoloing a hopefully maybe sufficiently large stack. I guess the sticking point is that you seem to think alloca is obviously the wrong thing, when it’s not even close to obvious to me what the right thing is.
(Once upon a time, MSLU, a Microsoft-provided Unicode compatibility layer for Windows 9x, used stack-allocated buffers to convert strings from WTF-16 to the current 8-bit encoding. That was also a bad idea.)
Was kind of the other way around, given the whole OS/2 versus Windows history, and that COM started as the evolution of OLE and VBX technologies, Windows 9X and Windows NT weren't as COM heavy as OS/2 was with SOM.
There was no COM to worry about on Windows 3.x back in 1991.
https://www.edm2.com/index.php/SOM_%26_DSOM_-_An_Introductio...
Also SOM was so much better, bettwen C++, Smalltalk and C, with support for meta-classes and proper inheritance implementation across such disparate languages.
Conceivably, an implementation of this `ResizableStruct` that uses an array buffer as backing rather than a heap allocation, and supports the defined layout of an extern struct, could be used to work across the ABI
moreover since the stack allocator is just an allocator, you can use it with any std (or user) datastructure that takes an allocator.
I mean you could also just abstract the allocation away and handle it after the function pointer to your bridge, right?
It's pretty clean at this imo: Less metaprogramming but I think nicer to use in some cases.
const Connection = opaque {
pub const Header = struct {
host_len: usize,
// add more static fields or dynamic field lengths here
//buff_len: usize,
};
pub fn init(a: std.mem.Allocator, args: struct { host: []const u8 }) *@This() {
var this = a.allocWithOptions(u8, @sizeOf(Header) + host.len, @alignOf(Header), null);
@memcpy(this[@sizeOf(Header)..], host);
return this.ptr;
}
pub fn host(self: *const @This()) []const u8 {
const bytes: *u8 = @ptrCast(self);
const header: *Header = @ptrCast(self);
const data = bytes[@sizeOf(Header)..];
const host = data[0..header.host_len];
return host;
}
};
Because the solution outlined in the article stores the lengths alongside the pointer, instead of behind it, there is room for it to work across an ABI (though it currently does not). It's more like a slice in this way.
You could in theory implement your opaque approach using this as a utility to avoid the headache of alignment calculations. For this reason, I think that makes the approach outlined in the article more suitable as a candidate for inclusion in the standard library.
pub fn getPtrs(self: Self) struct {
client: *Client,
host: []u8,
read_buffer: []u8,
write_buffer: []u8,
} {
return .{
client: self.get(.client),
host: self.get(.host),
read_buffer: self.get(.read_buffer),
write_buffer: self.get(.write_buffer),
}
}
> opaque {} declares a new type with an unknown (but non-zero) size and alignment. It can contain declarations the same as structs, unions, and enums
So it can contain methods, just like structs. The only thing it cannot contain is fields (which the example above doesn't contain). An opaque type can only be used as a pointer, it's like a `typedef void* ...` in C but with the possibility to add declarations within a namespace
Edit: the documentation doesn't contain any example of an opaque type which happens to have declarations inside. But here is one: https://codeberg.org/andrewrk/daw/src/commit/38d3f0513bf9bfc...
why not to make it heap-only type? it seems such a useful addition to type system, why ignore it due to one usecase?
But its just kind of mediocre and you're better off actually dealing with the stack if you can actually deal with certain fixed sizes.
array-like storage with dynamic size has existed since forever - it's vector. over or undercommitting is a solved problem
VLA is the way to bring that into type system, so that it can be it's own variable or struct member, with compiler auto-magic-ing size reading to access members after it
From the article
>we now have everything we need to calculate the size, offset and alignment of every field, regardless of their positioning in the struct. >init to allocate the memory >get to get a pointer to a field >resize to resize the arrays >deinit to free the memory
You're now suggesting to do exactly what the article is about without being aware of it.
The author has described a metaprogramming utility for allocating a contiguous hunk of memory, carving this hunk into fields (in the article's example, a fixed-sized Client header, then some number of bytes for host, then some number of bytes for read_buffer, and then some for write_buffer). I'll acknowledge the syntax is convenient, but
1. we've done this since time immemorial in C. See https://learn.microsoft.com/en-us/windows/win32/api/evntcons...
2. you can implement this pattern ergonomically in C++, and even moreso once the C++26 reflection stuff comes online
3. the zig implementation is inefficient. It desugars to
const Connection = struct {
ptr: [*]u8,
lens: struct {
host: usize,
read_buffer: usize,
write_buffer: usize,
}
}
Would it be productive to jump into a thread on a Ruby article and puff your feathers about how you've always been able to do this in Perl, and also in Python 4 you can do XYZ? I don't think so.
For whatever reason, inevitably in threads on systems languages, someone comes in and postures themselves defensively like this. You might want to reflect on that.
> you can implement this pattern ergonomically in C++,
lmao
This, of course, but I also consciously made the decision to write for an audience less familiar with the language and concepts being discussed. That excitement can translate to engaging a less disgruntled and more curious reader than the OP.