superice

Fair enough, I was thinking of terminals one step smaller where reachstackers or straddle carriers directly drive to and from the quay. On bigger terminals it’s a much more interlinked process indeed.

See https://youtu.be/in2Q1KgqVIQ?si=7kzBKQGtrXyAbZEi

Agreed, with the asterisk that shipping companies and terminals will try to be the ones driving the government agendas on this. Government run does not necessarily equal neutral. But a neutral system I am generally in favor of.

You should look into Secure Container Release, Certified PickUp, Secure Chain, and a whole bunch of other initiatives doing this. Here is the Dutch one: https://www.portbase.com/en/programs/secure-chain/

Container terminals will take any minor efficiency win on their side, even if it comes at the cost of massive efficiency loss for truck transporters. It's optimizing for a local maximum. The market is structured in such a way that it is hard to correct for that, since the relation between trucking companies and container terminals is very indirect, and customers can't directly compare.

Also while shipping is a competitive market, the market for ports is not. You're either in a location or not. There are not hundreds of container terminals in a single port in competition because of economies of scale.

(The market for trucking companies IS competitive however, meaning that if you have to err on 'protecting' either party, you should probably pick that one)

That's a pretty reasonable mental model. The only real requirement during unloading is ship stability, other than that just use max concurrency with all the cranes and equipment to max throughput. Even just on the crane level, you can just keep unloading stuff to shore, and wait until vehicles pick them up. If they are slow, just keep on unloading until they catch up. Chance of stalling ops is close to zero.

Loading operations are much more variable, especially if your yard is not stacked well and you need to 'dig out' specific containers. If you run out of containers underneath your crane, your operations are stalled until the terminal vehicles catch up and bring you new boxes to load.

Yes, although that's the same for loading.

As a general rule, container ships are unloaded tier-by-tier, breadth-first if you will, not shaft-by-shift (depth-first), so this is not much of a problem in practice.

That does start to change if you want to do simultaneous loading and unloading operations, then you'd want to clear out a vertical shafts first so you can start loading operations as quickly as possible. Which is one of the many reasons dock workers hate that style of operations.

So about truck appointment systems, you should probably be thankful those are NOT the norm. Generally speaking container terminal operators and transport companies are antagonistic to eachother, since they are NOT in a direct business relationship. The truck transporter (or rail/barge transport companies) are hired either by the shipper directly, or by the shipping company, depending on whether you book a door-to-door or a port-to-port transport. This is also known as carrier haulage and merchant haulage. The container terminal generally works for the shipping line.

Long story short: the container terminal will always opt to please their customer (shipping line) over their non-customer (trucking companies). Truck appointment systems are usually used to force transporters to smooth out peak times not in the name of efficiency, but rather to lower the amount of dock workers the container terminal needs to hire. The truck companies generally end up footing the bill for this, both in increased workload and in detention/demurrage costs because they can't get their containers out and back in time. This money goes directly into the pocket of both the shipping line and container terminals as this is typically something they make heavy profits on.

Be very wary when container terminals and shipping lines start to push for centrally mandated appointment systems. They are much more consolidated than hinterland transport operators. I'm all for increasing efficiency but let's not even further increase market power for shipping lines and container terminals please.

That's not true in my experience. Loading outbound cargo is way more complex, since the stowage plan of the ship dictates where each container goes. Theoretically a lot of containers can be swapped as long as weight is similar, the container type is identical, and the port of discharge is the same. In practice it's still incredibly complex compared to just unloading stuff. While you may need to do less 'digging' on shore, the nitty gritty of the actual operations are way more complex than throwing some boxes ashore.

Import cargo is annoying in that it is mostly random access on pickup. For pickup by train, barge, or feeder ship, a vast minority, you typically don't have cargo manifests until a day or two before pickup at best, so in practice this is also random access-ish. The customs processes are also trickier.

My experience is mostly in Rotterdam and Antwerp, and I'd say the problems in the US probably don't have to do much with automation. Rotterdam and Antwerp have very different automation levels at the biggest container terminals, yet productivity is quite similar.

There is lots of low hanging fruit in optimizing operations, like more collaborative stowage planning, simultaneous unloading and loading operations, and 'modal shifting' from road to rail and water combined with early preannouncement of manifests for trains and barges.

Disclaimer: I'm in the business of consulting and building software for container terminals, so I'll generally be biased towards those solutions.

We’ve done this for an SME customer of ours. We were essentially building a new module on top of their existing in-house systems. We laid out our query patterns in advance so they could index properly, and only query specific views. Write-out is only in tables dedicated for our product.

It works surprisingly well, and is pretty resilient since it essentially acts as a message queue too. Then again, this is all low traffic stuff in a SME / B2B setting, with zero multitenancy involved.

I’ll still take a proper API or message queue any day though.

Edit: I suppose our biggest benefit is that our customer can actually change the interface with us fairly quickly. Then have database experts in-house, but devs who could do APIs. So the collaboration has been mostly smooth an account of that, and that is a huge advantage compared to them having to outsource any API work. Technically speaking I’m not a fan, but the non-technical results have been useful.

My cheesy tagline for this is “The customer is always right about their problems, rarely about the solution to it”. They know their business inside and out, and I’m not about to tell them otherwise. Metaphorically: if they need to cross a river, that doesn’t make them bridge engineers. But boy will they tell me if the bridge is in the wrong place.

Fair enough. To clarify, my point was not necessarily “I know better”, but that it is incredibly difficult to get a broad view of the market from a single perspective. This market is by definition a chain of steps with parties that may be antagonistic. It is atypical in that way, and writing software in-house may be more difficult in this sector than in others because of it.

While writing software in-house gives you software better tailored to your exact situation it is at the cost of losing the overview of the business and the perspective of the parties you interact with. Whether that trade-off is worth it I’m probably pretty biased on, so take my stance with a grain of salt, but imo the business is intrinsically relatively ill suited for in-house dev considering the high degree of collab with other parties.

Apologies if it came off as arrogance, that was not what I was going for.

I'd recommend against it, but then again, I'm building software products in this exact space with my launching customer being a 200-300M annual revenue logistics company. I don't think they could do it in-house. You don't give a lot of info on the exact niche you are in, but the username tells me that you're doing containers in European context, most likely shortsea or domestic, not deepsea. Me telling you the ISO6436 type code for your username is either LEG1 or LEGB depending on max stacking height should tell you I know my stuff :)

Your biggest pain point is most likely that you know your business very well, but you probably do not know enough about the business context of your partners. If you are running a small-ish container terminal for example, you really should know how the depot department of the major carriers are managing their empty stock so you can decide your stacking strategies in the yard. You should probably also have an understanding of benefits vs drawbacks of tower cranes vs gantry cranes found at bigger quays. That then influences what your TOS should be capable of. Same thing goes for intermodal transport planning, you really should know not only how detention/demurrage works, but also what tricks you can pull to optimize for it, and how shipping lines monitor this internally. If you are building stuff in-house, you're inevitably going to be limited in your understanding of the wider market. Short term that's a huge improvement (more tailored software, better flexibility), long term that might cause huge issues.

I can certainly understand the frustration with third-party products. Your description of 'they are understaffed and the platform is stagnant' could realistically apply to pretty much any software provider in this space. There is a huge opportunity for somebody to swoop in and build something, the bar for this sector is mostly something that doesn't straight up suck. And I'm going to be honest: we're trying to be that somebody.

If you'd like to talk, my contact details are in my profile. I'm happy to show you what we're building, and tell you what that is like behind the scenes. If you go ahead with your plan to in-house it, let's stay in contact, perhaps we can learn from eachother.

I'm a little confused about the point-in-time restore functionality, I'm pretty sure there must be a way to not have to force those one minute WAL boundaries. DigitalOceans managed PostgreSQL for instance just allows you to specify a timestamp and restore, and when looking into the PostgreSQL docs I remember seeing an option to specify a timestamp as well.

Yes, CSS variables. Defining a preset of variables on :root for sizes and color ranges goes a long way.

The design philosophies behind Tailwind are sound, but like so many I dislike the technical implementation. I feel like there is a lot of halo effect happening with Tailwind where judgement of technical merits is influenced by judgement on visual merits.

Yes, and that's leaving aside the explainability problem. Let say for arguments sake that you have an optimal planning algorithm, if you cannot explain to the people using it why it generates a certain solution, it will not get adopted.

As an employee at a terminal, you have a) no reason to trust some software people who may have never seen one of them steel boxes up close, b) if this works, you or your colleague gets laid off, and c) if it doesn't work, you're now stuck cleaning up the mess the computer will inevitably make for you, which is way less fun than planning things yourself.

The 5%-10% invoices being incorrect is probably on the low end, I'd probably say it's 5-10% in total invoice VALUE. Because of the sheer amount of containers, you need automated invoice calculation. But if you miss an edge case ("We are allowed to invoice a 5 buck surcharge if the container is filled with at least 100kg of explosive goods, AND was loaded during the night shift because dangerous goods handling requirements now require us to have a safety officer on call, which we normally don't have during the night shift") you are leaving serious amounts of money uninvoiced, and you will never know. One of the reasons not that much effort goes into optimization algorithms is that there is so much low hanging fruit left on getting your invoices out correctly...

For the customers in my size category (50k to 1 million TEU / year) the answer is that they may have 2 or 3 planners, optimizing berthing slots and stowage plans, but other than that there is no coordinated effort.

You have to realize that these terminals are typically a lot smaller in terms of manpower than you might think. A terminal handling 1 million TEU/year may operate with 2-3 crews in a 16 hour/day schedule for handling ships and trucks, plus some extra people handling the empties. Let's say it's 60 dock workers a day. A typically office might only employ about 12 people in overhead: one for customs/quality stuff, one for invoicing and finance, one HR, a bunch of order entry / customer service people, and 2 or 3 yard/berth/stowage planners.

Unless you look at the biggest of biggest terminals, you have to essentially regard them as SMEs in a very non-sexy business, and they treat tech and fundamental research into optimization accordingly.

There is a talk by the Technical University of Denmark (DTU) online by a researcher in this space, which is a pretty good introduction to the problem of stowage: https://www.youtube.com/watch?v=9ltz4G-lPdg

As somebody actively involved in the space, there are lots of ways you can make life easier for yourself. Planning in blocks per hatch cover, grouping containers by destination, size, and weight and treating those as mostly interchangeable are table stakes. You then want to send a plan from the ship to the terminal before start of operations, so the terminal can also optimize and shuffle given they know the position of containers in the yard.

Stowage planning is a lot easier if you decide you're going to ignore the details of each container and only occupy yourself with the groups. The end result is very similar for way less work, and you give a lot more flexibility to the terminal to optimize operations.

Same here, I'm in the business of terminals as well as hinterland transportation companies. There is no shortage of people trying to solve the 'sexy' academic problems like stowage plan optimization, berth assignment problems, crane splits, etc, but only very little gets implemented in practice. It is a tough business, with an even tougher political landscape, since dock workers traditionally have very strong unions.

All of the problems you can neatly divide and name are interrelated in practice, and trying to have algorithms "magically" figure it out for users is pretty much guaranteed to fail. This business eats software hotshots who think they can swoop in and "just" run some algos, "because it's just a TSP / Constraint Solver / [insert your approach here] right?". This business can definitely use bright minds, but start humbly, and start by talking to real world users please.

You don't appear to have any contact details on your profile, but if you ever would like to exchange thoughts with another company in this business, shoot me an e-mail. I think we're doing some interesting stuff for the <1 million TEU/year segment of terminals, especially if they're heavy into multi modal.

No, probably not, but SQL and ORMs aren’t used just for consumer grade applications ;)

I’m sure N+1 is not a big problem in those use cases, but saying that thereforr the problem doesn’t exist is overgeneralizing a bit. B2B is not a small market.

At risk of sounding like the Dropbox-guy: it's not that hard to implement these yourself. I've had to write a service mimicking a PostgreSQL database for... reasons, and expanding the Postgres connection protocol to have functionalities for load balancing/caching/cookies would definitely be possible and not even crazy hard, especially if you at some point in your connection can just start proxying to an actual PostgreSQL server so you only have to intercept the handshake.

The bigger question is: what would that solve? Caching is mostly relevant if you are repeatedly executing the same, relatively heavy, query. Load balancing is mostly interesting if you have read-only queries that you can easily execute against any of X replicas, in which case you might as well do that application side instead of having some transparent layer in between. And I don't get why you'd need cookies for something that is a persistent connection instead of individual requests?

The tabular data thing is tricky though, you're right. The impedance mismatch between tables and nested object structures isn't easy to solve.

The reality of the situation is that ODBC isn't going away any time soon, and proposing a breaking change on top of protocols like that is not realistically going to see lots of adoption unless you come up with a really great alternative AND some good marketing around that.

I've come to the conclusion that the best way to spend my efforts is to work with the ecosystem and build tools that try and fix the impedance mismatch is easier than hoping for something better to come along. Making it easier to write performant code is yielding more results for me than trying to adapt SQL-protocols to horizontally scale. A single box with some well placed indexes and well written queries is ridiculously fast.

I wasn't aware of Joist until I read a comment of yours somewhere else in this comment section, interesting stuff! I think Joist is the closest thing I've seen to an actual fix to this problem. The way you use the JS event loop as your 'scope' is pretty clever, and satisfies most use-cases I think.

It aligns pretty closely to what we do internally. We don't use classes or annotations to do entity definitions, instead we'll have something like this:

    const User = {
        fields: {
            id: defineIdField(),
            email: defineField('string', false), // false = NOT NULL
        },
        manyToOne: {
            role: { entity: 'Role', nullable: false },
        },
    };

Then the way to query and save is something like this:

    datasource.inTransaction(async (orm) => {
        const users = await orm.findMany('User', { where: sql`...`, orderBy: sql`email asc` });
    
        // Await in this loop might look bad, but only blocks for the first user
        for (const user of users) {
            console.log(user.email, (await user.role).name);
        }

        const adminRole = await orm.findOne('Role', 1); // by ID, throws if not found
        return orm.saveMany('User', users.map(u => ({ ...u, role: adminrole }));
    });

Which looks different from what Joist is doing, but the fetching mechanics under the hood are very similar I think. Our 'scope' is just somewhat more explicit, since we don't use the event loop mechanics of JS but rather use the inTransaction callback as the boundary.

(The reason we haven't open sourced this yet is that I'd like to iron out some of the ugly parts first, like the string entity identifiers, and the forced usage of an id column, and the lack of type safe where clauses right now)

Happy to name a concrete use case: showing a manifest of containers to load/unload from a ship. Search needs to happen client-side for many reasons, ballpark figures may be 1000 loads, 1000 unloads, the info about the shipvisit, and a stowage plan of 1000 records loading/unloading (separate from the manifest). Give you 4001 individual entities, clocking in at at least 20 attributes per entity (container number, type, port of loading/discharge, cargo description, weight etc)

N+1 just isn't cutting it for that use case, because while a manifest may be paginated, you pretty much just need your entire stowage plan in memory to do any useful operation on it.

Oh it definitely isn't. Data volume may be small, but latency will kill you, especially if your database server isn't on the same machine as your application. If you fetch an entity A with a one-to-many to entity B, with a typical size of 100 B's for every A, you are looking at 101 separate queries if you access any relations on B, which assuming 1ms of network latency give you a lowerbound of 100ms for an operation that may have taken well below 10ms in total had you fetched in one go.

Well, looking at something like the PostgreSQL protocol, it's decently efficient already. You could probably run a layer of compression on top of it, like HTTP gzip or so, but I doubt that will give a significant performance gain.

If you look at the end-to-end problem of 'what is the minimum amount of data I need during this request' vs 'how much data do I fetch, and what is my total latency / number of roundtrips to the db doing so?' I think for most ORM patterns that use lazy loading your primary target is reducing roundtrips, and for most hand rolled queries or ORMs tweaked to do eager loading, the primary target is deduplicating the results.

My take on this is that a decent approximation is a query per relation you're fetching, so if you have 10 entities A in a transaction, and each has 20 entities B attached, ideally you want 2 queries: one for the 10 entities A, and one for the 200 entities B. Lazy loading will give you 1 query for A + 10 queries for B, and eager loading will duplicate the 10 A entities data 20 times each (and that problem gets worse as your graph gets bigger with more one-to-many relations).

Once you run into the raw data transfer between database and backend being the limit, trying to optimize that protocol comes into play, but at least in the use cases I tend to have this is not usually a bottleneck. Besides, I'll typically serialize the data fetched to send out over HTTP again, which essentially has the same challenges if you're not using protobuf or so.

I see all these comments stating 'oh ORMs are bad' and 'just write some SQL'. Yes, you should probably not be afraid of SQL, and yes, using an ORM for everything is probably not great, and no, ORMs aren't a full replacement for writing SQL occasionally, but taking an extremist pro-SQL point-of-view is not doing any favors to this debate.

There are very real reasons why writing raw SQL is a pain. You can make arguments about typesafety of queries, maintainability, mapping into application level data structures/types yadayada. Imho the primary argument is that you cannot efficiently write a query that doesn't suffer from duplication of data. If you have an entity A that has many B's, and that same entity A also has many C's, then if you join on both you now are loading A * B * C rows. This is going to be slow too, and is difficult to detangle.

ORMs in their current form are terrible at tackling this problem, and people are right to call them out for that. "Just" lazy loading or "just" eager loading both do not scale, nor does doing firing event listeners on a per-entity basis. But rawdogging SQL is not a viable solution either, and I wish people advocating that would stop this attitude of 'oh you prissy dev using your puny ORM, not willing to write a line of SQL, look at me with my glorious hair on my chest and superior intellect writing pristine queries'.

Interesting. Does it do this separate query for all entities in scope, or per entiry like Hibernates SELECT FetchMode? I find a separate SELECT is usually possible, but doesn’t quite solve this in the general case.

Perhaps I missed Django, but as far as I could tell, Hibernate and jooq can’t do this, Ecto can’t do it, ActiveRecord can’t, and the entirety of ORMs for JS can’t (TypeORM, Prisma, Drizzle, Sequelize and a bunch more).

This seems fundamentally unfixable with current ORMs. You either have to pick between lazy loading per entity, or eager load and carthesian product everything, which breaks badly if you are dealing with multiple one-to-many relations.

Our solution was to write our own ORM-like system that “remembers” which entities you’ve loaded within the context of the transaction, and then will fetch the relation for all of them. So if you access a1.b then it will also fetch a2.b and cache it if you loaded a2 within that same transaction. The call to a2.b will then resolve instantly. So instead of n queries with n being the number of entities loaded in lazy loading, you’re doing n queries with n being the number of relations touched.

The one bad failure case is if you update entities one-by-one but also accessing their relations in a loop since you invalidate the cache for that entity type on every write.

Tech debt is more like being in debt to violent mobsters, who will show up at your house unannounced at the most inconvenient of times, during which they will hit you with a wrench until you pay whatever is in your wallet right now. They will return soon enough, because whatever payment you just made on the spot is never quite enough to settle the debt.

Oh that’s easy to fake with type narrowing:

function expectType<A, B extends A>() {}

expectType<never, typeof yourUnion>();

The function call will fail at compile time if yourUnion is anything more than never, which you can use in your else case of if statements that narrow the discriminated union.

I have been programming since 2008, but I truly started my professional webdev career in 2014. The company I started at was providing software in a B2B-sector. Very boring, very non-startupy. They were considered innovators for doing webapps and SaaS in a market where everybody was used to win32 UIs made in Delphi or C++. The tech stack my employer used was Java EE, PostgreSQL, with a frontend in EmberJS. The prevailing view at the time was that we knew how to do applications on the backend with MVC, and that 'if we just figured out how to MVC in a JS-framework, everything would be fine'. Some went with Angular, some with EmberJS. Build steps for frontend were in its infancy, we used Gruntfiles to just concat everything together essentially.

When React started to gain traction, everybody scoffed at them. JSX was weird, uni-directional data flow was weird, ES6 classes were weird, too many building steps were weird. But it quickly became clear that this was the way forward. Traditional MVC approaches quickly failed once you got out of the example code and into more complex use cases. Which brings me to my point: React was better IF-AND-ONLY-IF your webapp needed to do more than basic forms.

React was never meant for consumer websites. It was never meant to be the default tool for anything on the web. If you wrote a web application, something complex, something with forms, planboards, custom UI, then yeah, it would be a sensible choice. It was meant to solve the problem that doing two-way binding of data is incredibly difficult if you have anything more than some standard form components.

The reality of the situation is that innovation in frontend websites (as opposed to webapps) has mostly been standing still, as everybody just switched to using React. CSS has gotten much better, but doing something like a form rendered with PHP + some validation with JS is still a pretty painful experience. HTMX is probably the best attempt I've seen, along with Elixir and the Phoenix LiveViews.

Most of the effort seems to go into writing a bunch of libraries for React to solve the complexities of having to deal with a complex set of tools for writing complex webapplications. These abstractions tend to either be complex themselves, or are leaky.

I realize my next sentence might as well be 'get off my lawn', but I really hope that devs stop focussing on writing the next state management framework, or routing library, or even rewriting existing things, and just try something to actually move dev forward. Open source a kit of sensible standard components including styling for webapps. Write a simple JS tool for some form validation. Show your barebones NodeJS app rendering HTML and processing forms in an PHP-esque way, give it a logo and a fancy name, chuck it up on GitHub. Don't try and combat complexity with more complexity. We have all the tools we need to write complex webapps. Please make simple websites easier, so people can stop complaining about the complexity of the complex tools meant for complex use cases that they don't need for their simple case.