Aluminum hydroxide and phosphoric acid, I think, mixed with enough suitable aggregate that it doesn't explode when you mix it. Look up recipes for castable refractory. Refractory materials aren't
always more durable at ordinary temperatures — olivine sand in particular will serpentinize if allowed to weather — but many of them are.
There are actually lots of artificial materials that weather more slowly than rebar-reinforced portland-cement concrete. But buildings are huge — a small house commonly contains many tons of building materials, and a large building contains thousands of tons — so the overriding consideration for building materials is that they be cheap as dirt.
In fact, buildings — then and now — are to a very large extent literally made of dirt. And rocks. Bricks? That's clay, which is a kind of dirt. Mortar? That's sand, which is a kind of dirt, glued together with typically a small amount of ordinary portland cement with an even smaller amount of additives. Drywall? That's gypsum, which is the kind of dirt White Sands National Monument is made out of (as well as, in its mineral form, many stunning ancient Egyptian vases) although a significant amount of modern gypsum is sourced from flue-gas desulfurization. Plaster? That's sand glued together with cooked gypsum, except when it's sand glued together with quicklime, which is made by cooking chalk. And of course adobe is sand glued together with unfired clay.
The issue is that costs that seem totally reasonable when you're dealing with gram or kilogram quantities of a material become totally unmanageable when you need a hundred tonnes of it. You can use glass, wood, or steel (the cheapest metal) in more modest amounts, but they generally can't replace OPC.
So, there's a pretty short list of materials that are both strong enough and cheap enough to use as building cements, and which can additionally be synthesized on a building site between grains of sand (as opposed to, say, in a furnace isolated for oxygen, or at 500°). You have:
· the above gypsum, quicklime, ordinary portland cement ("OPC"), and unfired clay;
· tar (asphalt);
· geopolymer cements (milled metakaolin polymerized with waterglass);
· waterglass itself;
· Sorel cement (magnesium oxychloride);
· magnesium oxysulfate;
· calcium aluminates; and
· the phosphates of aluminum, magnesium, and occasionally calcium or even zinc.
Of these, some have worse weathering properties than OPC: unfired clay, Sorel cement, magnesium oxysulfate, and (if cured improperly, in a famous nationwide disaster in Spain a few decades ago which wasn't discovered for years) calcium aluminates.
But there are all kinds of extremely stable materials that would last much, much longer than concrete in the weather, but which are just too expensive to make buildings out of, or which nobody knows how to make big enough pieces of. Soda-lime glass (though foamed glass and fiberglass have uses), aluminum, titanium, aluminum bronze, epoxy "granite", gold, silver, polytetrafluorethylene, sapphire, yttria-stabilized zirconia, yttria, carborundum, silicon, the rubber used for car tires, and so on.