Hacker News Reader

Comment by ardit33 | Hacker News Reader

And the circle of life continues! I have seen exactly this method in 2004-2006 but with XML.

If you are reading this, and you think it is a good idea, please don't follow it as it is not.

Back in 2006-2009 I worked at MobiTV, which we had the same exact way/method of programing our mobile apps, on J2ME/feature phones. The made up language was called CEF which was based on XML. While the idea is sound in paper, This turns really fast into an ugly, debug everywhere type of scenario, where it is hard to both develop and debug properly.

The only way this is ok, is when you have some fully automated UI tool, that stores some kind of state, and retrieves it back, but with no human intervention and some very very limited logic. Otherwise you are in a world of pain. If you try to include real application logic into something like this it is going to hurt your business long term.

It will turn your business logic like some Maven configuration, full of gotchas and more.

View on HN

I am not saying that you are wrong. Not at all.

However, something I’ve come to learn is that some ideas I thought were bad were actually not bad. They were poorly executed and the exact reasoning why they did not work well in a given application was not well-understood.

So I can see an argument for “this was horrible back when it was done in the early 2000s” simultaneously with “this could actually still be a good idea for some use cases.”

Currently reading about agriculture throughout human history. It made me wonder how many people failed to domesticate a single kind of crop and assumed crop domestication as a whole isn’t a good idea.

I realize that people need to make decisions based on their personal experience, and I don’t have a good rule-of-thumb for when to try a variation of a “bad” idea.

‘Many of life’s failures are people who did not realise how close they were to success when they gave up' -- Thomas Edison.

said the man who tested something like 6000 different kinds of plant materials as filaments for a light bulb

Three cheers for brute force methods.

Sometimes you need the BFH to get to the elegant solution.

Said the man who lied, cheated, and stole his way to fortune and historical fame.

You’re just jealous

Agriculture is difficult because it is literally a fight against nature/entropy. And it still has to be proven that it is a viable way to drive human civilization in the long term.

In nature, species populations compete with each other and are kept in a dynamic equilibrium. If population size of a species increases, then its predators will have more opportunities to predate, and their population size increases as well. This leads to more predation pressure, which should negate the population increase and decreases opportunities for predators as well. "Predators" is to be understood in a very general way. It can also apply to fungi and other microbial forms of life.

Changes to this equilibrium arise when prey species evolve to resist the predation pressure by predators. And vice versa if the predators improve. Then, ecosystems change, and these changes can be very disruptive.

Agriculture is a rebellion against this dynamic equilibrium because it creates irresistible opportunities for predators. They have to be held in check either by brute force (hunting, pesticides, containment) or by hacking the ecosystem (for lack of a better term) to ensure that helpful predators are around. The former is probably unsustainable in the long term as it is not at all clear that we will forever be able to outsmart the evolutionary potential of predator species. The latter is difficult as well.

Apart from predation, the other challenge to agriculture is that it depletes the soil in the long term. Fertile soil arises as a result of a ecosystem where the decomposers cannot keep up. Most forms of agriculture interrupt such ecosystems and thus deplete the soil in various ways.

To keep this in check, different methods were evolved, such as slash-and-burn, crop rotation and artificial fertilization. But it is not at all sure that we will always be able to stop depletion of our soils just in time to prevent civilizational collapse. Currently, availability of phosphorus is a looming danger because most of it is mined. Once we hit peak phosphorus, we better have solutions in place to get it from somewhere else.

Sharing code between frontend and backend isn't a bad idea, yes. However, using Json for it is. And the problems and limitations with json can't be "fixed" in any way.

> Sharing code between frontend and backend isn't a bad idea, yes. However, using Json for it is.

I don't see why serializing close to JSON is any worse than doin so to, say, S-expressions, or Algol-style syntax, or WASM.

If you aren't manually coding it, but generating it, the capabilities matter, the syntax matters a lot less.

I think there are very good reasons why languages have their own custom formats and none of them ever decided to serialize into json, even when just "generating it".

No I don't think there is a good reason. You're completely missing that programming language syntax is designed for humans. You can still serialize all languages to json syntax trees, just like you can do so with lisp notation, Polish notation etc.

I'm not saying that you _cannot_ do it. But tell me: why has no programming-language done that? They all could have chosen to serialize their code or AST into json but they didn't. (except JsonLogic I guess)

Because someone didn't have the time to do it? Or they didn't think of it? Just because nobody has had success at an idea, doesnt mean that the idea can't work.

Or do you think we are at our peak?

I mean, you are not wrong. But I think the chance for all these people just missing out or not having the idea seems quite low to me.

> all these people just missing out or not having the idea seems quite low to me.

You mean like every single idea/project/etc that ever existed before their invention?

Unfortunately, it has taken me 40 years to learn that most people have no clue about innovation. I mean, imagine believing there are no good ideas left, all the good songs have been written.

From now on, when someone tells me something is a bad idea, I'll take it mean that I should examine it further.

People did do it (JsonLogic by your own admission) and similar things (every single Lisp based language every) so if your only argument is "people didn't do it before, therefore it cannot be a good idea" it doesn't work.

> I think there are very good reasons

But what reasons and are they relevant to this use? For instance, I can think of one really good reason (actually several different ones justifying different formats getting used depending on the actual use case, but they can all be summarized the same way when it comes to not using JSON): Shipping an AST between endpoints one of which is a web page and therefore has JSON as its best supported generic object serialization format isn’t the use case.

I think the key point on this is `some use cases`.

For people who've spent a lot of time with ANT, you might be familiar with this article from its creator: https://web.archive.org/web/20041217023752///x180.net/Journa...

It was possible and even made sense to use XML as the format for a build tool, however, it got stretched way to far. The idea was applied in use cases that were less and less appropraite, as a result years later you end up with something that is unwieldy and doesn't make sense and, generally, thought of as "terrible".

There are use cases were JsonLogic might make a lot of sense, and for those that is great.

Generally, I'm sceptical as there is high risk of increasing couplings in a way that will make change much harder. This will be particularly true if it is applied to use cases that don't truly warrent this kind of solution.

One valid use case for solutions like this is if users are allowed to run their queries inside a database or api.

I'm often in the situation where you try to add all kinds of parameters to a query-field in the UI but no matter how hard you try there will always be a power user with the request to do some combination of AND/OR/CONTAINS/NOT with nested parenthesis that is not supported.

After hundred such requests you get sick of it and just want to expose the raw DB-query to the front end, which obviously is a bad idea for security reasons. Having a safe intermediate representation that is sophisticated enough for some logic can be a good idea. Instead of reinventing your own, using this language can be done, as long as it has a mapping into each database query language.

> there will always be a power user with the request to do some combination of AND/OR/CONTAINS/NOT with nested parenthesis that is not supported.

If that power user is properly skilled, authorized and authenticated, why not give them a read-only connection to the DB and let them write their own SQL?

How many times has the UI been nothing but an impediment?

> exactly this method in 2004-2006 but with XML.

Of course! JSON is the modern equivalent of XML. In 10 years everybody will mock JSON just like today we mock XML.

JSON mania has given me an appreciation for XML not having been so bad after all.

I dunno, to me it's pretty clear that JSON has improvements over XML; XML:

- does not entirely make it clear what should go into a tag attribute versus a nested tag

- conversion of those two features into a unified internal datatype can be tricky (for example, this property led to choices that for erlang's xmerl causes a security regression if you are handling user-submitted xml).

- first class support for text inside of a tag is not what you want in 90% of data interchange events, and support for it creates a burden when deserializing.

- it's confusing what should be a text inside a tag versus a tag attribute. Or should that text be parsed into a number? At least JSON is strongly typed, you know what datatype is what just by looking at it.

- because there aren't good "best practices" around it, everyone rolls their XML schemas with their own ideas about what sort of thing should go where.

The major benefit of XML is its obession with schematization means that at least in many case you can grab a schema and figure out an appropriate declarative command to obtain what you want (if you have the right library), but this has basically been conquered in JSONland using JsonSchema/OpenAPI, and stuff like JsonLogic.

> - it's confusing what should be a text inside a tag versus a tag attribute. Or should that text be parsed into a number? At least JSON is strongly typed, you know what datatype is what just by looking at it.

JSON's types are pretty anemic compared to the XMLSchema's datatypes[0]. Even it you don't use all of XSD's complex type system, having these basic datatypes is extremely useful and I wish JSON had similar.

[0] https://www.w3.org/TR/xmlschema11-2/

> JSON's types are pretty anemic compared to the XMLSchema's datatypes

The same thing is true if you replace “JSON” and “XMLSchema” with “XML” and “JSON Schema”, respectively.

And also, though less so, if you do the same but replace “XML Schema” with “JSON” instead of “JSON Schema”.

- JSON just has UTF-8. XML has encoding madness.

- JSON has just one escape format and all unicode codepoints can be used. XML generally allows only certain unicode codepoints, at some places like element names its further restricted (Quick: Which characters are not allowed? What's the difference between XML 1.0 and 1.1 here?). XML escapes at every place differently. In comments you need to escape less than elsewhere. In attributes you can use newlines but they will get normalized to spaces, you have to hex-encode newlines. There's Cdata.

- JSON doesn't have the entity madness with potential security issues.

- JSON doesn't have namespaces that add a lot of complexity with little benefit

JSON maps nicely into a python dictionary.

JSON is arguably easier to read than XML

JSON can't express ordering or duplicate fields in a consistent way.

Consider how you'd write these 2 bits of XML in JSON:

vs.

Without a schema, in the first case, how do you determine whether B should be "foo" or ["foo"]? In the second how do you preserve the relative ordering of B #1, C and B #2?

I worked on a project where we had a parse tree that we wanted to serialize out, and for this reason XML was the natural choice.

{ "a": [ { "b": "foo" }, { "c": "bar" } ] }

vs.

{ "a": [ { "b": "too" }, { "c": "bar" }, { "b": "uh oh" } ] }

Yes, so in essence:

- Only ever have a single key in each JSON object (treating them as tuples)

- Make all object values arrays.

It makes for some very unnatural looking JSON.

The only thing that doesn't look natural to me is that it's all on one line.

Otherwise it's very clear about the data's format and when there's ordering required

> It makes for some very unnatural looking JSON.

Not to me; its a fairly standard and natural JSON pattern. It’s tue that it use is usually quite limited in any given representation, but usually the thing you are modelling in JSON isn’t general XML with no further knowledge of application domain or logical schema.

> { "a": [ { "b": "foo" }, { "c": "bar" } ] }

This looks like line noise to me. All that punctuation and quoting looks ridiculous. What is the problem with straightforward and trivial to parse things like

    [a]
    b = foo
    c = bar

or even

    a.b = foo
    a.c = bar

So this is exactly a symptom of unclarity of the roles that xml causes in the first place. It seems like You're trying using the tags as keys instead of as information categories. In json you might encode these sorts of data as something like (pardon my lack of quotes, I'm on mobile):

    {a_list:[{text: too, type: b}, {text: bar, type: c}, {text: uh_oh, type: b}]}}

But generally you shouldn't have a mixed type list in json unless it's a deliberate choice, the "limitations" of json are exactly what devs want because it corresponds to a clarity of thought and purpose that simply isn't there in XML.

In my case the parse tree had nodes that corresponded with grammar rules (mapped to XML elements) and nodes that corresponded to the preserved but otherwise uninteresting fluff that surrounded them (mapped to XML PCDATA).

We wanted a parse tree that could be serialized out to produce the original document byte for byte, but still reflected the grammar.

Things get uglier for JSON in the real world case:

    "5+7 #comment"

becomes:

    <sum><integer>5</integer><operation>+</operation><integer>7</integer> #comment</sum>

The XML output is essentially the original input + grammar markup. Your solution also works but is a data structure representation with a lot more line noise for humans to read.

It just looks bad because I made it one line; by comparison if I hadn't seen the pre-parsed form 5+7 I would not be able to immediately grok what is going on in your parsed one-line XML, either; there's a lot of "line noise" there too. Moreover it feels like the XML you have has lost the structure of the meaning, by having the tokens 5, +, and 7 be linearized. You probably want a tree with + at the root and 5 and 7 as leaves. That XML encourages these sorts of "oddly flat" representations is one of its weaknesses.

+ at the root would indicate an abstract syntax tree rather than a concrete or "full" syntax tree (terminology varies slightly)

This was primarily a mechanism to load parsed documents in to code that didn't have access to the recursive descent parser that parsed the document, for either security or performance reasons.

Loading the XML with the appropriate parse flags, doing a pre-order traversal on all the leaf PCDATA, and then printing it out, would regenerate the original document byte for byte.

Perhaps I should go back and implement JSON output and see how I feel about it... thought I'm pretty sure once you introduce a bunch of "text" and "type" fields to the JSON tree it's going to look a lot less readable than XML markup.

There are other fields as well like line and column numbers (output as XML attributes)

While I agree that XML has muddy-thinking whether tags are types within a collection or properties of an object, I've often felt that Json's inability to store heterogeneous object arrays meaningfully is a big problem. The lack of a meaningful "type" attribute in JSON schema is a pain.

I've had to roll my own hack for such cases.

> Without a schema, in the first case, how do you determine whether B should be "foo" or ["foo"]? In the second how do you preserve the relative ordering of B #1, C and B #2?

Well, if you are literally developing a model for recasting general XML in JSON, then you know the children of an entity should be a list because duplication os possible and order is signifcant. (Given the close relationships between XML and HTML and between JS and JSON, the obvious starting point for a general solution is to copy the HTML DOM, using only the properties.)

If you are trying not to reserialize XML generally but to solve the same problem as the original XML, then you probably aren’t doing it without a (logical) schema—you know what the object model is that you are trying to stuff into JSON, and you decide based on that.

I like JSON a lot more than XML for what JSON is used for. XML did have some advantages, like more standards for schemas and templating, but the JSON ecosystem has at least decent facsimiles for these facilities. But the real star of the show is the data model: XML is a markup language. Jamming data structures into it is awkward. Not so for JSON, which has two aggregate types, a list type and a map type, followed by the usual suspects of primitive types: string, number, boolean, null.

I can implement a full, strict, compliant JSON parser in an afternoon, maybe even less time. It is simple enough to parse that you can even do it in pure C with relative ease. XML is a lot more complex and a lot of implementations skimped pretty hard on the details. I mean I have fond memories of tinyxml2, and if you take apart any app made in the 2000s you have a good chance of finding reference to it (or on Windows-only apps, MSXML3, or on Linux, libxml...) but it certainly let you do a lot of surprising things. I saw a lot of apps doing XML documents with multiple root nodes. And Apple's plist format is a nightmare. It's so easy to fuck up interpreting XML plist, that Apple did, and it lead to one of the more entertaining privilege escalation bugs in iOS history.

I have some fond memories, but I just don't miss XML. :)

One thing I miss from XML is namespaces. Made it easy to extend an existing format with some custom extensions, with the likelihood that tools that didn't know the extensions would just ignore them, and that your extensions wouldn't conflict with anybody else's or with future versions of the underlying format.

I haven't seen any widely-adopted equivalent in JSON. I've seen some proposals floating around but none of them seem to have really taken off.

God no. No one handles XML namespaces well. They are an absolute nightmare to deal with in almost every language.

We already have this in JSON: add a hashmap key with a prefix. It's at least as robust provided tools do the sensible thing and ignore keys they don't care about. But as with XML, sometimes, they don't, or do, or something.

> add a hashmap key with a prefix

Maybe I'm paranoid, but I worry someone else will pick the same prefix as me. If you make the prefix something like a domain name, e.g example.com:myExtension, nobody else is going to use the same prefix assuming you own example.com. That avoids the risk, but it can be a lot of typing. Whereas, if the prefix is something short and obviously meaningful (e.g "doc:"), there is a risk someone else will use the same prefix in a conflicting way.

What I liked about XML namespaces is the indirection: the prefixes can be whatever you want, and then what makes them unique is the URIs they are defined to map to in each document.

The one point at which I think it didn't work very well, was when people started using prefixed names in attribute values. You can't tell whether "foo:bar" is a reference to the XML namespace with prefix "foo:", or some unrelated string of "foo:bar", without looking at the schema.

In practice though this doesn't really happen: nothing stops someone coming up with the same seemingly unique namespace URI, or using the same namespace prefix.

But none of that is really the problem - the problem is that the implementations and parsing are evidently sufficiently difficult that people just don't get them right - hence the hassle.

I would argue also that having indirection like this is just a negative all around - having to look elsewhere in the file to figure out what I'm looking at just creates problems, and in practice it doesn't look like this capability was ever really used - the only times I run into it are when I'm dealing with some vendor-specific implementation (that Python or Go or whatever can't handle so string-templating winds up being easier) and it turns out that what I need to have happen is to match the semantics of example XML exactly to get anything to work.

Which is really the point: the extra complexity in parsing means the probability you're dealing with a broken parser shoots through the roof, at which point you've lost any advantages. Whereas with JSON, the minimum bar is arrays and maps and the basic types work. Stay within those bounds, and whatever namespacing that happens will "just work".

> nothing stops someone coming up with the same seemingly unique namespace URI

If people use URLs, then DNS gives uniqueness to URLs. Of course each registrant has to enforce uniqueness within their registered domain – for a standards organisation or large software vendor, they often adopt a formal system (spreadsheet, database, naming convention, etc) for tracking which parts of the namespace under their domain name are reserved for which products/standards/etc. Of course, that relies on the vendor/organisation to do it properly – but if (e.g.) Microsoft were to create a mess by letting different Microsoft products use conflicting namespace URIs under microsoft.com, well that would be Microsoft's fault, and not the responsibility of any standard or technology to fix it.

(There is still one issue–what if the domain changes ownership? Some standards, for example ISO 19770, have tried to address this by constructing namespaces based on a domain name with its registration date, so if the domain later expires and is reused by a different organisation they won't clash since they'll have a new registration date.)

> the extra complexity in parsing means the probability you're dealing with a broken parser shoots through the roof

A lot of that is because namespaces were added to XML later, rather than being there from the start; they are effectively an optional feature. So then you have some parsers that support them, some parsers that don't, some parsers where it depends on a flag you have to set (which may default to true or false). I guess trying to add namespaces to JSON now would encounter the same problems. But if someone was to design a new serialisation format from scratch, and had namespace support in it from day 1, I don't think these issues would happen.

Namespaces in XML might be ok in theory.

In practice not so much.

You will curse namespaces when you encounter some XML with multiple namespaces and your favorite parser starts choking on them.

> I can implement a full, strict, compliant JSON parser in an afternoon, maybe even less time.

Can you really? Doesn’t look like anyone else can: http://seriot.ch/parsing_json.php

If you actually look at this, the results are not nearly as bad as title suggests.

There are are many parsers which perfectly parse every valid JSON document (like every Perl, Python and PHP parser and many others). They got non-perfect score because they accepted some JSON which standard says they should not have -- like a NaN value or a trailing comma.

While using such parsers can potentially cause problems -- perhaps because your data generator has a bug, and your not-strict-enough parser ignores this bug -- this happens very rarely in practice.

Maybe? :) I haven’t tried.

I did do this one actually in an afternoon:

https://github.com/pangbox/rugburn/blob/master/src/json.c

... but I will say that it is intentionally very limited in its API due to its intended use case and desired footprint. (And because it’s designed to compile with OpenWatcom 2 with no libc for only Win32, but those particular constraints are not very interesting for this.)

I would imagine if I did try, I would have a fighting chance at making something properly strict. But I admittedly don’t know how to handle recursion in a stackless manner.

Edit: specifically for C, it would be hard to do it properly unless I had access to an implementation of Clinger and Steele & White. Go would be easier since it has better floating point algorithms with strconv, but relying on C strtod/etc. is probably a bad idea due to variability. That having been said, maybe you could do Clinger in the same afternoon; I’d be worried about edge cases, but simply put, I haven’t tried.

I am not a programmer, and I have written one over the course of a week. It passed all the tests in the particular JSON test suite mentioned in the article except one, by design. It allowed bad unicode in parts of the document, because that was the data it was being fed.

I am not even a programmer by profession. Anyone that can write a recursive descent parser can write a JSON parser.

I dunno, for me JSON will always be better than XML due to it just being nicer to read / work with.

You have clearly only ever had to deal with very simple JSON

Haha nope, I've dealt with JSON files that were deeply nested and in the 100s of MBs.

When you're dealing with complex data, JSON isn't very fun, but XML is still worse.

I've worked with Will on a large XML based system. I'll try to briefly describe our workflow. Our company created reference books. Those books ended up both as printed books and as HTML in a web application service people could subscribe too. Various people would create the content, some in WYSIWIG editors, others editing raw html to tag content to give it structure, and tools that fall somewhat in between. XML tags and annotations can do things like indicate where the title is, sub headings, links to other books or content. etc etc.

This annotation structure not only lets us transform the xml into HTML that resembles the physical book via CSS, but also lets us search the content with awareness of what the content is. Simple example, if a search term appears in a title give it more weight.

When a book is done the XML goes off to a printer to become a book, AND into our XML database. A nice feature of which is a lot of our content is in the DB as XML, you ask for a book page and the DB returns the HTML for the page, backend just forwards that HTML to the browser which displays it. ZERO parsing/transformation anywhere once the DB sends it! Just fling the payload at the browser. Super cool at times.

Anyway we all work with these formats differently, some of us never touch the json and xml, some of us only tweak values in config files. Those of us who work with it like that, the only differences between XML and JSON are "how nice are the serialization apis and how fast are they? how big is the payload?"

But when working WITH the format directly there can be more meaningful differences. And the tools that actually exist for the things you are doing may push you towards one format or the other, even if in principle it wouldn't have really mattered.

note: I keep mixing tenses here but this was all in the past, we got bought, don't do any of this anymore!

Gotcha, I was thinking more in terms of JSON as a serialization / interchange format (given the OP), rather than use cases such as the one you describe above.

To clarify: no, I do not believe that JSON is universally better than XML. I believe JSON is better than XML for the majority of use cases where people are currently using JSON today, esp. as an interchange format.

This. Once a document is big or complex enough, the benefit of human readability fades away and you are left with a brittle and very slow serialization format.

How much slower is JSON serialization than XML serialization?

Roughly in the same order of magnitude, depending on the implementation.

Light-years away from zero-copy serialization formats.

See https://en.wikipedia.org/wiki/Comparison_of_data-serializati...

Random example: https://google.github.io/flatbuffers/md__benchmarks.html

Speed is not everything: being able to self-describe data structures (without the data) enables generation of parsers, documentation and tests.

Streaming support also matters.

It's not. The grammar for JSON is vastly simpler than XML.

(I'm certain one can find a slow JSON serializer out performed by a fast XML serializer, but that's not the point.)

It's just arrays and maps. Regardless of how large or deeply nested json gets, it's always very simple.

With equal samples in the two formats, JSON wins on readability. I can draw up bad examples in both. Since you've seen back JSON already, I give you real-world XML (I actually saw this example, yes), describing whether a structure has a roof, or not.

  <roof>
    <true>
      1
    </true>
    <false>
      1
    </false>
  </roof>

(Normally, exactly one of either the <true> or <false> tags should have a "1", and the other a "0". Normally.)

(The really horrible thing was that there were … I don't want to call them "reasons" because that's a strong word, so let's say "cause and effect" that led to this format… and once you came to know them, you "understood", in a sort of horrible sense, the format. But give me a JSON bool any day over it.)

You don’t state your reasons for this very bad XML, but your comment suggests to the uninformed that XML doesn’t support Boolean types, which isn’t the case.

https://www.w3.org/TR/xmlschema-2/#built-in-primitive-dataty...

XML doesn’t support booleans. XML Schema, one of several separate add-on schema languages for XML does, but comparing XML + XML Schema to bare JSON does not make sense.

XML Schema isn't only for validating XML post-hoc, as you suggest. The point is that if you are using XML, it doesn't make sense to invent arbitrary solutions when there is already first class support in standardized toolchains. Comparing the markup specs, specifically, is an unreasonably narrow comparison. The discussion is about "using XML" and "using JSON", and the environments in which they're used are at least as important as the specs, independently.

If you are somehow in an environment where you are working with data nominally encoded according to XML spec, yet unable to leverage any other parts of the standard XML ecosystem, then I would certainly agree that XML is a bad choice!

People mock JSON today. No comments, hideous multiline strings, and weak schema support.

The problem is XML is _markup_ language and it was forced into data serialization -- hence endless mocks and people hating it.

There are people who actually use JSON for markup... the resulting schemas are usually very bad, and they disappear quickly without even gaining fraction.

When used properly -- XML for markup, JSON for serialization -- each format is pretty nice, and will probably last forever.

LOL, I am just going to leave this here, but to those that complain about JSON or XML or any future format X: It's always going to be shitty, messy and hard to read. You can only do so much to get rid of extraneous symbols but at the end of the day the format is hard to read because there is hierarchy and repetition. It seems like everyone is waiting for a format that will make reading 2000 configurations "easy". I'll have to say, good luck.

It sounds like your language didn't steer clear of Turing completeness.

That's usually how this type of thing goes wrong. I've seen similar things both work and fail for that reason.

This language made it an explicit goal, though.

> If you are reading this, and you think it is a good idea

I came to the comment section here hoping that the site was actually a joke...

I have successfully implemented this exact project in the past. The project allowed for runtime logic modification however tracking states was built into the platform to make debugging simple. As with every other library or framework, JsonLogic is a great tool if you implement it responsibly.

Maybe it's a kind of having your cake and eating it too: enjoying the flexibility of passing code around and running it with eval, while not having the security, stability and runtime problems implied by eval.

There are limited scopes in which this can work but it seems to be more about the environment or code loader than about the programming language, per se: for example, plugin systems for build tools and servers. The plugin might be isolated by runtime features (like in the JVM) or by processes (as some web servers do, or as Postgres does) but it's not language features that tend to be useful.

Tensorflow models and CSS selectors are both examples of fairly useful plugins that still have limited enough semantics to be sandboxed effectively.

I agree that there's basically never a good use case for shipping logic from server to client or vice versa in a formal application setting. BUT. I could definitely see a scenario where something like this would be handy for farming distributed processing out to naive nodes or workers that don't / shouldn't have access to the core logic.

Maybe you can tell me if there's a better solution. I'm open to hearing any options because I'm at a loss of how to do this otherwise:

Say we have a schema which defines certain data. Let's say it's a "credential" which is a "COVID-19 Back to Work Credential." The credential is valid in the case that someone either 1) has a COVID vaccination or 2) has a negative COVID test in the past 72 hours. That data might look like this:

  {
    "credentialName": "COVID-19 Back to Work Credential",
    "credentialIssuer": "Acme Labs Inc.",
    "credentialData": {
      "rapidSarsCov2PCRTest": {
        "hasTestResult": true,
        "testResult": "negative",
        "testTime": 1622145851
      },
      "covid19Vaccination": {
        "hasVaccination": false
      }
    }
  }

Now, say I want to define the conditions under which this "credential" is valid or invalid, but I want to include this "matrix" within the schema (not any business logic, just matrix logic that pertains to the data within the schema itself).

Optimally, I'd be able to do this within the same schema or file.

With JsonLogic, that might look like this (probably flawed logic, just a demo, not using this in production, may cause your computer to go up in flames, etc.):

  {
    ...

    "parsingMatrix": {
      "if": [
        {
          "or": [
            {
              "===": [
                { "var": "credentialData.rapidSarsCov2PCRTest.testResult" },
                "negative"
              ]
            },
            {
              "===": [
                { "var": "credentialData.covid19Vaccination.hasVaccination" },
                true
              ]
            }
          ]
        },
        "valid",
        "invalid"
      ]
    }
  }

Can you think of another way to do this that doesn't rely on using JsonLogic or something like it (or a cleaner method for doing it in JSON)?

I'm not sure how to express how much I loathe everything about this "solution."

To answer your question, what you have is (effectively) code, and just use a programming language for this, call it a function, give it a name, arguments, test suite and register it as a plugin to your platform. Oh, and documentation, change history, etc.

Seriously - the "overhead" more than pays for itself when 100 other yahoos do this nonsense and you end up with data files containing untestable code that breaks when combined with other code, breaks when the platform is changed, etc etc

Ok, I take your point, but let's say we need to share these credential objects between different systems that do not have prior knowledge of how to evaluate the credential data. Also, let's say we constrain the logic to only allow for evaluation of any given part of the credential data against a binary "pass | fail" matrix.

What is a better way to share this important part of the schema (the matrix) without resorting to letting the "business logic" perform the evaluation? Optimally, any matrix should be able to run against the same business logic for evaluation.

Also, though I understand why you say that, we won't have "100 other yahoos doing this nonsense," and each schema would go through a pretty thorough review process.

If your business logic doesn't meet the needs of your business without the schema, the schema is a part of your business logic, that should be subjected to a suite of automated tests.

If your concern is the sharability, consider making a library or service that provides the business logic.

We'd like to be able to include evaluation matrices within credentials as an extension to the Verifiable Credentials standard (see https://www.w3.org/TR/vc-data-model/). Optimally, different libraries and different services will be able to evaluate a credential using the credential's included matrix (i.e. the evaluation matrix should not be reliant on any business logic from any particular party). I can't get too deep into the use-case but having these parts bundled together is important for what we're working on.

> but let's say we need to share these credential objects between different systems that do not have prior knowledge of how to evaluate the credential data

Well, they _must_ have prior knowledge about it, because they must have to understand the concept of JsonLogic beforehand. So instead, you can just make these systems understand a proper programming language that can produce outputs. Something like https://github.com/dhall-lang/dhall-lang would probably be a much better option for this usecase.

With the standard that we're working with (the W3C's "Verifiable Credentials" standard [0]) having something that's in JSON or at least "JSON-compatible" is quite important.

[0]: https://www.w3.org/TR/vc-data-model/

No problem, just serialize the dhall into a json string.

What about an "isValid" field returned by your API?

Your problem is that you need the same logic in two places, and you can solve it by centralizing this logic in one place.

Sometimes you need to embed an expression interpreter in your application, for example to allow users to define them at runtime. In cases like that one, having a library with a well defined specification could save a lot of time.

It's much more likely that the library would do more harm than good.

Libraries tend to cover broader spectrum of use cases than you'd need. Usually that's okay (eg: URL parsing, string manipulation, HTTP client, etc) but in cases like this the logic schema should be as absolutely minimal as possible.

If you're building an interpreter and need an interchangeable format then you can always generate that out of your own bespoke and very constrained format.

How about representing the logic as a JavaScript expression:

    (credentialData.rapidSarsCov2PCRTest.testResult==='negative')
      || credentialData.covid19Vaccination.hasVaccination) 
          ? 'valid' : 'invalid'

Seem easier to read and maintain to me.

I'd love to do this, the only problem there is we're sharing this data between systems which use different runtimes. A client who runs a Python environment might not easily be able to make use of that logic. With JsonLogic, that would theoretically be possible.

You can easily write a parser for this in Python. As long as you only support expressions it should be pretty straightforward.

I see your angle. I'll consider this. Thanks for the ideas!

I posted this in another thread, but it seems relevant to what you are looking for: https://stopa.io/post/265

It walks through the process of building a Lisp in JSON and the associated parser in JavaScript. It could be a good starting point to build out a cross-platform approach to sending logic over the wire.

Nice, looks quite neat. I'll dive into it a bit deeper!

I can't help but think that at the end of this road is yet another Lisp waiting to be discovered.

Well JsonLogic is basically Lisp expressions in Json syntax which is kind of the worst of both worlds IMHO.

It seems weird to me to pass a syntax tree over the network rather than just the syntax. But if the expression is not supposed to be viewed or edited as code at all, but only in some kind of GUI expression builder, then it might make sense.

Lisp is at least human-writable, but if it is supposed to be written by humans why not just use straightforward expression syntax like "a + b".

Btw google have something almost exactly like this: https://github.com/google/cel-spec

Straw man.

With all due respect, you're making a classic mistake: presenting the solution to a problem without tracing the problem back to the use case.

Why do you need to include the logic matrix in the schema? I can promise you that the real world use cases where you'd need to this are virtually non-existent. You wouldn't even do this for some extreme case like Mars rover software.

Your business logic should have a single source of truth. The vast majority of the time you can just write an API for other software to synchronize with.

Otherwise just ship new builds of your software package to the client instead of a new schema. If you need to be modular then use modular primitives such as shared libraries.

Again, there's virtually no real world scenario where you need to ship a logic schema (in any format) to a client. You might as well ship a shared library, importable module, a completely new binary, etc, etc.

Interoperability is the main reason. We need this data and the evaluation matrix to be portable across different "credential wallets" as part of a standard model for matrix evaluation of Verifiable Credentials (as an extension that we're working on to the existing data model https://www.w3.org/TR/vc-data-model/).

You didn't state the use case initially. The context of the discussion is evaluating the value of the approach. What I'm saying is that you can't make that evaluation without stating the use case. Obviously one can think up a scenario where JsonLogic can be useful - that doesn't mean anything.

> Interoperability is the main reason. We need this data and the evaluation matrix to be portable across different "credential wallets" as part of a standard model for matrix evaluation of Verifiable Credentials (see also https://www.w3.org/TR/vc-data-model/).

Representing verification logic in the schema flawed. Just because the W3C publishes a spec doesn't make it good.

So I guess one of the few valid use cases for JsonLogic (or similar) is if your company makes a bad technical decision and you're forced to implement something?

> Representing verification logic in the schema flawed

The "verification" required is one of tamper proof, cryptographic safe certificate chain verification, something like the one done using Certificate Authorities, certificate revogation list and the surrounding infrastructure used in TLS.

Encoding anything Turing complete is out-of-spec.

What about:

    ["if", ["or", ["===", ["var", "credentialData.rapidSarsCov2PCRTest.testResult"  ], "negative"],
                  ["===", ["var", "credentialData.covid19Vaccination.hasVaccination"],  true     ] ],
           "valid",
           "invalid"]

?

That's nice, I like it! We don't really need anything more than this.

It's hard to gauge whether or not it's better to go with an existing, possibly inferior standard, or to create a new one, like this...

https://xkcd.com/927/

I mean it is just lisp denoted in javascript lists, but sure we can call it a new standard if you like.

If this works out we'll either be adopting a previous standard or defining a new one for these evaluation matrices!

But yes, does look like other people have grazed this idea as well: https://www.i-programmer.info/programming/javascript/2380-ja....

Clojure Spec works really well for this sort of thing. It doesn't serialize to JSON, but it would be able to check for it.

You could also think about Dhall as a front-end. You could represent the whole credential as a union type. This isn't deeply thought out, but this should give an idea as to how this would look:

    let TestResult = < Positive | Negative >

    let ScreeningTestDetails =
          { hasTestResult : Bool, testResult : TestResult, testTime : Natural }

    let CredentialData =
          < Vaccinated | RapidSarsCov2PCRTest : ScreeningTestDetails >

    let Credential =
          { credentialName : Text
          , credentialIssuer : Text
          , credentialData : CredentialData
          }

    let result
        : Credential
        = { credentialName = "COVID-19 Back to Work Credential"
          , credentialIssuer = "Acme Labs Inc."
          , credentialData = CredentialData.Vaccinated
          }

    let result2
        : Credential
        = { credentialName = "COVID-19 Back to Work Credential"
          , credentialIssuer = "Acme Labs Inc."
          , credentialData =
              CredentialData.RapidSarsCov2PCRTest
                { hasTestResult = True
                , testResult = TestResult.Negative
                , testTime = 1622145851
                }
          }

     let prop0 =
      λ(x : ScreeningTestDetails) →
        assert : greaterThan x.testTime 1522145851 ≡ True


    in  result2

Notice here that the type for credential data can only be vaccinated or a test result. It can't even represent unvaccinated. You could probably take it farther with dependent types and assertions.

Edit: added an assertion to model the 72 hour requirement.

I cant help but imagine what it would like like in EDN:

    {... 
     parsingMatrix
     (if (or (=== (var "credentialData.rapidSarsCov2PCRTest.testresult")
                  "negative")
             (=== (var "credentialData.covid19Vaccination.hasVaccination")
                  true))
         "valid"
         "invalid")}

I presume jsedn is your library [0]? I was heretofore unaware of EDN, but it looks like a very clean fit for our needs. I will examine this more in depth, thank you.

[0]: https://github.com/shaunxcode/jsedn

It is, but to be clear it is out-moded by the fact that cljs exists now!

> Can you think of another way to do this that doesn't rely on using JsonLogic or something like it (or a cleaner method for doing it in JSON)?

This strikes me as nice:

    (parsing-matrix
     (if (or (= (var "credentialData" "rapidSarsCov2PCRTest" "testResult") "negative")
             (= (var "credentialData" "covid19Vaccination" "hasVaccination") "negative"))
         "valid"
         "invalid"))

> With JsonLogic, that might look like this

Here is the thing...There is already a crappy way to describe JSON Schemas[1,2].

It is hard to grok by visual inspection and expressing conditions like "this element can either be a list containing a subset of these values or a hash with keys that belong to the same set" etc make it unwieldy.

The schema ends up being really brittle and hard to modify. 100% green-lit test coverage don't mean a thing when it is soooo easy to end up with a false negative.

Now take that and add logic to it? One has to question the existence of all the programming languages in use if we are going to end up subjecting ourselves to that torture.

[1]: https://json-schema.org/ [2]: https://json-schema.org/draft/2020-12/json-schema-core.html

Json schema is very useful and is not the same thing as this.

Json schema validates the data structure, while this seems to be more about writing algorithms in json

Precisely. If you want to do logic, use a programming language, not a data format.

Reply on news.ycombinator.com