Red Programming Language: Plans for 2019

I'm not the OP (and not associated with red an anyway, other than being a fan of REBOL), and I agree the wording deserved to be called out. "cult" and "Czech wizardry" aren't exactly positive framing words. Hopefully this doesn't spiral into a discussion about PC comments, because red looks to be cool and deserves a good technical discussion; but I don't think 9214 was off base calling out the comment.

Okay, but just an FYI that was probably a complement.

Wizardry and magic has positive vibes in American culture. I can see how that might not be the case in others.

I don't recall anyone fighting to the death over someones syntactic preferences. Code written by experienced users (and Red codebase in particular) is a hard nut to crack, that's true.

Regarding development: I'd appreciate if you elaborate over your critique. Concrete suggestions on how development and testing processes can be improved are always welcomed.

Development is far from haphazard, but we're always open to examples of how other projects we can learn from, as long as it's a fair comparison and the resource and organization aspects align enough that we can leverage things we learn.

Not liking Red is understandable. Programming languages, as largely subjective HCI, cannot appeal to every single software developer. Though, that is not a reason to be dishonest, and trying to paint a bad picture of our work (even in fallacious ways, as demonstrated below).

> One of the reasons is that the Red community is very small and entrenched.

"Entrenched" is just your opinion, not a fact. Given the rest of your post(s), your opinions about the Red project are demonstrably uninformed and highly subjective.

> They know everything that comes out of the language, and to them everything is easy and understandable.

If the core Red team and main contributors didn't "know everything that comes out of the language, and to them everything is easy and understandable.", that would be very worrying, don't you think so?

> They will fight you to the death if you even try to say that "english-like syntax with no parantheses" is the opposite of readable or understandable when you don't know the system.

Exaggerated, like your whole post. We have civil discussions with people who are actually interested in discussing about Rebol and Red. [1] From the way you sound, I am wondering if you might be one of the few arrogant people who show up on our online channels from time to time, trying to lecture us about "their right way" using a condescending tone...usually, that does not end well.

Fortunately, such bad encounters are rare. We often get lispers showing up from CL, Scheme, or even language authors, like Joe Armstrong (Erlang's author) on our Gitter chats in 2017 IIRC, who liked Red and labelled it a "fun" language. Though, Joe was later upset by our branching into the blockchain world, like a few others in the community, but I hope they see now that we did it for good reasons.

> Go into any of Red's source files and you will end up looking at dozens to hundreds of lines of code with no comments and which often look like this:

    return old-gradient-pen DC cmds start tail cmd sym catch?
    OS-draw-brush-pattern DC size crop-1 crop-2 word as red-block! cmd sym = fill-pen
    _read opfd hash alg-digest-size type

Patching together cherry-picked lines of code from different Red/System files, with zero context...how dishonest. (I give a detailed context and explanation about those lines you have extracted from different sources on a separate page [8], as it is quite long. It also debunks your "no comments" claim as the first line you have picked has 3 comments in the neighbouring lines.)

So, you went to cherry-pick low-level code parts in Red/System, involving OS API calls, for what? For making a preposterous statement like "These are obviously impossible to look up and find."?... Those lines of code are not mine, though it took me only a few of seconds using a basic code editor (not even pulling VSCode with the Red plugin) to fetch the information needed to fully understand those expressions...so much for the "obviously impossible to look up and find"...

People are using free-ranging evaluation in Red and Rebol code since 20 years, and readability is not something users complain about (they were certainly many other issues to complain about, but that is not one). On the contrary, it is one of the Rebol features I have found the most attractive, as have many others. I can understand that your mind is not wired for that, and that you don't like it, but that's not a reason for getting angry at us, nor at Red. The sad part is that you don't even bother asking why such design choice was done in the first place, and what is the aimed trade-off behind it...that speaks volume about your real agenda here.

> The whole development is haphazard as well.

Great, another armchair expert on Red! You are obviously not a Red contributor, I doubt you are a Red user either, so you really have no clue about the project's history, nor goals, nor organization, nor its real-world constraints (at this point, I don't think you care at all anyway, you are just here for bashing). Funny how you jumped from a feature of the language syntax to development process. Your post is really not meant to inform anyone, but only to cast negativity over the Red project...for no good reason.

> Everything hinges on two may be three people who know Red internals wihch frantically move from subsystem to subsystem developing features and patching wholes.

Let me rephrase that properly for you: "The core Red team is working very hard, despite being small, on many different sub-systems, restlessly implementing new features and fixing issues, making the Red users and community happier every day!".

So now being a small, but very capable team is something bad? As you praise a pretty much solo-project later, I don't get the rationale there...double standard I guess. Moreover, you don't even know what you are talking about (again). I can count at least 7 people in the Red community, including the core team, capable of working on the Red internals, adding features or fixing bugs. Maybe it has escaped your understanding, but we are building a stack of programming languages (with lot of system programming), and not a web site. The skills and experience required for properly doing such work are very rare. The few capable people also have real-life constraints, so the permanent core team capable on working on the internals is currently 3 people, with additional 2 or 3 more working on their spare time. We are lucky and grateful to have that many capable people with us already. We are still in the early days, with an alpha Red, so having a small compact team is rather an advantage, as there is still significant design to do.

> a fix is written (with zero comments even for the complex issues) and merged

From 10 days ago, ticket #3692 [2] referenced a hard to reproduce complex crash (an heiseinbug), and includes a few hundred lines of detailed explanations about the debugging process, including some screenshots.

I usually only comment some fixes when the explanation or fix is not obvious for the team. Sometimes, the comment is just put in the commit log. Systematically commenting all fixes would be useless anyway, as many fixes are straighforward, or simply don't deserve any comment (like the "Malignant output from react/link" [3] one you've pointed out below). Moreover, we have a strong presence on our Gitter chat, and are always happy to help contributors ramp up when needed. If someone thinks that a particular fix requires a comment or explanation, we are always glad to provide it.

> About 0.001% of "tested" issues contain any tests. The majority of fixes are apparently only fixed and manually tested against the reported issue.

You are implying here that tests for bugfixes are non-existent. Let's put your extravagant claims to the test (pun intended):

- 1076 tickets with `status.tested` and `type.bug` tags [4]

- 52 tickets with the extra `test.written` tag [5]

That gives a 4.83% result...so about 5000 times more than your claim! That's what happens when you are pulling fake numbers out of your ass. Moreover, as you are wilfully ignorant about Red project (confirmed by your other posts), you don't know that:

* the tests are sometimes provided in a later commit (using a `TEST:` prefix in the log), not necessarily within the fix commit.

* some tickets get a test written, but the `test.written` tag is not set (no stats there, not sure how many)

* we do try to address in batches the missing regression tests for fixes, from time to time.

The last point has led to the creation of a specific test file [6] that gathers tests written in those batch sessions. I count 242 more tests there, so it gives a total of 294 written tests for fixes, about 27.3% as the final result. So much for the "0.001%"...

That result is surely not satisfying though, even if 100% is not a goal there, as explained above, we should get a much more complete coverage for bugfixes.

> etc. etc.

Typical ending for a FUD-spreading post... The only honest statement from you I can find is your Twitter account's baseline [7]: "Opinions on things I know nothing about". That sums up well your posts here about Red.

[1] http://lambda-the-ultimate.org/node/1240#comment-48582

[2] https://github.com/red/red/issues/3692

[3] https://github.com/red/red/issues/3713

[4] https://github.com/red/red/issues?q=is%3Aissue+is%3Aclosed+l...

[5] https://github.com/red/red/issues?q=is%3Aissue+is%3Aclosed+l...

[6] https://github.com/red/red/blob/master/tests/source/units/re...

[7] https://twitter.com/dmitriid

[8] https://pastebin.com/jezLyDVt

Here are a some facts about Red to debunk your attempt at spreading false information, like the "no tests" and "no comments". For the record, we have written a large number of unit tests (in addition to the regression tests mentioned above) since the beginning:

* 22196 tests for Red (with ~10k of them running once for the compiler, once for the interpreter)

* 9847 tests for Red/System

Without this large test suite, the development of Red would simply be impossible. Though, the features coverage is still not exhaustive, as we are still lacking tools for measuring it (contributions welcome!).

Now for the supposedly total absence of comments, here are some stats from red/red repo extracted using a simple Red script [1]:

    -- Type -- FileCount -- LOC -- Comments -- Cmts/LOC (rounded to nearest integer)

       .r         39       25186      3507        14%
       .red       48       16495       776         5%
       .reds     154       85742      7687         9%

Basically, .r represents our toolchain code in Rebol, while all the runtime and standard library are in .red and .reds (Red and Red/System respectively).

Those ratios are not bad, but could be better (10-15% in each category would be a good goal). More comments could be still added in some complex code sections, though embedding a full documentation of the code, in the code, is not our approach (see more explanations below). As an extra note, I have noticed a pattern with wannabee contributors with no system programming skills, of thinking that heavy commenting in the runtime low-level code would magically make them become system programmers...

The plan has always been to provide architecture-oriented documents and code maps to ease the navigation for core contributors. Though, the core team does not have the resources for now to do that, as our short/mid-term roadmap is already overloaded. Some of our contributors have documented some parts, but those docs are scattered around, and in multiple languages, not necessarily in English.

I am also taking this opportunity to address those points from a higher perspective. Given:

* the unusually large spectrum of the project for a programming language (stack of multiple languages, very rich standard library, cross-platform, self-contained,...),

* the bootstrapping process that would lead us to the aimed selfhosted Red in 2.0,

* the funding needs to sustain the project both in short/mid/long-terms,

* our business goals (in addition to the purely technical ones),

* expectations of low number of contributors until 1.0 (due to some unique aspects of our stack),

we have strived to find ways to go as fast as possible through the bootstrapping phase. So since the beginning, we have taken shortcuts, to implement the features needed for 1.0 as quickly as possible. That means less comments in the code (as the team is small and work at the same location), postponing the writing of many regression tests, postponing some design decisions and some edge cases processing, writing only minimal survival docs for the team and close contributors.

So, FYI, there is a "catch-up" stage planned between 0.9 and 1.0, where we will address those gaps, as we want a rock-solid and properly documented, 1.0. Moreover, when all design decisions are not set in stone, systematically testing/documenting all the features or code parts that may be dropped later is not winning move. In practice, regressions are rare in Red codebase, so the missing regression tests are not a show-stopper either, and we can live with that until 0.9.

[1] https://gist.github.com/dockimbel/2af078e02e56fcc092aea771de...

Thanks for responding politely to my somewhat accusatory comment.

It sounds like Red is driven by a small but very dedicated community, and the way it's both designed and promoted makes me think of TempleOS. Terry Davis' constant eulogies about how his own eccentric OS compares with Linux were obviously delusional (for a start, TempleOS doesn't do networking at all). Similarly comparing Red to Clojure when Red doesn't run on 64-bit machines is also misguided.

However, taken on its own terms TempleOS is an amazing piece of art, and I'm also sure that Red does something new and interesting. Perhaps that style of "PR bait" is just how you roll, but I am certainly glad you took your time to write a more down-to-earth description of the project. Reading that makes it easier for me to appreciate Red as more of an esoteric language that is of great use within its community.

I've followed Red for a long time, occasionally downloaded builds, and currently hold some of the token.

As of right now I have these thoughts:

* I think the focus on crypto could be a long shot, because my observation is that it's premised on capturing the notoriously fickle dev tools market...

* ...But in the end, Red has to get into a niche and own it and the team is, as some in this thread are suggesting, fanatical enough about what they're doing and confident in its strengths, that they are likely to go farther than anyone else had considered. Which is good enough to make me speculate on it.

* Although Red produces some great demos right now, the lineage of Rebol has its downsides too, since with a lot of these demos, when you get past the trivial and start customizing the UX in fine grain detail, a brick wall of unconventional paradigms is exposed which means that even if I'm experienced in other systems, I start needing a higher level of documentation and handholding. This is not an issue exclusive to Red and frequently impacts software libraries, but one way in which a lot of projects cheat on this problem is to emulate an existing API with lots of adoption and say, "Look, we're the same!" And then they don't have to document, they just steal the existing users.

* Regarding the pace of announcements, there's a rule of thumb Valve worked out for updates of Team Fortress 2 years ago(discussed in one of the articles or talks from then) that I see as more-or-less applicable to all software: Roughly every two months, make a large release with a detailed summary narrative. Project news is a great marketing tool when it's sized like this. Smaller releases tend to not make headlines, larger ones don't have a larger impact. So that's a proposed benchmark for what Red could be doing. The fact that this post blew up is a sign that something is going well, though :)

> encumbered by mainstream language syntax and development procedures which introduce an extraordinary volume of unnecessary complexity at many levels.

This clearly states about Clojure that its syntax AND it's procedure of development (another grand claim) introduce said complexity.

That whole write-up is just about as incorrect as it can get:

> Virtually every other well known development tool in popular use is built upon bloated layers of existing technology, [...] Most developers have just never experienced the sort of self dependent architecture,

Never mind that Red is actually implemented in another abandoned proprietary language[0]!

That entire sales pitch is a barefaced lie.

[0]: https://news.ycombinator.com/item?id=18846518

Sorry about that. Looked official to me and was linked to from the Red page and from your Github, but sorry about the mistake. s/creators/community/

Can you back up your acqusations with palpable evidence? Neither Red author, nor any of official project representatives "made a go" on Clojure or any other language and its community, as far as I'm concerned.

As for "defensive lashing" - you should make a distinction between informed, well-posed critique and arrogant, unbacked ranting. Everyone accepts the former and does not tolerate the latter - the team is no exception. And, again - either provide an evidence for what you say or back off your claims. Calling project megalomaniacal for fighting FUD is completely off base, and every criticism raised in this thread was handled politely. Heck, I can pull out any blog post from one of Clojure community members and start claiming that it represents opinions of Hickey. Would that be OK? I guess not.

Even if there is a cost in using Rebol2 as bootsrapping language, it will be gone with 1.0 release. And right now Rebol source code takes roughly a quarter of whole codebase (compilers and linkers for Red and Red/System) - the rest is pure Red and its dialects. So your whole paragraph is completely irrelevant and misleading: lack of 64-bit support has nothing to do with Rebol2, bold "non-free dependency" claim does not have any weight to it (since when bootstrapping language is a dependency?) and "SSL library" passage tells me that you don't have an idea of what you're talking about.

And use official resources (Red and Rebol websites) as references, would you? Wiki page is an extremely weak testimony to whatever you have to say.

No project is ideal, but Red's terms are completely different from what you try to impose here. I do not say that we don't have holes in the housing - we do, and we have all what is necessary to close them.

As has been noted by others, we have never "had a go" at Clojure. Language designers and the people who build them have our respect, even if our choices differ. If we criticize them (and we do at times, being human, and thinking we're right ;^), we try to be fair. Rich Hickey's talks are some of my favorite to watch, and I have nothing but respect and admiration for him.

Of course everything has a cost, but to call us megalomaniacs is a bit harsh.

You didn't say you understood the reason, but I pointed it out in another reply here. By bootstrapping from Rebol, and being 95% Rebol compatible, we proved that Red could self host, and also leverage almost all the existing code when we get to that point.

To ask if we understand those issues comes across as insulting. Sure, we could be tech smart but miss obvious things, but that one we know about. ;^)

Ah, so we were more or less on the same page from the beginning.

BTW, is there a child-safe dialect for Red? How easy or hard would it be to create an environment like Processing or p5.js, without the foot-shooting risks?

https://processing.org/

http://p5js.org/

> Red/Rebol might actually be too ad hoc

No offence, but very few people who raise such concerns actually take time to learn the language (or even launch a REPL at least once) and understand its design, so I appreciate you digging deeper. "Things are thrown against the wall" only in terms of constant search for a sound business model, and this, I believe, is the struggle that any startup (esp. programming languages) faces.

> Is there means to access the lexer directly as a means to explicate how, exactly, Red is interpreting?

I think you can start by, well, reading lexer code, which is written in Parse dialect [1], but specification I showed you might be more approachable. But really, just grab the latest build and start playing, I'll give some very basic examples below.

Now, to the main point: from what I know, Rebol (and Red) are based on research in denotational semantics that Carl Sassenrath did. I'll try to briefly explain the main points.

As you already know, everything starts with a UTF-8 encoded string. Each valid token in this string is converted to an internal data representation - a boxed structure, called a value slot or sometimes a cell.

Value slot is composed of a header and a payload. Header contains various flags and datatype ID, payload specifies exact content of the value. If content doesn't fit in one value slot, then payload contains a pointer to an external buffer (an array of value slots, bytes, or other units + offset and start/end addresses IIRC) with extra data.

So, lexer converts string representation to a tree of value slots (this phase is called "loading"), which is essentially a concrete syntax tree (CST) - this is the crux of homoiconicity.

  >> "6 * 7"
  == "6 * 7"
  >> type? "6 * 7"
  == string!
  >> load "6 * 7"
  == [6 * 7]
  >> type? load "6 * 7"
  == block!
  >> first load "6 * 7"
  == 6
  >> type? first load "6 * 7"
  == integer!
 

Everything is a (first-class) value, and every value has a datatype (we have roughly 50 of them right now). And there's no code - only this data structure, which is just a block, which you can freely manipulate at will (so as any other value).

  >> reverse [6 * 7]
  == [7 * 6]
  >> append reverse [6 * 7] [+ 1]
  == [7 * 6 + 1]
  >> skip append reverse [6 * 7] [+ 1] 2
  == [6 + 1]

What interpreter does is just a "walk" over this tree of values, dictated by a set of simple evaluation rules (expressions are evaluated left to right, operators take precedence over functions and have a more tight left side, literals evaluate to themselves, functions are applied to a fixed set of arguments, symbolic values of type "set-word!" [more on this later] are bound to the result of expression that follows them, etc) but there are a couple of catches.

The first catch is that some values are symbolic - that is, they indirectly refer to some other values via a context (namespace). You can modify this reference (called binding) freely at runtime, and thus change the meaning of symbolic values and of an entire block that contains them.

So, the "meaning" of a given block is always relative to some context(s) - this is what relative expression means (RE in REBOL). And context itself is just an environment of key/value pairs (key is a "symbol", value is its "meaning") represented as an object (O in REBOL).

  >> block: [6 * 7]             ; "block:" is a value of type "set-word!"
  == [6 * 7]
  >> type? second block
  == word!                      ; words are one of the symbolic values I've mentioned
  >> do block
  == 42
  >> bind block object [*: :+] ; now "multiplication" means "addition"
  == [6 * 7]
  >> do block
  == 13

The second catch is that you are not restricted by default interpreter (represented by "do" function) and can use any other one or even implement your own, thus making an embedded DSL - a dialect in Red/Rebol parlance.

* Red/System takes a block of C-level code and does the prescribed job.

* View takes a block that specifies GUI layout and shows a fancy window.

* Draw takes a block of drawing commands and renders an image.

* Parse takes an input series and a block of PEG grammar, and parses the input.

* Math takes a block and interprets it with common operator precedence rules.

Sky is the limit, and its dialects all the way down.

To reiterate: the basic building block (no pun intended) is a block of values, which can represent either code (relative expression which, upon evaluation, will yield a value) or data (just a bunch of values arranged in a specific format - such micro-formats are considered to be dialects too †). Block can also contain symbolic values (called words) which can change their binding during evaluation, and thus alter the semantics of expression.

There's a lot hiding behind the facade, as you can see. And what is there is hardly an ad-hoc hodge-podge slapped together.

[1]: https://github.com/red/red/blob/master/environment/lexer.red

(†): one example of such micro-format dialect is function specification, e.g.

  spec: [
      "Add two numeric values together"
      x [number!]
      y [number!]
  ]

is a specification (or an "interace") for a function that performs addition. Here's a block that expresses addition of two specific numbers:

  [1 + 2]

If we wish to abstract over it, we can substitute 1 and 2 for words:

  expression: [x + y]

And then we can alter bindings of these words to actual arguments we wish to add together:

   bind expression object [x: 1 y: 2]

We then can evaluate such expression and yield a resulting value:

   == do expression
   >> 3

The trick is that functions are just abstraction over evaluation of expression in some environment, that is, a syntax sugar for

  do bind [...] object [...]

with some additional optimizations and type-checking. So, addition instead can be expressed as:

  >> add: func spec expression
  == func [
    "Add two numeric values together" 
    x [number!] y [number!]
  ][x + y]
  >> type? :add ; ":add" is a value of type "get-word!" which, on evaluation, yields function's value referred by word "add" as-is, without triggering its application.
  == function!
  >> add 1 2
  == 3

Minecraft Java Edition is 100% Java, but it's also notoriously slow, and experiences lag spikes, which are often attributed to garbage collection. I think an OS written in a GC language can certainly have adequate latency and throughput, but I don't think Minecraft is a good example of that.