Joe Armstrong: "In my opinion Erlang is brilliant at handling text"
groups.google.com
groups.google.com
Yes, regexes aren't part of the language syntax, but then that's not true for, say, python, either. Having to use a library to do regex is not going to be the difference maker in productivity for your app.
Besides, while I love regex, I still use them as a last resort. You're going to want to use a real parser for reliable structured text processing.
Generally, erlang programmers keep strings in binaries, which are compact. Most modules for handling string type tasks allow you to do this, e.g., the re module understands a unicode_binary type.
In 6 months of programming erlang professionally, in a domain dominated by scripting languages like python and ruby, I've certainly never been tempted to bolt over string handling issues. Erlang's flexible distribution, concurrency, and reliability model is just too compelling.
To get competitive performance in a reasonable amount of programmer time for concurrent applications in other languages you're limited to the subset of tasks that, say, twisted or tornado makes easy. The program I'm writing now couldn't have been done with either of them.
Frankly, if it's the choice between built-in support for regex and built-in primitives for distribution, concurrency, and fault tolerance, there's no question in my mind which is more important.
Another minor reason is to display; people prefer reading sequence of characters in a string syntax. If you have a statically typed language it is easy to display a list of chars in string syntax instead of list syntax. For a dynamically typed language with heterogeneous lists, it can be a performance penalty to check whether a list entirely consists of characters or not at runtime. So, in a sense, it is also about a performance. (Note: Having a syntax for strings has nothing to do with having distinct type for strings. The string syntax can be just a syntax sugar.)
But performance is important, of course. One thing very common in string (a list of characters) but not very common in general lists is concatenation. To be precise, lazy language programmers use list concatenation without a guilt, but eager language programmers tend to avoid it since it may cause unnecessary copying of lists. So for the eager evaluation languages, it makes sense to have a string type that has very cheap concatenation operation (e.g. using tree representation) internally.
In one world, you get an object of type [Char] ---means a list of characters--- and you can apply all sorts of list operations on it, and all sorts of operations specialized to [Char]. You can add a type alias String to the type [Char]. In the source file you can write "string" and it is read as a list of six characters. On the output the same list is printed as "string".
In another world, you get an object of type String, which is distinct type from a list of characters. Type String has all sorts of useful operations. But if you want to apply a generic list algorithm, you have to either duplicate the code, or coerce the string into a list. Conversely, if you have a list of characters and want to pass it to a string library function, say, regexp matcher, you have to coerce it to a string.
Which is easier?
As nostrademons commented, one way is to implement a generic interface so that you can write a generic algorithm on top of both list of characters and Strings, but that's actually the same thing I'm saying. I say "list" as some data structure on which you can peek the head, the tail, and you can add an element in front of existing one. I don't care how it is represented---if the runtime or the compiler can find out the list only contains ASCII characters, it can freely store the entire list in an octet array. In a sense, I say "list" as "data structures that implement the list interface".
Now, suppose if you have such a smart runtime/compiler. Suppose you can have specialized functions on [Char], apart from generic list. Do you still think having distinct string type is for ease of use?
In reality we don't have such sufficient smart runtime/compiler, so we compromise. That's the distinct string type.
I agree that conflating [Char] and [Integer] is not good. That's a different story.
I can't find the reasoning to back up your claim in your posts; if I miss it, could you point it? I think I explained a few points that a language does not need distinct string type, except from performance reasons.
Note that I've never said that strings shouldn't be a first class citizen. A list of characters is a first class citizen. You can have rich string library on top of lists of characters, plus generic list operators works on them. So, why do you want a string type disjoint from lists?
Is there a case where this still breaks down?
Except this falls apart when dealing with Unicode -- if your regex wants to match "ä", for example, treating strings as lists means you'll miss at least one possible way of representing that in Unicode (it may be a single code point, or it may be two -- an "a" with a combining diaresis).
Built-in string types can do things like warn that literal strings aren't convertible into target string type, at compile time; that an assignment from one string type to another may lose information, etc. Different string types may use different encodings, etc.
Delphi has multiple string types to handle all the backward compatibility issues. Ancient Pascal strings are limited to 255 characters; AnsiString (current code page), WideString (a COM BSTR), UnicodeString, and things like Utf8String (magic UTF-8 code page), etc. Assignments between the different strings perform conversions, and cause warnings for possible data-loss.
The more you learn about how strings work, including the international aspects, legacy aspects, OS-specific aspects, conversions at source code -> executable -> runtime -> I/O boundaries, etc., the more you appreciate the situation really isn't trivially reducible to simple lists of characters.
Some data, represented in lists, may have extra constraints. If elements have meaning in its order relative to each other, reversing such a list doesn't make sense. If every elements need to be power of 2, putting 3 into such list doesn't make sense. Yet, having constraints doesn't mean they need to be a distinct type from lists. You can implement them in a general lists and handle constraints with libraries. Why do strings need to differ? You don't want to reverse string character-by-character, or arbitrary indexing into a string may not make sense, but yet a general list operations like fold, filter, take-while, etc. may be useful.
Now, to avoid further confusion, let's separate CES and CCS. Utf-8, utf-16, ucs-4 are all CES variations of Unicode CCS. They can be converted freely without loss of information, and there are no reason that implementation can handle it implicitly, except performance issues.
If you need extra constraints, such as ascii-only or length-limited, you can create a specialized type wrapping the basic string and implement constraints there. That can be done in user-level and doesn't require such special types to be built-in. E.g. if a legacy library only accepts ascii-only string, it takes an argument of type AsciiOnly [Char]. Check can be enforced at type coercion.
Composed characters are headache, but the library to deal with them can be built on top of list of unicode codepoints (if we adopt unicode codepoints as Char). What's important here is that you can build some algorithm, say normalization, on top of list-of-char view. If you, as a language user, want to try out a new normalization scheme, you can do that, and you can have all the list manipulating tools in your hand. And you can wrap the resulting normalized string in a specialized type if you want to keep integrity.
Incompatible CCSes are more of a problem; e.g. there are several different conversions between Unicode and other Japanese character sets, and that have caused loss of information. I wish I can write CCS-neutral code, but eventually I need to deal with differences. This could be handled by having distinct UnicodeChar and JISChar, maybe.
Actually, there is a more fundamental question: What is a character? Depending on an application you may want different unit as a character. I think that's a good argument against "list-of-character" view. So another way is to have an opaque string type, which can show different level of abstraction depending on what an application asks (e.g. it may return graphemes, fully composed characters, unicode codepoints, etc.). That is plausible. Although I argue that it can be implemented as a library on top of list of basic characters (e.g. Unicode codepoints).
The way to get around that is to have some concept of interfaces in the language, and then have a generic Sequence type that strings, arrays, lists, and a bunch of other concrete types all implement. This makes your sequence operations even more polymorphic, eg. you can have them work on ropes, iterators, tree traversals, etc.
The real problem with the list representation for strings is memory consumption. English UTF-8 text in a byte array takes one byte per code point. English unicode text in a list where each code point is an immediate 32-bit int takes up 8 bytes per character. That's a factor of 8 difference in memory consumption.
And more memory means you can't fit as much into cache. There's a huge difference between being able to keep all your strings in L2 cache vs. having to go to main memory, particularly for string manipulations, which may need to touch a lot of different memory locations without a lot of locality.
"Some people emphasize importance of O(1) access of string access by index, but using integer index is also a performance hack. If search operations can return some way to point to the substring you don't need integer indexes."
Also, UTF-8 usually means you have to give up O(1) indexing. Your string may have multi-byte characters, which you can't discover unless you iterate through it.
In practice, this doesn't matter. The vast majority of string slices chop a few characters off from either the beginning or the end. You can do Boyer-Moore on byte sequences and return the byte offset instead of the character offset. Most other indexing operations involve some sort of iteration over the string anyway. The only real pathological case is when you need to find the midpoint of a string repeatedly.
For example, you might think that reversing a string is equivalent to reversing its characters. It's not.
Similarly, you might think that indexing a string is the same as indexing a list. It's not.
Etc etc.
I'm not sure what you mean by reversing a string not being equal to reversing its characters. It's certainly not always equal to reversing its bytes (depending on the encoding). Same goes for Strings that are implemented with UTF-16 characters (e.g. Java, win32 wchar_t etc.) all suffer from bad abstractions driven by implementation/efficiency concerns.
From a programming language perspective, I think it would be ideal if strings could always be viewed as lists of encoding independent characters, such that reversing this list is equivalent to reversing the string.
If you want to be able to maintain a one-to-one mapping to unicode, you will need to use unique characters for an "ä" and an "a with combining-diaeresis" but ideally that should be hidden from the user of the language.
Thus, in my ideal world, both "LATIN-SMALL-LETTER-O-WITH-DIAERESIS" and "LATIN-SMALL-LETTER-O COMBINING-DIAERESIS" would each be one single element of a list of characters which is a string.
foo\r\n\bar
rab\r\n\oofA string is more like a bytecode program that can be executed to evaluate to a piece of human-readable text.
I'm by no means an Erlang expert, nor have I ever written an editor; but I don't think it self-evident that, say, language support for regular expressions is a significant aid. (Automatic memory management may be helpful, but Erlang has that and it's not what we're talking about anyway.)
However my familiarity is limited and whether this is true or not I cannot comment on, but comparing erlang to C in performing text based operations is probably silly if the other person has an option such as say perl available to them.
C itself is bogged down by its own horrible string representation, the nul termination, where most operations need to traverse the string up to the terminator before they know where it ends. This causes all sorts of horrible buffering tricks, conditional tests on every character, and other unpleasant things. Pascal had length-prefixed strings from day one, and DJB created Netstrings to make network programming easier on people. See http://c2.com/cgi/wiki?LeasedString
As to snark - his reply was snark, I just replied in kind. Regexes are incredibly useful for all kinds of things, most especially in parsing data from web services. The fact that he didn't need them isn't 'funny,' it means we were doing different things.
In any case - using Erlang for something like this is so much win. The POE, and threaded implementations got real ugly real fast as we scaled it up. I knew Erlang could do it - across boxen, without a problem. It sounds like the regex libs have improved, and I look forward to using Erlang again in the future.
Happy? :D
Scroll to the bottom of the page http://common-lisp.net/project/cl-irregsexp/
If you have been following some recent papers on regexp performance, there is consensus that things could be a lot faster with better algorithms. I expect the game to change dramatically soon.
The benchmark compares the different implementations based a single trivial regular expression: /indecipherable|undecipherable/. You simply cannot claim a regex engine is faster than another with such a poor experiment. It is evident that Boyer–Moore string search algorithm will outshine any engine on that regular expression.
FWIW, if anybody can recommend a good benchmark, I would be happy to do a write up since I am proud of the regex performance of the other CL library (cl-ppcre.)
The majority of my professional programming is in Perl with many other programmers, and the number of times I see people do something like $settings =~ /read/ over a string containing settings, often complete with more than one setting that has the substring "read" even though they're only looking for one particular one... oi. Makes me sick.
Your point implies that exposing a powerful mechanism that is easily abused should be avoided. I disagree. I think this is a matter of culture and not "law".
Which is silly because if you make a list of 32 bit integers that have ASCII equivalents then Erlang assumes it is a string.
To Erlang, it is a list of 32 bit integers. There is no difference between a list of integers and a list of characters.
The shell will transform this list of 32 bit integers with ASCII equivalents into text for your convenience, but it doesn't do any sort of conversion or typing.
Sorry if I sound a bit pedantic, but there is a lot more to good Unicode support than using wide characters.