Hacker News Reader

Comment by VoidWhisperer | Hacker News Reader

Just to clarify something from the article and the github repo code (sorry if this is readily explained in the repo, C code using AVX stuff is not something I have any experience with): does the SIMD version still handle things like escaped quotes inbetween quotes denoting a string in a field in the csv file? It isn't mentioned in the article

View on HN

The readme in the source code mentions RFC 4180 compliance.

This snippet of the code shows that it is correctly handled:

    // from init_tables()
    } else {
        // When in a quoted string, a quote char transitions to the escape state.
        st[S_QUOTED][q] = S_QUOTE_ESC;

        // By default, any character seen after the escape state's quote
        // will transition back to the unquoted state (ending the field).
        memset(st[S_QUOTE_ESC], S_UNQUOTED, 256);

        // CRITICAL OVERRIDE: If the character is another quote, it was an
        // escaped quote, so we go back to the quoted state.
        st[S_QUOTE_ESC][q] = S_QUOTED;
    }

Reply on news.ycombinator.com