foo,"bar,baz"
However, the tools will treat it as 3 columns: $ echo 'foo,"bar,baz"' | awk -F, '{print NF}'
3 foo,"bar,baz"
However, the tools will treat it as 3 columns: $ echo 'foo,"bar,baz"' | awk -F, '{print NF}'
3[1] Maybe someday they won't.
In my current workflow, I generate JSON from the excel files and use the really awesome JQ command (http://stedolan.github.io/jq/) to process
You can write one-off (or reuseable) filters in minutes.
lex/flex should be in every UNIX distribution that has a C compiler, but maybe that's changing.
If I'm working with a datafile where I expect the delimiter to be in one of the fields, there is something wrong.
This is one reason why I always work with tab delimited files. Having an actual tab character isn't very common in free-text fields, at least in the data that I work with. Commas on the other hand, are quite common. Why one would select a field separator that was common in your data is beyond me (I know it's historical).
Your data files might be different, in which case, maybe you should select a different field separator.
Otherwise, no, there is no work around. If you have to quote fields, then you can't use the normal unix command line tools that tokenize fields.
https://github.com/dbro/csvquote
csvquote allows UNIX tools to work properly with quoted fields that contain delimiters inside the data. It is a simple translation tool that temporarily replaces the special characters occurring inside quotes with harmless non-printing characters. You do it as a first step in the pipeline, then do the regular operations using UNIX tools, and the last step of of the pipeline restores those troublesome characters back inside the data fields.