Factors are not first-class citizens in R
win-vector.com
win-vector.com
Python's excellent h5py module and the official HDF5 tools like "h5dump" understand that a dataset is boolean if it has the type H5T_ENUM with two values "FALSE" (0) and "TRUE" (1).
But R works a different way: if you save a boolean vector from R to HDF5 (using the rhdf5 package), it will create a dataset of type H5T_STD_I32LE, which takes 4x the storage space. And if you read the boolean vector as understood by other tools, you will get factors. Then, the most amazing thing happens: R uses the string names of the enum ("FALSE" and "TRUE"), but remaps them to factors with 1 and 2. So if you do as.integer(h5read(...)) you will get a vector of ones and twos instead of zeros and ones. And of course R itself treats 1 and 2 as truthy values, so now your data is well and truly corrupted: all the elements are truthy!
That is: as.logical(as.integer(h5read(...))) always produces all TRUE values when the input is boolean written by a program other than R. This is a lovely source of bugs in real software.
I don't think R support for HDF5 is "limited," I think it's just poorly implemented in a few ways (another is that you need a separate package to get 64-bit integer support, which is another problem with R in general).
I have gotten used to its behavior but only because I use factors sparingly, having set stringsAsFactors=FALSE as my default. But reshape2::melt() requires a separate argument (factorsAsStrings) if you don't want automatic conversion, and with plyr::ldply() you can't prevent the index column conversion to a factor at all. So factors creep in periodically into my data frames and burns me every now and then.
Oh, wow, thanks for that. I generally prefer data.table to plyr, but I do use reshape2 of course so that's great to know.
https://stat.ethz.ch/pipermail/r-help/2007-February/125438.h...
I have adopted this approach as well, and for all the scripts I distribute I set options(stringsAsFactors=FALSE) at the top.
I'm going to look more into setting this option, but is it consumable by others? I write R that will be consumable by customers, presumably I can set this at the top of my R source files?
cat("options(stringsAsFactors=FALSE)\n",file="~/.Rprofile",append=TRUE)
at the R console.
And to the GP, adding that to consumable code, particularly a distributed package, is generally seen as impolite behavior. What happens if I load your package and then another one which assumes the standard default?
If you absolutely must mess with peoples options, the way to do it is capture the original state, modify the option, make your affected call and then restore the original options.
I do agree that for my purposes I generally would prefer that as the default but even after using R for ~15 years I don't see it as some huge burden to specify it each time.