Validating variable names when writing data #46

Activity

1. evanmiller commented on Jan 31, 2016

   

   evanmiller

   on Jan 31, 2016

   Contributor

   More actions

   I am in favor of validating and sanitizing the variable names. One potential issue will be ensuring that the sanitized column names are unique within the data set (e.g. filter_$ and filter_% might both be squashed to filter_, which will wreak other kinds of havoc within Stata). So to do this properly we'll need an algorithm to assign unique names in the event of collisions.
2. wbuchanan commented on Feb 24, 2016

   

   wbuchanan

   on Feb 24, 2016

   Author

   More actions

   The simplest solution for those cases might be to append a numeric value instead of truncating the illegal characters. For example, filter_$ and filter_% would become filter_1 and filter_2 assuming that was the order in which they were stored in the underlying file. Not sure if C/C++ have any objects similar to a Set in Java, but that class only allows unique values of the collection (e.g., filter_ and filter_ would become filter_ and one of the instances would be lost), it might be helpful as a comparator for testing whether the number of unique variable names and the number of variable names parsed from the dataset are the same.
3. diogocp commented on Feb 26, 2016

   

   diogocp

   on Feb 26, 2016

   More actions

   For reference, here is what Stata does when reading CSV files:

   . type "stata_names_test.csv"
   filter_$,filter_%
   1,a
   2,b
   3,c
   
   . import delimited "stata_names_test.csv"
   (2 vars, 3 obs)
   
   . describe
   
   Contains data
     obs:             3                          
    vars:             2                          
    size:             6                          
   ------------------------------------------------------------
                 storage   display    value
   variable name   type    format     label      variable label
   ------------------------------------------------------------
   filter_         byte    %8.0g                 filter_$
   v2              str1    %9s                   filter_%
   ------------------------------------------------------------
4. evanmiller commented on Jun 21, 2016

   

   evanmiller

   on Jun 21, 2016

   Contributor

   More actions

   Addressed in 2baad22

   The library will now return an error when attempting to write an illegal column name -- it is the responsibility of the user to provide a legal / sanitized version of the column name.
5. evanmiller

   closed this as completedon Jun 21, 2016