Tests don't handle non-UTF-8 locales well #7217

Activity

1. MichaelChirico

   added

   tests

   encodingissues related to Encodingissues related to Encoding

   on Jul 28, 2025
2. jangorecki commented on Sep 21, 2025

   

   jangorecki

   on Sep 21, 2025

   Member

   More actions

   Related PR: #7335
   Skips tests that are now failing on our vanilla test job.
3. MichaelChirico commented on Jul 27, 2026

   

   MichaelChirico

   on Jul 27, 2026

   Last edited by MichaelChirico

   Member

   More actions

   Related: tests 2298.* don't seem to be skipping as intended:

   Test 2298.1 produced 1 errors but expected 0
   Expected: 
   Observed: Column 1 ['x'] of item 2 is missing in item 1. Use fill=TRUE to fill with NA (NULL for list columns), or use.names=FALSE to ignore column names.
   Test 2298.2 produced 1 errors but expected 0
   Expected: 
   Observed: Column 1 ['ä'] of item 2 is missing in item 1. Use fill=TRUE to fill with NA (NULL for list columns), or use.names=FALSE to ignore column names.
   Test 2298.3 ran without errors but failed check that x equals y:
   > x = rbindlist(list(x, y), use.names = TRUE, fill = TRUE) 
       ä  ö  ü  x  y  z
   1:  1  2  3 NA NA NA
   2: NA NA NA  4  5  6
       [Key= Types=dou,dou,dou,dou,dou,dou Classes=num,num,num,num,num,num]
   1:                                                                      
   2:                                                                      
   > y = data.table(ä = c(1, NA), ö = c(2, 4), ü = c(3, 5)) 
       ä ö ü  [Key= Types=dou,dou,dou Classes=num,num,num]
   1:  1 2 3                                              
   2: NA 4 5                                              
   Different number of columns
   Different column names
   Test 2298.4 ran without errors but failed check that x equals y:
   > x = rbindlist(list(y, x), use.names = TRUE, fill = TRUE) 
       x  y  z  ä  ö  ü
   1:  4  5  6 NA NA NA
   2: NA NA NA  1  2  3
       [Key= Types=dou,dou,dou,dou,dou,dou Classes=num,num,num,num,num,num]
   1:                                                                      
   2:                                                                      
   > y = data.table(ö = c(4, 2), ü = c(5, 3), ä = c(NA, 1)) 
      ö ü  ä  [Key= Types=dou,dou,dou Classes=num,num,num]
   1: 4 5 NA                                              
   2: 2 3  1                                              
   Different number of columns
   Different column names
   
   Mon Jul 27 22:27:25 2026  endian==little, sizeof(long double)==16, capabilities('long.double')==TRUE, longdouble.digits==64, sizeof(pointer)==8, TZ=='UTC', Sys.timezone()=='UTC', Sys.getlocale()=='LC_COLLATE=C;LC_CTYPE=English_United States.1252;LC_MONETARY=English_United States.1252;LC_NUMERIC=C;LC_TIME=English_United States.1252', l10n_info()=='MBCS=FALSE; UTF-8=FALSE; Latin-1=TRUE; codepage=1252; system.codepage=1252', getDTthreads()=='OpenMP version (_OPENMP)==201511; omp_get_num_procs()==4; R_DATATABLE_NUM_PROCS_PERCENT==unset (default 50); R_DATATABLE_NUM_THREADS==unset; R_DATATABLE_THROTTLE==unset (default 1024); omp_get_thread_limit()==2; omp_get_max_threads()==4; OMP_THREAD_LIMIT==2; OMP_NUM_THREADS==unset; RestoreAfterFork==true; data.table is using 2 threads with throttle==1024. See ?setDTthreads.', .libPaths()=='C:/Users/runneradmin/AppData/Local/Temp/RtmpqGmwcN/RLIBS_16507c6637cc','C:/R/library', zlibVersion()==1.2.11 ZLIB_VERSION==1.2.11
   Error in test.data.table() : 
     4 errors out of 12775. Search tests/tests.Rraw for test numbers 2298.1, 2298.2, 2298.3, 2298.4. Duration: 00:01:03 elapsed (54.7s cpu).
   

   https://github.com/Rdatatable/data.table/blob/5c91aadf7306fcb6d27fe97d4f6428e620b49a80/inst/tests/tests.Rraw#L20331

   See https://github.com/Rdatatable/data.table/actions/runs/30308520863/job/90118454982?pr=7832.

   It seems to be related to older versions of R: I only see the errors on Windows x {3.5,3.6,4.0,4.1}. It's somewhat surprising to see ä et al render correctly; on older R it doesn't (https://github.com/Rdatatable/data.table/actions/runs/30308520863/job/90118454967?pr=7832):

   Test 2298.1 produced 1 errors but expected 0
     Expected: 
     Observed: Column 1 ['x'] of item 2 is missing in item 1. Use fill=TRUE to fill with NA (NULL for list columns), or use.names=FALSE to ignore column names.
     Test 2298.2 produced 1 errors but expected 0
     Expected: 
     Observed: Column 1 ['�'] of item 2 is missing in item 1. Use fill=TRUE to fill with NA (NULL for list columns), or use.names=FALSE to ignore column names.
     Test 2298.3 ran without errors but failed check that x equals y:
     > x = rbindlist(list(x, y), use.names = TRUE, fill = TRUE) 
         �  �  �  x  y  z
     1:  1  2  3 NA NA NA
     2: NA NA NA  4  5  6
         [Key= Types=dou,dou,dou,dou,dou,dou Classes=num,num,num,num,num,num]
     1:                                                                      
     2:                                                                      
     > y = data.table(� = c(1, NA), � = c(2, 4), � = c(3, 5)) 
         � � �  [Key= Types=dou,dou,dou Classes=num,num,num]
     1:  1 2 3                                              
     2: NA 4 5                                              
     Different number of columns
     Different column names
   
4. MichaelChirico commented on Jul 28, 2026

   

   MichaelChirico

   on Jul 28, 2026

   Last edited by MichaelChirico

   Member

   More actions

   Gemini diagnosis roughly makes sense from the evidence

   • On Windows R ≤ 4.1 (MSVCRT / Windows-1252): R's parser on pre-UCRT Windows did not tag string literals in the 0x80–0xFF range as UTF-8; it tagged both "\u00e4" and enc2native("\u00e4") as Latin-1 byte 0xe4.
   • Later iconv(..., from = "UTF-8", to = "latin1") sets Latin-1 column names on y, then rbindlist() attempts to hash and match UTF-8 column names against Latin-1 column names.
   • translateCharUTF8 behavior was updated/improved in the UCRT effort for R 4.2.0

   I'm trying to reproduce & then improve utf8_check() on 3.5.0 devcontainer with encoding ISO-8859-1 now.
5. aitap

   linked a pull request that will close this issueGitHub Action for non-UTF-8 locales #7821on Jul 30, 2026
6. MichaelChirico

   closed this as completedin #7821on Aug 8, 2026