[kou]

GitHub: Fix GH-381

A max_field_size error skipped only the rest of the physical line. In parse_quotable_robust, the rest of a record that has a quoted field spanning lines was returned as the next row or raised another error. In parse_quotable_loose and parse_no_quote, the line was already read, so the next line was skipped too, and CSV#line raised TypeError when the error was in the first row.

Validate field sizes in emit_row instead. All parsers call it after reading the whole record, so nothing needs to be skipped and CSV#line returns the whole broken record.

If the record has another error after the long field, such as an unclosed quoted field, that error is raised instead, as it is without max_field_size.

The field_size_limit document said that it limits how much CSV reads ahead and prevents DoS attacks. It wasn't true even with the old line based parser: it read a whole line without limit and stopped reading only lines of an unclosed quoted field. The current parser doesn't stop reading at all. Remove it from the document and add a document for max_field_size that says it doesn't limit how much CSV reads.

Reported by William T. Nelson. Thanks!!!

[wtn]

With headers: true, if the header row has a field that's too long, the row after it becomes the header when the error is rescued.

Perhaps that error should be fatal.

[kou]

Good point. Let's work on it as a separated task.